Sight line estimation method and device, electronic equipment and storage medium

Abstract

The invention provides a sight line estimation method and device, electronic equipment and a storage medium. The method comprises the following steps: determining a to-be-estimated face image and an eye image in the face image; feature extraction is carried out on the face image and the eye image, mixed attention interaction is carried out based on face features and eye features obtained through feature extraction, self-attention interaction is carried out based on the face features, and eye fusion features and face posture features are obtained; and performing sight line estimation based on the eye fusion features and the face posture features to obtain the sight line direction corresponding to the face image, thereby overcoming the defect that the accuracy of the estimation result is poor due to the fact that the current sight line estimation is susceptible to interference of various factors, deeply mining complementary information through mixed attention interaction, and realizing information complementation through complementary feature fusion. On the basis, sight line estimation is carried out, so that the maximum utilization of information can be realized, the fineness and accuracy of sight line estimation can be improved, and an accurate and reliable sight line direction can be obtained.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a line of sight estimation method, device, electronic device and storage medium. Background Art

[0002] Eye tracking, as an important branch of computer vision, has shown great potential in many fields such as virtual reality, game control, medical equipment operation, security systems, etc. It achieves non-contact interactive control by capturing and analyzing eye movements, thereby improving the interactive experience and convenience of operation.

[0003] There are two main types of gaze estimation schemes at present, namely eye input and face input. Eye input relies on the visual features of the eyes and estimates the gaze direction by capturing changes in the appearance of the eyes. However, this method is easily restricted by factors such as lighting and occlusion, which affects the stability and accuracy of the estimation. The face input scheme uses the face image as input and extracts the global features of the face to estimate the gaze, but it is also affected by factors such as expression and head posture, and the accuracy is poor. Summary of the invention

[0004] The present invention provides a line of sight estimation method, device, electronic device and storage medium, which are used to solve the defects of the prior art that line of sight estimation is easily disturbed, not comprehensive and accurate enough, achieve more precise line of sight estimation, and improve the accuracy of estimation.

[0005] The present invention provides a line of sight estimation method, comprising: Determining a face image to be estimated, and an eye image in the face image; Performing feature extraction on the face image and the eye image, and performing mixed attention interaction based on the face features and eye features obtained by the feature extraction, and performing self-attention interaction based on the face features to obtain eye fusion features and facial posture features; The line of sight is estimated based on the eye fusion feature and the facial posture feature to obtain the line of sight direction corresponding to the facial image.

[0006] According to a line of sight estimation method provided by the present invention, the feature extraction of the face image and the eye image is performed, and mixed attention interaction is performed based on the face features and eye features obtained by the feature extraction, and self-attention interaction is performed based on the face features to obtain eye fusion features and facial posture features, including: Performing image cropping on the face image and the eye image to obtain a plurality of face regions and a plurality of eye regions; Based on the feature extraction layer in the gaze estimation model, feature extraction is performed on each face region and each eye region to obtain features of each face region and each eye region; Based on the attention interaction layer in the gaze estimation model, mixed attention interaction is performed on the features of each face region and the features of each eye region, and self-attention interaction is performed on the features of each face region to obtain eye fusion features and facial posture features; The gaze estimation model is obtained by training based on the visual converter, sample face images and their corresponding sample gaze directions and sample face postures.

[0007] According to a sight line estimation method provided by the present invention, the attention interaction layer includes an eye branch and a face branch; The eye branch is used to determine a mixed attention parameter based on the features of each face region and the features of each eye region, and to determine the eye fusion feature based on the mixed attention parameter and the features of each eye region; The face branch is used to determine the self-attention parameter based on the features of each face region, and to determine the facial posture feature based on the self-attention parameter and the features of each face region.

[0008] According to a line of sight estimation method provided by the present invention, the line of sight estimation model is trained based on the following steps: Determining a sample face image and a sample eye image in the sample face image; Based on the initial feature extraction layer in the initial model, feature extraction is performed on the sample face image and the sample eye image to obtain sample face features and sample eye features; Based on the initial attention interaction layer in the initial model, performing mixed attention interaction on the sample facial features and the sample eye features, and performing self-attention interaction on the sample facial features, to obtain sample eye fusion features and sample facial posture features; Based on the initial gaze estimation layer in the initial model, applying the sample eye fusion features and the sample facial posture features to perform gaze estimation and facial posture prediction, and obtain a predicted gaze direction and a predicted facial posture corresponding to the sample facial image; Based on the predicted sight line direction and predicted facial posture corresponding to the sample facial image, and the sample sight line direction and sample facial posture corresponding to the sample facial image, the initial model is iterated with parameters to obtain the sight line estimation model.

[0009] According to a sight line estimation method provided by the present invention, the predicted sight line direction includes a first predicted sight line direction and a second predicted sight line direction; The initial sight line estimation layer in the initial model is based on the sample eye fusion feature and the sample facial posture feature to perform sight line estimation and facial posture prediction to obtain the predicted sight line direction and predicted facial posture corresponding to the sample facial image, including: Based on the line of sight estimation branch of the initial line of sight estimation layer, applying the sample eye fusion feature to perform line of sight estimation to obtain a first predicted line of sight direction; Based on the posture prediction branch of the initial line of sight estimation layer, applying the sample facial posture features to perform facial posture prediction to obtain a predicted facial posture; Based on the global prediction branch of the initial gaze estimation layer, the sample facial posture features and the sample eye fusion features are applied to perform gaze estimation to obtain a second predicted gaze direction.

[0010] According to a gaze estimation method provided by the present invention, the initial model is parameterized based on the predicted gaze direction and predicted facial posture corresponding to the sample facial image, and the sample gaze direction and sample facial posture corresponding to the sample facial image, to obtain the gaze estimation model, including: Determining a first loss based on the first predicted sight line direction and the sample sight line direction; determining a second loss based on the second predicted sight line direction and the sample sight line direction; determining a third loss based on the predicted facial pose and the sample facial pose; Based on the first loss, the second loss, and the third loss, parameters of the initial model are iterated to obtain the line of sight estimation model.

[0011] According to a line of sight estimation method provided by the present invention, the sample face image and its corresponding sample face posture, and the sample eye image in the sample face image are determined based on the following steps: Obtain a sample human body image; Performing face detection on the sample human body image, and determining a sample human face image from the sample human body image based on the face detection result; Performing facial key point detection on the sample facial image, and determining a sample eye image from the sample facial image based on the facial key point detection result; Facial gesture detection is performed based on the sample facial image to obtain a sample facial gesture corresponding to the sample facial image.

[0012] The present invention also provides a sight line estimation device, comprising: An image determination unit, used to determine a face image to be estimated, and an eye image in the face image; an attention interaction unit, configured to perform feature extraction on the face image and the eye image, and perform mixed attention interaction based on the face features and eye features obtained by the feature extraction, and perform self-attention interaction based on the face features to obtain eye fusion features and facial posture features; The sight line estimation unit is used to perform sight line estimation based on the eye fusion feature and the facial posture feature to obtain the sight line direction corresponding to the facial image.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, a line of sight estimation method as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the line of sight estimation method as described in any one of the above is implemented.

[0015] The line of sight estimation method, device, electronic device and storage medium provided by the present invention perform feature extraction on a face image and an eye image in the face image, and perform mixed attention interaction based on the face features and eye features obtained by the feature extraction, and perform self-attention interaction based on the face features to obtain eye fusion features and facial posture features, and perform line of sight estimation based on the eye fusion features and facial posture features to obtain the line of sight direction corresponding to the face image, thereby overcoming the defect that line of sight estimation in traditional schemes is easily interfered by various factors, resulting in poor accuracy of estimation results, deeply mining complementary information through mixed attention interaction, and realizing information complementarity through complementary feature fusion, and performing line of sight estimation on this basis, which can not only realize the maximum utilization of information, but also improve the precision and accuracy of line of sight estimation, and can obtain accurate and reliable line of sight direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 It is a flow chart of the sight line estimation method provided by the present invention; Figure 2 It is a structural schematic diagram of the initial model provided by the present invention; Figure 3 is an example diagram of key points of a face provided by the present invention; Figure 4is a structural schematic diagram of a sight line estimation device provided by the present invention; Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Eye tracking technology, as an important branch of computer vision, has shown great potential and value in many fields. For example, in the field of virtual reality, you can use your eyes instead of your hands to control games, thereby reducing fatigue; in the medical field, some patients have difficulty moving due to illness, in this case you can use your eyes instead of your hands to control the device; in the security field, you can rely on eye tracking technology to develop eye unlocking devices. Unlike iris recognition unlocking, users can customize their eye movement status to replace passwords for verification.

[0020] Current gaze estimation schemes are mainly divided into two categories: eye input and face input. Among them, the eye input scheme mainly relies on the visual features of the eyes and estimates the gaze direction by capturing subtle changes in the appearance of the eyes. This type of method usually uses a convolutional neural network to extract features from the grayscale image of the eye and perform gaze estimation based on this. However, relying solely on eye information may be limited by factors such as lighting and occlusion, affecting the stability and accuracy of the estimation.

[0021] In contrast, the facial input scheme extracts global features of the face to perform gaze estimation. This method can show better robustness and performance in complex scenes, but it will also be disturbed by factors such as expression changes and head posture, affecting the accuracy of the estimation.

[0022] In this regard, the present invention provides a line of sight estimation method, which aims to combine facial input and eye input, and perform line of sight estimation through a hybrid attention mechanism. It can make full use of the complementarity of eye and facial information to perform line of sight estimation, which not only realizes deep mining of information and maximizes the use of complementary information, but also improves the precision and accuracy of line of sight estimation, and can obtain more accurate and reliable line of sight estimation results.

[0023] Figure 1 is a flow chart of the sight line estimation method provided by the present invention, such as Figure 1 As shown, the method includes: Step 110, determining a face image to be estimated and an eye image in the face image; Step 120, extracting features from the face image and the eye image, and performing mixed attention interaction based on the face features and eye features obtained by the feature extraction, and performing self-attention interaction based on the face features to obtain eye fusion features and facial posture features; Step 130 , performing line of sight estimation based on the eye fusion features and the facial posture features to obtain the line of sight direction corresponding to the face image.

[0024] Specifically, considering the problems of low estimation precision and poor accuracy in both eye input and face input schemes in current line of sight estimation schemes, an embodiment of the present invention proposes to combine face input and eye input to jointly perform line of sight estimation. In this process, mixed attention MA (Mixed Attention) is used to effectively extract and integrate facial and eye features to achieve more precise prediction, which can not only maximize the use of facial and eye information for estimation, but also improve the accuracy and stability of line of sight estimation.

[0025] It is understandable that in actual application, before performing line of sight estimation, the first thing that needs to be determined is the image to be estimated, that is, the "face input" and the "eye input", which are referred to here as the face image to be estimated, and the eye image of the face image. The eye image here is the image of the eye area in the face image, which is cropped from the face image; specifically, it can be through facial key point detection on the face image, and the eye image is cropped from the face image according to the facial key point detection result; it can also be manually delineated in the face image. The eye area is cropped to obtain the eye image, or the area is located by eye features, the eye area is located and cropped, or it can be obtained by other methods, which are not specifically limited in the embodiments of the present invention.

[0026] Among them, the facial image can be a real-life image containing a face, or an image downloaded through a network search, or an image containing a face area searched from a database with pre-stored images, or an image containing a face obtained by other means, and the embodiments of the present invention do not specifically limit this.

[0027] As for the image containing a face that is initially obtained, in an embodiment of the present invention, it is usually necessary to further process it, for example, perform face detection on it to obtain the face area, and perform face key point detection to further obtain the eye area therein, so as to obtain a face image and a corresponding eye image.

[0028] Here, the face image corresponds to the eye image, and corresponding to the human body structure, the eye image may include a left eye image and a right eye image. Since the left and right eyes are not absolutely symmetrical and completely consistent, in an embodiment of the present invention, in order to ensure the accuracy of the estimation, the left eye image and the right eye image are selected to perform line of sight estimation at the same time to avoid the directional deviation caused by monocular estimation.

[0029] After determining the face image and the eye image, in an embodiment of the present invention, feature extraction can be performed on the face image and the eye image to deeply mine their respective information, and feature fusion can be performed through mixed attention to obtain complementary information of the face and the eye. Estimation can be performed based on this, which can not only maximize the utilization of information, but also improve the precision and accuracy of the estimation. That is, feature extraction is first performed on the face image and the eye image to extract rich facial information and eye information, thereby obtaining face features and eye features. Here, the eye image contains rich line of sight detail information, and the face image contains global information, including not only line of sight information, but also head posture information, facial attribute information, etc.; the face image can make up for the global information missing from the eye image, thereby improving the accuracy of subsequent estimation.

[0030] After that, attention interaction can be performed on the facial features and eye features to obtain eye fusion features and facial posture features; here, specifically, mixed attention interaction is performed on the facial features and eye features, and self-attention interaction is performed on the facial features, so as to perform feature fusion through mixed attention interaction and self-attention interaction respectively, so as to learn the information of the face and eyes, and explore the correlation and complementarity between the two, to obtain the fusion features of the eyes and the face, that is, the eye fusion features, and learn the unique head posture information of the face to obtain the head posture features, which can also be called facial posture features.

[0031] Here, in the process of mixed attention interaction, eye features can provide fine features centered on the line of sight, while facial features provide global features containing rich contextual information. Based on this, mixed attention interaction can deeply mine the complementary information between facial posture and eye line of sight, so that more accurate prediction can be made in the subsequent line of sight estimation process.

[0032] Furthermore, after obtaining independent, uncoupled, but complementary eye fusion features and facial posture features through attention interaction, in an embodiment of the present invention, the complementary features can be fused, and the direction of the line of sight can be finely and accurately predicted based on the fused features.

[0033] Specifically, here, the line of sight is estimated based on the eye fusion features and facial posture features obtained by attention interaction to obtain the line of sight direction corresponding to the face image. This process actually involves first fusing the features obtained by attention interaction, that is, fusing the eye fusion features and facial posture features to obtain the fused features; the feature fusion here realizes the complementarity of information, and the fused features can more completely and comprehensively reflect the line of sight, head posture and other information; feature fusion can be addition, splicing, weighted fusion, etc., which can be set according to actual conditions. Then, the line of sight can be estimated based on the fused features to estimate the direction of the eye line of sight, thereby obtaining an accurate and reliable line of sight estimation result, that is, the line of sight direction corresponding to the face image.

[0034] Compared with the solution of separate eye input and face input, the embodiment of the present invention adopts face input and eye input for line of sight estimation, which can achieve better results. In addition, the hybrid attention mechanism and the self-attention mechanism are used for preliminary feature fusion, which can learn from two perspectives to learn head posture information and face and eye information respectively, and mine the complementary information between the two. On this basis, complementary feature fusion is performed and line of sight estimation is performed, which can greatly improve the precision and accuracy of line of sight estimation.

[0035] The line of sight estimation method provided by the present invention extracts features of a face image and an eye image in the face image, performs mixed attention interaction based on the face features and eye features obtained by the feature extraction, and performs self-attention interaction based on the face features to obtain eye fusion features and facial posture features, performs line of sight estimation based on the eye fusion features and facial posture features, and obtains the line of sight direction corresponding to the face image, thereby overcoming the defect that line of sight estimation in traditional schemes is easily interfered by various factors, resulting in poor accuracy of estimation results, deeply mines complementary information through mixed attention interaction, realizes information complementarity through complementary feature fusion, and performs line of sight estimation on this basis, which can not only realize the maximum utilization of information, but also improve the precision and accuracy of line of sight estimation, and can obtain accurate and reliable line of sight direction.

[0036] Based on the above embodiment, step 120 includes: Performing image cropping on the face image and the eye image to obtain multiple face regions and multiple eye regions; Based on the feature extraction layer in the gaze estimation model, feature extraction is performed on each face region and each eye region to obtain features of each face region and each eye region; Based on the attention interaction layer in the gaze estimation model, mixed attention interaction is performed on the features of each face region and each eye region, and self-attention interaction is performed on the features of each face region to obtain eye fusion features and facial posture features. Among them, the gaze estimation model is trained based on the visual converter based on sample face images and their corresponding sample gaze directions and sample facial postures.

[0037] Specifically, the above process of extracting features from face images and eye images, performing mixed attention interaction based on the extracted face features and eye features, and performing self-attention interaction based on face features can be implemented through a gaze estimation model. The specific process includes: First, the face image and the eye image can be cropped separately to divide the face image and the eye image into multiple blocks, thereby obtaining multiple image blocks corresponding to each other, each image block corresponds to an area in the original image. Therefore, the multiple image blocks obtained by division can also be called multiple image areas, that is, multiple face areas corresponding to the face image and multiple eye areas corresponding to the eye image can be obtained through image cropping.

[0038] The number of the face regions and eye regions obtained by the division can be 16 or 64. The specific number can be determined according to the actual situation, and the embodiment of the present invention does not specifically limit this. However, the number of the face regions and the eye regions is the same, for example, the face regions, the left eye regions, and the right eye regions are all 8*8.

[0039] Here, the process of image cropping of face images and eye images can be implemented through a gaze estimation model, or it can be implemented through other methods, algorithms, etc. before processing with the help of the gaze estimation model, and then the cropped face area and eye area are input into the gaze estimation model for feature extraction and attention interaction.

[0040] Subsequently, the gaze estimation model can be used to extract features from multiple face regions and multiple eye regions to obtain regional features corresponding to each face region and regional features corresponding to each eye region, i.e., face region features and eye region features. In an embodiment of the present invention, the gaze estimation model includes multiple layers, and here, feature extraction is performed on each face region and each eye region through a feature extraction layer to obtain features of each face region and each eye region.

[0041] Here, when the feature extraction layer in the gaze estimation model performs feature extraction, it first performs linear projection, that is, each face area and each eye area is input into the linear projection layer in the feature extraction layer to perform feature mapping on the input, so as to obtain the features corresponding to each face area and each eye area; then, the position code and class token can be added, that is, the class token idea in the visual converter ViT (Vision Transformer) is borrowed to add the respective position codes and class tokens to the features of each face area and each eye area obtained by the feature mapping, so as to obtain the face area features corresponding to each face area, and the eye area features corresponding to each eye area.

[0042] After that, the attention interaction can be continued through the gaze estimation model. Specifically, through the attention interaction layer of the model, mixed attention interaction is performed on the features of each face region and each eye region, and self-attention interaction is performed on the features of each face region to obtain eye fusion features and facial posture features. In this process, the attention interaction of both angles is constrained, so that only the head posture information is paid attention to during self-attention interaction, and not only the eye gaze information but also the head posture information is paid attention to during mixed attention interaction, so as to explore the correlation and complementarity between the two as much as possible, thereby obtaining facial posture features and eye fusion features.

[0043] However, it is worth noting that before applying the line of sight estimation model for feature extraction and attention interaction, it is necessary to pre-train the line of sight estimation model. Here, specifically, the line of sight estimation model can be obtained by pre-training using sample face images and labels corresponding to the sample face images. However, taking into account the limitations of single eye input and facial input, in an embodiment of the present invention, in the face of eye input combined with facial input, auxiliary tasks are introduced on the basis of the line of sight estimation task, so as to train the model in combination with the auxiliary tasks, thereby improving the performance of the model. The auxiliary tasks here can be head posture prediction tasks, facial feature point detection tasks, etc. Such auxiliary tasks can help the model to perform line of sight estimation, so that the model performs better on the line of sight estimation task and has better performance.

[0044] As a preference, out of consideration for the consistency between line of sight direction and head posture under normal circumstances, in an embodiment of the present invention, the head posture prediction task is selected as an auxiliary task, and the model is trained in a multi-task manner, so that the model is more accurate in line of sight estimation.

[0045] Specifically, a large number of sample face images can be collected first, and the labels of the corresponding gaze estimation task, i.e., the sample gaze direction, can be obtained; at the same time, the labels of the auxiliary task, i.e., the sample head posture / sample face posture, need to be determined. Then, the initial model can be trained based on the sample face images, and the corresponding sample gaze directions and sample face postures, so as to obtain a model, which is the gaze estimation model.

[0046] Here, the initial model can be built on the basis of the visual transformer ViT. Specifically, the architecture of the visual transformer ViT is borrowed to improve the transformer, and a hybrid attention mechanism is used for attention interaction to better capture and fuse features, thereby obtaining more representative, complete and comprehensive features.

[0047] Based on the above embodiment, the attention interaction layer includes an eye branch and a face branch; The eye branch is used to determine a mixed attention parameter based on the features of each face region and each eye region, and to determine an eye fusion feature based on the mixed attention parameter and the features of each eye region; The face branch is used to determine the self-attention parameters based on the features of each face region, and to determine the facial posture features based on the self-attention parameters and the features of each face region.

[0048] Specifically, the attention interaction layer in the embodiment of the present invention has two branches, namely the eye branch and the face branch. The eye branch is used to perform mixed attention interaction on each face region feature and each eye region feature to obtain eye fusion features; the face branch is used to perform self-attention interaction on each face region feature to obtain facial posture features.

[0049] It is worth noting that in the process of attention interaction, the parameters used in the attention interaction of the face branch, namely the self-attention parameters Q, K and V (query, key value and value item), only come from the face region feature face token, so the face branch performs self-attention, that is, it is expected that the face branch only pays attention to the head posture information. Correspondingly, the mixed attention parameters of the eye branch come not only from the eye region features, but also from the face region features. Specifically, Q (query) comes from the eyes token, and K and V (key value and value item) come from the eyes token and the face token; so the eye branch performs mixed attention interaction, that is, it is expected that the eye branch not only pays attention to the eye sight information, but also can mine the complementary information with the face.

[0050] Based on the above embodiment, the line of sight estimation model is trained based on the following steps: Determine a sample face image and a sample eye image in the sample face image; Based on the initial feature extraction layer in the initial model, feature extraction is performed on the sample face image and the sample eye image to obtain sample face features and sample eye features; Based on the initial attention interaction layer in the initial model, mixed attention interaction is performed on the sample face features and sample eye features, and self-attention interaction is performed on the sample face features to obtain sample eye fusion features and sample face posture features; Based on the initial gaze estimation layer in the initial model, the sample eye fusion features and the sample facial posture features are used to perform gaze estimation and facial posture prediction, and the predicted gaze direction and predicted facial posture corresponding to the sample face image are obtained; Based on the predicted gaze direction and predicted facial posture corresponding to the sample face image, and the sample gaze direction and sample facial posture corresponding to the sample face image, the initial model is iterated on parameters to obtain a gaze estimation model.

[0051] Specifically, the training process of the sight line estimation model may include the following steps: First, it is necessary to determine the sample face image and the eye area in the sample face image, that is, the sample eye image. The sample face image here can be collected in advance, specifically, it can be obtained from an open source data set, and when obtaining the sample face image, the label of the corresponding gaze estimation task, that is, the sample gaze direction, can be determined simultaneously. The sample gaze direction can be manually annotated, or it can be specifically screened when obtaining the sample face image, and the image with the gaze direction label is selected, so that the sample face image and its corresponding sample gaze direction can be obtained at the same time.

[0052] At the same time, it is necessary to build the initial model used for training. Figure 2 It is a structural schematic diagram of the initial model provided by the present invention, such as Figure 2 As shown in the figure, the initial model is built on the architecture of the visual transformer ViT, specifically replacing the transformer in ViT with a mixed attention unit MAU (Mixed Attention Unit). Furthermore, considering that the current use of face input and eye input can produce better results than a single input, it is inevitable to encounter the problem of inconsistent head posture and line of sight when training the model, which not only affects the training speed, but also greatly reduces the training effect.

[0053] Based on this, in an embodiment of the present invention, when constructing the initial model, the model structure is further improved, specifically, the network head is improved, and a design of multiple network heads is adopted to correspond to different prediction tasks and obtain respective prediction results. In this way, decoupling between different tasks can be achieved, which can not only maximize the utilization of facial and eye information, but also decouple the learning of gaze direction and head posture, thereby implicitly modeling spatial geometric relationships.

[0054] In detail, when the model is trained based on a multi-task approach, namely, the head posture prediction task and the line of sight estimation task, when the head posture in the sample face image is consistent with the line of sight direction, the head posture can closely reflect the line of sight direction. In this case, the head posture can provide great help in line of sight estimation; on the contrary, in some cases, the head posture is not consistent with the line of sight direction, that is, although the head posture has significant changes, the line of sight direction remains unchanged, or the line of sight direction has obvious changes but the head posture is basically unchanged. At this time, the head posture is easy to mislead the line of sight estimation. In this regard, in an embodiment of the present invention, multiple network heads are selected to be set, and each network head adopts a double-layer multi-layer perceptron to correspond to different prediction tasks respectively, so as to avoid their mutual influence in the prediction process, resulting in inaccurate prediction results and poor model training effect.

[0055] After determining the initial model and the sample facial image, in the embodiment of the present invention, model training can be performed, that is, feature extraction is first performed using the feature extraction layer in the initial model. Since the feature extraction layer at this time has not been trained yet, it is called the initial feature extraction layer. Based on this, feature extraction is performed on the sample facial image and the sample eye image. Specifically, image cropping can be performed first, and then feature mapping is performed on each cropped image block, and position codes and category tokens are added to obtain corresponding sample facial features and sample eye features.

[0056] Next, the extracted sample facial features and sample eye features can be subjected to attention interaction, that is, the initial attention interaction layer is applied to perform mixed attention interaction on the sample facial features and sample eye features, and self-attention interaction is performed on the sample facial features. Here, specifically, the sample facial features and sample eye features are subjected to mixed attention interaction through the eye branch in the initial attention interaction layer, and the sample facial features are subjected to self-attention interaction through the face branch, so that the face branch only learns information related to the head posture, and the eye branch not only learns information related to the eye sight, but also learns complementary information between the head posture and the sight, and finally the sample eye fusion features (sample left eye fusion features and sample right eye fusion features) and sample facial posture features output by the two branches can be obtained.

[0057] Afterwards, the sample eye fusion features and the sample facial posture features can be used to perform gaze estimation and facial posture prediction. Specifically, on the basis of the sample eye fusion features and the sample facial posture features, the initial gaze estimation layer in the initial model is used to perform gaze estimation and facial posture prediction respectively, so as to obtain the predicted gaze direction and predicted facial posture corresponding to the sample face image; here, specifically, different network heads can be used for gaze estimation and facial posture prediction to predict the gaze direction and facial posture respectively, and the prediction results are output, thereby obtaining the predicted gaze direction and predicted facial posture.

[0058] Here, in the hybrid attention mechanism, the face branch only learns information related to the head posture through self-attention, and the eye branch not only learns information related to the eye gaze through mix-attention, but also learns complementary information between head posture and gaze, thus improving the accuracy of gaze estimation. On this basis, the dual supervision constraint mechanism of head posture and gaze estimation is explicitly added to decouple the features of the face branch and the eye branch, making the features more complementary, that is, decoupling the gaze estimation and head posture prediction tasks to obtain independent features, and fusing these decoupled features in a complementary way to achieve more precise predictions.

[0059] After that, the model training can be performed according to the predicted sight line direction and predicted facial posture output by the model, as well as the label of the sample face image. That is, according to the predicted sight line direction and predicted facial posture corresponding to the sample face image, as well as the sample sight line direction and sample facial posture, the parameters of the initial model are iterated, that is, according to the output prediction result and the label of the sample, the parameters of the initial model are adjusted so that the prediction result output by the model after the parameter adjustment is consistent with the label, or as close to the label as possible, and finally a trained initial model can be obtained, which is the sight line estimation model. Here, it should be noted that the label of the head posture prediction task corresponding to the sample face image, that is, the sample face posture, can be obtained by performing face posture detection on the sample face image, or can be obtained by manual annotation, or can be obtained by other methods, and the embodiment of the present invention does not make specific limitations on this.

[0060] Based on the above embodiment, the predicted sight line direction includes a first predicted sight line direction and a second predicted sight line direction; Based on the initial gaze estimation layer in the initial model, the sample eye fusion features and the sample facial posture features are used to perform gaze estimation and facial posture prediction, and the predicted gaze direction and predicted facial posture corresponding to the sample face image are obtained, including: Based on the gaze estimation branch of the initial gaze estimation layer, the sample eye fusion features are used to perform gaze estimation to obtain a first predicted gaze direction; Based on the posture prediction branch of the initial gaze estimation layer, the sample facial posture features are used to perform facial posture prediction to obtain the predicted facial posture; Based on the global prediction branch of the initial gaze estimation layer, the sample facial posture features and the sample eye fusion features are used to perform gaze estimation to obtain the second predicted gaze direction.

[0061] Specifically, the above-mentioned process of applying the initial line of sight estimation layer to perform line of sight estimation and facial posture prediction on the basis of the sample eye fusion features and the sample facial posture features can be specifically that, in an embodiment of the present invention, the line of sight estimation layer includes three branches, namely, a line of sight estimation branch, a posture prediction branch and a global prediction branch; here, based on the sample eye fusion features, the line of sight estimation branch is used to perform line of sight estimation to obtain a first predicted line of sight direction; based on the sample facial posture features, the posture prediction branch performs facial posture prediction to obtain a predicted facial posture; based on the sample facial posture features and the sample eye fusion features, the global prediction branch is used to perform line of sight estimation to obtain a second predicted line of sight direction.

[0062] Specifically, after MAU, the face image and the eye image have a corresponding feature Class Tokens, namely the sample eye fusion feature (sample left eye fusion feature and sample right eye fusion feature) and the sample face posture feature, which are recorded as , and , respectively represent the information of the entire image. In the embodiment of the present invention, a dual supervision constraint is designed through three network heads, and each head adopts a double-layer multi-layer perceptron structure. The dual supervision constraint includes supervision of the eye's line of sight and constraint of the head posture of the face.

[0063] Among them, the sight line estimation head / sight line estimation branch is spliced and The obtained features are used as input to estimate the line of sight and obtain the first predicted line of sight direction (including pitch angle and yaw angle), which is recorded as ; The pose prediction head / pose prediction branch only uses As input, facial posture prediction is performed to obtain the predicted facial posture (including pitch angle pitch and yaw angle yaw), which is recorded as ; Global supervision is applied to the third prediction head, which aggregates , and , that is, the global prediction branch is spliced , and The obtained features are used as input to predict the final sight direction, and the second predicted sight direction (including pitch angle and yaw angle) is output, which is recorded as .

[0064] It should be noted that three branches will be used for prediction during the model training process. However, after the training is completed, in actual applications, line of sight estimation is only based on the global prediction branch, and the result output by this branch is the final result of the application process.

[0065] Based on the above embodiment, based on the predicted sight line direction and predicted facial posture corresponding to the sample face image, and the sample sight line direction and sample facial posture corresponding to the sample face image, the initial model is iterated to obtain a sight line estimation model, including: Determining a first loss based on the first predicted sight line direction and the sample sight line direction; Determining a second loss based on the second predicted sight line direction and the sample sight line direction; determining a third loss based on the predicted face pose and the sample face pose; Based on the first loss, the second loss, and the third loss, the parameters of the initial model are iterated to obtain a line of sight estimation model.

[0066] Specifically, the process of iterating parameters of the initial model according to the predicted sight line direction and predicted facial posture corresponding to the sample facial image, and the sample sight line direction and sample facial posture corresponding to the sample facial image to obtain the sight line estimation model may specifically include: First, the loss measurement can be performed according to the first predicted sight line direction and the sample sight line direction, and according to the second predicted sight line direction and the sample sight line direction, respectively, to determine the loss of the model in the sight line estimation task, and obtain the first loss and the second loss. That is, the distance / closeness between the first predicted sight line direction and the sample sight line direction, and the distance / closeness between the second predicted sight line direction and the sample sight line direction are calculated, and the first loss and the second loss are determined accordingly.

[0067] At the same time, the loss of the model on the head word prediction task can be measured according to the predicted facial posture and the sample facial posture to obtain the third loss. That is, the distance / closeness between the predicted facial posture and the sample facial posture is calculated, and the third loss is determined accordingly.

[0068] Then, the first loss, the second loss and the third loss can be combined to determine the overall loss of the model. Specifically, the weighted sum of the three losses can be performed to obtain the overall loss. Based on the overall loss, the parameters of the initial model are iterated to obtain the line of sight estimation model.

[0069] In an embodiment of the present invention, in addition to adding a hybrid attention mechanism at the network design level to improve feature extraction and fusion, the loss function is explicitly constrained to force the face branch to only learn information related to head posture, while the eye branch can learn complementary information between head posture and line of sight. Finally, the decoupled features are spliced ​​to achieve more precise predictions.

[0070] Based on the above embodiment, the overall loss of the initial model can be expressed by the following formula: In the formula, and are the weights of the first loss and the third loss, both are 1, and They represent the sample gaze direction and sample face strip respectively.

[0071] Based on the above embodiment, the sample face image and its corresponding sample face posture, as well as the sample eye image in the sample face image are determined based on the following steps: Obtain a sample human body image; Performing face detection on the sample human body image, and determining a sample human face image from the sample human body image based on the face detection result; Performing facial key point detection on the sample face image, and determining a sample eye image from the sample face image based on the facial key point detection result; Facial gesture detection is performed based on the sample face image to obtain a sample face gesture corresponding to the sample face image.

[0072] Specifically, the sample face image, its corresponding sample face posture, and the sample eye image in the sample face image can be determined by the following steps: First, sample human body images can be obtained, which can be obtained from major public websites and open source data sets.

[0073] Next, face detection can be performed on the sample human image to detect the face area in the image, thereby obtaining the face detection result, which can include the face detection frame and the confidence level. After that, the face area can be determined from the sample human image according to the face detection result as the sample face image. Here, face detection of the sample human image can be achieved by the current mature face detection technology and face detection model; the sample face image obtained by detection can be scaled to a uniform size of 128*128.

[0074] Subsequently, facial key point detection can be performed on the sample face image to detect the facial key points, thereby locating the eye area in the face, thereby obtaining a sample eye image. Figure 3is an example diagram of key points of a face provided by the present invention, such as Figure 3 As shown, the face key point detection result can be obtained through face key point detection, that is, 68 key points in the face, of which the left eye and the right eye respectively contain six key points. Based on this, the eye area can be determined from the sample face image as a sample eye image.

[0075] The eye area crop process specifically includes: calculating the center point (mean) and long axis distance corresponding to the left eye key points 37-42 and the right eye key points 43-48 respectively, taking 1.5 times of the long axis distance as the side length and the center point as the center, cropping a square area, and uniformly scaling it to 128*128 to obtain sample eye images (sample left eye image and sample right eye image).

[0076] Afterwards, facial posture detection can be performed based on the sample face image to obtain the corresponding sample face posture. Here, 3DMM (3D Morphable Models) is used for facial posture detection to obtain the sample face posture corresponding to the sample face image.

[0077] Based on the above embodiment, the overall process of the sight line estimation method includes the following steps: First, a face image to be estimated and an eye image in the face image are determined.

[0078] Then, feature extraction is performed on the face image and the eye image, and mixed attention interaction is performed based on the face features and eye features obtained by the feature extraction, and self-attention interaction is performed based on the face features to obtain eye fusion features and facial posture features.

[0079] Here, feature extraction is performed on the face image and the eye image, and mixed attention interaction is performed based on the face features and eye features obtained by the feature extraction, and self-attention interaction is performed based on the face features to obtain eye fusion features and facial posture features, specifically including: image cropping is performed on the face image and the eye image to obtain multiple face areas and multiple eye areas; based on the feature extraction layer in the line of sight estimation model, feature extraction is performed on each face area and each eye area to obtain each face area feature and each eye area feature; based on the attention interaction layer in the line of sight estimation model, mixed attention interaction is performed on each face area feature and each eye area feature, and self-attention interaction is performed on each face area feature to obtain eye fusion features and facial posture features; wherein, the line of sight estimation model is trained based on sample face images and their corresponding sample line of sight directions and sample face postures on the basis of the visual converter.

[0080] Among them, the attention interaction layer includes an eye branch and a face branch; the eye branch is used to determine the mixed attention parameters based on the features of each face region and each eye region, and determine the eye fusion features based on the mixed attention parameters and the features of each eye region; the face branch is used to determine the self-attention parameters based on the features of each face region, and determine the facial posture features based on the self-attention parameters and the features of each face region.

[0081] Subsequently, gaze estimation is performed based on the eye fusion features and facial posture features to obtain the gaze direction corresponding to the face image.

[0082] Among them, the line of sight estimation model is trained based on the following steps: determining a sample face image and a sample eye image in the sample face image; based on the initial feature extraction layer in the initial model, performing feature extraction on the sample face image and the sample eye image to obtain sample face features and sample eye features; based on the initial attention interaction layer in the initial model, performing mixed attention interaction on the sample face features and the sample eye features, and performing self-attention interaction on the sample face features to obtain sample eye fusion features and sample facial posture features; based on the initial line of sight estimation layer in the initial model, applying the sample eye fusion features and the sample facial posture features to perform line of sight estimation and facial posture prediction to obtain the predicted line of sight direction and predicted facial posture corresponding to the sample face image; based on the predicted line of sight direction and predicted facial posture corresponding to the sample face image, and the sample line of sight direction and sample facial posture corresponding to the sample face image, performing parameter iteration on the initial model to obtain the line of sight estimation model.

[0083] Here, the predicted sight line direction includes a first predicted sight line direction and a second predicted sight line direction.

[0084] Among them, based on the initial line of sight estimation layer in the initial model, the sample eye fusion features and the sample facial posture features are used to perform line of sight estimation and facial posture prediction, so as to obtain the predicted line of sight direction and predicted facial posture corresponding to the sample face image, specifically including: based on the line of sight estimation branch of the initial line of sight estimation layer, the sample eye fusion features are used to perform line of sight estimation, so as to obtain the first predicted line of sight direction; based on the posture prediction branch of the initial line of sight estimation layer, the sample facial posture features are used to perform facial posture prediction, so as to obtain the predicted facial posture; based on the global prediction branch of the initial line of sight estimation layer, the sample facial posture features and the sample eye fusion features are used to perform line of sight estimation, so as to obtain the second predicted line of sight direction.

[0085] Furthermore, based on the predicted gaze direction and predicted facial posture corresponding to the sample facial image, and the sample gaze direction and sample facial posture corresponding to the sample facial image, the initial model is iterated on parameters to obtain a gaze estimation model, including: determining a first loss based on the first predicted gaze direction and the sample gaze direction; determining a second loss based on the second predicted gaze direction and the sample gaze direction; determining a third loss based on the predicted facial posture and the sample facial posture; and iterating on parameters of the initial model based on the first loss, the second loss, and the third loss to obtain a gaze estimation model.

[0086] Among them, the sample face image and its corresponding sample face posture, as well as the sample eye image in the sample face image are determined based on the following steps: obtaining a sample body image; performing face detection on the sample body image, and based on the face detection result, determining a sample face image from the sample body image; performing face key point detection on the sample face image, and based on the face key point detection result, determining a sample eye image from the sample face image; performing face posture detection based on the sample face image to obtain a sample face posture corresponding to the sample face image.

[0087] The method provided by the embodiment of the present invention performs feature extraction on a face image and an eye image in the face image, performs mixed attention interaction based on the face features and eye features obtained by the feature extraction, and performs self-attention interaction based on the face features to obtain eye fusion features and facial posture features, performs line of sight estimation based on the eye fusion features and facial posture features, and obtains the line of sight direction corresponding to the face image, thereby overcoming the defect that line of sight estimation in traditional schemes is easily interfered by various factors, resulting in poor accuracy of estimation results, deeply mines complementary information through mixed attention interaction, realizes information complementarity through complementary feature fusion, and performs line of sight estimation on this basis, which can not only realize the maximum utilization of information, but also improve the precision and accuracy of line of sight estimation, and can obtain accurate and reliable line of sight direction.

[0088] The line of sight estimation device provided by the present invention is described below. The line of sight estimation device described below and the line of sight estimation method described above can be referenced to each other.

[0089] Figure 4 is a schematic diagram of the structure of the sight line estimation device provided by the present invention, such as Figure 4 As shown, the device comprises: An image determination unit 410, configured to determine a face image to be estimated and an eye image in the face image; An attention interaction unit 420 is used to perform feature extraction on the face image and the eye image, and perform mixed attention interaction based on the face features and eye features obtained by the feature extraction, and perform self-attention interaction based on the face features to obtain eye fusion features and facial posture features; The sight line estimation unit 430 is used to perform sight line estimation based on the eye fusion feature and the facial posture feature to obtain the sight line direction corresponding to the facial image.

[0090] The line of sight estimation device provided by the present invention extracts features of a face image and an eye image in the face image, performs mixed attention interaction based on the face features and eye features obtained by the feature extraction, and performs self-attention interaction based on the face features to obtain eye fusion features and facial posture features, performs line of sight estimation based on the eye fusion features and facial posture features, and obtains the line of sight direction corresponding to the face image, thereby overcoming the defect that line of sight estimation in traditional schemes is easily interfered by various factors, resulting in poor accuracy of estimation results, deeply mines complementary information through mixed attention interaction, realizes information complementarity through complementary feature fusion, and performs line of sight estimation on this basis, which can not only realize the maximum utilization of information, but also improve the precision and accuracy of line of sight estimation, and can obtain accurate and reliable line of sight direction.

[0091] Based on the above embodiment, the attention interaction unit 420 is used to: Performing image cropping on the face image and the eye image to obtain a plurality of face regions and a plurality of eye regions; Based on the feature extraction layer in the gaze estimation model, feature extraction is performed on each face region and each eye region to obtain features of each face region and each eye region; Based on the attention interaction layer in the gaze estimation model, mixed attention interaction is performed on the features of each face region and the features of each eye region, and self-attention interaction is performed on the features of each face region to obtain eye fusion features and facial posture features; The gaze estimation model is obtained by training based on the visual converter, sample face images and their corresponding sample gaze directions and sample face postures.

[0092] Based on the above embodiment, the attention interaction layer includes an eye branch and a face branch; The eye branch is used to determine a mixed attention parameter based on the features of each face region and the features of each eye region, and to determine the eye fusion feature based on the mixed attention parameter and the features of each eye region; The face branch is used to determine the self-attention parameter based on the features of each face region, and to determine the facial posture feature based on the self-attention parameter and the features of each face region.

[0093] Based on the above embodiment, the device further includes a model training unit, which is used to: Determining a sample face image and a sample eye image in the sample face image; Based on the initial feature extraction layer in the initial model, feature extraction is performed on the sample face image and the sample eye image to obtain sample face features and sample eye features; Based on the initial attention interaction layer in the initial model, performing mixed attention interaction on the sample facial features and the sample eye features, and performing self-attention interaction on the sample facial features, to obtain sample eye fusion features and sample facial posture features; Based on the initial gaze estimation layer in the initial model, applying the sample eye fusion features and the sample facial posture features to perform gaze estimation and facial posture prediction, and obtain a predicted gaze direction and a predicted facial posture corresponding to the sample facial image; Based on the predicted sight line direction and predicted facial posture corresponding to the sample facial image, and the sample sight line direction and sample facial posture corresponding to the sample facial image, the initial model is iterated with parameters to obtain the sight line estimation model.

[0094] Based on the above embodiment, the predicted sight line direction includes a first predicted sight line direction and a second predicted sight line direction; The model training unit is used to: Based on the line of sight estimation branch of the initial line of sight estimation layer, applying the sample eye fusion feature to perform line of sight estimation to obtain a first predicted line of sight direction; Based on the posture prediction branch of the initial line of sight estimation layer, applying the sample facial posture features to perform facial posture prediction to obtain a predicted facial posture; Based on the global prediction branch of the initial gaze estimation layer, the sample facial posture features and the sample eye fusion features are applied to perform gaze estimation to obtain a second predicted gaze direction.

[0095] Based on the above embodiment, the model training unit is used to: Determining a first loss based on the first predicted sight line direction and the sample sight line direction; determining a second loss based on the second predicted sight line direction and the sample sight line direction; determining a third loss based on the predicted facial pose and the sample facial pose; Based on the first loss, the second loss, and the third loss, parameters of the initial model are iterated to obtain the line of sight estimation model.

[0096] Based on the above embodiment, the model training unit is further used for: Obtain a sample human body image; Performing face detection on the sample human body image, and determining a sample human face image from the sample human body image based on the face detection result; Performing facial key point detection on the sample facial image, and determining a sample eye image from the sample facial image based on the facial key point detection result; Facial gesture detection is performed based on the sample facial image to obtain a sample facial gesture corresponding to the sample facial image.

[0097] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the sight line estimation method, which includes: determining a face image to be estimated and an eye image in the face image; performing feature extraction on the face image and the eye image, and performing mixed attention interaction based on the face features and eye features obtained by feature extraction, and performing self-attention interaction based on the face features to obtain eye fusion features and facial posture features; performing sight line estimation based on the eye fusion features and the facial posture features to obtain the sight line direction corresponding to the face image.

[0098] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0099] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the line of sight estimation method provided by the above-mentioned methods, and the method includes: determining a face image to be estimated, and an eye image in the face image; performing feature extraction on the face image and the eye image, and performing mixed attention interaction based on the face features and eye features obtained by the feature extraction, and performing self-attention interaction based on the face features to obtain eye fusion features and facial posture features; performing line of sight estimation based on the eye fusion features and the facial posture features to obtain the line of sight direction corresponding to the face image.

[0100] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the line of sight estimation method provided by the above-mentioned methods, the method comprising: determining a face image to be estimated, and an eye image in the face image; performing feature extraction on the face image and the eye image, and performing mixed attention interaction based on the face features and eye features obtained by the feature extraction, and performing self-attention interaction based on the face features to obtain eye fusion features and facial posture features; performing line of sight estimation based on the eye fusion features and the facial posture features to obtain the line of sight direction corresponding to the face image.

[0101] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0102] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A line of sight estimation method, characterized in that: include: Determining a face image to be estimated, and an eye image in the face image; Performing feature extraction on the face image and the eye image, and performing mixed attention interaction based on the face features and eye features obtained by the feature extraction, and performing self-attention interaction based on the face features to obtain eye fusion features and facial posture features; The line of sight is estimated based on the eye fusion feature and the facial posture feature to obtain the line of sight direction corresponding to the facial image.

2. The line of sight estimation method according to claim 1, characterized in that: The step of extracting features from the face image and the eye image, performing mixed attention interaction based on the face features and eye features obtained through feature extraction, and performing self-attention interaction based on the face features to obtain eye fusion features and facial posture features includes: Performing image cropping on the face image and the eye image to obtain a plurality of face regions and a plurality of eye regions; Based on the feature extraction layer in the gaze estimation model, feature extraction is performed on each face region and each eye region to obtain features of each face region and each eye region; Based on the attention interaction layer in the gaze estimation model, mixed attention interaction is performed on the features of each face region and the features of each eye region, and self-attention interaction is performed on the features of each face region to obtain eye fusion features and facial posture features; The gaze estimation model is obtained by training based on the visual converter, sample face images and their corresponding sample gaze directions and sample face postures.

3. The line of sight estimation method according to claim 2, characterized in that: The attention interaction layer includes an eye branch and a face branch; The eye branch is used to determine a mixed attention parameter based on the features of each face region and the features of each eye region, and to determine the eye fusion feature based on the mixed attention parameter and the features of each eye region; The face branch is used to determine the self-attention parameter based on the features of each face region, and to determine the facial posture feature based on the self-attention parameter and the features of each face region.

4. The line of sight estimation method according to claim 2, characterized in that: The line of sight estimation model is trained based on the following steps: Determining a sample face image and a sample eye image in the sample face image; Based on the initial feature extraction layer in the initial model, feature extraction is performed on the sample face image and the sample eye image to obtain sample face features and sample eye features; Based on the initial attention interaction layer in the initial model, performing mixed attention interaction on the sample facial features and the sample eye features, and performing self-attention interaction on the sample facial features, to obtain sample eye fusion features and sample facial posture features; Based on the initial gaze estimation layer in the initial model, applying the sample eye fusion features and the sample facial posture features to perform gaze estimation and facial posture prediction, and obtain a predicted gaze direction and a predicted facial posture corresponding to the sample facial image; Based on the predicted sight line direction and predicted facial posture corresponding to the sample facial image, and the sample sight line direction and sample facial posture corresponding to the sample facial image, the initial model is iterated with parameters to obtain the sight line estimation model.

5. The line of sight estimation method according to claim 4, characterized in that: The predicted sight line direction includes a first predicted sight line direction and a second predicted sight line direction; The initial sight line estimation layer in the initial model is based on the sample eye fusion feature and the sample facial posture feature to perform sight line estimation and facial posture prediction to obtain the predicted sight line direction and predicted facial posture corresponding to the sample facial image, including: Based on the line of sight estimation branch of the initial line of sight estimation layer, applying the sample eye fusion feature to perform line of sight estimation to obtain a first predicted line of sight direction; Based on the posture prediction branch of the initial line of sight estimation layer, applying the sample facial posture features to perform facial posture prediction to obtain a predicted facial posture; Based on the global prediction branch of the initial gaze estimation layer, the sample facial posture feature and the sample eye fusion feature are applied to perform gaze estimation to obtain a second predicted gaze direction.

6. The line of sight estimation method according to claim 5, characterized in that: The step of performing parameter iteration on the initial model based on the predicted sight line direction and predicted facial posture corresponding to the sample facial image, and the sample sight line direction and sample facial posture corresponding to the sample facial image, to obtain the sight line estimation model comprises: Determining a first loss based on the first predicted sight line direction and the sample sight line direction; determining a second loss based on the second predicted sight line direction and the sample sight line direction; determining a third loss based on the predicted facial pose and the sample facial pose; Based on the first loss, the second loss, and the third loss, parameters of the initial model are iterated to obtain the line of sight estimation model.

7. The line of sight estimation method according to any one of claims 2 to 6, characterized in that: The sample face image and its corresponding sample face posture, and the sample eye image in the sample face image are determined based on the following steps: Obtain a sample human body image; Performing face detection on the sample human body image, and determining a sample human face image from the sample human body image based on the face detection result; Performing facial key point detection on the sample facial image, and determining a sample eye image from the sample facial image based on the facial key point detection result; Facial gesture detection is performed based on the sample facial image to obtain a sample facial gesture corresponding to the sample facial image.

8. A sight line estimation device, characterized in that: include: An image determination unit, used to determine a face image to be estimated, and an eye image in the face image; an attention interaction unit, configured to perform feature extraction on the face image and the eye image, and perform mixed attention interaction based on the face features and eye features obtained by the feature extraction, and perform self-attention interaction based on the face features to obtain eye fusion features and facial posture features; The sight line estimation unit is used to perform sight line estimation based on the eye fusion feature and the facial posture feature to obtain the sight line direction corresponding to the facial image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the line of sight estimation method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the line of sight estimation method according to any one of claims 1 to 7 is implemented.

Images