A gaze estimation method based on an adversarial network and an attention mechanism

Abstract

The application claims a gaze estimation method based on an adversarial network and an attention mechanism, which estimates the gaze direction through a feature fusion adversarial network FadNet, and the FadNet comprises two parts of an extraction network and a fusion network. An adversarial module is introduced in the extraction network, so that the related features of the gaze estimation are more accurately extracted by an adversarial optimization method; the fusion network can fuse the features of multiple channels, and the weights of the channels are adaptively distributed through an attention mechanism, so that a more accurate gaze direction is obtained. The method has better precision performance in unconstrained natural environment gaze estimation.

Description

Technical Field

[0001] This invention belongs to the field of gaze estimation research, and in particular, it is a gaze estimation method based on adversarial networks and attention mechanisms. Background Technology

[0002] The gaze estimation task, specifically defined, involves accurately predicting the three-dimensional direction vector of the gaze based on input image or video information, or directly determining the gaze point position on a two-dimensional plane. As an important branch of computer vision, this task not only encompasses the essence of deep learning and image processing but also integrates advanced machine learning techniques. Therefore, it has profound and undeniable significance for both academic research and practical applications.

[0003] The extraction of gaze information effectively opens a window into the psychological and physiological state of a target. By analyzing gaze information, we can infer various potential psychological states of the target, such as the distribution of attention and the process of cognitive behavior. This in-depth analysis and understanding further promotes the development of human-computer interaction technology, enabling us to interact with computers or other intelligent devices more naturally and efficiently. Simultaneously, gaze information plays a crucial role in many practical applications. For example, it can be used to detect driver fatigue, thereby ensuring driving safety; it can also assist doctors in disease diagnosis, improving the accuracy and efficiency of diagnosis.

[0004] Gaze estimation methods can be broadly categorized into two types based on their principles: model-based methods and appearance-based methods. Appearance-based methods directly utilize RGB or depth images as input, outputting the gaze direction through a trained regression function. In recent years, with the rapid development of deep learning and neural network technologies, appearance-based methods have achieved significant improvements in the accuracy of gaze estimation. More importantly, this method maintains high accuracy even in complex environments, such as under unconstrained conditions involving changing lighting conditions or large changes in head angle, giving it broader prospects and potential in practical applications. Therefore, appearance-based gaze estimation methods have attracted increasing attention and research from researchers.

[0005] The inventor's prior application CN114898453A discloses a gaze estimation method based on a cooperative network. This method includes the following steps: S1, preprocessing a face image, using the MTCNN algorithm for face detection and eye region localization, and extracting eye feature information; S2, estimating head pose using a spatial weight-based convolutional neural network; S3, constructing a cooperative network CI-Net using the eyes and face images as input; S4, using the CI-Net network for training to predict eye consistency; S5, using the predicted consistency to guide the inconsistency predicted by the CI-Net through a cross-attention module, thereby obtaining the gaze direction of both eyes.

[0006] This patent improves the accuracy of gaze estimation by selectively using input features, even when the eyes have different appearances or the face is partially occluded. However, this method selects relatively high-quality monocular images from both eyes and relies mainly on their extracted features to predict the gaze direction, suppressing and discarding other image parts, resulting in insufficient extraction of gaze-related information. Furthermore, it cannot select highly reliable input features when the entire face is simultaneously slightly occluded or affected by shadows. This method proposes an adversarial network for feature extraction, which does not rely primarily on a single eye or suppress low-quality images. Instead, it first removes redundant features from multiple channels simultaneously, maximizing the retention of gaze estimation-related features in the input image. Then, an attention module weights and fuses the multi-channel features. Even when multiple channels simultaneously input low-quality images in outdoor natural environments, this invention still maintains high robustness. Summary of the Invention

[0007] This invention aims to solve the problems of the prior art. It proposes a gaze estimation method based on adversarial networks and attention mechanisms. The technical solution of this invention is as follows:

[0008] A gaze estimation method based on adversarial networks and attention mechanisms includes the following steps:

[0009] S1. Facial image information is fed into the feature fusion adversarial network FadNet, which consists of two parts: an extraction network and a fusion network. The extraction network uses adversarial optimization to extract gaze estimation-related features more accurately, while the fusion network fuses features from multiple channels to obtain a more accurate gaze direction.

[0010] S2. In the extraction network, EyeCNN is used to extract features from the eyes, and FaceCNN is used to extract features from the face, so as to obtain preliminary features of the eyes and preliminary features of the face.

[0011] S3. Send the preliminary features of the eyes and the preliminary features of the face into the adversarial module, and after processing by the adversarial module, they become secondary features.

[0012] S4. The obtained secondary features are fed into the fusion network. The fusion network combines the three features through the cross-attention module to obtain the final gaze direction.

[0013] Furthermore, in step S2, EyeCNN is used to extract features from the eyes, and FaceCNN is used to extract features from the face, to obtain preliminary features from the eyes and the face, as detailed below:

[0014] The FaceCNN and EyeCNN networks extract preliminary features for the face and eyes, respectively. To obtain a larger receptive field, dilated convolutions are added to the VGG-Net base network for both CNNs. The output feature v of the dilated convolution is represented by the following formula:

[0015]

[0016] Where N×M×K is the kernel size, u(x,y) represents an element on the input feature map, m and n represent the positions in the kernel, k represents the input channel, w and b are the weights and biases respectively, and (r1,r2) is the dilation rate of the network. Both FaceCNN and EyeCNN networks use dilated convolution to improve the multi-channel architecture. The template image of both networks is set to 64×64. FaceCNN uses a higher dilation rate in the dilated convolution layer.

[0017] Furthermore, in step S3, the preliminary features of the face, eyes, and other three parts are sent to the adversarial module, where they are processed into secondary features, as follows:

[0018] Feature optimization is achieved indirectly using two adversarial loss functions. The adversarial module consists of a gaze estimator Dg(x; θ). g ) and an appearance estimator Da(x; θ) a The adversarial module consists of a gaze estimator Dg(·) and an appearance classifier Da(·) to optimize the feature extractor parameters so that it can obtain more robust features x`, and use x` for the final gaze estimation. Specifically, the gaze estimator and the appearance classifier each have their own optimization objectives. While controlling their own parameters, they must also counteract the adverse effects of parameter changes caused by the other. The optimization is implicitly achieved through the adversarial loss function.

[0019] The gaze estimator Dg(·) directly regresses the gaze direction and constructs a loss function based on the difference between the estimator and the true gaze direction. Its purpose is to estimate the gaze direction more accurately. The loss function is:

[0020]

[0021] Where yi The pitch and yaw angles represent the true values ​​in 3D space, x i E represents the initial features extracted by the dilated convolutional network, f is the learned feature representation, and E f The parameter is θ f Feature extractor, D g The parameter is θ g gaze estimator; n t ′ represents the number of test samples.

[0022] The appearance classifier Da(·) obtains a probability t between 0 and 1, which represents the likelihood that the input comes from the training domain. The loss function is:

[0023]

[0024] The appearance classification loss function is defined using the binary cross-entropy method, where Da(x; θ) a ) is a parameter of θ a An appearance classifier that outputs a scalar probability t, representing the probability that the input comes from the source domain, n. t n represents the number of test samples. s This represents the number of samples in the source domain. When the output probability of the classifier is close to 0.5, it indicates that the feature distribution p cannot be distinguished at all. x and source domain distribution p data The appearance classifier reaches the global optimum.

[0025] To enable the extracted features to more accurately estimate gaze distance and better distinguish the environment type, the loss functions of the two components are combined to obtain a joint loss function:

[0026]

[0027] Where λ a It is a positive balance factor; Let Dg(·) and Da(·) represent the loss functions of the gaze estimator and appearance classifier, respectively.

[0028] Since the two loss functions have different optimization objectives, they actually form an adversarial optimization relationship and together constitute an adversarial module.

[0029] Furthermore, in step S4, the obtained secondary features are fed into the fusion network. The fusion network combines the three features through a cross-attention module to obtain the final gaze direction, as detailed below:

[0030] The cross-attention module assigns different weights to input features based on their reliability. First, the features from both eyes pass through a self-attention sub-layer, then facial features are fused via residual connections. Finally, a cross-attention operation is performed between the features from both eyes. The cross-attention module can allocate weights between eye and facial features, selectively enhancing or suppressing input features. Based on a decoder and improved for gaze estimation tasks, the cross-attention module consists of two coupled feature fusion decoders, each comprising three sub-layers: a self-attention sub-layer, a cross-attention sub-layer, and a fully connected sub-layer. Facial features are added to the self-attention sub-layer, and the query values ​​of both eyes are swapped in the cross-attention sub-layer.

[0031] The attention function maps the input query, key, and value to the output. The weight of each value is obtained by processing its query and key through a dot product, division by the dimension parameter, and the softmax function. For two cross-attention modules, the corresponding eye feature is used as the key and value, and the other eye feature is used as the query.

[0032] Furthermore, for the two cross-attention modules, the corresponding eye feature is used as the Key and Value, and the other eye feature is used as the Query, as shown in the following formula:

[0033]

[0034]

[0035] Where K l K r V represents the key vector of the left and right eye features, respectively. l V r Let d represent the Value vectors of the left and right eye features, respectively. k K represents l K r The vector dimension is denoted by softmax(·), which indicates that the vector has been processed by the softmax function.

[0036] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements a gaze estimation method based on adversarial networks and attention mechanisms as described in any one of the claims.

[0037] A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the gaze estimation method based on adversarial networks and attention mechanisms as described in any one of the claims.

[0038] The advantages and beneficial effects of this invention are as follows:

[0039] The emergence of deep learning networks, represented by Convolutional Neural Networks (CNNs), has made it possible to estimate gaze using only input images. Furthermore, as the number of network layers increases, CNNs can extract more abstract, high-level features, further improving the accuracy of gaze estimation. However, in deep learning-based gaze estimation, models trained on large amounts of data often exhibit poor generalization performance, and overfitting is a major challenge limiting its applicability. When faced with unfamiliar environments or unseen individuals, the network's prediction accuracy drops significantly compared to the training environment. Current research typically employs methods such as rapid calibration to improve adaptability, but the results remain unsatisfactory.

[0040] Analysis revealed two main reasons limiting the model's generalization ability. First, it failed to remove invalid features relevant to the specific environment from all features. An ideal network should more accurately capture gaze-related factors in the input image; however, environmental features such as lighting conditions and background, and subject-related features such as the subject's appearance, lead to decreased accuracy when the usage scenario changes. Second, most current networks use only a single input, such as an eye image or a face image, or simply connect the two, failing to rationally fuse multiple extracted features. Research has shown that multi-feature fusion is beneficial for improving network robustness. Therefore, this invention proposes a Feature Fusion Adversarial Network (FadNet) to better extract effective features after removing environmental factors and organically fuse multiple features of the face and eyes to achieve better gaze estimation results. Attached Figure Description

[0041] Figure 1 This is an overall architecture diagram of the feature fusion adversarial network provided in a preferred embodiment of the present invention;

[0042] Figure 2 This is a structural diagram of the adversarial module in the extraction network provided by a preferred embodiment of the present invention;

[0043] Figure 3 This is a structural diagram of the cross-attention module in the fusion network provided by a preferred embodiment of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present invention.

[0045] The technical solution of the present invention to solve the above-mentioned technical problems is:

[0046] A gaze estimation method based on adversarial networks and attention mechanisms includes the following steps:

[0047] S1. The facial image information is fed into the feature fusion adversarial network FadNet, which consists of two parts: an extraction network and a fusion network.

[0048] S2. In the extraction network, EyeCNN and FaceCNN are used to extract facial features and eye features respectively to obtain preliminary features;

[0049] S3. The initial features of the face, eyes and other parts are sent to the adversarial module, and after processing by the adversarial module, they become secondary features.

[0050] S4. The obtained secondary features are fed into the fusion network. The fusion network combines the three features through the cross-attention module to obtain the final gaze direction.

[0051] 2. The gaze estimation method based on adversarial networks and attention mechanisms according to claim 1, characterized in that, in step S2, EyeCNN and FaceCNN are used respectively in the extraction network to extract facial features and eye features to obtain preliminary features, as follows:

[0052] The FaceCNN and EyeCNN networks extract preliminary facial and eye features respectively. To obtain a larger receptive field, dilated convolutions are added to the VGG-Net base network for both CNNs. Common methods to increase the receptive field include increasing the stride and using pooling layers, but these methods simultaneously reduce the feature map resolution, which can significantly negatively impact gaze estimation, a regression task. Dilated convolutions can maintain spatial resolution and keep the number of parameters from increasing significantly. Its output feature v can be expressed by the following formula:

[0053]

[0054] Where N×M×K is the kernel size, w and b are the weights and biases respectively, and (r1, r2) is the dilation rate of the network. Both FaceCNN and EyeCNN networks use dilated convolutions to improve multi-channel architecture. The template image for both networks is set to 64×64, and the backbone network is as follows: Figure 3 As shown, FaceCNN uses a higher dilation rate in its dilated convolutional layers.

[0055] 3. The gaze estimation method based on adversarial networks and attention mechanisms according to claim 1, characterized in that, in step S3, the preliminary features of the face, eyes, and other parts are fed into the adversarial module, and after processing by the adversarial module, they become secondary features, as follows:

[0056] In traditional supervised learning, features x are derived from the input image p. dataIn supervised learning, the goal is to extract features by inversely optimizing the loss function to obtain the feature x that minimizes the error. This paper uses two adversarial loss functions to indirectly optimize the features, such as... Figure 2 As shown, the adversarial module consists of a gaze estimator Dg(x; θ). g ) and an appearance estimator Da(x; θ) a The adversarial module does not directly use the feature x to estimate the gaze direction. Instead, it simultaneously uses the gaze estimator Dg(·) and the appearance classifier Da(·) to optimize the feature extractor parameters, enabling it to obtain more robust features x', which are then used for the final gaze estimation. Specifically, the gaze estimator and the appearance classifier each have their own optimization objectives. While controlling their own parameters, they must also counteract the adverse effects of parameter changes caused by the other. Optimization is implicitly achieved through an adversarial loss function approach.

[0057] The gaze estimator Dg(·) directly regresses the gaze direction and constructs a loss function based on the difference between the estimator and the true gaze direction. Its purpose is to estimate the gaze direction more accurately. The loss function is:

[0058]

[0059] Where y represents the pitch and yaw angles of the true values ​​in 3D space, f is the learned feature representation, and E f The parameter is θ f Feature extractor, E g The parameter is θ g A gaze estimator.

[0060] The appearance classifier Da(·) obtains a probability t between 0 and 1, which represents the likelihood that the input comes from the training domain. The loss function is:

[0061]

[0062] The appearance classification loss function is defined using the binary cross-entropy method, where Da(x; θ) a ) is a parameter of θ a The appearance classifier outputs a scalar probability t, representing the probability that the input comes from the source domain. When the output probability of the classifier is close to 0.5, it means that it cannot distinguish the feature distribution p at all. x and source domain distribution p data The appearance classifier reaches the global optimum.

[0063] To enable the extracted features to more accurately estimate gaze distance and better distinguish the type of environment, the loss functions of the two components are combined to obtain a joint loss function:

[0064]

[0065] Where λ a It is a positive balance factor.

[0066] Since the two loss functions have different optimization objectives, they actually form an adversarial optimization relationship and together constitute an adversarial module.

[0067] 4. The gaze estimation method based on adversarial networks and attention mechanisms according to claim 1, characterized in that, in step S4, the obtained secondary features are fed into a fusion network, and the fusion network combines the three features through a cross-attention module to obtain the final gaze direction, as follows:

[0068] In practical applications, due to factors such as facial angle, the features of both eyes often differ significantly. The role of the cross-attention module is to assign different weights to the input features based on their reliability. First, the features of each eye pass through a self-attention sub-layer, then facial features are fused through residual connections, and finally, a cross-attention operation is performed between the features of both eyes. Verification shows that the cross-attention module can allocate weights between eye features and facial features, selectively enhancing and suppressing input features. The Transformer architecture is a novel network architecture suitable for computer vision tasks. Researchers have demonstrated that integrating pure Transformers and traditional CNNs can achieve advanced performance. The cross-attention module in this method is based on a Decoder and improved for gaze estimation tasks. It consists of two coupled feature fusion Decoders, each comprising three sub-layers: a self-attention sub-layer, a cross-attention sub-layer, and a fully connected sub-layer. This paper incorporates facial features into the self-attention sub-layer and swaps the query values ​​of both eyes in the cross-attention sub-layer. The entire cross-attention module is shown in the attached figure. Figure 3 As shown.

[0069] The attention function maps the input Query, Key, and Value to the output. The weight of each Value is obtained by processing its Query and Key through a dot product, dividing by the dimension parameter, and then using the softmax function. For two cross-attention modules, this paper uses the corresponding eye feature as the Key and Value, and the other eye feature as the Query, as shown in the following formula:

[0070]

[0071]

[0072] Where K l K r V represents the key vector of the left and right eye features, respectively. l V rLet d represent the Value vectors of the left and right eye features, respectively. k K represents l K r The vector dimension is denoted by softmax(·), which indicates that the vector has been processed by the softmax function.

[0073] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0074] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0075] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0076] The above embodiments should be understood as illustrative only and not as limiting the scope of protection of the present invention. After reading the description of the present invention, those skilled in the art can make various alterations or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A gaze estimation method based on adversarial networks and attention mechanisms, characterized in that, Includes the following steps: S1. Facial image information is fed into the feature fusion adversarial network FadNet, which consists of an extraction network and a fusion network. The extraction network extracts gaze estimation-related features using adversarial optimization, and the fusion network fuses features from multiple channels to obtain a more accurate gaze direction. S2. In the extraction network, EyeCNN is used to extract features from the eyes, and FaceCNN is used to extract features from the face, so as to obtain preliminary features of the eyes and preliminary features of the face. S3. Send the preliminary features of the eyes and the preliminary features of the face into the adversarial module, and after processing by the adversarial module, they become secondary features. S4. The obtained secondary features are fed into the fusion network. The fusion network combines the three features through the cross-attention module to obtain the final gaze direction. Step S3 sends the preliminary features of the face, eyes, and other three parts to the adversarial module, which processes them into secondary features, as follows: Feature optimization is achieved indirectly using two adversarial loss functions. The adversarial module consists of a gaze estimator Dg(x; ) and an appearance estimator Da(x ; The adversarial module consists of a gaze estimator Dg(∙) and an appearance classifier Da(∙), which optimize the feature extractor parameters to obtain more robust features x`, and use x` for the final gaze estimation. The gaze estimator Dg(∙) directly regresses the gaze direction and constructs a loss function based on the difference between the estimator and the true gaze direction. The loss function is: ； in Represents the pitch and yaw angles as true values ​​in 3D space. This represents the initial features extracted by the dilated convolutional network, where f is the learned feature representation. The parameter is Feature extractor, The parameter is A gaze estimator; It is the number of test samples; The appearance classifier Da(∙) obtains a probability t between 0 and 1, which represents the likelihood that the input comes from the training domain. The loss function is: ； The appearance classification loss function is defined using the binary cross-entropy method, where Da(x ; ) is the parameter The appearance classifier outputs a scalar probability t, representing the probability that the input comes from the source domain. Indicates the number of test samples. This represents the number of samples in the source domain; when the output probability of the classifier is close to 0.5, it indicates that the feature distribution p cannot be distinguished at all. x and source domain distribution p data The appearance classifier reaches the global optimum. The joint loss function is obtained by combining the loss functions of the two components: ； in It is a positive balance factor; , Let Dg(∙) and Da(∙) represent the loss functions of the gaze estimator and appearance classifier, respectively. Since the two loss functions have different optimization objectives, they actually form an adversarial optimization relationship and together constitute an adversarial module.

2. The gaze estimation method based on adversarial networks and attention mechanisms according to claim 1, characterized in that, In step S2, EyeCNN is used to extract features from the eyes, and FaceCNN is used to extract features from the face, to obtain preliminary features of the eyes and the face, as detailed below: The FaceCNN and EyeCNN networks extract preliminary features for the face and eyes, respectively. To obtain a larger receptive field, dilated convolutions are added to the VGG-Net base network for both CNNs. The output feature v of the dilated convolution is represented by the following formula: ； Where N×M×K is the kernel size, u(x, y) represents an element on the input feature map, m and n represent the positions in the kernel, k represents the input channel, w and b are the weights and biases respectively, and (r1, r2) is the dilation rate of the network. Both FaceCNN and EyeCNN networks use dilated convolution to improve the multi-channel architecture. The template image of both networks is set to 64×64. FaceCNN uses a higher dilation rate in the dilated convolution layer.

3. The gaze estimation method based on adversarial networks and attention mechanisms according to claim 1, characterized in that, In step S4, the obtained secondary features are fed into the fusion network. The fusion network combines the three features through a cross-attention module to obtain the final gaze direction, as detailed below: The role of the cross-attention module is to assign different weights to the input features based on their reliability. First, the features of each eye are passed through a self-attention sub-layer, then facial features are fused through residual connections, and finally, a cross-attention operation is performed between the features of both eyes. The cross-attention module assigns weights to the eye features and facial features, selectively enhancing and suppressing the input features. The cross-attention module is based on the decoder and improves upon it for the gaze estimation task. It consists of two feature fusion decoders coupled together, each of which includes three sub-layers: self-attention sub-layer, cross-attention sub-layer, and fully connected sub-layer. Facial features are added to the self-attention sub-layer, and the query values ​​of the two eyes are swapped in the cross-attention sub-layer. The attention function is the process of mapping the input query, key, and value to the output. The weight of each value is obtained by processing its query and key through dot product, division by the dimension parameter, and softmax function. For two cross-attention modules, the corresponding eye feature is used as the key and value, and the other eye feature is used as the query.

4. The gaze estimation method based on adversarial networks and attention mechanisms according to claim 3, characterized in that, For the two cross-attention modules, the corresponding eye feature is used as the Key and Value, and the other eye feature is used as the Query, as shown in the following formula: ； ； Where K l K r V represents the key vector of the left and right eye features, respectively. l V r Let d represent the Value vectors of the left and right eye features, respectively. k K represents l K r The vector dimension is denoted by softmax(), which indicates that the vector has been processed by the softmax function.

5. An electronic device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the gaze estimation method based on adversarial networks and attention mechanisms as described in any one of claims 1 to 4.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the gaze estimation method based on adversarial networks and attention mechanisms as described in any one of claims 1 to 4.

Images