Domain generalization gaze estimation method mixing cnn and transformer

Abstract

The application belongs to the field of computer vision, and provides a domain generalization gaze estimation algorithm combining CNN and Transformer, including the following steps: firstly, ResNest-50 and ViT double-flow feature extraction network are used for feature extraction, and the multi-layer fusion features of the ResNest-50 branch are used as the input of the ViT network to replace the use of a full-face image as the input; subsequently, in order to alleviate the dimension and semantic difference of the output features of the ResNest branch and the ViT branch, a feature fusion enhancement module (FFEM) is designed to fuse the output features of the double branches; then, a domain generalization method based on an adversarial strategy is proposed to improve the cross-domain performance of the model. An additional image reconstruction task is designed to conduct adversarial learning with the gaze estimation task, and a mutual information neural estimator (MINE) is used to calculate the mutual information of the feature spaces of the image reconstruction task and the gaze estimation task to decouple the gaze-independent features (individual appearance, etc.) and the gaze-related features; finally, a hybrid loss function is used for deep supervision training.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a domain generalization gaze estimation method that combines CNN and Transformer. Background Technology

[0002] The eyes are the windows to the soul, and gazing is a typical form of nonverbal expression, containing important clues for understanding human cognition and behavior. Gaze estimation, as a crucial research topic in computer vision, plays a vital role in human-computer interaction, assisted driving, education, and commercial applications.

[0003] Gaze estimation methods can be categorized into model-based and appearance-based methods. Model-based methods require specialized equipment (such as infrared cameras) for feature extraction, resulting in high costs and low accuracy, and are generally only suitable for laboratory environments. Appearance-based gaze estimation methods directly use the entire image as input and implicitly extract mapping functions that contribute to gaze estimation, typically requiring only a regular camera. Appearance-based gaze estimation is more robust to low-quality images or relatively complex scenes. In recent years, with the development of deep learning, researchers have applied convolutional neural networks (CNNs) to appearance-based gaze estimation. Thanks to the availability of many large public datasets, CNN-based gaze estimation methods have demonstrated good performance, but they still fall far short of meeting practical needs.

[0004] Depending on the input data, appearance-based gaze estimation can be categorized into: 1) methods using only eye region images; 2) methods using only face images; and 3) methods using a combination of face and eye images. Using eye region images for gaze estimation requires additional computational resources to isolate the eye region. In reality, eye region images are not always available due to the influence of extreme head poses. Therefore, most current work focuses on using full-face images for gaze estimation. Research shows that full-face images can provide richer information for gaze estimation. However, the generalization performance of the model is significantly affected by the introduction of irrelevant information (individual appearance, lighting, etc.). Addressing the problem of poor model generalization ability has been a hot research topic in this field. Researchers have proposed several domain adaptation methods. However, domain adaptation methods require target domain data, which is difficult to implement in practice.

[0005] With the successful application of Transformer in other areas of computer vision, researchers have also attempted to introduce Transformer into gaze estimation. Currently, in order to better utilize the advantages of both CNN and Transformer, hybrid CNN and Transformer methods are receiving increasing attention. However, two main problems remain: 1) Transformer has a large number of parameters; 2) The output features of CNN and Transformer have dimensionality and semantic differences. Summary of the Invention

[0006] The purpose of this invention is to alleviate the above-mentioned problems. It proposes a domain generalization gaze estimation method (HCTDG-Net) that combines CNN and Transformer to improve the accuracy and generalization ability of gaze estimation models.

[0007] Considering the difficulty in acquiring images of the eye region, this invention uses only facial images for gaze estimation. A two-stream network composed of ResNest-50 and VisionTransformer (ViT) is used for feature extraction. To reduce the number of parameters in the ViT branch, multi-layer features from the ResNest-50 branch are fused and used as input to the ViT network instead of using a full-face image as input. To mitigate the dimensionality and semantic differences between the output features of the ResNest and ViT branches, a Feature Fusion Enhancement Module (FFEM) is designed to fuse the output features of both branches. Finally, an adversarial strategy-based domain generalization method is designed to improve the model's cross-domain performance. An additional image reconstruction task is designed to perform adversarial learning with the gaze estimation task, and a Mutual Information Neural Estimator (MINE) is used to compute the mutual information of the feature spaces of the image reconstruction and gaze estimation tasks to decouple gaze-independent features from gaze-related features.

[0008] The present invention achieves its objective using the following technical solution:

[0009] 1. The domain generalization gaze estimation method of this invention, which combines CNN and Transformer, employs a ResNest-50 and ViT dual-stream feature extraction network. The input to the ViT network is not a direct full-face image, but rather a multi-layer fusion feature from the ResNest network (the fusion feature formed by concatenating and fusing the multi-scale feature maps from the four stages of ResNest-50 after 1×1 convolution and pooling operations).

[0010] 1.1) The ETH-XGaze(E), Gaze360(G), and MPIIFaceGaze(M) datasets were used as the training and testing datasets for the model. To verify the generalization performance of the model, two domain generalization tasks were designed: ETH-XGaze→MPIIFaceGaze(E→M) and Gaze360→MPIIFaceGaze(G→M);

[0011] 1.2) ResNest-50 is an improved version of ResNet-50. Like ResNet-50, it contains four residual blocks (conv2, conv3, conv4, conv5). These four residual blocks generate feature maps with dimensions (number of channels × length × width) of 256×56×56, 512×28×28, 1024×14×14, and 2048×7×7, respectively. Define the four feature maps as A1-A4, and the multi-layer feature fusion process can be represented as follows:

[0012]

[0013] 1.3) The feature map F generated after fusing multiple features is used as the input feature of the ViT branch. In this invention, the number of ViT layers is set to 6.

[0014] 2. To alleviate the dimensionality and semantic differences between the output features of the ResNest branch and the ViT branch, the Feature Fusion Enhancement Module (FFEM) is designed to fuse the output features of the two branches;

[0015] 2.1) First, reshape the ViT output features, then use a 1×1 convolution to align the number of channels, making its dimension consistent with the output features of the ResNest branch. Define the output features of the ResNest branch as f c The output feature of the reshaped ViT is defined as f v Then the fused features f f It can be represented as:

[0016]

[0017] The fusion process enables interaction between features and automatically learns a weight map to control the weights of features at different locations and levels, highlighting useful features;

[0018] 2.2) FFEM finally uses regular convolutions of different scales to form a multi-branch structure and cascades them to horizontally expand the width of the network, increase the receptive field of the network, and enhance feature expression.

[0019] 3. An adversarial strategy-based domain generalization method is proposed to improve the cross-domain performance of the model. An additional image reconstruction task is designed to perform adversarial learning with the gaze estimation task, and a mutual information neural estimator (MINE) is used to compute the mutual information of the feature spaces of the image reconstruction task and the gaze estimation task to decouple gaze-independent features (such as individual appearance) from gaze-related features;

[0020] 3.1) The ResNet50 backbone network is used as the encoder for the image reconstruction task, and the decoder consists of 5 upsampling modules. The feature space generated by the encoder is defined as f. r The image generated after the i-th upsampling module is defined as I. i (i = 1, 2, 3, 4, 5), then the decoding process can be represented as follows:

[0021]

[0022] 3.2) Individual appearance differences have a significant impact on the model's generalization ability. This invention addresses this by minimizing the intermediate features (f) in the image reconstruction task. r ) and the intermediate features of the CNN branch in the gaze estimation task (f c The mutual information between features is used to decouple features and improve the model's cross-domain performance. In this invention, a Mutual Information Neural Estimator (MINE) is used to calculate the mutual information between features. It is a neural network based on KL divergence and Donsker-Varadhan representation. The mutual information I(F) estimated by MINE is used to decouple features and improve the model's cross-domain performance. c ;F r This can be represented as:

[0023]

[0024] in, It is F c and F r The joint distribution of T θ It is a neural network MINE,F with parameter θ. c and F r f c and f r A random variable.

[0025] 4. The model of this invention comprises three sub-networks: a gaze estimation network, an image reconstruction network, and a mutual information estimation network. The loss functions of the three sub-networks are defined as follows: and Deep supervised training is performed using a hybrid loss function.

[0026] 4.1) Use Loss function to optimize the gaze estimation subnetwork:

[0027]

[0028] Where g is the real direction. It predicts the direction;

[0029] 4.2) The loss function for the image reconstruction task is pixel-level MSE. Furthermore, the eye region and relative position are crucial for gaze estimation. To prevent the loss of relevant information during adversarial learning, an attention map M is added to the loss function of the image reconstruction task:

[0030]

[0031] Where I is the original image, M is the restored image, and M is the attention map;

[0032] 4.3) In order to maximize f r and f c To better decouple attention-related and irrelevant features, this invention utilizes a mutual information loss function, which can be written as:

[0033]

[0034] Among them, according to technical solution 3.2, f r The intermediate features, f, represent the image reconstruction task. c This represents the intermediate features of the CNN branch in the gaze estimation task, F c and F r f c and f r random variable, T θ This represents a neural network MINE with parameter θ.

[0035] Beneficial effects:

[0036] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention proposes a domain-generalized gaze estimation method that combines CNN and Transformer, and designs a dual-stream feature extraction module, a feature fusion enhancement module, and a domain generalization module based on adversarial strategies, achieving the expected results in experiments. This invention effectively mitigates the impact of individual appearance on the generalization performance of the gaze estimation model and further improves the gaze estimation accuracy. Attached Figure Description

[0037] Figure 1 This is a flowchart of the invention.

[0038] Figure 2 This invention relates to a dual-stream feature extraction module.

[0039] Figure 3 This invention relates to a feature fusion enhancement module.

[0040] Figure 4 This is a test result diagram within the domain of this invention.

[0041] Figure 5 This is a diagram showing the cross-domain testing effect of the present invention.

[0042] Figure 6 This is a visualization of the effects of the invention. Detailed Implementation

[0043] The technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings in the examples of the domain generalization gaze estimation method combining CNN and Transformer. Furthermore, the described embodiments are merely some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0044] The flowchart framework of this invention is as follows Figure 1 As shown, the domain generalization gaze estimation method of the present invention, which combines CNN and Transformer, is described in detail below:

[0045] 1. For example Figure 2 As shown, the domain generalization gaze estimation method of this invention, which combines CNN and Transformer, employs...

[0046] The ResNest-50 and ViT dual-stream feature extraction network are used as inputs. The input of the ViT network does not directly use the full face image, but uses the multi-layer fusion features of the ResNest network (the fusion features formed by concatenating and fusing the multi-scale feature maps of the four stages of ResNest-50 after 1×1 convolution and pooling operations).

[0047] 1.1) The ETH-XGaze(E), Gaze360(G), and MPIIFaceGaze(M) datasets were used as the training and testing datasets for the model. To verify the generalization performance of the model, two domain generalization tasks were designed: ETH-XGaze→MPIIFaceGaze(E→M) and Gaze360→MPIIFaceGaze(G→M);

[0048] 1.2) ResNest-50 is an improved version of ResNet-50. Like ResNet-50, it contains four residual blocks (conv2, conv3, conv4, conv5). These four residual blocks generate feature maps with dimensions (number of channels × length × width) of 256×56×56, 512×28×28, 1024×14×14, and 2048×7×7, respectively. The four feature maps are defined as A1-A4. The multi-layer feature fusion process can be represented as follows:

[0049]

[0050] 1.3) The feature map F generated after fusing multiple features is used as the input feature of the ViT branch. In this invention, the number of ViT layers is set to 6.

[0051] 2. For example Figure 3 As shown, to alleviate the dimensionality and semantic differences between the output features of the ResNest branch and the ViT branch, the Feature Fusion Enhancement Module (FFEM) is designed to fuse the output features of the two branches.

[0052] 2.1) First, reshape the ViT output features, then use a 1×1 convolution to align the number of channels, making its dimension consistent with the output features of the ResNest branch. Define the output features of the ResNest branch as f c The output feature of the reshaped ViT is defined as f v Then the fused features f f It can be represented as:

[0053]

[0054] The fusion process enables interaction between features and automatically learns a weight map to control the weights of features at different locations and levels, highlighting useful features;

[0055] 2.2) FFEM finally uses regular convolutions of different scales to form a multi-branch structure and cascades them to horizontally expand the width of the network, increase the receptive field of the network, and enhance feature expression.

[0056] 3. For example Figure 1 As shown, an adversarial strategy-based domain generalization method is proposed to improve the cross-domain performance of the model. An additional image reconstruction task is designed to perform adversarial learning with the gaze estimation task, and a mutual information neural estimator (MINE) is used to compute the mutual information of the feature spaces of the image reconstruction task and the gaze estimation task to decouple gaze-independent features (such as individual appearance) from gaze-related features;

[0057] 3.1) The ResNet50 backbone network is used as the encoder for the image reconstruction task, and the decoder consists of 5 upsampling modules. The feature space generated by the encoder is defined as f. r The image generated after the i-th upsampling module is defined as I. i (i = 1, 2, 3, 4, 5), then the decoding process can be represented as follows:

[0058]

[0059] 3.2) Individual appearance differences have a significant impact on the model's generalization ability. This invention addresses this by minimizing the intermediate features (f) in the image reconstruction task. r ) and the intermediate features of the CNN branch in the gaze estimation task (f c The mutual information between features is used to decouple features and improve the model's cross-domain performance. In this invention, a Mutual Information Neural Estimator (MINE) is used to calculate the mutual information between features. It is a neural network based on KL divergence and Donsker-Varadhan representation. The mutual information I(F) estimated by MINE is used to decouple features and improve the model's cross-domain performance. c ;F r This can be represented as:

[0060]

[0061] in, It is F c and F r The joint distribution of T θ It is a neural network MINE,F with parameter θ. c and F r f c and f r A random variable.

[0062] 4. The model of this invention comprises three sub-networks: a gaze estimation network, an image reconstruction network, and a mutual information estimation network. The loss functions of the three sub-networks are defined as follows: and Deep supervised training is performed using a hybrid loss function;

[0063] 4.1) Use Loss function to optimize the gaze estimation subnetwork:

[0064]

[0065] Where g is the real direction. It predicts the direction;

[0066] 4.2) The loss function for the image reconstruction task is pixel-level MSE. Furthermore, the eye region and relative position are crucial for gaze estimation. To prevent the loss of relevant information during adversarial learning, an attention map M is added to the loss function of the image reconstruction task:

[0067]

[0068] Where I is the original image, M is the restored image, and M is the attention map;

[0069] 4.3) In order to maximize f r and f c To better decouple gaze-related and gaze-irrelevant features, this invention utilizes a mutual information loss function, which can be written as:

[0070]

[0071] According to operating instructions 3.2, f r The intermediate features, f, represent the image reconstruction task. c This represents the intermediate features of the CNN branch in the gaze estimation task, F c and F r f c and f r random variable, T θ This represents a neural network MINE with parameter θ.

[0072] 5. To fully demonstrate the accuracy and generalization ability of the proposed method, this invention qualitatively and quantitatively compares the proposed hybrid CNN and Transformer domain generalization gaze estimation method with some existing methods, such as... Figure 4 , Figure 5 , Figure 6 As shown.

[0073] The above description is for preferred embodiments of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A domain generalization gaze estimation method combining CNN and Transformer, characterized in that the method comprises the following steps: 1) The domain-generalized gaze estimation method that combines CNN and Transformer employs a ResNest-50 and ViT dual-stream feature extraction network. The input to the ViT network is not a full-face image, but rather a multi-layered fused feature formed by concatenating and pooling the multi-scale feature maps from the four stages of ResNest-50. To mitigate the dimensionality and semantic differences between the output features of the ResNest and ViT branches, a Feature Fusion Enhancement Module (FFEM) is designed to fuse the output features of both branches. To improve the model's generalization performance and decouple gaze-independent features, an additional image reconstruction task is designed to perform adversarial learning with the gaze estimation task. A Mutual Information Neural Estimator (MINE) is used to minimize the mutual information between the feature spaces of the image reconstruction and gaze estimation tasks, thus achieving feature decoupling. Step 1) The specific method is as follows: 1.1) The ETH-XGaze(E), Gaze360(G), and MPIIFaceGaze(M) datasets were used as the training and testing datasets for the model. In order to verify the generalization performance of the model, two domain generalization tasks were designed: ETH-XGaze→MPIIFaceGaze(E→M) and Gaze360→MPIIFaceGaze(G→M). 1.2) The input to the ViT network comes from the fusion features of multi-layer features generated by the ResNest-50 network. ResNest-50 contains four residual blocks (conv2, conv3, conv4, conv5), which generate feature maps with dimensions (number of channels × length × width) of 256×56×56, 512×28×28, 1024×14×14, and 2048×7×7, respectively. These four feature maps are defined as follows: The multi-layer feature fusion process can then be represented as follows: (1) Feature map generated after fusing multiple feature layers The input features for the ViT branch are set to 6 layers. 1.3) The output features generated by the dual branches are fused using a Feature Fusion Enhancement Module (FFEM). The FFEM first reshapes the ViT output features, then uses a 1×1 convolution to align the number of channels, making its dimension consistent with the ResNest branch output features. The output features of the ResNest branch are defined as... The output features of the reshaped ViT are defined as follows: The features generated after fusion are defined as Therefore, the fusion process can be represented as: (2) The fusion process enables interaction between features and automatically learns weight maps to control the weights of features at different locations and levels, highlighting useful features. Finally, FFEM uses regular convolutions of different scales to form a multi-branch structure and cascades them to horizontally expand the width of the network, increase the receptive field of the network, and enhance feature representation. 1.4) Individual appearance and environmental differences significantly impact the model's generalization ability. To improve this ability, the proposed method employs an additional image reconstruction task to engage in adversarial learning with the gaze estimation task. The ResNet50 backbone network serves as the encoder for the image reconstruction task, and the decoder consists of five upsampling modules. The feature space generated by the encoder is defined as... After the first The image generated by the secondary upsampling module is defined as The decoding process can be represented as follows: (3) 1.5) The method described above minimizes the intermediate features of the image reconstruction task ( ) and intermediate features of the CNN branch of the gaze estimation task ( The mutual information between features is used to decouple features and improve the model's cross-domain performance. In this method, a Mutual Information Neural Estimator (MINE) is used to compute the mutual information between features. It is a neural network based on KL divergence and Donsker-Varadhan representation. MINE estimates the mutual information... It can be represented as: (4) in, yes and The joint distribution The parameter is The neural network MINE, and express and random variables; 2) The method comprises three sub-networks: a gaze estimation network, an image reconstruction network, and a mutual information estimation network. The loss functions of the three sub-networks are defined as follows: and Deep supervised training is performed using a hybrid loss function.

2. The domain generalization gaze estimation method combining CNN and Transformer as described in claim 1, characterized in that: The specific method for step 2) is as follows: 2.1) Use Loss function to optimize the gaze estimation subnetwork: (5) in It is the true direction. It predicts the direction; 2.2) The loss function for the image reconstruction task is pixel-level MSE. Furthermore, the eye region and relative position are crucial for gaze estimation. To prevent the loss of relevant information during adversarial learning, an attention map is added to the loss function of the image reconstruction task. : (6) in It is the original image. It is to restore the image. It is an attention map; 2.3) To maximize and The difference between them, namely, to better decouple attention-related and irrelevant features, is addressed by utilizing the mutual information loss function, which can be written as: (7) in Representing intermediate features for image reconstruction tasks, This represents the intermediate features of the CNN branch in the gaze estimation task. and express and random variables, The parameter is The MINE neural network.

Images