Gaze estimation and detection method based on attention cross and dual-path feature fusion network

Abstract

This invention belongs to the field of computer vision and provides a dual-path feature extraction network framework that combines global and local information. The framework includes the following steps: 1) A novel method is proposed to process gaze estimation in parallel using Convolutional Neural Networks (CNNs) and Transformers. The resulting architecture, CTA-Net, fully leverages the inductive bias of CNNs in modeling spatial correlations and the powerful ability of Transformers in modeling global relationships; 2) A complementary eye image aggregation gaze estimation attention mechanism, ACFusion, is proposed. This mechanism generates multiple attention feature maps, thereby achieving the aggregation and learning of attention feature information for binocular image interaction; 3) A novel fusion technique—the DFA module—is introduced to fully fuse feature information from different encoders, promote global information from Transformer branches, reduce noise that may be present in CNN features, and enable CTA-Net to better utilize the advantages of both models. Experimental data on publicly available datasets demonstrate the efficiency and superiority of this invention.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more specifically, to a gaze estimation and detection method based on attention crossover and dual-path feature fusion networks. Background Technology

[0002] The statements in this section only relate to the background technology in connection with this invention and do not necessarily constitute prior art.

[0003] Gaze estimation and detection techniques play a crucial role in computer vision, used to estimate the true gaze direction and locate the gaze point. These techniques primarily employ model-based and appearance-based approaches. Due to the development and application of deep learning, appearance-based methods have recently attracted widespread attention. They only require a standard RGB camera to capture images and directly learn the mapping function from facial appearance to human gaze. Because CNNs (Convolutional Neural Networks) demonstrate superior ability to learn highly complex mapping functions, some have proposed using CNNs to address the influence of various factors, such as head pose, on human eye appearance in the field.

[0004] In recent years, CNN-based gaze estimation methods have been booming. Since Zhang et al. first proposed a network to estimate gaze direction from eye images, CNN-based methods have paved the way for gaze estimation in everyday environments. However, these methods only use image information encoding from a single eye and ignore the auxiliary information brought by head pose, which cannot meet the actual needs.

[0005] Human gaze is closely related to information from the eyes. Eye images should be sufficient to estimate gaze direction, but deep learning-based methods can utilize additional information from other facial regions, such as head pose, to provide excellent guidance for gaze estimation. In fact, eye images provide local features focused on the gaze, while face images provide global features with coarse-grained details. Therefore, fully utilizing both facial and eye features to achieve more accurate gaze estimation is a significant challenge.

[0006] While CNNs offer significant performance improvements, gaze estimation methods for object detection always rely on dilated convolutions with varying padding rates to extract contextual information and infer the estimated direction. However, the introduction of dilated convolution padding rates leads to the loss of local information, and independent feature extraction methods result in a lack of correlation between contextual information. Therefore, it's difficult to guarantee the reliability of contextual information from different receptive fields, which is detrimental to gaze estimation detection across different datasets. Furthermore, how to utilize complementary information between multiple modalities is a key issue for accurate gaze estimation. Cross-attention mechanisms can achieve the fusion of eye features. Compared to fusing individual eye data, cross-fusion is more conducive to information complementarity. Summary of the Invention

[0007] To alleviate the aforementioned problems, we propose a CNN+Transformer dual-path feature fusion framework, CTA-Net, which explicitly embeds global context and local information for gaze estimation. Specifically, we design two encoders that run in parallel: one based on a transformer and the other on a CNN, embedding global context and local visual information respectively. Then, through our proposed DFA module, the output features of the different encoders are fused together for gaze regression. Furthermore, existing methods ignore the correlation between the two eyes and process eye images separately for the final output, resulting in limited ability to represent feature ranges in monocular images and poor detection results. We propose a complementary attention module for gaze estimation, ACFusion, which aggregates complementary information by leveraging different combination and interaction methods. Specifically, we generate multiple attention feature maps through an attention mechanism, effectively filtering redundant or noisy information in the feature channels. By interacting with binocular attention features, we further enhance the information within the features, thereby improving the performance of gaze estimation for object detection.

[0008] The technical solution of this invention provides a gaze estimation method based on dual-path feature fusion network and attention cross-fusion, which includes the following steps:

[0009] 1. Obtain and organize the gaze estimation dataset for training and testing.

[0010] 1.1 The obtained gaze estimation datasets, including the MPIIFACEGaze dataset, Gaze360 dataset, and Eth-XGaze dataset, are summarized and organized, and environmental factors are eliminated through data preprocessing methods to simplify the gaze regression problem. 1.2 In this invention, the Eth-XGaze dataset with 110 participants is used to train the model; the MPIIFACEGaze dataset and the Gaze360 dataset are used to test the model's generalization performance.

[0011] 1.3 Image Preprocessing: We first define a reference point, the center of the eye or face. Then, we rotate the virtual camera so that it points towards the reference point. This operation eliminates differences caused by the camera's position. We also rotate the virtual camera so that the appearance captured by the rotated camera faces forward. Finally, we scale the image to ensure that the distance between the subject and the camera is the same. Note that the above operations only explain image-level processing. The direction of the gaze also needs to be changed. Lighting also affects the appearance of the human eye. To address this, we use a grayscale eye image instead of an RGB eye image as input and apply histogram equalization to the grayscale image to enhance it.

[0012] 2. This invention utilizes the Transformer network and CNN network from deep learning as the backbone network for extracting facial features and eye features.

[0013] 2.1 A Transformer encoder is constructed to extract facial features. The Transformer encoder adopts a traditional encoder-decoder architecture, where the input image is initially divided into 16 blocks. These blocks are then flattened, and the output dimension is... A linear projection is used to generate the original embedding sequence. The generated embeddings As input to the Transformer encoder, the Transformer encoder consists of an L-layer multi-head self-attention (MSA) module and a multilayer perceptron (MLP). The self-attention module is the core of the Transformer encoder; it updates the state of each embedding patch by globally aggregating information from each layer using the following formula:

[0014] (1)

[0015] in , For the projection matrix, They are respectively and The i-th row. MSA connects multiple self-attention modules (SA) and projects the latent dimension to... To extend SA. MLP is a dense stack of layers, consisting of fully connected layers, GELU activation function, and Dropout. The first fully connected layer quadruples the number of input nodes, while the second fully connected layer restores the original number of nodes. Layer normalization is performed on the output of the last transformer layer to obtain the encoded sequence. . Reshaped to produce the final output .

[0016] 2.2 Constructing a CNN encoder for extracting eye features. Our CNN branch design relies on the ResNet network architecture. The traditional ResNet contains 5 layers, and we choose its second layer. The output is used as the feature information for subsequent fusion. First, we use bicubic interpolation to upsample the left and right eye images from 56×56 to 224×224. Then, the ResNet network extracts features to generate the original feature image pairs. and .

[0017] (2)

[0018] in Indicates an upsampling operation. This indicates a convolution operation.

[0019] 3. Based on the left and right eye features proposed in step 2, this invention designs an Attention Cross-fusion Module (ACFusion) to fuse the features of the left and right eyes. By refining the left and right eye feature maps layer by layer and the original feature map, an information-enhanced feature map is obtained.

[0020] 3.1 Based on left eye characteristics For example, we first calculate the weights of each channel of the input image, correct the features to retain valuable feature maps, and then... The feature map is obtained by feeding it into the channel attention module. .

[0021] (3)

[0022] in This indicates the channel attention module.

[0023] 3.2 The feature map obtained in step 3.1 Compared with the unprocessed feature map of the right eye The result is multiplied, then convolved. The ReLU function is used to calculate the convolved features. A residual connection is introduced to multiply the output with the initial input right-eye feature map, and then convolved together. The resulting feature map is then... .

[0024] (4)

[0025] in These represent the ReLU function and the batch normalization operation, respectively.

[0026] 3.3 Feature map obtained in step 3.2 Feature maps are obtained through the spatial attention module. .

[0027] (5)

[0028] in This represents the spatial attention module; finally, we will refine the left and right eye feature maps layer by layer. , Compared with the original feature map , Perform a connection and output to better grasp the global context information; the output result is... .

[0029] (6)

[0030] in, This indicates a join operation.

[0031] 4. In order to effectively utilize the encoded features of Transformer and CNN, the CTA model proposed in this invention introduces a new DFA module, which combines attention and multi-layer feature fusion mechanism.

[0032] 4.1 Based on the feature map obtained in step 2.2, spatial attention is used as a spatial filter combined with residual connection.

[0033] (7)

[0034] Based on the feature map obtained in step 2.1, spatial attention is used as a spatial filter combined with residual connections.

[0035] (8)

[0036] in This indicates a residual join operation.

[0037] 4.2 The outputs of the two branches from step 4.1 are used to perform element-wise direct addition between feature maps using Hadamard addition. Finally, interactive features... and focus features , It is connected and passes through the final fully connected layer. The final feature is obtained. .

[0038] (9)

[0039] (10)

[0040] in It represents Hadamajafa.

[0041] 5. Feature results predicted by this invention Compared with manually labeled ground truth map The loss function is calculated, and the parameter weights of the proposed model are gradually updated using the Adam optimizer and backpropagation method to finally determine the structure and parameter weights of the CTA model fixation estimation method.

[0042] 6. Based on the model structure and parameter weights determined in steps 2-5, test the data image pairs on the test set involved in step 1, generate a gaze estimation map, and evaluate it using the MAE evaluation metric.

[0043] This invention proposes a novel method for parallel processing of gaze estimation using Convolutional Neural Networks (CNNs) and Transformers. The resulting architecture, CTA-Net, fully leverages the inductive bias of CNNs in modeling spatial relevance and the powerful capabilities of Transformers in modeling global relationships. We introduce a novel fusion technique—the DFA module—which allows CTA-Net to better utilize the advantages of both models. The DFA module further enhances the ability of CNNs to model spatial relevance and improves the efficiency of Transformers in handling global relationships. Furthermore, we introduce a fusion module to enhance feature extraction, including the channels and spatial axes of the feature map, thereby achieving a significant performance improvement. Compared to previous gaze estimation methods, this invention offers the following benefits:

[0044] (1) This invention proposes CTA-Net, a dual-path feature extraction network framework that combines global and local information, taking into account both local details of features and global high-level semantic information. (2) This invention proposes a DFA module to fully integrate feature information from different encoders, promote global information from the Transformer branch, and reduce noise that may be present in CNN features. (3) This invention proposes a complementary eye image aggregation gaze estimation attention mechanism, ACFusion, which generates multiple attention feature maps, thereby realizing the aggregation learning of attention feature information for binocular image interaction. Attached Figure Description

[0045] Figure 1 This diagram shows the overall structure of the invention and the schematic diagram of the dual-path fusion module.

[0046] Figure 2 This diagram illustrates the attention cross-fusion module proposed in this invention.

[0047] Figure 3 A comparison table showing the results of this invention with other gaze estimation methods.

[0048] Figure 4 A visualization showing the results of this invention compared to other gaze estimation methods. Detailed Implementation

[0049] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Furthermore, the described examples are merely a part of the examples of the present invention, and not all of them. All other examples obtained by those skilled in the art based on the examples in this invention without inventive effort are within the scope of protection of this invention.

[0050] Reference Appendix Figure 1 A gaze estimation network based on a dual-path feature fusion framework mainly includes the following steps:

[0051] 1. Obtain and organize the gaze estimation dataset for training and testing.

[0052] 1.1 The obtained gaze estimation datasets, including the MPIIFACEGaze dataset, Gaze360 dataset, and Eth-XGaze dataset, are summarized and organized, and environmental factors are eliminated through data preprocessing methods to simplify the gaze regression problem. 1.2 In this invention, the training dataset with 110 participants, the Eth-XGaze dataset, is used to train the model; the test dataset, the MPIIFACEGaze dataset and the Gaze360 dataset, is used to test the generalization performance of the model.

[0053] 1.3 In image preprocessing, we use a reference point (eye center or face center) to normalize the entire image. First, we align the virtual camera with the reference point and rotate it to eliminate differences caused by the camera position. Next, we rotate the camera to face forward to capture the appearance of the subject. Finally, we scale the image to ensure a consistent distance between the subject and the camera. It is important to note that we need to consider not only image normalization but also changes in the gaze direction.

[0054] Furthermore, lighting conditions also affect the appearance of the human eye. To address this issue, we use grayscale eye images instead of RGB images as input and perform histogram equalization on the grayscale images to enhance image quality.

[0055] 2. This invention utilizes the Transformer network and CNN network from deep learning as the backbone network for extracting facial features and eye features.

[0056] 2.1 A Transformer encoder is constructed to extract facial features; the branch design follows a typical encoder-decoder architecture. Specifically, the input image... First, they were divided into Block patching, where S is typically set to 16. The patch is then flattened and passed to the output dimension. In the linear projection layer, the original embedding sequence is obtained. In order to better utilize prior knowledge, in The same learnable position embeddings were added. The resulting embeddings are then used as input to the transformer encoder. It comprises an L-layer multi-head self-attention (MSA) and a multilayer perceptron (MLP). The core of the transformer encoder is the self-attention module, which updates the state of each embedded patch by globally aggregating information in each layer.

[0057] (1)

[0058] In the formula , It is a projection matrix, a vector. They are respectively and The i-th row. yes An extension that connects multiple self-attention modules. And project the potential dimensions to In this context, the MLP is a dense set of layers consisting of fully connected layers, a GELU activation function, and Dropout. It's important to note that the first fully connected layer quadruples the number of input nodes, while the second fully connected layer restores the original number of nodes. Layer normalization is then applied to the output of the final transformer layer to obtain the encoded sequence. Then we put Remodeling to form the final output Then, it is fused with the feature maps output by the CNN branch through a fully connected layer.

[0059] 2.2 Constructing a CNN encoder for extracting eye features. Our CNN branch design relies on the ResNet network architecture. The traditional ResNet contains 4 layers, and we choose its second layer. The output is used as the feature information for subsequent fusion. First, we use bicubic interpolation to upsample the left and right eye images from 56×56 to 224×224. Then, the ResNet network extracts features to generate the original feature image pairs. and Then, the paired features are input into our proposed feature-enhancing attention module. This is to integrate the features of the left and right eyes. (2)

[0060] in Represents the original feature map. Indicates an upsampling operation. This represents the convolution operation. Features obtained from the CNN branch. The feature maps then input together with the feature maps output from the Transformer branch are fed into the DFA module for full fusion.

[0061] 3. Based on the left and right eye features proposed in step 2, this invention designs an Attention Cross-fusion Module (ACFusion) to fuse the features of the left and right eyes. By refining the left and right eye feature maps layer by layer and the original feature map, an information-enhanced feature map is obtained.

[0062] 3.1 Based on left eye characteristics For example, we first calculate the weights of each channel of the input image, correct the features to retain valuable feature maps, and then... The feature map is obtained by feeding it into the channel attention module. .

[0063] (3)

[0064] in This indicates the channel attention module.

[0065] 3.2 The feature map obtained in step 3.1 Compared with the unprocessed feature map of the right eye The result is multiplied, then convolved. The ReLU function is used to calculate the convolved features. A residual connection is introduced to multiply the output with the initial input right-eye feature map, and then convolved together. The resulting feature map is then... .

[0066] (4)

[0067] in These represent the ReLU function and the batch normalization operation, respectively.

[0068] 3.3 Feature map obtained in step 3.2 Feature maps are obtained through the spatial attention module. .

[0069] (5)

[0070] in This represents the spatial attention module; finally, we will refine the left and right eye feature maps layer by layer. , Compared with the original feature map , Perform a connection and output to better grasp the global context information; the output result is... .

[0071] (6)

[0072] in, This indicates a join operation.

[0073] 4. In order to effectively utilize the encoded features of Transformer and CNN, the CTA model proposed in this invention introduces a new DFA module, which combines attention and multi-layer feature fusion mechanism.

[0074] 4.1 Based on the feature map obtained from the branch in step 2.2, since high-level features in CNNs usually have a large receptive field and strong semantic information, we propose to use spatial attention as a spatial filter combined with residual connections to enhance local details and suppress irrelevant regions.

[0075] (7)

[0076] Based on the feature map obtained from the branch in step 2.1, channel attention combined with residual connections is implemented in SE-Block to facilitate global information from the Transformer branch.

[0077] (8)

[0078] in This indicates a residual join operation.

[0079] 4.2 The outputs of the two branches in step 4.1 are used to implement element-wise addition between feature maps to reduce the gradient vanishing problem, preserve the positional information of the original feature maps, and enhance the expressive power of the model. Finally, interactive features... and focus features , It is connected and passes through the final fully connected layer. The resulting features Effectively capture both the global and local context of the current spatial resolution.

[0080] (9)

[0081] (10)

[0082] Among them | | is Hadamard addition, and Conv is a 3x3 convolutional layer.

[0083] 5. Feature results predicted by this invention Compared with manually labeled ground truth map Calculate the loss function. The loss function is defined as follows:

[0084]

[0085] Where α, γ, and β are adjustable hyperparameters. It is a positive value on the ground. The entire network uses weighted IoU loss and binary cross-entropy loss. The parameter weights of the proposed model are gradually updated using the Adam optimizer and backpropagation method, ultimately determining the structure and parameter weights of the CTA model gaze estimation method.

[0086] 6. Based on the model structure and parameter weights determined in steps 2-5, test the data image pairs on the test set involved in step 1, generate a gaze estimation map, and evaluate it using the MAE evaluation metric.

[0087] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A gaze estimation method based on dual-path feature fusion network and attention cross-fusion, characterized in that, The model corresponding to the method is a CTA model, which includes a Transformer encoder, a CNN encoder, an attention cross-fusion module ACFusion, and a DFA module; the method includes the following steps: (1) Obtain and organize the gaze estimation dataset for training and testing; (1.1) The obtained gaze estimation datasets, including the MPIIFACEGaze dataset, Gaze360 dataset, and Eth-XGaze dataset, are summarized and organized, and environmental factors are eliminated through data preprocessing methods to simplify the gaze regression problem; (1.2) The Eth-XGaze dataset with 110 participants was used as the training dataset for model training; the test datasets from the MPIIFACEGaze dataset and the Gaze360 dataset were used to test the generalization performance of the model. (1.3) Image preprocessing: First, a reference point is defined, either the center of the eyes or the center of the face; then, the virtual camera is rotated so that it points towards the reference point, thus eliminating the difference caused by the camera position; the virtual camera is also rotated so that the appearance captured by the rotated camera faces forward; finally, the image is scaled to ensure that the distance between the subject and the camera is the same; the above operations are only performed on the image for correction, and the direction of gaze is changed synchronously. To address the impact of lighting on the appearance of the human eye, grayscale eye images are used instead of RGB eye images as input, and histogram equalization is applied to the grayscale images to enhance the image. (2) The Transformer network and CNN network in deep learning are used as the backbone network to extract facial features and eye features; (2.1) A Transformer encoder is constructed to extract facial features. The Transformer encoder adopts an encoder-decoder architecture. The input image is first divided into 16 blocks; these blocks are then flattened and linearly projected onto them with an output dimension of D0 to generate the original embedding sequence; the generated embedding z 0 ∈Rᴺ×ᴰ 0 As input to the Transformer encoder, the Transformer encoder consists of L layers of multi-head self-attention (MSA) and multilayer perceptrons (MLP); the self-attention module globally aggregates information from each layer using the following formula to update the state of each embedding patch: (1) in , For the projection matrix, They are respectively and In the i-th row, MSA connects multiple self-attention modules (SA) and projects the latent dimension to... To extend SA, MLP is a dense stack of layers, consisting of fully connected layers, GELU activation function, and Dropout. The first fully connected layer quadruples the number of input nodes, the second fully connected layer restores the original number of nodes, and the output of the last transformer layer is normalized to obtain the encoded sequence. , Reshaped to produce the final output ; (2.2) Construct a CNN encoder to extract eye features. The CNN branch is based on the ResNet network architecture, which contains 5 layers. Select the second layer. The output is used as the feature information for subsequent fusion; firstly, bicubic interpolation is used to upsample the left and right eye images from 56×56 to 224×224, and then the ResNet network is used for feature extraction to generate original feature image pairs. and , (2) in Indicates an upsampling operation. Indicates the convolution operation; (3) Based on the left and right eye features obtained in step (2), an attention cross-fusion module (ACFusion) is set up to fuse the features of the left and right eyes. By refining the left and right eye feature maps layer by layer and the original feature map, an information-enhanced feature map is obtained. (3.1) Based on left eye features For example, firstly, the weights of each channel of the input image are calculated, and the features are corrected to retain valuable feature maps. The feature map is obtained by feeding it into the channel attention module. , (3) in This indicates the channel attention module; (3.2) The feature map obtained in step (3.1) Compared with the unprocessed feature map of the right eye The result is multiplied, then convolved. The ReLU function is used to calculate the convolved features. A residual connection is introduced to multiply the output with the original input right-eye feature map, and then convolved together to obtain the feature map. , (4) in These represent the ReLU function and the batch normalization operation, respectively. (3.3) The feature map obtained in step (3.2) Feature maps are obtained through the spatial attention module. , (5) in This represents a spatial attention module; Finally, the feature maps of the left and right eyes, refined layer by layer, are... , , and the original feature map , Perform a connection and output to better grasp the global context information; the output result is... (6) in, Indicates a connection operation; (4) To effectively utilize the encoding features of Transformer and CNN, a DFA module is set up, which combines an attention mechanism and a multi-layer feature fusion mechanism; (4.1) Based on the feature map obtained from the branch in step (2.2), spatial attention is used as a spatial filter combined with residual connections. (7) Based on the feature map obtained in step (2.1), spatial attention is used as a spatial filter combined with residual connections. (8) in Indicates a residual join operation; (4.2) The outputs of the two branches in step (4.1) are used to perform element-wise direct addition between feature maps using Hadamard addition. Finally, the interactive features are... and focus features , The final features are obtained by connecting and passing through the final fully connected layer. , (9) (10) in This refers to the Dharma of Hadama; (5) The predicted feature result g′ is compared with the manually labeled ground truth map G to calculate the loss function, and the parameter weights of the model are updated step by step through the Adam optimizer and backpropagation method. Finally, the structure and parameter weights of the CTA model gaze estimation method are determined. (6) Based on the model structure and parameter weights determined in steps (2) to (5), test the image data in the test set involved in step (1), generate a gaze estimation map, and evaluate it using the MAE evaluation index.

Images