Sight line estimation method and system for state space and Transform adaptive hybrid modeling

Abstract

The invention discloses a state space and Transform adaptive hybrid modeling sight line estimation method and system, and the method comprises the following steps: 1, obtaining a to-be-detected face image, and carrying out the preprocessing of the to-be-detected face image; 2, performing feature extraction on the preprocessed face image by using a deep convolutional neural backbone network; 3, inputting the extracted features into an adaptive context enhancement module, and performing reweighting on the features through a dependency relationship between modeling channels to obtain a calibrated feature map; 4, performing spatial flattening processing on the calibrated feature map, and adding position codes to obtain a feature sequence; 5, inputting the feature sequence into a self-adaptive hybrid encoder, and extracting high-dimensional sight line features; and 6, sending the high-dimensional sight line features into the sight line regression network to obtain a two-dimensional sight line direction vector. According to the invention, complementation of multi-view features is realized through a multi-branch parallel architecture, the difficulty level of image input is dynamically perceived by using an adaptive fusion mechanism, and the precision and robustness of sight line estimation in a complex scene are effectively improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, and in particular to a gaze estimation method and system based on state space and Transformer adaptive hybrid modeling. Background Technology

[0002] Gaze estimation aims to infer the direction of a person's gaze from images of a face or eyes, and has broad application prospects in fields such as human-computer interaction, intelligent driving monitoring, and virtual reality.

[0003] Existing gaze estimation methods primarily rely on deep convolutional neural networks (CNNs). While CNNs excel at extracting local features, their receptive field size limits their ability to effectively capture long-range dependencies between the eyes and the entire face, as well as between different facial regions. To address this issue, researchers have introduced the Transformer architecture, utilizing self-attention mechanisms to capture global features. However, the computational complexity of the Transformer increases quadratically with sequence length, resulting in slow inference speeds and making it difficult to meet the demands of real-time applications.

[0004] Recently, State Space Models (SSMs) such as Mamba have gained attention for their linear computational complexity and excellent sequence modeling capabilities. However, a single SSM model may not be as robust as the Transformer's self-attention mechanism when dealing with extremely complex and detailed visual features.

[0005] Therefore, how to combine the feature extraction capabilities of CNNs, the efficient sequence modeling capabilities of SSMs, and the global attention advantages of Transformers to construct a high-precision and computationally efficient gaze estimation model is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] The purpose of this invention is to provide a gaze estimation method that uses adaptive hybrid modeling of state space and Transformer. By combining a convolutional neural network backbone, an adaptive context enhancement module, and an adaptive hybrid encoder, a multi-branch architecture is used to adaptively hybrid model the state space model and the Transformer, which effectively improves the accuracy and robustness of gaze estimation.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] Step S1: Obtain the face image to be detected and perform preprocessing;

[0009] Step S2: Use a deep convolutional neural backbone network to extract features from the preprocessed face image to obtain the original feature map;

[0010] Step S3: Input the original feature map into the Adaptive Context Enhancement Module (ACEM), and reweight the features by modeling the dependencies between channels to obtain the calibrated feature map;

[0011] Step S4: Spatial flattening and positional encoding are added to the calibrated feature map to obtain the feature sequence;

[0012] Step S5: Input the feature sequence into the adaptive hybrid encoder;

[0013] The adaptive hybrid encoder constructs a parallel hybrid architecture of state space model and Transformer, including a pure vision state space model (Vision Mamba) branch, a pure Transformer encoding branch and a serial hybrid branch set in parallel. The adaptive fusion module dynamically adjusts the weight of each branch according to the input feature sequence to achieve adaptive hybrid modeling of multi-view features and extract high-dimensional gaze features.

[0014] Step S6: Input the obtained high-dimensional gaze features into the gaze regression network to obtain a two-dimensional gaze direction vector.

[0015] The specific implementation of step S1 includes:

[0016] Step S1-1: Obtain the original image containing the face, and use a face detection algorithm to crop out the face region;

[0017] Step S1-2: Adjust the resolution of the cropped face images to a uniform value. And perform normalization processing; in this embodiment, , ;

[0018] The face detection algorithm includes: Multi-Task Cascaded Convolutional Neural Network (MTCNN), RetinaFace algorithm, or feature detection algorithm based on Histogram of Oriented Gradients (HOG). In this embodiment, the mature MTCNN algorithm is preferred. This algorithm consists of three cascaded deep convolutional networks: P-Net, R-Net, and O-Net. By constructing an image pyramid input, it sequentially completes the rapid generation of candidate windows, the filtering and fine-tuning of candidate windows, and the precise localization of facial key points, thereby ensuring that the face region cropped from the original image contains complete facial features and is accurately positioned.

[0019] Further, step S2 specifically includes: using a ResNet50 network as the backbone network and loading pre-trained weights; extracting the feature map output from the last convolutional layer of the backbone network, and utilizing... The convolutional layer performs dimensionality reduction mapping on the feature map, adjusting the number of channels to C, resulting in a channel count of C and a spatial resolution of C. The original feature map (the overall feature tensor shape is...) In this embodiment, .

[0020] Furthermore, the specific processing procedure of the Adaptive Context Enhancement Module (ACEM) in step S3 includes the following sub-steps:

[0021] 1) Feature preprocessing: The original feature map is first processed by a... Convolution performs channel projection, followed by a... The depthwise convolution is used to obtain an intermediate feature map containing preliminary contextual information;

[0022] 2) Parallel extraction via two branches: The intermediate feature map is simultaneously fed into a local convolutional branch and a global attention branch for parallel processing.

[0023] Local convolutional branch: Reshapes features from intermediate feature maps, through... Convolution performs channel feature transformation, and then... Deep convolution extracts local spatial detail features from images;

[0024] Global attention branch: Separates the intermediate feature map into query vector (Q), key vector (K), and value vector (V) along the channel dimension; calculates the global self-attention weight matrix using the query vector and key vector, normalizes it using the Softmax function, and applies it to the value vector; and then... Convolutional projection and SiLU activation function processing are used to extract global long-range semantic features of the image;

[0025] 3) Feature Adaptive Fusion: The local spatial detail features output by the local convolution branch and the global long-distance semantic features output by the global attention branch are added element-wise to output the calibrated feature map.

[0026] Furthermore, the feature sequence construction in step S4 is implemented as follows:

[0027] Before inputting into the adaptive hybrid encoder, the calibrated feature map is spatially flattened to obtain the corresponding flattened sequence. A learnable classification token ([CLS] Token) is concatenated at the beginning of the flattened sequence, and positional encoding is added to obtain the feature sequence. .

[0028] Furthermore, the adaptive hybrid encoder in step S5 adopts a "multi-branch parallel extraction and adaptive fusion" architecture, specifically including:

[0029] Step S5-1. Parallel extraction of features from multiple perspectives;

[0030] feature sequence Simultaneously input three parallel encoding branches:

[0031] The first branch (pure visual state space model branch): consists of N layers of stacked visual state space model units. This branch utilizes a bidirectional scanning mechanism to efficiently capture feature sequences with linear computational complexity. The long-range contextual dependency is denoted as the output feature. .

[0032] The second branch (pure Transformer coding branch): consists of M stacked Transformer coding units. This branch utilizes a multi-head self-attention mechanism to capture feature sequences. The global fine-grained association between any two tokens is denoted as the output feature. .

[0033] The third branch (serial hybrid branch): adopts a serial connection structure of "Mamba first, then Transformer", containing N / 2 layers of visual state space model units and M / 2 layers of Transformer encoding units. This branch aims to extract comprehensive features that take into account both long-range dependencies and fine-grained local concerns. The output features are denoted as... .

[0034] The described visual state space model is a visual Mamba unit constructed based on a selective state space model. Its function is to utilize a bidirectional scanning mechanism to perform forward and reverse flattening of the two-dimensional image feature sequence, followed by inputting it into the state space model for feature extraction and fusion. This mechanism can efficiently capture the long-distance contextual dependencies of the sequence with linear computational complexity, while overcoming the single causality limitation of traditional state space models in image processing through bidirectional paths, thereby obtaining visual features encompassing the global receptive field. This belongs to the prior art.

[0035] The Transformer encoding unit, a feature encoding layer incorporating Multi-Head Self-Attention (MHSA) and a Feedforward Neural Network (FFN), utilizes the self-attention mechanism to calculate the correlation weights between any two tokens in the feature sequence, thereby capturing the global fine-grained associations and semantic details of the input features. A specially designed "per-layer positional encoding injection" strategy is employed. Unlike traditional Transformers that only add positional encoding once at the model input, this embodiment explicitly superimposes positional encoding onto the query and key vectors during the computation of each Transformer encoding unit. This improvement enhances the model's ability to perceive the spatial geometrical positional information of faces and eyes in deep networks, preventing the loss of positional information during deep feature transmission, thus making it more suitable for geometrically sensitive regression tasks such as gaze estimation.

[0036] Step S5-2. Adaptive weighted fusion;

[0037] The adaptive fusion module is used to dynamically select the output features of the three branches:

[0038] S5-2-1. Feature Aggregation: This involves combining the output features of the three branches. , and By performing aggregation, we obtain an aggregated feature map that contains information from all branches. ;

[0039] S5-2-2. Weight Generation: The aggregated feature map... The input is gated network, and adaptive weights for the three branches are generated through a "global compression-nonlinear mapping" mechanism. , and The specific process includes:

[0040] 1) Global descriptor extraction: First, the aggregated feature map is processed... Perform global average pooling to compress the spatial dimension to This yields a channel descriptor vector containing global semantic information;

[0041] 2) Feature Mapping and Scoring: The channel descriptor vector is input into a multilayer perceptron (MLP); firstly, the channel dimension is compressed (dimensionality reduction) through the first fully connected layer, and key nonlinear features are extracted through the ReLU activation function; then, the feature vector is mapped into a scoring vector with a dimension of 3 through the second fully connected layer.

[0042] 3) Weight Normalization: The scoring vector is normalized using the Softmax function, outputting three adaptive weights whose values ​​are between (0,1) and sum to 1, thus satisfying the condition... + + =1.

[0043] S5-2-3. Weighted Output: The output features of the corresponding branches are weighted and summed according to the generated adaptive weights to obtain the final high-dimensional gaze features. .

[0044] Furthermore, the visual state space model unit utilizes a bidirectional scanning mechanism to capture long-distance dependencies in the sequence. The specific calculation process includes: the input feature sequence is first processed by layer normalization and positional encoding superposition, and then divided into two branches for processing: the forward branch maintains the original order of the feature sequence, and after calculation by the internal state space model operator, the forward feature is obtained; the reverse branch flips the feature sequence in the time dimension, and after calculation by the state space model operator, it is flipped again to obtain the reverse feature; finally, the forward and reverse features are added element-wise and fused, and the bidirectional fused feature is passed through a Dropout layer and then connected with the input feature through a first residual connection to obtain the intermediate feature; the intermediate feature is normalized by a layer and then input into a feedforward neural network layer, and the output result is connected with the intermediate feature through a second residual connection to obtain the final output feature of the unit.

[0045] Furthermore, the gaze regression network includes a fully connected layer that maps the feature vector corresponding to the classification token into a two-dimensional vector containing yaw and pitch angles, and is trained using the L1 loss function.

[0046] Furthermore, the present invention also provides a gaze estimation system that uses adaptive hybrid modeling of state space and Transformer. This system combines a convolutional neural network backbone, an adaptive context enhancement module, and an adaptive hybrid encoder, and utilizes a multi-branch architecture to adaptively hybrid model the state space model and Transformer, effectively improving the accuracy and robustness of gaze estimation.

[0047] The system includes a data acquisition and preprocessing module, a feature extraction module, a calibration module, a location coding module, and an adaptive hybrid coding module.

[0048] The aforementioned acquisition and preprocessing module is used to acquire the face image to be detected and perform preprocessing.

[0049] The feature extraction module uses a deep convolutional neural backbone network to extract features from the preprocessed face image to obtain the original feature map.

[0050] The calibration module inputs the original feature map into the adaptive context enhancement module (ACEM), and reweights the features by modeling the dependencies between channels to obtain the calibrated feature map;

[0051] The location encoding module performs spatial flattening on the calibrated feature map and adds location encoding to obtain the feature sequence;

[0052] The adaptive hybrid coding module described above achieves adaptive hybrid coding by using a "multi-branch parallel extraction and adaptive fusion" architecture for feature sequences.

[0053] The beneficial effects of this invention are as follows:

[0054] This invention utilizes the ResNet50 backbone network to extract deep semantic features of images and adaptively reweights channel features through the ACEM module, effectively enhancing the expression of key features and suppressing noise interference.

[0055] This invention innovatively designs an adaptive hybrid encoder, constructing a parallel hybrid architecture of a state-space model and a Transformer. By setting up pure Mamba branches, pure Transformer branches, and hybrid branches in parallel, the model can simultaneously acquire long-range contextual features with linear complexity and global fine-grained features based on a self-attention mechanism. Furthermore, an adaptive fusion module dynamically adjusts the fusion weights of each branch based on the semantic information of the input image, achieving adaptive hybrid modeling of multi-view features. This architecture retains the advantages of each individual model while endowing the model with dynamic perception capabilities for samples of varying difficulty, significantly improving the robustness and accuracy of gaze estimation in complex scenes.

[0056] This invention employs a bidirectional scanning strategy in the Mamba unit. Compared to unidirectional scanning, this strategy enables the model to utilize both forward and reverse sequence information simultaneously, thereby overcoming the limitations of a single causal perspective in image feature extraction and significantly improving the model's ability to perceive the global context of the eyes and face. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the overall process of the method of the present invention;

[0058] Figure 2 This is a schematic diagram of the Adaptive Context Enhancement Module (ACEM) in this invention;

[0059] Figure 3 This is a schematic diagram of the overall model structure of the present invention; Detailed Implementation

[0060] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0061] like Figure 1 As shown, a gaze estimation method based on state-space and Transformer adaptive hybrid modeling employs the following technical steps:

[0062] Step S1: Obtain the face image to be detected and perform preprocessing;

[0063] Step S2: Use a deep convolutional neural network backbone to extract features from the preprocessed image to obtain the original feature map;

[0064] Step S3: Input the original feature map into the Adaptive Context Enhancement Module (ACEM), and reweight the features by modeling the dependencies between channels to obtain the calibrated feature map;

[0065] Step S4: Spatial flattening and positional encoding are added to the calibrated feature map to obtain the feature sequence;

[0066] Step S5: Input the feature sequence into the adaptive hybrid encoder; the adaptive hybrid encoder constructs a parallel hybrid architecture of state space model and Transformer, including a pure visual state space model branch, a pure Transformer encoding branch and a serial hybrid branch set in parallel, and dynamically adjusts the weight of each branch according to the input features through the adaptive fusion module to realize adaptive hybrid modeling of multi-view features and extract high-dimensional gaze features.

[0067] Step S6: Input the obtained high-dimensional gaze features into the gaze regression network to obtain a two-dimensional gaze direction vector;

[0068] The specific implementation of step S1 includes: step S1-1: acquiring the original image containing the face, and cropping out the face region using a face detection algorithm;

[0069] Step S1-2: Adjust the resolution of the cropped face images to a uniform value. And perform normalization processing; in this embodiment, , The specific implementation of step S2 includes: using a ResNet50 network as the backbone network and loading pre-trained weights; extracting the feature map output from the last convolutional layer of the network, and mapping the channel dimension of the feature map to... The dimension is obtained as The original feature map; in this embodiment, .

[0070] Step S3 specifically includes: the Adaptive Context Enhancement Module (ACEM) receiving dimension as follows: The original feature map; Step S3-1: The original feature map is first processed by a... Convolution performs channel projection, followed by a... The depthwise convolution is used to obtain an intermediate feature map containing preliminary contextual information;

[0071] Step S3-2: The intermediate feature map is simultaneously fed into the local convolution branch and the global attention branch for parallel processing.

[0072] Local convolutional branch: Reshapes features from intermediate feature maps, through... Convolution performs channel feature transformation, and then... Deep convolution extracts local spatial detail features from images;

[0073] Global attention branch: Separates the intermediate feature map into query vector (Q), key vector (K), and value vector (V) along the channel dimension; calculates the global self-attention weight matrix using the query vector and key vector, applies it to the value vector after Softmax normalization, and then... Convolutional projection and SiLU activation function processing are used to extract global long-range semantic features; Step S3-3: The local spatial detail features output by the local convolutional branch and the global long-range semantic features output by the global attention branch are added element-wise to output the calibrated feature map.

[0074] Step S5 specifically includes: the adaptive hybrid encoder adopts a "three-branch parallel and adaptive fusion" architecture; before the feature sequence is input into the adaptive hybrid encoder, a learnable classification token is appended to the beginning of the sequence, at which point the sequence length is... Then, three parallel branches are input simultaneously:

[0075] The first branch (pure state-space model branch) contains N layers of visual state-space model units, used to capture long-distance dependencies of sequences with linear complexity; in this embodiment, N=6.

[0076] The second branch (pure Transformer branch): contains M layers of Transformer coding units, used for fine-grained global feature modeling using a self-attention mechanism; in this embodiment, M=6.

[0077] The third branch (hybrid serial branch): adopts a serial connection structure of "Mamba first, then Transformer", which includes N / 2 (i.e. 3) layers of visual state space model units and M / 2 (i.e. 3) layers of Transformer coding units, used to extract comprehensive features;

[0078] Adaptive fusion: The output features of the three branches are input into the adaptive fusion module; the module uses a gating network to dynamically generate the weight coefficients corresponding to the three branches, and performs a weighted summation of the output features of the three branches to obtain the final output features.

[0079] The specific calculation process of the visual state space model unit includes: inputting a feature sequence. First, layer normalization and positional encoding are superimposed; then, the process is divided into two branches: forward and reverse.

[0080] Forward branch: Preserving the feature sequence The original order is input into the internal state-space model operator for calculation to obtain positive features. Reverse branch: This involves using the feature sequence... The input state-space model operator is flipped in the time dimension for calculation, and the calculation result is flipped again in the time dimension to obtain the reverse feature. Feature fusion: combining positive features and reverse features Element-wise addition is performed to obtain bidirectional fused features; these bidirectional fused features are then processed through a Dropout layer and combined with the input features. A first residual connection is performed to obtain intermediate features; these intermediate features are then normalized and input into a feedforward neural network layer. The output of this layer is then connected to the intermediate features via a second residual connection to obtain the final output features of the unit. The internal parameters of the state-space model are configured as follows: state dimension... kernel size expansion factor .

[0081] The Transformer encoding unit includes a multi-head self-attention mechanism; the number of heads in the multi-head self-attention mechanism is configured as follows: The hidden layer dimension of the feedforward network is configured as follows: The Dropout ratio is configured to 0.1; the input features pass through a multi-head self-attention layer and a feedforward network layer in sequence, with layer normalization and residual connections set before and after each layer.

[0082] Step S6 specifically includes: extracting the feature vector corresponding to the classification token in the output sequence of the hybrid encoder, the dimension of which is... The feature vector is input into a gaze regression network, which contains a fully connected layer that reduces the dimension from... Mapped to 2, the output is a predicted line-of-sight vector containing yaw and pitch angles. During the training phase, an L1 loss function is constructed to calculate the predicted gaze vector. With real-view label The error between them is calculated using the following formula:

[0083]

[0084] The model parameters are updated by minimizing the loss using the backpropagation algorithm.

[0085] In practical application, the invention uses Adam, which has a fast convergence speed, as the optimizer, adopts the L1 loss function (mean absolute error) as the loss function, sets the initial learning rate to 5e-4, and sets the batch size to 256.

[0086] Table 1 compares the method of this invention with other mainstream gaze estimation methods on the MPIIFaceGaze dataset. The metric is the mean angle error, expressed in degrees (°). A lower mean angle error indicates higher gaze estimation accuracy and better method performance. The method of this invention exhibits the lowest error on the MPIIFaceGaze dataset, reaching 3.53°, demonstrating the superiority of this invention.

[0087] Table 1

[0088]

[0089] Table 2

[0090]

[0091] Table 2 shows the configurations of different experimental groups and their corresponding line-of-sight estimation errors, with detailed analysis as follows:

[0092] 1. Effectiveness Analysis of the Adaptive Context Enhancement Module (ACEM): By comparing experimental groups D and E, it can be seen that, under the premise of both adopting the "three-branch parallel encoder architecture" proposed in this invention, experimental group E (error 3.53°) with the introduction of the adaptive context enhancement module (ACEM) shows a significant improvement in prediction accuracy compared to experimental group D (error 3.66°) without the module. This fully demonstrates that the ACEM module can effectively calibrate the backbone network features, capture local and global contextual information, thereby reducing gaze estimation error.

[0093] 2. Advantages of the Three-Branch Parallel Architecture: By comparing experimental groups A, B, and C with experimental group E, the superiority of the encoder architecture proposed in this invention can be verified.

[0094] Compared with the pure Transformer architecture (A vs E): Experimental group A uses a single Transformer coding architecture, which has global modeling capabilities, but lacks the ability to capture local details and complement multi-view features, with an error of 3.75°; while the present invention (experimental group E) introduces Mamba branches and hybrid branches, and the error is greatly reduced to 3.53°.

[0095] Compared to the pure visual Mamba architecture (B vs E): Experimental group B employed a pure visual Mamba encoder architecture with an average angular error of 3.63°. While the Mamba architecture demonstrates excellent linear computational efficiency in capturing long-range dependencies in sequences due to its unique state-space model, its lack of a global self-attention mechanism like that of the Transformer makes it difficult to capture the fine-grained global correlation between any two tokens when dealing with tasks like face gaze estimation, which are extremely sensitive to spatial geometry. In contrast, this invention (experimental group E) addresses the shortcomings of the single Mamba architecture in global feature extraction by introducing a Transformer branch that complements the Mamba branch in parallel, thereby further reducing the error to 3.53°.

[0096] Compared to the serial hybrid architecture (C vs E): Experimental group C used a simple "Mamba first, then Transformer" serial connection method, with an error of 3.67°. Although this result is slightly better than the pure Transformer architecture (experimental group A, 3.75°), it shows that the linear long-distance modeling capability of introducing Mamba does indeed bring some performance gains to the model. However, this result is still significantly worse than the three-branch parallel architecture proposed in this invention (experimental group E, 3.53°). This further shows that simple serial stacking can only superimpose features to a certain extent, but it cannot, like the "three-branch parallel extraction + adaptive gating fusion" strategy of this invention, allow the model to dynamically adjust the dependence weights on local details, global semantics, and linear sequence features according to the complexity of the input image, thus failing to tap the maximum potential of multi-view feature complementarity.

[0097] The "three-branch parallel extraction + adaptive gating fusion" strategy adopted in this invention can simultaneously utilize Mamba's linear long-distance modeling capability and Transformer's global attention advantage, and achieve optimal performance (3.53°) by dynamically adjusting the weights of each branch through a gating network, demonstrating the creativity and practicality of this architecture design.

[0098] Finally, it should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the claims.

Claims

1. A gaze estimation method using state-space and Transformer adaptive hybrid modeling, characterized in that, Includes the following steps: Step S1: Obtain the face image to be detected and perform preprocessing; Step S2: Use a deep convolutional neural backbone network to extract features from the preprocessed face image to obtain the original feature map; Step S3: Input the original feature map into the adaptive context enhancement module, and reweight the features by modeling the dependencies between channels to obtain the calibrated feature map; Step S4: Spatial flattening and positional encoding are added to the calibrated feature map to obtain the feature sequence; Step S5: Input the feature sequence into the adaptive hybrid encoder; The adaptive hybrid encoder constructs a parallel hybrid architecture of state space model and Transformer, including a pure visual state space model branch, a pure Transformer encoding branch and a serial hybrid branch set in parallel. The adaptive fusion module dynamically adjusts the weight of each branch according to the input feature sequence to achieve adaptive hybrid modeling of multi-view features and extract high-dimensional gaze features. Step S6: Input the obtained high-dimensional gaze features into the gaze regression network to obtain a two-dimensional gaze direction vector.

2. The gaze estimation method based on state-space and Transformer adaptive hybrid modeling according to claim 1, characterized in that... Step S1 is implemented as follows: Step S1-1: Obtain the original image containing the face, and use a face detection algorithm to crop out the face region; Step S1-2: Adjust the resolution of the cropped face images to a uniform value. And then normalize it.

3. The gaze estimation method based on state-space and Transformer adaptive hybrid modeling according to claim 1, characterized in that... Step S2 is implemented as follows: A ResNet50 network is used as the backbone network, with pre-trained weights loaded; the feature map output of the last convolutional layer of the backbone network is extracted and utilized. The convolutional layer performs dimensionality reduction mapping on the feature map, adjusting the number of channels to C, resulting in a channel count of C and a spatial resolution of C. Original feature map.

4. The gaze estimation method based on state-space and Transformer adaptive hybrid modeling according to claim 1, characterized in that... The specific implementation steps of the adaptive context enhancement module in step S3 are as follows: 3-1. Feature Preprocessing: The original feature map is first processed through a... Convolution performs channel projection, followed by a... The depthwise convolution is used to obtain an intermediate feature map containing preliminary contextual information; 3-2. Parallel Extraction via Two Branches: The intermediate feature maps are simultaneously fed into the local convolutional branch and the global attention branch for parallel processing. Local convolutional branch: Reshapes features from intermediate feature maps, through... Convolution performs channel feature transformation, and then... Deep convolution extracts local spatial detail features from images; Global attention branch: Separates the intermediate feature map into query vector, key vector, and value vector along the channel dimension; calculates the global self-attention weight matrix using the query vector and key vector, normalizes it using the Softmax function, and applies it to the value vector; and then... Convolutional projection and SiLU activation function processing are used to extract global long-range semantic features of the image; 3-3. Feature Adaptive Fusion: The local spatial detail features output by the local convolution branch are added element-wise with the global long-range semantic features output by the global attention branch to output a calibrated feature map.

5. The gaze estimation method based on state-space and Transformer adaptive hybrid modeling according to claim 4, characterized in that... The feature sequence construction in step S4 is implemented as follows: Before inputting into the adaptive hybrid encoder, the calibrated feature map is spatially flattened to obtain the corresponding flattened sequence. A learnable classification token is concatenated at the beginning of the flattened sequence, and positional encoding is added to obtain the feature sequence. .

6. The gaze estimation method based on state-space and Transformer adaptive hybrid modeling according to claim 4, characterized in that... The adaptive hybrid encoder in step S5 adopts an architecture of "multi-branch parallel extraction and adaptive fusion", including step S5-1 multi-view feature parallel extraction: feature sequence Simultaneously input three parallel encoding branches: The first branch is a pure visual state space model branch: it consists of N layers of stacked visual state space model units; this branch utilizes a bidirectional scanning mechanism to efficiently capture feature sequences with linear computational complexity. The long-range contextual dependency is denoted as the output feature. ; The second branch is a pure Transformer coding branch: it consists of M stacked Transformer coding units; this branch utilizes a multi-head self-attention mechanism to capture feature sequences. The global fine-grained association between any two tokens is denoted as the output feature. ; The third branch is a serial hybrid branch: it adopts a serial connection structure of "Mamba first, then Transformer", and includes N / 2 layers of visual state space model units and Layer Transformer coding unit; the output feature of this branch is denoted as .

7. The gaze estimation method according to claim 6, characterized in that, The visual state space model is a visual Mamba unit constructed based on a selective state space model. Its function is to use a bidirectional scanning mechanism to flatten the two-dimensional image feature sequence in the forward and reverse directions respectively, and then input it into the state space model for feature extraction and fusion. The Transformer encoding unit is a feature encoding layer that includes a multi-head self-attention mechanism and a feedforward neural network. Its function is to use the self-attention mechanism to calculate the correlation weight between any two tokens in the feature sequence, and to adopt a specially designed "per-layer positional encoding injection" strategy.

8. The gaze estimation method according to claim 6, characterized in that, The adaptive hybrid encoder in step S5 adopts a "multi-branch parallel extraction and adaptive fusion" architecture, and also includes step S5-2. Adaptive weighted fusion, which uses the adaptive fusion module to dynamically select the output features of the three branches: S5-2-1. Feature Aggregation: This involves combining the output features of the three branches. , and By performing aggregation, we obtain an aggregated feature map that contains information from all branches. ; S5-2-2. Weight Generation: The aggregated feature map... The input is gated network, and adaptive weights for the three branches are generated through a "global compression-nonlinear mapping" mechanism. , and ; S5-2-3. Weighted Output: The output features of the corresponding branches are weighted and summed according to the generated adaptive weights to obtain the final high-dimensional gaze features. .

9. The gaze estimation method according to claim 8, characterized in that, The generation of adaptive weights in step S5-2-2 is as follows: 1) Global descriptor extraction: First, the aggregated feature map is processed... Global average pooling is performed to compress the spatial dimension, resulting in a channel descriptor vector containing global semantic information; 2) Feature mapping and scoring: The channel descriptor vectors are input into the multilayer perceptron; firstly, the channel dimension is compressed through the first fully connected layer, and then the key nonlinear features are extracted through the ReLU activation function; The feature vector is then mapped to a 3-dimensional score vector through a second fully connected layer. 3) Weight Normalization: The scoring vector is normalized using the Softmax function, outputting three adaptive weights whose values ​​are between (0, 1) and sum to 1, thus satisfying the condition... + + =1.

10. A gaze estimation system based on state-space and Transformer adaptive hybrid modeling, characterized in that, The system is used to implement any one of the methods as claimed in claims 1-9. The system includes a data acquisition and preprocessing module, a feature extraction module, a calibration module, a position encoding module, and an adaptive hybrid encoding module. The aforementioned acquisition and preprocessing module is used to acquire the face image to be detected and perform preprocessing. The feature extraction module uses a deep convolutional neural backbone network to extract features from the preprocessed face image to obtain the original feature map. The calibration module inputs the original feature map into the adaptive context enhancement module (ACEM), and reweights the features by modeling the dependencies between channels to obtain the calibrated feature map; The location encoding module performs spatial flattening on the calibrated feature map and adds location encoding to obtain the feature sequence; The adaptive hybrid coding module described above achieves adaptive hybrid coding by using a "multi-branch parallel extraction and adaptive fusion" architecture for feature sequences.

Images