A video line-of-sight estimation method, system, storage medium and device

Abstract

The application discloses a video gaze estimation method, comprising: performing feature extraction on input binocular splicing images and full-face images frame by frame to obtain eye feature sequences and face feature sequences arranged in time sequence; based on the two sequences, splicing the eye features and the face features of each frame to obtain joint features of the frame; taking the feature map center of the joint features as an initial reference point, obtaining a learnable offset amount for driving the dynamic offset of the reference point based on the joint features; aligning the eye features by using the offset amount and the initial reference point to generate aligned eye features; calculating the difference between the aligned eye features of adjacent two frames to obtain inter-frame difference features; performing gated fusion on the aligned eye features, the inter-frame difference features and the face feature sequences to obtain an adaptively fused feature sequence, and mapping the feature sequence on a time dimension to obtain a time sequence enhanced feature representation; and performing regression calculation on the feature representation to determine the yaw angle and the pitch angle of the gaze direction.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a video gaze estimation method, system, storage medium, and device. Background Technology

[0002] Video gaze estimation is a key technology in the fields of computer vision and human-computer interaction, aiming to accurately infer the direction of human eye gaze from a sequence of images containing consecutive frames. This technology has wide application value in many scenarios such as driver fatigue monitoring, augmented reality / virtual reality (AR / VR) interaction, psychological and behavioral analysis, and intelligent human-computer interfaces.

[0003] Current mainstream video gaze estimation methods mainly follow two technical approaches. One approach is to extract and process features from both eyes and the full face image separately, then perform simple stitching or early fusion before feeding them into a subsequent network for prediction. This type of method fails to fully consider the spatial correlation between eye and facial features and is insufficient in modeling the subtle, subpixel-level eye movements between frames. The other approach is to introduce optical flow or attention mechanisms to capture temporal dynamics. However, traditional optical flow methods are inaccurate in low-texture eye regions, while existing attention mechanisms often have fixed sampling positions when processing video sequences, lacking the ability to perform precise subpixel-level alignment of eye features across frames. Summary of the Invention

[0004] Therefore, it is necessary to propose a video gaze estimation method to address the above problems.

[0005] A video gaze estimation method, the method comprising the following steps: Feature extraction is performed frame by frame on the input stitched eye image and full face image to obtain eye feature sequence and face feature sequence arranged in chronological order; Based on the eye feature sequence and face feature sequence, the eye features and face features of each frame are concatenated to obtain the joint features of that frame; Using the center of the feature map of the joint features as the initial reference point, a learnable offset that drives the dynamic shift of the reference point is obtained based on the joint features. The eye features are spatially aligned using learnable offsets and initial reference points to generate aligned eye features. Calculate the difference between the aligned eye features of two adjacent frames to obtain the inter-frame difference features; Gated fusion is performed on the aligned eye features, the inter-frame difference features, and the facial feature sequence to obtain an adaptively fused feature sequence; The adaptively fused feature sequence is mapped onto the time dimension to obtain a temporally enhanced feature representation. Regression calculations are performed on the time-enhanced feature representation to determine the yaw angle and pitch angle that characterize the line-of-sight direction.

[0006] In the above scheme, the step of concatenating the eye features and face features of each frame based on the eye feature sequence and face feature sequence, and then filtering the concatenated features to obtain the joint features of that frame, specifically includes: The facial features and eye features of frame t are concatenated to obtain preliminary joint features. :

[0007] in, For preliminary joint features, For the facial features in frame t, The eye features of frame t; Regarding the aforementioned preliminary joint features By performing efficient channel attention mechanism operations, joint features are obtained:

[0008] Among them, ECA stands for Efficient Channel Attention Mechanism. For preliminary joint features, This is a joint feature.

[0009] In the above scheme, obtaining the learnable offset based on the joint features to drive the dynamic offset of the reference point specifically includes: For the joint features of the t-th frame Joint features with frame t+1 Global average pooling is performed on each frame and then incorporated into the position embedding P to obtain the pooled representations for frame t and frame (t+1). and :

[0010] in, For the pooling representation of frame t, Let P be the pooled representation of frame t+1, where P is the preset position embedding. For the facial features in frame t, For the eye features of frame t, For the facial features in frame t+1, Let GAP(·) represent the eye features of frame t+1, and let GAP(·) represent the global average pooling operation. According to the pooling representation of the t-th frame Generate a query vector q, and determine a learnable offset based on the query vector q using a multilayer perceptron (MLP):

[0011]

[0012]

[0013] Where q is the query vector. Let LN(·) be the pooling representation of frame t, LN(·) denotes the layer normalization operation, off be the initial offset, and α be the learned weight coefficients. This indicates element-wise multiplication, where offset is the learnable offset.

[0014] In the above scheme, the step of spatially aligning eye features using a learnable offset and an initial reference point to generate aligned eye features specifically includes: Using the center of the feature map as the initial reference point, the learnable offset is applied to this reference point to obtain the offset sampling position; Pooling representation of the (t+1)th frame at the offset sampling position Sampling is performed, and the sampling results, after linear transformation, are used as keys and values. These are then cross-attentionally calculated with the query vector q to obtain the attention-weighted intermediate feature c.

[0015] Where c represents the attention-weighted intermediate feature. Let H represent the value vector corresponding to the i-th attention head, and let H represent the number of attention heads. The projection matrix; The intermediate feature c is added to the query vector q, and then feature enhancement is performed using a feedforward neural network (FFN) to generate aligned eye features. :

[0016] in, To generate aligned eye features, c represents attention-weighted intermediate features, and FFN(·) denotes a feedforward neural network.

[0017] In the above scheme, the step of calculating the difference between the aligned eye features of two adjacent frames to obtain inter-frame difference features specifically includes: The inter-frame difference features are determined by subtracting the aligned eye features of two adjacent frames:

[0018] in, For inter-frame difference features, Generate aligned eye features for frame t+1. Generate aligned eye features for frame t.

[0019] In the above scheme, the step of performing gated fusion on the aligned eye features, the inter-frame difference features, and the facial feature sequences to obtain an adaptively fused feature sequence specifically includes: Based on the eye features aligned to the t-th frame Generate spatial attention map A t :

[0020] Among them, A t This is a spatial attention map. For the t-th aligned eye feature, Conv att (·) Attention map convolution processing; According to the spatial attention map A t Modulation of facial features in frame t To obtain facial features for facial attention regulation Eye features aligned to the t-th frame Perform convolution transformation to obtain the joint feature components of the residuals. The inter-frame difference features Shape expansion is performed to obtain inter-frame difference components. :

[0021] in, Facial features that regulate facial attention. For the joint characteristic components of the residuals, For inter-frame difference components, For the facial features in frame t, A t For spatial attention maps, Conv res (·) performs convolution processing on the residual components, and Expand(·) performs shape expansion processing; For the facial features of the t-th frame , Generate aligned eye features in frame t and inter-frame difference features Perform global average pooling separately, and then concatenate them along the channel dimension to obtain the gated input z. t Dynamic gating weights ω are generated using a multilayer perceptron (MLP) and a sigmoid activation function. t :

[0022] Where [,] represents channel splicing, and B represents the batch size. , These are the joint characteristic components of the residuals. Inter-frame difference components The dynamic weighting coefficients, Generate aligned eye features for frame t. For the facial features in frame t, z t For gated input, ω t Dynamic gating weights; Facial features for facial attention regulation based on the dynamic weighting coefficients Joint characteristic components of residuals Inter-frame difference components We perform weighted fusion to obtain the adaptive fusion features of frame t. :

[0023] The adaptively fused feature sequence is obtained based on the adaptive fusion features of all frames:

[0024] in, For the adaptively fused feature sequence, Let be the adaptive fusion feature of the i-th frame.

[0025] In the above scheme, the step of performing regression calculations based on the time-enhanced feature representation to determine the yaw and pitch angles representing the line-of-sight direction specifically includes: The time-enhanced feature representation is input into a multilayer perceptron model for mapping and regression, and the output is the yaw angle and pitch angle representing the line of sight direction; The multilayer perceptron model is optimized according to the following joint loss function until the iteration condition is met:

[0026] Where B is the batch size. and These represent the predicted and labeled gaze vectors, respectively. and This represents the predicted gaze angle vector and the label gaze angle vector. and This represents hyperparameters.

[0027] This application also proposes a video gaze estimation system, which includes: a feature extraction unit, a feature processing unit, a differential feature acquisition unit, a gated adaptive extraction unit, and a gaze regression output unit; The feature extraction unit is used to extract features from the input eye stitched image and full face image frame by frame, and obtain the eye feature sequence and face feature sequence arranged in chronological order; The feature processing unit is used to concatenate the eye features and face features of each frame based on the eye feature sequence and face feature sequence, and filter the concatenated features to obtain the joint features of the frame; using the center of the feature map of the joint features as the initial reference point, and based on the joint features, obtain a learnable offset that drives the dynamic offset of the reference point; and use the learnable offset and the initial reference point to spatially align the eye features to generate aligned eye features. The difference feature extraction unit is used to calculate the difference between the aligned eye features of two adjacent frames to obtain the inter-frame difference features. A gated adaptive fusion unit is used to perform gated fusion on the aligned eye features, the inter-frame difference features, and the facial feature sequence to obtain an adaptively fused feature sequence. The line-of-sight regression output unit is used to map the adaptively fused feature sequence onto the time dimension to obtain a temporally enhanced feature representation; and to perform regression calculation on the temporally enhanced feature representation to determine the yaw angle and pitch angle representing the line-of-sight direction.

[0028] This application also proposes a readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the following steps: Feature extraction is performed frame by frame on the input stitched eye image and full face image to obtain eye feature sequence and face feature sequence arranged in chronological order; Based on the eye feature sequence and face feature sequence, the eye features and face features of each frame are concatenated to obtain the joint features of that frame; Using the center of the feature map of the joint features as the initial reference point, a learnable offset that drives the dynamic shift of the reference point is obtained based on the joint features. The eye features are spatially aligned using learnable offsets and initial reference points to generate aligned eye features. Calculate the difference between the aligned eye features of two adjacent frames to obtain the inter-frame difference features; Gated fusion is performed on the aligned eye features, the inter-frame difference features, and the facial feature sequence to obtain an adaptively fused feature sequence; The adaptively fused feature sequence is mapped onto the time dimension to obtain a temporally enhanced feature representation. Regression calculations are performed on the time-enhanced feature representation to determine the yaw angle and pitch angle that characterize the line-of-sight direction.

[0029] This application also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor in the following steps: Feature extraction is performed frame by frame on the input stitched eye image and full face image to obtain eye feature sequence and face feature sequence arranged in chronological order; Based on the eye feature sequence and face feature sequence, the eye features and face features of each frame are concatenated to obtain the joint features of that frame; Using the center of the feature map of the joint features as the initial reference point, a learnable offset that drives the dynamic shift of the reference point is obtained based on the joint features. The eye features are spatially aligned using learnable offsets and initial reference points to generate aligned eye features. Calculate the difference between the aligned eye features of two adjacent frames to obtain the inter-frame difference features; Gated fusion is performed on the aligned eye features, the inter-frame difference features, and the facial feature sequence to obtain an adaptively fused feature sequence; The adaptively fused feature sequence is mapped onto the time dimension to obtain a temporally enhanced feature representation. Regression calculations are performed on the time-enhanced feature representation to determine the yaw angle and pitch angle that characterize the line-of-sight direction.

[0030] The embodiments of this invention have the following beneficial effects: First, by extracting features frame-by-frame from the stitched eye image and the full-face image, dual-path feature sequences for the eyes and face are obtained, preserving complementary information for subsequent fusion. Based on this, the eye and face features of each frame are stitched together to form a joint feature, and a learnable offset is learned using the center of the joint feature map as the initial reference point. This offset is then used to spatially align the eye features with the reference point, thereby dynamically compensating for spatial misalignment of the eye region caused by head movement. This ensures that the eye features of adjacent frames correspond precisely at the sub-pixel level, and the inter-frame difference is obtained by calculating the difference between the aligned eye features of adjacent frames. Heterogeneous features can capture dynamic changes in gaze more cleanly and reduce motion artifacts. Furthermore, by gating and fusing aligned eye features, inter-frame difference features, and facial feature sequences, the model can adaptively adjust the contribution ratio of eye details and facial semantics according to the input quality of the current frame. In occluded or blurred scenes, it prioritizes head pose information and makes full use of eye cues in clear scenes. Finally, the adaptively fused feature sequence is mapped to the time dimension for temporal modeling and regression. Through a multilayer perceptron, smooth and accurate yaw and pitch angles are output, thus achieving robust, accurate, and adaptive video gaze estimation in dynamic environments. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] in: Figure 1 This is a schematic diagram of the video gaze estimation method in one embodiment; Figure 2 This is a schematic diagram of the process of stitching eye features and facial features and filtering channel attention in one embodiment of the present invention; Figure 3 This is a schematic diagram of a cross-frame spatial alignment mechanism based on learnable offsets in one embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a gated adaptive fusion module in one embodiment of the present invention; Figure 5 This is a schematic diagram of the architecture of the temporal enhancement and gaze angle regression network in one embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention; however, it will be apparent to those skilled in the art that the invention may be practiced without one or more of these details; in other instances, certain technical features well-known in the art have not been described in order to avoid confusion with the invention. It should be understood that the invention can be practiced in different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided to make the disclosure thorough and complete and to fully convey the scope of the invention to those skilled in the art.

[0035] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. When used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms, unless the context clearly indicates otherwise. The terms “comprising” and / or “including,” when used in this specification, identify the presence of said features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups. When used herein, the term “and / or” includes any and all combinations of the associated listed items.

[0036] To fully understand the present invention, a detailed structure will be presented in the following description in order to illustrate the technical solution proposed by the present invention; optional embodiments of the present invention are described in detail below, however, in addition to these detailed descriptions, the present invention may have other embodiments.

[0037] like Figure 1 As shown, in one embodiment, a video gaze estimation method is provided, which includes steps S101 to S108, detailed below: S101. Extract features from the input eye-stitched image and full-face image frame by frame to obtain eye feature sequence and face feature sequence arranged in chronological order; Specifically, for the input video sequence, the eye stitched region image and the full face region image are extracted for each frame. A pre-trained convolutional neural network is used to extract features from the two types of images, resulting in a time-ordered sequence of eye features and a sequence of face features.

[0038] S102. Based on the eye feature sequence and the face feature sequence, the eye features and face features of each frame are concatenated to obtain the joint features of the frame. As shown in Figure 2, for frame t, the extracted facial features and eye features are first concatenated along the channel dimension to form preliminary joint features. These are then input into an efficient channel attention (ECA) module. This module adaptively learns the importance weights of each channel through one-dimensional convolution and performs weighted calibration on the original features, ultimately outputting the filtered joint features. This process effectively suppresses redundant information unrelated to gaze direction, such as illumination changes and background noise, while highlighting eye details and facial orientation features strongly correlated with gaze direction, providing a cleaner and more discriminative joint feature representation for subsequent cross-frame spatial alignment.

[0039] like Figure 2 As shown, in some embodiments, based on eye feature sequences and face feature sequences, the eye features and face features of each frame are concatenated, and the concatenated features are filtered to obtain the joint features of the frame, specifically including: The facial features and eye features of frame t are concatenated to obtain preliminary joint features. :

[0040] in, For preliminary joint features, For the facial features in frame t, The eye features of frame t; For preliminary joint features By performing efficient channel attention mechanism operations, joint features are obtained:

[0041] Among them, ECA stands for Efficient Channel Attention Mechanism. For preliminary joint features, This is a joint feature.

[0042] By performing efficient channel attention (ECA) on the stitched eyelid features, the importance weights of each channel can be adaptively learned, thereby suppressing redundant information such as background and lighting that is irrelevant to the gaze direction and highlighting eye details and facial orientation features that are strongly related to the gaze direction. At the same time, the ECA mechanism uses one-dimensional convolution to achieve cross-channel interaction, avoiding the information compression loss that may be caused by dimensionality reduction operations in traditional attention modules. It achieves lightweight computation while maintaining the integrity of subtle eye features, providing cleaner and more discriminative joint features for subsequent cross-frame spatial alignment, thereby improving the accuracy and robustness of gaze estimation.

[0043] S103. Using the center of the feature map of the joint features as the initial reference point, and based on the joint features, obtain the learnable offset that drives the dynamic offset of the reference point. In some embodiments, a learnable offset for driving the dynamic offset of the reference point is obtained based on joint features, specifically including: Joint features of frame t Joint features with frame t+1 Global average pooling is performed on each frame and then incorporated into the position embedding P to obtain the pooled representations for frame t and frame (t+1). and :

[0044] in, For the pooling representation of frame t, Let P be the pooled representation of frame t+1, where P is the preset position embedding. For the facial features in frame t, For the eye features of frame t, For the facial features in frame t+1, Let GAP(·) represent the eye features of frame t+1, and let GAP(·) represent the global average pooling operation. Based on the pooling representation of frame t Generate a query vector q, and determine the learnable offset based on the query vector q using a multilayer perceptron (MLP):

[0045]

[0046]

[0047] Where q is the query vector. Let LN(·) be the pooling representation of frame t, LN(·) denotes the layer normalization operation, off be the initial offset, and α be the learned weight coefficients. This indicates element-wise multiplication, where offset is the learnable offset.

[0048] Specifically, by generating a query vector from the joint features of frame t and using a multilayer perceptron to regress the learnable offset, the model can dynamically predict the spatial offset of the eye region between adjacent frames caused by head movement or viewpoint changes. Compared with fixed sampling or global attention mechanisms, this query-based adaptive offset learning enables the model to flexibly adjust the sampling position for different input content, thereby more accurately locating discriminative regions strongly related to the line of sight. This lays the foundation for subsequent sub-pixel-level cross-frame alignment and effectively avoids motion artifacts and background noise introduced by spatial misalignment.

[0049] S104. Spatial alignment of eye features is performed using learnable offsets and initial reference points to generate aligned eye features. Figure 3 This paper demonstrates the complete process of generating aligned eye features from the joint features of frame t and frame t+1. First, global average pooling is performed on the joint features of adjacent frames, and positional embedding is added to obtain pooled representations. Then, a query vector is generated using the pooled representation of frame t, and a learnable offset is obtained through multilayer perceptron regression. This offset is applied to a position with the feature map center as the initial reference point, forming an adaptive sampling position. Next, features are extracted from the pooled representation of frame t+1 according to this sampling position, and after linear transformation, they are used as keys and values. Multi-head cross-attention calculation is performed with the query vector, and enhanced by a feedforward neural network, ultimately outputting eye features that are spatially precisely aligned with frame t. This mechanism can dynamically compensate for cross-frame spatial misalignment caused by head movement or viewpoint changes, achieving sub-pixel-level precise alignment.

[0050] like Figure 3As shown, in some embodiments, eye features are spatially aligned using a learnable offset and an initial reference point to generate aligned eye features, specifically including: Using the center of the feature map as the initial reference point, the learnable offset is applied to this reference point to obtain the offset sampling position; Pooling representation of frame t+1 at the offset sampling position Sampling is performed, and the sampling results, after linear transformation, are used as keys and values. These are then cross-attentionally calculated with the query vector q to obtain the attention-weighted intermediate feature c.

[0051] Where c represents the attention-weighted intermediate feature. Let H represent the value vector corresponding to the i-th attention head, and let H represent the number of attention heads. The projection matrix; The intermediate feature c is added to the query vector q, and then feature enhancement is performed through a feedforward neural network (FFN) to generate aligned eye features. :

[0052] in, To generate aligned eye features, c represents attention-weighted intermediate features, and FFN(·) denotes a feedforward neural network.

[0053] The above embodiment obtains the adaptive sampling position by using the feature map center as the initial reference point and applying a learnable offset. After sampling, cross-attention calculation is performed on the features of adjacent frames, which can achieve sub-pixel level spatial alignment of eye features in adjacent frames. Since the offset is dynamically predicted based on the query vector of the current frame, the model can flexibly adjust the sampling position according to the actual eye movement. Compared with fixed grid sampling or global attention, this mechanism effectively compensates for cross-frame spatial misalignment caused by head movement.

[0054] Meanwhile, by weighting and aggregating the sampled features through multi-head cross-attention, and introducing residual connections and feedforward networks to enhance feature expression, the aligned eye features retain the original query information and integrate the discriminative regions most relevant to the line of sight in adjacent frames. This provides a precise feature basis for subsequent inter-frame difference calculation and significantly reduces motion artifacts and noise interference caused by spatial misalignment.

[0055] S105. Calculate the difference between the aligned eye features of two adjacent frames to obtain the inter-frame difference features; In some embodiments, the difference between the aligned eye features of two adjacent frames is calculated to obtain inter-frame difference features, specifically including: The inter-frame difference features are determined by subtracting the aligned eye features of two adjacent frames:

[0056] in, For inter-frame difference features, Generate aligned eye features for frame t+1. Generate aligned eye features for frame t.

[0057] By directly subtracting the aligned eye features from adjacent frames, dynamic changes in gaze between frames can be effectively extracted. Since the features of the two frames involved in the difference have been spatially aligned at the subpixel level with learnable offset guidance, the spatial misalignment caused by head movement is pre-compensated. Therefore, the calculated inter-frame difference features more purely reflect real gaze changes such as pupil movement and eye rotation, rather than false displacements introduced by head shaking. As an independent component in subsequent gating fusion, this difference feature provides the model with clear temporal dynamic cues, enabling the model to use inter-frame change information to judge the trend of gaze direction changes on the basis of static eye details, thereby enhancing robustness to dynamic scenes such as rapid gaze movement or brief occlusion.

[0058] S106. Gated fusion is performed on the aligned eye features, inter-frame difference features and facial feature sequences to obtain the adaptive fused feature sequence. Figure 4 This is a schematic diagram of the gated adaptive fusion module in one embodiment of the present invention. The input of this module includes three parts: spatially aligned eye features, inter-frame difference features obtained by subtracting the aligned eye features from adjacent frames, and the original facial feature sequence. The module internally has three parallel processing branches. The first branch uses the aligned eye features to generate a spatial attention map, and then uses this attention map to modulate the original facial features, making the model pay more attention to facial regions related to the gaze direction, thus obtaining facial attention modulation features. The second branch performs nonlinear mapping on the aligned eye features through convolutional transformation, extracting residual detail information related to the gaze direction, thus obtaining residual joint feature components. The third branch expands the shape of the inter-frame difference features to make its spatial dimension consistent with the other two branches, thus obtaining inter-frame difference components.

[0059] Meanwhile, the module performs global average pooling on the original facial features, aligned eye features, and inter-frame difference features, compressing each feature map into a global descriptive vector, and concatenating these three vectors along the channel dimension. The concatenated composite vector is input into a multilayer perceptron, processed by a sigmoid activation function, and generates two dynamic gating weight coefficients.

[0060] Finally, the module uses facial attention modulation features as the base components, and weights the residual joint feature components and inter-frame difference components with two dynamic gating weight coefficients respectively. The three components are then summed to obtain the adaptive fusion features of the current frame. The fusion features of all frames are arranged in chronological order to form the adaptively fused feature sequence.

[0061] Through this gating mechanism, the model can automatically adjust the fusion ratio of eye details and facial semantics based on the actual input quality of each frame. When the eyes are clearly visible, the model assigns higher weights to the residual joint feature components to fully utilize subtle gaze cues; when the eyes are occluded or the image is blurred, the model automatically reduces the weight of this component, instead relying on facial features that carry head pose information; simultaneously, it dynamically adjusts the contribution of inter-frame difference components according to the drastic changes in gaze. This adaptive fusion strategy significantly improves the robustness and accuracy of gaze estimation in complex dynamic environments such as changes in lighting, partial occlusion, and large head movements.

[0062] like Figure 4 As shown, in some embodiments, gating fusion is performed on the aligned eye features, inter-frame difference features, and facial feature sequences to obtain an adaptively fused feature sequence, specifically including: Eye features aligned to frame t Generate spatial attention map A t :

[0063] Among them, A t This is a spatial attention map. For the t-th aligned eye feature, Conv att (·) Attention map convolution processing; According to spatial attention diagram A t Modulation of facial features in frame t To obtain facial features for facial attention regulation Eye features aligned to frame t Perform convolution transformation to obtain the joint feature components of the residuals. Inter-frame difference features Shape expansion is performed to obtain inter-frame difference components. :

[0064] in, Facial features that regulate facial attention. For the joint characteristic components of the residuals, For inter-frame difference components, For the facial features in frame t, A t For spatial attention maps, Convres (·) performs convolution processing on the residual components, and Expand(·) performs shape expansion processing; Facial features in frame t , Generate aligned eye features in frame t and inter-frame difference features Perform global average pooling separately, and then concatenate them along the channel dimension to obtain the gated input z. t Dynamic gating weights ω are generated using a multilayer perceptron (MLP) and a sigmoid activation function. t :

[0065] Where [,] represents channel splicing, and B represents the batch size. , These are the joint characteristic components of the residuals. Inter-frame difference components The dynamic weighting coefficients, Generate aligned eye features for frame t. For the facial features in frame t, z t For gated input, ω t Dynamic gating weights; Facial features for facial attention modulation based on dynamic weighting coefficients Joint characteristic components of residuals Inter-frame difference components We perform weighted fusion to obtain the adaptive fusion features of frame t. :

[0066] The adaptively fused feature sequence is obtained based on the adaptive fusion features of all frames:

[0067] in, For the adaptively fused feature sequence, Let be the adaptive fusion feature of the i-th frame.

[0068] Specifically, by constructing three components—attention-modulated facial features, residual eye features, and inter-frame difference features—and introducing a gating mechanism to dynamically adjust the fusion weights of each component, adaptive feature fusion based on input quality is achieved. Specifically, firstly, aligned eye features are used to generate a spatial attention map to modulate the original facial features, enabling the model to focus on gaze-related facial regions and suppress irrelevant backgrounds, resulting in a facial component that preserves global semantics. Simultaneously, aligned eye features are convolutionally transformed to retain eye details as a residual component, and inter-frame difference features are extended and aligned as a dynamically changing component. These three components form a complementary representation of coarse-grained semantics, fine-grained static details, and temporal dynamic cues. Building upon this foundation, dynamic gating weights are learned by pooling facial features, aligned eye features, and inter-frame difference features. This allows the model to assign higher weights to the residual eye component when the eye is clear, thus fully utilizing fine cues. When the eye is occluded or blurred, the weight of this component is automatically reduced, and the model instead relies on the facial component carrying head pose information. Simultaneously, the contribution of the inter-frame difference component is dynamically adjusted according to the degree of gaze change. This input quality-based gating adaptive fusion strategy overcomes the problem of poor adaptability of traditional static weighted fusion methods in different scenarios, significantly improving the robustness and accuracy of gaze estimation in complex dynamic environments such as lighting changes, partial occlusion, and large head movements.

[0069] S107. Map the adaptively fused feature sequence onto the time dimension to obtain a temporally enhanced feature representation; Figure 5 This is a schematic diagram of the architecture of a temporal enhancement and gaze angle regression network in one embodiment of the present invention. Figure 5 As shown above, this step specifically includes: Global average pooling is applied to the fused features of each frame to compress the spatial dimension into a global descriptor, resulting in a compressed sequence. This sequence is then input into a two-layer Transformer encoder, where a multi-head self-attention mechanism is used to model the contextual dependencies between time steps. Learnable positional encodings are also superimposed to preserve inter-frame temporal order information. After processing by the two Transformer encoder layers, a temporally enhanced feature representation is output. This representation integrates the temporal contextual information of consecutive frames, maintaining a stable feature representation even in cases of eye occlusion or blurring in individual frames, thanks to reliable information from adjacent frames.

[0070] S108. Perform regression calculations on the time-enhanced feature representations to determine the yaw and pitch angles that characterize the line-of-sight direction.

[0071] In some embodiments, regression calculations are performed based on time-enhanced feature representations to determine the yaw and pitch angles representing the line-of-sight direction, specifically including: The temporally enhanced feature representation is input into the multilayer perceptron model for mapping and regression, and the output is the yaw angle and pitch angle representing the line of sight direction; The multilayer perceptron model is optimized using the following joint loss function until the iteration condition is met:

[0072] Where B is the batch size. and These represent the predicted and labeled gaze vectors, respectively. and This represents the predicted gaze angle vector and the label gaze angle vector. and This represents hyperparameters.

[0073] By inputting temporal enhancement features into a multilayer perceptron for regression mapping, high-dimensional temporal features can be efficiently and nonlinearly transformed to a low-dimensional angular space, directly outputting the yaw and pitch angles representing the line-of-sight direction. Simultaneously, a joint loss function combining geometric angle error and L1 constraint is used for model optimization. The geometric angle error directly optimizes the geometric accuracy of line-of-sight estimation by calculating the spatial angle between the predicted and true line-of-sight directions, which is highly consistent with the physical nature of the line-of-sight estimation task. The L1 constraint enhances the stability and smoothness of the regression values ​​by regularizing the absolute difference between the predicted and true angles, avoiding severe jitter in single-frame prediction.

[0074] This application also proposes a video gaze estimation system, which includes: a feature extraction unit, a feature processing unit, a differential feature acquisition unit, a gated adaptive extraction unit, and a gaze regression output unit; The feature extraction unit is used to extract features from the input eye stitched image and full face image frame by frame, and obtain the eye feature sequence and face feature sequence arranged in chronological order; The feature processing unit is used to concatenate the eye features and face features of each frame based on the eye feature sequence and face feature sequence, and to filter the concatenated features to obtain the joint features of the frame; using the center of the feature map of the joint features as the initial reference point, a learnable offset is obtained based on the joint features to drive the dynamic offset of the reference point; the eye features are spatially aligned using the learnable offset and the initial reference point to generate aligned eye features. The difference feature extraction unit is used to calculate the difference between the aligned eye features of two adjacent frames to obtain the inter-frame difference features. The gated adaptive fusion unit is used to perform gated fusion on the aligned eye features, inter-frame difference features and facial feature sequences to obtain the adaptively fused feature sequence; The line-of-sight regression output unit is used to map the adaptively fused feature sequence onto the time dimension to obtain a temporally enhanced feature representation; regression calculation is performed on the temporally enhanced feature representation to determine the yaw angle and pitch angle representing the line-of-sight direction.

[0075] This application also proposes a readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the following steps: Feature extraction is performed frame by frame on the input stitched eye image and full face image to obtain eye feature sequence and face feature sequence arranged in chronological order; Based on the eye feature sequence and the face feature sequence, the eye features and face features of each frame are concatenated to obtain the joint features of that frame; Using the center of the feature map of the joint features as the initial reference point, a learnable offset that drives the dynamic shift of the reference point is obtained based on the joint features. The eye features are spatially aligned using learnable offsets and initial reference points to generate aligned eye features. Calculate the difference between the aligned eye features of two adjacent frames to obtain the inter-frame difference features; Gated fusion is performed on the aligned eye features, inter-frame difference features, and facial feature sequences to obtain an adaptively fused feature sequence; The adaptively fused feature sequence is mapped onto the time dimension to obtain a temporally enhanced feature representation; Regression calculations are performed on the temporally enhanced feature representations to determine the yaw and pitch angles that characterize the line-of-sight direction.

[0076] This application also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor in the following steps: Feature extraction is performed frame by frame on the input stitched eye image and full face image to obtain eye feature sequence and face feature sequence arranged in chronological order; Based on the eye feature sequence and the face feature sequence, the eye features and face features of each frame are concatenated to obtain the joint features of that frame; Using the center of the feature map of the joint features as the initial reference point, a learnable offset that drives the dynamic shift of the reference point is obtained based on the joint features. The eye features are spatially aligned using learnable offsets and initial reference points to generate aligned eye features. Calculate the difference between the aligned eye features of two adjacent frames to obtain the inter-frame difference features; Gated fusion is performed on the aligned eye features, inter-frame difference features, and facial feature sequences to obtain an adaptively fused feature sequence; The adaptively fused feature sequence is mapped onto the time dimension to obtain a temporally enhanced feature representation; Regression calculations are performed on the temporally enhanced feature representations to determine the yaw and pitch angles that characterize the line-of-sight direction.

[0077] Those skilled in the art will understand that implementing all or part of the processes in the above embodiments can be accomplished by instructing related hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0078] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] The embodiments described above are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application's patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. The embodiments disclosed above are merely preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made according to the claims of this invention are still within the scope of this invention.

Claims

1. A video gaze estimation method, characterized in that, The method includes: Feature extraction is performed frame by frame on the input stitched eye image and full face image to obtain eye feature sequence and face feature sequence arranged in chronological order; Based on the eye feature sequence and face feature sequence, the eye features and face features of each frame are concatenated to obtain the joint features of that frame; Using the center of the feature map of the joint features as the initial reference point, a learnable offset that drives the dynamic shift of the reference point is obtained based on the joint features. The eye features are spatially aligned using learnable offsets and initial reference points to generate aligned eye features. Calculate the difference between the aligned eye features of two adjacent frames to obtain the inter-frame difference features; Gated fusion is performed on the aligned eye features, the inter-frame difference features, and the facial feature sequence to obtain an adaptively fused feature sequence; The adaptively fused feature sequence is mapped onto the time dimension to obtain a temporally enhanced feature representation. Regression calculations are performed on the time-enhanced feature representation to determine the yaw angle and pitch angle that characterize the line-of-sight direction.

2. The video gaze estimation method according to claim 1, characterized in that, Based on the eye feature sequence and face feature sequence, the eye features and face features of each frame are concatenated, and the concatenated features are filtered to obtain the joint features of that frame, specifically including: The facial features and eye features of frame t are concatenated to obtain preliminary joint features. : in, For preliminary joint features, For the facial features in frame t, The eye features of frame t; Regarding the aforementioned preliminary joint features By performing efficient channel attention mechanism operations, joint features are obtained: Among them, ECA stands for Efficient Channel Attention Mechanism. For preliminary joint features, This is a joint feature.

3. The video gaze estimation method according to claim 2, characterized in that, The process of obtaining a learnable offset based on the joint features to drive the dynamic offset of the reference point specifically includes: For the joint features of the t-th frame Joint features with frame t+1 Global average pooling is performed on each frame and then incorporated into the position embedding P to obtain the pooled representations for frame t and frame (t+1). and : in, For the pooling representation of frame t, Let P be the pooled representation of frame t+1, where P is the preset position embedding. For the facial features in frame t, For the eye features of frame t, For the facial features in frame t+1, Let GAP(·) represent the eye features of frame t+1, and let GAP(·) represent the global average pooling operation. According to the pooling representation of the t-th frame Generate a query vector q, and determine a learnable offset based on the query vector q using a multilayer perceptron (MLP): Where q is the query vector. Let LN(·) be the pooling representation of frame t, LN(·) denotes the layer normalization operation, off be the initial offset, and α be the learned weight coefficients. This indicates element-wise multiplication, where offset is the learnable offset.

4. The video gaze estimation method according to claim 3, characterized in that, The step of spatially aligning eye features using a learnable offset and an initial reference point to generate aligned eye features specifically includes: Using the center of the feature map as the initial reference point, the learnable offset is applied to this reference point to obtain the offset sampling position; Pooling representation of the (t+1)th frame at the offset sampling position Perform sampling and record the sampling results After linear transformation, these are used as keys and values, and cross-attention calculation is performed with the query vector q to obtain attention-weighted intermediate features c: Where c represents the attention-weighted intermediate feature. Let H represent the value vector corresponding to the i-th attention head, and let H represent the number of attention heads. The projection matrix; The intermediate feature c is added to the query vector q, and then feature enhancement is performed using a feedforward neural network (FFN) to generate aligned eye features. : in, To generate aligned eye features, c represents attention-weighted intermediate features, and FFN(·) denotes a feedforward neural network.

5. The video gaze estimation method according to claim 4, characterized in that, The calculation of the difference between the aligned eye features of two adjacent frames to obtain inter-frame difference features specifically includes: The inter-frame difference features are determined by subtracting the aligned eye features of two adjacent frames: in, For inter-frame difference features, Generate aligned eye features for frame t+1. Generate aligned eye features for frame t.

6. The video gaze estimation method according to claim 5, characterized in that, The step of performing gated fusion on the aligned eye features, the inter-frame difference features, and the facial feature sequence to obtain an adaptively fused feature sequence specifically includes: Based on the eye features aligned to the t-th frame Generate spatial attention map A t : Among them, A t This is a spatial attention map. For the t-th aligned eye feature, Conv att (·) Attention map convolution processing; According to the spatial attention map A t Modulation of facial features in frame t To obtain facial features for facial attention regulation Eye features aligned to the t-th frame Perform convolution transformation to obtain the joint feature components of the residuals. The inter-frame difference features Shape expansion is performed to obtain inter-frame difference components. : in, Facial features that regulate facial attention. For the joint characteristic components of the residuals, For inter-frame difference components, For the facial features in frame t, A t For spatial attention maps, Conv res (·) performs convolution processing on the residual components, and Expand(·) performs shape expansion processing; For the facial features of the t-th frame , Generate aligned eye features in frame t and inter-frame difference features Perform global average pooling separately, and then concatenate them along the channel dimension to obtain the gated input z. t Dynamic gating weights ω are generated using a multilayer perceptron (MLP) and a sigmoid activation function. t : Where [,] represents channel splicing, and B represents the batch size. , These are the joint characteristic components of the residuals. Inter-frame difference components The dynamic weighting coefficients, Generate aligned eye features for frame t. For the facial features in frame t, z t For gated input, ω t Dynamic gating weights; Facial features for facial attention regulation based on the dynamic weighting coefficients Joint characteristic components of residuals Inter-frame difference components We perform weighted fusion to obtain the adaptive fusion features of frame t. : The adaptively fused feature sequence is obtained based on the adaptive fusion features of all frames: in, For the adaptively fused feature sequence, Let be the adaptive fusion feature of the i-th frame.

7. The video gaze estimation method according to claim 6, characterized in that, The regression calculation based on the time-enhanced feature representation to determine the yaw and pitch angles representing the line-of-sight direction specifically includes: The time-enhanced feature representation is input into a multilayer perceptron model for mapping and regression, and the output is the yaw angle and pitch angle representing the line of sight direction; The multilayer perceptron model is optimized according to the following joint loss function until the iteration condition is met: Where B is the batch size. and These represent the predicted and labeled gaze vectors, respectively. and This represents the predicted gaze angle vector and the label gaze angle vector. and This represents hyperparameters.

8. A video gaze estimation system, characterized in that, The system includes: a feature extraction unit, a feature processing unit, a differential feature acquisition unit, a gated adaptive extraction unit, and a gaze regression output unit; The feature extraction unit is used to extract features from the input eye stitched image and full face image frame by frame, and obtain the eye feature sequence and face feature sequence arranged in chronological order; The feature processing unit is used to concatenate the eye features and face features of each frame based on the eye feature sequence and face feature sequence, and filter the concatenated features to obtain the joint features of the frame; using the center of the feature map of the joint features as the initial reference point, and based on the joint features, obtain a learnable offset that drives the dynamic offset of the reference point; and use the learnable offset and the initial reference point to spatially align the eye features to generate aligned eye features. The difference feature extraction unit is used to calculate the difference between the aligned eye features of two adjacent frames to obtain the inter-frame difference features. A gated adaptive fusion unit is used to perform gated fusion on the aligned eye features, the inter-frame difference features, and the facial feature sequence to obtain an adaptively fused feature sequence. The line-of-sight regression output unit is used to map the adaptively fused feature sequence onto the time dimension to obtain a temporally enhanced feature representation; and to perform regression calculation on the temporally enhanced feature representation to determine the yaw angle and pitch angle representing the line-of-sight direction.

9. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, the processor performs the steps of the method as described in any one of claims 1 to 7.

10. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 7.

Images