A driver gaze direction estimation method based on structural inductive bias

Abstract

This invention provides a driver gaze direction estimation method based on structural inductive bias, relating to the technical field of driving safety. The method includes: extracting multi-scale features from face images using a backbone network; introducing an adaptive directional compression attention enhancement module, while simultaneously enhancing local information using convolutional auxiliary kernels; introducing an ordered multi-scale aggregator module, achieving directed cross-scale information fusion through sequential recursion and gating mechanisms, followed by mean pooling along the sequence length dimension to obtain a global representation; employing LayerNorm for sample-level normalization at the regression head entry point to ensure stable input distribution under mini-batch and strong domain bias conditions; and using a small perceptron consisting of two fully connected layers and GELU to map yaw and pitch angles, configuring isomorphic auxiliary heads for each scale to achieve gaze estimation. This invention's method has linear complexity and does not rely on global aggregation tokens or explicit positional encoding.

Description

Technical Field

[0001] This invention relates to the technical field of driving safety, and more particularly to a method for estimating driver gaze direction based on structural inductive bias. Background Technology

[0002] Existing in-vehicle gaze estimation methods mainly fall into three categories, all of which have significant limitations and are difficult to meet the needs of complex driving scenarios. Feature-based methods rely on facial feature point detection and head pose estimation, which is complex and prone to error accumulation, and has weak robustness to occlusion. For example, while feature point detection-based methods achieve a classification accuracy of 94.9%, they are not robust to extreme head poses. Model-based methods require 3D modeling and precise camera calibration, resulting in high computational complexity and poor cross-individual generalization. For example, 3D modeling-based methods use 3D head models to decouple facial movements, achieving an EOR detection accuracy of over 90%, but significantly increasing cross-individual errors, and multimodal dependence increases deployment costs. Deep learning-based methods use pre-trained backbone networks, over-relying on general priors and underutilizing multi-scale features. For example, GazePTR uses only top-level features, leading to decreased accuracy in occluded scenes. Furthermore, the models are bulky (e.g., GazeDPTR reaches 103MB) or rely on multimodal inputs, resulting in significant errors in occluded scenarios such as when wearing sunglasses (e.g., GazeTR error 17.49°). Common defects include difficulty in balancing accuracy and robustness, imbalance between computational efficiency and model volume, and insufficient anti-interference ability of multimodal dependence or single-modal mode. This invention breaks through the above bottlenecks by multi-scale feature interaction, lightweight design and single-modal end-to-end regression. Summary of the Invention

[0003] Based on the technical problems mentioned in the background section, this paper proposes a driver gaze direction estimation method based on structural inductive bias. A novel task-aligned collaborative optimization method is presented for end-to-end in-vehicle gaze estimation using only facial images as input. Experimental results show that the proposed method achieves state-of-the-art (SOTA) performance on multiple metrics and achieves best performance in real-world in-vehicle scenarios while maintaining competitive computational complexity.

[0004] The technical means employed in this invention are as follows:

[0005] A method for estimating driver gaze direction based on structural inductive bias includes the following steps: Step 1: Extract multi-scale features from face images using a densely connected backbone network; Step 2: Introduce an adaptive row compression enhanced attention module. First, perform row compression along the horizontal direction, and then apply multi-head self-attention along the vertical axis to model cross-row long-range dependencies. At the same time, enhance local information based on the auxiliary kernel of convolution. Step 3: Introduce an ordered multi-scale aggregator module, which uses a selective state-space model and structured convolution kernels as the core. It achieves directed cross-scale information fusion through sequential recursion and gating mechanism. Then, it performs mean pooling in the sequence length dimension to obtain the global representation. Step 4: At the regression head entry point, LayerNorm is used for sample-level normalization to keep the input distribution stable under mini-batch and strong domain offset conditions. Subsequently, a small perceptron consisting of two fully connected layers and GELU is used to complete the mapping of yaw and pitch angles, and an auxiliary head is configured with the same structure for each scale to achieve gaze estimation.

[0006] Furthermore, the backbone network is DenseNet.

[0007] Furthermore, step 2 includes the following steps: Step 21: Obtain the query Q, key K, and value V of the input feature map through 1×1 convolution and BatchNorm; Step 22, for By performing mean compression along the W dimension, we obtain , ;in, Queries representing horizontal row information; The key that represents horizontal row information; This represents the value of the horizontal row information; ; ; and These represent the dimensions of the key and value, respectively. Step 23, for and Perform position embedding and obtain the position-embedded sequence. ; Step 24: After embedding at the specified position, establish long-range dependencies between rows using standard scaled dot product attention. Step 25: In the attention weight calculation of step 24, a learnable adaptive scaling factor is introduced to handle and adapt to different types and levels of noise by adjusting the size of the scaling factor. Step 26: Embed the position into the sequence Merge multiple positions and perform a linear mapping; Step 27: Expand it in W dimensions using a broadcast operation, and then... After element-wise addition, a 1×1 convolution is performed to inject a row-level global context at each pixel position; Step 28: Use a convolution-based auxiliary kernel to enhance spatial details.

[0008] Furthermore, step 28 includes the following steps: Step 281, , and Concatenate along the channel dimension and use a 3×3 depthwise separable convolution to aggregate local information within each channel; Step 282: Adjust the dimension from (2Cqk+Cv) to C using linear projection, activation function, and batch normalization, and generate detail enhancement weights. ; Step 283: Generate gating weights for the output Y of the row-direction squeezing attention branch using the Sigmoid function, and combine them with the detail enhancement weights. Perform element-wise multiplication to obtain the combined output.

[0009] Furthermore, step 3 includes the following steps: Step 31: Replace the Transformer encoder with a single-layer Mamba2; the continuous system passes through intermediate hidden states. Input sequence Mapped to output : ; in, Represents the rate of change of state; Represents the state transition matrix; Indicates projection parameters; Step 32: Introduce a time step to the continuous system. Perform zero-order preserved discretization, and convert the continuous parameters , Convert to discrete parameters , : ; in, Representing the identity matrix; thus, the discrete model is obtained as follows: ; in, , Representing discrete time points Input and output; recursively equivalent to one-dimensional causal convolution: ; in, Represents the input sequence Length, Represents a structured convolution kernel; Step 33: Introduce state-space duality to Mamba2, rewrite the time recursion as a linear operator centered on matrix multiplication, and... Simplifying to scalars to improve parallelism and numerical stability; its causal mapping is: ; in, Represents the input sequence; Indicates the output sequence; The matrix is ​​a strictly lower triangular matrix generated from system parameters; SSD reveals an intrinsic equivalence with linear attention:

[0010] in, Represents the "query" projection. Represents the projection of the "key". Represents "value"; It is a strictly lower triangular causal linear operator; Step 34: Finally, perform mean pooling on the length dimension as the global representation.

[0011] Compared with the prior art, the present invention has the following advantages: This invention employs a densely connected backbone network to achieve cross-layer feature reuse, continuously applying shallow details to deeper layers, thereby shortening the gradient path and providing structural support for subsequent modules. The layer-normalized multilayer perceptron regression head suppresses fluctuations in the input distribution to achieve stable regression, while the isomorphic auxiliary head enhances semantic consistency and detail visibility by improving nonlinear and robust gradient propagation.

[0012] This invention introduces an adaptive row compression enhanced attention module that combines row compression with vertical axis multi-head self-attention to explicitly model long-distance dependencies across rows. A convolution-based detail enhancement kernel preserves local edges and textures, improving the semantic consistency of tokens across scales and increasing [the effectiveness of this technology].

[0013] This invention introduces an ordered multi-scale aggregator, which performs directed cross-scale fusion through ordered loops and gating mechanisms, constructing a stable and accurate feature representation. This method has linear complexity and does not rely on global aggregation tokens or explicit positional encoding. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of the overall process of the present invention.

[0016] Figure 2 This is a network structure diagram of the backbone network of this invention. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0018] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] like Figure 1 As shown, this invention provides a method for estimating driver gaze direction based on structural inductive bias, comprising the following steps: Step 1: Extract multi-scale features from face images using a densely connected backbone network (DenseNet); Step 2: Introduce an adaptive row compression enhanced attention module. First, perform row compression along the horizontal direction, then apply multi-head self-attention along the vertical axis to model long-range dependencies across rows. Simultaneously, enhance local information based on an auxiliary kernel of convolution. Step 2 includes the following steps: Step 21: Obtain the query Q, key K, and value V of the input feature map through 1×1 convolution and BatchNorm; Step 22, for By performing mean compression along the W dimension, we obtain , ;in, Queries representing horizontal row information; The key that represents horizontal row information; This represents the value of the horizontal row information; ; ; and These represent the dimensions of the key and value, respectively. Step 23, for and Perform position embedding and obtain the position-embedded sequence. ; Step 24: After embedding at the specified position, establish long-range dependencies between rows using standard scaled dot product attention. Step 25: In the attention weight calculation of step 24, a learnable adaptive scaling factor is introduced to handle and adapt to different types and levels of noise by adjusting the size of the scaling factor. Step 26: Embed the position into the sequence Merge multiple positions and perform a linear mapping; Step 27: Expand it in W dimensions using a broadcast operation, and then... After element-wise addition, a 1×1 convolution is performed to inject a row-level global context at each pixel position; Step 28: Enhance spatial details using a convolution-based auxiliary kernel. Step 28 includes the following steps: Step 281, , and Concatenate along the channel dimension and use a 3×3 depthwise separable convolution to aggregate local information within each channel; Step 282: Adjust the dimension from (2Cqk+Cv) to C using linear projection, activation function, and batch normalization, and generate detail enhancement weights. ; Step 283: Generate gating weights for the output Y of the row-direction squeezing attention branch using the Sigmoid function, and combine them with the detail enhancement weights. Perform element-wise multiplication to obtain the combined output.

[0020] Step 3: Introduce an ordered multi-scale aggregator module, using a selective state-space model and structured convolutional kernels as the core. Directed cross-scale information fusion is achieved through sequential recursion and gating mechanisms. Subsequently, mean pooling is performed along the sequence length dimension to obtain the global representation. Step 3 includes the following steps: Step 31: Replace the Transformer encoder with a single-layer Mamba2; the continuous system passes through intermediate hidden states. Input sequence Mapped to output : ; in, Represents the rate of change of state; Represents the state transition matrix; Indicates projection parameters; Step 32: Introduce a time step to the continuous system. Perform zero-order preserved discretization, and convert the continuous parameters , Convert to discrete parameters , : ; in, Representing the identity matrix; thus, the discrete model is obtained as follows: ; in, , Representing discrete time points Input and output; recursively equivalent to one-dimensional causal convolution: ; in, Represents the input sequence Length, Represents a structured convolution kernel; Step 33: Introduce state-space duality to Mamba2, rewrite the time recursion as a linear operator centered on matrix multiplication, and... Simplifying to scalars to improve parallelism and numerical stability; its causal mapping is: ; in, Represents the input sequence; Indicates the output sequence; The matrix is ​​a strictly lower triangular matrix generated from system parameters; SSD reveals an intrinsic equivalence with linear attention:

[0021] in, Represents the "query" projection. Represents the projection of the "key". Represents "value"; It is a strictly lower triangular causal linear operator; Step 34: Finally, perform mean pooling on the length dimension as the global representation.

[0022] Step 4: At the regression head entry point, LayerNorm is used for sample-level normalization to keep the input distribution stable under mini-batch and strong domain offset conditions. Subsequently, a small perceptron consisting of two fully connected layers and GELU is used to complete the mapping of yaw and pitch angles, and an auxiliary head is configured with the same structure for each scale to achieve gaze estimation.

[0023] Example 1 The connection relationships and working process of the control circuit in this invention are described in detail with examples from practical application fields.

[0024] The dataset was constructed and divided into training and testing sets: The IVGaze dataset contains 44,705 images from 125 subjects, all captured by a 1280×800 resolution infrared camera in real-world driving scenarios. In the preprocessing stage, a general face detector was used to locate facial regions, and the detected faces were cropped and normalized. Furthermore, the ground truth dataset includes multimodal information such as 3D gaze vectors, category labels, camera pose, and head pose. The dataset includes various lighting conditions, facial occlusion, and head pose variations. The gaze angle covers the horizontal direction within the normalized space. 50° to 90°, vertical direction The LBW dataset contains 123,297 driver facial images from 28 subjects, 3D gaze vectors acquired and calibrated using a head-mounted eye tracker, road-facing scene images, and corresponding 2D gaze points. During preprocessing, facial images were uniformly cropped to a fixed size of 224×224. Data collection covered various road types and driving conditions, exhibiting wide variations in lighting and head posture. The gaze angle covers the horizontal direction within the normalized space. 50° to 30°, vertical direction 20° to 10°.

[0025] Hyperparameter settings: The experiment was conducted on a Windows 10 operating system, using CUDA 12.6, OpenCV 4.10.0, and PyCharm 2025.1.2. The hardware platform included an Intel Core i9-14900 CPU, an NVIDIA RTX 3090 GPU, and 32GB of RAM. For hyperparameters, the number of threads and batch size were 4 and 64, respectively, with 100 epochs. The initial learning rate was set to 0.001 and halved at the 60th epoch; all other hyperparameters remained at their default values.

[0026] according to Figure 2 A multi-scale adaptive feature modulation in-vehicle gaze estimation model was constructed, including a backbone network, ARSEA, OMSA modules, and RegressionHead and AuxiliaryHeads modules. The network was then trained using the dataset constructed in section 1. During training, the number of threads was set to 4, the batch size to 64, and the total number of iterations to 100. The optimizer was Adam, with an initial learning rate of 0.001, which was reduced to half its initial value at the 60th training epoch.

[0027] Evaluation Metrics: In in-vehicle gaze estimation tasks, the mean angle error (MAE) is typically used as the primary performance metric. The angle error represents the angle between the predicted gaze vector and the ground truth gaze vector, and MAE represents the average angle error of all samples in the validation set. A smaller MAE indicates better model performance. To more precisely characterize the error distribution, mean accuracy is used as a supplementary metric. A sample is considered a correct prediction when its angle error is less than k°. Mean accuracy is defined as the proportion of correct samples in the validation set. Commonly used thresholds k ∈ {2, 4, 6, 8}.

[0028] To evaluate the model's performance on the aforementioned datasets, three-fold and five-fold cross-validation based on subject splitting were employed on IVGaze and LBW, respectively. This setup ensures that the validation set for each fold includes only subjects not present in the training set, thus providing a more objective measure of the model's generalization ability to new individuals. This evaluation protocol conforms to standard practices in the field of gaze estimation, facilitating fair comparisons with existing work and future research.

[0029] The test set was evaluated both subjectively and objectively. This invention comprehensively compared the latest in-vehicle gaze estimation models in terms of mean angle error (MAE), mean accuracy, model size, and FLOPs, including Fullface series, DGW, Gaze360, Xgaze, GazeDPTR series, UnMoDE, FIFA, and STAGE. All models were initialized with officially released pre-trained weights in the backbone network.

[0030] The proposed method significantly outperforms other state-of-the-art in-vehicle gaze estimation models across several key metrics. To verify the model's generalizability and versatility, it was trained and tested on the LBW dataset. Results show that the proposed model achieves state-of-the-art performance across five ranges. Furthermore, the method does not rely on multimodal information such as head or camera pose, facilitating deployment and application on mobile devices.

[0031] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the above embodiments of the present invention, the descriptions of each embodiment have their own emphasis; parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the disclosed technical content in the several embodiments provided in this application can be implemented in other ways.

[0032] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for estimating driver gaze direction based on structural inductive bias, characterized in that, Includes the following steps: Step 1: Extract multi-scale features from face images using a densely connected backbone network; Step 2: Introduce an adaptive row compression enhanced attention module. First, perform row compression along the horizontal direction, and then apply multi-head self-attention along the vertical axis to model cross-row long-range dependencies. At the same time, enhance local information based on the auxiliary kernel of convolution. Step 3: Introduce an ordered multi-scale aggregator module, which uses a selective state-space model and structured convolution kernels as the core. It achieves directed cross-scale information fusion through sequential recursion and gating mechanism. Then, it performs mean pooling in the sequence length dimension to obtain the global representation. Step 4: At the regression head entry point, LayerNorm is used for sample-level normalization to keep the input distribution stable under mini-batch and strong domain offset conditions. Subsequently, a small perceptron consisting of two fully connected layers and GELU is used to complete the mapping of yaw and pitch angles, and an auxiliary head is configured with the same structure for each scale to achieve gaze estimation.

2. The driver gaze direction estimation method based on structural inductive bias according to claim 1, characterized in that, The backbone network is DenseNet.

3. The driver gaze direction estimation method based on structural inductive bias according to claim 1, characterized in that, Step 2 includes the following steps: Step 21: Obtain the query Q, key K, and value V of the input feature map through 1×1 convolution and BatchNorm; Step 22, for By performing mean compression along the W dimension, we obtain , ;in, Queries representing horizontal row information; The key that represents horizontal row information; This represents the value of the horizontal row information; ; ; and These represent the dimensions of the key and value, respectively. Step 23, for and Perform position embedding and obtain the position-embedded sequence. ; Step 24: After embedding at the specified position, establish long-range dependencies between rows using standard scaled dot product attention. Step 25: In the attention weight calculation of step 24, a learnable adaptive scaling factor is introduced to handle and adapt to different types and levels of noise by adjusting the size of the scaling factor. Step 26: Embed the position into the sequence Merge multiple positions and perform a linear mapping; Step 27: Expand it in W dimensions using a broadcast operation, and then... After element-wise addition, a 1×1 convolution is performed to inject a row-level global context at each pixel position; Step 28: Use a convolution-based auxiliary kernel to enhance spatial details.

4. The driver gaze direction estimation method based on structural inductive bias according to claim 1, characterized in that, Step 28 includes the following steps: Step 281, , and Concatenate along the channel dimension and use a 3×3 depthwise separable convolution to aggregate local information within each channel; Step 282: Adjust the dimension from (2Cqk+Cv) to C using linear projection, activation function, and batch normalization, and generate detail enhancement weights. ; Step 283: Generate gating weights for the output Y of the row-direction squeezing attention branch using the Sigmoid function, and combine them with the detail enhancement weights. Perform element-wise multiplication to obtain the combined output.

5. The driver gaze direction estimation method based on structural inductive bias according to claim 1, characterized in that, Step 3 includes the following steps: Step 31: Replace the Transformer encoder with a single-layer Mamba2; the continuous system passes through intermediate hidden states. Input sequence Mapped to output : ； in, Represents the rate of change of state; Represents the state transition matrix; Indicates projection parameters; Step 32: Introduce a time step to the continuous system. Perform zero-order preserved discretization, and convert the continuous parameters , Convert to discrete parameters , : ； in, Representing the identity matrix; thus, the discrete model is obtained as follows: ； in, , Representing discrete time points Input and output; recursively equivalent to one-dimensional causal convolution: ； in, Represents the input sequence Length, Represents a structured convolution kernel; Step 33: Introduce state-space duality to Mamba2, rewrite the time recursion as a linear operator centered on matrix multiplication, and... Simplifying to scalars to improve parallelism and numerical stability; its causal mapping is: ； in, Represents the input sequence; Indicates the output sequence; The matrix is ​​a strictly lower triangular matrix generated from system parameters; SSD reveals an intrinsic equivalence with linear attention: in, Represents the "query" projection. Represents the projection of the "key". Represents "value"; It is a strictly lower triangular causal linear operator; Step 34: Finally, perform mean pooling on the length dimension as the global representation.

Images