A method for estimating line of sight by fusing binocular geometric features
By fusing binocular geometric features into a gaze estimation method, and using a geometric prior network and a residual network for dynamic calibration, the problem of feature interference caused by physiological convergence and divergence and optical distortion in near-field interactions is solved, and high-precision gaze estimation is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JIANGXI NORMAL UNIV
- Filing Date
- 2026-05-11
- Publication Date
- 2026-06-09
AI Technical Summary
Existing 3D gaze estimation techniques suffer from feature interference due to binocular physiological convergence and divergence movements and optical distortions during near-field interactions, leading to decreased estimation accuracy. Furthermore, the lack of an effective binocular non-conjugate calibration mechanism makes it difficult to maintain high accuracy in complex environments.
By fusing binocular geometric features, a geometric prior network is used to extract multi-scale left and right eye geometric feature tensors, calculate residual feature maps and perform dynamic affine transformations, and combine residual networks and cross-attention modules to perform semantic association fusion to generate a three-dimensional gaze feature vector.
Without requiring manual annotation of key points, it effectively eliminates the effects of physiological convergence and divergence and optical distortion, improving the robustness and accuracy of line-of-sight estimation, and exhibiting good adaptability, especially in complex interactive environments.
Smart Images
Figure CN122176784A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of behavioral feature recognition technology, and in particular relates to a gaze estimation method that integrates binocular geometric features. Background Technology
[0002] Current 3D gaze estimation is used to infer the gaze direction of the human eye in 3D space. The mainstream technical approach is to acquire human eye images, extract appearance features using deep neural networks, and convert them into gaze direction vectors. With the increase in near-field interaction applications, the system needs to process visual information within 60 cm. Under near-field conditions, the human eye produces physiological convergence and divergence movements, resulting in an angle between the visual axes of the left and right eyes. At this time, the geometric distribution of the binocular images no longer satisfies the mirror symmetry relationship. Combined with the asymmetry caused by unilateral illumination and the difference in lens refraction, the features of the left and right eyes exhibit non-conjugate spatial offset in the latent space. Existing binocular fusion strategies mostly adopt direct stitching at the feature level. Their underlying logic presupposes that the features of both eyes have ideal conjugate symmetry. This scheme, which forcibly superimposes feature tensors with physical deviations, produces feature signal interference within the model. When dealing with near gaze or unilateral strong light interference, the gaze regression network is affected by the misalignment of the feature manifold, resulting in a decrease in estimation accuracy.
[0003] Geometric model-based methods rely on high-precision eye key point annotation, but the cost of obtaining large-scale iris boundary or eyelid contour labels in real-world environments is high. Existing self-supervised improvement paths lack effective binocular non-conjugate calibration mechanisms, making it difficult to eliminate feature interference introduced by physiological convergence and optical distortion from the underlying mechanism. Software-level feature fusion logic also has shortcomings. For example, Chinese invention patent application CN107545302A discloses a method for calculating the gaze direction of human left and right eye images by using dual-channel and single-channel deep neural networks to extract binocular features and implement connection fusion.
[0004] Therefore, the technical problem to be solved by this invention is how to extract binocular non-conjugate residuals and construct a feature manifold calibration mechanism to solve the feature interference problem caused by physiological convergence and divergence motion and optical distortion without the need for manual annotation of key points, thereby improving the estimation accuracy of the system under complex interactive conditions. Summary of the Invention
[0005] This invention proposes a gaze estimation method that integrates binocular geometric features, comprising the following steps: Step 101: Obtain a face image containing the regions of both eyes; Step 102: Extract multi-scale left eye geometric feature tensors and multi-scale right eye geometric feature tensors from face images through a pre-trained geometric prior network. The geometric prior network is a feature extraction model based on a variational inference architecture. Step 103: Perform feature shift residual calculation on the multi-scale left eye geometric feature tensor and the multi-scale right eye geometric feature tensor to obtain a residual feature map characterizing the physiological convergence and divergence shift of the two eyes and the optical distortion component. The residual feature map is composed of the absolute value of the element-wise difference between the multi-scale left eye geometric feature tensor and the multi-scale right eye geometric feature tensor. Step 104: The residual feature map is input to the parameter mapping unit and subjected to global average pooling and linear transformation to obtain dynamic affine transformation parameters for calibrating binocular asymmetry. Step 105: Spatial resampling and feature point alignment are performed on the multi-scale right eye geometric feature tensor according to the dynamic affine transformation parameters to obtain the joint geometric feature tensor aligned with the multi-scale left eye geometric feature tensor under the reference visual axis. Step 106: Extract appearance feature tensors from face images using a residual network, and input the appearance feature tensors and joint geometric feature tensors into a cross-attention module for semantic association fusion to obtain a three-dimensional gaze feature vector. Step 107: Input the 3D gaze feature vector into the regression network to obtain the 3D gaze direction parameters.
[0006] Preferably, step 103 includes the following sub-steps: Step 1031, determining whether the resolutions of the multi-scale left-eye geometric feature tensor and the multi-scale right-eye geometric feature tensor are consistent; Step 1032, when the determination result is that the resolutions are inconsistent, scaling the multi-scale left-eye geometric feature tensor and the multi-scale right-eye geometric feature tensor to a uniform size using a bilinear interpolation algorithm; Step 1033, calculating the element-wise difference between the multi-scale left-eye geometric feature tensor and the multi-scale right-eye geometric feature tensor after scaling to a uniform size and taking the absolute value to obtain the residual feature map.
[0007] Preferably, the geometric prior network in step 102 is trained in the following way: the probability distribution of the eye training image is projected onto the latent space using an encoder, and the statistical constraint ability of the geometric prior network on the eye geometric topology is based on minimizing the Kohlbek-Leibler divergence between the probability distribution and the preset Gaussian distribution.
[0008] Preferably, step 106 includes the following sub-steps: Step 1061: Using the joint geometric feature tensor as the query vector, and the appearance feature tensor as the key vector and value vector; Step 1062: Identifying local feature regions in the appearance feature tensor affected by ambient illumination by calculating the cross-correlation coefficient between the query vector and the key vector; Step 1063: Performing a weighted summation of the value vector based on the cross-correlation coefficient, and using the structured prior information in the joint geometric feature tensor to complete the features of the local feature regions, thereby obtaining the three-dimensional line-of-sight feature vector.
[0009] Preferably, the residual network in step 106 adopts a hierarchical downsampling structure to extract semantic feature components containing iris texture and scleral reflectivity from the face image, and maps the semantic feature components to a feature space consistent with the dimension of the joint geometric feature tensor.
[0010] Preferably, step 105 further includes: constructing a 2×3 affine transformation matrix based on dynamic affine transformation parameters, and using the affine transformation matrix to perform coordinate mapping on the multi-scale right eye geometric feature tensor, so that the multi-scale right eye geometric feature tensor presents mirror conjugate symmetry with respect to the multi-scale left eye geometric feature tensor in the feature space.
[0011] Preferably, the regression network in step 107 includes cascaded fully connected layers and nonlinear activation units, which map the three-dimensional line-of-sight feature vectors to horizontal yaw angles and vertical pitch angles in degrees.
[0012] Preferably, after acquiring the face image, the method further includes histogram equalization and normalization to a 224×224 pixel size to eliminate the interference of ambient light non-uniformity on feature extraction.
[0013] Preferably, step 105 eliminates the binocular visual axis non-parallelism caused by the fixation distance in the range of 30cm to 50cm at the feature space level by establishing a nonlinear coordinate mapping in the multi-scale right eye geometric feature tensor.
[0014] Compared with existing technologies, the gaze estimation method of this invention, which integrates binocular geometric features, has the following advantages: 1. In gaze estimation by fusing binocular geometric features, a dynamic calibration mechanism based on non-conjugate residuals is established before binocular feature fusion. The pixel-level difference between the left and right eye geometric feature tensors is used to resolve the residual feature map that represents the physiological convergence and divergence of the two eyes and unilateral optical distortion. A two-dimensional affine transformation matrix is generated based on the residual feature map. Mesh resampling and coordinate alignment are performed on one of the eye features in the latent space. This eliminates the feature manifold misalignment caused by the non-parallelism of the visual axes of the two eyes in near-field gaze scenarios at the data level. It avoids the gradient cancellation and feature interference caused by traditional stitching methods when processing heterogeneous features, so that the subsequent fusion operator can stably capture the physically consistent binocular geometric correlation.
[0015] 2. Establish a multi-mechanism collaborative modeling path between geometric topological constraints and visual texture information. Input the implicit geometric structure features extracted by the prior network and the facial appearance features extracted by the residual network into the cross-attention module. Use the structured prior of geometric distribution to guide the semantic alignment of appearance details. When facing non-ideal imaging conditions such as unilateral strong light reflection, lens refraction or partial occlusion, the system can adaptively complete and adjust the features of the disturbed area through binocular symmetry logic. Thus, without increasing the external perception hardware, the robustness of the gaze estimation process in complex interactive environments is improved by utilizing the complementarity between features.
[0016] 3. A geometric prior network architecture based on probability distribution modeling is adopted. During the training phase, the divergence constraint between the posterior and prior networks is used to enable the encoder to learn the statistical distribution law and kinematic constraints of the eye geometry. During the inference phase, only the original image data needs to be input to output multi-scale geometric features with structural consistency. This reduces the dependence of the gaze estimation model on large-scale manually annotated key point data in real scenes from the algorithm logic level, shortens the migration and deployment cycle of the system on different terminal devices, and demonstrates good engineering application value. Attached Figure Description
[0017] Figure 1 This is a flowchart of the gaze estimation method of the present invention that integrates binocular non-conjugate feature calibration; Figure 2 This is a system operation architecture diagram of the present invention that combines hardware awareness and multi-mechanism feature fusion. Detailed Implementation
[0018] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.
[0019] It should be noted that all directional and positional terms used in this invention, such as: up, down, left, right, front, back, vertical, horizontal, inner, outer, top, bottom, transverse, longitudinal, center, etc., are only used to explain the relative positional relationship and connection between components in a specific state (as shown in the accompanying drawings). They are only for the convenience of describing this invention and do not require that this invention be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention. In addition, the descriptions of "first," "second," etc., in this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated.
[0020] In the description of this invention, unless otherwise explicitly specified and limited, the terms installation, connection, and linking should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections; they can refer to direct connections or indirect connections through an intermediate medium; they can refer to the internal connection of two components. For those skilled in the art, the specific meaning of the above terms in this invention can be understood according to the specific circumstances.
[0021] In the description of this specification, references to the terms "an embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example, and the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0022] A gaze estimation method incorporating binocular geometric features includes the following steps: Step 101: Obtain a face image containing the regions of both eyes; Step 102: Extract multi-scale left eye geometric feature tensors and multi-scale right eye geometric feature tensors from face images through a pre-trained geometric prior network. The geometric prior network is a feature extraction model based on a variational inference architecture. Step 103: Perform feature shift residual calculation on the multi-scale left eye geometric feature tensor and the multi-scale right eye geometric feature tensor to obtain a residual feature map characterizing the physiological convergence and divergence shift of the two eyes and the optical distortion component. The residual feature map is composed of the absolute value of the element-wise difference between the multi-scale left eye geometric feature tensor and the multi-scale right eye geometric feature tensor. Step 104: The residual feature map is input to the parameter mapping unit and subjected to global average pooling and linear transformation to obtain dynamic affine transformation parameters for calibrating binocular asymmetry. Step 105: Spatial resampling and feature point alignment are performed on the multi-scale right eye geometric feature tensor according to the dynamic affine transformation parameters to obtain the joint geometric feature tensor aligned with the multi-scale left eye geometric feature tensor under the reference visual axis. Step 106: Extract appearance feature tensors from face images using a residual network, and input the appearance feature tensors and joint geometric feature tensors into a cross-attention module for semantic association fusion to obtain a three-dimensional gaze feature vector. Step 107: Input the 3D gaze feature vector into the regression network to obtain the 3D gaze direction parameters.
[0023] Preferably, step 103 includes the following sub-steps: Step 1031, determining whether the resolutions of the multi-scale left-eye geometric feature tensor and the multi-scale right-eye geometric feature tensor are consistent; Step 1032, when the determination result is that the resolutions are inconsistent, scaling the multi-scale left-eye geometric feature tensor and the multi-scale right-eye geometric feature tensor to a uniform size using a bilinear interpolation algorithm; Step 1033, calculating the element-wise difference between the multi-scale left-eye geometric feature tensor and the multi-scale right-eye geometric feature tensor after scaling to a uniform size and taking the absolute value to obtain the residual feature map.
[0024] Preferably, the calculation of the residual feature map in step 103 follows the formula below: ,in, For residual feature maps, For the multi-scale left eye geometric feature tensor, For the multi-scale right eye geometric feature tensor.
[0025] Preferably, the geometric prior network in step 102 is trained in the following way: the probability distribution of the eye training image is projected onto the latent space using an encoder, and the statistical constraint ability of the geometric prior network on the eye geometric topology is based on minimizing the Kohlbek-Leibler divergence between the probability distribution and the preset Gaussian distribution.
[0026] Preferably, step 106 includes the following sub-steps: Step 1061: Using the joint geometric feature tensor as the query vector, and the appearance feature tensor as the key vector and value vector; Step 1062: Identifying local feature regions in the appearance feature tensor affected by ambient illumination by calculating the cross-correlation coefficient between the query vector and the key vector; Step 1063: Performing a weighted summation of the value vector based on the cross-correlation coefficient, and using the structured prior information in the joint geometric feature tensor to complete the features of the local feature regions, thereby obtaining the three-dimensional line-of-sight feature vector.
[0027] Preferably, the residual network in step 106 adopts a hierarchical downsampling structure to extract semantic feature components containing iris texture and scleral reflectivity from the face image, and maps the semantic feature components to a feature space consistent with the dimension of the joint geometric feature tensor.
[0028] Preferably, step 105 further includes: constructing a 2×3 affine transformation matrix based on dynamic affine transformation parameters, and using the affine transformation matrix to perform coordinate mapping on the multi-scale right eye geometric feature tensor, so that the multi-scale right eye geometric feature tensor presents mirror conjugate symmetry with respect to the multi-scale left eye geometric feature tensor in the feature space.
[0029] Preferably, the regression network in step 107 includes cascaded fully connected layers and nonlinear activation units, which map the three-dimensional line-of-sight feature vectors to horizontal yaw angles and vertical pitch angles in degrees.
[0030] Preferably, after acquiring the face image, the method further includes histogram equalization and normalization to a 224×224 pixel size to eliminate the interference of ambient light non-uniformity on feature extraction.
[0031] Preferably, step 105 eliminates the binocular visual axis non-parallelism caused by the fixation distance in the range of 30cm to 50cm at the feature space level by establishing a nonlinear coordinate mapping in the multi-scale right eye geometric feature tensor.
[0032] Example 1: In close-range interaction with a handheld mobile terminal, the user's gaze distance is 40cm. Due to physiological convergence-divergence shift of the visual axes of both eyes, the left and right eye regions in the acquired face image exhibit asymmetrical geometric deformation. A pre-trained geometric prior network is used to extract the multi-scale left eye geometric feature tensor from the face image. With multi-scale right eye geometric feature tensor Calculate the absolute value of the element-by-element difference between the two to determine the residual feature map characterizing the view axis offset component. Residual feature map The calculation follows the formula below: ;in, For residual feature maps, For the multi-scale left eye geometric feature tensor, For the multi-scale right eye geometric feature tensor.
[0033] Residual feature map The input parameter mapping unit uses global average pooling and linear transformation to output 2×3 dynamic affine transformation parameters. And based on the dynamic affine transformation parameters Multiscale right eye geometric feature tensor Spatial resampling is performed to determine the aligned multi-scale right-eye geometric feature tensor, thereby aligning the right-eye features to a feature space based on the left-eye visual axis. It should be noted that this two-dimensional dynamic affine resampling at the feature space level does not directly act on the original image pixel planes with three-dimensional perspective distortion, but rather on the geometric manifold space after high-level abstraction and dimensionality reduction by a geometric prior network. Since the preceding nonlinear encoding operation has compressed and folded the three-dimensional curved surface structure features and spherical kinematics of the human eye into a high-dimensional feature tensor with local linear flattening properties, two-dimensional affine transformation matrix operations can be used in the local tangent space coordinate system of this feature tensor to mathematically compensate for the three-dimensional nonlinear topological distortion caused by near-field gaze convergence and divergence in the physical world, thus aligning the multi-scale left-eye geometric feature tensor... The binocular geometric consistency fusion features are then extracted by concatenating the aligned multi-scale right-eye geometric feature tensor with the joint geometric feature tensor and using a 1×1 convolution operator. Then, the binocular geometric consistency features are fused. The cross-attention module is used to input the appearance feature tensor extracted from the face image. It uses the geometric structure distribution law to guide the semantic alignment of appearance details. The regression network outputs the three-dimensional gaze direction parameters, which solves the problem of decreased gaze estimation accuracy caused by the non-parallelism of the visual axes of the two eyes in near-field interaction from 30cm to 50cm.
[0034] Example 2: In a visual verification experiment simulating near-field interaction with a handheld terminal, to verify the system's performance in counteracting visual axis shift and suppressing environmental noise interference, an experimental environment was constructed consisting of a 12-megapixel CMOS visual sensor, a computing platform with a main frequency of 2.8 GHz, and a controlled LED light source. A publicly available face dataset containing 5,000 gaze samples was selected as the data source. To simulate illumination fluctuations and thermal noise of the image acquisition equipment in real-world conditions, Gaussian white noise with a signal-to-noise ratio of 25 dB was superimposed on the original acquired face images, and a non-uniform illumination distribution with a brightness deviation of 15% was introduced to evaluate the multi-scale left eye geometric feature tensor. With multi-scale right eye geometric feature tensor Feature stability under non-ideal conditions; for multi-scale left-eye geometric feature tensors With multi-scale right eye geometric feature tensor The setting of spatial resolution involves a technical trade-off between ensuring the precision of geometric structure representation and controlling computational resource consumption. If the resolution is lower than 32×32 pixels, the residual feature map... Insufficient spatial sampling frequency leads to the loss of geometric gradients at the pupil edge and corneal reflection points, causing subsequent parameter mapping units to fail to accurately capture visual axis offset. Furthermore, when the resolution exceeds 128×128 pixels, the increase in feature dimensions causes an exponential increase in computational load, and performance saturation occurs in improving the accuracy of 3D gaze estimation. Therefore, based on the Nyquist sampling criterion and combined with the minimum feature size of the ocular anatomy, [the following is missing from the original text]. and The resolution is fixed at 64×64 pixels to ensure that the non-conjugate geometric deformation of both eyes can be represented while maximizing the feature entropy.
[0035] The experiment included a control group and the prototype group of this invention. The control group used a feature stitching method without affine calibration, while the prototype group of this invention employed a complete geometric residual analysis and dynamic affine calibration process. In the baseline setting stage, a set of original images acquired at a 40cm gaze distance were input. Due to physiological convergence and divergence movements, the eyes exhibited an asymmetric deformation of 1.52 pixels on the image plane. In the core derivation stage, the prototype group of this invention calculated... In the determined residual feature map, high response values are concentrated in the edge regions of the inner and outer canthi. The parameter mapping unit outputs dynamic affine transformation parameters based on this residual distribution. This parameter includes a scaling factor of 0.081 and a rotation component of 1.22 radians. By implementing bilinear interpolation resampling, the geometric features of the right eye are aligned with the visual axis of the left eye in the feature manifold space. In the actual derivation of this data benchmark, the system substituted the statistically representative 63 mm average binocular interpupillary distance constant of the adult population. Under the geometric projection model with a focal distance of 40 cm, the angle limit generated at the camera's photosensitive plane when the visual axes of the left and right eyes naturally intersect is calculated using the inverse trigonometric function. The angular deformation benchmark of 1.22 radians is calculated, and the optimal size compensation difference of 0.081 is calculated by combining the empirical correction attenuation coefficient for lens edge perspective distortion. This constitutes the core guiding target for the iterative convergence of affine transformation parameters. In the value closure stage, the data shows that the average angular error of the control group under noise interference is 4.54 degrees, while the sample group of this invention extracts binocular geometric consistency fusion features. By incorporating a cross-attention mechanism, the estimation error of the three-dimensional gaze direction parameters was stabilized at 1.82 degrees, confirming the scheme's ability to suppress near-field non-conjugate bias. To investigate the impact of parameter boundaries on system stability, the nonlinear evolution trend of observation performance was adjusted by changing the gaze distance. Experimental data showed that when the gaze distance moved within a preset working window of 30cm to 60cm, the standard deviation of the error fluctuation of the sample group of this invention was less than 0.12 degrees, demonstrating good adaptability to various working conditions. However, as the gaze distance decreased to below the extreme boundary of 10cm, the extreme near-field perspective distortion generated by the eye image exceeded the dynamic affine transformation parameters. The scope of compensation led to Semantic alignment with the appearance feature tensor failed, and the gaze estimation error increased to 7.23 degrees. The appearance of this performance inflection point confirms that the defined parameter range of 30cm to 50cm is the optimal window that takes into account both physiological convergence and divergence laws and algorithm compensation capabilities. When the user's gaze distance exceeds the upper limit of 50cm, the visual axes of both eyes tend to be approximately parallel. At this time, the magnitude of the angle interference offset will be attenuated and submerged by the background white noise of the sensor. Triggering the affine network at this time will introduce useless computational clutter. When the distance from the camera focal plane is less than the lower limit of 30cm, the convergence and divergence angle of the two visual axes increases nonlinearly and sharply (breaking through the threshold of about 15 degrees), causing topological tearing of the appearance texture of the pupil area of both eyes, which exceeds the physical limit that can be restored by mathematical fitting by the current prior compensation mechanism.
[0036] Example 3: In the offline calibration process of the visual perception system, to determine the transformation logic from the pixel space of the eye image to the geometric manifold space, a geometric training library consisting of 2000 sets of eye topology samples with known 3D mesh annotations is constructed. In this process, the backbone feature encoding architecture for constructing the geometric prior network uses a deep residual network based on ResNet-18. In the actual modification, the fully connected classification mapping head of the original architecture is removed, retaining one large-kernel initial convolutional layer, one pooling layer, and four sets of cascaded residual modules containing direct shortcuts as the core backbone. This ensures that the dense pixel matrix of the input image can be processed step-by-step. Layer-wise nonlinear compression ultimately outputs a deep structure tensor pointing to a 512-dimensional latent variable space. Variational inference logic is used to statistically model the eye contour, pupil center coordinates, and corneal edge curvature within the latent space. Reparameterization is employed to convert high-dimensional image information into mean and variance vectors, and the convergence state of the geometric prior network is constrained by minimizing the geometric consistency loss function. The calibration process determines convergence by monitoring the divergence between the latent space distribution and the standard normal distribution over 100 training epochs. When the divergence value is below 0.05, the weight coefficients of the geometric feature extraction units are locked to ensure the multi-scale left eye geometric feature tensor... With multi-scale right eye geometric feature tensor The output points to a definite spatial physical attribute. Based on the latent variable generation constraint law of variational inference theory, the geometric prior network converges by relying on the auxiliary reconstruction mechanism to solidify the topological information physical mapping relationship. The computing platform constructs a decoding unit containing multi-level deconvolution operations. The mean vector and variance vector in the latent space are extracted to recover and generate the eye prediction distribution map. The pixel-level relative entropy between the eye prediction distribution map and the input original image is calculated as the reconstruction loss parameter. The control system calculates the linear weighted sum of the reconstruction loss parameter and the divergence constraint term, and initiates backpropagation to iteratively update the encoder node weights. Based on the Taylor expansion approximation principle of small spatial displacement, the system sets a spatial alignment procedure in the training closed loop, extracts the corresponding batch of left eye geometric feature tensors and the right eye geometric feature tensors after spatial resampling, and calculates their root mean square error. As a weight optimization link for the independent injection of inverse gradient signals into the parameter mapping unit, it constrains the output of the internal linear transformation layer nodes to stably converge to the physical affine values representing the rotation and scaling components of the two-dimensional plane. After obtaining the linear affine values that have converged under the above constraints, the system does not perform a simple global rigid translation at the feature level. Instead, it uses the 2×3 affine transformation matrix as the basic addressing module parameter of the spatial transformation network and imports its output reference coordinate system into a dynamic mesh generator with Sigmoid normalization activation. The continuous response value of the bias position is obtained through bilinear interpolation. This mechanism enables the affine matrix, which was originally built based on pure linear operators, to establish a nonlinear coordinate mapping relationship with gradient feedback between discrete pixel feature mesh points, thus achieving elastic alignment of local features.
[0037] When the system is deployed to a new mobile interactive terminal, a geometric baseline calibration procedure based on camera intrinsic parameters is initiated. Five frames of gaze images of the user at a preset guided position are acquired, and the average pixel difference between the binocular regions in each frame is calculated. The average pixel difference is normalized using a camera projection model to determine the geometric offset reference value caused by lens distortion in this terminal environment. This reference value is then injected as a bias term into the linear transformation layer of the parameter mapping unit. Based on the principle of linear inverse mapping from two-dimensional coordinates of the image plane to high-dimensional feature manifolds, the single-dimensional geometric offset scalar must undergo dimensional transformation to adapt to the heterogeneous space. The control platform extracts the initial feature projection layer weight matrix of the geometric prior network to calculate the generalized inverse matrix. The geometric offset reference value and the generalized inverse matrix are extracted and subjected to tensor dot product operations to determine the relationship with the linear transformation. The latent space compensation bias vector with consistent layer dimensions is replaced by the direct bias value and superimposed on each output node of the linear transformation layer. Considering that the initial feature projection layer weight matrix is a non-square matrix structure with unequal number of rows and columns, conventional inversion operators cannot solve it directly. The system embeds a singular value decomposition (SVD) algorithm to independently decompose the weight matrix, thereby deriving its corresponding Moore-Ponros pseudo-inverse matrix as the generalized inverse matrix mentioned in this paper. This pseudo-inverse matrix mathematically establishes the least squares reverse mapping link from the high-dimensional latent space to the one-dimensional geometric scalar, enabling the originally single-dimensional geometric bias scalar to be back-distributed to each node of the feature space according to the tensor dot product operation rules and different weight ratios. By synchronously monitoring the residual feature map during the aforementioned guidance process, The root mean square response value is used to lock the dynamic affine transformation parameters when the standard deviation of the response value fluctuation is less than 0.015 for three consecutive frames. The initial gain coefficient enables the system to cancel out geometric distortions caused by specific hardware during the spatial resampling stage and output three-dimensional line-of-sight parameters during the semantic association stage.
[0038] Example 4: In the calibration scenario for establishing a three-dimensional gaze deviation compensation benchmark, the system instructs the user to sequentially gaze at nine preset benchmark points from a gaze position 40cm away from the screen. For each benchmark point, the system extracts the initial gaze vector output by the regression network and calculates the spatial residual between the initial gaze vector and the actual gaze vector of the benchmark point. The system then averages the spatial residual across the nine sampling points to determine the individual gaze compensation vector. The system assesses calibration stability by calculating the standard deviation of the residual distribution at nine reference points. When the standard deviation is below 0.5 degrees, it completes the baseline setting of individual geometric attributes and utilizes the individual gaze compensation vector during fixation. As a constant additive term at the output of the regression network, it performs in-situ deviation correction on the three-dimensional line-of-sight direction parameters.
[0039] In the engineering procedure for optimizing the weight parameters of the geometric prior network, the system employs a variational coding architecture consisting of four downsampled residual blocks to process 128×128 pixel grayscale image data, utilizing a latent variable space with a latent dimension of 512 to represent the probability distribution of the eye topology. During the model training phase, the system determines the geometric consistency loss function by calculating the L2 norm between the predicted feature point coordinates and the response center of the geometric feature map. The system will use the geometric consistency loss function. Combined with the variational inference term, it forms the total loss function, with an initial learning rate of 1×10. -4 Furthermore, gradient updates are performed using a batch size of 32 samples. When the relative decrease rate of the total loss function is less than 0.001 over five consecutive training epochs, the multi-scale left-eye geometric feature tensor is locked. With multi-scale right eye geometric feature tensor The extracted model parameters ensure a definite physical correspondence between the feature space and the anatomical structure.
[0040] The embodiments of this application have been described above with reference to the accompanying drawings. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. This application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit of this application and the scope of protection of this invention, and all of these forms are within the protection scope of this application.
Claims
1. A gaze estimation method that integrates binocular geometric features, characterized in that, Includes the following steps: Step 101: Obtain a face image containing the regions of both eyes; Step 102: Extract multi-scale left eye geometric feature tensors and multi-scale right eye geometric feature tensors from face images through a pre-trained geometric prior network. The geometric prior network is a feature extraction model based on a variational inference architecture. Step 103: Perform feature shift residual calculation on the multi-scale left eye geometric feature tensor and the multi-scale right eye geometric feature tensor to obtain a residual feature map characterizing the physiological convergence and divergence shift of the two eyes and the optical distortion component. The residual feature map is composed of the absolute value of the element-wise difference between the multi-scale left eye geometric feature tensor and the multi-scale right eye geometric feature tensor. Step 104: The residual feature map is input to the parameter mapping unit and subjected to global average pooling and linear transformation to obtain dynamic affine transformation parameters for calibrating binocular asymmetry. Step 105: Spatial resampling and feature point alignment are performed on the multi-scale right eye geometric feature tensor according to the dynamic affine transformation parameters to obtain the joint geometric feature tensor aligned with the multi-scale left eye geometric feature tensor under the reference visual axis. Step 106: Extract appearance feature tensors from face images using a residual network, and input the appearance feature tensors and joint geometric feature tensors into a cross-attention module for semantic association fusion to obtain a three-dimensional gaze feature vector. Step 107: Input the 3D gaze feature vector into the regression network to obtain the 3D gaze direction parameters.
2. The gaze estimation method fusing binocular geometric features according to claim 1, characterized in that, Step 103 includes the following sub-steps: Step 1031, determine whether the resolutions of the multi-scale left-eye geometric feature tensor and the multi-scale right-eye geometric feature tensor are consistent; Step 1032, when the determination result is that the resolutions are inconsistent, use the bilinear interpolation algorithm to scale the multi-scale left-eye geometric feature tensor and the multi-scale right-eye geometric feature tensor to a uniform size; Step 1033, calculate the element-wise difference between the multi-scale left-eye geometric feature tensor and the multi-scale right-eye geometric feature tensor after scaling to a uniform size and take the absolute value to obtain the residual feature map.
3. The gaze estimation method fusing binocular geometric features according to claim 1, characterized in that, The geometric prior network in step 102 is trained in the following way: the probability distribution of the eye training image is projected onto the latent space using the encoder, and the statistical constraint ability of the geometric prior network on the eye geometry topology is based on minimizing the Kohlbek-Leibler divergence between the probability distribution and the preset Gaussian distribution.
4. The gaze estimation method fusing binocular geometric features according to claim 1, characterized in that, Step 106 includes the following sub-steps: Step 1061: Use the joint geometric feature tensor as the query vector and the appearance feature tensor as the key vector and value vector; Step 1062: Identify the local feature regions in the appearance feature tensor affected by ambient illumination by calculating the cross-correlation coefficient between the query vector and the key vector; Step 1063: Perform a weighted summation on the value vector based on the cross-correlation coefficient, and use the structured prior information in the joint geometric feature tensor to complete the features of the local feature regions, thereby obtaining the three-dimensional line-of-sight feature vector.
5. The gaze estimation method fusing binocular geometric features according to claim 1, characterized in that, The residual network in step 106 adopts a hierarchical downsampling structure to extract semantic feature components containing iris texture and scleral reflectivity from the face image, and maps the semantic feature components to a feature space with the same dimension as the joint geometric feature tensor.
6. The gaze estimation method fusing binocular geometric features according to claim 1, characterized in that, Step 105 further includes: constructing a 2×3 affine transformation matrix based on dynamic affine transformation parameters, and using the affine transformation matrix to perform coordinate mapping on the multi-scale right eye geometric feature tensor, so that the multi-scale right eye geometric feature tensor presents mirror conjugate symmetry with respect to the multi-scale left eye geometric feature tensor in the feature space.
7. The gaze estimation method fusing binocular geometric features according to claim 1, characterized in that, The regression network in step 107 includes cascaded fully connected layers and nonlinear activation units, which map the three-dimensional line-of-sight feature vectors to horizontal yaw angles and vertical pitch angles in degrees.
8. The gaze estimation method fusing binocular geometric features according to claim 1, characterized in that, After acquiring the face image, the process also includes histogram equalization and normalization to a 224×224 pixel size to eliminate the interference of ambient lighting non-uniformity on feature extraction.
9. The gaze estimation method fusing binocular geometric features according to claim 1, characterized in that, Step 105 eliminates binocular visual axis non-parallelism caused by fixation distance in the range of 30cm to 50cm at the feature space level by establishing a nonlinear coordinate mapping within the multi-scale right eye geometric feature tensor.