A distribution offset-aware binocular RGB-D gaze tracking method

Abstract

The application provides a binocular RGB-D gaze tracking method with distribution shift perception. First, left and right eye RGB images, corresponding depth images and head pose data are collected by an RGB-D sensor device to construct an eye RGB-D dataset, and binocular geometric enhanced features are extracted by a parameter sharing backbone network. Then, the standardization distribution shift is calculated based on the head pose statistics of the source domain reference data and the target domain data, and the ordinary backbone network or the backbone network with a spatial adaptive sampling module is adaptively selected according to the shift. Next, the aligned head pose features and the binocular geometric enhanced features are constructed into a common query, and the cross-modal fusion features under the pose condition constraint are learned through a double-channel cross-attention mechanism. Finally, the gaze angle is obtained by regression based on the fusion features, and is converted into a three-dimensional unit gaze vector. The stable three-dimensional gaze direction is output through multi-checkpoint vector fusion and sliding window smoothing processing. The application can effectively adapt to the gaze tracking task under complex head poses and continuous scenes.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, human-computer interaction, and gaze tracking technology, specifically to a gaze tracking method that utilizes binocular RGB-D images and head pose data for distribution offset perception, cross-modal fusion, and stable inference. Background Technology

[0002] Non-contact gaze tracking captures images of the face or eyes using camera devices and determines the user's gaze direction using image processing or learning models. It can be applied to scenarios such as human-computer interaction, driving status analysis, and assisted medical care. Existing appearance-based methods typically use monocular or binocular RGB images as the main input and utilize convolutional networks to learn the mapping relationship between eye texture and gaze direction.

[0003] However, RGB images of the eyes are easily affected by changes in lighting, reflections, occlusion, and individual differences in eye shape, making it difficult for 2D textures to independently provide stable spatial geometric constraints. Significant differences in head pose distribution may also exist between different objects or data partitions, making it difficult for a fixed-structure convolutional backbone to simultaneously accommodate both normal poses and large-amplitude deflections. Furthermore, the data forms and semantic levels of eye visual features and head pose features differ, making direct concatenation insufficient to fully model their conditional relationship. For continuous video, frame-by-frame prediction is also prone to unnatural jumps due to single-frame noise. Therefore, a gaze tracking scheme is needed that can jointly utilize binocular appearance, depth geometry, and head pose information, and handle pose distribution changes and continuous output fluctuations. Summary of the Invention

[0004] To address the shortcomings of existing gaze tracking methods, which primarily rely on RGB visual information of the eyes, lack adaptability to changes in head posture distribution, suffer from insufficient fusion of eye visual features and head posture features, and are prone to fluctuations in continuous gaze output, this invention provides a binocular RGB-D gaze tracking method based on distribution offset awareness. This invention jointly utilizes binocular RGB images, eye depth images, and head posture data. It selects appropriate feature extraction backbones based on the degree of head posture distribution offset in the target data, and outputs the three-dimensional gaze direction of the object to be tracked through posture-conditional cross-modal fusion and 3D gaze vector stabilization processing.

[0005] The technical solution of the present invention includes the following steps:

[0006] Step a: Acquire and preprocess the left and right eye RGB images, corresponding depth images, and head pose data of the object to be tracked;

[0007] Step b: Based on the head pose distribution offset between the reference data and the target data, and a preset threshold, determine the general feature extraction backbone or the feature extraction backbone containing the spatial adaptation module, including the following steps:

[0008] Step b1: Using the mean and standard deviation of the head pose of the reference data, standardize the difference between the mean head pose of the target data and the reference data, and calculate the head pose distribution offset.

[0009] Step b2: Compare the head pose distribution offset with a preset threshold. If it is lower than the preset threshold, select the normal feature extraction backbone; otherwise, select the feature extraction backbone that includes the spatial adaptation module.

[0010] Step b3: Predict the two-dimensional sampling offset using the spatial adaptive module and perform deformable convolution, and weight the deformable convolution output by channel weights;

[0011] Step c: Construct binocular geometric enhancement features, including the following steps:

[0012] Step c1: Combine the left and right eye RGB images with the corresponding depth images along the channel dimension to form left and right eye RGB-D data;

[0013] Step c2: Using the determined parameters, extract the features of the left eye and the right eye respectively by sharing the binocular trunk;

[0014] Step c3: Concatenate the features of the left and right eyes along the channel dimension, and then perform 1×1 convolution, normalization and non-linear activation to obtain the geometric enhancement features of both eyes;

[0015] Step d: Perform pose-conditional cross-modal fusion of binocular geometric enhancement features and head pose data, including the following steps:

[0016] Step d1: Perform feature encoding and dimension alignment on the head pose data, and combine the obtained pose condition features with the binocular geometric enhancement features to form a common query;

[0017] Step d2: Using the common query as the query feature, and using the binocular geometric enhancement feature and pose condition feature as the key feature and value feature of the two cross-attention branches respectively, perform dual-channel cross-attention;

[0018] Step d3: Aggregate and feedforward the outputs of the two cross-attention branches, and add the residuals with the binocular geometric enhancement features to obtain the pose-conditional cross-modal fusion features;

[0019] Step e: Predict and stably output the 3D gaze direction based on the cross-modal fusion features of the attitude conditions, including the following steps:

[0020] Step e1: Predict the line-of-sight yaw angle and line-of-sight pitch angle based on the cross-modal fusion features of attitude conditions, and convert the line-of-sight yaw angle and line-of-sight pitch angle into a three-dimensional line-of-sight vector;

[0021] Step e2: Average and normalize the 3D line-of-sight vectors output from multiple model checkpoints to obtain the checkpoint fused line-of-sight vector;

[0022] Step e3: Average and normalize the fused gaze vectors of the checkpoints within the sliding time window to stably output the 3D gaze direction of the object to be tracked.

[0023] This invention supplements the spatial geometric information of the eyes by using joint RGB-D input from both eyes, and drives the selection between a normal backbone network and a spatially adaptive backbone network by using the head pose distribution offset, so that the model's feature extraction capability matches the degree of pose change in the target data. An explicit correlation between eye vision and head pose conditions is established through a dual-channel cross-attention mechanism driven by common queries, and the fluctuation of the results in the continuous prediction process is reduced by multi-checkpoint fusion and temporal smoothing at the three-dimensional gaze vector level, thereby improving the adaptability and stability of gaze tracking results in complex poses and continuous scenes. Attached Figure Description

[0024] Figure 1 This is a flowchart of the binocular RGB-D gaze tracking method with distribution offset perception in the method of the present invention.

[0025] Figure 2 This is a diagram of the overall network framework of the binocular RGB-D gaze tracking method with distribution offset perception in the present invention.

[0026] Figure 3 This is a diagram of the attitude distribution offset backbone selection and spatial adaptive module in the method of the present invention.

[0027] Figure 4 This is a diagram of the dual-channel cross-attention fusion module for attitude conditions in the method of this invention.

[0028] Figure 5 This is a correlation distribution diagram showing the predicted and actual values ​​of yaw angle and pitch angle in the method of this invention.

[0029] Figure 6 This is a comparison diagram of the thermal distribution of spatial angle error of different subjects in the method of the present invention.

[0030] Figure 7 This is a comparison chart of the average angular error of the upper gaze of different subjects in the method of the present invention. Detailed Implementation

[0031] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific examples. It should be noted that the following examples are used to explain the present invention and not to limit the scope of protection of the present invention. Those skilled in the art can make equivalent substitutions or adaptive adjustments to the data acquisition device, network layer number, feature dimension, activation function, threshold, and time window length without departing from the technical concept of the present invention. The same or corresponding technical features in the various examples can be combined with each other, and the resulting technical solutions still fall within the scope disclosed in the present invention.

[0032] This invention provides a binocular RGB-D gaze tracking method with distribution offset awareness, mainly including data acquisition and preprocessing, binocular geometric enhancement feature construction, head pose distribution offset measurement and backbone selection, pose conditional cross-modal fusion, gaze regression training, and 3D gaze vector stable inference. This invention takes left and right eye RGB images, depth images aligned with the RGB image space, and head pose data as input, and outputs the 3D gaze direction in the camera coordinate system.

[0033] Step a: Acquire and preprocess RGB-D data for both eyes and head pose data:

[0034] Step a1: Use an RGB-D imaging device to acquire an RGB image of the face of the object to be tracked and a corresponding depth image. Preferably, the RGB image and the depth image have the same or a registrable spatial coordinate system.

[0035] Step a2: When the resolution, intrinsic parameters, or field of view of the two images are inconsistent, the depth image is mapped to the color image coordinate system according to the intrinsic and extrinsic parameters provided by the imaging device, so as to ensure that the color information and depth information at the same pixel position correspond to the same or approximately the same spatial position.

[0036] Step a3: Perform face detection and key point localization on the acquired face image. Determine the left and right eye regions based on key points near the corner of the eye, eyelid, or iris. Crop the left and right eye RGB images from the color image, and crop the corresponding left and right eye depth images from the registered depth image.

[0037] Step a4: Scale each eye image to a uniform size, such as 224×224 pixels, or set to other fixed sizes according to the backbone network structure; for pixels that are outside the effective depth range, lack depth, or are affected by reflection, use one or more of the following methods: neighborhood interpolation, median padding, and effective depth truncation, and normalize the depth values ​​to a preset range.

[0038] Step a5: Obtain the head pose data of the object to be tracked. Head pose can be calculated from facial landmarks and a 3D face model, or obtained from a standalone pose tracking device or a head pose network. Head pose includes at least yaw and pitch angles, and may also include roll angles if the device and data allow. Let the head pose vector of the i-th sample be:

[0039]

[0040] in, This represents the head yaw angle of the i-th sample. Indicates the head pitch angle. Indicates the head roll angle, superscript This represents the vector transpose. The actual pose components used should be consistent with the training data and model input.

[0041] Step b: Construct RGB-D data of the eyes and extract geometric enhancement features for both eyes:

[0042] Step b1: Combine the three-channel RGB image of each eye with the corresponding single-channel depth image along the channel dimension to form four-channel eye RGB-D data. The RGB-D inputs for the left and right eyes are represented as follows:

[0043]

[0044]

[0045] In this context, the superscripts l and r represent the left eye and right eye, respectively; , and These represent the red, green, and blue channels, respectively. Indicates depth channel; This indicates concatenation along the channel dimension. The RGB channel is used to describe the appearance information of the iris, sclera, eyelids, and periocular region, while the depth channel is used to describe the relative distances, local surface undulations, and spatial structure of the eye region. The two types of information are combined during the input stage, enabling depth geometry information to participate in the complete encoding process from shallow edges to high-level semantics.

[0046] Step b2: Extract the backbone of binocular features from the shared input parameters of the left-eye RGB-D data and the right-eye RGB-D data. Let the backbone of feature extraction be... ,in If we consider shared network parameters, then the features of the left and right eyes are represented as follows:

[0047]

[0048]

[0049] in, and These represent the features of the left and right eyes, respectively. The left and right eye branches use the same set of parameters, mapping the data from both eyes to a unified feature space. This reduces parameter redundancy caused by independent modeling of the two branches and facilitates subsequent comparison and fusion of corresponding semantic channels. Although the parameters are shared, the data from the left and right eyes pass through the backbone network separately, thus preserving their respective local observational differences. When one eye is affected by occlusion, reflection, or image blurring, the other eye can provide complementary information.

[0050] Step b3: Keep the forward encoding of the left and right eye data independent of each other in the shared backbone, so that the left eye features and right eye features retain the local observation information of the corresponding eyes respectively, and use parameter sharing constraints to keep them in a consistent semantic space; when one eye is affected by occlusion, reflection or blurring, the other eye can provide complementary iris, eyelid and depth structure information.

[0051] Step b4: Concatenate the features of the left and right eyes along the channel dimension, and obtain the binocular geometric enhancement features through 1×1 convolution, batch normalization, and nonlinear activation:

[0052]

[0053] in, Indicates binocular geometric enhancement features; Represents a 1×1 convolution kernel; * indicates a convolution operation; Indicates batch normalization; Represents ReLU or other non-linear activation functions. Channel concatenation preserves left and right eye features, and 1×1 convolution establishes cross-channel interactions and maps the concatenated channel count to the target dimension. The resulting... It also includes the RGB appearance, depth structure, and complementary relationship between the two eyes for both eyes.

[0054] Step c: Calculate the head pose distribution offset and select the feature extraction backbone.

[0055] Step c1: Determine the reference training data and target application data, and statistically analyze head pose during the model configuration phase of target data partitioning or application domain. Based on the statistical results, select either a standard feature extraction backbone or a spatially adaptive feature extraction backbone; this selection does not require switching the network structure during each frame of inference, thereby avoiding latency fluctuations caused by dynamically changing computation.

[0056] Step c2: Establish head pose sets for the reference training data and the target data. Let the head pose set for the reference training data be:

[0057]

[0058] The target data set of head poses is:

[0059]

[0060] Step c3: Calculate the attitude mean vector and standard deviation vector based on the reference data. Where N... tr Indicates the number of reference training samples. Indicates the number of target samples. and These represent the head pose vectors in the reference data and the target data, respectively:

[0061]

[0062]

[0063] Step c4: Calculate the target attitude mean vector based on the target data:

[0064]

[0065] Step c5: Standardize the difference between the target pose mean and the reference pose mean using the reference data standard deviation, and calculate the standardized head pose distribution offset. Among these steps, and These represent the mean head pose values ​​of the reference data and the target data, respectively. This represents the standard deviation of the reference data's attitude. The square, square root, and division are all dimension-wise operations.

[0066]

[0067] Where s represents the head pose distribution offset; Represents the smallest positive number that is protected against division by zero; This represents the Euclidean norm. The calculation first uses the standard deviation of the reference data to scale the different attitude components, and then combines the multidimensional offsets into a single scalar, which can reduce the impact of different yaw and pitch angle ranges on the comparison results.

[0068] Step c6: Set the preset threshold as... The head pose distribution offset is compared with a preset threshold, and the trunk selection rule is expressed as follows:

[0069]

[0070] in, This represents the backbone of general feature extraction. This represents the feature extraction backbone containing the spatially adaptive module. When the pose distribution of the target data is close to that of the reference data, fixed convolutional sampling usually provides a stable representation, so a normal backbone is chosen. When the offset reaches or exceeds a threshold, a spatially adaptive backbone is chosen to compensate for changes in the position of the iris, corner of the eye, eyelid, and depth boundaries caused by head rotation. Threshold The accuracy and computational cost can be determined by verifying the data, for example by comparing the trunk selection results and the average line-of-sight angle error under different thresholds, but the specific threshold does not constitute a limitation of the present invention.

[0071] Step c7: In the selected spatial adaptive backbone, insert the spatial adaptive module after the second convolution and batch normalization of the residual blocks, and before the summation of the residuals. Let the intermediate features generated by the residual main branch be... The spatial adaptive module first generates a two-dimensional offset field through offset prediction convolution:

[0072]

[0073] in, This represents the offset prediction function. This represents the two-dimensional offset corresponding to each convolutional sampling point. For a 3×3 convolutional kernel, there are 9 sampling points, each with two offsets, one horizontal and one vertical. Therefore, the offset field can include 18 offset channels. The deformable convolution output is represented as:

[0074]

[0075] in, Indicates the current output position. This represents the fixed offset of the k-th rule sampling point relative to the center position. This represents the additional offset obtained through learning. This represents the convolution weight of the k-th sampling point, where K represents the number of sampling points. This represents the features after adaptive sampling in the space. For non-integer sampling locations, bilinear interpolation can be used to obtain the feature values.

[0076] Step c8: Perform global average pooling and two-layer mapping based on spatial adaptive sampling features to generate channel weights:

[0077]

[0078]

[0079] in, Indicates global average pooling. and This represents the weights of the two-layer mapping. This represents the activation function. Limit the weights to between 0 and 1. Indicates channel weight, This represents element-wise multiplication broadcast by channel. Deformable sampling is used to determine where to sample, and channel weighting is used to determine which responses are more important; together, they improve the backbone's adaptability to attitude-related spatial changes.

[0080] Step d: Construct a common query and perform attitude-conditional cross-modal fusion.

[0081] Step d1: Input the head pose vector into the multilayer perceptron for encoding:

[0082]

[0083] Step d2: Perform channel projection and spatial expansion on the pose semantic vector. Represents the attitude semantic vector. This represents a head pose encoding network; since pose features are typically one-dimensional vectors, while binocular geometric enhancement features... Since it is a 3D feature map, the following dimension alignment is performed:

[0084]

[0085] in, This represents the attitude projection function. Indicates space broadcasting. Indicates and Attitude condition features with the same number of channels, height, and width.

[0086] Step d3: Add the binocular geometric enhancement features and pose conditional features element by element to form a common query:

[0087]

[0088] Step d4: Establish visual and geometric conditional branches. (Common query) It simultaneously includes ocular visual semantics and head posture conditions, serving as a unified retrieval basis for both branches; the visual branch uses... As a query, with As keys and values; geometric conditional branches are used as As a query, with As keys and values. For any branch m, generate the query, key, and value:

[0089]

[0090]

[0091]

[0092] Where m represents the visual branch or the geometric conditional branch; , and These represent the feature projection functions for the query, key, and value, respectively; in the vision branch In the geometric branch The projection function can be composed of 1×1 point convolution and 3×3 depth convolution. Point convolution is used for channel mapping, and depth convolution is used to preserve local spatial information.

[0093] Step d5, set Representing the number of spatial locations in the feature map, the query, key, and value features are rearranged as follows: Matrix, and calculate correlation along the channel dimension:

[0094]

[0095]

[0096] Where C represents the number of channels, and H and W represent the height and width of the feature map, respectively; , and This represents the rearranged query, key, and value; Indicates temperature parameter; Indicates size is The channel correlation matrix; This represents the attention-weighted result. This calculation establishes relationships between locations in the channel space rather than all spatial locations, reducing the sensitivity of attention calculations to image resolution.

[0097] Step d6: Aggregate the visual branch output and the geometric conditional branch output element-wise, and input them into the fusion feedforward network:

[0098]

[0099]

[0100]

[0101] Step d7: Add the residuals of the fusion feedforward network output and the original binocular geometric enhancement features to obtain the pose-conditional cross-modal fusion features. Among them, Indicates the visual branch output, This indicates the output of the geometric conditional branch. Indicates dual-path aggregation feature, Representation layer normalization, and These represent dimensionality-up projection and dimensionality-down projection, respectively. This indicates the output of the fused feedforward network. This represents pose-conditional cross-modal fusion features. Visual residual connections are used to preserve the original eye RGB-D information, enabling pose features to participate in fusion through conditional modulation and geometric compensation.

[0102] Step e: Train the multi-branch gaze regression network

[0103] Step e1: Set up a left-eye auxiliary regression head based on left-eye features, set up a right-eye auxiliary regression head based on right-eye features, and set up a master regression head based on pose-conditional cross-modal fusion features.

[0104] Step e2: Obtain the left eye auxiliary prediction. Right eye auxiliary prediction and main line of sight prediction and read the corresponding real label. , and .

[0105] Step e3: Concatenate the three prediction results and their corresponding ground truth labels separately, and construct a joint... Regression loss:

[0106]

[0107] Step e4, according to the union The regression loss is backpropagated and the network parameters are updated. Among these, This indicates the loss of visual return. express Norms. Left and right eye auxiliary branches allow monocular features to be directly supervised by gaze labels, while the main regression branch learns the synergistic relationship between binocular information and pose conditions. During the inference phase, only the main regression output can be retained, or the auxiliary outputs can be retained for reliability analysis depending on task requirements.

[0108] Step f: Regress the gaze angle and perform 3D gaze vector stabilization inference.

[0109] Step f1: Regress the yaw and pitch angles of the current frame based on the cross-modal fusion features of the attitude conditions. Let the pitch and yaw angles output by the k-th model checkpoint in the t-th frame be respectively... and .

[0110] Step f2: According to the camera coordinate system convention, convert the pitch angle and yaw angle into a three-dimensional line-of-sight vector:

[0111]

[0112] in, Let represent the 3D gaze vector output by the k-th checkpoint for the t-th frame. The signs of each component in the formula can be adaptively set according to the camera coordinate axis orientation, but the same convention should be used for training, label conversion, and inference.

[0113] Step f3: When using multiple model checkpoints for fusion, average the multiple 3D gaze vectors of the same frame:

[0114]

[0115]

[0116] Step f4: Normalize the average vector of checkpoints. Where K represents the number of checkpoints. Represents the average vector of checkpoints. This represents the normalized checkpoint fusion vector. Multiple checkpoints can also be averaged using a non-negative weighted average, but the sum of all weights should be 1.

[0117] Step f5: Set a sliding time window of length L and determine the starting position of the window corresponding to the current frame:

[0118]

[0119] Step f6: Calculate the actual number of frames within the window. The average of the checkpoint fusion vectors within the window is then normalized.

[0120]

[0121]

[0122] in, This represents the average vector within the time window. This represents the stable 3D line-of-sight direction of the final output of the current frame. A larger window length generally results in a stronger smoothing effect, but may also increase response latency; therefore, different window lengths can be selected based on real-time requirements. Since the fusion process is completed in 3D directional space, it avoids the angular periodicity and coordinate coupling problems that may occur when directly averaging yaw and pitch angles.

[0123] Step f7: When the application system needs to output the angle form, stabilize the three-dimensional line-of-sight vector. Reconverted to pitch and yaw angles:

[0124]

[0125]

[0126] in, Represents the arcsine function. This represents a two-parameter arctangent function with quadrant discrimination capability. The final output can be used for screen gaze mapping, human-computer interaction control, driver state analysis, or other applications requiring three-dimensional gaze direction.

[0127] In summary, this invention supplements eye depth geometry information through joint RGB-D input from both eyes, matches the pose variation of the feature extraction backbone with that of the target data by standardizing head pose distribution offset, establishes an explicit correlation between eye vision and head pose conditions through dual-channel cross-attention driven by a common query, and reduces continuous prediction fluctuations through checkpoint fusion and temporal smoothing at the 3D gaze vector level. These steps constitute a complete technical chain from sensor data input to stable 3D gaze direction output.

[0128] To verify the effectiveness of this invention, comparative experiments were conducted on the EyeDiap dataset using several existing mainstream gaze tracking models. The invention was compared with several advanced algorithms, including RGB-DGaze, GazeNet, FullFace, and GazeNAS-ETH, and the results are shown in Table 1. Experimental results show that existing methods mainly rely on RGB images as a single input source, with some methods supplemented by head pose or depth information. In contrast, this invention innovatively integrates RGB, depth, and head pose as inputs, fully utilizing the complementarity of multi-source information. Quantitative analysis shows that the average angle error of the method in this invention is reduced to 4.86°, demonstrating a significant performance improvement. This substantial performance enhancement verifies the effectiveness and superiority of introducing depth information and multi-modal fusion strategies in improving viewpoint tracking accuracy.

[0129] Table 1 Performance Evaluation of Eye-Tracking Methods

[0130]

[0131] Figure 7 The mean angular error of eye tracking across different subjects was compared between the baseline method and the method of this invention on the EyeDiap dataset. A lower mean angular error indicates higher eye tracking accuracy. Overall, the mean angular error of the method of this invention decreased from 3.77° of the baseline method to 3.19°, an absolute decrease of 0.58° and a relative decrease of approximately 15.38%, indicating that the method of this invention can effectively improve eye tracking performance across subjects. This means that the method of this invention has achieved good performance on the EyeDiap dataset, demonstrating its generalization and robustness.

Claims

1. A binocular RGB-D gaze tracking method based on distribution offset perception, characterized in that, Includes the following steps: Step a: Acquire and preprocess the left and right eye RGB images, corresponding depth images, and head pose data of the object to be tracked; Step b: Based on the head pose distribution offset between the reference data and the target data, and a preset threshold, determine the general feature extraction backbone or the feature extraction backbone containing the spatial adaptation module, including the following steps: Step b1: Using the mean and standard deviation of the head pose of the reference data, standardize the difference between the mean head pose of the target data and the reference data, and calculate the head pose distribution offset. Step b2: Compare the head pose distribution offset with a preset threshold. If it is lower than the preset threshold, select the normal feature extraction backbone; otherwise, select the feature extraction backbone that includes the spatial adaptation module. Step b3: Predict the two-dimensional sampling offset using the spatial adaptive module and perform deformable convolution, and weight the deformable convolution output by channel weights; Step c: Construct binocular geometric enhancement features, including the following steps: Step c1: Combine the left and right eye RGB images with the corresponding depth images along the channel dimension to form left and right eye RGB-D data; Step c2: Using the determined parameters, extract the features of the left eye and the right eye respectively by sharing the binocular trunk; Step c3: Concatenate the features of the left and right eyes along the channel dimension, and then perform 1×1 convolution, normalization and nonlinear activation to obtain the geometric enhancement features of both eyes; Step d: Perform pose-conditional cross-modal fusion of binocular geometric enhancement features and head pose data, including the following steps: Step d1: Perform feature encoding and dimension alignment on the head pose data, and combine the obtained pose condition features with the binocular geometric enhancement features to form a common query; Step d2: Using the common query as the query feature, and using the binocular geometric enhancement feature and pose condition feature as the key feature and value feature of the two cross-attention branches respectively, perform dual-channel cross-attention; Step d3: Aggregate and feedforward the outputs of the two cross-attention branches, and add the residuals with the binocular geometric enhancement features to obtain the pose-conditional cross-modal fusion features; Step e: Predict and stably output the 3D gaze direction based on the cross-modal fusion features of the attitude conditions, including the following steps: Step e1: Predict the line-of-sight yaw angle and line-of-sight pitch angle based on the cross-modal fusion features of attitude conditions, and convert the line-of-sight yaw angle and line-of-sight pitch angle into a three-dimensional line-of-sight vector; Step e2: Average and normalize the 3D line-of-sight vectors output from multiple model checkpoints to obtain the checkpoint fused line-of-sight vector; Step e3: Average and normalize the fused gaze vectors of the checkpoints within the sliding time window to stably output the 3D gaze direction of the object to be tracked.

Images