RGB-D sight tracking method fusing head posture

Abstract

The invention discloses an RGB-D sight line tracking method fusing head postures, and belongs to the field of computer sight line estimation. Facial feature extraction adopts a RetinaFace network as a detection core, a detected human face region is cropped through an affine transformation strategy, a cropped image is input into a 3D Dense Face Alliance Version 2 (3DDFAV2) model after being normalized, and an eye RGB-D image block and a calculated head posture Euler angle are output through registration. In order to effectively integrate complementary information of RGB and a depth mode, a lightweight multi-mode sight tracking network is provided and is divided into three main levels: a feature extraction layer, a feature fusion layer and a regression prediction layer. According to the network, a double-branch multi-mode framework is constructed, five input modes including a binocular RGB image, a depth map and a head posture are processed respectively, Kalman filtering smoothing is utilized, a multi-mode information fusion strategy combining an attention and feature linear modulation (FiLM) mechanism is designed, and finally high-precision sight line regression is achieved through a multi-layer perceptron (MLP).

Description

Technical Field

[0001] This invention, which integrates RGB-D gaze tracking based on head posture, belongs to the field of computer vision. Background Technology

[0002] Human-computer interaction (HCI) technology, by establishing a two-way information feedback mechanism between humans and machines, enables users to naturally control the computing environment, and has had a revolutionary impact on the field of medical rehabilitation. Its core lies in supporting non-precise interaction channels (such as posture, vision, and head motion tracking) and a multi-channel integration framework, the latter requiring real-time fusion and collaborative parsing of multi-source heterogeneous inputs. Tracking-based projection AR interactive systems display virtual objects in real time through projection devices and dynamically adjust their spatial pose and scale according to environmental changes. In surface projection AR applications, to ensure an immersive operating experience for doctors, the core challenge of eye-to-projection screen collaborative interaction needs to be overcome. HCI technologies in this field encompass eye-tracking technology and projection distortion correction technology.

[0003] Eye tracking, a crucial element of human-computer interaction, is currently widely used in fatigue monitoring and educational assessment, with a relatively low penetration rate in medical applications. Eye estimation techniques based on RGB images are a mainstream research direction, primarily focusing on predicting eye direction or gaze location through facial features. Based on the type of input data, existing methods can be categorized into three main technical approaches: monocular image input methods, full-face image input methods, and binocular image patch input methods. For monocular image input methods, the single-branch structure lacks robustness to complex scenarios such as eye movement deviation, eyelid occlusion, and free head rotation. The semantic ambiguity of RGB textures (e.g., when the head turns left, the left eye is obscured by the bridge of the nose, allowing only the sclera to be observed) makes it difficult for the model to infer the true eye direction, becoming a core obstacle to the practical application of this technology. Technical limitations lie in the sensitivity to head movements and the reliance on manually labeled data, while low model complexity leads to insufficient generalization ability. For full-face image input methods, due to the presence of numerous facial features unrelated to eye movement (such as hair and background), these methods often require complex attention mechanisms to focus on key areas, increasing model complexity and computational burden. For binocular image patch input methods, cross-individual generalization remains a challenge due to insufficient consideration of the asymmetry of binocular feature spaces.

[0004] To alleviate the semantic ambiguity of single RGB modality, researchers have recently focused on innovations in "multi-branch feature decoupling" and "multi-modal geometric constraints," propelling the technology into a multi-branch fusion stage. Firstly, multi-branch structure design has become a mainstream trend. Secondly, multi-modal information fusion supplements geometric constraints. This stage breaks through the scene generalization bottleneck of single RGB, gradually shifting from single eye features to multi-modal inputs (such as face + eye), and improving performance through optimized network structures (such as VGG and ResNet). However, it still relies on external tools (such as OpenFace for head pose annotation and RealSense for depth acquisition), laying the groundwork for future improvements in end-to-end learning. RGB-D gaze tracking shows significant potential in depth perception and cross-modal fusion. While RGB-D multimodality has theoretical advantages, it is still constrained by bottlenecks such as data scale, computational efficiency, and biometric adaptation, urgently requiring collaborative breakthroughs in lightweight fusion architecture design, physiological constraint models, and terminal-friendly depth sensors. Summary of the Invention

[0005] In light of the aforementioned technical background, the applicant has designed an RGB-D gaze tracking method that integrates head pose. The objective of this invention is achieved as follows:

[0006] 1. A network framework for face capture and processing oriented towards face obstruction by masks is constructed, characterized by the following steps:

[0007] Step a: Introduce the Microsoft Azure Kinect DK sensor to simultaneously acquire high-resolution RGB images and depth image data, and construct a dual-modal input data stream.

[0008] Step b: Use the RetinaFace network as the detection core. The end-to-end output includes additional supervised branch prediction of occlusion information and coordinates of 5 facial key points.

[0009] Step c: To ensure complete coverage of the surgical cap and mask area, the detection frames with a confidence level greater than 0.95 are expanded by 20% in width and 15% in height.

[0010] Step d: The detected face region is cropped and scaled to a 120×120 input size using an affine transformation strategy to ensure that key facial structures are fully preserved.

[0011] Step e: After the cropped image is normalized, it is input into the 3D Dense Face Alignment Version 2 (3DDFA_V2) model. The fully connected layer outputs a 62-dimensional 3D Facial Shape Model (3DMM) parameter vector.

[0012] Step f: For the RGB-D camera configuration, spatial geometric registration of the RGB image and the depth map is achieved through the camera intrinsic parameter matrix.

[0013] Step g: After performing bilateral filtering to reduce noise on the depth map, crop it to generate a 60×36 pixel four-channel data block, and output the registered eye RGB-D image block (60×36×4) and the head pose Euler angles calculated from the rotation matrix R.

[0014] 2. To overcome the physical limitations of RGB modality, a new RGB-D multimodal information feature fusion network structure design is proposed, characterized by the following steps:

[0015] Step a: Use the EfficientNetV2-s network as the backbone network of the RGB branch to extract the RGB image features of both eyes.

[0016] Step b: The lightweight MobileNetV3-Small was used as the backbone of the Depth branch to extract the Depth image features of both eyes.

[0017] Step c: Combine Kalman filtering to achieve smooth head pose tracking. Use Kalman filtering to post-process the estimation results to effectively suppress measurement noise and pose jitter.

[0018] Step d: The feature fusion layer combines attention and feature-wise linear modulation (FiLM) to achieve the fusion of multimodal information from the feature output RGB-D and head pose described in steps a, b, and c.

[0019] Step e: A model constructed using a multilayer perceptron (MLP) performs regression prediction on the fused information described in step d. gaze It includes a Mish activation function and a Dropout layer to prevent overfitting, and the final output layer predicts the pitch and yaw of the view.

[0020] Compared with the prior art, the present invention has the following technical effects:

[0021] 1. In the operating room environment, surgeons wearing masks significantly interfere with the accuracy of vision-based eye tracking. To effectively mitigate this problem, this invention introduces the Microsoft Azure Kinect DK sensor to simultaneously acquire high-resolution RGB and depth image data, constructing a multimodal input data stream, such as... Figure 1As shown. Based on this, the present invention designs and implements a framework for facial image acquisition and processing in scenarios where masks are worn, aiming to improve the robustness of facial feature acquisition and analysis under occlusion conditions.

[0022] 2. To effectively integrate complementary information from RGB and depth modalities, this invention proposes a lightweight multimodal gaze tracking network, consisting of three main layers: Feature Extraction, Feature Fusion, and Regression Head. This network employs a dual-branch multimodal architecture, processing five input modalities: binocular RGB images, depth maps, and head pose. Kalman filtering is used to smooth head pose tracking, and a multimodal information fusion strategy combining attention and FiLM mechanisms is designed. Specifically, RGB-D-CAFM, a Cross-Attention Fusion Module (CAFM) for RGB-D eye features, is proposed. This module dynamically fuses RGB and depth features through a cross-attention mechanism. This dynamic attention allocation mechanism allows the model to focus on the geometric regions most discriminative of gaze direction. To further enhance the responsiveness of feature channels, a Squeeze-and-Excitation (SE) module is introduced after the attention output. The SE module explicitly models the interdependencies between channels and adaptively recalibrates the channel feature responses, thereby suppressing noisy channels and enhancing key features. This invention utilizes the FiLM mechanism to adaptively inject head pose information into the fused features, thereby significantly improving the accuracy of gaze estimation and its robustness under extreme head poses. Finally, high-precision gaze regression is achieved through a multilayer perceptron (MLP).

[0023] 3. This invention can be applied to devices such as robots and laptops, and does not require users to wear any auxiliary devices. It has a wide range of applications and good economic and application prospects. Attached Figure Description

[0024] Figure 1 This is a flowchart of a specific embodiment of the present invention.

[0025] Figure 2 This is a framework diagram of face capture and processing for faces obscured by masks, according to a specific embodiment of the present invention;

[0026] Figure 3 This is a diagram of a lightweight multimodal gaze tracking network structure according to a specific embodiment of the present invention;

[0027] Figure 4 This is a visualization of the preprocessing results according to a specific embodiment of the present invention;

[0028] Figure 5 This is a heatmap of the average angular error of yaw and pitch angles according to a specific embodiment of the present invention; Detailed Implementation

[0029] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0030] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the illustrations only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0031] Some exemplary embodiments of the invention have been described for illustrative purposes. It should be understood that the invention may be implemented in other ways not specifically shown in the accompanying drawings.

[0032] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0033] The face capture and processing flow for individuals with mask-covered faces in this specific implementation method is as follows: Figure 2 As shown, it includes the following steps:

[0034] Step a: Introduce the Microsoft Azure Kinect DK sensor to simultaneously acquire high-resolution RGB and depth image data, and construct a dual-modal input data stream.

[0035] Step b: Use the RetinaFace network as the RGB image detection core described in step a. The end-to-end output includes additional supervised branch prediction of occlusion information and coordinates of 5 facial key points.

[0036] Step c: To ensure complete coverage of the surgical cap and mask area, the detection frames with a confidence level greater than 0.95 described in step b are expanded by 20% in width and 15% in height.

[0037] Step c1: The key points in the center of the human eye are directly predicted by the network, without the need for an additional key point detection algorithm.

[0038] Step c2: Calculate the translation vector of the gaze transformation, which represents the three-dimensional translation vector t from the origin of the world coordinate system to the center of the human eye. gaze The expression is as follows:

[0039]

[0040] Where (X) eye , Y eye Z eye The translation vector represents the 3D coordinates of the binocular center mean values ​​in the world coordinate system after registration. This translation vector is used to construct a rigid body transformation from the world coordinate system to the viewpoint coordinate system, providing a geometric reference for subsequent line-of-sight direction estimation.

[0041] Step d: The detected face region is cropped and scaled to a 120×120 input size using an affine transformation strategy to ensure that key facial structures are fully preserved.

[0042] Step e, in conjunction with step d, involves normalizing the cropped image and inputting it into the 3DDFA_V2 model. The fully connected layer outputs a 62-dimensional 3D facial shape model (3DMM) parameter vector.

[0043] Step e1: The parameters include the first 12 dimensions of camera parameters, the rotation matrix R, and the translation amount t.

[0044] Step e2: Reconstruct 68 3D key points using the Basel face model.

[0045] Step e3, Preliminary estimation Obtain Euler angles:

[0046]

[0047] V∈R 3×N There are N 3D facial coordinates.

[0048] Step f: For the RGB-D camera configuration described in step a, spatial geometric registration of the RGB image and the depth map is achieved through the camera intrinsic parameter matrix.

[0049] Step g: After performing bilateral filtering and noise reduction on the depth map described in step f, crop it to generate a 60×36 pixel four-channel data block. Output the registered RGB-D image patch of the eye (60×36×4) and the Euler angles of the head pose (yaw / pitch / roll) calculated from the rotation matrix R:

[0050]

[0051] The lightweight multimodal gaze tracking network in this specific implementation method two is as follows: Figure 3 As shown, it includes the following steps:

[0052] Step a: Use the EfficientNetV2-s network as the backbone network of the RGB branch to extract the RGB image features of both eyes.

[0053] Step a1: In this embodiment, the original fully connected layer is removed, and only the feature extraction part is retained.

[0054] Step a2: Project the output channel to 128 dimensions.

[0055] Step a3: Use the Mish activation function to output RGB features.

[0056] Step b: The lightweight MobileNetV3-Small was used as the backbone of the Depth branch to extract the Depth image features of both eyes.

[0057] Step b1: In this embodiment, the first convolutional layer is modified to have a single-channel input.

[0058] Step b2: Initialize using the mean of the pre-trained weights and output the Depth feature.

[0059] Step c: Combine Kalman filtering to achieve smooth head pose tracking, and use Kalman filtering to post-process the estimation results.

[0060] Step c1, State transition matrix Assuming linear continuity of attitude changes:

[0061]

[0062] Where I3 is a 3×3 identity matrix and Δt is the inter-frame time interval.

[0063] Step c2: The observation model uses the attitude prediction results of 3DDFA_V2 as the measurement values. enter:

[0064]

[0065] Observation matrix H k =[I3O3] extracts only the positional component.

[0066] Step c3, the Kalman filter prediction-update loop, ensures the spatiotemporal consistency of attitude estimation. Prior states are inferred during the prediction phase:

[0067]

[0068]

[0069] Q kThe process noise covariance is used to control the confidence level of the motion model.

[0070] Step c4: In the update phase, correct the state estimate using the measurement results of 3DDFA_V2:

[0071]

[0072]

[0073]

[0074] Kalman gain K k Based on dynamic adjustments to prediction uncertainty and measurement noise, the new information term ( () indicates the deviation between the measured value and the predicted value.

[0075] Step c4: By inputting the original pose estimation results into the Kalman filter, the system can obtain a more stable and reliable optimal estimate of head pose information. .

[0076]

[0077] Step d: The feature fusion layer combines attention and FiLM to achieve the fusion of multimodal information of feature output RGB-D and head pose as described in steps a, b, and c.

[0078] Step d1: Design RGB-D-CAFM, a CAFM for RGB-D eye features; including the following steps:

[0079] Step d11: Treat the RGB features as a query (Q) and the depth features as keys (K) and values ​​(V). The mathematical expression is as follows:

[0080]

[0081] Where, d k This is the scaling factor. Through this mechanism, the model can automatically retrieve corresponding geometric information from the depth feature map based on the eye appearance (such as pupil position and eyelid opening / closing) in the RGB image.

[0082] Step d12: Following the attention output described in step b11, an SE module is introduced. The channel feature responses are adaptively recalibrated. The left-eye RGB features and depth features are processed by the SE module to obtain the left-eye features, and the right-eye RGB features and depth features are processed by the SE module to obtain the right-eye features.

[0083] Step d13: Combine the binocular features described in step b12 and obtain the binocular feature vector F through Global Avg Pooling. eyes .

[0084] Step d2: The FiLM mechanism is introduced to inject head pose information. The binocular feature vectors F... eyes As the modulated object, using the head pose vector P head Generate scaling factor γ and translation factor β:

[0085]

[0086]

[0087] Where W represents the linear transformation layer.

[0088] Step e: The MLP-based model performs regression prediction on the fused information described in step d. gaze It includes a Mish activation function and a Dropout layer to prevent overfitting, and the final output layer predicts the pitch and yaw of the view.

[0089] Step e1, t gaze Positional information is used for temporal smoothing via Kalman filtering. The parameters of the Kalman filter were experimentally optimized. The process noise covariance Q = q·I⁴, where q = 0.0001, and the observation noise covariance R = r·I², where r = 0.1. Both I⁴ and I² are identity matrices. The stable head pose estimation results are input into the gaze estimation network and fused with the features of the left and right eye images. gaze Obtain the view direction vector:

[0090]

[0091] Where I left I right Features of left and right eye images, This represents the head pose after Kalman filtering and smoothing.

[0092] Step e2, final viewpoint pose vector From the perspective direction and head rotation matrix Combining the results, we get:

[0093]

[0094] To further illustrate the technical solution of the present invention, a comparative explanation is provided below in conjunction with specific experiments.

[0095] To verify the effectiveness of the method, experiments were conducted using the EyeDiap RGB-D dataset, which contains 96 video clips from 16 subjects. These videos were captured under two visual target conditions: screen targets and 3D floating targets. Due to the lack of CS video data for two subjects, the final experiment used 114,000 RGB-D images from 14 subjects, covering different lighting and head pose variations. This invention selects VGA videos of screen targets, sampling one image every 15 frames, and truncating the sampled images from each video to ensure an equal number of images.

[0096] The experiments were conducted using PyTorch 2.4.1 + cu121 on Ubuntu 20.04.6 LTS, with Python 3.8.20 (GPU acceleration for network training). The system was configured with a 13th Gen Intel® Core™ i5-13600KF 14-core CPU, 64GB of RAM, and an NVIDIA GeForce RTX 4090 graphics card. In all experiments, the batch size was set to 256, and the Adam optimizer was used. The learning rate was 0.0005, and the network was trained for a total of 100 epochs.

[0097] The dataset is standardized according to the current gaze point estimation paradigm, with the eye center as the reference point. After normalization, the eye images undergo Contrast Limited Adaptive Histogram Equalization (CLAHE) to adapt to different lighting conditions. Since the RGB camera and depth camera have different spatial positions and orientations, coordinate system transformation is required to achieve accurate registration of multimodal data. Visualization of eye localization and head pose registration is shown below. Figure 4 As shown.

[0098] To verify the accuracy and robustness of this invention, quadruple cross-validation was used on the EyeDiap dataset, and multiple experiments were conducted, with the mean and standard deviation calculated. Detailed ablation experiments were performed to verify the effectiveness of the multimodal fusion strategy and the impact of different input modalities on model performance. The experimental results are shown in Table 1, comparing the average error performance of Direct Concatenation, Adaptive Weighted Fusion, MLP Expanded Concatenation, and our invention under different modal combinations.

[0099]

[0100] As can be seen from the overall trend in Table 1, combining multiple modal inputs generally leads to performance improvements, regardless of the fusion method used. While the direct concatenation strategy is simple, its error is the highest among all trimodal results, indicating that simple feature stacking is insufficient to fully exploit intermodal interaction information. In contrast, adaptive weighted fusion and MLP extended concatenation, by introducing weighting mechanisms or nonlinear mappings, reduce the trimodal errors to 5.48° and 5.47° respectively, demonstrating the necessity of deep feature processing. This invention achieves optimal results in all settings. Particularly in the case of full trimodal input, this invention achieves the lowest average error of 4.85 ± 0.32°. Compared to the best-performing baseline method, MLP extended concatenation, the error is further reduced by approximately 10.6%. Furthermore, even in the bimodal setting using only RGB and Depth, the error of this invention is already superior to the trimodal results of some comparative methods. The fusion network proposed in this invention can more effectively integrate heterogeneous modal features, exhibiting stronger feature representation capabilities and robustness compared to traditional concatenation or weighting strategies, thus achieving optimal experimental performance.

[0101] To conduct a more in-depth analysis, Figure 5 The heatmaps showing the average angular error of the yaw and pitch angles of this invention under different input combinations are presented, intuitively illustrating that introducing multimodal data such as depth maps and head attitude can significantly improve the prediction accuracy of the model, strongly supporting the quantitative data in Table 1. Figure 5 As shown in (a), when using only RGB images for gaze tracking, the heatmap color rapidly fades as the gaze angle gradually deviates from the center, especially in the edge regions. This indicates that relying solely on 2D appearance features is insufficient to address gaze feature distortion at large angles, resulting in poor robustness of the model in extreme gaze directions. Introducing a depth map significantly shrinks the high-error region. Depth information provides crucial 3D facial geometry, helping the model more accurately perceive the relative spatial position of the eyeball and the camera. This geometric prior effectively alleviates the ambiguity of 2D images in perspective projection, improving the model's gaze prediction accuracy in non-central regions.

[0102] In gaze tracking tasks, the gaze direction is typically determined by both head pose and eye rotation relative to the head. Introducing head pose data provides the model with a strong prior, helping it effectively decouple eye movement from head movement. Therefore, even in complex gaze scenarios, the model can significantly reduce prediction errors using this prior knowledge. As can be seen, even in edge gaze regions with extremely high errors, this invention maintains a low error rate. This demonstrates that this invention successfully combines the geometric structure information of the depth map with the directional prior information of the head pose. Depth information corrects spatial location, while head pose assists in decoupling analysis; the two complement each other, minimizing the blind spots of a single modality in extreme gaze conditions. It intuitively proves that relying solely on RGB appearance features is insufficient in gaze tracking tasks. Through a multimodal fusion strategy, depth and pose information can be effectively integrated, significantly improving the model's prediction accuracy and robustness across the entire gaze range.

[0103] Table 2 shows the comparison results between the present invention and existing models.

[0104]

[0105] As can be seen from the experimental data of the above embodiments, the robustness, universality and accuracy of the improved estimation system and method of the present invention have been fully verified, achieving accurate eye tracking.

[0106] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A network framework for face capture and processing oriented towards face obstruction by masks is constructed, characterized in that, Includes the following steps: Step a: Introduce the Microsoft Azure Kinect DK sensor to simultaneously acquire high-resolution RGB images and depth image data, and construct a dual-modal input data stream.

2. Step b: Use the RetinaFace network as the detection core. The end-to-end output includes additional supervised branch prediction of occlusion information and coordinates of 5 facial key points.

3. Step c: To ensure complete coverage of the surgical cap and mask area, the detection frames with a confidence level greater than 0.95 are expanded by 20% in width and 15% in height.

4. Step d: The detected face region is cropped and scaled to a 120×120 input size using an affine transformation strategy to ensure that key facial structures are fully preserved.

5. Step e: After the cropped image is normalized, it is input into the 3D Dense Face Alignment Version 2 (3DDFA_V2) model. The fully connected layer outputs a 62-dimensional 3D facial shape model (3DMM) parameter vector.

6. Step f: For the RGB-D camera configuration, achieve spatial geometric registration between the RGB image and the depth map through the camera intrinsic parameter matrix.

7. Step g: After performing bilateral filtering to reduce noise on the depth map, crop it to generate a 60×36 pixel four-channel data block, and output the registered eye RGB-D image block (60×36×4) and the head pose Euler angles calculated from the rotation matrix R.

8. To overcome the physical limitations of RGB modality, a shift is made to the design of an RGB-D multimodal information feature fusion network structure, characterized by: Includes the following steps: Step a: Use the EfficientNetV2-s network as the backbone network of the RGB branch to extract the RGB image features of both eyes.

9. Step b: The lightweight MobileNetV3-Small was used as the backbone of the Depth branch to extract the Depth image features of both eyes.

10. Step c: Combine Kalman filtering to achieve smooth head pose tracking. Use Kalman filtering to post-process the estimation results to effectively suppress measurement noise and pose jitter.

11. Step d: The feature fusion layer achieves the fusion of multimodal information of the feature output RGB-D and head pose described in steps a, b, and c by combining attention and feature-wise linear modulation (FiLM). Step e: A model constructed using a multilayer perceptron (MLP) performs regression prediction on the fused information described in step d. gaze It includes a Mish activation function and a Dropout layer to prevent overfitting, and the final output layer predicts the pitch and yaw of the view.

Images