Lightweight graph neural network-based multi-modal gaze estimation method and system, electronic device and storage medium

Abstract

The application provides a multi-modal line-of-sight estimation method and system based on a lightweight graph neural network, an electronic device and a storage medium, and relates to the fields of human-computer interaction application and line-of-sight estimation technology. First, multi-modal heterogeneous data is acquired, and visual features are extracted from visual appearance data. Then, the visual features, geometric direction information and spatial positioning coordinates are projected into a 32-dimensional space. Next, a graph structure is constructed, and a multi-head attention graph neural network with a two-layer structure is used to process the constructed graph to obtain graph-level features. Further, the graph-level features are aggregated and fused with the visual features, geometric direction information and spatial positioning coordinates. Finally, a full connection layer is used to perform line-of-sight estimation prediction based on the fused features, and coordinate conversion is performed to convert the line-of-sight estimation prediction into a normalized three-dimensional line-of-sight vector. The application is suitable for mobile deployment of lightweight and accurate line-of-sight estimation, and can effectively realize multi-modal feature integration and rapid specific subject adaptation.

Description

Technical Field

[0001] This invention relates to the fields of human-computer interaction applications and gaze estimation technology, specifically to a multimodal gaze estimation method, system, electronic device, and storage medium based on a lightweight graph neural network. Background Technology

[0002] The demand for gaze estimation technology is growing in human-computer interaction, virtual and augmented reality systems, eye-tracking applications, and assistive technologies.

[0003] However, the growing demand has also exposed key limitations of existing solutions: 1) Current appearance-based gaze estimation systems typically process multimodal inputs through separate computational branches, preventing the effective integration of spatial relationships between facial features, eye position, and head orientation data; 2) Model-based methods require specialized hardware, such as infrared illumination and spot detection systems, making them unsuitable for consumer devices and unconstrained environments; 3) State-of-the-art systems typically require 7-10 million parameters, creating a huge computational burden and hindering the deployment of gaze estimation technologies in mobile devices, embedded systems, and AR / VR wearables; 4) Current architectures suffer from poor generalization across datasets and require numerous time-consuming and computationally expensive subject-specific calibration procedures.

[0004] Based on this, this application proposes a multimodal gaze estimation technique based on a lightweight graph neural network in order to solve one or more of the problems mentioned above. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides a multimodal gaze estimation method, system, electronic device, and storage medium based on a lightweight graph neural network. This lightweight and accurate gaze estimation method is suitable for mobile deployment, while also achieving effective multimodal feature integration and rapid subject-specific adaptation, thus solving one or more of the aforementioned problems.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] Firstly, this application proposes a multimodal gaze estimation method based on a lightweight graph neural network, the method comprising:

[0010] Acquire multimodal heterogeneous data that is related to and complementary to the gaze direction estimation, wherein the multimodal heterogeneous data includes visual appearance data, geometric orientation information and spatial positioning coordinates;

[0011] Visual features are extracted from visual appearance data based on depthwise separable convolution and squeeze excitation modules;

[0012] Visual features, geometric orientation information, and spatial positioning coordinates are projected into the same vector space;

[0013] Construct a graph structure, which contains several nodes representing multimodal heterogeneous data or features corresponding to multimodal heterogeneous data;

[0014] The graph is constructed based on a multi-head attention graph neural network with a two-layer structure to obtain graph-level features; the first layer of the two-layer structure expands the dimension of the node features, and the second layer reduces the dimension of the expanded features.

[0015] The graph-level features are aggregated and fused with visual features, geometric orientation information, and spatial positioning coordinates.

[0016] A gaze estimation prediction is performed using a fully connected layer and based on fused features.

[0017] Based on coordinate transformation, the gaze estimation prediction is converted into a normalized three-dimensional gaze vector.

[0018] In one embodiment, the visual appearance data includes facial image data, eye cropped image data, geometric orientation information including head rotation data, and spatial positioning coordinates including the three-dimensional eye center position.

[0019] In one embodiment, constructing the graph structure includes:

[0020] Construct a fully connected graph consisting of six nodes; the six nodes include: facial features, left eye features, right eye features, head rotation data, left eye 3D position, and the node corresponding to the right eye 3D position;

[0021] Learnable edge weights are established between the six node pairs.

[0022] In one embodiment, coordinate transformation includes converting spherical coordinates to Cartesian coordinates.

[0023] In one embodiment, the method further includes:

[0024] When estimating the line of sight for a specific subject, the multimodal heterogeneous data also includes subject-specific calibration data.

[0025] In a preferred embodiment, the specific subject calibration includes:

[0026] During calibration, only the specific subject calibration embedding and final fully connected layer parameters are updated, while all other network parameters remain frozen.

[0027] In one embodiment, the graph constructed based on a multi-head attention graph neural network with a two-layer structure is processed to obtain graph-level features, including:

[0028] Step 4.1: Initialize the six graph nodes using projection feature embedding;

[0029] Step 4.2: Calculate the attention coefficients for all node pairs using linear projection;

[0030] Step 4.3: Apply LeakyReLU activation and softmax normalization to the attention coefficients;

[0031] Step 4.4: Aggregate neighboring features using weighted attention coefficients;

[0032] Step 4.5: Determine if the current processing corresponds to the first GNN layer. If yes, proceed to step 4.6; otherwise, proceed to step 4.7.

[0033] Step 4.6: Extend the node features to 256 dimensions;

[0034] Step 4.7: Reduce node features to 32 dimensions;

[0035] Step 4.8: Apply the second-layer GNN for processing;

[0036] Step 4.9: Flatten all node features and connect them into a unified representation;

[0037] Step 4.10: Output the final graph-level feature vector.

[0038] In one embodiment, converting the gaze estimation prediction into a normalized three-dimensional gaze vector based on coordinate transformation includes:

[0039] An orthogonal normalized coordinate frame is constructed using cross product, and parametric rays emanating from the center of the 3D eye are generated.

[0040] Preferably, the parametric ray is capable of performing intersection calculations with objects including virtual objects, user interfaces, and physical surfaces in spatial applications.

[0041] In one embodiment, the multi-head attention comprises four attention heads that process in parallel, wherein:

[0042] The first is used to capture specific relationship patterns between the six input modalities represented within a six-node graph structure;

[0043] The first two are used to capture the substitution relationship patterns between input modes;

[0044] The first three are used to provide additional parallel processing capabilities for modeling comprehensive relationships;

[0045] The first four can be accomplished by providing a fourth independent attention mechanism to complete the parallel processing array.

[0046] Secondly, this application further provides a multimodal gaze estimation system based on a lightweight graph neural network, the system comprising:

[0047] The multimodal input module is configured to acquire multimodal heterogeneous data that is related to and complementary to the gaze direction estimation, the multimodal heterogeneous data including visual appearance data, geometric orientation information and spatial positioning coordinates;

[0048] The feature extraction module is configured to extract visual features from visual appearance data based on depthwise separable convolution and squeeze excitation module.

[0049] The projection module is configured to project the visual features, geometric orientation information, and spatial positioning coordinates into the same vector space;

[0050] The graph construction module is configured to construct a graph structure, which contains several nodes representing multimodal heterogeneous data or features corresponding to multimodal heterogeneous data.

[0051] Multi-head attention GNN is configured to obtain graph-level features based on the graph constructed by the multi-head attention graph neural network with a two-layer structure. In the two-layer structure, the first layer expands the dimension of the node features, and the second layer reduces the dimension of the expanded features.

[0052] The feature fusion module is configured to aggregate the graph-level features and fuse them with the visual features, geometric orientation information, and spatial positioning coordinates.

[0053] The regression module is configured to utilize fully connected layers and perform gaze estimation prediction based on fused features;

[0054] The 3D reconstruction module is configured to convert gaze estimation predictions into normalized 3D gaze vectors based on coordinate transformation.

[0055] Thirdly, this application further proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multimodal gaze estimation method based on a lightweight graph neural network as described in any of the preceding claims.

[0056] Fourthly, this application finally proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the multimodal gaze estimation method based on a lightweight graph neural network as described in any of the preceding claims.

[0057] (III) Beneficial Effects

[0058] This invention provides a multimodal gaze estimation method, system, electronic device, and storage medium based on a lightweight graph neural network. Compared with existing technologies, it has the following advantages:

[0059] 1. This application processes heterogeneous inputs, including facial images, cropped eye images, head pose data, and 3D eye center positions, through a unified feature encoding framework. This solves the problem in existing technologies where multimodal inputs are processed using independent branches of convolutional neural networks, leading to the independent processing of facial images, cropped eye images, and head pose data, which fails to meaningfully model cross-modal spatial relationships. This approach can effectively capture complex relationship structures and offers superior feature fusion and cross-modal interaction.

[0060] 2. This application constructs a fully connected graph with six nodes to represent different modalities and uses a multi-head attention mechanism to model cross-modal relationships and dependencies.

[0061] 3. This application can unify feature projection, mapping all heterogeneous inputs (visual, geometric, and calibration data) into a common 32-dimensional embedding space, enabling coherent reasoning across modalities. The two-layer graph neural network employs attention-based reasoning to capture the spatial and semantic relationships between facial components, head orientation, and eye positions.

[0062] 4. This application reconstructs 3D line-of-sight vectors and converts the predicted pitch and yaw angles into mathematically stable 3D rays, which are suitable for intersection calculations with screens, surfaces, and augmented or virtual reality coordinate frames.

[0063] 5. This application combines subject-specific calibration embedding, which enables rapid personalization with minimal training samples, improving cross-dataset generalization ability while maintaining computational efficiency.

[0064] 6. The complete system of this application has fewer parameters (approximately 3.48 million), achieves real-time performance, and is suitable for deployment on mobile devices, embedded systems, and low-power AR / VR platforms. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a flowchart of the multimodal gaze estimation method based on a lightweight graph neural network, as described in this application.

[0067] Figure 2 This is a system architecture diagram of the LightGazeNet system shown in the embodiment.

[0068] Figure 3 This is a flowchart illustrating a specific subject calibration method in the embodiments.

[0069] Figure 4 The flowchart of the graph neural network processing method shown in the embodiment is as follows. Figure 1 .

[0070] Figure 5 This is a flowchart illustrating the three-dimensional line-of-sight vector reconstruction method in the embodiment.

[0071] Figure 6 The feature embedding component architecture is shown in the embodiment.

[0072] Figure 7 This is a system diagram of the line-of-sight estimation architecture shown in the embodiment.

[0073] Figure 8 This is a flowchart illustrating the line-of-sight estimation process in the embodiment.

[0074] Figure 9 The flowchart of the graph neural network processing method shown in the embodiment is as follows. Figure 2 .

[0075] Figure 10 This is a flowchart of the line-of-sight estimation processing pipeline shown in the embodiment.

[0076] Figure 11 This is a sequence diagram of the multimodal line-of-sight estimation process shown in the embodiment.

[0077] Figure 12 This is a sequence diagram of the multimodal gaze estimation data processing workflow shown in the embodiment.

[0078] Figure 13 This is a sequence diagram of the graph neural network processing workflow shown in the embodiment.

[0079] Figure 14 This is a sequence diagram illustrating the multimodal feature integration and 3D line-of-sight vector reconstruction process shown in the embodiment.

[0080] Figure 15 This is a system diagram of the LightGazeNet deployment system shown in the embodiment.

[0081] Figure 16 This is a multi-head attention architecture for processing multimodal gaze estimation data, as shown in the embodiment. Detailed Implementation

[0082] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0083] It should be noted that the multimodal gaze estimation method, system, electronic device, and storage medium based on lightweight graph neural networks proposed in this application can be applied to, but are not limited to, the following scenarios:

[0084] Implemented in virtual and augmented reality systems, it provides accurate real-time eye tracking for immersive user interfaces, enabling natural interaction with virtual objects, menu navigation, and foveated rendering optimization, while reducing computational load by rendering high-resolution graphics only where the user is looking.

[0085] We propose a lightweight gaze estimation system for mobile devices and smartphones to enable hands-free control of applications, provide accessibility features for users with mobility impairments, and enhance the user experience in games and social media applications through gaze-based interaction mechanisms.

[0086] Implemented in automotive systems, it provides driver attention monitoring and drowsiness detection functions, improves vehicle safety by warning the driver when the driver's gaze pattern indicates distraction or fatigue, and enables a gaze-based infotainment system that allows the driver to stay focused on the road.

[0087] Implemented in human-computer interaction systems for desktop and laptop computers, it provides a more intuitive user interface through eye-based cursor movement, text selection, and application switching, which is particularly beneficial for users with physical disabilities who may have limited mobility or dexterity.

[0088] This provides a gaze estimation solution for eye-tracking research and medical diagnosis, enabling researchers and clinicians to study visual attention patterns, diagnose neurological disorders, assess cognitive function, and monitor rehabilitation progress using portable, cost-effective devices that do not require specialized infrared hardware.

[0089] Implemented in educational technology and e-learning platforms, this system analyzes students' attention and engagement levels during online courses, provides teachers with valuable feedback on the effectiveness of content, and enables an adaptive learning system that adjusts presentation based on students' focus of attention.

[0090] Implemented in retail and marketing applications, digital signage and interactive displays that respond to customer gaze patterns deliver personalized content recommendations, measure advertising effectiveness, and create engaging shopping experiences in stores and public spaces.

[0091] Example 1:

[0092] Firstly, this invention proposes a multimodal gaze estimation method based on a lightweight graph neural network, see [link to relevant documentation]. Figure 1 The method includes:

[0093] S1. Acquire multimodal heterogeneous data that is related to and complementary to the line-of-sight direction estimation, wherein the multimodal heterogeneous data includes visual appearance data, geometric orientation information and spatial positioning coordinates;

[0094] S2. Visual features are extracted from visual appearance data based on depthwise separable convolution and squeeze excitation modules;

[0095] S3. Project visual features, geometric orientation information, and spatial positioning coordinates into the same vector space;

[0096] S4. Construct a graph structure, which contains several nodes representing multimodal heterogeneous data or features corresponding to multimodal heterogeneous data;

[0097] S5. Obtain graph-level features by processing the graph constructed based on a multi-head attention graph neural network with a two-layer structure; wherein, the first layer of the two-layer structure expands the dimension of the node features, and the second layer reduces the dimension of the expanded features.

[0098] S6. Aggregate the graph-level features and fuse them with visual features, geometric orientation information, and spatial positioning coordinates;

[0099] S7. Utilize fully connected layers and perform gaze estimation and prediction based on fused features;

[0100] S8. Based on coordinate transformation, the gaze estimation prediction is converted into a normalized three-dimensional gaze vector.

[0101] This embodiment proposes a multimodal gaze estimation method based on a lightweight graph neural network. This method achieves real-time gaze estimation performance through system dimensionality transformations and lightweight processing operations implemented in sequential steps. It achieves computational efficiency suitable for mobile deployment while maintaining cross-modal relationship modeling capabilities, thus enhancing gaze estimation accuracy compared to traditional standalone processing methods.

[0102] The following is in conjunction with the appendix Figure 1-16 The following details the implementation process of an embodiment of the present invention, including explanations of the specific steps S1-S8.

[0103] S1. Acquire multimodal heterogeneous data that is related to and complementary to the line-of-sight estimation. The multimodal heterogeneous data includes visual appearance data, geometric orientation information, and spatial positioning coordinates.

[0104] Acquire multimodal heterogeneous data that provides complementary information for gaze direction estimation. This multimodal heterogeneous data includes visual appearance data, geometric orientation information, and spatial positioning coordinates from the facial region. In some embodiments, the visual appearance data from the facial region includes, but is not limited to, facial image data and eye-cropped image data; the geometric orientation information includes, but is not limited to, user head rotation data; and the spatial positioning coordinates include, but are not limited to, the three-dimensional eye center position.

[0105] S2. Visual features are extracted from visual appearance data based on depthwise separable convolution and squeeze excitation modules.

[0106] In this step, visual appearance data, including facial images and eye cropping data, is processed through lightweight convolution operations and a squeeze excitation module to extract visual features. In some embodiments, this step can ultimately generate 576-dimensional feature vectors that capture unique visual features from the input data of the facial and eye regions.

[0107] It should be noted that, by leveraging the squeeze-excitation module, and through squeezing (global information aggregation) and excitation (channel weight learning), the feature channel responses are adaptively recalibrated, thereby enhancing the feature representation capability of the convolutional neural network with almost no increase in computational cost. This operation reduces computational load while maintaining the effectiveness of feature extraction.

[0108] S3. Based on a linear transformation with a GELU activation function, the visual features, geometric orientation information, and spatial positioning coordinates are projected into the same vector space.

[0109] This step projects all heterogeneous inputs into a single vector space (preferably a common 32-dimensional embedding space) to convert them into a unified dimensional representation, enabling coherent cross-modal processing. Heterogeneous inputs include extracted visual features, geometric orientation information, and spatial positioning coordinates, which are then converted into a unified dimensional representation to facilitate coherent cross-modal processing. In some embodiments, a linear transformation with a GELU activation function is applied to map the 576-dimensional visual features into a standardized 32-dimensional space, while geometric orientation information (such as the head rotation matrix) and spatial positioning coordinates (such as 3D eye center coordinates) are processed through separate projection paths. For example, the head rotation data is flattened into a nine-dimensional 3x3 matrix, which is projected into a 32-dimensional embedding space. The 3D eye center position, including the left and right eye center coordinates, is also projected from 3D to 32D.

[0110] S4. Construct a graph structure, which contains several nodes representing multimodal heterogeneous data or features corresponding to multimodal heterogeneous data.

[0111] This step creates a graph structure, constructing a fully connected graph containing several nodes representing multimodal heterogeneous data or features corresponding to multimodal heterogeneous data, to represent the input of the multimodal heterogeneous data. In some embodiments, the fully connected graph contains at least six nodes, each corresponding to an input type for processing, specifically including facial features, left eye features, right eye features, head rotation data, left eye 3D position, and right eye 3D position. Preferably, learnable edge weights can be established between all node pairs to achieve comprehensive relationship modeling between different input modalities.

[0112] S5. Obtain graph-level features by processing the graph constructed based on a multi-head attention graph neural network with a two-layer structure; wherein, in the two-layer structure, the first layer expands the dimension of the node features, and the second layer reduces the dimension of the expanded features.

[0113] This step processes the fully connected graph constructed in the previous steps using a two-layer multi-head attention graph neural network to obtain graph-level features. An attention mechanism is applied, using linear projection, connection operations, and the LeakyReLU activation function to calculate the attention coefficients and other relational information for all node pairs to obtain graph-level features. In some embodiments, parallel attention heads are used to capture multiple relational patterns simultaneously. The first layer expands the node features to 256 dimensions, and the second layer reduces the features back to 32 dimensions to improve computational efficiency. Preferably, the multi-head attention graph neural network includes four parallel attention heads, which operate using independent parameter sets to simultaneously capture different relational patterns.

[0114] S6. Aggregate the graph-level features and fuse them with the visual features, geometric orientation information, and spatial positioning coordinates.

[0115] This step aggregates graph-level features and fuses them with the original CNN features and geometric input. Specifically, the relational information captured by the graph neural network processing (i.e., graph-level features) is combined with raw feature representations from the original CNN features (i.e., extracted visual features) and the original geometric features (i.e., geometric orientation information and spatial location coordinates). In some embodiments, a unified 384-dimensional relational vector from the graph processing is integrated with visual appearance features and spatial orientation data to create a comprehensive multimodal representation.

[0116] S7. Utilize fully connected layers and perform gaze estimation and prediction based on fused features.

[0117] This step uses fully connected layers to predict the estimated line-of-sight angles, which include at least the line-of-sight pitch and yaw angles. A fully connected processing network processes the fused feature representation, applying a nonlinear transformation to map the integrated features to the angular line-of-sight direction parameters. This generates pitch and yaw angle predictions, which represent the line-of-sight direction in spherical coordinates, suitable for subsequent 3D transformation operations.

[0118] S8. Based on coordinate transformation, the gaze estimation prediction is converted into a normalized three-dimensional gaze vector.

[0119] This step uses a spherical-to-Cartesian transformation to convert the predicted angle into a normalized 3D gaze vector. In some embodiments, the 3D gaze vector reconstruction module applies trigonometric functions to convert the angle prediction into a Cartesian coordinate representation, generating a unit vector representing the gaze direction in 3D space. This vector can work in conjunction with the orthogonal basis building unit and the camera coordinate transformation unit to produce parametric rays suitable for spatial interaction applications and coordinate system integration.

[0120] refer to Figure 2 This paper presents a comprehensive system architecture for multimodal gaze estimation based on a lightweight graph neural network, which transforms heterogeneous input data into accurate 3D gaze predictions through four sequential processing stages. The method implements a hierarchical structure that systematically processes diverse input modalities through specialized components, including feature extraction, dimensionality normalization, relation modeling, and vector reconstruction operations.

[0121] 2.1) Multimodal input processing.

[0122] Multimodal input processing receives and organizes six different types of input data for comprehensive gaze estimation processing. In specific implementations, multimodal input processing includes:

[0123] Facial image input provides overall facial appearance information captured from the user's facial regions. The facial image input can provide visual features that capture facial structure, orientation, and contextual information around the eye regions.

[0124] Left and right eye cropping provides specialized visual data from individual eye regions. Left and right eye cropping can provide detailed appearance information from each eye, enabling focused analysis of eye-specific features, including pupil position, iris pattern, and eyelid configuration. In some embodiments, left and right eye cropping can be generated through precise localization of eye keypoints, followed by cropping to extract a rectangular region centered on each eye.

[0125] Head rotation data can provide geometric orientation information representing the three-dimensional pose of the user's head relative to the camera coordinate system. Head rotation data can include rotation matrix information describing the spatial relationship between the user's head position and the imaging device's reference frame. Head rotation data can be represented as a flattened 9-dimensional vector derived from the 3x3 rotation matrix that captures head pose features.

[0126] The 3D eye center can provide spatial positioning coordinates, which represent the three-dimensional position of each eye center within the camera coordinate system. The 3D eye center can provide the coordinates of the left and right eye centers calculated through geometric analysis of facial key points combined with depth estimation techniques. In some embodiments, the 3D eye center can provide 6D coordinate data, representing the spatial position of the two eye centers in three-dimensional space.

[0127] Calibration embeddings can provide subject-specific personalized parameters, enabling personalized gaze estimation model adaptation. A calibration embedding can contain a 6-dimensional learnable embedding vector representing individual user characteristics and gaze patterns without modifying the underlying network architecture. Calibration embeddings can be rapidly personalized through selective parameter updates, which improve the accuracy of subject-specific gaze estimation.

[0128] 2.2) Unified feature coding.

[0129] Unified feature encoding transforms heterogeneous input data from multimodal input processing into a standardized dimensionality representation suitable for subsequent graph-based processing. Unified feature encoding applies system transformation operations to map diverse input modalities to a common embedding space while preserving the unique features of each input type. In specific implementations, unified feature encoding includes:

[0130] Depthwise separable convolution processes visual input from facial images, left-eye cropping, and right-eye cropping. It applies lightweight convolutional operations to extract unique visual features while maintaining computational efficiency suitable for mobile deployments. Compared to standard convolutional operations, depthwise separable convolution reduces the number of parameters and computational requirements while preserving the effectiveness of feature extraction.

[0131] The squeeze activation module can enhance the quality of feature representations generated by depthwise separable convolutions through a channel attention mechanism. The squeeze activation module can apply global average pooling, followed by a fully connected layer and sigmoid activation to generate channel attention weights. In some embodiments, the squeeze activation module can improve feature discriminativity by emphasizing informative channels while suppressing the extraction of less relevant feature channels in the visual representation.

[0132] The linear projection layer transforms the enhanced visual features from the squeeze excitation module into a unified dimensional space. It applies a learned linear transformation to map the 576-dimensional visual features extracted from facial and eye inputs into a standardized 32-dimensional embedding representation. This linear projection layer ensures dimensional compatibility between the visual features and the geometric and calibration data for subsequent graph processing operations.

[0133] GELU activation can provide nonlinear transformation capabilities within linear projection layers to enhance the quality of feature representations. GELU activation can apply Gaussian error linear unit activation functions, which provide smooth nonlinearity and improved gradient flow features compared to traditional activation functions. GELU activation can achieve efficient feature transformation while maintaining training stability and convergence properties.

[0134] 2.3) Graph Neural Networks.

[0135] Graph neural networks (Graph Neural Networks) model cross-modal relationships and dependencies between unified feature representations generated by unified feature encoding. Graph Neural Networks can implement attention-based processing operations that capture spatial and semantic interactions between different input modalities through system relationship modeling. In specific implementations, Graph Neural Networks include:

[0136] A six-node graph structure represents each input modality as a distinct node within a fully connected graph topology. This structure establishes comprehensive connections between all input modalities, allowing each node to directly influence every other node within the graph framework. In some embodiments, the six-node graph structure may correspond to facial features, left eye features, right eye features, head rotation data, left eye 3D position, and right eye 3D position, with learnable edge weights between all node pairs.

[0137] Multi-head attention can process six-node graph structures through parallel attention mechanisms that simultaneously capture multiple types of relation patterns. Multi-head attention can compute attention coefficients for all node pairs using learned linear projections, connection operations, and activation functions. It can operate on multiple parallel attention heads with independent parameter sets to achieve comprehensive cross-modal relation modeling.

[0138] Two-layer processing can achieve hierarchical feature refinement through dimensionality expansion and reduction operations, which enhance representational power while maintaining computational efficiency. Two-layer processing can expand node features to 256 dimensions in the first layer to capture complex relational patterns, and reduce the expanded features back to 32 dimensions in the second layer. In some embodiments, two-layer processing can apply a full attention mechanism in both processing layers to achieve systematic feature refinement across graph structures.

[0139] 2.4) Output generation.

[0140] Output generation transforms the processed graph representation into accurate 3D gaze vector predictions suitable for spatial applications. Output generation can integrate relational information captured through graph processing with complementary feature representations to generate comprehensive gaze direction predictions. Specifically, output generation includes:

[0141] The feature fusion module combines graph-level representations from graph neural networks with raw CNN features, raw geometric features, and subject-specific calibration embeddings. The feature fusion module integrates diverse feature representations through fusion operations that ensure appropriate weighting and combination of heterogeneous input modalities. The feature fusion module can create a unified feature representation that maximizes gaze estimation accuracy under different subject and environmental conditions.

[0142] The regression network can process integrated features from the feature fusion module to generate angular gaze direction predictions. The regression network can apply fully connected layers with non-linear activation functions to map the fused multimodal features to gaze direction parameters in spherical coordinates. In some embodiments, the regression network can produce predicted pitch and yaw angles, which represent the user's gaze direction relative to the camera coordinate system.

[0143] 3D vector reconstruction can transform angle predictions from regression networks into normalized 3D view vectors suitable for spatial applications and coordinate system integration. 3D vector reconstruction can use sine and cosine functions to apply trigonometric transformations to generate 3D coordinate components from predicted pitch and yaw angles. 3D vector reconstruction can work in conjunction with 3D view vector reconstruction modules, orthogonal basis building units, and camera coordinate transformation units to generate parametric rays with defined origins and direction vectors. These rays can be used to compute intersections with virtual objects, user interface elements, and physical surfaces in augmented reality and virtual reality environments.

[0144] A comprehensive system architecture for multimodal gaze estimation based on lightweight graph neural networks features a hierarchical structure that enables system transformation of heterogeneous input data through dedicated processing stages. These stages capture comprehensive cross-modal relationships while maintaining computational efficiency. This system architecture achieves real-time performance with approximately 3.48 million parameters, making it suitable for deployment on mobile devices, embedded systems, and low-power AR / VR platforms. Furthermore, it maintains gaze estimation accuracy through integrated processing of visual appearance features, geometric orientation information, spatial positioning coordinates, and subject-specific calibration parameters.

[0145] Example 2:

[0146] This embodiment, based on the multimodal gaze estimation method proposed in Embodiment 1 and its preferred embodiments, further includes a step of initializing a specific subject's six-dimensional calibration embedding vector, which enables personalized adaptation with minimal training samples. (Reference) Figure 3 The specific subject calibration method includes the following steps:

[0147] 3.1) Initialize the target subject's 6-dimensional calibration embedding vector.

[0148] Learnable parameters representing individual user characteristics and gaze patterns can be created without modifying the underlying network architecture.

[0149] 3.2) Collect the minimum number of calibration samples from the target subject.

[0150] Gaze data points are collected, which provide subject-specific training information for the personalized gaze estimation model. In some embodiments, as few as 32 calibration samples can be collected to achieve effective personalization while minimizing user burden and calibration time requirements.

[0151] 3.3) Determine if there are enough calibration samples available for effective model personalization.

[0152] After sample collection, the quantity and quality of the collected samples can be evaluated based on predetermined thresholds that ensure sufficient training data to achieve reliable calibration performance. This step provides a decision point that guides the calibration process down different paths based on the sample sufficiency assessment: if insufficient samples are determined, proceed to step 3.5; if sufficient samples are determined, proceed to step 3.4.

[0153] 3.4) Only update the calibration embedding and final FC layer parameters.

[0154] Modify the weights of the final fully connected layer within the subject-specific calibration embedding and fully connected processing network, while keeping all other network parameters frozen. In some embodiments, this step can avoid retraining the entire network architecture, thereby reducing computational requirements and preventing overfitting on limited calibration data.

[0155] It should be noted that by only updating the calibration embedding and the final FC (Fully Connected Layer) parameters, this selective parameter update method reduces the calibration time to less than 2 minutes.

[0156] 3.5) Request the target entity to provide additional calibration samples.

[0157] Prompt the user to provide more gaze data points to meet the minimum requirements for effective personalization. Once the user provides more gaze data points, return to step 3.2) to collect additional calibration samples, creating a feedback loop to ensure sufficient training data is obtained before model adaptation.

[0158] 3.6) Verify calibration performance on test samples.

[0159] The effectiveness of the personalized model is evaluated by testing gaze estimation accuracy on validation data not used during calibration parameter updates. This step measures the performance improvement achieved through subject-specific adaptation and verifies whether the calibration process successfully enhances gaze estimation accuracy for the target user.

[0160] 3.7) Deploy personalized gaze estimation models for the target subject.

[0161] The calibration system is activated for real-time eye-tracking applications using updated subject-specific calibration embedding parameters. In some embodiments, this step can enable the deployment of a personalized model that provides up to 11.8% error reduction compared to an uncalibrated baseline system.

[0162] The subject-specific calibration method proposed in this embodiment achieves rapid personalization by updating only the 6-dimensional calibration embedding vector and the final layer weights instead of retraining the entire network architecture. It can provide an efficient calibration workflow, minimize user interaction time, and maximize personalization benefits to improve gaze estimation accuracy for different subjects and usage scenarios.

[0163] Example 3:

[0164] This embodiment provides a specific implementation method for obtaining graph-level features based on a graph constructed using a two-layer multi-head attention graph neural network. (Reference) Figure 4 Graph neural network (GNN) processing operations transform multimodal input embeddings through an attention-based inference mechanism. Cross-modal relationships between heterogeneous input modalities are modeled through a two-layer GNN architecture with dimensionality expansion and reduction operations. Specifically, the GNN processing operations include the following steps:

[0165] 4.1) Initialize the six graph nodes using projection feature embedding.

[0166] The system receives a uniform 32-dimensional embedding generated through a previous projection operation and can assign each embedding to a corresponding node within a fully connected graph structure. In some embodiments, this step can establish initial node states that represent facial features, left eye features, right eye features, head rotation data, left eye 3D position, and right eye 3D position within a graph neural network processing framework.

[0167] 4.2) Use linear projection to calculate the attention coefficients for all node pairs.

[0168] Each attention head applies a learned linear transformation to the features of the source and target nodes, and can concatenate the transformed representations to create a combined feature vector for calculating the attention coefficients. The generated attention coefficients quantify the strength of the relationship between each pair of nodes within the fully connected graph structure.

[0169] 4.3) Apply LeakyReLU activation and softmax normalization to the attention coefficient.

[0170] After the attention coefficients are calculated, the raw attention coefficients are processed by the LeakyReLU activation function to introduce non-linearity, and softmax normalization can then be applied to all neighbor connections (adjacent nodes) of each node. In some embodiments, this step can ensure that the sum of the attention weights for each node is one, while maintaining gradient flow through the activation function during training operations.

[0171] 4.4) Use weighted attention coefficients to aggregate adjacent features.

[0172] Based on the normalized attention coefficients generated in the above steps, a weighted sum of features from adjacent nodes is calculated. This step can combine features from all connected nodes according to the attention weighted importance, enabling each node to incorporate information from relevant modalities within the graph structure.

[0173] 4.5) Determine whether the current processing corresponds to the first GNN layer.

[0174] This step provides a decision point, guiding the dimensionality transformation operation to different paths based on the layer position within the two-layer graph neural network architecture. Using this step, the current processing stage can be evaluated to determine the appropriate dimensionality scaling operation for the aggregated node features. If it is determined that the current processing corresponds to the first GNN layer, proceed to step 4.6); if it is determined that the current processing does not correspond to the first GNN layer, proceed to step 4.7.

[0175] 4.6) Extend node features to 256 dimensions.

[0176] A linear transformation is applied to increase the dimensionality of each node from the input 32-dimensional space to an extended 256-dimensional intermediate representation. In some embodiments, this step can achieve the capture of complex relational patterns by increasing representational capacity during the first-layer processing operation.

[0177] 4.7) Reduce node features to 32 dimensions.

[0178] A linear transformation is applied to reduce the dimensional representation from the intermediate 256-dimensional space back to a normalized 32-dimensional output format. This step maintains computational efficiency while preserving the enhanced relational information captured during the first-level expansion operation.

[0179] 4.8) Apply the second-layer GNN processing.

[0180] The extended 256-dimensional node representation is processed through the second layer of the graph neural network architecture. This step can use the extended dimensional representation as input features to repeat attention coefficient computation, normalization, and feature aggregation operations.

[0181] 4.9) Flatten all node features and connect them into a unified representation.

[0182] The six 32-dimensional nodes are embedded and transformed into a single flattened vector through a join operation. In some embodiments, this step can generate a 192-dimensional unified representation that captures processing relation information from all input modalities within a single feature vector.

[0183] 4.10) Output the final graph-level feature vector for fusion operation.

[0184] A unified graph representation is provided to downstream processing modules for integration with raw CNN features, raw geometric features, and subject-specific calibration embeddings. This step enables the multimodal feature integration layer to combine graph-based relational features with other feature representations for comprehensive gaze estimation processing.

[0185] It should be noted that the graph neural network processing method achieves hierarchical feature refinement through a two-layer processing structure. This structure expands representational capabilities in the first layer and integrates information in the second layer. In some embodiments, dimensionality expansion and reduction operations can capture complex cross-modal relationships while maintaining computational efficiency suitable for real-time deployment scenarios in mobile devices and embedded systems.

[0186] Example 4:

[0187] This embodiment provides a specific implementation method for converting gaze estimation prediction into a normalized 3D gaze vector based on coordinate transformation. The coordinate transformation steps include constructing an orthogonal coordinate frame using cross product and generating a parametric ray emanating from the 3D eye center position for intersection calculation with the augmented reality or virtual reality coordinate frame.

[0188] refer to Figure 5 This paper presents a 3D view vector reconstruction process that converts angular view prediction into a spatial orientation vector representation suitable for augmented reality and virtual reality applications. This method enables coordinate transformation and mathematical operations, converting spherical view direction parameters into parametric ray representations with defined origins and direction vectors. Specifically, the 3D view vector reconstruction process includes the following steps:

[0189] 5.1) Receive the predicted line-of-sight pitch and yaw angles from the regression module.

[0190] Angle predictions generated by a fully connected processing network are obtained, representing the line-of-sight direction in spherical coordinates. In some embodiments, this step receives pitch and yaw angle values ​​calculated through the multimodal feature fusion and regression operations described in the preceding processing stages.

[0191] 5.2) Use trigonometric transformations to convert spherical coordinates to Cartesian coordinates.

[0192] Sine and cosine functions are applied to the received pitch and yaw angles to generate three-dimensional coordinate components. In this step, the x, y, and z coordinate values ​​representing the line-of-sight vector in Cartesian space can be calculated, converting the angular representation into a format suitable for three-dimensional spatial calculations.

[0193] 5.3) Normalize the resulting 3D vector to unit length.

[0194] The magnitude of the Cartesian coordinate vector is calculated, and each coordinate component can be divided by the calculated magnitude to ensure unit vector properties. In some embodiments, this step can guarantee that the length of the normalized line-of-sight vector is exactly one unit, providing mathematical stability for subsequent coordinate frame construction operations.

[0195] 5.4) Use cross product to construct an orthogonal normalized coordinate frame.

[0196] Orthogonal basis vectors are generated by performing a cross product operation between the normalized view vector and the reference coordinate axes. This step creates a complete 3D coordinate system that establishes the spatial orientation relationship of the view vector within the target coordinate frame.

[0197] 5.5) Construct rotation matrices based on orthogonal basis vectors.

[0198] The generated orthogonal basis vectors are arranged into a rotation matrix format that enables coordinate system transformation. In some embodiments, this step may construct a 3x3 rotation matrix, which provides a mathematical framework for transforming the line-of-sight vector between different coordinate reference systems.

[0199] 5.6) Determine whether camera coordinate transformation is required.

[0200] The target application requirements are assessed to determine whether the gaze vector should be expressed in the camera coordinate space or preserved in the original coordinate system. This step provides a decision point, guiding the processing flow to different paths based on the coordinate system requirements of the downstream application. If it is determined that a camera coordinate transformation is required, proceed to step 5.7); if it is determined that a camera coordinate transformation is not required, proceed to step 5.8.

[0201] 5.7) Use a rotation matrix to transform the gaze vector into camera coordinates.

[0202] The rotation matrix constructed in step 5.5) transforms the normalized gaze vector from the original coordinate system to a camera-aligned coordinate space. This step enables integration with imaging systems and spatial tracking applications that operate within the camera coordinate frame.

[0203] 5.8) Output the line-of-sight vector in the original coordinate system.

[0204] This provides a normalized line-of-sight vector without coordinate system transformation, preserving the original spatial orientation relationships. In some embodiments, this step may be applicable to applications operating within the same coordinate framework as the initial line-of-sight estimation process.

[0205] 5.9) Generate a parametric ray with the center of the 3D eye as the origin.

[0206] Following the coordinate transformation decision and processing in step 5.7) or 5.8), the processed gaze direction vector is combined with the 3D eye center coordinates to create a complete ray representation. This step can define the starting point and direction vector of the gaze ray, enabling intersection calculations with virtual objects, user interface elements, and physical surfaces in spatial applications.

[0207] It should be noted that the 3D gaze vector reconstruction method achieves accurate 3D gaze vector reconstruction by applying trigonometric transformations, normalization operations, and coordinate system transformations. This method can work collaboratively with a 3D gaze vector reconstruction module, orthogonal basis building units, and camera coordinate transformation units to provide a mathematically stable gaze ray representation suitable for augmented reality, virtual reality, and spatial interaction applications. In some embodiments, this method can generate parametric rays that support gaze plane intersection calculations for user interface control, diagnostic applications, and immersive environment interaction systems.

[0208] Example 5:

[0209] This embodiment provides a specific implementation method for projecting the visual features, geometric orientation information, and spatial positioning coordinates into the same vector space based on a linear transformation with a GELU activation function.

[0210] refer to Figure 6 This embodiment provides a feature embedding component architecture that processes heterogeneous multimodal inputs for gaze estimation applications through four dedicated processing modules. Different input modalities are converted into unified dimensional representations, enabling coherent cross-modal inference and lightweight deployment on mobile devices and embedded systems. Specifically, the feature embedding component includes:

[0211] 6.1) Visual Feature Processing Module: This module processes appearance-based information from facial and eye region inputs. The Visual Feature Processing Module handles visual data through specialized extraction operations that capture unique features from facial images and eye cropping inputs while maintaining computational efficiency suitable for real-time applications.

[0212] The facial feature extractor within the visual feature processing module can process facial region images to extract appearance-based features that capture overall facial structure, orientation, and contextual information. The facial feature extractor can apply depthwise separable convolutions and squeeze activation modules to generate feature representations that capture facial appearance patterns related to gaze orientation estimation. In some embodiments, the facial feature extractor can process facial image input through lightweight convolutional operations, which reduce computational requirements while maintaining the effectiveness of feature extraction.

[0213] The eye feature extractor within the visual feature processing module can process individual eye region images to extract detailed appearance information from each eye. The eye feature extractor can handle left and right eye cropping through specialized processing operations that capture eye-specific features, including pupil position, iris pattern, and eyelid configuration. The eye feature extractor can apply convolutional processing operations similar to those used in facial feature extractors, while focusing on isolating unique visual features present within the eye region.

[0214] The facial feature extractor and eye feature extractor can generate a 576D feature vector that captures visual features from the processed facial and eye region inputs. The 576D feature vector can represent a combination of appearance-based information extracted from the facial image and eye cropping input through convolutional processing operations. In some embodiments, the 576D feature vector can contain unique visual features that provide texture, shape, and appearance information complementary to the geometric and spatial input modalities.

[0215] The 32D projection layer within the visual feature processing module can transform 576D feature vectors into a normalized dimensional space for subsequent processing operations. This 32D projection layer can apply a linear projection layer with GELU activation to map high-dimensional visual features into a unified 32-dimensional embedding space. The 32D projection layer ensures dimensional compatibility between visual features and geometric and calibration data, facilitating subsequent graph processing operations within the graph neural network.

[0216] 6.2) Geometric Feature Processing Module: This module processes head orientation data through system transformation operations. The geometric feature processing module can process three-dimensional rotation matrix information representing the spatial relationship between the user's head position and the camera coordinate system.

[0217] The head rotation processor within the geometric feature processing module can receive head rotation data containing three-dimensional rotation matrix information describing head pose features. The head rotation processor can process the rotation matrix data to extract geometric orientation parameters that provide spatial context for gaze direction estimation. In some embodiments, the head rotation processor can process a rotation matrix describing the angular relationship between the user's head orientation and an imaging device reference frame.

[0218] The 9D matrix flattening component within the geometric feature processing module can convert a 3x3 rotation matrix into a flattened vector representation suitable for subsequent processing operations. This component can convert a 3x3 rotation matrix into a 9D vector format that preserves geometric orientation information while enabling linear processing. Furthermore, the 9D matrix flattening component ensures that head rotation information is represented in a format compatible with dimension normalization operations applied to other input modalities.

[0219] The 32D embedding space component within the geometric feature processing module projects flattened geometric data into a unified dimensional framework used across all input modalities. The 32D embedding space component can apply linear transformation operations to map the 9-dimensional head rotation vector to a normalized 32-dimensional space. In some embodiments, the 32D embedding space component can ensure that geometric orientation data shares the same dimensional features as visual features and spatial positioning information for coherent cross-modal processing within the graph neural network.

[0220] 6.3) Spatial Feature Processing Module: This module manages 3D eye position data through specialized coding operations. The spatial feature processing module can process the spatial coordinates representing the 3D position of each eye center within the camera coordinate system.

[0221] The left-eye center component within the spatial feature processing module can process the 3D coordinate data of the left-eye center position. This component can process spatial positioning information calculated through geometric analysis of facial key points combined with depth estimation techniques. Ultimately, the left-eye center component manages the coordinate data representing the spatial position of the left-eye center within the 3D camera coordinate frame.

[0222] The right eye center component within the spatial feature processing module can process the three-dimensional coordinate data of the right eye center position. The right eye center component can process spatial positioning information complementary to that of the left eye center component to provide comprehensive eye positioning data. In some embodiments, the right eye center component can work in conjunction with the left eye center component to provide 6-dimensional coordinate data representing the positions of the two eye centers within a 3D eye center input modality.

[0223] The 3D position encoding component within the spatial feature processing module can process eye center coordinates from the left and right eye center components. This component applies encoding operations to convert the three-dimensional spatial coordinates into a representation suitable for dimension normalization. Furthermore, the 3D position encoding component ensures that spatial positioning information is correctly formatted for subsequent projection into a unified embedding space.

[0224] The 32D Unified Spatial Component within the Spatial Feature Processing module can convert encoded spatial information into a common embedding dimension used across all input modalities. The 32D Unified Spatial Component can apply linear transformation operations to map 6D eye center coordinate data to a standardized 32D space. In some embodiments, the 32D Unified Spatial Component can ensure dimensionality compatibility between spatial positioning information and visual feature and geometric orientation data for efficient integration within a six-node graph structure.

[0225] 6.4) Calibration Component Module: This module provides personalization capabilities through a specific subject adaptation mechanism. The Calibration Component Module enables personalized gaze estimation model adaptation without requiring full network retraining.

[0226] The subject-specific embedding component within the calibration module can generate personalized parameters for each user, capturing the gaze patterns and characteristics of a specific subject. This component can create learnable embedding vectors representing individual user features without modifying the underlying network architecture. Furthermore, the subject-specific embedding component can achieve rapid personalization through selective parameter updates, which improve the accuracy of gaze estimation for that specific subject.

[0227] The 6D parameter vector component within the calibration component module can represent calibration data in a standardized format suitable for integration with other input modalities. The 6D parameter vector component can contain learnable parameters optimized for the target subject through the calibration process. In some embodiments, the 6D parameter vector component can provide a calibration embedding that achieves subject-specific adaptation while maintaining compatibility with the uniform dimensional framework used throughout the system.

[0228] The personalization module within the calibration component allows for rapid adaptation to specific subjects without requiring full system retraining. This module implements a selective parameter update method that modifies only the subject-specific calibration embedding and final layer weights while keeping all other network parameters frozen. Furthermore, the personalization module provides a calibration workflow that minimizes user interaction time while maximizing personalization benefits to improve gaze estimation accuracy across different subjects and usage scenarios.

[0229] The four processing modules described above can work collaboratively to transform heterogeneous input modalities into unified dimensional representations, which enable efficient cross-modal inference within the graph neural network. The visual feature processing module processes appearance-based information through facial and eye feature extractors, while the geometric feature processing module processes head orientation data through a head rotation processor and dimensional transformation operations. The spatial feature processing module manages eye positioning coordinates through left and right eye center components, while the calibration component module provides personalization capabilities through subject-specific embedding components and personalization module components. In some embodiments, system dimensionality normalization implemented across all four modules ensures that diverse input modalities can be effectively integrated within a six-node graph structure for comprehensive cross-modal relationship modeling and accurate gaze direction prediction.

[0230] Example 6:

[0231] Secondly, the present invention also provides a multimodal gaze estimation system based on a lightweight graph neural network, the system comprising:

[0232] The multimodal input module is configured to acquire multimodal heterogeneous data that is related to and complementary to the gaze direction estimation, the multimodal heterogeneous data including visual appearance data, geometric orientation information and spatial positioning coordinates;

[0233] The feature extraction module is configured to extract visual features from visual appearance data based on depthwise separable convolution and squeeze excitation module.

[0234] The projection module is configured to project the visual features, geometric orientation information, and spatial positioning coordinates into the same vector space;

[0235] The graph construction module is configured to construct a graph structure, which contains several nodes representing multimodal heterogeneous data or features corresponding to multimodal heterogeneous data.

[0236] Multi-head attention GNN is configured to obtain graph-level features based on the graph constructed by the multi-head attention graph neural network with a two-layer structure. In the two-layer structure, the first layer expands the dimension of the node features, and the second layer reduces the dimension of the expanded features.

[0237] The feature fusion module is configured to aggregate the graph-level features and fuse them with the visual features, geometric orientation information, and spatial positioning coordinates.

[0238] The regression module is configured to utilize fully connected layers and perform gaze estimation prediction based on fused features;

[0239] The 3D reconstruction module is configured to convert gaze estimation predictions into normalized 3D gaze vectors based on coordinate transformation.

[0240] refer to Figure 7 The system can provide a multimodal gaze estimation architecture based on a lightweight graph neural network, which processes heterogeneous inputs to predict gaze direction. The system includes:

[0241] The multimodal input module receives various types of input data, including facial images, cropped eye images, head rotation data, and 3D eye center positions. The multimodal input module captures image data via the device's camera and processes the captured data through a normalized camera module that performs data normalization.

[0242] A feature extraction module receives normalized facial and eye image data from a multimodal input module. The feature extraction module can generate facial and eye features from the processed image input using depthwise separable convolution and a squeeze excitation module. In some embodiments, the feature extraction module can process 3D eye center data obtained from the captured image and head rotation matrix information representing head pose data.

[0243] The projection module can receive input from multiple sources within the system, including facial and eye features from the feature extraction module. It can convert heterogeneous input modalities into a unified representation through embedding operations. The projection module can generate calibrated embeddings represented as vectors and can create facial feature embeddings, 3D eye center embeddings, head rotation embeddings, and eye feature embeddings, mapping different input types to a common dimensional space.

[0244] The graph construction module receives the embedded output from the projection module. This module can construct a fully connected graph structure with six interconnected nodes to represent different input modalities. In some embodiments, the six nodes can correspond to right eye, left eye, face, right eye center, left eye center, and head rotation data. The graph construction module can establish directed edges between the nodes to indicate relationships between modalities.

[0245] Multi-head attention GNNs can handle graph structures created by graph building modules. A multi-head attention GNN can include a multi-head attention processing mechanism followed by attention aggregation operations. Multi-head attention GNNs can model cross-modal relationships and dependencies through attention-based inference that captures spatial and semantic interactions between facial components, head orientation, and eye positions.

[0246] The feature fusion module combines the outputs from multiple processing paths. It receives calibration embeddings, 3D eye center data, and context-aware encodings from a multi-head attention GNN from the projection module. The feature fusion module integrates diverse inputs into a unified representation that combines relational information from graph processing with the original feature representation.

[0247] The regression module can process the fused features from the feature fusion module through a fully connected output layer. The regression module can produce a normalized line-of-sight prediction expressed in yaw and pitch angles. In some embodiments, the regression module can generate an angle prediction that represents the line-of-sight direction in a format suitable for subsequent 3D transformation.

[0248] The 3D reconstruction module receives angular predictions from the regression module. It uses a spherical-to-Cartesian transformation to convert normalized view direction angles into a 3D view ray representation. The 3D reconstruction module can generate view vectors suitable for spatial applications and coordinate system integration.

[0249] The system boasts a total parameter count of approximately 3.48 million, making it 50-70% smaller than competing transformers and CNN models. In some embodiments, the reduced parameter count enables real-time performance on mobile devices, embedded systems, and low-power AR / VR platforms while maintaining gaze estimation accuracy. The system can work with RGB webcams, mobile front-facing cameras, and world cameras on head-mounted displays without requiring dedicated infrared hardware, enabling deployment in a wide range of consumer devices and unconstrained environments.

[0250] refer to Figure 8 The camera device can initiate the gaze estimation process by capturing image data from the user. The camera device can then transmit the captured image data to a normalized camera interface, which standardizes the input data for subsequent processing. The normalized camera interface performs normalization operations to ensure consistent data format across different camera types and imaging conditions.

[0251] The normalized camera interface can forward the processed image data to the data processing module, which performs comprehensive preprocessing operations. The data processing module can perform facial landmark detection, eye position recognition, resizing, cropping, and normalization operations on the received image data. In some embodiments, the data processing module can generate multiple output streams that branch to parallel processing paths for different input modalities.

[0252] The data processing module can generate head rotation vectors representing 3D head orientation information. It can also generate facial images capturing overall facial appearance features and create cropped left and right eye images for detailed analysis, isolating the eye region. Furthermore, the module can provide 3D position data for the left and right eyes, representing the spatial coordinates of each eye center in 3D space.

[0253] The feature extractor receives facial images, left-eye cropping, and right-eye cropping from the data processing module. It processes the visual input using depthwise separable convolution combined with a squeeze activation module, extracting 576-dimensional features from the facial and eye croppings. The feature extractor can apply global average pooling to generate facial and eye features that capture unique visual characteristics from the input image.

[0254] The feature extractor can project 576-dimensional visual features into a unified embedding space using a linear transformation with GELU activation. In some embodiments, the feature extractor can convert heterogeneous visual data into a standardized dimensional representation, enabling coherent cross-modal processing in subsequent stages.

[0255] The multistream linear module can receive output from the data processing module and the feature extractor. It can process head rotation vectors, facial features, eye features, left eye 3D position, and right eye 3D position data to produce a unified embedding. The multistream linear module can generate head rotation embeddings, facial feature embeddings, eye feature embeddings, and 3D eye center embeddings, which represent various input modalities in a common dimensional space.

[0256] Multistream linear modules can enable coherent cross-modal reasoning by mapping heterogeneous input types to standardized embedding representations. In some embodiments, multistream linear modules can facilitate subsequent graph-based processing for attention-based relation modeling by ensuring that all input modalities share compatible dimensional features.

[0257] refer to Figure 9 The multimodal input module can provide node embeddings, including face, left and right eye, head rotation, and eye position data, to initiate graph neural network processing. The multimodal input module can also provide heterogeneous embedding representations transformed into a unified dimensional space through previous feature extraction and projection operations.

[0258] The graph building module can receive node embeddings from the multimodal input module and can create a fully connected network structure with six nodes. The graph building module can establish a static topology where all nodes are connected to all other nodes via learnable edge weights. In some embodiments, the graph building module can model comprehensive relationships between different input modalities by ensuring that each node can directly influence every other node in the network.

[0259] The graph construction module can forward the constructed graph structure to a multi-head attention mechanism for attention-based processing. The multi-head attention mechanism can perform linear transformations on source and target features and compute attention coefficients using learned linear projections. The multi-head attention mechanism can then apply concatenations of the transformed features, followed by LeakyReLU activation, to generate attention coefficients with learnable parameters.

[0260] Multi-head attention mechanisms can use parallel attention heads to simultaneously capture multiple relational patterns between graph nodes. In some embodiments, multi-head attention mechanisms can enable the system to model different types of relationships and dependencies in the same processing step, allowing for comprehensive cross-modal interaction analysis.

[0261] The attention normalization module can receive computed attention coefficients from a multi-head attention mechanism and apply coefficient normalization on a per-node basis. It can perform exponential scaling of the scores and softmax normalization among neighbors to ensure proper weighting of node influence. The module generates normalized attention coefficients representing the relative importance of each node's connections.

[0262] The feature aggregation module can process the normalized attention coefficients from the attention normalization module to perform node feature processing. The feature aggregation module can compute a weighted sum of neighbor features and can apply nonlinear transformations while preserving the original node dimensions. In some embodiments, the feature aggregation module can preserve the spatial and semantic relationships between input modalities through attention-weighted feature combinations.

[0263] The feature aggregation module can provide aggregated node features to the multi-head processing module to generate enhanced representations.

[0264] The multi-head processing module can operate on multiple parallel attention heads with independent parameter sets, and can aggregate the results by concatenation to form rich node representations. The multi-head processing module can simultaneously process different aspects of relationships within a graph structure.

[0265] The two-layer inference module can receive rich node representations from the multi-head processing module for hierarchical processing operations. In the first layer, the two-layer inference module can convert node features into intermediate representations and expand them to 256 dimensions during this initial processing stage. The two-layer inference module can then convert the intermediate representations into output embeddings in the second layer and can reduce the expanded features back to 32 dimensions to maintain computational efficiency.

[0266] The two-layer inference module can apply a full attention mechanism across two processing layers to achieve hierarchical feature refinement across graph structures. In some embodiments, dimensionality expansion and reduction operations can allow the two-layer inference module to capture... It acquires complex relationship patterns while maintaining manageable computing requirements for mobile deployment scenarios.

[0267] The graph generation module receives processed embeddings from the two-layer inference module for final feature vector creation. It flattens and concatenates all six updated node embeddings into a unified graph representation vector. The module produces a unified 384-dimensional relation vector that captures cross-modal dependencies and relationships across all input modalities.

[0268] The graph generation module can output the final graph-level feature vector for subsequent fusion with other feature representations in downstream processing modules. In some embodiments, the unified relation vector can provide comprehensive cross-modal relation information, enhancing gaze estimation accuracy compared to the independent processing methods used in traditional systems.

[0269] refer to Figure 10 The multimodal feature ensemble layer can receive multiple input streams to create a composite representation for gaze estimation processing. It can combine diverse input modalities from four different sources, which provide different types of feature information for comprehensive gaze direction prediction.

[0270] Graph neural network outputs can provide relational features to multimodal feature integration layers. The outputs can contain graph-based relational information that captures cross-modal dependencies and spatial relationships between facial components, eye positions, and head orientation data processed through attention mechanisms.

[0271] The raw CNN features can provide appearance-based features from the face and eye processing branches to the multimodal feature integration layer. The raw CNN features can include visual features extracted from face regions and eye cropping, which provide texture, shape, and appearance information that is complementary to the relational features of graph processing.

[0272] The raw geometric features can transfer rotation and eye position data to the multimodal feature integration layer. These raw geometric features can include head rotation matrix information and 3D eye center coordinates, providing spatial orientation and localization context for gaze direction estimation.

[0273] Subject-specific calibration embeddings can provide personalized calibration parameters to multimodal feature integration layers. Subject-specific calibration embeddings can contain a 6-dimensional learnable embedding vector that enables subject-specific adaptation and personalization without requiring full network retraining.

[0274] A multimodal feature integration layer can combine graph neural network outputs, raw CNN features, raw geometric features, and subject-specific calibration embeddings into a unified feature representation. In some embodiments, the multimodal feature integration layer can ensure that heterogeneous input modalities are appropriately weighted and integrated to maximize gaze estimation accuracy under different subject and environmental conditions.

[0275] Fully connected processing networks can receive ensemble features from multimodal feature ensemble layers for angle prediction. These layers perform nonlinear transformations on the composite feature representations, mapping the fused features to line-of-sight (LOS) parameters. The networks can then generate predicted pitch and yaw angles, represented in spherical coordinates to indicate the LOS direction.

[0276] A fully connected processing network can output the angle prediction to a 3D line-of-sight vector reconstruction module for 3D vector transformation. The 3D line-of-sight vector reconstruction module can use a spherical-to-Cartesian transformation operation to convert the predicted pitch and yaw angles into normalized 3D vectors. In some embodiments, the 3D line-of-sight vector reconstruction module can apply trigonometric functions to convert the angle representation into Cartesian coordinate vectors suitable for space applications.

[0277] The 3D Reconstruction module can work in conjunction with the 3D View Vector Reconstruction module to perform comprehensive 3D view vector processing. The 3D Reconstruction module can use a spherical-to-Cartesian transformation to convert pitch and yaw angles into normalized 3D vectors, ensuring mathematical stability and unit vector properties.

[0278] Orthogonal basis building elements can receive normalized view vectors from the 3D view vector reconstruction module to establish a coordinate frame. Orthogonal basis building elements can construct orthogonal coordinate frames using cross product and vector normalization operations. They can perform cross product calculations to derive orthogonal axes and can perform orthogonality normalization and verification to establish a stable coordinate reference system.

[0279] The camera coordinate transformation unit can process orthogonal basis vectors from the orthogonal basis building unit for final coordinate system alignment. The camera coordinate transformation unit can transform the normalized line-of-sight vector into camera coordinates using a rotation matrix constructed from the orthogonal basis vectors. The camera coordinate transformation unit can construct rotation matrices from basis vectors and can invert these matrices to produce camera-aligned output suitable for integration with imaging systems and spatial tracking applications.

[0280] The 3D gaze vector reconstruction module can output parametric rays emanating from the center of the 3D eye, enabling intersection calculations with screens, surfaces, and AR / VR coordinate frames. In some embodiments, the parametric ray representation can provide origin and direction information, allowing downstream applications to calculate the intersection points of the gaze with virtual objects, user interface elements, and physical surfaces in augmented and virtual reality environments.

[0281] refer to Figure 11The system can coordinate a series of integrated processing operations, transforming multimodal input data into accurate 3D gaze vector representations through module interaction. The system can orchestrate data flows between specialized processing components that sequentially handle feature extraction, dimensionality normalization, graph-based relational modeling, and vector reconstruction operations.

[0282] The multimodal input module can initiate a processing sequence by providing multimodal inputs (including facial images, eye cropping, head rotation data, and 3D eye center positions) to the system. In some embodiments, heterogeneous input data capturing visual appearance features, geometric orientation information, and spatial positioning coordinates forms the basis for subsequent processing operations. In some embodiments, it ensures that all necessary input modalities are available for comprehensive gaze estimation processing.

[0283] The system forwards the received image input to the feature extraction module for visual feature processing.

[0284] Facial images and eye cropping data are guided to specialized processing operations that extract unique visual features from the input images. The feature extraction module processes the image input using depthwise separable convolution and a squeeze excitation module, and can return the processed features to the system.

[0285] The extracted visual features capture appearance-based information from facial regions and eye cropping. In some embodiments, when the processed features are returned to the system, 576-dimensional feature vectors can be generated, representing unique visual features suitable for subsequent dimensionality transformation operations. The feature extraction module can perform visual processing operations and can transmit the extracted features back to the system for integration with other processing paths.

[0286] The system then forwards all heterogeneous inputs to the projection module for dimensional standardization.

[0287] Visual features, geometric data, and calibration parameters are guided to unified embedding operations, which transform different input modalities into a common dimensional space. The projection module can project all inputs into a common 32-dimensional embedding space and then return the unified embedding to the system.

[0288] When performing this step, the projection module can provide standardized dimensional representations that enable coherent cross-modal processing in subsequent graph-based operations. The projection module can apply a linear transformation with a GELU activation function to map heterogeneous input types to a unified 32-dimensional space. In some embodiments, visual features, head rotation matrices, and 3D eye center coordinates can be represented in a compatible dimensional format for graph neural network processing.

[0289] The system sends the projection embedding to the graph building module to create the graph structure.

[0290] The unified embedding guides graph building operations, which establish relationships between different input modalities. The graph building module can construct a fully connected graph with six nodes, representing the input modalities, and can return the graph structure to the system.

[0291] When performing related operations, the graph construction module can provide a fully connected graph structure, where each node corresponds to an input modality, including facial features, left eye features, right eye features, head rotation data, left eye 3D position, and right eye 3D position. In some embodiments, learnable edge weights can be established between all node pairs to achieve comprehensive relationship modeling between different input modalities within the graph neural network framework.

[0292] The system forwards the constructed graph to a multi-head attention GNN for relation processing.

[0293] This approach guides graph structures towards attention-based processing operations that model cross-modal relationships and dependencies through parallel attention mechanisms. Multi-head attention GNNs can process graphs through a two-layer structure with attention mechanisms and return graph-level features to the system.

[0294] Multi-head attention GNNs, when performing related operations, can provide processed relational features that capture cross-modal dependencies and spatial relationships between different input modalities. Multi-head attention GNNs can apply attention coefficient computation, normalization, and feature aggregation operations in two processing layers. In some embodiments, this process can generate a unified 384-dimensional relation vector representing the comprehensive relational information captured through graph-based attention processing.

[0295] The system sends graph features, CNN features, and geometric inputs to the feature fusion module for integration.

[0296] This module guides multiple feature representations to fusion operations that combine relational information from graph processing with appearance-based feature and geometric data. The feature fusion module can aggregate and fuse all feature representations and return a unified representation to the system.

[0297] When performing related operations, the feature fusion module can provide an integrated feature representation that combines the graph neural network output, raw CNN features, raw geometric features, and subject-specific calibration embeddings into a comprehensive multimodal representation. In some embodiments, different input modalities can be appropriately weighted and integrated to maximize gaze estimation accuracy under different subject and environmental conditions.

[0298] The system forwards the fused features to the regression module for angle prediction.

[0299] The integrated feature representations are guided to regression operations, which fuse the features and map them to line-of-sight (LOS) parameters. The regression module can use fully connected layers to predict LOS pitch and yaw angles and can return these angle predictions to the system.

[0300] When performing related operations, the regression module can provide predicted pitch and yaw angles, which represent the line-of-sight direction in spherical coordinates, suitable for subsequent 3D transformation operations. The regression module can generate angle predictions from fused multimodal features by applying nonlinear transformations through a fully connected processing network. In some embodiments, line-of-sight direction parameters can be generated, which serve as inputs to the 3D vector reconstruction process.

[0301] The system sends the predicted angle to the 3D reconstruction module for vector conversion.

[0302] The angle prediction is guided to 3D reconstruction operations, which convert spherical coordinates into Cartesian vector format. The 3D reconstruction module can use the spherical-to-Cartesian transformation to convert angles into normalized 3D gaze vectors and can return the final gaze vectors to the system.

[0303] When performing related operations, the 3D reconstruction module can provide normalized 3D gaze vectors, which represent the gaze direction in Cartesian coordinate space, suitable for spatial applications and coordinate system integration. The 3D reconstruction module can work in conjunction with the 3D gaze vector reconstruction module, the orthogonal basis building unit, and the camera coordinate transformation unit to generate parametric rays with defined origins and direction vectors. In some embodiments, gaze ray representations are generated that can perform intersection calculations with virtual objects, user interface elements, and physical surfaces in augmented reality and virtual reality environments.

[0304] refer to Figure 12 The camera device captures image data from the user and transmits the captured image data to the normalized camera interface to initiate the multimodal gaze estimation data processing workflow.

[0305] The initial data acquisition phase of the processing workflow is established by providing raw image input containing information on facial features, eye areas, and head orientation. The camera device can work with a variety of imaging hardware, including RGB webcams, mobile front-facing cameras, and world cameras on head-mounted devices, without requiring a dedicated infrared illumination system.

[0306] A normalized camera interface can receive captured image data from a camera device and perform standardization operations to ensure a consistent data format across different camera types and imaging conditions. A normalization process is applied that takes into account variations in camera resolution, color space representation, and geometric distortion characteristics. In some embodiments, image data can be prepared for subsequent processing by establishing uniform data attributes that enable reliable feature extraction and analysis across different imaging hardware configurations.

[0307] The normalized camera interface can also forward standardized image data to the data processing module for comprehensive preprocessing operations. Normalized image input is guided to specialized processing operations that extract various types of information from the captured image. The data processing module can perform facial landmark detection, eye position recognition, resizing, cropping, and normalization operations on the received image data to generate multiple output streams that provide different aspects of the visual and geometric information contained in the original image.

[0308] The data processing module can generate a head rotation vector representing 3D head orientation information and output this vector to the multistream linear module. During this process, geometric orientation data can also be provided, capturing the spatial relationship between the user's head position and the camera coordinate system. The head rotation vector can be represented as a flattened 9-dimensional vector derived from a 3x3 rotation matrix, which describes the head pose relative to the imaging device's reference frame.

[0309] After the head rotation vector is generated, the data processing module can produce a facial image that captures the overall facial appearance features and can transmit the facial image to the feature extractor.

[0310] The system provides cropped facial region data that isolates facial regions from the original captured image while preserving sufficient contextual information for appearance-based feature extraction. In some embodiments, the cropped facial image can be generated with normalized dimensions adjusted to suit subsequent convolutional neural network processing operations.

[0311] The data processing module can create cropped left and right eye images for detailed analysis by isolating individual eye regions, and can send the cropped eye images to the feature extractor.

[0312] It provides specialized eye region data, which enables focused feature extraction by estimating the most informative areas from the gaze direction. Left and right eye cropped images are generated by precisely locating eye keypoints, followed by cropping to extract rectangular regions centered on each eye while maintaining consistent spatial relationships and scale factors.

[0313] The data processing module can also provide the multistream module with the spatial coordinate data of the left and right eye 3D positions.

[0314] It provides three-dimensional eye center coordinates, which represent the spatial position of each eye within the camera coordinate system. 3D eye position data can be calculated through geometric analysis of facial key points combined with depth estimation techniques, which provide spatial localization information complementary to appearance-based features extracted from cropped eye images.

[0315] The feature extractor can receive facial images, left eye cropping, and right eye cropping from the data processing module, and can process visual input using depthwise convolution, squeeze activation module, and global average pooling operations.

[0316] Lightweight convolutional operations are applied to extract unique visual features while maintaining computational efficiency suitable for mobile deployment scenarios. Depthwise separable convolutions reduce the number of parameters and computational requirements compared to standard convolutional operations, while the squeeze excitation module enhances the quality of feature representations through channel attention mechanisms.

[0317] The feature extractor can generate facial and eye features that capture unique visual features from the processed image and can transfer the extracted features to the multistream linear module.

[0318] A 576-dimensional feature vector is provided, representing appearance-based information extracted from facial and eye region inputs. In some embodiments, feature representations are provided that capture texture, shape, and appearance features complementary to the geometric information provided by head rotation vectors and 3D eye position data.

[0319] The multistream linear module can receive all outputs from the data processing workflow, including head rotation vectors, facial features, eye features, and 3D eye positions, for unified processing.

[0320] The different input modalities generated through the aforementioned processing operations are integrated into a single processing module, which handles dimensionality normalization and embedding generation. The multistream linear module can process heterogeneous input types through parallel processing paths, including geometric orientation data, appearance-based visual features, and spatial positioning coordinates, while preserving the unique characteristics of each input modality.

[0321] The multistream linear module generates head rotation embedding, facial feature embedding, eye feature embedding, and 3D eye center embedding in a unified dimensional space.

[0322] Standardized 32-dimensional embedding representations are generated for all input modalities through linear transformation operations, which map heterogeneous input data into a common dimensional framework. A separate linear projection layer with a GELU activation function is applied to transform the 9-dimensional head rotation vector, 576-dimensional visual features, and 6-dimensional eye position coordinates into a unified 32-dimensional embedding space, which enables coherent cross-modal inference in subsequent graph neural network processing operations.

[0323] refer to Figure 13 The multimodal input module provides node embeddings (including face, left and right eye, head rotation, and eye position data) to the graph construction module to initiate the graph neural network processing workflow.

[0324] It provides heterogeneous embedding representations transformed into a unified 32-dimensional space through previous feature extraction and projection operations. The multimodal input module ensures that all six input modalities are properly formatted and dimensionally normalized before the graph construction operation begins.

[0325] The graph building module can receive node embeddings from the multimodal input module and create a fully connected network of six nodes with a static topology, where all nodes are connected to all other nodes via learnable edge weights.

[0326] Synthetic connections are established across all input modalities, enabling each node to directly influence every other node within the graph structure. The graph building module initializes learnable parameters for each edge connection, which are optimized during training to capture meaningful relationships between different input modalities.

[0327] After graph construction, the graph construction module can send the constructed graph structure to the multi-head attention mechanism for processing. Step S906 can guide the fully connected graph to attention-based processing operations, which will compute the relationship strength between all node pairs. The multi-head attention mechanism can receive a graph structure with initialized node embeddings and edge weights for subsequent attention coefficient calculation.

[0328] Multi-head attention mechanisms can perform linear transformations of source and target features and can use parallel attention heads to compute attention coefficients with learnable parameters.

[0329] Learned linear projections are applied to the features of the source and target nodes, and the transformed representations can be concatenated to create a combined feature vector for attention coefficient computation. In some embodiments, multiple parallel attention heads are operated simultaneously to capture different types of relationship patterns between graph nodes.

[0330] The multi-head attention mechanism can forward the calculated attention coefficients to the attention normalization module for coefficient normalization.

[0331] The raw attention coefficients are guided to normalization operations that ensure proper weighting of the influence of nodes within the graph structure. The attention normalization module can receive attention coefficients from all parallel attention heads for subsequent exponential scaling and softmax normalization operations.

[0332] The attention normalization module applies exponential scaling to the scores and performs softmax normalization on a per-node basis with respect to neighboring nodes. It processes the raw attention coefficients using an exponential function, followed by a softmax operation, normalizing the attention weights to a sum of one for each node. This attention normalization module ensures that the attention distribution across each node's neighbors forms an effective probability distribution, while maintaining gradient flow for training operations.

[0333] After normalization, the attention normalization module can send the normalized attention coefficients to the feature aggregation module for node feature processing. Appropriately weighted attention coefficients are provided, quantifying the relative importance of each node connection within the graph structure.

[0334] The feature aggregation module can receive normalized attention weights and original node features for weighted feature combination operations.

[0335] The feature aggregation module can compute a weighted sum of neighbor features and apply nonlinear transformations while preserving the original node dimensions. Features from all connected nodes are combined based on attention-weighted importance, enabling each node to incorporate information from relevant modalities within the graph structure. In some embodiments, a 32-dimensional representation of each node is maintained, while feature content is enriched through attention-weighted aggregation of neighbor information.

[0336] The feature aggregation module can provide aggregated node features to the multi-head processing module to generate enhanced representations.

[0337] It provides attention-aggregated node features containing rich relational information from the first stage of graph processing. The multi-head processing module can receive updated node representations for parallel multi-head attention processing, which captures additional relational aspects within the graph structure.

[0338] The multi-head processing module can operate on multiple parallel attention heads with independent parameter sets and aggregate the results through concatenation. By processing aggregated node features using individual attention mechanisms operating with different learning parameters, it is possible to capture different relational patterns simultaneously. The multi-head processing module can concatenate the outputs from all attention heads to form a rich node representation that includes multiple perspectives across modal relationships.

[0339] The multi-head processing module can send rich node representations to the two-layer inference module for hierarchical processing. It provides enhanced node features processed through multiple attention mechanisms to capture comprehensive relational information. The two-layer inference module can receive rich representations for system dimensional transformation via the hierarchical two-layer architecture.

[0340] The two-layer inference module transforms node features into intermediate representations in the first layer and into output embeddings in the second layer. This operation enables dimensionality expansion and reduction operations, allowing for hierarchical feature refinement within the graph structure. In the first layer, the two-layer inference module can expand node features to 256 dimensions to capture complex relationship patterns, while in the second layer, it can reduce the expanded features back to 32 dimensions to maintain computational efficiency for subsequent processing operations.

[0341] After hierarchical processing, the two-layer inference module can forward the processed embeddings to the graph-level generation module for final feature vector creation. This provides fine-grained node embeddings through the two-layer processing of the hierarchical inference architecture. The graph-level generation module can receive the final processed node representations for integration into a unified graph-level feature vector.

[0342] The graph generation module can embed, flatten, and connect all six updated nodes into a unified graph representation vector.

[0343] Individual 32-dimensional node embeddings are transformed into single flattened vectors through concatenation operations, which preserve processing relational information from all input modalities. The graph-level generation module generates a unified representation that captures the comprehensive relational information processed by the graph neural network architecture, providing an integrated feature vector suitable for integration with other feature representations in downstream processing modules.

[0344] Figure 13 The workflow illustrated enables a systematic transformation of heterogeneous input embeddings through specialized graph processing operations that model cross-modal relationships and dependencies. The sequential processing steps described above ensure that information about the relationships between different input modalities is captured and refined through attention mechanisms, dimensionality transformations, and hierarchical processing operations, which enhance the representational quality of the final graph-level feature vectors.

[0345] refer to Figure 14 The multimodal feature ensemble layer coordinates a series of integrated processing operations that systematically integrate heterogeneous input modalities and convert the integrated features into an accurate 3D gaze vector representation. The multimodal feature ensemble layer can receive multiple input streams from specialized processing components that provide different types of feature information for integrated gaze direction prediction and spatial vector reconstruction.

[0346] The output of a graph neural network can provide relational features to a multimodal feature integration layer.

[0347] It provides graph-based relational information that captures cross-modal dependencies and spatial relationships between facial components, eye positions, and head orientation data, which have been processed through an attention mechanism within a graph neural network architecture. The graph neural network output can contain a unified 384-dimensional relation vector, representing the comprehensive relational information captured through two layers of graph processing operations.

[0348] The original CNN features can transfer appearance-based features from the face and eye branches to the multimodal feature integration layer. Step S922 can provide visual features extracted from the face region and eye cropping, which provide texture, shape, and appearance information complementary to the relational features of graph processing. In some embodiments, step S922 can provide 576-dimensional feature vectors that capture unique visual features from the face image, left-eye cropping, and right-eye cropping inputs processed by the depthwise separable convolution and squeeze activation module.

[0349] The original geometric features can send rotation and eye position data to the multimodal feature integration layer.

[0350] It provides head rotation matrix information and 3D eye center coordinates, which provide spatial orientation and localization context for gaze direction estimation. The raw geometric features may include a 9D flattened head rotation vector and 6D eye center position data, which represent the geometric relationship between the user's head pose and eye position within the camera coordinate system.

[0351] Subject-specific calibration embedding can provide personalized calibration parameters to the multimodal feature integration layer.

[0352] It provides a 6-dimensional learnable embedding vector that enables subject-specific adaptation and personalization without requiring full network retraining. In some embodiments, personalized parameters are provided that are optimized for the target subject through a calibration process.

[0353] Multimodal feature ensemble layers can combine diverse input modalities from graph neural network outputs, raw CNN features, raw geometric features, and subject-specific calibration embeddings into a composite representation. By integrating relational features, appearance-based features, geometric data, and personalized parameters through fusion operations, these operations ensure the appropriate weighting and combination of heterogeneous input modalities. Multimodal feature ensemble layers can create a unified feature representation that maximizes gaze estimation accuracy under different subject and environmental conditions.

[0354] Following feature integration, the multimodal feature integration layer can forward the composite representation to the fully connected processing network in step S930. Step S930 can then guide the integrated features to regression processing operations, which convert the fused multimodal representation into angular gaze direction parameters. The fully connected processing network can receive the synthesized feature representation, which combines all input modalities for subsequent nonlinear transformation operations.

[0355] The fully connected processing network can perform nonlinear transformations on the composite feature representation and can generate predicted pitch and yaw angles in step S932. Step S932 can apply a fully connected layer with a nonlinear activation function to map the integrated features to gaze direction parameters in spherical coordinate format. The fully connected processing network can process the fused multimodal features through learned transformations, which produce angular predictions representing the user's gaze direction relative to the camera coordinate system.

[0356] The fully connected processing network transmits the angle prediction to the 3D gaze vector reconstruction module.

[0357] The system provides predicted pitch and yaw angles, representing the line-of-sight direction in spherical coordinates, suitable for subsequent 3D vector transformation operations. In some embodiments, angle parameters are provided as input to the system coordinate transformation process, which converts the spherical representation to a Cartesian vector format.

[0358] The 3D view vector reconstruction module can convert spherical coordinates to Cartesian coordinates and normalize the vectors.

[0359] Using sine and cosine functions, trigonometric transformations are applied to generate three-dimensional coordinate components from the received pitch and yaw angles. The 3D line-of-sight vector reconstruction module can calculate the x, y, and z coordinate values ​​and normalize the resulting vector to unit length to ensure the mathematical stability of subsequent coordinate frame construction operations.

[0360] The 3D gaze vector reconstruction module can send normalized gaze vectors to orthogonal basis building units.

[0361] Provides a unit-length line-of-sight direction vector, converted from spherical coordinates to Cartesian coordinates. Orthogonal basis building blocks can receive normalized 3D vectors for subsequent orthogonal coordinate frame establishment operations.

[0362] Orthogonal basis building blocks can perform cross product operations and establish orthogonal coordinate frames.

[0363] Orthogonal basis vectors are generated by calculating the cross product between the normalized view vector and the reference coordinate axes. Orthogonal basis building blocks can create a complete 3D coordinate system that establishes the spatial orientation relationship of the view vectors within the target coordinate frame through orthogonality normalization and verification operations.

[0364] The orthogonal basis construction unit can transfer the orthogonal basis to the camera coordinate transformation unit in step S942. Orthogonal basis vectors defining a stable coordinate reference system for the direction of the gaze vector space can be provided. In some embodiments, a mathematical framework is provided that implements the coordinate system transformation between the original gaze vector space and the camera-aligned coordinate representation.

[0365] The camera coordinate transformation unit can construct a rotation matrix and transform vectors into camera coordinates.

[0366] The orthogonal basis vectors received from the orthogonal basis building unit construct a 3x3 rotation matrix, which can be applied to transform the normalized line-of-sight vector into a camera-aligned coordinate space. The camera coordinate transformation unit enables integration with imaging systems and spatial tracking applications operating within the camera coordinate framework.

[0367] The camera coordinate transformation unit can output the final 3D line-of-sight ray for space applications.

[0368] A parametric ray representation is provided, which combines the processed gaze direction vector with the three-dimensional eye center coordinates to create a complete ray definition.

[0369] The origin and direction vectors of the gaze rays are generated, enabling intersection calculations with virtual objects, user interface elements, and physical surfaces in augmented reality and virtual reality environments. In some embodiments, mathematically stable gaze ray representations are generated, which support gaze plane intersection calculations in user interface controls, diagnostic applications, and immersive environment interaction systems.

[0370] refer to Figure 15 The LightGazeNet deployment system offers multi-platform implementation capabilities, enabling gaze estimation based on lightweight graph neural networks across diverse hardware configurations and deployment scenarios. The LightGazeNet deployment system demonstrates the versatility and adaptability of the gaze estimation architecture through three different platform configurations, accommodating varying computing resources, imaging hardware, and application requirements, while maintaining consistent gaze estimation functionality across all deployment environments.

[0371] The LightGazeNet deployment system can include a mobile device platform that provides gaze estimation capabilities for smartphone and tablet applications. The mobile device platform can achieve real-time gaze tracking on consumer mobile devices through optimized processing operations that leverage the lightweight architecture of graph neural networks and efficient feature extraction operations implemented via depthwise separable convolutions and squeeze activation modules.

[0372] Smartphone cameras within mobile device platforms can capture visual input data for gaze estimation processing using standard front-facing camera hardware common in consumer mobile devices. Smartphone cameras can provide RGB image data without requiring a dedicated infrared illumination system or additional hardware components. In some embodiments, smartphone cameras can work with a variety of mobile camera specifications and resolutions while maintaining compatibility with normalized camera interface processing operations that standardize the input data for subsequent feature extraction and analysis.

[0373] Smartphone cameras can be connected to embedded processors, which handle the computational requirements of gaze estimation algorithms within the mobile device environment. Embedded processors can leverage the processing power of mobile devices to perform multimodal input processing, unified feature encoding, graph neural networks, and output generation operations. Embedded processors can achieve computational efficiency suitable for mobile deployment scenarios by utilizing a reduced parameter count of approximately 3.48 million parameters.

[0374] An embedded processor can be connected to a real-time engine that manages the real-time processing and output generation of a mobile application. The real-time engine can coordinate sequential processing operations and ensure that gaze estimation results are generated at a frame rate suitable for the interactive application. In some embodiments, the real-time engine can achieve real-time performance exceeding 30 frames per second on mobile devices through lightweight processing operations implemented across the feature extraction, graph neural network, and vector reconstruction stages.

[0375] The LightGazeNet deployment system can further include an AR / VR platform that provides specialized capabilities for augmented reality and virtual reality applications. The AR / VR platform can contain components designed for immersive environments that require accurate 3D gaze vector reconstruction and spatial tracking integration.

[0376] Head-mounted displays within AR / VR platforms can provide visual output to users while also serving as a platform for integrating gaze-tracking hardware. These displays can house imaging components and processing hardware that enables gaze estimation within immersive virtual environments. They can also collaborate with 3D vector reconstruction operations to provide gaze-ray representations suitable for interacting with virtual objects and user interface elements within augmented and virtual reality coordinate frameworks.

[0377] The head-mounted display can be connected to a panoramic camera, which captures environmental and user visual data for integrated gaze estimation processing. The panoramic camera provides imaging capabilities that capture the user's facial features and eye regions, as well as environmental context information. In some embodiments, the panoramic camera can operate within the power-constrained and computationally-intensive environments typical of head-mounted display systems, while maintaining the quality of visual data required for accurate feature extraction via facial and eye feature extractors.

[0378] Both head-mounted displays and panoramic cameras can be connected to a spatial tracker that handles 3D localization and gaze vector reconstruction for immersive applications. The spatial tracker can integrate gaze estimation results with a spatial tracking system that monitors head position and orientation within a virtual environment. It can work in conjunction with a 3D gaze vector reconstruction module, orthogonal basis building units, and camera coordinate transformation units to generate parametric rays that can perform intersection calculations with virtual objects and surfaces within augmented reality and virtual reality coordinate systems.

[0379] The LightGazeNet deployment system can additionally include a desktop platform, which provides a traditional computer-based implementation through the integration of standard webcam hardware and desktop applications. The desktop platform can provide gaze estimation capabilities for desktops and laptops without requiring specialized hardware beyond common webcam devices.

[0380] A webcam interface within a desktop platform can capture user visual data using standard webcam hardware common on desktop and laptop systems. The webcam interface can provide RGB image data, which is processed using the same multimodal input processing operations used in other platform configurations. In some embodiments, the webcam interface can accommodate various webcam specifications and installation configurations while maintaining compatibility with feature extraction and processing operations implemented throughout the gaze estimation pipeline.

[0381] The webcam interface can connect to the application layer, which handles software integration and processing coordination within the desktop computing environment. The application layer manages the execution of unified feature encoding, graph neural networks, and output generation operations within the desktop application framework. It coordinates the data flow between webcam input processing and gaze estimation algorithms, while providing an interface for integration with desktop software applications.

[0382] The application layer can connect to a user interface that manages user interaction and system control functions within the desktop application. The user interface can provide control over calibration operations, system configuration, and visualization of line-of-sight estimation results. In some embodiments, the user interface can provide access to subject-specific calibration capabilities implemented through calibration embeddings and can provide feedback on line-of-sight estimation performance and accuracy.

[0383] The LightGazeNet deployment system can include cross-platform connectivity capabilities, illustrated by connections between the real-time engine, spatial trackers, and application layers. These connections enable interoperability and data sharing between different platform implementations, allowing the gaze estimation system to run on multiple device types while maintaining consistent functionality. Cross-platform connectivity facilitates deployment scenarios where gaze estimation results are shared between mobile devices, AR / VR systems, and desktop platforms for collaborative applications or multi-device interaction systems.

[0384] The real-time engine can communicate with the spatial tracker to enable integration between mobile device gaze estimation and AR / VR spatial tracking systems. This connection can support applications that use mobile devices as input sources for immersive environments, or applications that transfer gaze estimation results between mobile and AR / VR platforms. In some embodiments, the connection between the real-time engine and the spatial tracker can enable synchronous gaze tracking across multiple device types within a shared virtual environment.

[0385] The real-time engine can also communicate with the application layer to enable integration between mobile device gaze estimation and desktop application environments. This connectivity supports scenarios where mobile devices provide gaze input to desktop applications, or where gaze estimation results are processed on both mobile and desktop platforms. The connection between the real-time engine and the application layer facilitates cross-platform data sharing and coordinated processing operations that leverage the computing power of various device types.

[0386] The LightGazeNet deployment system achieves consistent gaze estimation across all three platform configurations through a unified architecture implemented in graph neural networks and standardized processing operations applied in multimodal input processing, unified feature encoding, and output generation stages. The system maintains a model size of less than 5 megabytes across all deployment platforms, enabling cloud-free device gaze estimation that operates independently of network connectivity requirements. In some embodiments, the LightGazeNet deployment system can operate at less than 500 milliwatts of power, making it suitable for battery-powered mobile devices, embedded systems, and low-power AR / VR platforms, while maintaining real-time performance characteristics across all supported hardware configurations.

[0387] refer to Figure 16 Multi-head attention architectures can provide specialized attention-based processing capabilities that enable comprehensive cross-modal relationship modeling within graph neural networks. Multi-head attention architectures can implement systematic attention mechanisms that capture diverse relationship patterns between heterogeneous input modalities through parallel processing operations and coordinated feature transformation stages.

[0388] Multi-head attention architectures can include an attention head array containing four parallel processing units designed to simultaneously capture different types of relational patterns and dependencies within the input data. The attention head array enables the system to model multiple aspects of cross-modal relations through independent attention mechanisms that operate with different learning parameters while processing the same input representation.

[0389] An attention head array may include a first head that processes input node embeddings through a first set of learned attention parameters. The first head may capture specific relational patterns between six input modalities represented within a six-node graph structure, including relationships between facial features, eye features, head rotation data, and spatial localization coordinates. In some embodiments, the first head may focus on specific types of cross-modal dependencies that differ from those captured by other attention heads within the array.

[0390] The second head within the attention head array can operate using independent parameter sets to capture alternative relational patterns between input modalities. The second head can process the same node embeddings as the first head while applying different learning transformations that capture complementary relational information. The second head can work in parallel with the first head, providing diverse perspectives on cross-modal interactions present within graph structures.

[0391] The attention head array may further include a third head, which provides additional parallel processing capability for comprehensive relation modeling. The third head may apply different attention mechanisms that capture relational aspects not addressed by the first or second head. In some embodiments, the third head can enable a multi-head attention architecture to model complex multipath relationships between different combinations of input modalities through specialized attention parameter configurations.

[0392] The first four attention heads within the attention head array can complete the parallel processing array by providing a fourth independent attention mechanism. The fourth head can capture additional relational patterns that complement the processing operations performed by the first, second, and third heads. The fourth head can ensure comprehensive coverage of cross-modal relational modeling through the combined operations of all four parallel attention heads.

[0393] Multi-head attention architectures can include a linear projection layer that receives input from all four attention heads within the attention head array. The linear projection layer transforms the input features into query, key, and value representations, which are used for subsequent attention computation operations. The linear projection layer can apply learned linear transformations to prepare node embeddings for attention coefficient computation and feature aggregation processes.

[0394] The linear projection layer may include a query projection that transforms input node features into query representations for attention coefficient computation. The query projection may apply a learned linear transformation to the node embeddings received from the attention head array to generate a query vector representing the information-seeking aspect of each node within the attention mechanism. In some embodiments, the query projection may create query representations that enable each node to identify relevant information from other nodes within the graph structure.

[0395] Key projection within a linear projection layer transforms input node features into key representations, which serve as the target for attention coefficient computation. Key projection can apply various linear transformations to generate key vectors representing aspects of information provided by each node within the attention framework. Key projection can work in conjunction with query projection to compute attention coefficients, which quantify the strength of relationships between different node pairs.

[0396] The linear projection layer may further include value projection, which transforms the input node features into value representations containing actual information aggregated through an attention mechanism. Value projection may apply learned transformations to generate value vectors representing feature content, which is combined based on attention weights computed through interactions between query and key representations. In some embodiments, value projection may provide feature information weighted and aggregated according to attention weights to produce a final attention-based node representation.

[0397] A multi-head attention architecture may include an attention computation module that processes projected features from a linear projection layer to generate an attention-weighted output. This module can implement the system's attention coefficient calculation, normalization, and feature aggregation operations, which combine query, key, and value representations into enhanced node embeddings that capture cross-modal relationship information.

[0398] The attention calculation module may include a scoring calculator that computes attention coefficients between different input elements using query and key representations generated by the linear projection layer. The scoring calculator can compute the dot product or other similarity measures between the query vector from the query projection and the key vector from the key projection to generate the raw attention coefficients. The scoring calculator can quantify the strength of the relationships between each pair of nodes within the six-node graph structure through system score calculation operations.

[0399] The Softmax normalizer within the attention computation module processes the raw attention coefficients from the score calculator to create a normalized probability distribution. The Softmax normalizer can apply exponential scaling followed by normalization operations, ensuring that the sum of the attention weights for each node is one, while maintaining an appropriate gradient flow for training operations. In some embodiments, the Softmax normalizer can generate normalized attention coefficients that represent the relative importance of connections between each node within the graph structure.

[0400] The attention computation module may further include a feature aggregator that combines weighted features to produce a final attention output. The feature aggregator may receive normalized attention coefficients from a Softmax normalizer and value representations from value projections to compute a weighted combination of node features. The feature aggregator may generate enhanced node embeddings that incorporate information from relevant modalities based on attention-weighted importance determined through a score computation and normalization process.

[0401] The connections within a multi-head attention architecture establish a system information flow from the attention heads through linear projection to the attention computation operations. Heads one, two, three, and four can all be connected to the linear projection layer, providing parallel processing inputs for feature transformation operations. Parallel connections enable the simultaneous processing of the same input embedding through multiple attention mechanisms, while maintaining an independent parameter set for each attention head.

[0402] Both query projections and key projections can be connected to the scoring calculator to provide the query and key representations required for attention coefficient computation. Connectivity enables the scoring calculator to compute similarity measures between query and key vectors, quantifying the strength of relationships between different node pairs within the graph structure. In some embodiments, connectivity can facilitate the systematic computation of attention coefficients that capture cross-modal dependencies between heterogeneous input modalities.

[0403] Value projections can be directly connected to feature aggregators to provide feature content that will be combined according to computed attention weights. This connection allows the feature aggregator to access value representations containing the actual information aggregated through attention-weighted combination operations. Direct connections ensure that the feature content from value projections is properly integrated with attention coefficients generated through fractional computation and normalization processes.

[0404] The score calculator can be connected to a Softmax normalizer to provide the raw attention coefficients for the normalization operation. This connection allows the computed attention coefficients to be transferred from the score calculation stage to the normalization stage, where exponential scaling and the softmax operation generate an appropriately weighted attention distribution. The Softmax normalizer can then be connected to a feature aggregator to provide normalized attention coefficients for the weighted feature combination operation.

[0405] The linear projection layer can be connected to the attention computation module to establish a holistic information flow from feature transformation to attention-based aggregation. The connection coordinates the transfer of query, key, and value representations from the projection operation to the attention computation process that generates the final enhanced node embeddings. In some embodiments, the connection ensures that the transformed features from the linear projection layer are properly integrated within the attention computation framework implemented by the attention computation module.

[0406] The multi-head attention architecture can capture diverse relationship patterns through parallel operations of four attention heads within an attention head array, combined with system feature transformation and aggregation operations implemented through a linear projection layer and attention computation module. This architecture can provide comprehensive cross-modal relationship modeling capabilities that enhance the representation quality of node embeddings within a six-node graph structure while maintaining computational efficiency suitable for real-time deployment on mobile device platforms, AR / VR platforms, and desktop platforms.

[0407] It is understood that the multimodal gaze estimation system based on lightweight graph neural networks provided in this embodiment of the invention corresponds to the multimodal gaze estimation method based on lightweight graph neural networks described above. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the multimodal gaze estimation method based on lightweight graph neural networks, and will not be repeated here.

[0408] Example 7:

[0409] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multimodal gaze estimation method based on a lightweight graph neural network as described in any of the above embodiments and their preferred embodiments.

[0410] It is understood that the multimodal gaze estimation electronic device based on lightweight graph neural networks provided in this embodiment of the invention corresponds to the multimodal gaze estimation method and system based on lightweight graph neural networks described above. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the multimodal gaze estimation method and system based on lightweight graph neural networks, and will not be repeated here.

[0411] Example 8:

[0412] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal gaze estimation method based on a lightweight graph neural network as described in any of the above embodiments and their preferred embodiments.

[0413] It is understood that the multimodal gaze estimation storage medium based on lightweight graph neural networks provided in this embodiment of the invention corresponds to the multimodal gaze estimation method and system based on lightweight graph neural networks described above. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the multimodal gaze estimation method and system based on lightweight graph neural networks, and will not be repeated here.

[0414] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal gaze estimation method based on a lightweight graph neural network, characterized in that, The method includes: Acquire multimodal heterogeneous data that is related to and complementary to the gaze direction estimation, wherein the multimodal heterogeneous data includes visual appearance data, geometric orientation information and spatial positioning coordinates; Visual features are extracted from visual appearance data based on depthwise separable convolution and squeeze excitation modules; Visual features, geometric orientation information, and spatial positioning coordinates are projected into the same vector space; Construct a graph structure, which contains several nodes representing multimodal heterogeneous data or features corresponding to multimodal heterogeneous data; The graph is constructed based on a multi-head attention graph neural network with a two-layer structure to obtain graph-level features; the first layer of the two-layer structure expands the dimension of the node features, and the second layer reduces the dimension of the expanded features. The graph-level features are aggregated and fused with visual features, geometric orientation information, and spatial positioning coordinates. A gaze estimation prediction is performed using a fully connected layer and based on fused features. Based on coordinate transformation, the gaze estimation prediction is converted into a normalized three-dimensional gaze vector.

2. The method as described in claim 1, characterized in that, The visual appearance data includes facial image data, eye cropped image data, geometric orientation information including head rotation data, and spatial positioning coordinates including the three-dimensional eye center position.

3. The method as described in claim 1, characterized in that, The construction of the graph structure includes: Construct a fully connected graph consisting of six nodes; the six nodes include: facial features, left eye features, right eye features, head rotation data, left eye 3D position, and the node corresponding to the right eye 3D position; Learnable edge weights are established between the six node pairs.

4. The method as described in claim 1, characterized in that, Coordinate transformation includes converting spherical coordinates to Cartesian coordinates.

5. The method as described in claim 1, characterized in that, The method further includes: When estimating the line of sight for a specific subject, the multimodal heterogeneous data also includes subject-specific calibration data.

6. The method as described in claim 5, characterized in that, The specific subject calibration includes: During calibration, only the specific subject calibration embedding and final fully connected layer parameters are updated, while all other network parameters remain frozen.

7. The method as described in claim 3, characterized in that, The graph constructed using a two-layer multi-head attention graph neural network is used to obtain graph-level features, including the following steps: Step 4.1: Initialize the six graph nodes using projection feature embedding; Step 4.2: Calculate the attention coefficients for all node pairs using linear projection; Step 4.3: Apply LeakyReLU activation and softmax normalization to the attention coefficients; Step 4.4: Aggregate neighboring features using weighted attention coefficients; Step 4.5: Determine if the current processing corresponds to the first GNN layer. If yes, proceed to step 4.6; otherwise, proceed to step 4.

7. Step 4.6: Extend the node features to 256 dimensions; Step 4.7: Reduce node features to 32 dimensions; Step 4.8: Apply the second-layer GNN for processing; Step 4.9: Flatten all node features and connect them into a unified representation; Step 4.10: Output the final graph-level feature vector.

8. The method as described in claim 1, characterized in that, The conversion of line-of-sight estimation prediction into a normalized 3D line-of-sight vector based on coordinate transformation includes: An orthogonal normalized coordinate frame is constructed using cross product, and parametric rays emanating from the center of the 3D eye are generated.

9. The method as described in claim 8, characterized in that, The parametric ray can perform intersection calculations with objects including virtual objects, user interfaces, and physical surfaces in spatial applications.

10. The method of claim 1, characterized in that, The multi-head attention comprises four attention heads that process in parallel, wherein: The first is used to capture specific relationship patterns between the six input modalities represented within a six-node graph structure; The first two are used to capture the substitution relationship patterns between input modes; The first three are used to provide additional parallel processing capabilities for modeling comprehensive relationships; The first four can be accomplished by providing a fourth independent attention mechanism to complete the parallel processing array.

11. A multimodal gaze estimation system based on a lightweight graph neural network, characterized in that, The system includes: The multimodal input module is configured to acquire multimodal heterogeneous data that is related to and complementary to the gaze direction estimation, the multimodal heterogeneous data including visual appearance data, geometric orientation information and spatial positioning coordinates; The feature extraction module is configured to extract visual features from visual appearance data based on depthwise separable convolution and squeeze excitation module. The projection module is configured to project the visual features, geometric orientation information, and spatial positioning coordinates into the same vector space; The graph construction module is configured to construct a graph structure, which contains several nodes representing multimodal heterogeneous data or features corresponding to multimodal heterogeneous data. Multi-head attention GNN is configured to obtain graph-level features based on the graph constructed by the multi-head attention graph neural network with a two-layer structure. In the two-layer structure, the first layer expands the dimension of the node features, and the second layer reduces the dimension of the expanded features. The feature fusion module is configured to aggregate the graph-level features and fuse them with the visual features, geometric orientation information, and spatial positioning coordinates. The regression module is configured to utilize fully connected layers and perform gaze estimation prediction based on fused features; The 3D reconstruction module is configured to convert gaze estimation predictions into normalized 3D gaze vectors based on coordinate transformation.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the multimodal gaze estimation method based on a lightweight graph neural network as described in any one of claims 1-10.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the multimodal gaze estimation method based on a lightweight graph neural network as described in any one of claims 1-10.

Images