Metaverse eye movement tracking system and method based on improved yolov3 and sub-pixel refinement and application

Abstract

The application discloses a real-time eye movement tracking method and system based on a YOLOv13 key point model and sub-pixel refinement, responds to the urgent demand of meta-universe application for high frame rate and high precision human-computer interaction, and constructs an intelligent eye movement tracking platform integrating high-speed acquisition and real-time accurate positioning. The system can be deployed in immersive interactive scenes such as meta-universe virtual cockpits, and the closed-loop processing flow from 'color image acquisition' to 'YOLOv13 coarse positioning' to'sub-pixel accurate positioning' and finally 'output coordinates' significantly improves the accuracy (≤), frame rate (≥200Hz) and low delay (≤10ms) of eye movement tracking. At the same time, due to the robustness and high precision characteristics of the algorithm, it can be quickly migrated to other fields (such as auxiliary driving and cognitive science research) requiring high-performance human-computer interaction, and provides a standardized eye movement tracking solution for multi-field professional applications.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction, specifically relating to a metaverse eye-tracking system, method, and application based on improved YOLOv13 and subpixel refinement. Background Technology

[0002] Eye tracking is a core technology in human-computer interaction, virtual reality (VR), augmented reality (AR), and metaverse applications (such as virtual cockpits). Existing eye tracking methods are mainly divided into two categories: those based on infrared light sources and those based on pure vision.

[0003] While infrared-based methods offer high accuracy, they suffer from high hardware costs and power consumption, and prolonged exposure to infrared light can cause user discomfort. Traditional purely visual methods, such as those based on feature points (e.g., Haar, HOG) or conventional image processing (e.g., Hough transform), struggle to meet the demands of next-generation human-computer interaction in terms of processing speed, robustness (especially in low light, under conditions of eye occlusion, or rapid eye movements), and tracking accuracy.

[0004] In recent years, deep learning methods have become the state-of-the-art (SOTA) choice, especially in the field of general object detection, where single-stage detectors, represented by the YOLO series, have achieved great success. For example, the YOLOv13 model represents the current state of technology. To achieve its superior detection performance, advanced models like YOLOv13 typically introduce extremely complex and innovative modules, including but not limited to:

[0005] ① Advanced Backbone Network: Its backbone network may employ modules such as "HyperACE" (HyperGraph Enhancement Adaptive Core), replacing the traditional CSP or C2f structure. This structure treats the channels and spatial locations of the feature map as nodes in a hypergraph, and aggregates features by dynamically learning the correlations between nodes, thereby achieving context awareness and feature representation capabilities far exceeding those of static convolution.

[0006] ② FullPAD Neck: Its neck may adopt a structure such as "full-process aggregation and distribution" (as shown in the attached image). Figure 3 (As shown in "FullPAD Tunnel"). Unlike the static feature concatenation of traditional PANet, FullPAD efficiently fuses feature maps from different levels (such as P3, P4, P5) through a carefully designed aggregation and distribution path, making it perform better when processing multi-scale targets.

[0007] ③ Highly efficient decoupled detection head: Its detection head adopts a decoupled design, separating the classification and regression tasks, effectively resolving the conflict between the two.

[0008] However, despite the high performance of the aforementioned models on general object detection (such as the COCO dataset), it is not feasible to directly apply such state-of-the-art models to specific scenarios—namely, "real-time eye tracking on smart glasses (ESP32)"—and faces the following two core and fatal limitations:

[0009] The task paradigms are fundamentally mismatched: YOLOv13 is designed for "object detection," and its entire architecture (including the head design and loss function) is optimized for "bounding box" regression and "class" prediction. Eye tracking, on the other hand, is essentially "keypoint detection," aiming to predict precise "pupil center coordinates." These two approaches differ drastically in the physical meaning of their outputs, their dimensionality, and the required network structure.

[0010] The scale mismatch in accuracy requirements: General-purpose object detectors, in pursuit of speed, often perform high-level downsampling (e.g., predicting large targets on 1 / 32 or even 1 / 64 of a feature map). This design is sufficient for "boxing" a "person" or "vehicle," but disastrous for "locating" a "pupil." On a 1 / 16 feature map, a one-pixel deviation magnified back to the original image results in a 16-pixel error, far exceeding the sub-pixel accuracy required for eye tracking.

[0011] Therefore, there is an urgent need for a new eye-tracking method that cannot simply abandon the powerful feature representation capabilities of state-of-the-art models (such as YOLOv13). Instead, it must be "surgically" modified and "extremely" lightweighted to balance model accuracy and inference efficiency while retaining its core advantages, and provide a real-time and accurate interactive experience in immersive scenarios such as the metaverse. Summary of the Invention

[0012] The technical problem to be solved by this invention is to provide a metaverse eye-tracking system, method and application based on improved YOLOv13 and sub-pixel refinement, which solves the problem of scale mismatch between task paradigm and accuracy requirements in the prior art.

[0013] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0014] The Metaverse eye-tracking system based on the improved YOLOv13 and subpixel refinement includes a smart glasses terminal, which integrates at least one ESP32 vision module for acquiring visible light images and a low-latency wireless communication module for data transmission.

[0015] Metaverse application scenario, the metaverse application scenario is an immersive virtual cockpit based on first-person perspective, used to receive eye tracking results and provide high-fidelity real-time eye interaction feedback;

[0016] An edge computing control terminal is connected to the smart glasses terminal via a low-latency wireless communication module and is equipped with an eye-tracking algorithm module;

[0017] The eye-tracking algorithm module is deployed on the edge computing control terminal and is used to process the eye color RGB image data stream acquired by the ESP32 vision module in real time, and calculate high-precision sub-pixel pupil center coordinates as gaze point coordinates;

[0018] The edge computing control terminal outputs the high-precision sub-pixel pupil center coordinates to the metaverse application scenario output through a closed-loop real-time feedback mechanism, based on the calculation results.

[0019] The eye-tracking algorithm module includes a composite algorithm pipeline of serialization processing and parallel computing, which includes at least:

[0020] Image preprocessing unit: used to process the raw color RGB image data stream acquired by the ESP32 vision module;

[0021] Eye Image ROI Fast Localization Unit: Employs a lightweight detector combined with a tracking algorithm to efficiently locate the eye region in the entire frame;

[0022] YOLOv13 key point detection unit: used to process the ROI color RGB image region and directly predict the pixel-level pupil center coordinates;

[0023] Sub-pixel refinement unit: used to refine the pixel-level pupil center coordinates to obtain high-precision pupil center coordinates at the sub-pixel level;

[0024] Coordinate output unit: Used to send the final coordinates output by the sub-pixel refining unit to the meta-universe application scenario.

[0025] The YOLOv13 keypoint detection unit includes detection head conversion, feature extraction, feature fusion modification, and platform inference optimization units; wherein...

[0026] The detection head conversion retains the original detection branch of the YOLOv13 object detection baseline model detection head and adds a dedicated Pose branch, realizing the conversion from Detect head to Pose head;

[0027] Feature extraction unit: The HyperACE module in the backbone network of the YOLOv13 baseline model is retained and used for feature extraction;

[0028] Feature fusion modification: Modify the feature fusion neck of the baseline model by forcing it to fuse and upsample high-resolution feature maps to obtain the modified baseline model;

[0029] Platform inference optimization: The modified baseline model is deployed on the edge computing control terminal and deeply optimized by inference acceleration, INT8 quantization or structured pruning, so that it can run with extremely low latency on the target hardware.

[0030] The refining process of the sub-pixel refining unit includes the following steps:

[0031] Step a: Using the pixel-level coordinates output by the YOLOv13 keypoint detection unit Centered on the color RGB window, the image is converted into a single-channel grayscale image, and a high-resolution N×N neighborhood window is extracted, retaining only the light and dark boundary information used for geometric fitting.

[0032] Step b: Apply bilinear interpolation algorithm to upsample the N×N grayscale neighborhood window to generate a smooth, continuous sub-pixel grayscale field;

[0033] Step c: On the interpolated continuous grayscale field, apply edge detection or adaptive thresholding to extract the contour point set of the pupil edge. ;

[0034] Step d: Apply the least squares circular fitting algorithm to fit the set of contour points extracted in step c. Utilize the geometric prior knowledge that "the pupil is approximately circular" to calculate a robust center coordinate by minimizing the algebraic distance. This coordinate is the final subpixel-level pupil center.

[0035] The metaverse application scenario is a virtual cockpit, which uses the sub-pixel level pupil center coordinates output by the edge computing control terminal to achieve the following human-computer interaction functions:

[0036] Dynamic gaze point rendering: Using the gaze point coordinates, the instantaneous area of ​​the user's gaze focus is rendered at full resolution, while the edge area of ​​the gaze is rendered at low resolution to optimize rendering performance and GPU load.

[0037] Hands-free eye-tracking interaction: Users can select and confirm information on the instrument panel, central control screen buttons, or head-up display in the virtual cockpit by "gazing".

[0038] The "gaze-satiety" method refers to determining whether the gaze point remains in a certain area for more than a threshold time.

[0039] It also includes a personalized calibration module, which is activated when the user first uses it. The user focuses on several points in the metaverse scene that appear in a specific time sequence to calibrate the target. The module also automatically calculates a high-order polynomial or neural network model based on the pupil coordinate data pairs collected in real time, which serves as a personalized mapping function between the user's gaze vector and the input pupil coordinates.

[0040] The high-precision eye-tracking method for the metaverse virtual cockpit based on the aforementioned system includes the following steps:

[0041] Step 1: Startup and Calibration. Start the smart glasses terminal and control terminal, and perform initialization calibration.

[0042] Step 2: Acquisition and preprocessing. The ESP32 vision module is used to acquire the eye's color RGB image stream in real time and perform preprocessing.

[0043] Step 3: ROI localization. The eye region is cropped from the preprocessed image by quickly locating the ROI unit in the eye image.

[0044] Step 4: Coarse localization of YOLOv13 key points. The color RGB image of the eye ROI is input into the modified YOLOv13 key point detection unit to directly predict the pixel-level coordinates of the pupil center. ;

[0045] Step 5: Sub-pixel refinement. The sub-pixel refinement unit is activated to refine the pixel-level coordinates of the pupil center. Refined calculations were performed to obtain high-precision pupil center coordinates. ;

[0046] Step 6: Coordinate output and interaction, outputting the high-precision pupil center coordinates. Send directly to the metaverse virtual cockpit for foveated rendering or gaze interaction.

[0047] Step 5 specifically includes:

[0048] Step 5.1, with Centered on a high-resolution ROI, extract neighborhood windows and convert them to grayscale images;

[0049] Step 5.2: Upsample the grayscale image using bilinear interpolation to obtain the grayscale field;

[0050] Step 5.3: Apply the Canny operator or Otsu's method to extract the edge contour point set of the pupil from the interpolated grayscale field. ;

[0051] Step 5.4: Using the edge contour point set extracted in Step 5.3 as input, apply the least squares circle fitting algorithm to calculate the final pupil center coordinates. .

[0052] Based on the improved YOLOv13 and the subpixel refined Metaverse eye-tracking system, this system is applied to immersive interactive scenarios including human-computer interaction, virtual reality, augmented reality, and the Metaverse.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. Hardware Collaboration: Utilize ESP32 as a low-power, high-frame-rate acquisition terminal and Jetson AGX Orin (edge ​​computing unit) as a high-performance processing terminal.

[0055] Algorithm collaboration: YOLOv13-pose is used to process color RGB images to achieve robust recognition and coarse localization; classic CV (S9-S11) is used to process grayscale images to achieve high-precision geometric localization.

[0056] 2. Optimized Collaboration: Utilizing optimizations such as "Detect-then-Track" and TensorRT, end-to-end latency on edge computers is ensured. Thus, High frame rate.

[0057] 3. A high-precision system was implemented on the Jetson AGX Orin platform. High frame rate and low latency Eye-tracking system.

[0058] 4. This system can be deployed in immersive interactive scenarios such as the Metaverse Virtual Cockpit. At the same time, due to the robustness and high precision of the algorithm, it can be quickly migrated to other fields (such as assisted driving and cognitive science research) that require high-performance human-computer interaction, providing standardized eye-tracking solutions for professional applications in multiple fields. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the hardware architecture of the eye-tracking system provided in an embodiment of the present invention.

[0060] Figure 2 This is an overall flowchart of the eye-tracking method provided in an embodiment of the present invention.

[0061] Figure 3 This is a schematic diagram of the head modification of the YOLOv13 model in an embodiment of the present invention.

[0062] Figure 4This is a flowchart illustrating the sub-pixel refining algorithm in an embodiment of the present invention.

[0063] The labels in the diagram are: 1-Smart Glasses; 2-ESP32 Vision Module; 3-Virtual Cockpit; 4-Metaverse Interactive Terminal; 5-Edge Computer. Detailed Implementation

[0064] The structure and working process of the present invention will be further described below with reference to the accompanying drawings.

[0065] The purpose of this invention is to overcome the shortcomings of the prior art, particularly to resolve the fundamental contradiction between the "massive computing power requirements of high-performance state-of-the-art models (such as YOLOv13)" and the "lightweight / low-power consumption limitations of wearable devices such as smart glasses (such as ESP32)". Existing technologies either suffer from insufficient accuracy due to model simplicity or are too complex for lightweight glasses terminals. This invention aims to provide a "separation of acquisition and computation" system architecture to achieve high-precision, high-frame-rate real-time eye tracking while ensuring lightweight and low-power wearable devices.

[0066] The Metaverse eye-tracking system based on the improved YOLOv13 and subpixel refinement includes a smart glasses terminal, which integrates at least one ESP32 vision module for acquiring visible light images and a low-latency wireless communication module for data transmission.

[0067] Metaverse application scenario, the metaverse application scenario is an immersive virtual cockpit based on first-person perspective, used to receive eye tracking results and provide high-fidelity real-time eye interaction feedback;

[0068] An edge computing control terminal is connected to the smart glasses terminal via a low-latency wireless communication module and is equipped with an eye-tracking algorithm module;

[0069] The eye-tracking algorithm module is deployed on the edge computing control terminal and is used to process the eye color RGB image data stream acquired by the ESP32 vision module in real time, and calculate high-precision sub-pixel pupil center coordinates as gaze point coordinates;

[0070] The edge computing control terminal outputs the high-precision sub-pixel pupil center coordinates to the metaverse application scenario output through a closed-loop real-time feedback mechanism, based on the calculation results.

[0071] The high-precision eye-tracking method for the Metaverse virtual cockpit includes the following steps:

[0072] Step 1: Startup and Calibration. Start the smart glasses terminal and control terminal, and perform initialization calibration.

[0073] Step 2: Acquisition and preprocessing. The ESP32 vision module is used to acquire the eye's color RGB image stream in real time and perform preprocessing.

[0074] Step 3: ROI localization. The eye region is cropped from the preprocessed image by quickly locating the ROI unit in the eye image.

[0075] Step 4: Coarse localization of YOLOv13 key points. The color RGB image of the eye ROI is input into the modified YOLOv13 key point detection unit to directly predict the pixel-level coordinates of the pupil center. ;

[0076] Step 5: Sub-pixel refinement. The sub-pixel refinement unit is activated to refine the pixel-level coordinates of the pupil center. Refined calculations were performed to obtain high-precision pupil center coordinates. ;

[0077] Step 6: Coordinate output and interaction, outputting the high-precision pupil center coordinates. Send directly to the metaverse virtual cockpit for foveated rendering or gaze interaction.

[0078] Specific embodiments, such as Figures 1 to 4 As shown:

[0079] This embodiment discloses a metaverse eye-tracking system based on improved YOLOv13 and sub-pixel refinement, including a smart glasses terminal, a metaverse application scenario, and an edge computing control terminal. The core of this solution is a "collection-computation separation" architecture.

[0080] Hardware layer: A pair of smart glasses with a built-in image sensor (with an ESP32-based vision module) as the image acquisition unit; an edge computing unit (Jetson AGX Orin series computer) that communicates with the smart glasses (wireless / wired) to receive and process data acquired by the image sensor; a lightweight acquisition terminal responsible for high frame rate image capture; and a human-computer interaction terminal for presenting the metaverse virtual cockpit scene.

[0081] Algorithm layer: The program embedded in the edge computing unit (e.g., Jetson AGX Orin) is used to execute the eye-tracking method. The high-performance edge computing control terminal (such as Jetson AGX Orin) acts as the algorithm processing center, responsible for receiving image data streams and running the improved YOLOv13 keypoint model and sub-pixel refinement algorithm described in this invention. In this way, this invention offloads the heavy computational tasks from the wearable device, resolving the aforementioned contradiction between computing power and power consumption.

[0082] The specific method for implementing real-time eye tracking using this system will be explained in detail, taking the Metaverse virtual cockpit scene as an example. The specific steps include:

[0083] Step S1: High frame rate image acquisition. Activate the image sensor on the smart glasses (vision module with ESP32 as the core) to continuously capture original color RGB image frames (Frame_t) of the user's eye area at a high frame rate of ≥200HZ.

[0084] Step S2: Data stream transmission. The raw image frame data stream acquired by the image sensor is transmitted in real time to the memory of the edge computing unit (Jetson AGX Orin) via a low-latency wireless (Wi-Fi) interface.

[0085] Step S3: Region of Interest (ROI) localization and tracking. To maximize the efficiency of the neural network without sacrificing accuracy, ROI processing is performed on the received image frames at the edge computing unit. This step is preferably performed using a "detection-tracking" strategy.

[0086] a) Initialization detection: When tracking loss is detected during system initialization or loop, a dlib face detector is called to determine the approximate location of the eyes.

[0087] b) High frame rate tracking: Once locked, in subsequent frames, taking advantage of the minimal inter-frame displacement at high frame rates (200Hz), the ROI of the current frame is directly "blindly cut" near the ROI position of the previous frame using Kalman filtering or high-speed template matching, based on the ROI position of the previous frame. This avoids the computational overhead of running expensive detectors in every frame and is one of the key optimizations for achieving a 200Hz high frame rate.

[0088] Step S4: Normalization processing. The color RGB ROI image patch (including a single eye) obtained in Step S3 is uniformly resized and padded to the fixed input size (640×640 pixels) required by the YOLOv13 keypoint detection model, and grayscale value normalization is performed to convert it to a format suitable for the model input. Tensors in the specified format are ready to be fed into the neural network.

[0089] Step S5: Accelerate model inference. The preprocessed ROI image tensor from Step S4 is fed into the YOLOv13 keypoint detection model deployed on Jetson AGX Orin. A key point is that the model has been pre-optimized for inference depth (INT8 quantization or FP16 half-precision conversion) using tools such as NVIDIA TensorRT to fully utilize the GPU and DLA (Deep Learning Accelerator) hardware computing power of Orin, achieving extremely low-latency inference and compressing the inference latency of YOLOv13 to the extreme.

[0090] Step S6: Backbone Network and Neck Feature Extraction. The color RGB ROI tensor first flows through the modified YOLOv13 model's backbone network and feature fusion neck. The innovation lies in the fact that, during the modification process, the powerful feature representation capabilities of advanced modules such as HyperACE are retained (to cope with complex situations such as lighting and occlusion), and high-resolution feature maps are forcibly fused and upsampled in the neck area, strengthening the fusion path of high-resolution feature maps and preserving the necessary spatial details for accurate localization in S7.

[0091] Step S7: Pose estimation head regression. The high-resolution fused feature map output from step S6 is input into the modified YOLOv13 pose estimation head. A keypoint detection branch is added to transform it into a keypoint detection model. This head directly predicts the pixel-level coordinates of the pupil center. and its confidence / visibility This step is to achieve The key to the accuracy of the viewing angle.

[0092] Step S8: High-resolution neighborhood extraction, using the pixel-level coordinates obtained in step S7. Centered on the original RGB ROI image cropped in step S3, convert it to grayscale, and then extract an N×N (N≥16) neighborhood pixel window.

[0093] Step S9: Neighborhood Interpolation and Gray-Scale Conversion for Continuity. To overcome the discreteness of pixels, a bilinear interpolation algorithm is applied to upsample the N×N (N≥16) neighborhood window extracted in step S8, generating a smoother and finer continuous single-channel gray-scale field. Its beneficial effect is that after YOLOv13 (S7) completes the "recognition" task using color information, the "precise localization" task in steps (S9-S11) only needs to rely on "bright and dark boundaries." Converting to grayscale completely eliminates interference from iris color (such as blue, brown) and other information, making the subsequent geometric fitting (S11) more robust.

[0094] Step S10: Pupil contour point set extraction. On the interpolated grayscale field generated in step S9, Canny edge detection or adaptive thresholding is applied to extract the contour point set of the pupil edge. .

[0095] Step S11: Subpixel coordinate fitting. The least squares circular fitting algorithm is applied to fit the set of M contour points extracted in step S10. This fitting algorithm utilizes the geometric prior that "the pupil is circular" and calculates a robust center coordinate that is not severely affected by individual noise points (such as eyelash obstruction or Purchiner reflections) by minimizing the algebraic distance from all contour points to the assumed center. These coordinates are the sub-pixel level pupil center coordinates that are ultimately output by this invention.

[0096] Step S12: Gaze calculation and output, converting the sub-pixel pupil center coordinates obtained in step S11 into... The data is input into a pre-calibrated gaze estimation model, which calculates the instantaneous gaze coordinates of the user in the metaverse virtual cockpit scene. The coordinate data is then sent to the human-computer interaction terminal via socket communication to drive the interaction.

[0097] Output and Visualization: The edge computing unit (Orin) sends the gaze coordinates to the interactive terminal via the network. On the Orin's debug display, the system can display the sub-pixel coordinates obtained by S11. As a marker point (e.g., a green cross), it is overlaid in real time onto the original color RGB video stream acquired in step S1 for developers to visualize and verify.

[0098] Furthermore, the "modification" of YOLOv13 is a deep adaptation rather than a simple replacement. This modification retains the core feature extraction capabilities of the YOLOv13 backbone (e.g., the HyperACE module described in the background section) after pruning and lightweighting, to fully leverage its powerful visual representation learning capabilities (e.g., handling illumination changes and occlusion). The innovation lies in:

[0099] ① A completely new pose estimation head was designed. Based on the original detection branch 002, a dedicated Pose branch (responsible for "point" prediction) was added, realizing a shift in the task objective. This changes the computational endpoint of the model from "surface" prediction to "point" prediction, solving the "task paradigm mismatch" problem in the background technology.

[0100] ② The modified model has been deeply optimized for high-performance edge AI computing platforms such as Jetson AGX Orin. By employing techniques such as NVIDIA TensorRT for inference acceleration, INT8 quantization, and structured pruning, its computational load and parameter count are significantly compressed, enabling it to run on the target hardware (Jetson AGX Orin) with extremely low latency (e.g., ≤10ms), thereby achieving the ultra-high frame rate real-time processing requirement of ≥200Hz. This solves the problem in the background technology where, although the state-of-the-art (SOTA) model has high computational overhead, it can still meet the stringent real-time performance requirements at the edge after optimization by this invention.

[0101] Furthermore, the training process of the YOLOv13 keypoint detection model in the keypoint detection step includes:

[0102] a) In the pre-training stage, large-scale public face data (Flicker-Faces-HQ dataset) is used to crop the eye ROI of face images to form an eye keypoint dataset. The YOLOv13 keypoint model is then pre-trained. This stage uses rich data augmentation, including affine transformation and strong lighting / dark lighting simulation, to enable it to learn a general representation of eye features.

[0103] (b) In the fine-tuning stage, the model is fine-tuned using a highly targeted "private" color RGB dataset collected by the inventors of this invention. This dataset is specific to smart glasses wearing scenarios (including different lighting, angles, and occlusion conditions) and is collected in actual application scenarios to enhance the model's generalization and robustness under target hardware and real-world application scenarios. This dataset covers "corner cases" such as strong light (pupil constriction), dim light (pupil dilation), extreme rotation (saccade), blinking, and occlusion. This step is crucial for adapting the model to the specific imaging characteristics of the ESP32 sensor and the near-eye shooting perspective.

[0104] Furthermore, the loss function used in the fine-tuning stage of step b) is specifically designed for this task. Since the original loss function of YOLOv13 is no longer fully applicable, this invention upgrades the original loss function as follows:

[0105]

[0106] in, For the total loss function, For classification loss, binary cross-entropy loss is used to measure the accuracy of the model's prediction of the target class. For bounding box regression loss, , , , , where are the weight coefficients of each loss term, all of which are non-negative real numbers; CIoU Loss is used to measure the degree of matching between the predicted bounding box and the ground truth bounding box, serving as an auxiliary loss to help train the backbone network, and is also used to predict the pupil bounding box; This is a distributed focus loss used to assist in the regression of the bounding box; This is the keypoint loss term, using mean squared error loss (MSE).

[0107] Furthermore, the sub-pixel refining step is implemented using a multi-stage algorithm, specifically including:

[0108] a) Bilinear interpolation application: First, a small neighborhood window is constructed around the coarse pupil center coordinates output by the model, and the gray value or feature map response within the window is upsampled using the bilinear interpolation algorithm to obtain refined information of non-integer coordinate positions.

[0109] b) Least squares circular fitting application: Next, in the interpolated neighborhood, the edge contour point set of the pupil is extracted by edge detection or threshold segmentation, and then the least squares method is used to perform circular fitting on the point set. The center coordinates of the fitted circle are regarded as the final sub-pixel level pupil center coordinates.

[0110] Furthermore, this invention employs a "two-stage" refining scheme, rather than a single centroid method or heatmap argmax, which has the following advantages:

[0111] a) Overcoming the "pixel lock-in" effect: The output of a neural network (whether it's regression coordinates or a heatmap) is inherently discrete, and its results tend to "lock in" to integer pixel coordinates. Bilinear interpolation breaks this pixel grid constraint by mathematically creating a smooth, continuous sub-pixel grayscale field.

[0112] b) Enhancing robustness by leveraging prior knowledge: The simple centroid method is extremely sensitive to light reflection (Pulchinjer spot) or eyelash occlusion (dark spot). Circular fitting, on the other hand, utilizes the geometric prior knowledge that "the pupil is approximately circular." It is a global fit that finds an "optimal circle" that minimizes the error of all edge points, thus exhibiting strong robustness to individual noise points (such as a small missing edge due to eyelash occlusion).

[0113] Furthermore, in the sub-pixel refining step, the mathematical principle of bilinear interpolation is as follows: for a non-integer coordinate... Its grayscale value From its four surrounding integer pixels grayscale value Decision. First, perform linear interpolation in the x-direction:

[0114]

[0115]

[0116] Then perform linear interpolation in the y-direction:

[0117]

[0118] Furthermore, in the sub-pixel refinement step, the mathematical principle of least squares circular fitting is as follows: Let the set of pupil edge points be... The goal is to find the center of a circle. and radius This minimizes the sum of the squares of the distances from all points to the center of the circle, i.e., minimizes the error function. To simplify calculations, algebraic distance minimization is typically employed, i.e., solving... In This can be constructed as an overdetermined system of linear equations, and solved by finding its normal equations. To obtain The least squares solution, where It is by The matrix formed It is by The vector formed. The final center coordinates are... .

[0119] Furthermore, when the method is run on a Jetson AGX Orin series edge computer, it achieves end-to-end performance metrics: eye tracking accuracy ≤0.7°, single-frame processing latency ≤10ms, and system frame rate ≥200Hz.

[0120] In summary, the specific embodiments of the present invention solve the problems in the background art through the following innovative combinations:

[0121] Hardware collaboration: Utilizing ESP32 as a low-power, high-frame-rate acquisition terminal and Jetson AGX Orin (edge ​​computing unit) as a high-performance processing terminal.

[0122] Algorithm collaboration: YOLOv13-pose is used to process color RGB images to achieve robust recognition and coarse localization; classic CV (S9-S11) is used to process grayscale images to achieve high-precision geometric localization.

[0123] Optimized collaboration: By leveraging optimizations such as "Detect-then-Track" and TensorRT, end-to-end latency on Orin is ensured. Thus, High frame rate.

[0124] Ultimately, this invention implements a high-precision system on the Jetson AGX Orin platform. High frame rate and low latency Eye-tracking system.

[0125] This invention proposes a real-time eye-tracking method and system based on the YOLOv13 keypoint model and sub-pixel refinement. Responding to the urgent need for high frame rate and high-precision human-computer interaction in Metaverse applications, it constructs an intelligent eye-tracking platform integrating high-speed acquisition and real-time precise positioning. The system comprises two core modules: an intelligent glasses acquisition terminal and an edge computing controller.

[0126] The system supports a closed-loop processing flow from "color image acquisition" to "YOLOv13 coarse localization," then to "sub-pixel fine localization," and finally "output coordinates." The system uses a modified YOLOv13 model to process color RGB images. By modifying the target detection head into a pose estimation head and retaining the backbone (e.g., HyperACE), robust pixel-level coarse localization is achieved. Subsequently, a hybrid vision refinement algorithm is used to convert the coarse localization neighborhood into a grayscale image. Sub-pixel accuracy is obtained through bilinear interpolation, edge extraction, and circular fitting. The hardware adopts an "acquisition-computation separation" architecture, with a lightweight ESP32 vision module handling high-frame-rate acquisition and a high-performance Orin platform handling TensorRT accelerated inference.

[0127] Its core technologies include:

[0128] YOLOv13 Pose Head Modification Technology: Transforms the target detection head into a key point detection head and forces the fusion of high-resolution feature maps (such as P2 level) to solve the mismatch between "task" and "accuracy" in the background technology.

[0129] Hybrid vision-based fine localization pipeline: YOLOv13 (RGB color) is used for robust coarse localization, while classic CV (grayscale) is used for high-precision geometric fine localization, achieving algorithm synergy.

[0130] Two-stage subpixel circular fitting: Improve the robustness of the refinement by "extracting edge points with Canny / Otsu" and applying "least squares method to fit the center of the circle".

[0131] Hardware and software co-optimization: Employing ROI strategies and TensorRT inference acceleration, a high frame rate of ≥200Hz is achieved on the Jetson AGX Orin platform.

[0132] This system can be deployed in immersive interactive scenarios such as the Metaverse Virtual Cockpit, significantly improving the accuracy of eye tracking (≤ High frame rate (≥200Hz) and low latency (≤10ms). Furthermore, due to the algorithm's robustness and high precision, it can be quickly migrated to other fields (such as assisted driving and cognitive science research) requiring high-performance human-computer interaction, providing standardized eye-tracking solutions for professional applications across multiple fields.

[0133] The various embodiments described herein have different specific implementation methods and application scenarios. Those skilled in the art can combine the technical features of different embodiments or combine the technical solutions of different embodiments in specific implementations according to the ideas of the present invention, thereby forming new embodiments, all of which do not depart from the ideas disclosed in the present invention.

Claims

1. A metaverse eye-tracking system based on improved YOLOv13 and sub-pixel refinement, characterized by: The device includes a smart glasses terminal, which integrates at least one ESP32 vision module for acquiring visible light images and a low-latency wireless communication module for data transmission. Metaverse application scenario, the metaverse application scenario is an immersive virtual cockpit based on first-person perspective, used to receive eye tracking results and provide high-fidelity real-time eye interaction feedback; An edge computing control terminal is connected to the smart glasses terminal via a low-latency wireless communication module and is equipped with an eye-tracking algorithm module; The eye-tracking algorithm module is deployed on the edge computing control terminal and is used to process the eye color RGB image data stream acquired by the ESP32 vision module in real time, and calculate high-precision sub-pixel pupil center coordinates as gaze point coordinates; The edge computing control terminal outputs the high-precision sub-pixel pupil center coordinates to the metaverse application scenario output through a closed-loop real-time feedback mechanism, based on the calculation results.

2. The metaverse eye-tracking system based on improved YOLOv13 and sub-pixel refinement according to claim 1, characterized in that: The eye-tracking algorithm module includes a composite algorithm pipeline of serialization processing and parallel computing, which includes at least: Image preprocessing unit: used to process the raw color RGB image data stream acquired by the ESP32 vision module; Eye Image ROI Fast Localization Unit: Employs a lightweight detector combined with a tracking algorithm to efficiently locate the eye region in the entire frame; YOLOv13 key point detection unit: used to process the ROI color RGB image region and directly predict the pixel-level pupil center coordinates; Sub-pixel refinement unit: used to refine the pixel-level pupil center coordinates to obtain high-precision pupil center coordinates at the sub-pixel level; Coordinate output unit: Used to send the final coordinates output by the sub-pixel refining unit to the meta-universe application scenario.

3. The metaverse eye-tracking system based on improved YOLOv13 and sub-pixel refinement according to claim 2, characterized in that: The YOLOv13 keypoint detection unit includes detection head conversion, feature extraction, feature fusion modification, and platform inference optimization units; wherein... The detection head conversion retains the original detection branch of the YOLOv13 object detection baseline model detection head and adds a dedicated Pose branch, realizing the conversion from Detect head to Pose head; Feature extraction unit: The HyperACE module in the backbone network of the YOLOv13 baseline model is retained and used for feature extraction; Feature fusion modification: Modify the feature fusion neck of the baseline model by forcing it to fuse and upsample high-resolution feature maps to obtain the modified baseline model; Platform inference optimization: The modified baseline model is deployed on the edge computing control terminal and deeply optimized by inference acceleration, INT8 quantization or structured pruning, so that it can run with extremely low latency on the target hardware.

4. The metaverse eye-tracking system based on improved YOLOv13 and sub-pixel refinement according to claim 2, characterized in that: The refining process of the sub-pixel refining unit includes the following steps: Step a: Using the pixel-level coordinates output by the YOLOv13 keypoint detection unit Centered on the color RGB window, the image is converted into a single-channel grayscale image, and a high-resolution N×N neighborhood window is extracted, retaining only the light and dark boundary information used for geometric fitting. Step b: Apply bilinear interpolation algorithm to upsample the N×N grayscale neighborhood window to generate a smooth, continuous sub-pixel grayscale field; Step c: On the interpolated continuous grayscale field, apply edge detection or adaptive thresholding to extract the contour point set of the pupil edge. ; Step d: Apply the least squares circular fitting algorithm to fit the set of contour points extracted in step c. Utilize the geometric prior knowledge that "the pupil is approximately circular" to calculate a robust center coordinate by minimizing the algebraic distance. This coordinate is the final subpixel-level pupil center.

5. The metaverse eye-tracking system based on improved YOLOv13 and sub-pixel refinement according to claim 1, characterized in that: The metaverse application scenario is a virtual cockpit, which uses the sub-pixel level pupil center coordinates output by the edge computing control terminal to achieve the following human-computer interaction functions: Dynamic gaze point rendering: Using the gaze point coordinates, the instantaneous area of ​​the user's gaze focus is rendered at full resolution, while the edge area of ​​the gaze is rendered at low resolution to optimize rendering performance and GPU load. Hands-free eye-tracking interaction: Users can select and confirm information on the instrument panel, central control screen buttons, or head-up display in the virtual cockpit by "gazing".

6. The metaverse eye-tracking system based on improved YOLOv13 and sub-pixel refinement according to claim 5, characterized in that: The "gaze-satiety" method refers to determining whether the gaze point remains in a certain area for more than a threshold time.

7. The metaverse eye-tracking system based on improved YOLOv13 and sub-pixel refinement according to claim 1, characterized in that: It also includes a personalized calibration module, which is activated when the user first uses it. The user focuses on several points in the metaverse scene that appear in a specific time sequence to calibrate the target. The module also automatically calculates a high-order polynomial or neural network model based on the pupil coordinate data pairs collected in real time, which serves as a personalized mapping function between the user's gaze vector and the input pupil coordinates.

8. A high-precision eye-tracking method for a metaverse virtual cockpit based on the system described in any one of claims 1 to 7, characterized in that: Includes the following steps: Step 1: Startup and Calibration. Start the smart glasses terminal and control terminal, and perform initialization calibration. Step 2: Acquisition and preprocessing. The ESP32 vision module is used to acquire the eye's color RGB image stream in real time and perform preprocessing. Step 3: ROI localization. The eye region is cropped from the preprocessed image by quickly locating the ROI unit in the eye image. Step 4: Coarse localization of YOLOv13 key points. The color RGB image of the eye ROI is input into the modified YOLOv13 key point detection unit to directly predict the pixel-level coordinates of the pupil center. ; Step 5: Sub-pixel refinement. The sub-pixel refinement unit is activated to refine the pixel-level coordinates of the pupil center. Refined calculations were performed to obtain high-precision pupil center coordinates. ; Step 6: Coordinate output and interaction, outputting the high-precision pupil center coordinates. Send directly to the metaverse virtual cockpit for foveated rendering or gaze interaction.

9. The high-precision eye-tracking method for the metaverse virtual cockpit according to claim 8, characterized in that: Step 5 specifically includes: Step 5.1, with Centered on a high-resolution ROI, extract neighborhood windows and convert them to grayscale images; Step 5.2: Upsample the grayscale image using bilinear interpolation to obtain the grayscale field; Step 5.3: Apply the Canny operator or Otsu's method to extract the edge contour point set of the pupil from the interpolated grayscale field. ; Step 5.4: Using the edge contour point set extracted in Step 5.3 as input, apply the least squares circle fitting algorithm to calculate the final pupil center coordinates. .

10. The application of a metaverse eye-tracking system based on improved YOLOv13 and sub-pixel refinement, characterized by: This system is applied to immersive interactive scenarios, including human-computer interaction, virtual reality, augmented reality, and the metaverse.

Images