Eye movement tracking method and system based on stacked hourglass network and homography transformation

Abstract

The invention provides an eye movement tracking method and system based on a stacked hourglass network and homography transformation. The method comprises the steps that a real-time eye image is collected; eye features are generated based on a stacked hourglass network model; calculating an instantaneous sight angle according to the eye features; mapping the sight line angle into a screen drop point coordinate by using a pre-trained homography transformation model; oneEuro time filtering is applied to the coordinates of the original screen; performing noise reduction on the smoothed coordinate sequence; and generating a KDE attention heat map. According to the invention, high-precision key point extraction of the eyes is realized through the stacked hourglass network, nonlinear mapping of the sight line and the drop point is realized through homography transformation, adaptive smoothing is realized through OneEuro filtering, and the naked eye tracking precision and robustness are remarkably improved.

Description

Technical Field

[0001] This disclosure provides an eye-tracking method, system, and apparatus based on stacked hourglass networks and homography transformation, specifically relating to an eye-tracking method and corresponding system for predicting gaze points using computer vision in a human-computer interaction environment. Background Technology

[0002] Eye-tracking technology aims to estimate a user's gaze point in real time without contact, and has been widely used in fields such as human-computer interaction, market analysis, medical diagnosis, driver assistance, and virtual reality (VR / AR).

[0003] Existing technical solutions are mainly divided into two categories: solutions based on dedicated hardware and naked-eye tracking solutions based on standard cameras. While the former offers high accuracy, its equipment is expensive, deployment is complex, and it requires users to wear specific devices or be in specific, fixed environments, making large-scale adoption difficult. On the other hand, naked-eye tracking solutions based on standard cameras have become a current research hotspot due to their low cost, lack of need for additional equipment, and ease of integration. However, in practical applications, existing naked-eye tracking technologies based on standard cameras suffer from limitations in accuracy and stability due to insufficient accuracy and robustness in extracting key eye points, limited fitting ability of the gaze mapping model, and difficulty in balancing jitter and latency in the original gaze signal.

[0004] Therefore, an eye-tracking method that can simultaneously solve the technical problems of inaccurate key point extraction, poor nonlinear mapping fitting ability, and difficulty in balancing gaze jitter and delay in existing naked-eye tracking technologies is urgently needed. Summary of the Invention

[0005] The purpose of this disclosure is to provide an eye-tracking method and system based on stacked hourglass networks and homography transformation, aiming to improve the problems of inaccurate key point extraction, poor nonlinear mapping fitting ability of gaze-to-screen gaze point, severe jitter of the original gaze point, and high latency introduced by filtering and smoothing in the prior art.

[0006] The eye-tracking method and system based on stacked hourglass networks and homography transformation proposed in this disclosure have the following specific steps:

[0007] An eye-tracking method based on stacked hourglass networks and homography transformation includes:

[0008] Acquire real-time eye images;

[0009] Based on the real-time eye images, corresponding eye features are generated using a pre-trained stacked hourglass network model;

[0010] The instantaneous gaze angle is obtained based on the described eye features;

[0011] Predict the coordinates of the original landing point of the gaze on the screen based on the instantaneous gaze angle;

[0012] The filter cutoff frequency is adjusted on the original screen coordinates to output a smoothed gaze point coordinate sequence;

[0013] Spatial clustering is performed on the smoothed gaze point coordinate sequence to obtain the denoised gaze point coordinates;

[0014] An attention heatmap is generated from the denoised coordinate distribution to complete eye tracking.

[0015] The step of predicting the original landing point coordinates of the line of sight on the screen based on the instantaneous line of sight angle includes:

[0016] Establish calibration data pairs between viewing angle and screen coordinates;

[0017] The homography transformation model is trained using the calibration data to obtain a homography transformation matrix that characterizes the projection relationship from the viewing angle space to the screen coordinate space, and the trained homography transformation matrix is ​​stored.

[0018] Based on the instantaneous line-of-sight angle, a projection transformation is performed on the instantaneous line-of-sight angle using the trained homography transformation matrix to predict the coordinates of the original landing point.

[0019] The establishment of calibration data pairs between the viewing angle and screen coordinates includes:

[0020] Display the calibration target point with known coordinates on the screen;

[0021] When the user is looking at the calibration target point, the viewing angle and the calibration target point are stored as a coordinate pair.

[0022] The step of training the homography transformation model using the calibration data includes:

[0023] A calibration data list is constructed using the calibration data.

[0024] Obtain the average viewing angle and all corresponding known screen coordinates from the calibration data list; and use the average viewing angle as the source point array and the known screen coordinates as the target point array.

[0025] Using the source point array and the target point array as input, a homography transformation matrix is ​​constructed to characterize the projection transformation relationship from the viewing angle space to the screen coordinate space;

[0026] The homography transformation matrix is ​​used as the parameter of the homography transformation model to obtain the homography transformation model.

[0027] The step of generating corresponding eye features based on the real-time eye image using a pre-trained stacked hourglass network model includes:

[0028] Based on the real-time eye images, a heatmap of key eye points is generated by the pre-trained stacked hourglass network model.

[0029] The heatmap of key eye points is converted into the coordinates of key points in the eye features;

[0030] The eyeball radius in the eye features is obtained by regression through a fully connected layer based on the coordinates of the key points.

[0031] The step of obtaining the instantaneous gaze angle based on the eye features includes:

[0032] Extract the iris center coordinates, eyeball center coordinates, and eyeball radius from the eye features;

[0033] The relative coordinates of the iris center and the relative coordinates of the eyeball center on the real-time eye image are obtained respectively.

[0034] Obtain the displacement vector between the relative coordinates of the iris center and the relative coordinates of the eyeball center;

[0035] The trigonometric function relationship between the displacement vector and the eyeball radius is obtained to get the pitch angle and yaw angle, which are used as the instantaneous line of sight angle.

[0036] The step of adjusting the filter cutoff frequency of the original screen coordinates to output a smoothed gaze point coordinate sequence includes:

[0037] Get the current time original coordinates And calculate the time interval between the current sample and the previous sample. ;

[0038] The current velocity estimate of the eye-tracking coordinates is calculated based on the first-order difference.

[0039] By fixing the cutoff frequency as The first-order low-pass filter is used for filtering to determine the filter coefficients of the low-pass filter. The smoothed speed is obtained after filtering. and ;

[0040] The cutoff frequency transverse component of the position filter is dynamically determined based on the smoothing velocity. Longitudinal component ,in The preset minimum cutoff frequency, This is the speed adjustment coefficient. and These are the absolute values ​​of the smoothing velocities in the horizontal and vertical directions, respectively.

[0041] The filter coefficients of the position filter are calculated based on the cutoff frequency, and the original coordinates are then subjected to a first-order low-pass filter using these coefficients to output smoothed gaze point coordinates; wherein, Represents the timestamp of the previous sample; Represents the original screen gaze coordinates at the current time t.

[0042] The step of spatially clustering the smoothed gaze point coordinate sequence to obtain the denoised gaze point coordinates includes:

[0043] The smoothed gaze coordinate sequence is processed using density-based noisy spatial clustering, including:

[0044] Set a coordinate density threshold;

[0045] Regions with a density higher than the coordinate density threshold in the smoothed gaze point coordinate sequence are identified as core clusters, representing effective gaze;

[0046] Sparse points in the smoothed gaze point coordinate sequence that do not belong to the core cluster are marked as outliers.

[0047] By filtering out any of the outliers using a mask, the coordinates of the gaze point after noise reduction are obtained.

[0048] The step of generating an attention heatmap from the denoised coordinate distribution includes:

[0049] A Gaussian kernel function is applied to the coordinates of any of the denoised gaze points.

[0050] By superimposing all the Gaussian kernel functions, the continuous probability density of the gaze point in screen space is obtained;

[0051] The continuous probability density is rendered as a visual heatmap.

[0052] The acquisition of real-time eye images includes:

[0053] Acquire real-time video frames;

[0054] Face detection is performed on the video frame to obtain face bounding boxes; and the face detection is executed in a separate asynchronous background thread to avoid blocking the main processing flow.

[0055] Based on the face detection results, the eye images are located and standardized.

[0056] The step of locating and standardizing the eye image based on the face detection results includes:

[0057] Locate key facial points within the face bounding box.

[0058] The rotation angle and center point of the eye region are calculated based on the key points, and affine transformation is applied to perform rotation correction and size normalization on the eye region; histogram equalization and pixel value normalization are then performed sequentially.

[0059] An eye-tracking system based on the eye-tracking method using stacked hourglass networks and homography transformation as described above includes:

[0060] The image acquisition module is used to acquire real-time eye images;

[0061] The image preprocessing module interacts with the image acquisition module to perform face detection and eye image standardization preprocessing based on the real-time eye images.

[0062] The gaze angle calculation module interacts with the image preprocessing module to generate eye features through a pre-trained stacked hourglass network model and obtain the instantaneous gaze angle based on the eye features.

[0063] The first processing module interacts with the line-of-sight angle calculation module to predict the original landing point coordinates of the line of sight on the screen based on the instantaneous line-of-sight angle.

[0064] The second processing module interacts with the first processing module to adjust the filtering cutoff frequency of the original screen coordinates in order to output a smoothed gaze point coordinate sequence.

[0065] The gaze data analysis module interacts with the second processing module to perform spatial clustering on the smoothed gaze point coordinate sequence to obtain the noise-reduced gaze point coordinates.

[0066] The output module interacts with the gaze data analysis module to generate an attention heatmap from the denoised coordinate distribution, thereby completing eye tracking.

[0067] The first processing module includes:

[0068] The user calibration module interacts with the viewing angle calculation module to call the viewing angle calculation module and establish a calibration data pair between the viewing angle and the screen coordinates.

[0069] The mapping model selection and system training module interacts with the user calibration module to select the corresponding algorithm to train the homography transformation model according to the user's instructions, obtain the trained homography transformation model, and use the trained homography transformation model to map the viewing angle to the screen landing point coordinates.

[0070] The second processing module includes:

[0071] The filtering selection module interacts with the mapping model selection and system training module in the first processing module to adjust the filtering cutoff frequency of the original screen coordinates.

[0072] The real-time mapping module interacts with the filtering and selection module to perform projection transformation on the instantaneous line-of-sight angle and predict the coordinates of the original landing point.

[0073] Beneficial effects:

[0074] The beneficial effects of this disclosure include at least the following:

[0075] The eye-tracking method based on stacked hourglass networks and homography transformation disclosed herein firstly improves the accuracy of eye keypoint localization and extraction by utilizing the multi-scale feature fusion and progressive refinement capabilities of the stacked hourglass network model, thus addressing the inaccuracy of eye keypoint extraction in existing methods. Secondly, through a trained and stored homography transformation model, it effectively fits the nonlinear projection relationship between the gaze angle plane and the screen coordinate plane, and accurately fits distortions caused by individual physiological differences and device layout based on user-specific calibration data, significantly improving the prediction accuracy of gaze points and addressing the poor nonlinear fitting capability of existing methods. Thirdly, this disclosure employs a OneEuro filter, adaptively adjusting the cutoff frequency according to the rate of change of the original coordinates, achieving a high-efficiency balance between smoothness and real-time performance, overcoming the difficulty of simultaneously addressing jitter and latency in existing methods. Finally, the DBSCAN density denoising and KDE heatmap generation introduced after tracking automatically remove outliers and visualize gaze hotspots, enhancing the robustness and intuitiveness of eye-tracking data analysis and facilitating data visualization analysis. Attached Figure Description

[0076] Figure 1 This is a flowchart illustrating the steps of the eye-tracking method based on stacked hourglass networks and homography transformation as described in this disclosure;

[0077] Figure 2 It is a flowchart of standardized image conditions;

[0078] Figure 3 This is an architecture diagram of a stacked neural network;

[0079] Figure 4 This is a schematic diagram illustrating the working principle of the homography transformation model;

[0080] Figure 5 This is an application diagram of the homography transformation model;

[0081] Figure 6This is a functional block diagram of the eye-tracking data processing system according to an embodiment of the present invention. Detailed Implementation

[0082] Various embodiments of this disclosure will be described more fully below. This disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of this disclosure to the specific embodiments disclosed herein, but rather this disclosure should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of this disclosure.

[0083] In the following, the terms “comprising” or “may include”, which may be used in various embodiments of this disclosure, indicate the presence of the disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. Furthermore, as used in various embodiments of this disclosure, the terms “comprising,” “having,” and their cognates are intended only to indicate a particular feature, number, step, operation, element, component, or combination of the foregoing, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or the possibility of adding one or more combinations of features, numbers, steps, operations, elements, components, or combinations of the foregoing.

[0084] The terms used in the various embodiments of this disclosure (such as "first," "second," etc.) may modify various components in the various embodiments, but do not limit the corresponding components. For example, the above terms do not limit the order and / or importance of the components. The above terms are only used for the purpose of distinguishing one component from others. For example, a first user device and a second user device refer to different user devices, although both are user devices. For example, a first component may be referred to as a second component without departing from the scope of the various embodiments of this disclosure, and similarly, a second component may also be referred to as a first component.

[0085] It should be noted that if a description is made of "connecting" one component to another, then the first component can be directly connected to the second component, and a third component can be "connected" between the first and second components. Conversely, when a component is "directly connected" to another component, it can be understood that there is no third component between the first and second components.

[0086] The term "user" as used in various embodiments of this disclosure may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0087] The terminology used in the various embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the various embodiments of this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this disclosure pertain. Terms (such as those defined in a generally used dictionary) are to be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and are not to be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this disclosure.

[0088] Existing naked-eye tracking technologies based on standard cameras have the following drawbacks in practical applications:

[0089] First, the accuracy and robustness of eye keypoint extraction are insufficient. Traditional appearance-based computer vision methods are highly sensitive to factors such as lighting conditions, subtle changes in the user's head posture, whether glasses are worn, and eye occlusion when extracting keypoints such as the pupil center and corners of the eyes. These interfering factors can cause deviations and jitters in keypoint localization, which are the source of error for all subsequent calculations and directly limit the final accuracy of eye tracking.

[0090] Secondly, the fitting ability of gaze mapping models is limited. The relationship between eye movements and the user's 2D gaze point on the screen is a complex nonlinear geometric relationship. Most existing technologies use simple linear regression or polynomial fitting for calibration. The drawback is that this linear model makes overly simplistic assumptions and cannot accurately describe the nonlinear distortions caused by perspective transformations and individual physiological differences in the eyes. Especially in the screen edge area, the mapping error will increase significantly, making it impossible for users to accurately measure the specific gaze points at the four corners of the screen using existing methods.

[0091] Third, balancing jitter and latency in the original gaze point signal is difficult. Whether using traditional methods or emerging deep learning methods, the output of the original gaze point coordinates inevitably contains high-frequency noise. This noise originates from both the physiological tremors of the human eye and minor errors in sensor noise and algorithm models. This noise manifests on the screen as severe cursor jitter. To eliminate jitter, existing techniques often employ Kalman filtering or simple moving average filters. However, these filters have an inherent contradiction: excessive smoothing can eliminate jitter, but this introduces significant signal latency; conversely, to ensure real-time response, jitter cannot be effectively suppressed.

[0092] Based on this, the present disclosure provides the following embodiments, which are intended to solve the above-mentioned technical problems. Specific Implementation Example 1:

[0094] This disclosure provides an embodiment:

[0095] The core idea of ​​this disclosure includes the following steps:

[0096] S10: Acquire real-time eye images;

[0097] S20: Generate eye features using a pre-trained stacked hourglass network model;

[0098] S30: Calculate the instantaneous line of sight angle based on the described eye features;

[0099] S40-S50: Training the homography transformation model;

[0100] S60: Loads the homography transformation model trained by S40 and S50, and predicts the original landing point coordinates of the gaze on the screen based on the homography transformation model;

[0101] S70: Apply OneEuro temporal filtering to the original screen coordinates. The filtering includes adaptively adjusting the filter cutoff frequency based on the temporal derivative of the original screen coordinates to output smoothed gaze point coordinates.

[0102] S80: Apply DBSCAN to the smoothed gaze point coordinate sequence;

[0103] S90: Apply KDE to the denoised coordinate distribution to generate an attention heatmap.

[0104] This embodiment provides an eye-tracking method based on stacked hourglass networks and homography transformation, such as... Figure 1 The diagram shown is a flowchart of the overall method steps in this embodiment. This flowchart illustrates the complete data processing chain from raw video input to final heatmap visualization, mainly comprising three parts: a common processing stage, a model training stage, and a real-time tracking stage.

[0105] In the common processing stage, as described in S10-S30 above, real-time eye images are first acquired through step S10, which includes acquiring video frames, performing face detection, and the crucial step S13 standardizing the eye images; then, step S20 inputs the processed standard eye image into a pre-trained stacked hourglass network, which generates keypoint coordinates and eyeball radius through multi-scale feature fusion; next, step S30 calculates the instantaneous gaze angle based on these features.

[0106] During the model training and storage phase, as described in S40-S50 above, step S40 establishes calibration data pairs, which involves collecting the user's gaze angle when looking at a specific point on the screen. and compare it with the known screen coordinates. Pairing. Then, step S50 is performed to train and store the homography transformation model, and the best-fit homography matrix is ​​calculated using the collected data. The model parameters are then stored for later use.

[0107] During the real-time tracking phase, as described in steps S60-S90 above, the trained homography matrix is ​​loaded, and step S60 is executed to predict the original gaze point coordinates, projecting the current instantaneous gaze angle onto the screen space. To address the jitter issue in the original signal, step S70 applies OneEuro temporal filtering, adaptively adjusting the smoothness based on the signal's rate of change to output stable gaze point coordinates. Finally, for data analysis, step S80 applies DBSCAN density denoising to remove outliers, and step S90 generates a KDE attention heatmap to visualize the user's gaze distribution.

[0108] Preferably, step S10, acquiring real-time eye images, specifically includes: S11 acquiring real-time video frames; S12 performing face detection on the video frames; and S13 locating and standardizing eye images based on the face detection results.

[0109] Preferably, step S12 includes: performing the face detection in a separate asynchronous background thread to avoid blocking the main processing flow.

[0110] Preferably, step S13 specifically includes: locating facial key points within the face bounding box obtained in S12; calculating the rotation angle and center point of the eye region based on the key points, and applying affine transformation to perform rotation correction and size normalization on the eye region; and sequentially performing histogram equalization and pixel value normalization.

[0111] Preferably, step S20 specifically includes: S21 generating a key point heatmap from the stacked hourglass network model; S22 applying the Soft-argmax function to convert the heatmap into key point coordinates; and S23 regressing the eyeball radius based on the key point coordinates.

[0112] Preferably, step S30 specifically includes: extracting the iris center coordinates, eyeball center coordinates, and eyeball radius from the eye features; and calculating the pitch angle and yaw angle using the displacement vector between the iris center coordinates and the eyeball center coordinates on the image, relative to the trigonometric function relationship of the eyeball radius, as the instantaneous line of sight angle.

[0113] Preferably, step S60 specifically includes: S61 continuously repeating steps S10 to S30 to obtain the user's current instantaneous viewing angle; S62 loading the homography transformation model and performing a projection transformation on the instantaneous viewing angle; S63 calculating and outputting the original screen coordinates.

[0114] Preferably, step S70 specifically includes: using adaptive adjustment of the filter cutoff frequency; when the signal coordinates change slowly, the cutoff frequency is reduced to perform strong smoothing; when the signal coordinates change rapidly, the cutoff frequency is increased to reduce the smoothing intensity and reduce signal delay.

[0115] Preferably, step S80 specifically includes: identifying regions with a density higher than a preset threshold in the smoothed gaze point coordinate sequence as core clusters, representing effective gaze; marking sparse points in the sequence that do not belong to the core clusters as outliers; and removing all outliers by masking to obtain the noise-reduced gaze point coordinates.

[0116] Preferably, step S90 specifically includes: S91 placing a Gaussian kernel function on each gaze point coordinate after noise reduction; S92 superimposing all the Gaussian kernel functions to calculate the continuous probability density of the gaze point in the screen space; and S93 rendering the continuous probability density into a visualized heatmap.

[0117] To provide a more detailed explanation of the eye-tracking method based on stacked hourglass networks and homography transformation provided in Specific Embodiment 1, this embodiment will provide a detailed implementation process of the above steps.

[0118] The following is a detailed explanation of each step in the above method:

[0119] S10 Acquiring Real-Time Eye Images: In this embodiment, the purpose of step S10, acquiring real-time eye images, is to convert the raw video stream obtained by an image acquisition device, such as a standard camera, into a standardized eye diagram that meets the input requirements of the stacked hourglass network model in the subsequent step S20, through a series of preprocessing steps. Specifically, this includes the following steps S11-S13:

[0120] S11: Acquire real-time video frames: Started through the data source interface module, calling the standard camera to capture the real-time video stream in a loop at a set frame rate of 30 FPS to obtain the original BGR format video frames.

[0121] S12: Perform face detection: such as Figure 2 In this embodiment, the BGR video frames acquired in S11 are converted into grayscale images, and optionally scaled to improve processing speed. Then, a face detector provided by the dlib library is used to detect faces on the grayscale image, and the bounding boxes of the face regions are output. Figure 2 The document details how to transform the "raw eye image" output in step S12 into a "standardized eye image" that meets the input requirements of the deep learning model in step S20.

[0122] Furthermore, to improve the smoothness of the real-time system, the face detection in step S12 will be executed in an independent asynchronous background thread. The main processing flow will only retrieve the latest face detection results from the queue, thereby avoiding main process blocking and screen lag caused by face detection time consumption.

[0123] S13: Standardized Eye Image: After obtaining the face bounding box in step S12, specifically, dlib is used to accurately locate five key points on the face, namely the four corners of the eyes and the tip of the nose. Subsequently, the center point, width, and rotation angle of the eye region are calculated based on the coordinates of the key points, and the eye region is segmented and transformed from the original grayscale image. Affine transformation is then applied to the eye region to perform combined translation, rotation correction, scaling matrix, and size normalization to an image of 36×60 pixels. Finally, histogram equalization is performed on the standard eye image to enhance contrast, and the pixel values ​​are normalized to the range of [-1.0, 1.0], and the channel dimension is increased to serve as the tensor for the final input of the stacked neural network in step S20.

[0124] In summary, this step is as follows Figure 2 As shown, the system first receives the raw eye image, including the eye region, output from the face detection step. This image typically suffers from head pose distortion and inconsistent scale. Next, the system uses the located facial key points to calculate the rotation angle and center position, and performs an affine transformation. This transformation simultaneously performs two tasks: rotation correction, correcting the tilted eyes to a horizontal orientation; and size normalization, scaling the image to a fixed 36×60 pixel size.

[0125] Subsequently, histogram equalization is performed on the corrected image to enhance image contrast, highlight the edge features of the iris and pupil, and reduce the impact of illumination variations. Finally, a normalization operation is performed, linearly mapping the pixel intensity values ​​of the image from [0, 255] to a floating-point range of [-1.0, 1.0]. The normalized eye image output after the above processing is the standard input tensor of the stacked hourglass network in S20.

[0126] S20: Generate eye features using a stacked hourglass network model. In this embodiment, a stacked hourglass network model is constructed for high-precision gaze estimation. This network is specifically designed to extract high-precision geometric features from low-resolution eye images, and specifically includes:

[0127] S21: Generating keypoint heatmaps from the stacked hourglass network model: In this embodiment, two hourglass modules are preferably stacked in series. A symmetrical encoder-decoder structure is adopted for each hourglass module. 1. The resolution is reduced by progressively downsampling through pooling layers to capture a wide range of high-level semantic features; 2. The resolution is restored by progressively upsampling through bilinear interpolation; 3. Multi-scale feature fusion is achieved by fusing feature maps from the encoder path at the corresponding resolution through skip connections at each upsampling stage of the decoder path. Subsequent modules receive the preliminary eye features and heatmaps output from the previous module as input, and perform iterative refinement and error correction based on the previous module to generate the final probabilistic heatmaps of 18 eye keypoints.

[0128] S22: Apply Soft-argmax to convert the heatmap into eye key point coordinates: In this embodiment, the model uses Soft-argmax to create a normalized reference coordinate grid for each pixel position of the heatmap in step S21, converting the heatmap into a probability distribution. Then, by calculating the weighted average of the probability and the coordinate grid, the expected positions of 18 differentiable eye key points in the normalized coordinate system are obtained, and finally converted back to pixel coordinates.

[0129] S23: Regressing the eyeball radius based on the keypoint coordinates: In this embodiment, the 18 keypoint coordinate tensors (N, 18, 2) output in step S22 are flattened and input into three fully connected layers, ultimately regressing a single scalar value to obtain the eyeball radius. Thus, step S20 obtains eye features containing the eye keypoint coordinates described in step S22 and the eyeball radius described in step S23.

[0130] Please refer to the specific network structure. Figure 3 . Figure 3 This is a diagram of the overall architecture of the stacked neural network in this embodiment. The network mainly consists of four parts: a preprocessing module, a stacked hourglass body, an intermediate supervision module, and a prediction output head.

[0131] 1. Preprocessing module:

[0132] like Figure 3 As shown on the far left, the network first receives a normalized eye image with a tensor of 36×60×1 (H×W×1) output from step S13. The input data first passes through a convolutional layer with a kernel size of 7×7 and a stride of 1, combined with batch normalization (BN) and ReLU activation functions, to expand the number of image channels to 32 channels and extract preliminary shallow texture features. Subsequently, the data flows through two cascaded residual modules. Each residual module contains three convolutional layers (1×1, 3×3, 1×1) and a skip connection, used to deepen the network without losing information and extract more abstract semantic features.

[0133] 2. Stacked hourglass body:

[0134] like Figure 3 As shown in the middle, this embodiment uses two cascaded hourglass modules. Each hourglass module has a symmetrical "U" shaped structure, which aims to capture multi-scale spatial information. Its specific structure is as follows: (1) Downsampling path (left half): The feature map resolution is gradually halved through max pooling layer until the lowest resolution (4×4) is reached to capture global context information. (2) Upsampling path (right half): The feature map resolution is gradually restored through bilinear interpolation. (3) Skip connection (middle horizontal line): At each resolution level, the features of the downsampling path are processed by the residual module and then directly added element-wise with the corresponding features of the upsampling path.

[0135] 3. Intermediate Supervision Module:

[0136] like Figure 3 As shown at the connection point of the two hourglass modules, to alleviate gradient vanishing and improve training efficiency, this invention introduces an intermediate supervision mechanism. The feature map output by the first hourglass module is processed by a 1×1 convolution to generate an intermediate heatmap, which is used to calculate the intermediate loss. Simultaneously, this intermediate heatmap, after remapping, is fused with the input and output features of the first hourglass module through a three-way summation. The fused result serves as the input to the second hourglass module.

[0137] 4. Predictive output header:

[0138] like Figure 3 As shown on the far right, the second hourglass module outputs a final keypoint probability heatmap containing 18 channels. This heatmap is not directly output; instead, it is converted into precise keypoint coordinates (x, y) through a soft-argmax layer. Finally, these coordinates are flattened and fed into a set of fully connected layers to regress and predict the scalar form of the eyeball radius.

[0139] S30: Calculate the instantaneous line of sight angle based on the described eye features.

[0140] In this embodiment, specifically:

[0141] S31: In the eye features described in this embodiment, the coordinates of the iris center are extracted. Coordinates of the center of the eyeball And obtain the eyeball radius in step S23. .

[0142] S32: In this embodiment, the pitch angle is calculated using geometric trigonometric relationships and the following formula. and yaw angle This ultimately outputs the line-of-sight angle G:

[0143] ;

[0144] S40: Establish calibration data pairs between viewing angle and screen coordinates.

[0145] This embodiment specifically includes:

[0146] S41: In this embodiment, in full-screen mode, 30 target points are selected sequentially from a pre-generated 5×6 calibration point grid and visible marks are drawn on the screen;

[0147] S42: This embodiment prompts the user to look at the point and waits for the user to confirm by pressing a key. After the user confirms, the complete process of steps S10 to S30 is continuously called within a fixed preset time of 2 seconds to obtain the instantaneous gaze angle sample of the current screen. .

[0148] S43: After the data acquisition is completed, this embodiment analyzes the n instantaneous viewpoint angle samples of the current image acquired in step S42. Where k=1, ..., n, the arithmetic mean is calculated to improve stability and obtain the average viewing angle. That is, P eye :

[0149]

[0150] Then, the average line of sight angle The known screen coordinates P of the target point screen (x, y) as a data pair ( , P screen ), and store it in the calibration data list.

[0151] S44: Repeat steps S41 to S43 until all calibration target points have been traversed.

[0152] Please refer to the attached image for details. Figure 4 .exist Figure 4 The paper provides the working principle of the homography transformation model, which mainly includes two steps: calibration data acquisition and matrix solving.

[0153] Step 1: Calibrate data acquisition, i.e., step S40 in this embodiment:

[0154] This embodiment uses a 5×6 grid calibration scheme to cover the main viewing area of ​​the screen.

[0155] First, the system generates 30 uniformly distributed calibration target points in full-screen mode.

[0156] The system displays a target point on the screen in sequence as the user looks at it, prompting the user to look at it. Once the user confirms the view, the system collects and calculates the current instantaneous gaze angle at high frequency within a 2-second window.

[0157] To eliminate the jitter error caused by microscanning, an arithmetic mean was calculated for all sampling points within the time window to obtain a stable average viewing angle. .

[0158] like Figure 4 As shown, this average viewing angle is related to the known screen coordinates of the current target point. (x, y) are bound together to form a mapping data pair. This is then recorded and added to the calibration list. This process is repeated until all 30 calibration points have been traversed. These two data points correspond one-to-one in the index list.

[0159] Step 2: Matrix solution and model establishment, i.e., step S52;

[0160] This embodiment utilizes The source set and Given a set of target points, train an optimal 3×3 transformation matrix. .like Figure 5 The application diagram of the homography transformation model is shown. This matrix describes the perspective projection relationship between the plane containing the viewing angle and the screen plane. The goal is to minimize the reprojection error.

[0161] ;

[0162] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and preferred embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0163] S50: Train and store a homography transformation mapping model, which in this embodiment specifically includes:

[0164] S51: In this embodiment, all average line-of-sight angles are separated from the calibration data list in S40 to form a source point set. And separate the corresponding known screen coordinates to form a set of target points. .

[0165] S52: As Figure 4 The data collected in step S51 above is used as training data to calculate a 3×3 homography transformation matrix. , making satisfy This matrix achieves the optimal description of the projection transformation relationship from the source plane to the target plane. This refers to the homography transformation model obtained through training.

[0166] S53: As Figure 5 After step S52 is completed, the homography transformation matrix is ​​assigned a value and stored in the eye-tracking system. Subsequent real-time tracking steps can load and apply the matrix in real time to perform the projection transformation.

[0167] S60: Perform real-time tracking and predict the original screen coordinates. In this embodiment, this specifically includes:

[0168] S61: This embodiment enters a real-time loop, continuously calling the complete process from steps S10 to S30 to obtain the user's current instantaneous viewing angle. .

[0169] S62: In this embodiment, the homography mapping model trained in S50 is called, and the homography matrix stored in step S53 is loaded. And through the projection transformation function, for Perform matrix transformation operations to calculate and output the original coordinates. :

[0170] ;

[0171] S70: In this embodiment, a OneEuro filter is applied to the original coordinates to output smoothed gaze point coordinates. Specifically, this includes:

[0172] S71: Collect the original coordinates of the current time t output in step S60. Calculate the time interval between the current sample and the previous sample. .

[0173] S72: Velocity estimation of eye-tracking coordinates based on first-order difference calculation, where the lateral velocity is... The longitudinal velocity is .

[0174] S73: The above velocity estimates are respectively tested through a fixed cutoff frequency of... A first-order low-pass filter (set to 1Hz) is used for filtering, and the filter coefficients of the low-pass filter are... From the formula The smoothed speed is obtained after filtering. and .

[0175] S74: Dynamically determine the cutoff frequency of the position filter using an adaptive method. Specifically, the horizontal component Longitudinal component ,in The minimum cutoff frequency is set to 0.2. The speed adjustment coefficient is set to 0.001.

[0176] S75: Calculate the filter coefficients of the final position filter based on the adaptive cutoff frequency. The horizontal component is The vertical component is .

[0177] S76: Apply a first-order low-pass filter to the current original eye movement position using the above filtering coefficients to obtain the filtered position. and The two are then used as the output of eye-tracking data.

[0178] Preferably, in step S74, the cutoff frequency is small when the coordinate change is slow. It is also low, the filter performs strong smoothing to eliminate jitter; when the coordinates change rapidly, the cutoff frequency is high. As a result, the filter reduces smoothness to decrease latency and lag.

[0179] S80: After real-time tracking ends, the DBSCAN algorithm is applied to perform density denoising on the gaze point coordinate sequence. In this embodiment, the specific content is as follows:

[0180] The DBSCAN algorithm identifies regions with a density higher than a preset threshold of 30 among the gaze points recorded in step S70 as core clusters, while marking sparse points as outliers. This step filters the data using a mask to remove all outliers, obtaining the denoised gaze point coordinates.

[0181] S90: Generate an attention heatmap using the KDE algorithm. In this embodiment, the specific content is as follows:

[0182] In this embodiment, the KDE algorithm is used to place a Gaussian kernel function on each clean gaze point output in step S80, and then all kernel functions are superimposed to calculate the continuous probability density of the gaze point in screen space; then the density map generated by KDE is rendered as a heatmap, which is preferably superimposed on a specified scene image.

[0183] In summary, the eye-tracking method described in this embodiment acquires eye features through the high-precision stacked hourglass network model described in step S20, achieves accurate mapping by combining the homography transformation with individualized calibration in step S50, and achieves an excellent balance between smooth jitter and low latency response by utilizing the adaptive OneEuro filter in step S70. Finally, robust offline data analysis capabilities are provided through DBSCAN in step S80 and KDE in step S90, thus solving the problems in the prior art. Specific Implementation Example 2:

[0185] This disclosure also provides an embodiment:

[0186] like Figure 6 An eye-tracking system based on the eye-tracking method based on stacked hourglass networks and homography transformation as described in Specific Embodiment 1 includes: an image acquisition module 100, an image preprocessing module 200, a gaze angle calculation module 300, a first processing module 400, a second processing module 500, a gaze data analysis module 600, and an output module 700; wherein, the image acquisition module 100 is used to acquire real-time eye images; the image preprocessing module 200 interacts with the image acquisition module 100 to perform face detection and eye image standardization preprocessing based on the real-time eye images; the gaze angle calculation module 300 interacts with the image preprocessing module 200 to generate eye features through a pre-trained stacked hourglass network model and, based on... The instantaneous gaze angle is obtained from the eye features; the first processing module 400 interacts with the gaze angle calculation module 300 to predict the original landing point coordinates of the gaze on the screen based on the instantaneous gaze angle; the second processing module 500 interacts with the first processing module 400 to adjust the filtering cutoff frequency of the original screen coordinates to output a smoothed gaze point coordinate sequence; the gaze data analysis module 600 interacts with the second processing module 500 to perform spatial clustering on the smoothed gaze point coordinate sequence to obtain denoised gaze point coordinates; the output module 700 interacts with the gaze data analysis module 600 to generate an attention heatmap from the denoised coordinate distribution to complete eye tracking.

[0187] Furthermore, the first processing module 400 includes: a user calibration module 401 and a mapping model selection and system training module 402; wherein, the user calibration module 401 interacts with the viewing angle calculation module 300 to call the viewing angle calculation module 300 and establish a calibration data pair between the viewing angle and the screen coordinates; the mapping model selection and system training module 402 interacts with the user calibration module 401 to select the corresponding algorithm to train the homography transformation model according to the user instruction, obtain the trained homography transformation model, and use the trained homography transformation model to map the viewing angle to the screen landing point coordinates.

[0188] Furthermore, the second processing module 500 includes a filtering selection module 501 and a real-time mapping module 502; wherein, the filtering selection module 501 interacts with the mapping model selection and system training module 402 in the first processing module 400 to adjust the filtering cutoff frequency of the original screen coordinates; the real-time mapping module 502 interacts with the filtering selection module 501 to perform projection transformation on the instantaneous viewing angle and predict the original landing point position coordinates.

[0189] It should be noted that:

[0190] The image acquisition module 100 can acquire raw video streams from various data sources such as real-time webcams or local video files; the image preprocessing module 200 can call real-time camera data and read local video files; in this embodiment, the mapping model selection and system training module 402 has built-in multiple optional mapping model algorithms and training data, which users can select from the built-in options through the drop-down menu of the graphical user interface; at the same time, in this embodiment, the filtering selection module 501 provides multi-mode selection. This module has built-in multiple optional temporal filter algorithms, which users can select from the following options through the drop-down menu of the GUI to load the trained mapping model and perform prediction of the raw screen coordinates.

[0191] Alternatively, all modules involved in this embodiment can be encapsulated within a graphical user interface (GUI) module, which can be built using the tkinter library. This GUI module provides a parameter configuration interface and task execution control. To ensure interface responsiveness, this module is implemented using the threading and queue libraries, placing time-consuming tasks into background threads for execution and safely transferring logs, progress, and other status information, as well as the final results, between the main thread and background threads via a queue. After the task is completed, this module is also responsible for loading the final generated analysis graph and previewing it on the GUI page. Specific Implementation Example 3:

[0193] This disclosure also provides an embodiment:

[0194] An electronic device for eye tracking includes: a storage medium and a processing unit; wherein the storage medium is used to store a computer program; the processing unit exchanges data with the storage medium and is used to execute the computer program during eye tracking to perform the steps of the eye tracking method based on stacked hourglass networks and homography transformation as described in Specific Embodiment 1.

[0195] The CPU described above can perform various appropriate actions and processes according to the program stored in the storage medium. The electronic device also includes the following peripherals, including input sections such as a keyboard and mouse, and output sections such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker.

[0196] This disclosure also provides an embodiment:

[0197] A readable storage medium: the readable storage medium stores a computer program; when the computer program is run, it executes the steps of the eye-tracking method based on stacked hourglass networks and homography transformation as described in Specific Embodiment 1.

[0198] In this embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, the computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0199] The above disclosures only cover a few specific implementation scenarios. However, this disclosure is not limited to these, and any variations that can be conceived by those skilled in the art should fall within the protection scope of this disclosure. The serial numbers in this disclosure are for descriptive purposes only and do not represent the superiority or inferiority of the implementation scenarios.

Claims

1. An eye tracking method based on a stacked hourglass network and homography transformation, characterized in that, The method comprises the following steps: Collecting real-time eye images; Generating corresponding eye features through a pre-trained stacked hourglass network model according to the real-time eye images; Obtaining an instantaneous line-of-sight angle according to the eye features; Predicting original landing position coordinates of the line-of-sight on a screen according to the instantaneous line-of-sight angle; Adjusting a filter cutoff frequency on the original screen coordinates to output a smoothed gaze point coordinate sequence; Performing spatial clustering on the smoothed gaze point coordinate sequence to obtain a denoised gaze point coordinate; Generating an attention heat map for the denoised coordinate distribution to complete eye movement tracking.

2. The eye tracking method based on stacked hourglass network and homography transformation according to claim 1, wherein, The step of predicting original landing position coordinates of the line-of-sight on a screen according to the instantaneous line-of-sight angle comprises the following steps: Establishing a calibration data pair of line-of-sight angle and screen coordinates; Training a homography transformation model using the calibration data pair to obtain a homography transformation matrix for representing a projection relationship from a line-of-sight angle space to a screen coordinate space, and storing the trained homography transformation matrix; Performing projection transformation on the instantaneous line-of-sight angle using the trained homography transformation matrix to predict the original landing position coordinates. The step of establishing a calibration data pair of line-of-sight angle and screen coordinates comprises the following steps: Displaying a calibration target point with known coordinates on the screen; Storing the line-of-sight angle and the calibration target point as a coordinate pair when a user gazes at the calibration target point.

3. The eye tracking method based on stacked hourglass network and homography transformation according to claim 2, characterized in that, The step of training a homography transformation model using the calibration data pair comprises the following steps: Constructing a calibration data list through the calibration data pair; Obtaining an average line-of-sight angle and all corresponding known screen coordinates from the calibration data list; and taking the average line-of-sight angle as a source point array and the known screen coordinates as a target point array; Taking the source point array and the target point array as inputs to construct a homography transformation matrix for representing a projection transformation relationship from a line-of-sight angle space to a screen coordinate space; Taking the homography transformation matrix as a parameter of the homography transformation model to obtain the homography transformation model.

4. The eye tracking method based on stacked hourglass network and homography transformation of claim 1, wherein, The step of generating corresponding eye features through a pre-trained stacked hourglass network model according to the real-time eye images comprises the following steps: Generating an eye key point heat map from the pre-trained stacked hourglass network model according to the real-time eye images; Converting the eye key point heat map into key point coordinates in the eye features; Obtaining an eyeball radius in the eye features through a fully connected layer regression according to the key point coordinates.

5. The eye tracking method based on stacked hourglass network and homography transformation of claim 1, wherein, The step of obtaining an instantaneous line-of-sight angle according to the eye features comprises the following steps: Extracting an iris center coordinate, an eyeball center coordinate and the eyeball radius from the eye features; Respectively obtaining an iris center relative coordinate and an eyeball center relative coordinate of the iris center and the eyeball center on the real-time eye images; Obtaining a displacement vector between the iris center relative coordinate and the eyeball center relative coordinate; Obtaining a trigonometric function relationship of the displacement vector relative to the eyeball radius to obtain a pitch angle and a yaw angle as the instantaneous line-of-sight angle.

6. The eye tracking method based on stacked hourglass network and homography transformation of claim 1, wherein, The step of adjusting a filter cutoff frequency on the original screen coordinates to output a smoothed gaze point coordinate sequence comprises the following steps: Get the current time original coordinates And calculate the time interval between the current sample and the previous sample. ; Calculating a current speed estimation value of eye movement coordinates based on a first-order difference; The low-pass filter is a first-order low-pass filter with a fixed cut-off frequency of The filter coefficient of the low-pass filter is determined , and the smoothed speed in the horizontal direction and the vertical direction is obtained respectively after filtering and ; determining a cut-off frequency lateral component of a position filter dynamically based on the smoothed velocity , a longitudinal component wherein is a pre-set minimum cut-off frequency, is a velocity adjustment coefficient, and are absolute values of the smoothed velocity in lateral and longitudinal directions, respectively; According to the cutoff frequency, filter coefficients of the position filter are calculated, and the original coordinates are first-order low-pass filtered by using the filter coefficients to output smoothed gaze point coordinates; wherein, represents a timestamp of the previous sample; represents a current time t of the original screen gaze point coordinates.

7. The eye tracking method based on stacked hourglass network and homography transformation of claim 1, wherein, The spatial clustering of the smoothed gaze point coordinate sequence is performed to obtain noise-reduced gaze point coordinates, including: Adopting a noise application spatial clustering based on density to process the smoothed gaze point coordinate sequence includes: A coordinate density threshold is set; Regions in the smoothed gaze point coordinate sequence with a density higher than the coordinate density threshold are identified as core clusters, representing effective gazes; Sparse points in the smoothed gaze point coordinate sequence that do not belong to the core clusters are marked as outliers; Any outlier is removed by mask screening to obtain noise-reduced gaze point coordinates.

8. The eye tracking method based on stacked hourglass network and homography transformation of claim 1, wherein, The attention heat map is generated according to the noise-reduced coordinate distribution, including: A Gaussian kernel function is placed for any noise-reduced gaze point coordinate; All the Gaussian kernel functions are superimposed to obtain the continuous probability density of the gaze point in the screen space; The continuous probability density is rendered into a visual heat map.

9. An eye tracking system based on the eye tracking method based on a stacked hourglass network and homography transformation according to any one of claims 1-8, characterized in that, It includes: An image acquisition module is configured to acquire real-time eye images; An image preprocessing module is configured to interact with the image acquisition module to perform face detection and eye image standardization preprocessing according to the real-time eye images; A line of sight angle calculation module is configured to interact with the image preprocessing module to generate eye features through a pre-trained stacked hourglass network model and to obtain an instantaneous line of sight angle according to the eye features; A first processing module is configured to interact with the line of sight angle calculation module to predict the original landing position coordinate of the line of sight on the screen according to the instantaneous line of sight angle; A second processing module is configured to interact with the first processing module to adjust the filter cutoff frequency of the original screen coordinate to output a smoothed gaze point coordinate sequence; A gaze data analysis module is configured to interact with the second processing module to perform spatial clustering on the smoothed gaze point coordinate sequence to obtain noise-reduced gaze point coordinates; An output module is configured to interact with the gaze data analysis module to generate an attention heat map according to the noise-reduced coordinate distribution to complete eye movement tracking.

10. The eye tracking system of claim 9, wherein, The first processing module includes: A user calibration module is configured to interact with the line of sight angle calculation module to call the line of sight angle calculation module to establish a calibration data pair of the line of sight angle and the screen coordinate; A mapping model selection and system training module is configured to interact with the user calibration module to select a corresponding algorithm to train a homography transformation model according to a user instruction, to obtain a trained homography transformation model, and to map the line of sight angle to the screen landing point coordinate by using the trained homography transformation model; The second processing module includes: A filter selection module is configured to interact with the mapping model selection and system training module in the first processing module to adjust the filter cutoff frequency of the original screen coordinate; A real-time mapping module, in data interaction with the filter selection module, is configured to perform a projection transformation on the instantaneous line-of-sight angle to predict the original impact point coordinate.

Images