Universal eye movement tracking method and system

Abstract

The present invention belongs to the technical field of data analysis and processing. Provided is a universal eye movement tracking method. The method comprises: by means of an electronic device, collecting facial data within a cycle; on the basis of collected facial features and by means of a deep learning model, detecting the facial data and determining facial feature points, and using a depth estimation model to acquire facial depth information; on the basis of the facial feature points and depth information data, performing parsing to obtain 3D facial feature points; inputting the 3D facial feature points and the depth information data into a 3D engine, and inputting position and rotation data of a camera, and the size, position and rotation data of a screen into the 3D engine, so as to complete the setting for a digital twin environment; and using a standardized facial model and the collected facial feature points to calculate a rotation vector and a translation vector of the head. A system of the present invention can immediately run on any device, such that it is unnecessary to perform individual calibration for each user, thereby simplifying the setting process, and saving on time; and the system can be immediately used and deployed in various environments.

Description

Universal eye tracking method and system Technical Field

[0001] The present invention relates to the technical field of data analysis and processing, and in particular to a universal eye tracking method and system. Background Art

[0002] Eye tracking and gaze estimation technologies have made significant progress in recent years. However, some shortcomings and flaws still exist, limiting their effectiveness and applicability. The following are some of the main shortcomings and flaws observed in the background technology and current methods:

[0003] 1. Hardware Limitations: Existing eye tracking and gaze estimation systems often rely on specific hardware configurations, such as dedicated cameras or specialized sensors. These hardware dependencies limit the widespread adoption and integration of eye tracking technology into a variety of devices and applications. This creates a barrier for developers, manufacturers, and users who require eye tracking capabilities but cannot easily integrate specialized hardware into their systems.

[0004] 2. Limited compatibility: Many current eye tracking solutions are designed only for specific platforms or operating systems, making them incompatible with other systems or devices. This lack of cross-platform compatibility hinders the seamless integration and interoperability of eye tracking technology across different hardware and software environments.

[0005] Summary of the Invention

[0006] (1) Technical problems solved

[0007] In view of the deficiencies of the prior art, the present invention provides a universal eye tracking method and system to solve the problems raised in the above background technology.

[0008] (2) Technical solution

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions: A universal eye tracking method comprises the following steps:

[0010] S100: collecting facial data of one cycle through an electronic device;

[0011] S200: Detecting facial data and determining facial feature points using a deep learning model based on the collected facial features, and obtaining facial depth information using a depth estimation model;

[0012] S300: Analyzing 3D facial feature points based on facial feature points and depth information data;

[0013] S400: Input 3D facial feature points and depth information data into the 3D engine, input camera position, rotation data, screen size, position and rotation data into the 3D engine, and complete the setting of the digital twin environment;

[0014] S500: Calculating the rotation and translation vectors of the head using the normalized facial model and the collected facial feature points, and determining the rotation of the eyes and the corresponding translation vectors by using the normalized facial model and the eye feature points extracted from the facial feature points;

[0015] S600: Using the rotation and translation vectors of the head and eyes as input, the parameters obtained by calculating through the neural network model are determined to determine the user's gaze position.

[0016] As a preference of this embodiment, the step S3 includes:

[0017] S301: Determine the location of facial feature points based on facial features

[0018] S302: Extracting facial edge contours and feature point contours using an edge extraction algorithm;

[0019] S303: By using PFLD as a facial key point detection algorithm, the loss function is continuously optimized iteratively to train the neural network and extract facial features;

[0020] S304: Determine 3D facial feature points based on the facial feature vector, the facial edge contour, and the feature point contour.

[0021] As a preferred embodiment of the present invention, the process of determining the rotation of the eye and the corresponding translation vector by using the normalized facial model and the eye feature points extracted from the facial feature points in S500 includes:

[0022] M501: Detects and estimates eye feature points using a deep learning model, acquires depth information using a depth estimation model, and obtains 3D eye feature points using the eye feature point and depth information data.

[0023] M502: Inputs 3D eye feature points and depth data into the 3D engine, as well as the world camera video. This completes the setup of the digital twin environment by providing the 3D engine with camera position and rotation data.

[0024] M503: Calculate the rotation and translation vectors of each eye using the normalized eye model and eye feature points.

[0025] As a preferred embodiment of this invention, the parameters calculated by the neural network model in step S600 need to be calibrated before determining the user's gaze position. The specific calibration process is as follows:

[0026] Electronic devices process captured facial images, detect and locate eyes, and use techniques such as image filtering, edge detection, and feature extraction to identify key eye features;

[0027] By having users focus on a specific point on the screen or move their eyes in a specific pattern and recording their eye movements, we use this calibration data to create a personalized model that maps eye movements to gaze location.

[0028] As a preference of this embodiment, the eye features include iris, pupil and canthus.

[0029] As a preference of this embodiment, the process of determining the user's gaze position in 600 includes: using eye tracking data or parameters to use a mathematical model to determine the gaze direction, and mapping the determined gaze direction onto the screen.

[0030] As a preferred embodiment of this invention, a general eye tracking system includes an electronic device, wherein the electronic device includes a memory and a processor, the memory being coupled to the processor and configured to store a computer program. When the computer program is executed by the electronic device, the electronic device performs the following operations:

[0031] Collect facial data for a period of time through electronic devices;

[0032] Based on the collected facial features, the deep learning model is used to detect facial data and determine facial feature points, and the depth estimation model is used to obtain facial depth information;

[0033] Analyze 3D facial feature points based on facial feature points and depth information data;

[0034] Input 3D facial feature points and depth information data into the 3D engine, input camera position, rotation data, screen size, position, and rotation data into the 3D engine to complete the setup of the digital twin environment;

[0035] calculating the rotation and translation vectors of the head using the normalized facial model and the collected facial landmarks, and determining the rotation of the eyes and the corresponding translation vectors by using the normalized facial model and the eye landmarks extracted from the facial landmarks;

[0036] Using the rotation and translation vectors of the head and eyes as input, the parameters calculated by the neural network model determine the user's gaze position.

[0037] As a preferred embodiment of this invention, a computer program product is provided. When the computer program product is run on a computer, the computer is enabled to execute the general eye tracking method as described above.

[0038] (3) Beneficial effects

[0039] The present invention provides a universal eye tracking method and system, which has the following beneficial effects:

[0040] 1. No calibration required: Unlike traditional eye tracking systems that require a time-consuming calibration process, the system can be run immediately on any device without the need for individual calibration for each user, simplifying the setup process, saving time, and can be used and deployed immediately in various environments.

[0041] 2. Reduced reliance on hardware: The system reduces reliance on specific eye tracking hardware or sensors. By leveraging the advantages of computer vision technology, deep learning models, and 3D geometric calculations, the system can use existing cameras or near-eye cameras on the device for eye tracking and gaze estimation. This eliminates the need for dedicated hardware, making the technology more accessible, cost-effective, and easily deployed on a variety of devices.

[0042] 3. Improved user experience: The system provides accurate and calibration-free eye tracking and gaze estimation, improving the user experience in many aspects. It enables users to use gaze to interact with digital content intuitively and naturally, and enables users to control applications or systems through gaze, which can improve accessibility, efficiency and immersion in virtual reality environments. In addition, the system's compatibility with various devices ensures a consistent user experience on different platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] FIG1 is a flow chart of a general eye tracking method according to the present invention. DETAILED DESCRIPTION

[0044] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limiting the present invention.

[0045] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like, indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the said features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0046] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections, electrical connections, or mutual communication; direct connections or indirect connections through an intermediate medium; and internal communication between two components or interaction between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0047] In the present invention, unless otherwise expressly specified or limited, a first feature being "above" or "below" a second feature may include the first and second features being in direct contact, or the first and second features not being in direct contact but being in contact via another feature between them. Furthermore, a first feature being "above," "above," and "above" a second feature may include the first feature being directly above or diagonally above the second feature, or may simply mean that the first feature is higher in level than the second feature. A first feature being "below," "below," and "below" a second feature may include the first feature being directly below or diagonally below the second feature, or may simply mean that the first feature is lower in level than the second feature.

[0048] The disclosure below provides many different embodiments or examples for realizing different structures of the present invention. In order to simplify the disclosure of the present invention, the components and settings of specific examples are described below. Of course, they are merely examples and are not intended to limit the present invention. In addition, the present invention may repeat reference numbers and / or reference letters in different examples. Such repetition is for the purpose of simplicity and clarity and does not in itself indicate the relationship between the various embodiments and / or settings discussed. In addition, the present invention provides examples of various specific processes and materials, but those skilled in the art will recognize the application of other processes and / or the use of other materials.

[0049] An embodiment of the present invention provides a general eye tracking method, comprising the following steps:

[0050] S100: collecting facial data of one cycle through an electronic device;

[0051] S200: Detecting facial data and determining facial feature points using a deep learning model based on the collected facial features, and obtaining facial depth information using a depth estimation model;

[0052] S300: Analyzing 3D facial feature points based on facial feature points and depth information data;

[0053] S400: Input 3D facial feature points and depth information data into the 3D engine, input camera position, rotation data, screen size, position and rotation data into the 3D engine, and complete the setting of the digital twin environment;

[0054] S500: Calculating the rotation and translation vectors of the head using the normalized facial model and the collected facial feature points, and determining the rotation of the eyes and the corresponding translation vectors by using the normalized facial model and the eye feature points extracted from the facial feature points;

[0055] S600: Using the rotation and translation vectors of the head and eyes as input, the parameters obtained by calculating through the neural network model are determined to determine the user's gaze position.

[0056] For example, taking a desktop setting as an example, a deep learning model is used to detect the face and estimate facial feature points, and a depth estimation model is also used to obtain depth information. By utilizing these facial feature points and depth data, 3D facial feature points can be obtained.

[0057] In order to establish the digital twin environment, the 3D facial feature points and depth data are input into the 3D engine, the camera position, rotation data, screen size, position and rotation data are input into the 3D engine, and the setting of the digital twin environment is completed.

[0058] The camera is a camera that comes with the electronic device or any video camera that can transmit data to the electronic device.

[0059] Using the normalized facial model and facial landmarks, the head rotation (pitch, yaw, roll) and translation vectors are calculated. In addition, the eye rotation (pitch, yaw, roll) and corresponding translation vectors are determined by using the normalized facial model and eye landmarks extracted from the facial landmarks.

[0060] To predict the point of gaze (PoG), the rotation and translation vectors of the head and eyes are used as input and calculated by a neural network model. This neural network model can accurately estimate the user's gaze position based on the calculated parameters.

[0061] In the near-eye camera setting, we use a deep learning model to detect and estimate eye landmarks. Similarly, we use depth estimation

[0062] The model obtains depth information. By using these eye feature points and depth data, we can obtain 3D eye feature points.

[0063] The near-eye camera setup is similar to the desktop setup, with 3D eye landmarks and depth data fed into the 3D engine. In addition, the world camera video is fed into the 3D engine. The digital twin environment is set up by providing the camera position and rotation data to the 3D engine. Using the normalized eye model and eye landmarks, we calculate the rotation (pitch, yaw, roll) and translation vectors for each eye. These eye rotation and translation vectors serve as input to the neural network model, accurately predicting the point of gaze (PoG).

[0064] By integrating concepts from deep learning, 3D geometry, 3D engines, and digital twins, the proposed eye tracking method achieves accurate, real-time gaze estimation without calibration. This system can be seamlessly implemented across various device types and environments, enhancing user experience and enabling widespread application in areas such as human-computer interaction, virtual reality, and assistive technologies.

[0065] Furthermore, the step S3 includes:

[0066] S301: Determine the location of facial feature points based on facial features

[0067] S302: Extracting facial edge contours and feature point contours using an edge extraction algorithm;

[0068] S303: By using PFLD as a facial key point detection algorithm, the loss function is continuously optimized iteratively to train the neural network and extract facial features;

[0069] S304: Determine 3D facial feature points based on the facial feature vector, the facial edge contour, and the feature point contour.

[0070] Furthermore, the process of determining the rotation of the eye and the corresponding translation vector by using the normalized facial model and the eye feature points extracted from the facial feature points in S500 includes:

[0071] M501: Detects and estimates eye feature points using a deep learning model, acquires depth information using a depth estimation model, and obtains 3D eye feature points using the eye feature point and depth information data.

[0072] M502: Inputs 3D eye feature points and depth data into the 3D engine, as well as the world camera video. This completes the setup of the digital twin environment by providing the 3D engine with camera position and rotation data.

[0073] M503: Calculate the rotation and translation vectors of each eye using the normalized eye model and eye feature points.

[0074] Furthermore, the parameters calculated by the neural network model in S600 need to be calibrated before determining the user's gaze position. The specific calibration process is as follows:

[0075] Electronic devices process captured facial images, detect and locate eyes, and use techniques such as image filtering, edge detection, and feature extraction to identify key eye features;

[0076] By having users focus on a specific point on the screen or move their eyes in a specific pattern and recording their eye movements, we use this calibration data to create a personalized model that maps eye movements to gaze location.

[0077] Furthermore, the eye features include iris, pupil and canthus.

[0078] Furthermore, the process of determining the user's gaze position in 600 includes: using eye tracking data or parameters to determine the gaze direction using a mathematical model, and mapping the determined gaze direction onto the screen.

[0079] This embodiment further provides a general eye tracking system, including an electronic device, wherein the electronic device includes a memory and a processor, the memory being coupled to the processor and configured to store a computer program. When the computer program is executed by the electronic device, the electronic device performs the following operations:

[0080] Collect facial data for a period of time through electronic devices;

[0081] Based on the collected facial features, the deep learning model is used to detect facial data and determine facial feature points, and the depth estimation model is used to obtain facial depth information;

[0082] Analyze 3D facial feature points based on facial feature points and depth information data;

[0083] Input 3D facial feature points and depth information data into the 3D engine, input camera position, rotation data, screen size, position, and rotation data into the 3D engine to complete the setup of the digital twin environment;

[0084] calculating the rotation and translation vectors of the head using the normalized facial model and the collected facial landmarks, and determining the rotation of the eyes and the corresponding translation vectors by using the normalized facial model and the eye landmarks extracted from the facial landmarks;

[0085] Using the rotation and translation vectors of the head and eyes as input, the parameters calculated by the neural network model determine the user's gaze position.

[0086] Specifically, the universal eye tracking system combines the concepts of deep learning, 3D geometry, 3D engine and digital twins to achieve accurate and calibration-free eye tracking and gaze estimation capabilities. The system is suitable for any device, does not require manual calibration, and can run immediately after setup.

[0087] As a preferred embodiment of this invention, a computer program product is provided. When the computer program product is run on a computer, the computer is enabled to execute the general eye tracking method as described above.

[0088] In summary, the universal eye tracking method and system of the present application does not require calibration operations: unlike traditional eye tracking systems that require a time-consuming calibration process, the system can run immediately on any device without the need for individual calibration for each user, simplifying the setup process, saving time, and can be immediately used and deployed in various environments.

[0089] Second, reduced dependence on hardware: The system reduces dependence on specific eye tracking hardware or sensors. By leveraging the advantages of computer vision technology, deep learning models, and 3D geometric calculations, the system can use existing cameras or near-eye cameras on the device for eye tracking and gaze estimation, which eliminates the need for dedicated hardware, making the technology more accessible, cost-effective, and easily deployed on a variety of devices.

[0090] And, improve user experience: The system provides accurate and calibration-free eye tracking and gaze estimation, improving the user experience in many aspects. It enables users to use gaze to interact with digital content intuitively and naturally, and enables users to control applications or systems through gaze, which can improve accessibility, efficiency and immersion in virtual reality environments. In addition, the system's compatibility with various devices ensures a consistent user experience on different platforms.

[0091] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A general eye tracking method, characterized in that: The steps include: S100: collecting facial data of one cycle through an electronic device; S200: Detecting facial data and determining facial feature points using a deep learning model based on the collected facial features, and obtaining facial depth information using a depth estimation model; S300: parsing 3D facial feature points based on facial feature points and depth information data; S400: Input the 3D facial feature points and depth information data into the 3D engine, input the camera position, rotation data, screen size, position and rotation data into the 3D engine, and complete the setting of the digital twin environment; S5 00: Calculate the rotation and translation vector of the head using the standardized facial model and the collected facial feature points, and determine the rotation of the eyes and the corresponding translation vector by using the standardized facial model and the eye feature points extracted from the facial feature points; S600: Using the rotation and translation vectors of the head and eyes as input, the parameters obtained by calculating through the neural network model determine the user's gaze position.

2. A universal eye tracking method according to claim 1, characterized in that: The steps of S3 include: S301: Determine the position of facial feature points based on facial features S302: extracting facial edge contours and feature point contours through an edge extraction algorithm; S303: By adopting PFLD as a facial key point detection algorithm, the loss function is continuously optimized iteratively to train a neural network and extract facial feature vectors; S304: Determine 3D facial feature points based on the facial feature vector, the facial edge contour and the feature point contour.

3. A universal eye tracking method according to claim 1, characterized in that: The process of determining the rotation of the eye and the corresponding translation vector by using the normalized facial model and the eye feature points extracted from the facial feature points in S500 includes: M501: Use the deep learning model to detect and estimate eye feature points, use the depth estimation model to obtain depth information, and obtain 3D eye feature points through eye feature points and depth information data; M502: Input 3D eye feature points and depth data into the 3D engine, as well as the world camera video into the 3D engine. Complete the setup of the digital twin environment by providing the camera position and rotation data to the 3D engine. M503: Using the normalized eye model and eye feature points, calculate the rotation and translation vectors of each eye.

4. A universal eye tracking method according to claim 1, characterized in that: The parameters calculated by the neural network model in S600 need to be calibrated before determining the user's gaze position. The specific calibration process is as follows: Electronic devices process captured facial images to detect and locate eyes, using techniques such as image filtering, edge detection, and feature extraction to identify key eye features; By having users focus on a specific point on the screen or move their eyes in a specific pattern and recording their eye movements, we use this calibration data to create a personalized model that maps eye movements to gaze location.

5. A universal eye tracking method according to claim 4, characterized in that: The ocular features include the iris, pupil and canthus.

6. A universal eye tracking method according to claim 5, characterized in that: The process of determining the user's gaze position in 600 includes: using eye tracking data or parameters to use a mathematical model to determine the gaze direction, and mapping the determined eye movement gaze direction onto the screen.

7. A universal eye tracking system, characterized in that: The electronic device includes a memory and a processor, wherein the memory is coupled to the processor, and the memory is used to store a computer program. When the computer program is executed by the electronic device, the electronic device performs the following operations: Collect facial data for a period of time through electronic devices; Based on the collected facial features, the deep learning model is used to detect facial data and determine facial feature points, and the depth estimation model is used to obtain facial depth information; Analyze 3D facial feature points based on facial feature points and depth information data; Input the 3D facial feature points and depth information data into the 3D engine, input the camera position, rotation data, screen size, position and rotation data into the 3D engine, and complete the setting of the digital twin environment; Calculating the rotation and translation vector of the head using the normalized facial model and the collected facial feature points, and determining the rotation of the eyes and the corresponding translation vector by using the normalized facial model and the eye feature points extracted from the facial feature points; Using the rotation and translation vectors of the head and eyes as input, the parameters calculated by the neural network model determine the user's gaze position.

8. A computer program product, characterized in that: When the computer program product runs on a computer, the computer is enabled to execute the general eye tracking method according to any one of claims 1 to 7.

Images