An eye-tracking method, device, equipment, and medium

Abstract

This invention discloses an eye-tracking method, apparatus, device, and medium, relating to the field of image processing technology. The method utilizes a deep learning model trained on a large amount of data to improve the prediction accuracy of eye-tracking. This high accuracy helps ensure excellent performance of the eye-tracking system across different scenarios and individuals, providing reliable data support for medical applications. The deep learning model is adaptive, capable of adjusting according to the eye movement characteristics of different patients and environments. This makes the eye-tracking system more versatile, applicable to a wide range of medical applications, and adaptable to individual differences. The high efficiency of the deep learning model enables the eye-tracking system to process and analyze eye movement data in real time.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an eye-tracking method, apparatus, device, and medium based on point tracking and deep learning. Background Technology

[0002] Eye-tracking technology is a key human-computer interaction method that accurately measures the focus of a user's gaze by tracking the movement of their eyes. Traditional methods involve illuminating the eyeball with an infrared light source and capturing the pupil's reflection using a camera to obtain eye movement information. Modern methods, however, utilize computer vision technology and image processing algorithms to track the pupil, corneal reflection, and eyeball contour, thereby achieving more precise eye movement data extraction.

[0003] Eye-tracking technology captures and analyzes eye movement data to track and record an individual's gaze focus in real time. In the field of disability healthcare, this technology is widely used to help patients who are unable to communicate through traditional means due to movement disorders or other physical limitations.

[0004] Despite significant progress in eye-tracking technology, several challenges remain. For example, inconsistent performance in low-light conditions, individual differences, and the real-time processing of eye-tracking data limit its widespread adoption in practical applications. These limitations underscore the urgent need for innovation and improvement.

[0005] "Eye-tracking data processing method and electronic device CN116661587A": This application provides an eye-tracking data processing method and electronic device. The method includes: acquiring an image to be tested; correcting a first calibration image based on head pose data of the image to be tested to obtain multiple second calibration images and the calibration position of the gaze point of each second calibration image; determining the predicted position of the gaze point of the image to be tested based on the image to be tested, the multiple second calibration images, and the calibration position of the gaze point of each second calibration image; and triggering the electronic device to perform corresponding operations based on the predicted position of the gaze point of the image to be tested.

[0006] "A Content Interaction Method Based on MR Smart Glasses CN116909408A": This invention discloses a content interaction method based on MR smart glasses, relating to the field of intelligent interaction technology. The method includes determining the user's state and needs attributes, and acquiring a local image of the visual gaze point area; performing artificial intelligence processing on the acquired image content; prioritizing notification icons of relevant applications, services, and information based on the processing results; and retrieving answers from a database or opening relevant applications based on the user's needs attributes and priority ranking, and presenting the results on an optical display.

[0007] "An eye-tracking method, apparatus, device, and storage medium CN111966219A": This invention discloses an eye-tracking method, apparatus, device, and storage medium, comprising: acquiring eye images of people wearing glasses and people without glasses as a dataset; training a neural network model based on the dataset; preprocessing the eye images of people wearing glasses based on the neural network model; and performing eye tracking based on the preprocessed images.

[0008] Despite significant advancements in vision-based eye-tracking technology across many fields, several drawbacks and limitations remain, which may have varying degrees of impact in different application scenarios. Here are some common drawbacks:

[0009] 1. Sensitivity is affected by the environment: Factors such as lighting conditions, background noise, and ambient light can affect the sensitivity of vision-based eye-tracking systems. In complex or uncontrolled environments, the system may exhibit instability.

[0010] 2. Individual differences: Differences exist between individuals, such as pupil size and iris texture, which can affect the accuracy of eye-tracking systems. Some systems may require personalized calibration, which can be inconvenient for users.

[0011] 3. Motion artifacts: Rapid eye movements can cause motion artifacts, making it difficult for the system to accurately track the eye's position. This can be a challenge for applications requiring high precision, such as virtual reality and simulation.

[0012] 4. Equipment Cost and Complexity: High-precision eye-tracking devices are typically expensive, and some complex systems may require specialized hardware and software support. This increases the deployment and maintenance costs of the system.

[0013] 5. Requires additional equipment: Most vision-based eye-tracking technologies require users to wear additional equipment, such as eye trackers or cameras, which may not be suitable for some special groups or long-term use scenarios.

[0014] 6. Impact of Head Movements: Head movements can cause changes in the position of the eyes relative to the camera, negatively impacting the accuracy of eye tracking. Some systems require head movement tracking for correction.

[0015] 7. Real-time requirements: For some applications with high real-time requirements, such as driver assistance systems, eye-tracking systems may require faster response times, and some systems may face challenges in this regard.

[0016] 8. Unnatural experimental environments: Some experimental environments may be unnatural, causing users to behave differently when using eye-tracking systems than they do in daily life, thus reducing the reliability of experimental results. Summary of the Invention

[0017] In view of the above problems, the present invention provides an eye-tracking method, apparatus, device, and medium for overcoming or at least partially solving the above problems. It overcomes the limitations of the prior art and improves robustness in complex environments by introducing advanced image processing algorithms.

[0018] This invention provides the following solution:

[0019] An eye-tracking method includes:

[0020] The RGB image of the eye of the organism under test in the current frame, the RGB image of the previous frame, and the heat map predicted in the previous frame are acquired; the heat map is used to represent the position and confidence level of the target center point in the previous frame.

[0021] The RGB image of the current frame, the RGB image of the previous frame, and the heatmap are input into a deep learning point tracking network to obtain predictions of key eye movement feature points and form eye movement trajectories.

[0022] The deep learning point tracking network is used to perform the following operations:

[0023] A fused feature map is obtained by bitwise addition of the RGB image of the current frame, the RGB image of the previous frame, and the heatmap using convolutional layers and batch normalization layers.

[0024] The fused feature map is input into a fully convolutional neural network (WCNN) so that the WCNN can predict heatmap branches, offset branches, size branches, and correlation branches. The heatmap branches are used to output the position and confidence of the center point of different object categories. The offset branches are used to output the offset of the target center point relative to the grid center. The size branches are used to output the width and height of the target. The correlation branches are used to output the displacement vector of the target center point relative to the center point of the previous frame.

[0025] Several peak points are extracted from the heatmap branches of each category;

[0026] Extract the number of points from the previous target at several of the aforementioned peak points as the detected center points;

[0027] For each center point, the bounding box coordinates of each center point in the current frame and the previous frame are calculated based on the outputs of the corresponding offset branch, the size branch and the association branch, as well as the association relationship between each center point and the center point of the previous frame.

[0028] Preferably, if the value of a point extracted from the heatmap branch is greater than or equal to the value of a point within its 8-neighborhood, then this point is determined to be a peak point.

[0029] Preferably, the target quantity is 100.

[0030] Preferably, the deep learning point tracking network includes a backbone network, an object detection head, and an object tracking head; the backbone network is used to acquire high-level semantic information of the image; the object detection head is used to generate an information map of the object's center point, bounding box, and key points; and the object tracking head is used to achieve long-term tracking of the object using object information from previous frames.

[0031] Preferably, the backbone network includes either the Hourglass algorithm or the ResNet algorithm.

[0032] Preferably, the loss function of the network used during the training of the deep learning point tracking network is expressed by the following formula:

[0033]

[0034] In the formula: t represents the t-th frame, and t-1 represents the frame preceding the current frame. This indicates the offset of the target center point position between two consecutive frames.

[0035] Preferably:

[0036]

[0037] In the formula: This indicates the detected center point of the target.

[0038] An eye-tracking device for performing the above-described eye-tracking method, the device comprising:

[0039] The image acquisition unit is used to acquire the RGB image of the current frame of the eye of the organism under test, the RGB image of the previous frame, and the predicted heat map of the previous frame; the heat map is used to represent the position and confidence level of the target center point in the previous frame.

[0040] An eye-tracking unit is used to input the RGB image of the current frame, the RGB image of the previous frame, and the heatmap into a deep learning point tracking network to obtain predictions of key eye-tracking feature points and form eye-tracking trajectories.

[0041] The deep learning point tracking network is used to perform the following operations:

[0042] A fused feature map is obtained by bitwise addition of the RGB image of the current frame, the RGB image of the previous frame, and the heatmap using convolutional layers and batch normalization layers.

[0043] The fused feature map is input into a fully convolutional neural network (WCNN) so that the WCNN can predict heatmap branches, offset branches, size branches, and correlation branches. The heatmap branches are used to output the position and confidence of the center point of different object categories. The offset branches are used to output the offset of the target center point relative to the grid center. The size branches are used to output the width and height of the target. The correlation branches are used to output the displacement vector of the target center point relative to the center point of the previous frame.

[0044] Several peak points are extracted from the heatmap branches of each category;

[0045] Extract the number of points from the previous target at several of the aforementioned peak points as the detected center points;

[0046] For each center point, the bounding box coordinates of each center point in the current frame and the previous frame are calculated based on the outputs of the corresponding offset branch, the size branch and the association branch, as well as the association relationship between each center point and the center point of the previous frame.

[0047] An eye-tracking device, the device comprising a processor and a memory:

[0048] The memory is used to store program code and transmit the program code to the processor;

[0049] The processor is used to execute the eye-tracking method described above according to the instructions in the program code.

[0050] A computer-readable storage medium for storing program code for performing the eye-tracking method described above.

[0051] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0052] This application provides an eye-tracking method, apparatus, device, and medium that utilizes a deep learning model trained on a large amount of data to improve the prediction accuracy of eye-tracking. This high accuracy helps ensure excellent performance of the eye-tracking system across different scenarios and individuals, providing reliable data support for medical applications. The deep learning model is adaptive, capable of adjusting according to the eye movement characteristics of different patients and environments. This makes the eye-tracking system more versatile, applicable to a wide range of medical applications, and adaptable to individual differences. The high efficiency of the deep learning model enables the eye-tracking system to process and analyze eye movement data in real time.

[0053] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0055] Figure 1 This is a flowchart of an eye-tracking gaze method provided in an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the network processing flow and network structure provided in the embodiments of the present invention;

[0057] Figure 3 This is a flowchart of model training and eye tracking provided in an embodiment of the present invention;

[0058] Figure 4 This is a schematic diagram of an eye-tracking device provided in an embodiment of the present invention;

[0059] Figure 5 This is a schematic diagram of an eye-tracking device provided in an embodiment of the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0061] See Figure 1This invention provides an eye-tracking gaze method, such as... Figure 1 As shown, the method may include:

[0062] S101: Acquire the RGB image of the current frame of the eye of the organism under test, the RGB image of the previous frame, and the heat map predicted in the previous frame; the heat map is used to represent the position and confidence level of the target center point in the previous frame;

[0063] S102: Input the RGB image of the current frame, the RGB image of the previous frame, and the heatmap into the deep learning point tracking network to obtain predictions of key eye movement feature points and form eye movement trajectories.

[0064] The deep learning point tracking network is used to perform the following operations:

[0065] A fused feature map is obtained by bitwise addition of the RGB image of the current frame, the RGB image of the previous frame, and the heatmap using convolutional layers and batch normalization layers.

[0066] The fused feature map is input into a fully convolutional neural network (WCNN) so that the WCNN can predict heatmap branches, offset branches, size branches, and correlation branches. The heatmap branches are used to output the position and confidence of the center point of different object categories. The offset branches are used to output the offset of the target center point relative to the grid center. The size branches are used to output the width and height of the target. The correlation branches are used to output the displacement vector of the target center point relative to the center point of the previous frame.

[0067] Several peak points are extracted from the heatmap branches of each category; in specific implementation, the embodiments of this application can determine that a point is a peak point if the value of a point extracted from the heatmap branch is greater than or equal to the value of a point within its 8-neighborhood.

[0068] The number of points representing the previous target quantity is extracted from several peak points and used as the detected center point; in a specific implementation, the target quantity can be 100.

[0069] For each center point, the bounding box coordinates of each center point in the current frame and the previous frame are calculated based on the outputs of the corresponding offset branch, the size branch and the association branch, as well as the association relationship between each center point and the center point of the previous frame.

[0070] Furthermore, the deep learning point tracking network includes a backbone network, an object detection head, and an object tracking head; the backbone network is used to acquire high-level semantic information of the image; the object detection head is used to generate information maps of the object's center point, bounding box, and key points; the object tracking head is used to achieve long-term tracking of the object using object information from previous frames. The backbone network includes any one of the Hourglass algorithm and the ResNet algorithm.

[0071] The loss function of the deep learning point tracking network used during training is expressed by the following formula:

[0072]

[0073] In the formula, t represents the t-th frame, and t-1 represents the frame preceding the current frame. This indicates the offset of the target center point position between two consecutive frames.

[0074]

[0075] In the formula: This indicates the detected center point of the target.

[0076] The eye-tracking method provided in this application uses a keypoint tracking network for pupil localization and tracking. This method organically integrates detection and tracking tasks, achieving efficient and accurate target tracking. It uses center points to represent targets and predicts their association with the previous frame. The advantages of this method are its simplicity, speed, and accuracy; it does not require anchor boxes or non-maximum suppression, nor does it require reinitializing lost tracking.

[0077] This method achieves efficient detection and accurate tracking of eye movements by introducing a deep learning-based keypoint tracking network. This network can adapt to various scenarios and complex environments, providing a reliable foundation for eye-tracking. It can be applied in the medical field for people with disabilities, providing eye-tracking support for patients with impaired functions. This includes, but is not limited to, patients with movement disorders and speech disorders, providing them with more convenient and flexible interaction methods to promote full functional recovery. The introduction of deep learning networks enables higher real-time performance and accuracy in eye-tracking. This is crucial for medical applications, ensuring patients receive timely support and feedback, thereby maximizing treatment outcomes.

[0078] This method aims to be applied in the field of healthcare for people with disabilities to improve their quality of life and the effectiveness of medical services. Current eye-tracking technologies have limitations in accuracy, real-time performance, and applicability in certain situations, particularly for the needs of people with disabilities.

[0079] By combining deep learning algorithms and point tracking technology, this method aims to overcome the limitations of traditional eye-tracking techniques and improve the accuracy and sensitivity of tracking the eye movements of patients with disabilities. This method can be applied in the medical field for people with disabilities, such as rehabilitation therapy, communication assistance, and the control of intelligent assistive devices, providing patients with disabilities with a more precise, convenient, and personalized eye-tracking interaction experience.

[0080] The application of this method aims to provide greater independence and improve the quality of life for people with disabilities, while also offering healthcare professionals more powerful tools to better understand and support the rehabilitation process for patients with disabilities. It seeks to advance eye-tracking technology, bringing it closer to medical practice and providing innovative solutions to the field of disability healthcare.

[0081] The point-tracking-based deep learning eye-tracking method proposed in this application, based on a keypoint tracking network, aims to solve the technical challenges of eye-tracking detection and tracking in the medical field for people with disabilities. The key to this technology lies in its comprehensive application of advanced deep learning algorithms, particularly keypoint tracking networks, to achieve accurate and real-time tracking of eye movements.

[0082] The key aspects of the method provided in this application's embodiments include, but are not limited to, the following:

[0083] 1. Keypoint Tracking Network: By introducing a keypoint tracking network, efficient detection and accurate tracking of eye movements can be achieved. This network is renowned for its superior target tracking performance and can adapt to various scenarios and complex environments, thus providing a reliable foundation for eye-tracking.

[0084] 2. Application of Point Tracking Technology: By employing point tracking technology, high-precision reconstruction of eye movement trajectories can be achieved through precise calibration and tracking of key points during eye movements. This technology enables more detailed analysis of eye movement behavior, providing more accurate functional recovery support for the medical field of disability care.

[0085] 3. Widespread Applications in the Medical Field for People with Disabilities: This method is primarily applied in the medical field for people with disabilities, providing customized eye-tracking support for patients with impaired functions. This includes, but is not limited to, patients with movement disorders and speech disorders, providing them with more convenient and flexible interaction methods to promote full functional recovery.

[0086] 4. Improved Real-Time Performance and Accuracy: The introduction of deep learning networks has enabled eye-tracking to achieve higher real-time performance and accuracy. This is crucial for medical applications, ensuring patients receive timely support and feedback, thereby maximizing treatment outcomes.

[0087] The unique combination of the above key technologies constitutes the innovation and practicality of this patent. The core of the protection lies in this entire technological system and its application in the field of medical care for people with disabilities, bringing significant progress and application prospects to the field of eye-tracking technology.

[0088] The principle and process of the deep learning point tracking network provided in this application embodiment are as follows:

[0089] The deep learning point tracking network takes three inputs: the RGB image of the current frame, the RGB image of the previous frame, and the heatmap predicted from the previous frame. The heatmap represents the position and confidence of the target center point in the previous frame, providing temporal information. These three inputs are then passed through several convolutional layers and batch normalization layers, and finally summed bit-by-bit to obtain a fused feature map.

[0090] Deep learning point tracking networks employ various fully convolutional neural networks with different encoder-decoder structures, such as ResNet, DLA, and Hourglass, to predict four output branches from the fused feature map:

[0091] Heatmap branches, size Output the position and confidence level of the center point of different categories of objects.

[0092] Offset branch, size is Output the offset of the target center point relative to the grid center to improve positioning accuracy.

[0093] Size branch, size is Output the width and height of the target to generate the bounding box.

[0094] Related branches, size is Output the displacement vector of the target center point relative to the center point of the previous frame, which is used to achieve temporal correlation.

[0095] During the inference phase, peak points are first extracted from the heatmap branches of each category. If a point's value is greater than or equal to the values ​​of its 8 neighboring points, then that point is considered a peak point. The top 100 peak points are then selected as the detected center points. For each detected center point, its bounding box coordinates in the current and previous frames, as well as its association with the center point of the previous frame, are calculated based on the outputs of its offset, size, and association branches. Ultimately, all target information is obtained directly through the above center point estimation method, without requiring IoU-based non-maximum suppression or other post-processing operations.

[0096] Overall, such as Figure 2 As shown, the structure of a deep learning point tracking network consists of three main parts:

[0097] 1. Backbone Network:

[0098] This network employs advanced backbone networks (such as Hourglass or ResNet) to extract features from images in order to obtain high-level semantic information from the images.

[0099] 2. Target Detection Head:

[0100] Following the backbone network, an information map of the target's center point, bounding box, and key points is generated using the target detection head. This step involves performing multi-scale convolution operations on the feature map of the backbone network to effectively capture multi-scale information about the target.

[0101] 3. Tracking Head:

[0102] This network introduces a target tracking head, which enables long-term target tracking by utilizing target information from previous frames. This part further improves the accuracy of target tracking by generating the target's motion vectors and appearance features.

[0103] The output of the entire network structure includes the target's center point, bounding box, key points, and motion information, enabling efficient target detection and long-term tracking within a unified framework. This fusion of detection and tracking methods makes this network perform exceptionally well in target tracking tasks, especially maintaining robust target tracking even in complex scenes. The network's loss function is:

[0104]

[0105] in This represents the detected target center point, where t represents the specific location in the time series, i.e., the t-th frame, and t-1 represents the frame preceding the current frame. This indicates the offset of the target center point position between two consecutive frames, i.e.

[0106]

[0107] like Figure 2 The diagram shows the network processing flow and network structure.

[0108] The method provided in this application includes training a deep learning model and an eye-tracking implementation step, such as... Figure 3 As shown, the detailed steps for both parts are as follows:

[0109] 1. Dataset Preparation: Eye movement data of patients in different scenarios were collected using high-resolution camera equipment. Each data sample includes an eye movement image and corresponding real eye movement annotations, which are used to guide the training of the deep learning point tracking network.

[0110] 2. Network Architecture Design: Design the architecture of the deep learning point tracking network to ensure that the network can accurately capture the features of eye movements. The network output should include accurate predictions of key eye movement feature points, forming eye movement trajectories.

[0111] 3. Loss Function Definition: Define the loss function used for training, which measures the difference between the model's predictions and the actual eye-tracking annotations. The loss function typically includes a regression loss to ensure that the model can accurately regress the eye-tracking trajectory.

[0112] 4. Data Augmentation: Augment the eye-tracking data by rotating, scaling, and flipping to expand the training set and improve the model's generalization ability.

[0113] 5. Model Training: Using the prepared eye-tracking dataset, the model is trained through a deep learning point-tracking network. During training, the model continuously optimizes the network parameters through backpropagation, gradually improving its accuracy in predicting eye-tracking behaviors.

[0114] 6. Validation and tuning: Use the validation set to evaluate the model's performance during training, and tune the model parameters based on the validation results to improve the model's generalization performance.

[0115] 7. Eye-tracking: A trained deep learning point-tracking network is embedded into the eye-tracking system. In practical applications, the deep learning model can track the patient's eye movement trajectory in real time using eye movement data collected by the system, generating accurate eye-tracking results.

[0116] 8. Real-time Feedback and Personalized Optimization: Through real-time feedback from the deep learning model, the system not only provides accurate eye-tracking results but also performs personalized optimization based on real-time data analysis. This helps adjust the parameters of the tracking model to suit the eye movement characteristics and needs of different patients.

[0117] 9. Results Presentation and Analysis: Through accurate tracking by the deep learning model, the system can present visualized results of the patient's eye movement behavior, including eye trajectory maps and trends in key point changes. This provides medical professionals with clear and intuitive eye movement data, supporting applications in medical decision-making and rehabilitation training.

[0118] In summary, the eye-tracking method provided in this application utilizes a deep learning model trained on a large amount of data, thereby improving the prediction accuracy of eye-tracking. This high accuracy helps ensure that the eye-tracking system performs well across different scenarios and individuals, providing reliable data support for medical applications. The deep learning model is adaptive, capable of adjusting according to the eye movement characteristics of different patients and environments. This makes the eye-tracking system more versatile, applicable to a wide range of medical applications, and adaptable to individual differences. The high efficiency of the deep learning model enables the eye-tracking system to process and analyze eye movement data in real time.

[0119] See Figure 4 This application embodiment can also provide an eye-tracking device, such as... Figure 4 As shown, the device for performing the above-described eye-tracking method may include:

[0120] Image acquisition unit 401 is used to acquire the RGB image of the current frame of the eye of the organism under test, the RGB image of the previous frame, and the heat map predicted in the previous frame; the heat map is used to represent the position and confidence level of the target center point in the previous frame.

[0121] The eye-tracking unit 402 is used to input the RGB image of the current frame, the RGB image of the previous frame, and the heatmap into a deep learning point tracking network to obtain predictions of key eye-tracking feature points and form an eye-tracking trajectory.

[0122] The deep learning point tracking network is used to perform the following operations:

[0123] A fused feature map is obtained by bitwise addition of the RGB image of the current frame, the RGB image of the previous frame, and the heatmap using convolutional layers and batch normalization layers.

[0124] The fused feature map is input into a fully convolutional neural network (WCNN) so that the WCNN can predict heatmap branches, offset branches, size branches, and correlation branches. The heatmap branches are used to output the position and confidence of the center point of different object categories. The offset branches are used to output the offset of the target center point relative to the grid center. The size branches are used to output the width and height of the target. The correlation branches are used to output the displacement vector of the target center point relative to the center point of the previous frame.

[0125] Several peak points are extracted from the heatmap branches of each category;

[0126] Extract the number of points from the previous target at several of the aforementioned peak points as the detected center points;

[0127] For each center point, the bounding box coordinates of each center point in the current frame and the previous frame are calculated based on the outputs of the corresponding offset branch, the size branch and the association branch, as well as the association relationship between each center point and the center point of the previous frame.

[0128] This application embodiment can also provide an eye-tracking device, the device including a processor and a memory:

[0129] The memory is used to store program code and transmit the program code to the processor;

[0130] The processor is used to execute the steps of the eye-tracking method described above according to the instructions in the program code.

[0131] like Figure 5 As shown in the figure, an eye-tracking device provided in this application embodiment may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0132] In the embodiments of this application, the processor 10 may be a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0133] The processor 10 can call programs stored in the memory 11. Specifically, the processor 10 can execute operations in the embodiments of the eye-tracking method.

[0134] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions:

[0135] The RGB image of the eye of the organism under test in the current frame, the RGB image of the previous frame, and the heat map predicted in the previous frame are acquired; the heat map is used to represent the position and confidence level of the target center point in the previous frame.

[0136] The RGB image of the current frame, the RGB image of the previous frame, and the heatmap are input into a deep learning point tracking network to obtain predictions of key eye movement feature points and form eye movement trajectories.

[0137] The deep learning point tracking network is used to perform the following operations:

[0138] A fused feature map is obtained by bitwise addition of the RGB image of the current frame, the RGB image of the previous frame, and the heatmap using convolutional layers and batch normalization layers.

[0139] The fused feature map is input into a fully convolutional neural network (WCNN) so that the WCNN can predict heatmap branches, offset branches, size branches, and correlation branches. The heatmap branches are used to output the position and confidence of the center point of different object categories. The offset branches are used to output the offset of the target center point relative to the grid center. The size branches are used to output the width and height of the target. The correlation branches are used to output the displacement vector of the target center point relative to the center point of the previous frame.

[0140] Several peak points are extracted from the heatmap branches of each category;

[0141] Extract the number of points from the previous target at several of the aforementioned peak points as the detected center points;

[0142] For each center point, the bounding box coordinates of each center point in the current frame and the previous frame are calculated based on the outputs of the corresponding offset branch, the size branch and the association branch, as well as the association relationship between each center point and the center point of the previous frame.

[0143] In one possible implementation, the memory 11 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as file creation or data read / write). The data storage area may store data created during use, such as initialization data.

[0144] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0145] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0146] Of course, it should be noted that, Figure 5 The structure shown does not constitute a limitation on the eye-tracking device in the embodiments of this application. In practical applications, the eye-tracking device may include devices that are more advanced than those described above. Figure 5 More or fewer components as shown, or combinations of certain components.

[0147] This application embodiment may also provide a computer-readable storage medium for storing program code for performing the steps of the above-described eye-tracking method.

[0148] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0149] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0150] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0151] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. An eye-tracking gaze method, characterized in that, include: Acquire the RGB image of the eye of the organism under test in the current frame, the RGB image of the previous frame, and the predicted heatmap of the previous frame; The heatmap is used to represent the position and confidence level of the target center point in the previous frame; The RGB image of the current frame, the RGB image of the previous frame, and the heatmap are input into a deep learning point tracking network to obtain predictions of key eye movement feature points and form eye movement trajectories. The deep learning point tracking network is used to perform the following operations: A fused feature map is obtained by bitwise addition of the RGB image of the current frame, the RGB image of the previous frame, and the heatmap using convolutional layers and batch normalization layers. The fused feature map is input into a fully convolutional neural network so that the fully convolutional neural network predicts heatmap branches, offset branches, size branches, and association branches; the heatmap branches are used to output the position and confidence of the center point of different categories of objects; the offset branches are used to output the offset of the target center point relative to the grid center; the size branches are used to output the width and height of the target. The associated branch is used to output the displacement vector of the target center point relative to the center point of the previous frame; Several peak points are extracted from the heatmap branches of each category; Extract the number of points from the previous target at several of the aforementioned peak points as the detected center points; For each center point, the bounding box coordinates of each center point in the current frame and the previous frame are calculated based on the outputs of the corresponding offset branch, the size branch and the association branch, as well as the association relationship between each center point and the center point of the previous frame.

2. The eye-tracking method according to claim 1, characterized in that, If the value of a point extracted from the heatmap branch is greater than or equal to the value of a point within its 8-neighborhood, then this point is determined to be a peak point.

3. The eye-tracking method according to claim 1, characterized in that, The target quantity is 100.

4. The eye-tracking method according to claim 1, characterized in that, The deep learning point tracking network includes a backbone network, an object detection head, and an object tracking head; the backbone network is used to acquire high-level semantic information of the image; the object detection head is used to generate an information map of the object's center point, bounding box, and key points; and the object tracking head is used to achieve long-term tracking of the object using object information from previous frames.

5. The eye-tracking method according to claim 4, characterized in that, The backbone network includes either the Hourglass algorithm or the ResNet algorithm.

6. The eye-tracking method according to claim 1, characterized in that, The loss function of the deep learning point tracking network used during training is expressed by the following formula: In the formula: t represents the t-th frame, and t-1 represents the frame preceding the current frame. This indicates the offset of the target center point position between two consecutive frames.

7. The eye-tracking method according to claim 6, characterized in that, In the formula: This indicates the detected center point of the target.

8. An eye-tracking device, characterized in that, The apparatus for performing the eye-tracking method according to any one of claims 1-7, the apparatus comprising: The image acquisition unit is used to acquire the RGB image of the current frame of the eye of the organism under test, the RGB image of the previous frame, and the predicted heat map of the previous frame; the heat map is used to represent the position and confidence level of the target center point in the previous frame. An eye-tracking unit is used to input the RGB image of the current frame, the RGB image of the previous frame, and the heatmap into a deep learning point tracking network to obtain predictions of key eye-tracking feature points and form eye-tracking trajectories. The deep learning point tracking network is used to perform the following operations: A fused feature map is obtained by bitwise addition of the RGB image of the current frame, the RGB image of the previous frame, and the heatmap using convolutional layers and batch normalization layers. The fused feature map is input into a fully convolutional neural network (WCNN) so that the WCNN can predict heatmap branches, offset branches, size branches, and correlation branches. The heatmap branches are used to output the position and confidence of the center point of different object categories. The offset branches are used to output the offset of the target center point relative to the grid center. The size branches are used to output the width and height of the target. The correlation branches are used to output the displacement vector of the target center point relative to the center point of the previous frame. Several peak points are extracted from the heatmap branches of each category; Extract the number of points from the previous target at several of the aforementioned peak points as the detected center points; For each center point, the bounding box coordinates of each center point in the current frame and the previous frame are calculated based on the outputs of the corresponding offset branch, the size branch and the association branch, as well as the association relationship between each center point and the center point of the previous frame.

9. An eye-tracking device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the eye-tracking method according to any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for performing the eye-tracking method according to any one of claims 1-7.

Images