Eye movement sight tracking method, device, equipment and medium

Abstract

The invention discloses an eye movement gaze tracking method, device and equipment and a medium, and relates to the technical field of image processing, the method uses a deep learning model to train through a large amount of data, and the prediction accuracy of eye movement gaze tracking is improved. The high accuracy is helpful for ensuring that the eye movement sight tracking system has excellent performance in different scenes and among individuals, and reliable data support is provided for medical application. The deep learning model has adaptivity and can be adjusted according to eye movement behavior characteristics of different patients and different environments. Therefore, the eye movement sight tracking system is more universal, is suitable for wide medical application scenarios, and can adapt to individual differences at the same time. The efficient performance of the deep learning model enables the eye movement gaze tracking system to process and analyze eye movement data in real time.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an eye movement and sight tracking method, device, equipment and medium based on point tracking and deep learning. Background Art

[0002] Eye tracking technology is a key means of human-computer interaction. This technology can accurately measure the focus of the user's gaze by tracking the movement of the user's eyes. Traditional methods include using infrared light to illuminate the eyeball and capturing the pupil reflection through a camera to obtain eye movement information. Modern methods use computer vision technology to track the pupil, corneal reflection and eyeball contour through image processing algorithms, thereby achieving more accurate eye movement data extraction.

[0003] Eye tracking technology is a technology that captures and analyzes eye movement data to track and record an individual's gaze focus in real time. In the field of medical care for the disabled, this technology is widely used to help patients who are unable to communicate through traditional methods due to motor disorders or other physical limitations.

[0004] Although eye tracking technology has made significant progress, there are still some challenges. For example, unstable performance in low-light environments, differences between individuals, and real-time processing of eye movement data limit the widespread promotion of technology in practical applications. These limitations make it urgent to seek innovation and improvement.

[0005] "Eye movement data processing method and electronic device CN116661587A": The present application provides an eye movement data processing method and electronic device, the method comprising: acquiring an image to be tested; correcting a first calibration image according to head posture data of the image to be tested to obtain a plurality of second calibration images and a calibrated position of a gaze point of each second calibration image; determining a predicted position of a gaze point of the image to be tested according to the image to be tested, a plurality of second calibration images, and the calibrated position of a gaze point of each second calibration image; triggering an electronic device to perform a corresponding operation according to the predicted position of the gaze point of the image to be tested.

[0006] "A content interaction method based on MR smart glasses CN116909408A": The present invention discloses a content interaction method based on MR smart glasses, which relates to the field of intelligent interaction technology, including judging the user's state and the user's demand attributes, and obtaining a local image of the visual gaze area; performing artificial intelligence processing on the obtained image content, and according to the processing results, prioritizing the notification icons of related applications, services and information; according to the user's demand attributes and priority ranking, searching the database to obtain answers or opening related applications, and presenting them on the optical display.

[0007] "An eye tracking method, device, equipment and storage medium CN111966219A": The present invention discloses an eye tracking method, device, equipment and storage medium, including: obtaining eye images of people wearing glasses and eye images of people without glasses as data sets; training a neural network model based on the data sets; preprocessing the eye images of people wearing glasses based on the neural network model; and performing eye tracking based on the preprocessed images.

[0008] Although vision-based eye tracking technology has made significant progress in many fields, there are still some shortcomings and deficiencies, which may have different degrees of impact in different application scenarios. The following are some common shortcomings:

[0009] 1. Sensitivity is affected by the environment: Factors such as lighting conditions, background noise, and ambient light may affect the sensitivity of vision-based eye tracking systems. In complex or uncontrolled environments, the system may behave unstable.

[0010] 2. Individual differences: There are differences between different individuals, such as pupil size, iris texture, etc., which may affect the accuracy of the eye tracking system. Some systems may require personalized calibration, which may be inconvenient for users.

[0011] 3. Motion artifacts: Rapid eye movements can cause motion artifacts, which can make it difficult for the system to accurately track the position of the eye. This can be a challenge for some applications that require high precision, such as virtual reality and simulation.

[0012] 4. Equipment cost and complexity: High-precision eye tracking equipment is usually expensive, and some complex systems may require professional hardware and software support, which increases the deployment and maintenance costs of the system.

[0013] 5. Requires additional equipment: Most vision-based eye tracking technologies require users to wear additional equipment, such as an eye tracker or camera, which may not be suitable for some special groups or long-term usage scenarios.

[0014] 6. Impact of head movement: Head movement may cause changes in the position of the eyeball relative to the camera, which can negatively affect the accuracy of eye tracking. Some systems need to track head movement for correction.

[0015] 7. Real-time requirements: For some applications with high real-time requirements, such as driver assistance systems, eye tracking systems may require faster response times, and some systems may have challenges in this regard.

[0016] 8. Unnatural experimental environment: Some experimental environments may be unnatural, causing users to behave differently from their daily lives when using the eye tracking system, thereby reducing the reliability of the experimental results. Summary of the invention

[0017] In view of the above problems, the present invention provides an eye movement and sight tracking method, device, equipment and medium for overcoming the above problems or at least partially solving the above problems. The limitations of the prior art are solved and the robustness in complex environments is improved by introducing advanced image processing algorithms.

[0018] The present invention provides the following scheme:

[0019] An eye movement and sight tracking method, comprising:

[0020] Obtaining an RGB image of the current frame of the eye of the life object to be tested, an RGB image of the previous frame, and a thermal map predicted by the previous frame; the thermal map is used to indicate the position and confidence of the target center point in the previous frame;

[0021] Inputting the RGB image of the current frame, the RGB image of the previous frame, and the heat map into a deep learning point tracking network to obtain a prediction of key eye movement feature points to form an eye movement trajectory;

[0022] The deep learning point tracking network is used to perform the following operations:

[0023] Using a convolution layer and a batch normalization layer to add the RGB image of the current frame, the RGB image of the previous frame, and the heat map bit by bit to obtain a fused feature map;

[0024] The fused feature map is input into a fully convolutional neural network so that the fully convolutional neural network predicts a heat map branch, an offset branch, a size branch, and an association branch; the heat map branch is used to output the position and confidence of the center point of objects of different categories; the offset branch is used to output the offset of the target center point relative to the grid center; the size branch is used to output the width and height of the target; the association branch is used to output the displacement vector of the target center point relative to the center point of the previous frame;

[0025] Extracting and obtaining a number of peak points from the heat map branch of each category;

[0026] Extracting the number of points of the front target from the peak points as the detected center points;

[0027] For each of the center points, the bounding box coordinates of each of the center points in the current frame and the previous frame, as well as the association relationship between each of the center points and the center point of the previous frame are calculated based on the outputs of the corresponding offset branch, the size branch, and the association branch.

[0028] Preferably: if the value of a point extracted from the branch of the heat map is greater than or equal to the values ​​of points in its 8-neighborhood, then the point is determined to be a peak point.

[0029] Preferably, the target number is 100.

[0030] Preferably: the deep learning point tracking network includes a backbone network, a target detection head and a target tracking head; the backbone network is used to obtain high-level semantic information of the image; the target detection head is used to generate information graphs of target center points, bounding boxes and key points; the target tracking head is used to utilize target information of previous frames to achieve long-term tracking of the target.

[0031] Preferably, the backbone network includes any one of the Hourglass algorithm and the ResNet algorithm.

[0032] Preferably: the loss function of the network used in the deep learning point tracking network training process is expressed by the following formula:

[0033]

[0034] Where: t represents the tth frame, t-1 represents the previous frame of the current frame Indicates the offset of the target center point between the previous and next frames.

[0035] Preferably:

[0036]

[0037] Where: Represents the detected center point of the target.

[0038] An eye movement and sight tracking device, used to execute the above-mentioned eye movement and sight tracking method, the device comprising:

[0039] An image acquisition unit, used to acquire an RGB image of the current frame of the eye of the life body to be tested, an RGB image of the previous frame, and a thermal map predicted by the previous frame; the thermal map is used to indicate the position and confidence of the target center point in the previous frame;

[0040] An eye tracking unit, used for inputting the RGB image of the current frame, the RGB image of the previous frame and the heat map into a deep learning point tracking network to obtain a prediction of key eye movement feature points to form an eye movement trajectory;

[0041] The deep learning point tracking network is used to perform the following operations:

[0042] Using a convolution layer and a batch normalization layer to add the RGB image of the current frame, the RGB image of the previous frame, and the heat map bit by bit to obtain a fused feature map;

[0043] The fused feature map is input into a fully convolutional neural network so that the fully convolutional neural network predicts a heat map branch, an offset branch, a size branch, and an association branch; the heat map branch is used to output the position and confidence of the center point of objects of different categories; the offset branch is used to output the offset of the target center point relative to the grid center; the size branch is used to output the width and height of the target; the association branch is used to output the displacement vector of the target center point relative to the center point of the previous frame;

[0044] Extracting and obtaining a number of peak points from the heat map branch of each category;

[0045] Extracting the number of points of the front target from the peak points as the detected center points;

[0046] For each of the center points, the bounding box coordinates of each of the center points in the current frame and the previous frame, as well as the association relationship between each of the center points and the center point of the previous frame are calculated based on the outputs of the corresponding offset branch, the size branch, and the association branch.

[0047] An eye tracking device, comprising a processor and a memory:

[0048] The memory is used to store program code and transmit the program code to the processor;

[0049] The processor is used to execute the above-mentioned eye movement and gaze tracking method according to the instructions in the program code.

[0050] A computer-readable storage medium is used to store program codes, and the program codes are used to execute the above-mentioned eye movement and gaze tracking method.

[0051] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0052] An eye movement and gaze tracking method, apparatus, device and medium provided in the embodiments of the present application use a deep learning model to train with a large amount of data, thereby improving the prediction accuracy of eye movement and gaze tracking. This high accuracy helps to ensure that the eye movement and gaze tracking system has excellent performance in different scenarios and individuals, providing reliable data support for medical applications. The deep learning model is adaptive and can be adjusted according to the eye movement behavior characteristics of different patients and different environments. This makes the eye movement and gaze tracking system more versatile, suitable for a wide range of medical application scenarios, and can adapt to individual differences. The efficient performance of the deep learning model enables the eye movement and gaze tracking system to process and analyze eye movement data in real time.

[0053] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0055] Figure 1 is a flow chart of an eye movement and sight tracking method provided by an embodiment of the present invention;

[0056] Figure 2 It is a processing flow and a schematic diagram of a network structure provided by an embodiment of the present invention;

[0057] Figure 3 is a flow chart of model training and eye tracking provided by an embodiment of the present invention;

[0058] Figure 4 is a schematic diagram of an eye movement and sight tracking device provided by an embodiment of the present invention;

[0059] Figure 5 It is a schematic diagram of an eye movement and gaze tracking device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0061] See also Figure 1, is an eye movement and sight tracking method provided by an embodiment of the present invention, such as Figure 1 As shown, the method may include:

[0062] S101: obtaining an RGB image of a current frame, an RGB image of a previous frame, and a predicted heat map of the previous frame of the eye of the life object to be tested; the heat map is used to represent the position and confidence of the target center point in the previous frame;

[0063] S102: Inputting the RGB image of the current frame, the RGB image of the previous frame, and the heat map into a deep learning point tracking network to obtain a prediction of key eye movement feature points to form an eye movement trajectory;

[0064] The deep learning point tracking network is used to perform the following operations:

[0065] Using a convolution layer and a batch normalization layer to add the RGB image of the current frame, the RGB image of the previous frame, and the heat map bit by bit to obtain a fused feature map;

[0066] The fused feature map is input into a fully convolutional neural network so that the fully convolutional neural network predicts a heat map branch, an offset branch, a size branch, and an association branch; the heat map branch is used to output the position and confidence of the center point of objects of different categories; the offset branch is used to output the offset of the target center point relative to the grid center; the size branch is used to output the width and height of the target; the association branch is used to output the displacement vector of the target center point relative to the center point of the previous frame;

[0067] A number of peak points are extracted from the heat map branch of each category; in specific implementation, the embodiment of the present application can provide that if the value of a point extracted from the heat map branch is greater than or equal to the value of the points in its 8 neighborhoods, then this point is determined to be a peak point.

[0068] The points of the preceding target quantity are extracted from the plurality of peak points as the detected central points; in specific implementation, the embodiment of the present application may provide that the target quantity is 100.

[0069] For each of the center points, the bounding box coordinates of each of the center points in the current frame and the previous frame, as well as the association relationship between each of the center points and the center point of the previous frame are calculated based on the outputs of the corresponding offset branch, the size branch, and the association branch.

[0070] Furthermore, the deep learning point tracking network includes a backbone network, a target detection head and a target tracking head; the backbone network is used to obtain high-level semantic information of the image; the target detection head is used to generate information maps of target center points, bounding boxes and key points; the target tracking head is used to utilize target information of previous frames to achieve long-term tracking of the target. The backbone network includes any one of the Hourglass algorithm and the ResNet algorithm.

[0071] The loss function of the network used in the deep learning point tracking network training process is expressed as follows:

[0072]

[0073] Where t represents the tth frame, and t-1 represents the previous frame of the current frame. Indicates the offset of the target center point between the previous and next frames.

[0074]

[0075] Where: Represents the detected center point of the target.

[0076] The eye tracking method provided in the embodiment of the present application uses key points and a tracking network to locate and track pupils. The method achieves efficient and accurate target tracking by organically integrating detection and tracking tasks. It uses center points to represent targets and predicts their association with the previous frame. The advantages of this method are that it is simple, fast, and accurate, and does not require the use of anchor boxes and non-maximum suppression, nor does it require reinitialization of lost tracking.

[0077] This method achieves efficient detection and accurate tracking of eye movement behavior by introducing a key point tracking network based on deep learning. The network can adapt to a variety of scenarios and complex environments, thus providing a reliable basis for eye movement tracking. It can be applied to the medical field of the disabled to provide eye movement support for patients with impaired functions. This includes but is not limited to patients with movement disorders and language disorders, providing them with a more convenient and flexible way of interaction and promoting the full recovery of functions. The introduction of deep learning networks makes eye movement tracking more real-time and accurate. This is crucial for medical applications to ensure that patients can receive timely support and feedback, thereby maximizing the effectiveness of treatment.

[0078] This method is intended to be applied in the medical field of disabled people to improve the quality of life and medical service effect of disabled patients. The current eye tracking technology has limitations in accuracy, real-time and applicability in certain situations, especially for the needs of disabled people.

[0079] By combining deep learning algorithms with point tracking technology, this method aims to overcome the limitations of traditional eye tracking technology and improve the accuracy and sensitivity of eye movement behavior of disabled patients. This method can be applied to the medical field of disabled people, such as rehabilitation therapy, communication assistance, and control of intelligent auxiliary equipment, providing disabled patients with a more accurate, convenient, and personalized eye movement interaction experience.

[0080] Through the application of this method, it is expected to provide more independence and improve the quality of life for people with disabilities, and provide medical professionals with more powerful tools to better understand and support the rehabilitation process of disabled patients. It aims to promote the development of eye movement technology, make it closer to medical practice, and bring innovative solutions to the medical field of people with disabilities.

[0081] The deep learning eye movement and gaze tracking method based on point tracking proposed in the embodiment of the present application is based on the key point tracking network, and aims to solve the technical problems of eye movement and gaze detection and tracking in the medical field of the disabled. The key point of this technology is that it comprehensively uses advanced deep learning algorithms, especially the key point tracking network, to achieve accurate and real-time tracking of eye movement behavior.

[0082] The key points of the method provided in the embodiment of the present application include but are not limited to the following aspects:

[0083] 1. Keypoint Tracking Network: By introducing the keypoint tracking network, efficient detection and accurate tracking of eye movement behavior can be achieved. This network is known for its excellent target tracking performance and can adapt to a variety of scenarios and complex environments, thus providing a reliable foundation for eye movement tracking.

[0084] 2. Application of point tracking technology: By using point tracking technology, we can accurately calibrate and track key points in the eye movement process to achieve high-precision restoration of eye movement trajectories. The application of this technology enables us to analyze eye movement behavior more carefully and provide more accurate functional recovery support for the medical field of the disabled.

[0085] 3. Wide application in the medical field of the disabled: This method is mainly used in the medical field of the disabled to provide customized eye movement support for patients with impaired functions. This includes but is not limited to patients with movement disorders and language disorders, providing them with a more convenient and flexible way of interaction and promoting the full recovery of functions.

[0086] 4. Improved real-time performance and accuracy: Through the introduction of deep learning networks, eye tracking has higher real-time performance and accuracy. This is crucial for medical applications, ensuring that patients can receive timely support and feedback, thereby maximizing treatment effectiveness.

[0087] The unique combination of the above key technical points forms the innovation and practicality of this patent. The core to be protected lies in this whole set of technical systems and its application in the medical field of disabled people, which brings significant progress and application prospects to the field of eye movement and gaze tracking technology.

[0088] The principles and processes of the deep learning point tracking network provided by the embodiment of the present application are as follows:

[0089] The deep learning point tracking network has three inputs, namely the RGB image of the current frame, the RGB image of the previous frame, and the heat map predicted by the previous frame. The heat map represents the position and confidence of the target center point in the previous frame, which is used to provide timing information. After these three inputs pass through some convolutional layers and batch normalization layers, they are bitwise added to obtain a fused feature map.

[0090] The deep learning point tracking network uses some fully convolutional neural networks with different encoding-decoding structures, such as ResNet, DLA, Hourglass, etc., and predicts four output branches from the fused feature map, namely:

[0091] Heat map branch, size is Output the location and confidence of the center points of objects of different categories.

[0092] Offset branch, size Output the offset of the target center point relative to the grid center to improve positioning accuracy.

[0093] Dimension branch, size Outputs the width and height of the object, used to generate the bounding box.

[0094] The associated branch is Output the displacement vector of the target center point relative to the center point of the previous frame to achieve timing association.

[0095] In the inference stage, the peak points are first extracted from the heat map branch of each category. If the value of a point is greater than or equal to the value of the points in its 8 neighborhoods, then this point is considered to be a peak point, and then the first 100 points are selected from these candidate peak points as the detected center points. For each detected center point, the bounding box coordinates of the center point in the current frame and the previous frame are calculated according to the output of its offset branch, size branch, and association branch, as well as its association with the center point of the previous frame. In the end, all target information is directly obtained through the above-mentioned center point estimation method without the need for IoU-based non-maximum suppression or other post-processing operations.

[0096] Overall, if Figure 2 As shown in Figure 2, the structure of the deep learning point tracking network consists of three main parts:

[0097] 1. Backbone network:

[0098] This network uses advanced backbone networks (such as Hourglass or ResNet) to extract image features to obtain high-level semantic information in the image.

[0099] 2. Target Detection Head:

[0100] After the backbone network, the object detection head generates information maps of the object center point, bounding box, and key points. This step involves multi-scale convolution operations on the feature map of the backbone network to effectively capture the multi-scale information of the object.

[0101] 3. Tracking Head:

[0102] The network introduces a target tracking head, which achieves long-term tracking of the target by utilizing the target information of the previous frame. This part further improves the accuracy of target tracking by generating the target's motion vector and appearance features.

[0103] The output of the entire network structure is the center point, bounding box, key points and motion information of the target, which enables the network to achieve efficient detection and long-term tracking of the target in a unified framework. This method of integrating detection and tracking enables the network to perform well in target tracking tasks, especially in complex scenes. The loss function of the network is:

[0104]

[0105] in represents the detected target center point, t represents the specific position of the time series, that is, the tth frame, t-1 represents the previous frame of the current frame, Indicates the offset of the target center point position between the previous and next two frames, that is,

[0106]

[0107] like Figure 2 As shown, the processing flow and network structure of the network are shown.

[0108] The method provided in the embodiment of the present application includes the training of a deep learning model and the step of tracking eye movements, such as Figure 3 The detailed steps of the two parts are as follows:

[0109] 1. Dataset preparation: The eye movement data of patients in different scenarios are collected through high-resolution camera equipment. Each data sample includes an eye movement image and the corresponding real eye movement annotation, which is used to guide the training of the deep learning point tracking network.

[0110] 2. Network architecture design: Design the architecture of the deep learning point tracking network to ensure that the network can accurately capture the characteristics of eye movement behavior. The output of the network should contain accurate predictions of key eye movement feature points to form eye movement trajectories.

[0111] 3. Loss function definition: Define the loss function used for training to measure the difference between the model prediction results and the actual eye movement annotations. The loss function usually includes regression loss to ensure that the model can accurately regress the eye movement trajectory.

[0112] 4. Data enhancement: Perform data enhancement on eye movement data, including rotation, scaling, flipping, etc., to expand the training set and improve the generalization ability of the model.

[0113] 5. Model training: Using the prepared eye movement dataset, the model is trained through a deep learning point tracking network. During the training process, the model continuously optimizes the network parameters through the back propagation algorithm, so that it gradually improves the prediction accuracy of eye movement behavior.

[0114] 6. Validation and tuning: Use the validation set to evaluate the performance of the model during training, and tune the model parameters based on the validation results to improve the generalization performance of the model.

[0115] 7. Eye tracking: The trained deep learning point tracking network is embedded in the eye tracking system. In practical applications, through the eye movement data collected by the system in real time, the deep learning model can track the patient's eye movement trajectory in real time and form accurate eye tracking results.

[0116] 8. Real-time feedback and personalized optimization: The system not only provides accurate eye tracking results through real-time feedback from the deep learning model, but also performs personalized optimization based on real-time data analysis. This helps adjust the parameters of the tracking model to adapt to the eye movement characteristics and needs of different patients.

[0117] 9. Results presentation and analysis: Through accurate tracking of the deep learning model, the system can present visual results of the patient's eye movement behavior, including eye movement trajectory diagrams and changing trends of key points. This provides medical professionals with clear and intuitive eye movement data to support applications in medical decision-making and rehabilitation training.

[0118] In summary, the eye movement and gaze tracking method provided in this application uses a deep learning model to train with a large amount of data, thereby improving the prediction accuracy of eye movement and gaze tracking. This high accuracy helps to ensure that the eye movement and gaze tracking system has excellent performance in different scenarios and individuals, providing reliable data support for medical applications. The deep learning model is adaptive and can be adjusted according to the eye movement behavior characteristics of different patients and different environments. This makes the eye movement and gaze tracking system more versatile, suitable for a wide range of medical application scenarios, and can adapt to individual differences. The efficient performance of the deep learning model enables the eye movement and gaze tracking system to process and analyze eye movement data in real time.

[0119] See also Figure 4 , the embodiment of the present application can also provide an eye movement tracking device, such as Figure 4 As shown, the device for executing the above-mentioned eye movement and sight tracking method may include:

[0120] The image acquisition unit 401 is used to acquire the RGB image of the current frame of the eye of the life body to be tested, the RGB image of the previous frame, and the thermal map predicted by the previous frame; the thermal map is used to represent the position and confidence of the target center point in the previous frame;

[0121] An eye tracking unit 402, configured to input the RGB image of the current frame, the RGB image of the previous frame, and the heat map into a deep learning point tracking network to obtain a prediction of key eye movement feature points to form an eye movement trajectory;

[0122] The deep learning point tracking network is used to perform the following operations:

[0123] Using a convolution layer and a batch normalization layer to add the RGB image of the current frame, the RGB image of the previous frame, and the heat map bit by bit to obtain a fused feature map;

[0124] The fused feature map is input into a fully convolutional neural network so that the fully convolutional neural network predicts a heat map branch, an offset branch, a size branch, and an association branch; the heat map branch is used to output the position and confidence of the center point of objects of different categories; the offset branch is used to output the offset of the target center point relative to the grid center; the size branch is used to output the width and height of the target; the association branch is used to output the displacement vector of the target center point relative to the center point of the previous frame;

[0125] Extracting and obtaining a number of peak points from the heat map branch of each category;

[0126] Extracting the number of points of the front target from the peak points as the detected center points;

[0127] For each of the center points, the bounding box coordinates of each of the center points in the current frame and the previous frame, as well as the association relationship between each of the center points and the center point of the previous frame are calculated based on the outputs of the corresponding offset branch, the size branch, and the association branch.

[0128] The present application may also provide an eye tracking device, the device comprising a processor and a memory:

[0129] The memory is used to store program code and transmit the program code to the processor;

[0130] The processor is used to execute the steps of the above-mentioned eye movement and gaze tracking method according to the instructions in the program code.

[0131] like Figure 5 As shown, an eye tracking device provided by an embodiment of the present application may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 communicate with each other through the communication bus 13.

[0132] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices, etc.

[0133] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in an embodiment of the eye movement and gaze tracking method.

[0134] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:

[0135] Obtaining an RGB image of the current frame of the eye of the life object to be tested, an RGB image of the previous frame, and a thermal map predicted by the previous frame; the thermal map is used to indicate the position and confidence of the target center point in the previous frame;

[0136] Inputting the RGB image of the current frame, the RGB image of the previous frame, and the heat map into a deep learning point tracking network to obtain a prediction of key eye movement feature points to form an eye movement trajectory;

[0137] The deep learning point tracking network is used to perform the following operations:

[0138] Using a convolution layer and a batch normalization layer to add the RGB image of the current frame, the RGB image of the previous frame, and the heat map bit by bit to obtain a fused feature map;

[0139] The fused feature map is input into a fully convolutional neural network so that the fully convolutional neural network predicts a heat map branch, an offset branch, a size branch, and an association branch; the heat map branch is used to output the position and confidence of the center point of objects of different categories; the offset branch is used to output the offset of the target center point relative to the grid center; the size branch is used to output the width and height of the target; the association branch is used to output the displacement vector of the target center point relative to the center point of the previous frame;

[0140] Extracting and obtaining a number of peak points from the heat map branch of each category;

[0141] Extracting the number of points of the front target from the peak points as the detected center points;

[0142] For each of the center points, the bounding box coordinates of each of the center points in the current frame and the previous frame, as well as the association relationship between each of the center points and the center point of the previous frame are calculated based on the outputs of the corresponding offset branch, the size branch, and the association branch.

[0143] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required for at least one function (such as a file creation function, a data reading and writing function), etc.; the data storage area can store data created during use, such as initialization data, etc.

[0144] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0145] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0146] Of course, it should be noted that Figure 5 The structure shown does not constitute a limitation on the eye movement and sight tracking device in the embodiment of the present application. In actual applications, the eye movement and sight tracking device may include Figure 5 More or fewer components than shown, or combinations of certain components.

[0147] The embodiment of the present application may also provide a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the steps of the above-mentioned eye movement and gaze tracking method.

[0148] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0149] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0150] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0151] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. An eye movement tracking method, characterized in that: include: Obtaining the RGB image of the current frame, the RGB image of the previous frame, and the predicted thermal map of the previous frame of the eye of the life form to be tested; The heat map is used to represent the position and confidence of the target center point in the previous frame; Inputting the RGB image of the current frame, the RGB image of the previous frame, and the heat map into a deep learning point tracking network to obtain a prediction of key eye movement feature points to form an eye movement trajectory; The deep learning point tracking network is used to perform the following operations: Using a convolution layer and a batch normalization layer to add the RGB image of the current frame, the RGB image of the previous frame, and the heat map bit by bit to obtain a fused feature map; The fused feature map is input into a fully convolutional neural network so that the fully convolutional neural network predicts a heat map branch, an offset branch, a size branch, and an association branch; the heat map branch is used to output the position and confidence of the center point of objects of different categories; the offset branch is used to output the offset of the target center point relative to the grid center; the size branch is used to output the width and height of the target; The association branch is used to output a displacement vector of the target center point relative to the center point of the previous frame; Extracting and obtaining a number of peak points from the heat map branch of each category; Extracting the number of points of the front target from the peak points as the detected center points; For each of the center points, the bounding box coordinates of each of the center points in the current frame and the previous frame, as well as the association relationship between each of the center points and the center point of the previous frame are calculated based on the outputs of the corresponding offset branch, the size branch, and the association branch.

2. The eye movement tracking method according to claim 1, characterized in that: If the value of a point extracted from the heat map branch is greater than or equal to the value of the points in its 8-neighborhood, then this point is determined to be a peak point.

3. The eye movement tracking method according to claim 1, characterized in that: The target quantity is 100.

4. The eye movement tracking method according to claim 1, characterized in that: The deep learning point tracking network includes a backbone network, a target detection head and a target tracking head; the backbone network is used to obtain high-level semantic information of the image; the target detection head is used to generate information graphs of target center points, bounding boxes and key points; the target tracking head is used to utilize target information of previous frames to achieve long-term tracking of the target.

5. The eye movement tracking method according to claim 4, characterized in that: The backbone network includes any one of the Hourglass algorithm and the ResNet algorithm.

6. The eye movement tracking method according to claim 1, characterized in that: The loss function of the network used in the deep learning point tracking network training process is expressed as follows: Where: t represents the tth frame, t-1 represents the previous frame of the current frame, Indicates the offset of the target center point between the previous and next frames.

7. The eye movement tracking method according to claim 6, characterized in that: Where: Represents the detected target center point.

8. An eye movement tracking device, characterized in that: The device is used to execute the eye movement and sight tracking method according to any one of claims 1 to 7, comprising: An image acquisition unit, used to acquire an RGB image of the current frame of the eye of the life body to be tested, an RGB image of the previous frame, and a thermal map predicted by the previous frame; the thermal map is used to indicate the position and confidence of the target center point in the previous frame; An eye tracking unit, used for inputting the RGB image of the current frame, the RGB image of the previous frame and the heat map into a deep learning point tracking network to obtain a prediction of key eye movement feature points to form an eye movement trajectory; The deep learning point tracking network is used to perform the following operations: Using a convolution layer and a batch normalization layer to add the RGB image of the current frame, the RGB image of the previous frame, and the heat map bit by bit to obtain a fused feature map; The fused feature map is input into a fully convolutional neural network so that the fully convolutional neural network predicts a heat map branch, an offset branch, a size branch, and an association branch; the heat map branch is used to output the position and confidence of the center point of objects of different categories; the offset branch is used to output the offset of the target center point relative to the grid center; the size branch is used to output the width and height of the target; the association branch is used to output the displacement vector of the target center point relative to the center point of the previous frame; Extracting and obtaining a number of peak points from the heat map branch of each category; Extracting the number of points of the front target from the peak points as the detected center points; For each of the center points, the bounding box coordinates of each of the center points in the current frame and the previous frame, as well as the association relationship between each of the center points and the center point of the previous frame are calculated based on the outputs of the corresponding offset branch, the size branch, and the association branch.

9. An eye movement tracking device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the eye movement tracking method according to any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the eye movement tracking method described in any one of claims 1-7.

Images