A high-speed bright pupil eye movement detection and tracking method and system based on event camera

Abstract

The application provides a high-speed bright pupil eye movement tracking method based on an event camera, comprising the following steps: S1, data acquisition: using an event camera in cooperation with an infrared flicker light source to collect human eye data in different scenes; S2, data conversion: converting the data collected by the event camera into frame data; S3, data labeling: labeling the converted data to mark the position of the human eye pupil, and inputting the labeled data into step S4; S4, model training: training the backbone network of a feature extraction model, performing feature extraction training by using a bright pupil detection module, and setting the trained backbone network of the feature extraction model into a bright pupil detection convolutional network and a bright pupil tracking convolutional network; and S5, model conversion: converting the trained bright pupil detection convolutional network and the bright pupil tracking convolutional network. The application can realize dynamic and static data collection of the pupil by the event camera, and realize low-power and high-speed detection and tracking of the pupil by combining with a pulse neural network technology.

Description

Technical Field

[0001] This invention relates to a high-speed bright pupil eye movement detection and tracking method and system based on an event camera, and more particularly, to a high-speed bright pupil eye movement detection and tracking method and system based on an event camera. Background Technology

[0002] Eye-tracking technology determines the direction of a person's gaze by tracking the pupil. Traditional eye-tracking systems are generally divided into bright pupil and dark pupil tracking. These methods utilize an external light source to illuminate the eye. Because the pupil is a dark area with strong light absorption, it is easily distinguishable from the surrounding eye area, facilitating detection and tracking. Bright pupil means that the external light source and the eyeball are on the same light path, while dark pupil means that the external light source and the eyeball are not on the same light path.

[0003] Eye tracking has a wide range of applications. In VR / AR, by tracking eye gaze, the range of human attention can be determined, allowing for more computational resources to be allocated to render the areas of focus more precisely, resulting in a better visual experience. Furthermore, eye tracking has broad application value in customer interest analysis, fatigue detection, assessment of childhood motor disorders, motion analysis, and strabismus diagnosis. Internationally renowned manufacturers of eye tracking devices include Tobii and Pupil Labs. However, their products often suffer from issues such as large device size, high power consumption, and significant heat generation. Due to the rapid frequency of eye movements, a high sampling frequency is often required to obtain accurate data. Therefore, a higher sampling rate ensures more accurate sampling of real eye movements and reduces the chance of missing subtle eye movements.

[0004] However, the main problem is that eye tracking, as a wearable device widely used in VR, XR, and other fields, has always faced the challenge of balancing performance and power consumption using traditional visual tracking solutions. Good visual tracking solutions often suffer from high data volume and high power consumption, limiting the application and promotion of eye tracking. Furthermore, with the increase in sampling frequency, challenges are posed to camera power consumption and data transmission. In addition, due to the low dynamic range of traditional cameras, when using the bright-eye method, traditional cameras, with their limited dynamic range, cannot adapt to tracking different colored eyes.

[0005] Event cameras are a novel type of sensor inspired by human and animal vision. They feature high sampling frequencies, low power consumption, and high dynamic range, and respond only to moving targets, making them ideal for eye tracking. Spiking neural networks, a new type of network inspired by the human brain, can directly process pulse signals. They are a natural fit with event cameras, maximizing their advantages.

[0006] The difficulty in solving the above problems and defects lies in the fact that it is quite difficult to solve the above problems through traditional methods and strategies. Traditional methods cannot balance the contradiction between sampling frequency, data redundancy, and power consumption.

[0007] The significance of addressing the above problems and shortcomings is that it provides the industry with a new hardware and software algorithm. This novel solution effectively solves the problems of high device power consumption and excessive data redundancy in current pupil detection and tracking methods. The proposed solution uses a combination of an event camera and an infrared light source, which improves the accuracy of pupil detection and tracking, demonstrating novelty and advancement. Summary of the Invention

[0008] This invention provides a high-speed bright pupil eye movement detection and tracking method and system based on an event camera. It adopts a combination of an event camera and an infrared light source. The high sampling frequency and wide dynamic range of the event camera enable high-speed and low-power eye movement tracking. At the same time, the use of a flashing infrared light source enables the event camera to acquire dynamic and static data of the pupil. Combined with pulse neural network technology, it achieves low-power and high-speed detection and tracking of the pupil.

[0009] The technical solution of the present invention is as follows:

[0010] According to one aspect of the present invention, a high-speed pupil detection and tracking method based on an event camera is provided, comprising the following steps: S1. Data acquisition: using an event camera in conjunction with an infrared scintillation light source, human eye data in different scenarios is acquired, and the acquired data is filtered and denoised; S2. Data conversion: the data acquired by the event camera is converted into frame data; S3. Data annotation: the converted data is annotated to mark the location of the human pupil, and the annotated data is input into step S4 for training; S4. Model training: the backbone network of the feature extraction model is trained, the pupil detection module is used for feature extraction training, and the trained backbone network of the feature extraction model is set into the pupil detection convolutional network and the pupil tracking convolutional network; S5. Model conversion: the trained pupil detection convolutional network and the pupil tracking convolutional network are converted into a pulse network model.

[0011] Optionally, in the above-mentioned high-speed bright pupil eye movement detection and tracking method based on event camera, in step S1, the human eye data includes human eye data in indoor and outdoor scenes with different lighting conditions.

[0012] Optionally, in the above-mentioned high-speed bright pupil eye movement detection and tracking method based on event camera, in step S3, the location of the human eye pupil is marked by manual annotation.

[0013] Optionally, in the above-mentioned high-speed bright pupil eye movement detection and tracking method based on event camera, in step S4, the bright pupil detection convolutional network uses a convolutional network to detect bright pupils, transforms all neural network outputs, performs absolute value transformation on the network output of each layer, and sets all activation functions to ReLU activation functions. The bright pupil detection convolutional network mainly performs bright pupil detection, provides a tracking template for bright pupil tracking, and relocates the bright pupil position after tracking failure.

[0014] Optionally, in the above-mentioned high-speed bright-eye detection and tracking method based on event cameras, in step S4, the bright-eye tracking convolutional network adopts a Siamese network tracking model, the backbone network adopts the same convolutional network as the bright-eye detection convolutional network, and two parallel input networks are set up for data processing. The two parallel networks are divided into a template network and a target network, and the weights of the two networks are shared. The features of the template and the target are extracted respectively, and the position of the tracking target is determined by the cross-correlation operation of feature comparison.

[0015] According to another aspect of the present invention, a high-speed bright pupil eye movement detection and tracking system based on an event camera is provided, comprising: a light source and acquisition module, a bright pupil detection module, a bright pupil tracking module, and a system calibration module, wherein: the light source and acquisition module includes a front-facing camera, an event camera, and a flashing infrared light source, wherein the flashing infrared light source and the event camera are on the same optical path, the infrared light source illuminates the pupil, the event camera receives the light data returned by the pupil, the event camera is used to acquire pupil information, and the front-facing camera is used to acquire the scene observed by the human eye and fuse the position of the gaze in the system; the bright pupil detection module uses a convolutional network to detect bright pupils, providing a tracking template for bright pupil tracking and handling of tracking failures. The system repositions the target's position. The target tracking module uses a Siamese network tracking model. The backbone network uses the same convolutional network as the target detection module. Two parallel input networks are set up for data processing. The two parallel networks are a template network and a target network. The weights of the two networks are shared. The features of the template and the target are extracted respectively. The position of the tracking target is determined by the cross-correlation operation of feature comparison. The system calibration module performs gaze calibration for each user. The image features of the target are stored in the system. The target is placed in front of the user's eyes. The tester needs to focus on looking at the target and move randomly to multiple positions. The system automatically identifies the target and calibrates the parameters within the system. When the target position of the visual tracking point coincides multiple times, the system calibration is completed.

[0016] Optionally, in the above-mentioned high-speed bright pupil eye movement detection and tracking system based on event camera, the flashing frequency of the infrared light source is set to one-tenth of the sampling frequency, and the infrared light source is an invisible infrared light source.

[0017] The beneficial effects of the technical solution of the present invention are as follows:

[0018] This invention proposes a high-speed pupil detection and tracking method and system based on an event camera. It employs a brain-inspired event sensor and neural impulse network. This invention can achieve high-speed pupil tracking with high sampling frequency and low power consumption. This invention simultaneously achieves low power consumption and high efficiency in both data acquisition and intelligent data processing stages, reducing the cost of eye tracking and benefiting the industry's development.

[0019] To better understand and illustrate the concept, working principle, and effects of this invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments: Attached Figure Description

[0020] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below.

[0021] Figure 1 This is a flowchart of the high-speed bright pupil eye movement detection and tracking method based on an event camera according to the present invention;

[0022] Figure 2 This is a schematic diagram of the device layout of the high-speed bright pupil eye movement detection and tracking system based on an event camera according to the present invention;

[0023] Figure 3 This is the bullseye of the calibration procedure in this invention;

[0024] Figure 4 This is a flowchart of a preferred embodiment of the high-speed bright pupil eye movement detection and tracking method based on an event camera according to the present invention. Detailed Implementation

[0025] To make the objectives, technical methods, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific examples. These examples are merely illustrative and not intended to limit the invention.

[0026] This invention provides a high-speed bright pupil eye movement detection and tracking method and system based on an event camera. The invention acquires pupil data by combining an event camera and a flashing infrared light source, and uses a spiking neural network to detect and track the pupil.

[0027] The principle of this invention is:

[0028] A pupil tracking method combining event cameras and flashing infrared light is employed. Leveraging the event camera's ability to capture only moving targets, a flashing infrared light source is placed on the same optical path as the human pupil. The flashing of the light source is reflected off the pupil, allowing the event camera to accurately capture both static and dynamic pupil information. The information acquired by the event camera is input into a spiking neural network. After detecting the pupil, the spiking neural network efficiently tracks pupil information. The spiking neural network is derived from a pre-trained convolutional network through a model conversion module.

[0029] This invention employs pulse data and pulse network processing throughout the entire process, achieving a method for bright-eye tracking with low data acquisition power consumption, low data transmission bandwidth, and low data processing power consumption. Figure 1 As shown, the high-speed bright pupil eye movement detection and tracking method based on an event camera of the present invention includes the following steps:

[0030] S1. Data Acquisition: Using an event camera in conjunction with an infrared flashing light source, human eye data is collected in different scenarios, including indoor and outdoor scenarios with different lighting conditions. The collected data is then filtered and denoised.

[0031] S2. Data Conversion: Convert the data acquired by the event camera into frame data for subsequent annotation and training.

[0032] S3. Data labeling: Label the converted data using manual labeling to mark the location of the human eye pupil. Input the labeled data into step S4 for training.

[0033] S4. Model Training: The main training focuses on the backbone network of the feature extraction model. The feature extraction model is trained using the Mingtong detection module. The trained backbone network of the feature extraction model is then incorporated into the Mingtong detection convolutional network and the Mingtong tracking convolutional network.

[0034] The trained feature extraction backbone network can effectively extract the features of the bright eye. This feature extraction model backbone network can be used in the bright eye detection network and the bright eye tracking network (i.e., the bright eye tracking module).

[0035] The bright eye detection module (i.e., the bright eye detection convolutional network) uses a convolutional network to detect bright eyes. In order for the model conversion in step S5 to be completed smoothly, the existing ResNet (a network structure name) and YOLO (a network structure name) convolutional networks need to be specially designed to convert all neural network outputs, that is, to convert the absolute value of the network output of each layer and set all activation functions to ReLU activation functions. The bright eye detection module mainly performs bright eye detection, provides a tracking template for bright eye tracking, and relocates the bright eye position after tracking failure.

[0036] The Mingtong tracking module (i.e., the Mingtong tracking convolutional network) adopts a Siamese network tracking model. The backbone network uses the same convolutional network as the Mingtong detection module (Mingtong detection convolutional network). Two parallel input networks are set up for data processing. The two parallel networks are divided into a template network and a target network. The two networks share weights and extract the features of the template and the target respectively. The position of the tracking target is determined by the cross-correlation operation of feature comparison.

[0037] S5. Model Conversion: The trained pupil detection convolutional network and pupil tracking convolutional network are converted into neural spiking network models.

[0038] Since current spiking networks cannot perform gradient descent and the impulses are not differentiable, this invention uses a model conversion method by training a bright eye detection convolutional network and a bright eye tracking convolutional network. Specifically, the convolutional network model is converted into a spiking network model, and the neurons of the convolutional network are converted into an integrated-and-fire neuron model.

[0039] The high-speed bright pupil eye-tracking detection and tracking system based on an event camera of the present invention comprises five main modules: a light source and acquisition module, a bright pupil detection module, a bright pupil tracking module, and a system calibration module, wherein:

[0040] Light source and acquisition module: A scintillation infrared light source and an event camera are used on the same optical path. The scintillation frequency of the infrared light source is set to one-tenth of the sampling frequency. The device layout is as follows: Figure 2 As shown, the event camera and the infrared light source are on the same optical path. The infrared light source illuminates the pupil, and the event camera receives the light data returned by the pupil. The infrared light source is invisible to the human eye. The event camera is used to collect pupil information, while the front-facing camera is used to capture the scene observed by the human eye and fuse the location of the line of sight in the system.

[0041] The bright eye detection module uses a convolutional network for bright eye detection. To ensure smooth model conversion, the existing ResNet and YOLO convolutional networks need to be specially designed. All neural network outputs are converted, that is, the absolute value of the network output of each layer is converted, and the activation function is set to ReLU activation function. The bright eye detection module mainly detects bright eyes, provides a tracking template for bright eye tracking, and relocates the bright eye position after tracking failure.

[0042] The Mingtong tracking module employs a Siamese network tracking model. The backbone network uses the same convolutional network as the Mingtong detection module. Two parallel input networks are set up for data processing. These two parallel input networks are a template network and a target network, with shared weights. They extract the features of the template and the target respectively, and determine the position of the tracking target through the cross-correlation operation of feature comparison.

[0043] System calibration module: Due to differences in eye structure among individuals, gaze calibration is required for each individual user. Figure 3 The system displays the bullseye used for calibration procedures. The image features of the bullseye are stored in the system. The bullseye is placed in front of the test subject's eyes, and the test subject needs to focus on looking at the bullseye. The subject moves randomly to multiple positions, and the system automatically identifies the bullseye and calibrates the parameters within the system. When the bullseye positions of the visual tracking points coincide multiple times, the system calibration is completed.

[0044] like Figure 4 As shown, the present invention implements an improved method and system for pupil detection and tracking. It acquires data by combining an event camera and an infrared scintillation light source, and intelligently processes the acquired data using a pulse neural network. The specific implementation steps from start to finish are as follows:

[0045] Beginning: Position the equipment in front of the viewer and adjust the positions of the two cameras. The equipment should not obstruct the viewer's line of sight. After adjustment, perform system calibration.

[0046] System calibration: Perform calibration according to the aforementioned method. After successful calibration, proceed to the Bright Eye detection.

[0047] Bright Eye Detection: The bright eye is detected by the neural impulse detection network generated after conversion using the bright eye detection module. Once the bright eye is detected, the bright eye tracking is initiated.

[0048] Bright Eye Tracking: Using the detection results of bright eye detection as the initial frame, the neural impulse tracking network generated after conversion is used by the method in the bright eye tracking module to track the bright eye;

[0049] Successful tracking: Set the tracking comparison similarity threshold to 0.65. When the threshold is greater than 0.65, the tracking is considered successful and tracking continues. When the threshold is less than 0.65, the tracking is considered to have failed and the process returns to the Mingtong detection step to re-perform Mingtong detection.

[0050] Data merging: Extract the tracking results of the left and right eyes, merge the tracking results of both eyes, use triangulation to estimate the position of the gaze point using the angles of the left and right eyes, and fuse the position with the front camera;

[0051] Trajectory mapping: By summarizing the positions of all fixation points, the trajectory of human eye movement and attention is mapped.

[0052] The comparison between the present invention and the traditional method is shown in Table 1 below, which shows that the present invention is better than the traditional method of pupil detection and tracking in terms of both power consumption and data redundancy.

[0053] Table 1

[0054] feature Traditional method of pupil detection and tracking The Bright Eye Detection and Tracking of the Invention Sampling speed 1200HZ 100MHz Data volume 1.5GB / s 30MB / s Power consumption 120W 15mW weight 5.1kg 70g

[0055] The present invention relates to a high-speed pupil-based eye movement detection and tracking method and system based on an event camera. It employs a brain-inspired event sensor and neural impulse network. The present invention can achieve high-speed pupil tracking with high sampling frequency and low power consumption. The present invention simultaneously achieves low power consumption and high efficiency in both data acquisition and intelligent data processing stages, thereby reducing the cost of eye movement tracking and benefiting the development of the industry.

[0056] The above description represents the preferred embodiment based on the inventive concept and working principle. The above embodiments should not be construed as limiting the scope of protection of these claims; other embodiments and combinations of implementations based on the inventive concept are all within the scope of protection of this invention.

Claims

1. A high-speed bright pupil eye movement detection and tracking method based on an event camera, characterized in that, Includes the following steps: S1. Data Acquisition: Using an event camera in conjunction with an infrared flashing light source, human eye data is collected in different scenarios, and the collected data is filtered and denoised. S2. Data Conversion: Convert the data acquired by the event camera into frame data; S3. Data labeling: Label the converted data, marking the location of the human eye pupil, and input the labeled data into step S4 for training; S4. Model Training: Train the backbone network of the feature extraction model, use the Mingtong detection module for feature extraction training, and set the trained backbone network of the feature extraction model into the Mingtong detection convolutional network and the Mingtong tracking convolutional network; S5. Model Conversion: The trained pupil detection convolutional network and pupil tracking convolutional network are converted into a spiking network model.

2. The high-speed bright pupil eye movement detection and tracking method based on an event camera according to claim 1, characterized in that, In step S1, the human eye data includes human eye data in indoor and outdoor scenarios with different lighting conditions.

3. The high-speed bright pupil eye movement detection and tracking method based on an event camera according to claim 1, characterized in that, In step S3, the location of the human eye pupil is marked manually.

4. The high-speed bright pupil eye movement detection and tracking method based on an event camera according to claim 1, characterized in that, In step S4, the bright eye detection convolutional network uses a convolutional network to detect bright eyes, transforms all neural network outputs, performs absolute value transformation on the network output of each layer, and sets all activation functions to ReLU activation functions. The bright eye detection convolutional network mainly performs bright eye detection, provides a tracking template for bright eye tracking, and relocates the bright eye position after tracking failure.

5. The high-speed bright pupil eye movement detection and tracking method based on an event camera according to claim 1, characterized in that, In step S4, the Bright Eye Tracking Convolutional Network adopts a Siamese network tracking model, and the backbone network adopts the same convolutional network as the Bright Eye Detection Convolutional Network. Two parallel input networks are set up for data processing. The two parallel networks are divided into a template network and a target network. The two networks share weights and extract the features of the template and the target respectively. The position of the tracking target is determined by the cross-correlation operation of feature comparison.

6. A high-speed bright pupil eye movement detection and tracking system based on an event camera, used to implement the high-speed bright pupil eye movement detection and tracking method based on an event camera as described in any one of claims 1 to 5, characterized in that, include: The system comprises a light source and acquisition module, a pupil detection module, a pupil tracking module, and a system calibration module, among which: The light source and acquisition module include a front-facing camera, an event camera, and a flashing infrared light source. The flashing infrared light source and the event camera are on the same optical path. The infrared light source illuminates the pupil. The event camera receives the light data returned by the pupil to acquire pupil information. The front-facing camera is used to acquire the scene observed by the human eye and fuse the position of the line of sight in the system. The bright eye detection module uses a convolutional network to detect bright eyes, providing a tracking template for bright eye tracking and repositioning the bright eye after tracking failure. The Bright Eye tracking module adopts a twin network tracking model. The backbone network uses the same convolutional network as the Bright Eye detection module. Two parallel input networks are set up for data processing. The two parallel input networks are divided into a template network and a target network. The two networks share weights and extract the features of the template and the target respectively. The position of the tracking target is determined by the cross-correlation operation of feature comparison. The system calibration module performs line-of-sight calibration for each user. The image features of the bullseye are stored in the system. The bullseye is placed in front of the user's eyes, and the tester needs to focus on looking at the bullseye. The user moves randomly to multiple positions, and the system automatically identifies the bullseye and calibrates the parameters within the system. When the bullseye positions of the visual tracking points coincide multiple times, the system calibration is completed.

7. The high-speed bright pupil eye movement detection and tracking system based on an event camera according to claim 6, characterized in that, The flashing frequency of the infrared light source is set to one-tenth of the sampling frequency, and the infrared light source is an invisible infrared light source.

Images