Eye movement tracking system and augmented reality / virtual reality equipment

Abstract

The invention relates to an eye movement tracking system and augmented reality / virtual reality equipment, and the system comprises a data receiver which is used for receiving event stream data sent by a dynamic vision sensor; the pupil tracker is connected with the data receiver and used for receiving the event stream data sent by the data receiver and determining the pupil position according to the received event stream data; the sight line estimator is connected with the pupil tracker and used for receiving the pupil position sent by the pupil tracker and determining sight line data according to the received pupil position; and the system controller is respectively connected with the data receiver, the pupil tracker and the sight line estimator, and is used for receiving a data report message sent by the dynamic vision sensor and sending a control instruction to at least one of the data receiver, the pupil tracker and the sight line estimator.

Description

Technical Field

[0001] The present disclosure relates to the field of digital integrated circuit technologies, and particularly to an eye movement tracking system and an augmented reality / virtual reality device. Background Art

[0002] Augmented Reality (AR) will become the next wave of human-computer interaction, dominating the relationship between humans and the digital world within the next 50 years. For example, AR glasses are one of the most typical AR devices. AR glasses require AR sensors and eye movement tracking chips to support computer vision and AI functions, and have strict requirements for accuracy, power consumption, size, and weight. AR sensors require high accuracy, low power consumption, and small size to support all-day wearable usage scenarios for day and night, indoor and outdoor. The eye movement tracking chip is a core component in the AR sensor system and one of the most important interfaces for human-computer interaction in AR devices, which can account for more than 50% of the total power consumption of AR devices; therefore, the eye movement tracking chip in AR devices poses requirements for low latency and low power consumption.

[0003] However, in related technologies, the power consumption of the eye movement tracking chip in AR devices is relatively high, and the latency is relatively high. Summary of the Invention

[0004] To overcome the problems existing in related technologies, embodiments of the present disclosure provide an eye movement tracking system and an augmented reality / virtual reality device to solve the defects in related technologies.

[0005] According to a first aspect of embodiments of the present disclosure, an eye movement tracking system is provided, and the system includes:

[0006] A data receiver, configured to receive event stream data sent by a dynamic vision sensor;

[0007] A pupil tracker, connected to the data receiver, configured to receive the event stream data sent by the data receiver and determine a pupil position according to the received event stream data;

[0008] A gaze estimator, connected to the pupil tracker, configured to receive the pupil position sent by the pupil tracker and determine gaze data according to the received pupil position;

[0009] A system controller, respectively connected to the data receiver, the pupil tracker, and the gaze estimator, configured to receive a data reporting message sent by the dynamic vision sensor and send a control instruction to at least one of the data receiver, the pupil tracker, and the gaze estimator.

[0010] In some embodiments of the present disclosure, the data receiver includes a data queue, a preprocessing module, and a selector. The input end of the data queue is connected to the dynamic vision sensor, and the output ends are respectively connected to the input end of the preprocessing module and the input end of the selector. The output end of the preprocessing module is connected to the input end of the selector, and the output end of the selector is connected to the pupil tracker;

[0011] The data queue is configured to receive the event stream data sent by the dynamic vision sensor based on the control instruction of the system controller, and send the received event stream data to the preprocessing module and the selector;

[0012] The preprocessing module is configured to preprocess or not preprocess the received event stream data based on the control instruction of the system controller;

[0013] The selector is configured to send one of the event stream data sent by the data queue and the event stream data sent by the preprocessing module to the pupil tracker based on the control instruction of the system controller.

[0014] In some embodiments of the present disclosure, when the system controller receives the data reporting message sent by the dynamic vision sensor, it sends an instruction indicating data reception to the data queue.

[0015] In some embodiments of the present disclosure, the request terminal and the confirmation terminal of the system controller are respectively connected to the dynamic vision sensor, and the system controller is configured to:

[0016] Determine that it has received the data reporting message sent by the dynamic vision sensor when the level of the request terminal changes;

[0017] Switch the level of the confirmation terminal when it is determined that it has received the data reporting message sent by the dynamic vision sensor;

[0018] Restore the level of the confirmation terminal when the data queue has received all the event stream data sent by the dynamic vision sensor, so that the dynamic vision sensor restores the level of the request terminal.

[0019] In some embodiments of the present disclosure, the pupil tracker includes an event accumulation unit and a pupil tracking unit connected to each other. The input ends of the event accumulation unit and the pupil tracking unit are respectively connected to the data receiver;

[0020] The event accumulation unit is configured to receive the event stream data sent by the data receiver, update or not update the current event accumulation result based on the received event stream data according to the control instruction of the system controller, and send the updated event accumulation result to the pupil tracking unit;

[0021] The pupil tracking unit is configured to receive the event stream data sent by the data receiver and the event accumulation result sent by the event accumulation unit, and determine the pupil position based on the received event stream data and the received event accumulation result according to the control instruction of the system controller.

[0022] In some embodiments of the present disclosure, the pupil tracking unit is configured to send an update message to the event accumulation unit when the determined pupil position meets a preset condition, so that the event accumulation unit sends an update message to the system controller;

[0023] The system controller is configured to, when receiving the update message, send an instruction to the preprocessing module to preprocess the received event stream data, send an instruction to the selector to send the event stream data sent by the preprocessing module to the pupil tracker, and send an instruction to the event accumulation unit to update the event accumulation result.

[0024] In some embodiments of the present disclosure, the pupil tracking unit is configured to send a freeze message to the event accumulation unit when the determined pupil position does not meet a preset condition, so that the event accumulation unit sends a freeze message to the system controller;

[0025] The system controller is configured to, when receiving the freeze message, send an instruction to the preprocessing module not to preprocess the received event stream data, send an instruction to the selector to send the event stream data sent by the data queue to the pupil tracker, and send an instruction to the event accumulation unit not to update the event accumulation result.

[0026] In some embodiments of the present disclosure, the system controller is configured to send an instruction to the pupil tracking unit to determine the pupil position when receiving the data reporting message sent by the dynamic vision sensor.

[0027] In some embodiments of the present disclosure, the gaze estimator includes a calibration look-up table and a gaze estimation unit;

[0028] The calibration look-up table records the mapping relationship between the pupil position and the gaze data;

[0029] The gaze estimation unit is configured to determine the gaze data based on the calibration look-up table and the received pupil position according to the control instruction of the system controller.

[0030] In some embodiments of the present disclosure, the gaze data includes the projection position of the pupil gaze on the virtual screen.

[0031] According to a second aspect of the embodiments of the present disclosure, there is provided an augmented reality / virtual reality device, including a dynamic vision sensor and an eye movement tracking system provided in any embodiment of the first aspect.

[0032] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

[0033] The eye movement tracking system provided by the embodiments of the present disclosure is connected to the dynamic vision sensor, and includes a data receiver, a pupil tracker, and a gaze estimator connected in sequence. The data receiver can receive the event stream data sent by the dynamic vision sensor. The pupil tracker can receive the event stream data sent by the data receiver and determine the pupil position according to the received event stream data. The gaze estimator can receive the pupil position sent by the pupil tracker and determine the gaze data according to the received pupil position. It further includes a system controller respectively connected to the data receiver, the pupil tracker, and the gaze estimator, which can receive the data reporting message sent by the dynamic vision sensor and send a control instruction to at least one of the data receiver, the pupil tracker, and the gaze estimator to complete eye movement tracking based on the event stream data sent by the dynamic vision sensor. This system is connected to the dynamic vision sensor and receives and processes the event stream data generated by the dynamic vision sensor. Compared with traditional image sensors such as RGB sensors, the event stream data generated by the dynamic vision sensor only records the data of pixels where the light intensity changes, thereby reducing the data processing pressure of the eye movement tracking system, improving the processing efficiency to reduce latency, and being able to reduce power consumption. Description of the Drawings

[0034] The drawings here are incorporated into the description and form a part of this description, showing embodiments consistent with the present invention and used together with the description to explain the principles of the present invention.

[0035] Figure 1 is a schematic structural diagram of an eye movement tracking system shown in an exemplary embodiment of the present disclosure;

[0036] Figure 2 is a schematic structural diagram of an eye movement tracking system shown in an exemplary embodiment of the present disclosure;

[0037] Figure 3 is the eye movement tracking process of an AR device shown in an exemplary embodiment of the present disclosure;

[0038] Figure 4 is a schematic diagram of the control timing of an eye movement tracking system shown in an exemplary embodiment of the present disclosure;

[0039] Figure 5 is a block diagram of the structure of an electronic device shown in an exemplary embodiment of the present disclosure. Detailed Embodiments

[0040] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0041] The terms used in the present disclosure are for the purpose of describing particular embodiments only and are not intended to limit the present disclosure. The singular forms "a", "the", and "said" used in the present disclosure and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0042] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to a determination".

[0043] Augmented Reality (AR) will become the next wave of human-computer interaction, dominating the relationship between humans and the digital world in the next 50 years. For example, AR glasses are one of the most typical AR devices. AR glasses require AR sensors and eye-tracking chips to support computer vision and AI functions, and at the same time have strict requirements for accuracy, power consumption, size, and weight. AR sensors are required to have high accuracy, low power consumption, and small size to support all-day wearable usage scenarios day and night, indoors and outdoors. The eye-tracking chip is a core component in the AR sensor system and one of the most important interfaces for human-computer interaction in AR devices, which can account for more than 50% of the total power consumption of AR devices; therefore, the eye-tracking chip in AR devices poses requirements for low latency and low power consumption.

[0044] However, in related technologies, the power consumption and latency of the eye-tracking chip in AR devices are relatively high.

[0045] For example, the measurement results of AR devices in the related art show that their tracking latency usually exceeds 15 ms, which is sufficient to introduce visual artifacts in rendering scenarios based on line-of-sight estimation. In addition, AR devices in the related art use traditional frame-based image sensors, and their always-on feature causes the eye-tracking chip to always consume more than 2 W of power, consuming most (more than half) of the device's power budget. It can be seen that both the latency and total power consumption of AR devices are limited by the performance of the eye-tracking chip. Therefore, how to implement an eye-tracking chip with low latency and low power consumption is a key issue in promoting the progress of AR devices.

[0046] Based on this, in a first aspect, at least one embodiment of the present disclosure provides an eye-tracking system, which is applied to an AR device or a VR (Virtual Reality) device, and can achieve lower latency and lower power consumption, so as to ensure that the augmented reality device or the virtual reality device can achieve low-latency eye tracking with lower power consumption, and further can implement functions such as rendering images, and ensure that there are no influencing factors such as visual artifacts in these functions.

[0047] For example, the eye-tracking system can be an eye-tracking chip or other circuit forms, and the present disclosure does not limit this.

[0048] Please refer to the appendix Figure 1 , which exemplarily shows the structure of the eye-tracking system. The system includes a data receiver, a pupil tracker, a line-of-sight estimator, and a system controller. The data receiver is connected to the dynamic vision sensor and is used to receive the event stream data sent by the dynamic vision sensor; the pupil tracker is connected to the data receiver and is used to receive the event stream data sent by the data receiver and determine the pupil position according to the received event stream data; the line-of-sight estimator is connected to the pupil tracker and is used to receive the pupil position sent by the pupil tracker and determine the line-of-sight data according to the received pupil position; the system controller is respectively connected to the data receiver, the pupil tracker, and the line-of-sight estimator, and is used to receive the data reporting message sent by the dynamic vision sensor and send a control instruction to at least one of the data receiver, the pupil tracker, and the line-of-sight estimator.

[0049] Among them, the Dynamic Vision Sensor (DVS) is a sensor within the device where the eye-tracking system is located, and is used to collect event stream data within its field of view. If the brightness of a pixel within the field of view of the Dynamic Vision Sensor changes, it outputs the event stream data AER (Address-Event Representation). The event stream data includes the timestamp, location, and polarity of the pixel where the brightness change occurs. The timestamp represents the time when the event occurs, the location represents the location where the event occurs, that is, the coordinates of the pixel where the brightness change occurs, and the polarity represents the direction of the brightness change, such as brightness increase or brightness decrease. When the user wears the device where the eye-tracking system is located, the field of view of the Dynamic Vision Sensor covers the user's eyes.

[0050] The Dynamic Vision Sensor is connected to the data receiver and system controller of the eye-tracking system. When the Dynamic Vision Sensor generates event stream data, it can send a data reporting message to the system controller and send the generated event stream data to the data receiver.

[0051] The eye-tracking system processes based on the event stream data output by the Dynamic Vision Sensor, has the characteristic of event-driven, and does not need to process a frame of data including all pixel information, but only calculates the pixel data with changes in light intensity. That is, it only starts to work when the Dynamic Vision Sensor detects an event, otherwise it will remain in the standby state. Therefore, this event-driven processing method can effectively avoid the power consumption caused by redundant image data transmission and processing. In addition, this method can achieve extremely low tracking latency and meet the requirements of the AR scenario for eye tracking.

[0052] The eye-tracking system provided by the embodiments of the present disclosure is connected to the Dynamic Vision Sensor and receives and processes the event stream data generated by the Dynamic Vision Sensor. Compared with traditional image sensors such as RGB sensors, the event stream data generated by the Dynamic Vision Sensor only records the data of pixels where the light intensity changes, thereby reducing the data processing pressure on the eye-tracking system, improving the processing efficiency to reduce latency, and being able to reduce power consumption.

[0053] Please refer to the appendix Figure 2, in some embodiments of the present disclosure, the data receiver includes a data queue, a preprocessing module, and a selector. The input end of the data queue is connected to the dynamic vision sensor, and the output ends are respectively connected to the input end of the preprocessing module and the input end of the selector. The output end of the preprocessing module is connected to the input end of the selector, and the output end of the selector is connected to the pupil tracker. The data queue is configured to receive the event stream data sent by the dynamic vision sensor based on the control instruction of the system controller, and send the received event stream data to the preprocessing module and the selector. The preprocessing module is configured to preprocess or not preprocess the received event stream data based on the control instruction of the system controller. The selector is configured to send one of the event stream data sent by the data queue and the event stream data sent by the preprocessing module to the pupil tracker based on the control instruction of the system controller.

[0054] For example, the data queue can be the Figure 2 specifically shown FIFO (First In First Out) module, which can ensure that the events in the event stream data are transmitted in the order of generation. The preprocessing module can be the Figure 2 specifically shown event discarding module, which can discard some of the received event stream data and retain some data, thereby reducing the processing pressure of subsequent hardware.

[0055] Optionally, the system controller is configured to send an instruction indicating data reception to the data queue when receiving the data reporting message sent by the dynamic vision sensor, so that the data queue receives the event data stream sent by the dynamic vision sensor.

[0056] Wherein, the system controller may be provided with a request terminal REQ and an acknowledgment terminal ACK. The terminals can be pins of the system, and both the request terminal and the acknowledgment terminal are connected to the dynamic vision sensor. The dynamic vision sensor can send a data reporting message to the system controller by changing the level of the request terminal of the system controller. And the system controller can: determine that it has received the data reporting message sent by the dynamic vision sensor when the level of the request terminal changes; change the level of the acknowledgment terminal when determining that it has received the data reporting message sent by the dynamic vision sensor; restore the level of the acknowledgment terminal when the data queue has received the event stream data sent by the dynamic vision sensor, so that the dynamic vision sensor restores the level of the request terminal.

[0057] For example, the request terminal and the confirmation terminal are normally at a low level. When the dynamic vision sensor generates event stream data, it raises the level of the request terminal so that the system controller can confirm that it has received the data reporting message. Further, the system controller controls the data queue to receive the event stream data sent by the dynamic vision sensor, and at the same time raises the confirmation terminal to notify the dynamic vision sensor that the data reception has started. The data queue reports the progress of data reception to the system controller, and the system controller lowers the level of the confirmation terminal after the data queue has completed receiving the event stream data to notify the dynamic vision sensor that the data reception is complete. The dynamic vision sensor lowers the level of the request terminal.

[0058] The process of receiving data in this example can be called a four-phase handshake protocol, that is, the dynamic vision sensor and the system controller perform four protocol handshakes to correctly read the event stream data generated by the dynamic vision sensor into the data queue and complete data reception.

[0059] Please continue to refer to the attached Figure 2 In some embodiments of the present disclosure, the pupil tracker includes an event accumulation unit and a pupil tracking unit connected to each other. The input ends of the event accumulation unit and the pupil tracking unit are respectively connected to the data receiver. The event accumulation unit is configured to receive the event stream data sent by the data receiver, and update or not update the current event accumulation result based on the received event stream data according to the control instruction of the system controller, and send the updated event accumulation result to the pupil tracking unit. The pupil tracking unit is configured to receive the event stream data sent by the data receiver and the event accumulation result sent by the event accumulation unit, and determine the pupil position based on the received event stream data and the received event accumulation result according to the control instruction of the system controller.

[0060] For example, the event accumulation unit can be a Random Access Memory (RAM), that is, the event accumulation unit is an event accumulation RAM. For example, the event accumulation unit can accumulate the event stream data over a period of time to generate an event frame as the event accumulation result. For example, the pupil position can be the pixel coordinates of the pupil.

[0061] Among them, the pupil tracking unit determines the pupil position based on the received event stream data and the received event accumulation result, improving the tracking accuracy, that is, the accuracy of the determined pupil position.

[0062] Optionally, when receiving the data reporting message sent by the dynamic vision sensor, the system controller is configured to send an instruction for determining the pupil position to the pupil tracking unit. In other words, whenever the eye tracking system receives the event stream data generated by the dynamic vision sensor, it controls the pupil tracking unit to determine the pupil position based on the latest event stream data and the current event accumulation result, that is, to perform pupil tracking.

[0063] Among them, the pupil position can characterize the current eye state of the user, such as the blinking state, the eyeball rotation state, etc.

[0064] For example, when the determined pupil position meets a preset condition, the pupil tracking unit is configured to send an update message Update (not shown in the figure) to the event accumulation unit, so that the event accumulation unit sends an update message Update to the system controller; when receiving the update message Update, the system controller is configured to send an instruction for preprocessing the received event stream data to the preprocessing module, send an instruction for sending the event stream data sent by the preprocessing module to the pupil tracker to the selector, and send an instruction for updating the event accumulation result to the event accumulation unit.

[0065] Among them, when the pupil is not blocked, such as in a rapid rotation state, it can be confirmed that the pupil position meets the preset condition.

[0066] In other words, when the pupil tracking unit determines that the pupil is not blocked, the subsequent event stream data is valuable for pupil tracking and gaze estimation. Therefore, by sending instructions to the preprocessing module, the selector, and the event accumulation unit, the subsequent event stream data is preprocessed, and the event accumulation result is updated based on the preprocessed event stream data.

[0067] For example, when the determined pupil position does not meet the preset condition, the pupil tracking unit is configured to send a freeze message to the event accumulation unit, so that the event accumulation unit sends a freeze message to the system controller; when receiving the freeze message, the system controller is configured to send an instruction for not preprocessing the received event stream data to the preprocessing module, send an instruction for sending the event stream data sent by the data queue to the pupil tracker to the selector, and send an instruction for not updating the event accumulation result to the event accumulation unit.

[0068] Among them, when the pupil is blocked, such as when the user is in a blinking state, it can be confirmed that the pupil position does not meet the preset condition.

[0069] In other words, when the pupil tracking unit determines that the pupil is occluded, the subsequent event stream data is valuable for pupil tracking and gaze estimation. Therefore, by sending instructions to the preprocessing module, selector, and event accumulation unit, the subsequent event stream data is not preprocessed, and the event accumulation result is not updated based on the subsequent event stream data; that is, pupil tracking is performed only based on the subsequent unprocessed event stream data until the pupil position meets the preset conditions.

[0070] Preferably, if the pupil tracking unit does not send an update message Update to the event accumulation unit, the event accumulation unit regards that the pupil tracking unit has sent a freeze message; if the event accumulation unit does not send an update message Update to the system controller, the system controller regards that the event accumulation unit has sent a freeze message to the system controller.

[0071] Preferably, the EN_Drop terminal (such as a pin) of the system controller is connected to the preprocessing module and the selector respectively, and is used to send the following enable signals: if the system controller adjusts the level of the EN_Drop terminal to a high level, the preprocessing module regards that it has received an instruction to preprocess the received event stream data, and the selector regards that it has received an instruction to send the event stream data sent by the preprocessing module to the pupil tracker; if the system controller adjusts the level of the EN_Drop terminal to a low level, the preprocessing module regards that it has received an instruction not to preprocess the received event stream data, and the selector regards that it has received an instruction to send the event stream data sent by the data queue to the pupil tracker.

[0072] Preferably, the EN_ACC terminal (such as a pin) of the system controller is connected to the event accumulation unit, and is used to send the following enable signals: if the system controller adjusts the level of the EN_ACC terminal to a high level, the event accumulation unit regards that it has received an instruction to update the event accumulation result; if the system controller adjusts the level of the EN_ACC terminal to a low level, the event accumulation unit regards that it has received an instruction not to update the event accumulation result.

[0073] Preferably, the EN_Pup terminal (such as a pin) of the system controller is connected to the pupil tracking unit, and is used to send the following enable signals: if the system controller adjusts the level of the EN_Pup terminal to a high level, the pupil tracking unit regards that it has received an instruction to determine the pupil position; if the system controller adjusts the level of the EN_Pup terminal to a low level, the pupil tracking unit regards that it has received an instruction not to determine the pupil position.

[0074] Please continue to refer to the appendix Figure 2, in some embodiments of the present disclosure, the line-of-sight estimator includes a calibration look-up table and a line-of-sight estimation unit; the calibration look-up table records the mapping relationship between the pupil position and the line-of-sight data; the line-of-sight estimation unit is configured to determine the line-of-sight data based on the control instruction of the system controller according to the calibration look-up table and the received pupil position.

[0075] Line-of-sight estimation refers to establishing a mapping relationship between the obtained pupil position and the fixation point of the eye on the basis of the obtained pupil position, so as to obtain the line-of-sight angle. The line-of-sight data can be used to represent the fixation point coordinates of the eye on the screen, that is, the projection position of the pupil line of sight on the virtual screen, where the virtual screen is the virtual screen generated by the AR device.

[0076] For example, when receiving an update message, the system controller may send an instruction to the line-of-sight estimator to indicate line-of-sight estimation; when receiving a freeze message, the system controller may send an instruction to the line-of-sight estimator to indicate not to perform line-of-sight estimation.

[0077] Preferably, the EN_Gaze terminal (such as a pin) of the system controller is connected to the line-of-sight estimator for sending the following enable signal: if the system controller adjusts the level of the EN_Gaze terminal to a high level, the line-of-sight estimator is regarded as receiving an instruction to perform line-of-sight estimation; if the system controller adjusts the level of the EN_Gaze terminal to a low level, the line-of-sight estimator is regarded as receiving an instruction not to perform line-of-sight estimation.

[0078] In this embodiment, line-of-sight estimation is performed when the event accumulation unit updates the event accumulation result, so as to avoid energy consumption waste caused by regularly enabling the line-of-sight estimation step and further reduce power consumption.

[0079] Please refer to the appendix Figure 3 , which exemplarily shows the eye movement tracking process of an AR device including a dynamic vision sensor and the eye movement tracking system provided by the present disclosure.

[0080] First, it is judged whether the DVS (dynamic vision sensor) generates an event according to the state of the handshake signal. If no event is generated, the standby state is continued. If an event is generated, it is further judged whether to perform event discarding. If event discarding is performed, the remaining data after discarding some events is input into the event accumulation RAM for accumulation, and pupil tracking operation is performed based on the remaining events and the event accumulation result; if event discarding is not performed, accumulation is performed based on the received original event stream, and pupil tracking operation is performed based on the original event stream and the event accumulation result. Subsequently, after completing pupil tracking, a line-of-sight estimation operation is performed, and finally the predicted fixation point coordinates of the eye on the screen are output, thus completing the entire eye movement tracking process.

[0081] Please refer to the appendix Figure 4, which exemplarily shows the control timing of the eye tracking system provided by the present disclosure.

[0082] REQ and ACK are handshake signals between the eye tracking system and the DVS. Pulling REQ high indicates that the DVS has a data request. At this time, the data AER_DATA output by the DVS is received. After the reception is completed, the eye tracking system will pull the ACK signal low, thus completing a data transmission. EN_Drop determines whether to perform event discarding. If EN_Drop is high, event discarding is performed, and the remaining data after discarding is input into the event accumulation RAM for accumulation, and pupil tracking operations are performed based on the remaining events and the event accumulation results; if EN_Drop is low, event discarding is not performed, event accumulation is not performed, and pupil tracking operations are performed based on the original event stream and the current event accumulation results. Update is an update message output by the event accumulation RAM, indicating whether to discard or update the data. When the event data stream is received, EN_Pup will be pulled high to make the pupil tracking unit start working; when the pupil tracking unit finishes working and the event accumulation RAM updates the event accumulation results, EN_Gaze will be pulled high immediately to control the pupil tracking unit to start working and output out_data, identifying the gaze point coordinates of the eyes on the screen.

[0083] The eye tracking system provided by the present disclosure does not run continuously, but is in a standby mode and will only be activated and start to execute its functions when a data reception requirement or a data processing request inside or outside the system is detected. That is, when the FIFO receives the event stream data generated by the DVS, the system controller will activate data processing through various enable signals, realizing the working mode of the system "working when there are events and standby when there are no events", which can effectively reduce the power consumption of the system.

[0084] According to a second aspect of the embodiments of the present disclosure, there is provided an augmented reality / virtual reality device, including a dynamic vision sensor and the eye tracking system provided in any embodiment of the first aspect.

[0085] Please refer to the appendix Figure 5 , which exemplarily shows the block diagram of the device. For example, the device 500 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0086] Refer to Figure 5 , the device 500 may include one or more of the following components: a processing component 502, a memory 504, a power supply component 506, a multimedia component 508, an audio component 510, an input / output (I / O) interface 512, a sensor component 514, and a communication component 516.

[0087] The processing component 502 generally controls the overall operation of the device 500, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above methods. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0088] The memory 504 is configured to store various types of data to support the operation of the device 500. Examples of such data include instructions for any application or method operating on the device 500, contact data, phone book data, messages, pictures, videos, etc. The memory 504 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0089] The power component 506 provides power to various components of the device 500. The power component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 500.

[0090] The multimedia component 508 includes a screen that provides an output interface between the device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may not only sense the boundaries of touch or swipe actions, but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the device 500 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each of the front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0091] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC) that is configured to receive external audio signals when the device 500 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 504 or transmitted via the communication component 516. In some embodiments, the audio component 510 further includes a speaker for outputting audio signals.

[0092] The I / O interface 512 provides an interface between the processing component 502 and peripheral interface modules, and the peripheral interface modules may be a keyboard, a click wheel, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a power button, and a lock button.

[0093] The sensor component 514 includes one or more sensors for providing status assessments of various aspects of the device 500. For example, the sensor component 514 can detect the on / off state of the device 500, the relative positioning of components, such as the display and keypad of the device 500. The sensor component 514 can also detect a change in the position of the device 500 or a component of the device 500, the presence or absence of user contact with the device 500, the orientation or acceleration / deceleration of the device 500, and the temperature change of the device 500. The sensor component 514 may also include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 514 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 514 may further include an acceleration sensor, a gyro sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0094] The communication component 516 is configured to facilitate communication between the device 500 and other devices in a wired or wireless manner. The device 500 can access a wireless network based on communication standards, such as WiFi, 2G or 3G, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0095] In an exemplary embodiment, the device 500 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0096] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the disclosure herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common general knowledge or conventional technical means in the technical field not disclosed herein. The specification and examples are only exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0097] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes may be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. An eye tracking system, characterized in that: The system comprises: A data receiver, used for receiving event stream data sent by a dynamic vision sensor; a pupil tracker connected to the data receiver, configured to receive event stream data sent by the data receiver, and determine a pupil position based on the received event stream data; a sight line estimator, connected to the pupil tracker, for receiving the pupil position sent by the pupil tracker, and determining sight line data according to the received pupil position; The system controller is respectively connected to the data receiver, the pupil tracker and the line of sight estimator, and is used to receive the data reporting message sent by the dynamic vision sensor and send a control instruction to at least one of the data receiver, the pupil tracker and the line of sight estimator.

2. The eye tracking system according to claim 1, characterized in that: The data receiver includes a data queue, a preprocessing module and a selector, wherein the input end of the data queue is connected to the dynamic vision sensor, and the output end is respectively connected to the input end of the preprocessing module and the input end of the selector, the output end of the preprocessing module is connected to the input end of the selector, and the output end of the selector is connected to the pupil tracker; The data queue is used to receive the event stream data sent by the dynamic vision sensor based on the control instruction of the system controller, and send the received event stream data to the preprocessing module and the selector; The preprocessing module is used to preprocess or not preprocess the received event stream data based on the control instruction of the system controller; The selector is used to send one of the event stream data sent by the data queue and the event stream data sent by the pre-processing module to the pupil tracker based on the control instruction of the system controller.

3. The eye tracking system according to claim 2, characterized in that: The system controller is used for sending an instruction instructing to receive data to the data queue when receiving the data reporting message sent by the dynamic vision sensor.

4. The eye tracking system according to claim 3, characterized in that: The request terminal and the confirmation terminal of the system controller are respectively connected to the dynamic vision sensor, and the system controller is used for: determining, when the level of the request terminal changes, that a data reporting message sent by the dynamic vision sensor is received; Switching the level of the confirmation terminal when it is determined that the data reporting message sent by the dynamic vision sensor is received; When the data queue has finished receiving the event stream data sent by the dynamic vision sensor, the level of the confirmation terminal is restored, so that the dynamic vision sensor restores the level of the request terminal.

5. The eye tracking system according to claim 2, characterized in that: The pupil tracker comprises an event accumulation unit and a pupil tracking unit connected to each other, and an input end of the event accumulation unit and an input end of the pupil tracking unit are respectively connected to the data receiver; The event accumulation unit is used to receive the event stream data sent by the data receiver, and based on the control instruction of the system controller, update or not update the current event accumulation result according to the received event stream data, and send the updated event accumulation result to the pupil tracking unit; The pupil tracking unit is used to receive the event stream data sent by the data receiver and the event accumulation result sent by the event accumulation unit, and determine the pupil position according to the received event stream data and the received event accumulation result based on the control instruction of the system controller.

6. The eye tracking system according to claim 5, characterized in that: The pupil tracking unit is used to send an update message to the event accumulation unit when the determined pupil position meets a preset condition, so that the event accumulation unit sends an update message to the system controller; The system controller is used to send instructions to the preprocessing module to preprocess the received event stream data, send instructions to the selector to send the event stream data sent by the preprocessing module to the pupil tracker, and send instructions to the event accumulation unit to update the event accumulation results when receiving an update message.

7. The eye tracking system according to claim 5, characterized in that: The pupil tracking unit is used to send a freeze message to the event accumulation unit when the determined pupil position does not meet a preset condition, so that the event accumulation unit sends a freeze message to the system controller; The system controller is used to, when receiving a freeze message, send an instruction to the preprocessing module to instruct not to preprocess the received event stream data, send an instruction to the selector to instruct to send the event stream data sent by the data queue to the pupil tracker, and send an instruction to the event accumulation unit to instruct not to update the event accumulation result.

8. The eye tracking system according to claim 5, characterized in that: The system controller is used to send an instruction to the pupil tracking unit to determine the pupil position when receiving the data reporting message sent by the dynamic vision sensor.

9. The eye tracking system according to claim 1, characterized in that: The line of sight estimator comprises a calibration lookup table and a line of sight estimation unit; The calibration lookup table records the mapping relationship between pupil position and sight line data; The line of sight estimation unit is used to determine line of sight data according to the calibration lookup table and the received pupil position based on the control instruction of the system controller.

10. The eye tracking system according to claim 9, characterized in that: The sight line data includes the projection position of the pupil sight line on the virtual screen.

11. An augmented reality / virtual reality device, characterized in that: The invention comprises a dynamic vision sensor and an eye tracking system as claimed in any one of claims 1 to 10.

Images