Real-time eye movement tracking event and feature calculation system platform building method

Abstract

The invention relates to a real-time eye movement tracking event and feature calculation system platform building method and device, electronic equipment and a storage medium. The method comprises the following steps: establishing an eye movement online signal capturing system; based on the IO multiplexing model, generating a synchronous processing method of eye movement real-time signal processing, eye movement real-time event processing, eye movement real-time feature calculation and eye movement real-time data model algorithm; establishing a real-time alignment processing method for the eye movement labeling signal and the eye movement fixation data; establishing a method for converting an eye movement event into eye movement feature recognition, and completing a real-time eye movement feature calculation algorithm; and accessing the IVT offline model to a real-time stream, calling online operation by a variable threshold value, and completing real-time eye movement tracking event and feature calculation system platform construction. According to the invention, through model and algorithm updating, establishment of a strong real-time system platform for eye movement data acquisition, eye movement event calculation and eye movement feature calculation without delay is realized.

Description

Background Art

[0002] Eye tracking is the process of tracking a person's eye direction, movement and gaze on a stimulus through an eye tracking device. The monitoring of a person's real-time cognitive state is a hot research direction in the fields of computer applications, brain-computer engineering, human factors engineering, and intelligent politics and law. Eye tracking technology is an effective means of reflecting cognitive states, so real-time detection of human eye tracking is necessary.

[0003] Eye tracking devices are divided into desktop fixed eye trackers and portable eye trackers. Portable eye trackers are divided into glasses-type eye trackers and fixed-position eye trackers. Because of the different fixed positions of these eye trackers, the sensors for collecting eye data are also different, but the output data format for transmitting eye gaze data is generally {timestamp, gaze data{[gaze 2D (two-dimensional) coordinates], [gaze 3D (three-dimensional) coordinates], [left eye data: {eye source coordinates, eye direction, pupil diameter}, [right eye data: {eye source coordinates, eye direction, pupil diameter}}. For the study of eye tracking, especially the study of gaze, it is necessary to calculate eye movement events. Generally, eye movement events are divided into five types: gaze, saccade, blink, pupil diameter, and smooth pursuit. In the calculation of gaze and saccade, after collecting eye movement data, some models are used to calculate eye movement data offline. Salvucci (2000) generally has five eye movement event calculation models, namely velocity threshold recognition (I-VT) algorithm, discrete threshold recognition (I-DT algorithm), hidden Markov recognition (I-MM), and interest area algorithm (I-AOI algorithm). Dao (2014) tested the online gaze and saccade classification of velocity threshold recognition (I-VT), using the overall real-time data input method and putting it into cloud computing for testing. It was found that the delay was about 2000ms on a single machine and about 3000ms on more than ten machines. The real-time calculation of pupil diameter is relatively mature, and a large number of studies use pupil diameter as an indicator of arousal. In the calculation of blinking, real-time blinking event calculation is usually judged by machine learning, and in offline blinking event judgment, empty sample data above 75ms is used as blinking data.

[0004] In terms of eye movement features, in existing research, according to existing research, the commonly used eye movement feature extractions include nine eye movement features: fixation time per unit time, number of fixations, saccade time, number of saccades, blink time, number of blinks, average pupil diameter of both eyes, eye movement speed, and eye movement distance. However, they are all used for offline research and training, and there is no online platform that can process all eye movement events and feature calculations in real time, so it is impossible to achieve real-time processing of all eye movement events and real-time calculation of features.

[0005] Therefore, one or more methods are needed to solve the above problems.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0007] The purpose of the present disclosure is to provide a method, device, electronic device and computer-readable storage medium for building a real-time eye tracking event and feature calculation system platform, thereby overcoming one or more problems caused by the limitations and defects of related technologies at least to a certain extent.

[0008] According to one aspect of the present disclosure, a method for building a real-time eye tracking event and feature calculation system platform is provided, comprising:

[0009] Establish an online eye movement signal capture system based on a preset network framework built with a preset program library;

[0010] In the eye movement online signal capture system, based on the IO multiplexing model of the preset software development kit, a synchronous processing method for generating eye movement real-time signal processing, eye movement real-time event processing, eye movement real-time feature calculation, and eye movement real-time data model algorithm;

[0011] Establish a real-time method of controlling the marking data based on the event points of manually calling data packets, and establish a real-time method of aligning the eye movement marking signal with the eye movement gaze data;

[0012] Establish and update the IVT offline model, so that the updated IVT model is updated from two events of fixation and saccade to four events of fixation, saccade, pupil size, and blink, simulate the IVT offline data based on the IVT offline model to generate simulation results, and compare the simulation results with the online data;

[0013] Establish a method to convert eye movement events into eye movement feature recognition and complete a real-time eye movement feature calculation algorithm;

[0014] The IVT offline model is connected to the real-time stream and called to run online with a variable threshold to complete the construction of the real-time eye tracking event and feature calculation system platform.

[0015] In an exemplary embodiment of the present disclosure, the method further includes:

[0016] The preset network framework of the preset program library is an asynchronous web framework built based on aiohttp. The asynchronous web framework built based on aiohttp adopts tobbi's software development kit SDK to establish an eye movement online signal capture system.

[0017] In an exemplary embodiment of the present disclosure, the IO multiplexing model based on the preset software development kit in the method further includes:

[0018] Control the process of the IO multiplexing model through marking events and set an asynchronous data PP pool;

[0019] By setting up monitors corresponding to the user-defined data volume, data replication in the data pool is achieved and the data pool is updated in real time;

[0020] A thread for data calculation is added to the asynchronous data PP pool. The thread is set to have an event within the user-defined data volume collection time. When entering the thread, the data volume of each calculation package is locked.

[0021] Based on the asynchronous data PP pool, the computing packages are queued in the data pool and enter the computing channel.

[0022] In an exemplary embodiment of the present disclosure, the eye movement tagging signal alignment processing method of the method further includes:

[0023] The marking data is taken as a single channel lead, and the marking data is controlled based on the event points of the artificially called data packet to achieve real-time alignment of the eye movement marking signal and the eye movement gaze data.

[0024] In an exemplary embodiment of the present disclosure, the method further includes gaze, glance, blink, and pupil event calculation:

[0025] The calculation of each fixation time is edited as the fixation end timestamp minus the fixation start timestamp;

[0026] The calculation of each saccade time was edited as the saccade end timestamp minus the saccade start timestamp;

[0027] Edit each blink time to a blank time period greater than 75ms within the time period;

[0028] The pupil diameter at each time is edited as the average value of the valid data of the left and right eyes in the time period.

[0029] In an exemplary embodiment of the present disclosure, the method further includes calculating eye movement features:

[0030] Fixation duration was defined as the sum of fixation event times within the time period;

[0031] The number of fixations was defined as the sum of the number of fixation time events within the time period;

[0032] The saccade duration was defined as the sum of the saccade event times within the time period;

[0033] The number of saccades was defined as the sum of the number of saccade time events within the time period;

[0034] The distance is defined as the sum of the movement distances of the fixation point per unit time;

[0035] Eye movement velocity is defined as the sum of the speeds of gaze point movement per unit time;

[0036] The number of blinks was defined as the sum of the number of blink events in the time period;

[0037] Blink duration was defined as the sum of blink event times within the time period;

[0038] The pupil diameter is defined as the sum of pupil diameter event values ​​per unit time.

[0039] In an exemplary embodiment of the present disclosure, the method further includes:

[0040] The IVT offline model is connected to the real-time stream and retrieved with a variable threshold, the IVT offline data is simulated to generate simulation results, and the simulation results are compared with the online data to verify the real-time eye tracking event and feature calculation system platform.

[0041] In one aspect of the present disclosure, a device for building a real-time eye tracking event and feature calculation system platform is provided, comprising:

[0042] An eye movement signal capture module is used to establish an eye movement online signal capture system based on a preset network framework built by a preset program library;

[0043] A data synchronization processing module is used to generate an eye movement real-time signal processing, an eye movement real-time event processing, an eye movement real-time feature calculation, and an eye movement real-time data model algorithm in the eye movement online signal capture system based on an IO multiplexing model of a preset software development kit;

[0044] The tagging signal alignment module is used to establish a real-time method of controlling the tagging data based on the event points of the artificially called data packets, and to establish a real-time alignment processing method of the eye movement tagging signal and the eye movement gaze data;

[0045] An offline model verification module is used to establish an IVT offline model, simulate the IVT offline data based on the IVT offline model to generate simulation results, and compare the simulation results with online data;

[0046] The feature conversion module is used to establish a method for converting eye movement events into eye movement feature recognition and complete a real-time eye movement feature calculation algorithm;

[0047] The online operation module is used to connect the IVT offline model to the real-time stream and call the online operation with a variable threshold to complete the construction of the real-time eye tracking event and feature calculation system platform.

[0048] In one aspect of the present disclosure, there is provided an electronic device, comprising:

[0049] Processor; and

[0050] A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the method according to any one of the above items.

[0051] In one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method according to any one of the above items is implemented.

[0052] A method for building a real-time eye tracking event and feature calculation system platform in an exemplary embodiment of the present disclosure establishes an online eye movement signal capture system; based on an IO multiplexing model, a synchronous processing method for generating real-time eye movement signal processing, real-time eye movement event processing, real-time eye movement feature calculation, and real-time eye movement data model algorithm; a real-time eye movement tagging signal and eye movement gaze data alignment processing method is established; a method for converting eye movement events to eye movement feature recognition is established to complete a real-time eye movement feature calculation algorithm; the IVT offline model is connected to a real-time stream, and is called online with a variable threshold to complete the construction of a real-time eye tracking event and feature calculation system platform. The present disclosure realizes the construction of a strong real-time system platform for delay-free eye movement data collection, eye movement event calculation, and eye movement feature calculation through model and algorithm updates.

[0053] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The above and other features and advantages of the present disclosure will become more apparent by describing in detail example embodiments thereof with reference to the attached drawings.

[0055] Figure 1 A flow chart of a method for building a real-time eye tracking event and feature calculation system platform according to an exemplary embodiment of the present disclosure is shown;

[0056] Figure 2 A structural block diagram of a device for building a real-time eye tracking event and feature calculation system platform according to an exemplary embodiment of the present disclosure is shown;

[0057] Figure 3 A block diagram schematically shows an electronic device according to an exemplary embodiment of the present disclosure;

[0058] Figure 4 A schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0059] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0060] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, materials, devices, steps, etc. may be adopted. In other cases, known structures, methods, devices, implementations, materials or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0061] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or these functional entities or parts of functional entities may be implemented in one or more software hardened modules, or these functional entities may be implemented in different networks and / or processor devices and / or microcontroller devices.

[0062] In this example embodiment, a method for building a real-time eye tracking event and feature calculation system platform is first provided; Figure 1 As shown in , the method for building a real-time eye tracking event and feature calculation system platform may include the following steps:

[0063] Step S110, establishing an eye movement online signal capture system based on a preset network framework built by a preset program library;

[0064] Step S120, in the eye movement online signal capture system, based on the IO multiplexing model of the preset software development kit, a synchronous processing method for eye movement real-time signal processing, eye movement real-time event processing, eye movement real-time feature calculation, and eye movement real-time data model algorithm is generated;

[0065] Step S130, establishing a real-time method for controlling the marking data based on the event points of the artificially called data packets, and establishing a real-time method for processing the alignment of the eye movement marking signal and the eye movement gaze data;

[0066] Step S140, establishing and updating an IVT offline model, so that the updated IVT model is updated from two events of fixation and saccade to four events of fixation, saccade, pupil size, and blink, simulating the IVT offline data based on the IVT offline model to generate simulation results, and comparing the simulation results with online data;

[0067] Step S150, establishing a method for converting eye movement events into eye movement feature recognition, and completing a real-time eye movement feature calculation algorithm;

[0068] Step S160, connecting the IVT offline model to the real-time stream, and calling the online operation with a variable threshold, to complete the construction of the real-time eye tracking event and feature calculation system platform.

[0069] A method for building a real-time eye tracking event and feature calculation system platform in an exemplary embodiment of the present disclosure, wherein the method includes: establishing an online eye movement signal capture system; based on an IO multiplexing model, generating a synchronous processing method for real-time eye movement signal processing, real-time eye movement event processing, real-time eye movement feature calculation, and real-time eye movement data model algorithm; establishing a real-time eye movement tagging signal and an alignment processing method for eye movement gaze data; establishing a method for converting eye movement events to eye movement feature recognition, and completing a real-time eye movement feature calculation algorithm; connecting the IVT offline model to a real-time stream, and calling online operation with a variable threshold to complete the construction of a real-time eye tracking event and feature calculation system platform. The present disclosure realizes the construction of a strong real-time system platform for delay-free eye movement data collection, eye movement event calculation, and eye movement feature calculation through model and algorithm updates.

[0070] Next, a method for building a real-time eye tracking event and feature calculation system platform in this example embodiment will be further described.

[0071] Embodiment 1:

[0072] In step S110, an eye movement online signal capture system based on a preset network framework constructed by a preset program library may be established.

[0073] In the embodiment of this example, the method further includes:

[0074] The preset network framework of the preset program library is an asynchronous web framework built based on aiohttp. The asynchronous web framework built based on aiohttp adopts tobbi's software development kit SDK to establish an eye movement online signal capture system.

[0075] In step S120, in the eye movement online signal capture system, based on the IO multiplexing model of the preset software development kit, a synchronous processing method for real-time eye movement signal processing, real-time eye movement event processing, real-time eye movement feature calculation, and real-time eye movement data model algorithm can be generated.

[0076] In the embodiment of this example, the IO multiplexing model based on the preset software development kit in the method further includes:

[0077] Control the process of the IO multiplexing model through marking events and set an asynchronous data PP pool;

[0078] By setting up monitors corresponding to the user-defined data volume, data replication in the data pool is achieved and the data pool is updated in real time;

[0079] A thread for data calculation is added to the asynchronous data PP pool. The thread is set to have an event within the user-defined data volume collection time. When entering the thread, the data volume of each calculation package is locked.

[0080] Based on the asynchronous data PP pool, the computing packages are queued in the data pool and enter the computing channel.

[0081] In step S130, a real-time method of controlling the marking data based on the event points of the artificially called data packets can be established, and a real-time alignment processing method of the eye movement marking signal and the eye movement gaze data can be established.

[0082] In the embodiment of this example, the eye movement tagging signal alignment processing method of the method further includes:

[0083] The marking data is taken as a single channel lead, and the marking data is controlled based on the event points of the artificially called data packet to achieve real-time alignment of the eye movement marking signal and the eye movement gaze data.

[0084] In step S140, an IVT offline model can be established and updated, so that the updated IVT model is updated from two events, gaze and saccade, to four events, gaze, saccade, pupil size, and blink. Based on the IVT offline model, the IVT offline data is simulated to generate simulation results, and the simulation results are compared with the online data.

[0085] In the embodiment of this example, the method also includes gaze, glance, blink, and pupil event calculation:

[0086] The calculation of each fixation time is edited as the fixation end timestamp minus the fixation start timestamp;

[0087] The calculation of each saccade time was edited as the saccade end timestamp minus the saccade start timestamp;

[0088] Edit each blink time to a blank time period greater than 75ms within the time period;

[0089] The pupil diameter at each time is edited as the average value of the valid data of the left and right eyes in the time period.

[0090] In the embodiment of this example, the method further includes calculating eye movement features:

[0091] Fixation duration was defined as the sum of fixation event times within the time period;

[0092] The number of fixations was defined as the sum of the number of fixation time events within the time period;

[0093] The saccade duration was defined as the sum of the saccade event times within the time period;

[0094] The number of saccades was defined as the sum of the number of saccade time events within the time period;

[0095] The distance is defined as the sum of the movement distances of the fixation point per unit time;

[0096] Eye movement velocity is defined as the sum of the speeds of gaze point movement per unit time;

[0097] The number of blinks was defined as the sum of the number of blink events in the time period;

[0098] Blink duration was defined as the sum of blink event times within the time period;

[0099] The pupil diameter is defined as the sum of pupil diameter event values ​​per unit time.

[0100] In step S150, a method for converting eye movement events into eye movement feature recognition may be established to complete a real-time eye movement feature calculation algorithm.

[0101] In step S160, the IVT offline model can be connected to the real-time stream and called to run online with a variable threshold to complete the construction of the real-time eye tracking event and feature calculation system platform.

[0102] In the embodiment of this example, the IVT offline model is connected to the real-time stream, and after being called with a variable threshold, the IVT offline data is simulated to generate simulation results, and the simulation results are compared with the online data to realize the verification of the real-time eye tracking event and feature calculation system platform.

[0103] Embodiment 2:

[0104] In the embodiment of this example, the present disclosure uses the tobbi glass3 model eye tracker as the hardware device of the invention. In the software system, the storage card in the tobbi official SDK is used to call the real-time acquisition data, and then the eye movement events and feature calculation models are used to calculate the eye movement events and feature results in real time.

[0105] The framework of the overall platform is designed using an explicit concurrent model. To align tags, event-triggered synchronization tags are used in the calculation. In order to achieve real-time calculation of events (the definition of real-time calculation in the present invention is to reach 2 / eye tracker sampling rate per second as the fastest. Taking glass3 as an example, the sampling rate of glass3 is 100hz, and the shortest output of an eye movement event is 0.02s, and the user can set a real-time time valve according to their own needs), IVT is used as the event calculation framework in the accessed data packet. After the event calculation is completed, the commonly used features of eye movement are extracted based on the event, and in order to ensure the quality of eye movement data, fill-in-the-blank value and filtering preprocessing are added before IVT calculation. In real-time testing, the threshold of the above fill-in-the-blank value, the filtering threshold, the speed threshold of IVT, and the time valve are all set for users to choose according to their own needs.

[0106] In the embodiment of this example, based on the previous research of this disclosure, it is found that a framework for online eye movement capture without delay can be built. In some studies, it has been proved that a system for capturing an online eye movement signal can be used. By calling the hardware device IP set by glass3, gaze data can be retrieved through the web. There are two complete projects in guihub, one is the official SDK, and the other is the SDK compiled by Columbia University. In the official API, it is an asynchronous web framework built using aiohttp, and a real-time acquisition system designed using a concurrent process model; the other is a threaded web framework built using a client, and a real-time acquisition system designed using a concurrent state machine model. The web data retrieved in the data of this study belongs to a small data packet, a multi-level low-loss development system. When using threads, due to the large resource occupation, it may cause delay in capturing data, so the model used in this disclosure needs to test the delay. Before testing, this disclosure tested the existing online gaze data capture program and found that there was a delay in using the client framework provided by the non-official, and there was no delay in the tobbi official SDK built using the asynchronous web framework built using aiohttp.

[0107] In the embodiment of this example, the preliminary research of the present disclosure found that a method for synchronous processing of online eye movement signals and eye movement event processing algorithms can be realized. First, in the initial test, this study used a serial data collection and calculation method to calculate, and found that the signal system began to block when the second capture package was performed, and the collected packages and calculation results after the blockage could not generate online logs. Then, this study adopted the form of asynchronous I / O to perform calculations while collecting data. It was found that after entering the calculation thread, congestion occurred after calculating the calculation data of the first package. Finally, according to the characteristics of the SDK concurrent process model in tobii, the system module was developed using an explicit concurrent model based on this model, and the process was controlled by marking events, and an asynchronous data parallel process pool was set. After each user-defined data volume, a monitor was added to copy data in the data pool, and the data pool was cleared. A thread for data calculation was added to the asynchronous data parallel process pool. The thread set the event within the user-defined data volume collection time. Each time it entered the thread, a lock was added to lock the data volume of each calculation package. After the result is obtained, if the next package comes in again, it will enter the data pool again and enter the calculation channel. If the next packet comes in before the result comes out, it will enter the data pool and queue up to enter the calculation channel.

[0108] In the embodiment of this example, in the eye movement tagging, the tagging and data can be aligned in the alignment method. In the project using EEG, the tagging data is used as a single channel lead and the data is recorded at 1000 Hz. However, in the eye movement data, although the gaze data and tag data are both retrieved storage card data in the official interface SDK, due to the asynchronous thread, the two belong to channels with different data volumes. When the user issues an instruction, the two channel data are retrieved at the same time. Due to the different amounts of data packets returned after the request is sent in the asynchronous, there will be a certain difference in the transmission speed. In the comparison of the offline data stored in glass3 with the online data stored in the log of this disclosure, it is found that in 20 tests, there are about 3 times when the tag will be faster than the data packet by about 0.03s. In order to prevent the tag transmission speed from being faster than the data speed, when storing online, the event point of the artificial call data packet is used to control the tagging data. After testing, it is found that the tagging and data can be accurately matched.

[0109] In the embodiment of this example, in the offline model building, the first thing to be determined is that IVT is the most real-time data in the current eye movement event algorithm, but its disadvantage is that it is greatly affected by data quality. Therefore, in order to ensure that the data quality is passed, two layers of pre-processing are set up first, namely, null value filling and median filtering. In IVT, the gaze2d coordinates of the online data use resolution, but the speed threshold uses the viewing angle. According to the conversion formula of the viewing angle and the resolution, the resolution is converted into the viewing angle, and the speed threshold is calculated. By judging the speed threshold, it is determined whether the relationship between the two points is fixation or scanning. After the data passes through the IVT model, the start timestamp, end timestamp, and start and end timestamp of the segment will be output. In order to calculate the four events of fixation, scanning, blinking, and pupil size, the calculation of each fixation time is first edited as the end timestamp of fixation minus the start timestamp of fixation, the calculation of the scanning time is edited as the end timestamp of scanning minus the start timestamp of scanning, the blink time is edited as the blank time period greater than 75ms within the time period, and the pupil diameter is edited as the average value of the valid data of the left and right eyes within the time period. In terms of eye movement features, fixation time is defined as the sum of fixation time event times within a time period, fixation times are defined as the sum of fixation time event times within a time period, saccade time is defined as the sum of saccade time event times within a time period, saccade times are defined as the sum of saccade time event times within a time period, distance is defined as the sum of the moving distance of the fixation point per unit time (calculated in the middle stage of the I-VT model), eye movement speed is defined as the sum of the moving speed of the fixation point per unit time (calculated in the middle stage of the I-VT model), blink times are defined as the sum of the blink time event times within a time period, blink time is defined as the sum of the blink time event times within a time period, pupil diameter is defined as the sum of the pupil diameter event values ​​per unit time, and calculations are performed using the above definitions. In the offline model verification stage, data of a few seconds (such as 4s) is intercepted and verified in a loop by segmentation, and the final test results are compared with the tobbi I-VT algorithm in tobiiprolab using the same parameters, and the data results are found to be consistent. In the offline model verification stage, online data access and offline tobbi I-VT algorithm are compared with the same parameters, and the data results are found to be consistent.

[0110] In the embodiment of this example, after various parameters are changed, the offline Tobbi I-VT algorithm is used for comparison with the same parameters, and the data results are found to be consistent. The delay of the event characteristics is verified, and it is found that the pupil diameter, gaze distance, and eye movement speed parameters can be obtained at the same time as the data is entered. The saccade event and the number of saccades can be obtained at 20ms (100HZ eye tracker 10ms one gaze point), the gaze time and the number of gazes can be obtained at 70ms (a large number of studies require that the gaze time be greater than 70ms), and the number of blinks and the blink time can be obtained at 80ms (greater than 75ms is blinking), thus proving that there is no delay.

[0111] In the embodiment of this example, the present disclosure builds a platform for acquiring online data of eye movements and calculating eye tracking events and features in real time, which is valuable in the fields of personnel status monitoring, brain-computer interface control, and automation control, and solves the problem of hard real-time systems in these fields.

[0112] It should be noted that, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0113] In addition, in this exemplary embodiment, a device for building a real-time eye tracking event and feature calculation system platform is also provided. Figure 2 As shown, the real-time eye tracking event and feature calculation system platform construction device 200 may include: an eye movement signal capture module 210, a data synchronization processing module 220, a tag signal alignment module 230, an offline model verification module 240, a feature conversion module 250 and an online operation module 260. Among them:

[0114] The eye movement signal capture module 210 is used to establish an eye movement online signal capture system based on a preset network framework built by a preset program library;

[0115] The data synchronization processing module 220 is used to generate eye movement real-time signal processing, eye movement real-time event processing, eye movement real-time feature calculation, and eye movement real-time data model algorithms in the eye movement online signal capture system based on the IO multiplexing model of the preset software development kit;

[0116] The tagging signal alignment module 230 is used to establish a real-time method for controlling the tagging data based on the event points of the artificially called data packets, and to establish a real-time method for aligning the eye movement tagging signal with the eye movement gaze data;

[0117] The offline model verification module 240 is used to establish and update the IVT offline model, so that the updated IVT model is updated from the two events of fixation and saccade to the four events of fixation, saccade, pupil size, and blink, and simulate the IVT offline data based on the IVT offline model to generate simulation results, and compare the simulation results with the online data;

[0118] The feature conversion module 250 is used to establish a method for converting eye movement events into eye movement feature recognition and complete a real-time eye movement feature calculation algorithm;

[0119] The online operation module 260 is used to connect the IVT offline model to the real-time stream and call the online operation with a variable threshold to complete the construction of the real-time eye tracking event and feature calculation system platform.

[0120] The specific details of each of the above-mentioned real-time eye tracking event and feature calculation system platform building device modules have been described in detail in the corresponding real-time eye tracking event and feature calculation system platform building method, so they will not be repeated here.

[0121] It should be noted that although several modules or units of the real-time eye tracking event and feature calculation system platform building device 200 are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units for concretization.

[0122] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0123] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware embodiments, complete software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0124] Refer to the following Figure 3 The electronic device 300 according to such an embodiment of the present invention is described. Figure 3 The electronic device 300 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0125] like Figure 3 As shown, the electronic device 300 is in the form of a general computing device. The components of the electronic device 300 may include but are not limited to: at least one processing unit 310, at least one storage unit 320, a bus 330 connecting different system components (including the storage unit 320 and the processing unit 310), and a display unit 340.

[0126] The storage unit stores program codes, which can be executed by the processing unit 310, so that the processing unit 310 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 310 can perform the following steps: Figure 1Steps S110 to S160 shown in FIG.

[0127] The storage unit 320 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 3201 and / or a cache memory unit 3202 , and may further include a read-only memory unit (ROM) 3203 .

[0128] The storage unit 320 may also include a program / utility 3204 having a set (at least one) of program modules 3205, such program modules 3205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0129] Bus 330 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0130] The electronic device 300 may also communicate with one or more external devices 370 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 300, and / or communicate with any device that enables the electronic device 300 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 350. Furthermore, the electronic device 300 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 360. As shown, the network adapter 360 communicates with other modules of the electronic device 300 via a bus 330. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0131] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0132] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible embodiments, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to perform the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0133] refer to Figure 4 As shown, a program product 400 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0134] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0135] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0136] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0137] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0138] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0139] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0140] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for building a real-time eye tracking event and feature calculation system platform, characterized in that: The method comprises: Establish an online eye movement signal capture system based on a preset network framework built with a preset program library; In the eye movement online signal capture system, based on the IO multiplexing model of the preset software development kit, a synchronous processing method for generating eye movement real-time signal processing, eye movement real-time event processing, eye movement real-time feature calculation, and eye movement real-time data model algorithm; Establish a real-time method of controlling the marking data based on the event points of manually calling data packets, and establish a real-time method of aligning the eye movement marking signal with the eye movement gaze data; Establish and update the IVT offline model, so that the updated IVT model is updated from two events of fixation and saccade to four events of fixation, saccade, pupil size, and blink, simulate the IVT offline data based on the IVT offline model to generate simulation results, and compare the simulation results with the online data; Establish a method to convert eye movement events into eye movement feature recognition and complete a real-time eye movement feature calculation algorithm; The IVT offline model is connected to the real-time stream and called to run online with a variable threshold to complete the construction of the real-time eye tracking event and feature calculation system platform.

2. The method according to claim 1, characterized in that The method further comprises: The preset network framework of the preset program library is an asynchronous web framework built based on aiohttp. The asynchronous web framework built based on aiohttp adopts tobbi's software development kit SDK to establish an eye movement online signal capture system.

3. The method according to claim 1, characterized in that The IO multiplexing model based on the preset software development kit in the method also includes: Control the process of the IO multiplexing model through marking events and set an asynchronous data PP pool; By setting up monitors corresponding to the user-defined data volume, data replication in the data pool is achieved and the data pool is updated in real time; A thread for data calculation is added to the asynchronous data PP pool. The thread is set to have an event within the user-defined data volume collection time. When entering the thread, the data volume of each calculation package is locked. Based on the asynchronous data PP pool, the computing packages are queued in the data pool and enter the computing channel.

4. The method according to claim 1, characterized in that The eye movement tagging signal alignment processing method of the method also includes: The marking data is taken as a single channel lead, and the marking data is controlled based on the event points of the artificially called data packet to achieve real-time alignment of the eye movement marking signal and the eye movement gaze data.

5. The method according to claim 1, characterized in that The method also includes gaze, saccade, blink, and pupil event calculations: The calculation of each fixation time is edited as the fixation end timestamp minus the fixation start timestamp; The calculation of each saccade time was edited as the saccade end timestamp minus the saccade start timestamp; Edit each blink time to a blank time period greater than 75ms within the time period; The pupil diameter at each time is edited as the average value of the valid data of the left and right eyes in the time period.

6. The method according to claim 5, characterized in that The method also includes eye movement feature calculation: Fixation duration was defined as the sum of fixation event times within the time period; The number of fixations was defined as the sum of the number of fixation time events within the time period; The saccade duration was defined as the sum of the saccade event times within the time period; The number of saccades was defined as the sum of the number of saccade time events within the time period; The distance is defined as the sum of the distances moved by the fixation point per unit time; Eye movement velocity is defined as the sum of the speeds of gaze point movement per unit time; The number of blinks was defined as the sum of the number of blink events in the time period; Blink duration was defined as the sum of blink event times within the time period; The pupil diameter is defined as the sum of pupil diameter event values ​​per unit time.

7. The method according to claim 1, characterized in that The method further comprises: The IVT offline model is connected to the real-time stream and retrieved with a variable threshold, the IVT offline data is simulated to generate simulation results, and the simulation results are compared with the online data to verify the real-time eye tracking event and feature calculation system platform.

8. A device for building a real-time eye tracking event and feature calculation system platform, characterized in that: The device comprises: An eye movement signal capture module is used to establish an eye movement online signal capture system based on a preset network framework built by a preset program library; A data synchronization processing module is used to generate an eye movement real-time signal processing, an eye movement real-time event processing, an eye movement real-time feature calculation, and an eye movement real-time data model algorithm in the eye movement online signal capture system based on an IO multiplexing model of a preset software development kit; The tagging signal alignment module is used to establish a real-time method of controlling the tagging data based on the event points of the artificially called data packets, and to establish a real-time alignment processing method of the eye movement tagging signal and the eye movement gaze data; An offline model verification module is used to establish and update an IVT offline model, so that the updated IVT model is updated from two events of fixation and saccade to four events of fixation, saccade, pupil size, and blink, and based on the IVT offline model, the IVT offline data is simulated to generate simulation results, and the simulation results are compared with the online data; The feature conversion module is used to establish a method for converting eye movement events into eye movement feature recognition and complete a real-time eye movement feature calculation algorithm; The online operation module is used to connect the IVT offline model to the real-time stream and call the online operation with a variable threshold to complete the construction of the real-time eye tracking event and feature calculation system platform.

9. An electronic device, characterized in that: include Processor; and A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images