A method and system for measuring blink frequency using a finite state machine
Through the finite parent-child state machine and fault tolerance mechanism, the problems of large overhead calculation and sensitivity of blink frequency measurement in the prior art are solved, and an efficient and robust blink frequency measurement method is realized, reducing the dependence on deep learning models.
Patent Information
- Application Number
- CN202411239706.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-05
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-09-05
AI Technical Summary
The existing automated blink frequency measurement technology has problems such as large calculation overhead, strong data dependence, sensitive environmental changes and cumbersome processes. In particular, deep learning models require a large amount of calculation and labeling of data when acquiring eye key points, and are not reckless enough for environmental changes. Great.
A finite parent-child state machine is used to represent the changes in eye characteristics through continuous Boolean sequences, define the parent and child states, increase the fault tolerance mechanism, reduce calculation overhead, and improve the robustness of environmental changes, and avoid using deep learning models with large parameters.
It realizes more efficient blink frequency measurement, reduces calculation overhead and data labeling requirements, enhances the stability and adaptability of the system, and reduces the impact of external interference on detection accuracy.
Smart Images

Figure CN118762836B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of eye trackers, and in particular to a blink frequency determination method and system using a finite element state machine. Background Art
[0002] An eye tracker is a device used to track and record eye movements, and is widely used in cognitive science, medical diagnosis, education, and other fields. By tracking and analyzing the full process of eye movement data, it is possible to obtain multiple indicators such as fixation point, blink frequency, pupil diameter, fixation time, etc., providing researchers with a basis for in-depth understanding of visual behavior and cognitive processes.
[0003] The blinking frequency index not only reflects an individual's mental state, fatigue level, and attention distribution, but is also closely related to abnormalities in the nervous system, and is of significant importance in the field of medical diagnosis. Studies have shown that abnormal changes in blinking frequency can provide doctors with valuable diagnostic clues and discover some potential health problems. For example, patients with certain neurological diseases will have a change in blinking frequency as the disease progresses. By analyzing these changes, doctors can better understand the progression of the disease and adjust treatment plans. At the same time, blinking frequency data can also be used to evaluate treatment effects and help doctors determine whether a certain treatment method is effective. By monitoring and analyzing blinking frequency, key information about the patient's eye health and overall health status can be obtained, thereby improving the early detection rate and diagnostic accuracy of the disease.
[0004] Common eye movement information acquisition devices are divided into wearable and desktop types according to their placement. Wearable devices mainly fix the device on the user's head area, track the pupil or iris through a camera and infrared light source, and use an eye feature recognition algorithm or model to record the movement information of the eyeball in real time. The main difference between desktop eye trackers and wearables is that they are installed in different locations, usually near the display. Therefore, their application scenarios are also different. Wearables are more suitable for daily life due to their convenience and flexibility. Given the stability and high precision of desktop devices, they are often used in scientific fields such as psychology and medical auxiliary diagnosis.
[0005] The existing automated blink frequency measurement technology is the contact sensor type. The most common is to wear a contact blink detection device, which measures the blink frequency by recording and analyzing the electrophysiological signals of the eye, such as the electrooculogram (EOG) and the electroretinogram (ERG). The advantage of electrophysiological signal analysis is that it can provide high-precision data and is not affected by external light and facial expressions. However, this method requires complex electrodes and signal processing equipment, which is costly, and wearing electrodes may cause discomfort to the person being tested, and the experience is low in comfort. It is not suitable for special groups to wear, and this method will fail.
[0006] The existing automatic blink frequency measurement technology uses two sensors and is non-contact. The common ones can be divided into microwave radar method and video analysis method. The microwave radar method uses microwave radar technology to detect eye movement, and judges the opening and closing state of the eyelid by analyzing the reflected signal, thereby recording the number of blinks. The advantage is that it can penetrate some obstructions, is insensitive to external light, and is suitable for use in complex environments. The disadvantage is that the equipment cost is high, and the processing requirements for radar signals are high, requiring complex signal processing algorithms. The video analysis method uses a high frame rate camera to record the eye video of the subject, and then automatically detects and records the number of blinks through a video analysis algorithm. The existing algorithms include motion estimation based on the eye area, where the face and eyes are detected by a Viola-Jones type detector, and the movement of the eye area is estimated by optical flow sparse tracking or frame-by-frame intensity difference and adaptive threshold. Finally, a decision is made as to whether the eyes are covered by the eyelids. Another method is to infer the open state of the eyes from a single image, for example, by correlation matching with open and closed eye templates, by heuristic methods of horizontal or vertical image intensity projection of the eye area, by parameter fitting to find the eyelids, or by using active shape models, with relatively limited accuracy. With the popularity of deep learning in recent years, deep learning models are often used to identify key points in the face or eye area, and the eye area aspect ratio (EAR) is calculated based on the distance of specific key points to determine whether a blinking action occurs. The advantage is that it can provide relatively accurate blinking data and can monitor for a long time. However, the disadvantages of this method are:
[0007] (1) The initial amount of computation is large. Although the amount of computation required to calculate the aspect ratio of the eye area is not large, in order to obtain specific eye key points, the model needs to have good generalization ability. Deep learning models with large parameters are often selected, resulting in high computational overhead. Hardware resources such as GPU and NPU are required to meet performance requirements.
[0008] (2) Strong data dependence. The accuracy of the model usually depends on a large amount of labeled data. Training high-quality deep learning models requires a large amount of labeled image data, which may be difficult to obtain in some scenarios with high data privacy requirements.
[0009] (3) Sensitivity to environmental changes. Although deep learning models are much more adaptable to the environment, they may still be affected by changes in lighting, facial expressions, and postures. This may limit recognition under different environmental conditions, reduce the accuracy of related key points, and lead to errors in the calculation of eye aspect ratio.
[0010] (4) The process is cumbersome. The deep learning model used in this method usually requires a training process before deployment, which is time-consuming and cumbersome. Summary of the invention
[0011] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method and system for determining blinking frequency using a finite element state machine, which mainly solves the following problems: (1) Avoiding the use of specific eye key points to obtain blinking status information; the current solution for obtaining eye key points is a subset of the entire facial key feature points, which will result in a large amount of unnecessary calculations; (2) Avoiding the use of deep learning models with large parameters to reduce the overall computing overhead; at the same time, reducing the heavy physical labor such as cleaning and labeling of large amounts of data; the blinking frequency can be determined by a lightweight model and a small amount of data, or even without using a deep learning model; (3) Improving robustness to environmental changes; the present invention reduces errors caused by external interference by adding a fault-tolerant mechanism, reduces the impact of these changes on the accuracy of blink detection, and enhances stability and adaptability.
[0012] The purpose of the present invention is achieved through the following technical solutions:
[0013] In a first aspect, a method for measuring blink frequency using a finite element state machine is provided, the specific steps of which are:
[0014] Step 100: Obtain the real-time changes of eye features in a continuous time period, using a continuous Boolean sequence indicates; among them , The Boolean value of the eye feature at the i-th time point, 1-indicates that the eye feature is detected, 0-indicates that the eye feature is not detected;
[0015] Step 200: Update the current parent state, count through the counter, and store the state sequence Make updates;
[0016] Step 201: Define the state of the state machine: -Current state, which is the parent state; - Initial state / idle state, which is the parent state; -Eye feature detectable state, which is the mother state; -Eye features are undetectable, indicating the mother state; -Feature detection occasional mutation sub-state, specifically refers to a mutation from detecting eye features to not detecting eye features, or a mutation from not detecting eye features to detecting eye features, which is a sub-state and is also represented by a Boolean value;
[0017] Step 202: Parameter definition and initialization of the state machine:
[0018] Tolerance : The number of errors tolerated, specifically refers to the number of consecutive failures to detect eye features or consecutive detections of eye features. The initialization value is defined according to the eye feature recognition algorithm used;
[0019] Idle Threshold : The threshold for transitioning from the undetectable state of eye features to the idle state, with the initial value defined according to the eye feature recognition algorithm used;
[0020] counter : Detectable eye feature counter, initialized to 0;
[0021] counter : Undetectable eye feature counter, initialized to 0;
[0022] counter : Error counter, which can be used for both detected eye features and undetected eye features, initialized to 0, and starts from 0 every time it is reset;
[0023] Storage state sequence : Stores the determined valid state sequence of eye activity , initialized to an empty set ;in, represents the parent state of the jth change, if The corresponding parent state is but Use Boolean value 1 if The corresponding parent state is but Represented by Boolean value 0;
[0024] Step 203: The finite child state machine receives in real time the Boolean sequence after the eye feature recognition. , and perform state transfer changes:
[0025] (1) At the beginning, the current parent state is , that is, idle state:
[0026] a. If the eye feature has not been detected, then: the current mother state is always ;counter The count is 0, the counter The count is 0; the storage state sequence is an empty set;
[0027] b. If the eye feature is detected for the first time in the subsequent test, then: the parent state is immediately updated to ;counter Increase by 1; because the parent state changes, the parent state The corresponding Boolean value is updated to the storage state sequence , used to record the current state transfer;
[0028] (2) The current parent state is hour:
[0029] a. If the next test still detects eye features, the parent state remains ,counter Increase by 1; since the parent state has not changed, the state sequence is stored No update;
[0030] b. If the eye feature is not detected next time, enter the sub-state ; If the eye features are not detected for multiple consecutive times in the subsequent detection, and the number of consecutive times the eye features are not detected is less than or equal to the set tolerance , it indicates a false detection;
[0031] At this stage: mother state Remain unchanged; pass the counter Count the number of detections and add the obtained number to the counter ; Update all Boolean values in the continuous sequence of this stage to 1, and store the state sequence No update;
[0032] c. If the eye feature is not detected next time, enter the sub-state ; If the eye features are not detected for multiple consecutive times in the subsequent detection, and the number of consecutive times the eye features are not detected is greater than the set tolerance , it indicates that it is not a false detection;
[0033] At this stage: the parent state is updated to ; also through the counter Count the number of detections and add the obtained number to the counter ; Update all Boolean values in the continuous sequence of this stage to 0, and change the parent state The corresponding Boolean value is updated to the storage state sequence ;
[0034] (3) The current parent status is hour:
[0035] a. If the eye feature is not detected next time, the parent state remains ,counter Increase by 1; since the parent state has not changed, the storage state sequence is not updated ;
[0036] b. If the eye feature is detected next time, enter the sub-state If the eye features are detected multiple times in a row in the subsequent detection, and the number of times the eye features are detected multiple times is less than or equal to the set tolerance , it indicates a false detection;
[0037] At this stage: mother state Remain unchanged; pass the counter Count the number of detections and add the obtained number to the counter ; Update all Boolean values in the continuous sequence in this stage to 0 and store the state sequence No update;
[0038] c. If the eye feature is detected next time, enter the sub-state , if the eye features are detected multiple times in the subsequent detection, and the number of times the eye features are detected multiple times is greater than the tolerance , it indicates that it is not a false detection;
[0039] At this stage: the parent state is updated to ; Through the counter Count the number of detections and add the obtained number to the counter ; Update all Boolean values in the continuous sequence in this stage to 1, and change the parent state The corresponding person Boolean value is updated to the storage state sequence ;
[0040] Step 300: Blinking pattern determination;
[0041] According to the final storage state sequence obtained in the state machine , determine the blinking pattern and calculate the number of blinks ;
[0042] The formula for calculating the number of blinks is:
[0043]
[0044] Among them, Ⅱ represents the indicator function. When the condition is 1, Ⅱ returns 1, otherwise Ⅱ returns 0; N represents a natural number, represents the j-th parent state, represents the j-1th parent state, represents the j-2th parent state, j represents The index of the sequence.
[0045] In some embodiments, in (2) b above, specifically refers to:
[0046] If the eye feature is not detected next time, enter - is the sub-state, and the subsequent detection continues. If the eye features are not detected for multiple consecutive times in the subsequent detection, and the number of consecutive failures to detect the eye features is less than or equal to the set tolerance , it indicates a false detection; the period from the failure to detect eye features for multiple consecutive times to the first subsequent detection of eye features is considered Time period;
[0047] At this time Time period: Mother state Remain unchanged; counter The count starts from 0 and increases by 1 if the eye feature is not detected once. The count increases, and the increased count is 1 + counter Cumulative count; update all Boolean values in the continuous sequence within the time period to 1, and store the state sequence Not updated.
[0048] In some embodiments, in (2) c, specifically refers to:
[0049] If the eye feature is not detected next time, enter - is the sub-state, and the subsequent detection continues. If the eye features are not detected for multiple consecutive times in the subsequent detection, and the number of consecutive failures to detect the eye features is greater than the tolerance , it indicates that it is not a false detection; the time period in which the eye features are not detected for many consecutive times is regarded as Time period;
[0050] At this time Time period: Parent status updated to ;counter Start counting from 0, and increase by 1 each time the eye feature is not detected, and add the accumulated count of this time period to the counter ; Update all Boolean values in the continuous sequence within this time period to 0, The corresponding Boolean value is updated to the storage state sequence .
[0051] In some embodiments, in (3) b, specifically refers to:
[0052] b. If the eye feature is detected next time, enter - That is, the sub-state, subsequent continuous detection, if the eye features are detected multiple times in the subsequent detection, and the number of consecutive eye features detected is less than or equal to the set tolerance , it indicates a false detection; the period from the time when the eye features are detected multiple times in succession to the time when the eye features are not detected for the first time is considered as Time period;
[0053] At this time Time period: Mother state Remain unchanged; counter The counter starts counting from 0 and increases by 1 each time an eye feature is detected. The count increases, and the increased count is 1 + counter Cumulative count; update all Boolean values in the continuous sequence within the time period to 0, and store the state sequence Not updated.
[0054] In some embodiments, in (3) c, specifically refers to:
[0055] If the eye feature is detected next time, enter - is the sub-state, and the subsequent detection continues. If the eye features are detected multiple times in the subsequent detection, and the number of consecutive eye features detected is greater than the tolerance , it indicates that it is not a false detection; the time period in which the eye features are detected multiple times in succession is regarded as Time period;
[0056] At this time Time period: Parent status updated to ;counter Start counting from 0, and increase by 1 each time an eye feature is detected, and add the accumulated count of this time period to the counter ; Update all Boolean values in the continuous sequence within this time period to 1, The corresponding Boolean value is updated to the storage state sequence .
[0057] In some embodiments, in the blinking pattern determination, the blinking pattern determination needs to maintain a pointer to the beginning of the next window of the state machine, which is used to determine the state of the beginning of the window. It needs to start from the state of detecting eye features, and terminate the determination when encountering the end of the sequence or the end signal during the determination process.
[0058] In some embodiments, the eye feature is one of a pupil center, a pupil area, an iris, or a light spot generated by an infrared light source.
[0059] In a second aspect, a blink frequency determination system using a finite element state machine is provided, which is used to run any of the methods described above.
[0060] The present invention has the following advantages:
[0061] The technical solution provided by the present invention adopts a Boolean sequence based on whether the eye features are obtained continuously, avoids using the eye aspect ratio to represent the open and closed state of the eyes, and further abandons the method of using a deep learning model with large parameters to relatively accurately obtain the key points of facial and eye features. The acquisition of eye features can be selected in a flexible way according to the final performance, frame rate and accuracy requirements.
[0062] The present invention has three main states: idle state , Eye feature detectable status and eye feature undetectable state The state transition relationship between the parent states has clear rules and logical processes. There are only two transfer directions for the idle state. One is that if the eye feature is not detected, it can only transfer to its own idle state; the other is that if the eye feature is detected, it will immediately switch to the idle state. . It is detected that there is only one main transfer direction of the characteristic state. When the transfer condition is met, the current state will be transferred to There are two transfer directions when the characteristic state is not detected. One is to transfer to ; The other is that if no valid eye features are detected for a long time, it will switch to the idle state .
[0063] Furthermore, adding a fault-tolerant mechanism throughout the state transfer process can correct occasional errors in the selected eye feature recognition algorithm. This is mainly achieved by adding a sub-state with occasional mutations to the parent state, and realizing the correct conversion of the eye state through the combination of the parent and child states. When the parent state is in the state of detecting or not detecting eye features, when encountering the opposite detection sequence results, the state transfer is not performed directly, but by setting an error counter to cache the current possible abnormal results. When the tolerance is exceeded, it will be considered that there is no misidentification and the state transfer is performed, which has the effect of delaying the transfer of the parent state and avoiding mistransfer. Otherwise, no state transfer is performed.
[0064] During the state transition process of the entire finite-child state machine, the valid state will be saved to form an eye activity state sequence, and then through sliding window matching, the pattern sequence that meets the blinking requirements will be found. The required blinking frequency can be obtained through simple subsequent processing such as accumulation. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] Figure 1 A schematic diagram of a flow chart of an embodiment of the present invention;
[0066] Figure 2 Schematic diagram of eye features in the embodiment;
[0067] Figure 3 Schematic diagram for determining blinking patterns;
[0068] 200 - eyelid, 202 - sclera, 204 - iris, 206 - pupil area, 208 - infrared reflection spot. DETAILED DESCRIPTION
[0069] The present invention is further described below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following description.
[0070] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments may be combined with each other.
[0071] By analyzing the problem that the frame rate of current domestic eye trackers is difficult to improve, the present invention proposes a blink frequency measurement method that can take into account a variety of eye feature detection algorithms, avoiding the use of a method that uses a deep learning model with large computational complexity to obtain key feature points of the eyelid 200 and other surrounding features, so that blink frequency measurement is no longer a bottleneck in the selection of eye feature detection algorithms.
[0072] The main process of this embodiment is as follows Figure 1 and Figure 2 As shown, the data is mainly obtained by a high frame rate camera that captures facial information. The eye area image is captured by the camera, and the eye feature recognition algorithm selected by the user is applied. The eye features include but are not limited to the pupil center, sclera 202, iris 204, pupil area 206, infrared reflection spot 208, etc. This embodiment first needs to initialize the system, including various states, parameters, and thresholds. Then, based on the recognition of the pupil area 206 in the eye in the image as an eye feature, a Boolean sequence of eye features is formed through the recognition results of continuous frames. According to the Boolean sequence, the finite child state machine is continuously converted between states in an orderly manner. Then a valid eye activity sequence is formed. Finally, the determination of the blinking pattern and the acquisition of the final blinking frequency are completed.
[0073] In a specific embodiment of the present invention, refer to 1 and Figure 3 , a method for determining blink frequency using a finite state machine is provided, and the specific steps are as follows:
[0074] Step 100: Obtain a continuous Boolean sequence of eye features.
[0075] A high frame rate camera is used in conjunction with several infrared light source transmitters for facial information collection and eye area positioning. According to the selected eye feature recognition scheme, in addition to providing information such as the shape of the eye feature or key point coordinates, it is very convenient to obtain the existence of the features of the current eye area. The eye feature can be one of the pupil center, pupil area, iris or light spot generated by infrared light source, or other conventional eye features.
[0076] High-speed continuous frame recognition can obtain a continuous eye feature Boolean sequence of eye features indicates; among them , Indicates the Boolean value of the eye feature at the i-th time point, 1- indicates that the eye feature is detected, and 0- indicates that the eye feature is not detected.
[0077] Step 200: Update the current parent state, count through the counter, and store the state sequence to update.
[0078] Step 201: It should be noted that the state of the state machine needs to be defined. -Current status. -Initial state / idle state, which is the parent state. -Eye feature detectable status, which is the mother status. -Eye features are undetectable, which is the mother state. -Feature detection occasional mutation sub-state, specifically refers to a mutation from detected eye features to undetected eye features, or a mutation from undetected eye features to detected eye features, which is a sub-state and is also represented by a Boolean value.
[0079] Step 202: It should also be noted that the parameters of the state machine need to be defined. : The number of errors tolerated, specifically the number of consecutive failures to detect eye features or consecutive detections of eye features. The initialization value is defined based on the eye feature recognition algorithm used. Idle Threshold : The threshold for transitioning from the undetectable eye feature state to the idle state. The initial value is defined according to the eye feature recognition algorithm used. Counter : Detectable eye feature counter, initialized to 0. Counter : Undetectable eye feature counter, initialized to 0. Counter : Error counter, which can be used for both detected eye features and undetected eye features, initialized to 0, and counts from 0 after each reset. Storage state sequence : Stores the determined valid state sequence of eye activity, that is, a continuous Boolean sequence The processed valid state sequence is initialized to an empty set .
[0080] It should be noted that the finite state machine is initialized first. The current state of the current state machine The initial state is set to idle state ; Occasional mutant state is false; counter counter ,counter ,counter All are 0; tolerance and idle threshold The setting depends on the selected eye feature recognition algorithm. The setting principle is that the higher the recognition algorithm accuracy, the corresponding tolerance The smaller the idle threshold is, the smaller the eye feature activity is stored in the sequence Initialized to an empty set .
[0081] Step 203: The finite child state machine receives in real time the Boolean sequence after the eye feature recognition. , and perform state transfer transformation. The details are as follows:
[0082] (1) At the beginning, the current parent state is in idle state :
[0083] a. If the eye feature has not been detected, then: the current mother state is always ;counter The count is 0, the counter The count is 0; the storage state sequence is an empty set;
[0084] b. If the eye feature is detected next time, or when the received When it is 1, then: immediately switch the current mother state to ;counter Increase by 1; because the parent state changes and the parent state is ,Will Update to the storage state sequence as 1 .
[0085] For example, the above (1) can be expressed as:
[0086] Current state (initial state): S = SIdle , enter the update process,
[0087] If the eye feature is detected (isDetected=1), then:
[0088] Update current status ← ;
[0089] counter Add 1;
[0090] renew ← { };
[0091] otherwise:
[0092] Keep the current state unchanged.
[0093] Current status: = , enter the update process.
[0094] (2) The current parent state is hour:
[0095] a. If the next test still detects eye features, the parent state remains ,counter Increase by 1; since the parent state does not change, store the state sequence No update;
[0096] b. If the eye feature is not detected next time, enter - is the sub-state, at this time The Boolean value is true, and the subsequent detection continues. If the eye features are not detected for multiple consecutive times in the subsequent detection, and the number of consecutive failures to detect the eye features is less than or equal to the set tolerance , it indicates a false detection; the period from the failure to detect eye features for multiple consecutive times to the first subsequent detection of eye features is considered Time period;
[0097] At this time Time period: Mother state Remain unchanged; counter The count starts from 0 and the counter is incremented every time the eye feature is not detected. Increment 1, counter The count number is the number of times the eye features are detected in this time period; the counter Count up, counter Increased count = 1 + counter The accumulated number of times, where 1 is the first time that the eye feature is detected in the subsequent period; are all updated to 1; since the parent state does not change, the state sequence is stored at this time No update;
[0098] It should be noted that at this time Time period: counter Start counting from 0, and increase by 1 each time the eye feature is not detected. Stop counting when the eye feature is detected for the first time in the time period, and then add the accumulated count to the counter. Then initialize to 0; but after the time period is over, reset to false;
[0099] c. If the eye feature is not detected next time, enter - is the sub-state, at this time Boolean value is true, and subsequent detection continues. If the eye features are not detected for multiple consecutive times in the subsequent detection, and the number of consecutive failures to detect the eye features is greater than the tolerance , it indicates that it is not a false detection; the time period in which the eye features are not detected for many consecutive times is regarded as Time period;
[0100] At this time Time period: Parent status updated to ;counter The count starts from 0, and the counter is incremented every time the eye feature is not detected. Increment 1, counter The accumulated number is the number of times the eye feature is not detected during this time period; counter The count increases, the counter Increased count = counter within this time period The cumulative number of times; all are updated to 0; because the parent state changes and changes to , will correspond Update to 0 to store the status sequence ;
[0101] It should be noted that at this time Time period: counter Start counting from 0, increase by 1 each time the eye feature is not detected, and then add the accumulated technology to the counter , and then initialized to 0; but after the time period is over, Reset to false.
[0102] For example, the above (2) can be expressed as:
[0103] Current status: = , enter the update process,
[0104] ①If the eye feature is detected (isDetected=1), then:
[0105] If in the occasional mutation substate and the error counter is less than the tolerance, =1, < ,but: = + ;
[0106] Reset and , ←0, ←0; counter Add 1;
[0107] ②If the eye feature is not detected (isDetected=0), then:
[0108] If the error counter is less than the tolerance, ≤ ,but:
[0109] ← +1;
[0110] Always enter the occasional mutation state first , ←1;
[0111] Reset , ←0;
[0112] otherwise:
[0113] Update current status ← ;renew ← { };
[0114] counter ←( ≠0)?U + : +1 Reset and , ←0, ←0;
[0115] Reset ←0.
[0116] (3) The current parent status is hour:
[0117] a. If the eye feature is not detected next time, the parent state remains ,counter Increase by 1; since the parent state does not change, the state sequence is stored No update;
[0118] b. If the eye feature is detected next time, enter - That is, the sub-state, at this time The Boolean value is true, and the subsequent detection continues. If the eye features are detected multiple times in a row in the subsequent detection, and the number of consecutive eye features detected is less than or equal to the set tolerance , it indicates a false detection; the period from the time when the eye features are detected multiple times in succession to the time when the eye features are not detected for the first time is considered as Time period;
[0119] At this time Time period: Mother state Remain unchanged; counter The count starts from 0, and the counter increases every time an eye feature is detected. Increment 1, counter The accumulated number is the number of times the eye features are detected during this time period; counter Count up, counter Increased count = 1 + counter The accumulated number of times, where an increase of 1 is the first time that the eye feature is not detected in the subsequent period; all the times in the period are all updated to 0; since the parent state does not change, the state sequence is stored No update;
[0120] It should be noted that at this time Time period: counter Start counting from 0, and increase by 1 each time an eye feature is detected. Stop counting when the eye feature is not detected for the first time in the time period, and then add the accumulated count to the counter. Then initialize to 0; but after the time period is over, reset to false;
[0121] c. If the eye feature is detected next time, enter - is the sub-state, at this time The Boolean value is 1, and the subsequent detection continues. If the eye features are detected multiple times in the subsequent detection, and the number of consecutive eye features detected is greater than the tolerance , it indicates that it is not a false detection; the time period in which the eye features are detected multiple times in succession is regarded as Time period;
[0122] At this time Time period: Parent status updated to ;counter The count starts from 0, and increases by 1 each time an eye feature is detected, and the accumulated count during this time period is added to the counter. ; All the are updated to 1; because the parent state changes and the parent state is , so the corresponding Update to the storage state sequence as 1 ;
[0123] It should be noted that at this time Time period: counter Start counting from 0, increase by 1 each time an eye feature is detected, and then add the accumulated technology to the counter , and then initialized to 0; but after the time period is over, Reset to false.
[0124] For example, the above (3) can be expressed as:
[0125] Current status: = , enter the update process,
[0126] ① If the eye feature is not detected (isDetected=0), then:
[0127] If in the occasional mutation substate and the error counter is less than the tolerance, =1, < ,but: = + ;
[0128] Reset and , ECounter←0, ←0; counter r increases by 1;
[0129] If the counter Greater than the idle threshold IT, >T, then:
[0130] ← ;
[0131] ←0;
[0132] Reset the current child state and parameters, { ←0;
[0133] ←0, ←0;
[0134] renew ← { };
[0135] ②If the eye feature is detected (isDetected=1), then:
[0136] If the error counter is less than the tolerance, ≤T, then: counter Add 1;
[0137] Always enter the occasional mutation state first , ←1;
[0138] Reset , ←0;
[0139] otherwise:
[0140] Update the current state S← ;renew ← { };
[0141] counter ←( ≠0)? + : +1;
[0142] Reset and , ←0, ←0; reset ←0.
[0143] The above state machine runs until the end signal is issued. No matter which parent state it is in, the end signal can be issued, that is, the life cycle of the current finite child parent state machine instance ends.
[0144] Step 300: Blink pattern determination.
[0145] Storage state sequence The blinking pattern determination can be determined in real time in the above finite element state machine, or can be determined after the end. For ease of understanding, this embodiment adopts an offline determination method after the end to illustrate.
[0146] like Figure 3 As shown:
[0147] The eye activity after the finite letter state machine is an effective state sequence that eliminates occasional errors. The number of blinks per unit time can be obtained directly through a sliding window matching with a window of 3; according to the final storage state sequence obtained in the state machine , determine the blinking pattern and calculate the number of blinks ;
[0148] The formula for calculating the number of blinks is:
[0149]
[0150] Among them, Ⅱ represents the indicator function. When the condition is 1, Ⅱ returns 1, otherwise Ⅱ returns 0; N represents a natural number, represents the j-th parent state, represents the j-1th parent state, represents the j-2th parent state, j represents The index of the sequence.
[0151] It should be noted that the blinking pattern determination algorithm needs to maintain a pointer to the beginning of the next window of the state machine, which is used to determine the state of the beginning of the window. It needs to start from the state of detecting eye features and terminate the determination when encountering the end of the sequence or the end signal during the determination process.
[0152] The above embodiments only express preferred implementation modes, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention.
Claims
1. A method for measuring blink frequency using a finite element state machine, characterized in that: The following steps are involved: Step 100: Obtain the real-time changes of eye features in a continuous time period, using a continuous Boolean sequence indicates; among them , Indicates the Boolean value of the eye feature at the i-th time point, 1- indicates that the eye feature is detected, 0- indicates that the eye feature is not detected; Step 200: Update the current parent state, count through the counter, and store the state sequence Make updates; Step 201: Define the state of the state machine: - Current status; - Initial state / idle state; -Eye feature detectable state, which is the mother state; -Eye features are undetectable, indicating the mother state; -Feature detection occasional mutation sub-state, specifically refers to a mutation from detecting eye features to not detecting eye features, or a mutation from not detecting eye features to detecting eye features, which is a sub-state and is also represented by a Boolean value; Step 202: Parameter definition and initialization of the state machine: Tolerance : The number of errors tolerated, specifically refers to the number of consecutive failures to detect eye features or consecutive detections of eye features. The initialization value is defined according to the eye feature recognition algorithm used; Idle Threshold : The threshold for transitioning from the undetectable state of eye features to the idle state, with the initial value defined according to the eye feature recognition algorithm used; counter : Detectable eye feature counter, initialized to 0; counter : Undetectable eye feature counter, initialized to 0; counter : Error counter, which can be used for both detected eye features and undetected eye features, initialized to 0, and starts from 0 every time it is reset; Storage state sequence : Stores the determined valid state sequence of eye activity , initialized to an empty set ;in, represents the parent state of the jth change, if The corresponding parent state is but Use Boolean value 1 if The corresponding parent state is but Represented by Boolean value 0; Step 203: The finite child state machine receives in real time the Boolean sequence after the eye feature recognition. , and perform state transfer changes: (1) At the beginning, the current parent state is , that is, idle state: a. If the eye feature has not been detected, then: the current mother state is always ;counter The count is 0, the counter The count is 0; the storage state sequence is an empty set; b. If the eye feature is detected for the first time in the subsequent test, then: the parent state is immediately updated to ;counter Increase by 1; because the parent state changes, the parent state The corresponding Boolean value is updated to the storage state sequence , used to record the current state transfer; (2) The current parent state is hour: a. If the next test still detects eye features, the parent state remains ,counter Increase by 1; since the parent state has not changed, the state sequence is stored No update; b. If the eye feature is not detected next time, enter the sub-state ; If the eye features are not detected for multiple consecutive times in the subsequent detection, and the number of consecutive times the eye features are not detected is less than or equal to the set tolerance , it indicates a false detection; At this stage: mother state Remain unchanged; pass the counter Count the number of detections and add the obtained number to the counter ; Update all Boolean values in the continuous sequence of this stage to 1, and store the state sequence No update; c. If the eye feature is not detected next time, enter the sub-state ; If the eye features are not detected for multiple consecutive times in the subsequent detection, and the number of consecutive times the eye features are not detected is greater than the set tolerance , it indicates that it is not a false detection; At this stage: the parent state is updated to ; also through the counter Count the number of detections and add the obtained number to the counter ; Update all Boolean values in the continuous sequence of this stage to 0, and change the parent state The corresponding Boolean value is updated to the storage state sequence ; (3) The current parent state is hour: a. If the eye feature is not detected next time, the parent state remains ,counter Increase by 1; since the parent state has not changed, the storage state sequence is not updated ; b. If the eye feature is detected next time, enter the sub-state If the eye features are detected multiple times in a row in the subsequent detection, and the number of times the eye features are detected multiple times is less than or equal to the set tolerance , it indicates a false detection; At this stage: mother state Remain unchanged; pass the counter Count the number of detections and add the obtained number to the counter ; Update all Boolean values in the continuous sequence in this stage to 0 and store the state sequence No update; c. If the eye feature is detected next time, enter the sub-state , if the eye features are detected multiple times in the subsequent detection, and the number of times the eye features are detected multiple times is greater than the tolerance , it indicates that it is not a false detection; At this stage: the parent state is updated to ; Through the counter Count the number of detections and add the obtained number to the counter ; Update all Boolean values in the continuous sequence in this stage to 1, and change the parent state The corresponding person Boolean value is updated to the storage state sequence ; Step 300: Blinking pattern determination; According to the final storage state sequence obtained in the state machine , determine the blinking pattern and calculate the number of blinks ; The formula for calculating the number of blinks is: Among them, Ⅱ represents the indicator function. When the condition is 1, Ⅱ returns 1, otherwise Ⅱ returns 0; N represents a natural number, represents the j-th parent state, represents the j-1th parent state, represents the j-2th parent state, j represents The index of the sequence; In (2) b, it specifically means: if the eye feature is not detected next time, then enter - is the sub-state, and the subsequent detection continues. If the eye features are not detected for multiple consecutive times in the subsequent detection, and the number of consecutive failures to detect the eye features is less than or equal to the set tolerance , it indicates a false detection; the period from the failure to detect eye features for multiple consecutive times to the first subsequent detection of eye features is considered time period; now Time period: Mother state Remain unchanged; counter The count starts from 0 and increases by 1 if the eye feature is not detected once. The count increases, and the increased count is 1 + counter Cumulative count; update all Boolean values in the continuous sequence within the time period to 1, and store the state sequence No update; In (2) c, it specifically means: if the eye feature is not detected next time, then enter - is the sub-state, and the subsequent detection continues. If the eye features are not detected for multiple consecutive times in the subsequent detection, and the number of consecutive failures to detect the eye features is greater than the tolerance , it indicates that it is not a false detection; the time period in which the eye features are not detected for many consecutive times is regarded as time period; now Time period: Parent status updated to ;counter Start counting from 0, and increase by 1 each time the eye feature is not detected, and add the accumulated count of this time period to the counter ; Update all Boolean values in the continuous sequence within this time period to 0, The corresponding Boolean value is updated to the storage state sequence ; In (3) b, it specifically means: b. If the eye feature is detected next time, enter - That is, the sub-state, subsequent continuous detection, if the eye features are detected multiple times in the subsequent detection, and the number of consecutive eye features detected is less than or equal to the set tolerance , it indicates a false detection; the period from the time when the eye features are detected multiple times in succession to the time when the eye features are not detected for the first time is considered as time period; now Time period: Mother state Remain unchanged; counter The counter starts counting from 0 and increases by 1 each time an eye feature is detected. The count increases, and the increased count is 1 + counter Cumulative count; update all Boolean values in the continuous sequence within the time period to 0, and store the state sequence No update; In (3) c, it means: if the eye feature is detected next time, then enter - is the sub-state, and the subsequent detection continues. If the eye features are detected multiple times in the subsequent detection, and the number of consecutive eye features detected is greater than the tolerance , it indicates that it is not a false detection; the time period in which the eye features are detected multiple times in succession is regarded as time period; now Time period: Parent status updated to ;counter Start counting from 0, and increase by 1 each time an eye feature is detected, and add the accumulated count of this time period to the counter ; Update all Boolean values in the continuous sequence within this time period to 1, The corresponding Boolean value is updated to the storage state sequence .
2. The method for determining blink frequency using a finite element state machine according to claim 1, characterized in that: In the blinking pattern determination, the blinking pattern determination needs to maintain a pointer to the beginning of the next window of the state machine, which is used to determine the state of the beginning of the window. It needs to start from the state of detecting eye features, and terminate the determination when encountering the end of the sequence or the end signal during the determination process.
3. A method for measuring blink frequency using a finite element state machine according to any one of claims 1 or 2, characterized in that: The eye feature is one of the pupil center, pupil area, iris or light spot generated by infrared light source.
