A method of identifying blink frames in eye videos

Abstract

The present application relates to the technical field of video image analysis, in particular to a method for identifying blink frames in eye videos, comprising the following steps: 1) using a deep learning model to locate the pupil center of the eye video data, obtaining the pupil trajectory data in the video, and converting the pupil trajectory coordinate curve into a speed curve; 2) using a DBSCAN clustering algorithm to identify the blink frames according to the time series data characteristics of the pupil; 3) performing data cleaning according to the clustering algorithm result of step 2); 4) performing smoothing operation on the screened blink frames; compared with the prior art, the present application does not directly use image data in the video for judgment, but uses a machine learning clustering method to identify a video as a whole, thereby solving the problem of whether the blink frames in the medical video can be quickly and automatically identified in the signal dimension rather than the image dimension.

Description

[Technical Field]

[0001] This invention relates to the field of video image analysis technology, specifically a method for identifying blink frames in eye videos. [Background Technology]

[0002] Currently, deep learning has been widely applied in the medical field, especially in image-related areas. In the medical field of eye diseases, due to factors such as lighting, head movements, and the patient's own condition, the frequency of blinking frames in medical imaging videos is extremely high. This results in a large amount of noise in the video samples, making them unsuitable for direct use as training data. Furthermore, existing blink recognition methods rely on numerical computation, requiring significant time investment and failing to meet the requirements for processing blinks in long videos.

[0003] Specifically, the existing blink detection methods are as follows:

[0004] Option 1 uses facial landmarks for eye localization to calculate the eye aspect ratio (EAR). The calculated value is then used to determine if a blink has occurred. This value is compared to a set threshold; if a switch between open and closed eyes is observed, a blink is considered to have occurred. The drawback of this option is that it requires high landmark accuracy. However, in various complex application scenarios with different lighting conditions, postures, and high-speed movement, landmark recognition accuracy is low. Therefore, the blink detection accuracy calculated using this method is not high. Furthermore, it requires calculation of key positions on the entire face, resulting in a large computational load and high latency, making it unsuitable for long video processing.

[0005] Option 2 involves processing the video image using image binarization, then selecting the optimal threshold to extract the pupil in each frame, and judging whether there are any abnormalities based on the changes in pupil shape between adjacent frames. This method can quickly produce results, but it is very sensitive to the choice of threshold and heavily relies on the accurate extraction of the pupil. However, the threshold corresponding to each frame of a video is often not fixed, making it difficult to automate. At the same time, this method has high requirements for the image itself; problems such as reflection, blurring, and black and white will greatly affect the accuracy.

[0006] Furthermore, both of the above methods are image-based identification, requiring the analysis of the entire image. This makes it difficult to treat blink detection as an independent task, limiting the performance of blink detection algorithms in certain scenarios, such as video preprocessing. Based on these issues, this invention chooses not to directly use image data from the video for judgment, but instead treats a video segment as a whole and uses machine learning clustering methods for identification. [Summary of the Invention]

[0007] The purpose of this invention is to overcome the above-mentioned shortcomings and provide a method for identifying blink frames in eye videos. Instead of directly using image data from the video for judgment, this method treats a video as a whole and uses machine learning clustering methods to achieve rapid and automatic identification of blink frames in medical videos.

[0008] To achieve the above objectives, a method for identifying blink frames in eye videos is designed, comprising the following steps: 1) using a deep learning model to locate the pupil center in the eye video data, obtaining pupil trajectory data within the video, and converting the pupil trajectory coordinate curve into a velocity curve; 2) using the DBSCAN clustering algorithm to identify blink frames based on the time-series data characteristics of the pupil; 3) performing data cleaning based on the clustering algorithm results in step 2); 4) smoothing the selected blink frames.

[0009] Preferably, in step 1), the formula for the pupil trajectory velocity is:

[0010]

[0011] In the formula, x and y are the horizontal and vertical coordinates of the pupil center, respectively, and r is the pupil radius.

[0012] As a preferred step, in step 2), the data feature tuples (dx, dy, dr) of each frame in the pupil time series form a set D, specifying the search radius EPS and the minimum sample size Min Pts of the category. The specific steps are as follows: (1) Mark the core points. The definition of a core point is that the density search radius EPS contains at least Min Pts points, and the rest are marked as non-core points; (2) Randomly select an unprocessed frame from set D; (3) If the selected frame is a core point, start from the point and search for all reachable objects within the density radius EPS, merge with the adjacent core points to form a cluster; if the selected frame is a non-core point, skip it; (4) Repeat steps (2) and (3) until all data is processed; (5) Finally, output all generated clusters and select the largest cluster as the cleaned sample.

[0013] As a preferred embodiment, in step 3), based on the horizontal and vertical velocities in the pupil data, as well as the rate of change of the radius, and combined with information from the three feature dimensions, the DBSCAN clustering algorithm can identify the location of the blink frame.

[0014] As a preferred option, the specific steps of step 4) are as follows: (1) Using the blinking frame as the center, find the nearest non-blinking frame; (2) Use the signal value of the non-blinking frame as the replacement value and mark the blinking frame as a normal frame; (3) Repeat the process until all blinking frames have been processed.

[0015] Compared with the prior art, the present invention has the following advantages:

[0016] (1) The present invention selects to extract the pupil trajectory of the video first, and then selects the blinking frames according to the clustering algorithm. Its advantage is that it can detect all frames in the video at the same time, and use the correlation and difference between video frames to distinguish the data that needs to be screened out, so as to achieve real-time results.

[0017] (2) The present invention converts the image into time series data of the pupil, and entrusts the influencing factors such as reflection and noise in the image to the deep learning model for pupil localization, thereby reducing their influence.

[0018] (3) This invention uses DBSCAN as an unsupervised clustering method, which eliminates the need to repeatedly modify the threshold for different samples;

[0019] (4) This invention chooses not to directly use the image data in the video for judgment, but to treat a video as a whole and use machine learning clustering methods for identification;

[0020] (5) This invention solves the problem of whether blinking frames in medical videos can be quickly and automatically identified in the signal dimension rather than the image dimension. [Image Description]

[0021] Figure 1 This is a diagram showing the pupil trajectory and velocity curve of an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram illustrating the clustering and differentiation of blink frames according to an embodiment of the present invention;

[0023] Figure 3 This is a visual illustration of blink frame marking in an embodiment of the present invention;

[0024] Figure 4 This is a comparison chart of the smoothed time series curve and the original curve in an embodiment of the present invention. [Detailed Implementation]

[0025] The present invention will be further described below with reference to specific embodiments:

[0026] This invention provides a method for identifying blink frames in eye videos, the technical solution of which is described in detail below:

[0027] Step 1: Use deep learning algorithms to locate the pupil center in the eye video data, obtain the pupil trajectory data in the video as shown in Table 1 below, and convert the pupil trajectory coordinate curve into a velocity curve, as shown in the appendix. Figure 1 As shown.

[0028]

[0029] Where x and y are the horizontal and vertical coordinates of the pupil center, and r is the pupil radius.

[0030]

[0031] Table 1: Pupil Trajectory Data

[0032] Step 2: Use the DBSCAN clustering algorithm to identify blink frames. The specific implementation steps of this invention are as follows:

[0033] Input: Data feature tuples (dx, dy, dr) for each frame in the time series, forming a set D, specifying the search radius EPS and the minimum sample size Min Pts for each class.

[0034] 1. Mark core points. Core points are defined as points that contain at least Min Pts points within the density search radius EPS; the rest are marked as non-core points.

[0035] 2. Randomly select an unprocessed frame from set D;

[0036] 3. If the extracted frame is a core point, then start from that point and search for all reachable objects within the density radius EPS, and merge them with adjacent core points to form a cluster; if the extracted frame is not a core point, then skip it.

[0037] 4. Repeat steps 2 and 3 until all data has been processed.

[0038] Output: All generated clusters, with the largest cluster selected as the cleaned sample, as shown in the attached image. Figure 2 As shown, DBSCAN clustering clusters and distinguishes blink frames.

[0039] Step 3: Clean the data based on the clustering results.

[0040] Blinking is accompanied by abnormal changes in the coordinates of the pupil's center and a rapid decrease in the pupil's radius; head movements are accompanied by sudden accelerations in the horizontal and vertical directions, all of which differ significantly from normal pupil movements. (Appendix) Figure 3 The three curves in the image correspond to the horizontal and vertical velocities and the rate of change of the pupil's radius, respectively. Combining information from these three feature dimensions, the DBSCAN algorithm can accurately determine the location of the blink frame. (See attached image.) Figure 3 The blink frame markers are visualized, with asterisks indicating frames that need to be removed.

[0041] Step 4: Smooth the selected blink frames. The specific steps are as follows:

[0042] (1) Using the blink frame as the center, find the nearest non-blink frame;

[0043] (2) Use the signal value of the non-blinking frame as the replacement value, and mark the blinking frame as a normal frame;

[0044] (3) Process in a loop until all blink frames have been processed;

[0045] Appendix Figure 4 Comparing the smoothed time series curve with the original curve, it can be seen that the cleaned time series can eliminate the influence of blink signals while retaining the effect of normal nystagmus signals, which greatly improves the performance of deep learning models.

[0046] In summary, this invention transforms blink frame detection in videos from an image-based problem into a time-series problem, enabling batch data cleaning of videos. This solution uses a pupil center localization model combined with DBSCAN clustering to filter blink frames. While directly using deep learning models to predict blink frames can be an alternative, it requires extensive manual annotation of these frames.

[0047] The purpose of this invention is to filter out blinking frames in medical videos. This preprocessing operation requires an efficient implementation scheme to provide suitable deep learning modeling samples. In existing technologies, the scheme of calculating the aspect ratio of the eyes is applicable to single images, but it is too inefficient and time-consuming when processing video frame by frame. The binarization method requires manually setting thresholds for filtering and has high requirements for image quality.

[0048] This invention combines the characteristics of eye images in medical videos, employing a deep learning model to generate pupil trajectory data, and then using the DBSCAN algorithm (a machine learning algorithm) to detect blinks. DBSCAN is a density-based unsupervised clustering algorithm. Such density clustering algorithms assume that categories can be determined by the density of sample distribution; that is, samples of the same category are more closely connected. This algorithm can distinguish blinks from normal videos based on the time-series characteristics of pupil data (such as sudden changes in radius, disappearance of center coordinates, etc.), achieving the function of batch data cleaning.

[0049] This invention is not limited to the above-described embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of this invention shall be considered equivalent substitutions and shall be included within the scope of protection of this invention.

Claims

1. A method for identifying blink frames in an eye video, characterized in that, Includes the following steps: 1) A deep learning model is used to locate the pupil center in eye video data, obtain pupil trajectory data in the video, and convert the pupil trajectory coordinate curve into a velocity curve; 2) The DBSCAN clustering algorithm is used to identify blink frames based on the time-series data characteristics of pupils; 3) Perform data cleaning based on the clustering algorithm results from step 2); 4) Smooth the selected blink frames; In step 1), the formula for the pupil trajectory velocity is: In the formula, x and y are the x and y coordinates of the pupil center, and r is the pupil radius; In step 2), the data feature tuples (dx, dy, dr) of each frame in the pupil time series form a set D, specifying the search radius EPS and the minimum sample size Min Pts for each class. The specific steps are as follows: (1) Mark core points. The definition of a core point is that the point density search radius EPS contains at least Min Pts points, and the rest are marked as non-core points. (2) Randomly select an unprocessed frame from set D; (3) If the extracted frame is a core point, then start from that point and search for all reachable objects within the density radius EPS, and merge them with adjacent core points to form a cluster; if the extracted frame is not a core point, then skip it. (4) Repeat steps (2) and (3) until all data has been processed; (5) Finally, output all generated clusters and select the largest cluster as the cleaned sample.

2. The method for identifying blink frames in an eye video as described in claim 1, characterized in that: In step 3), based on the horizontal and vertical velocities in the pupil data, as well as the rate of change of the radius, and combined with information from the three feature dimensions, the DBSCAN clustering algorithm can identify the location of the blink frame.

3. The method for identifying blink frames in an eye video as described in claim 2, characterized in that, The specific steps for step 4) are as follows: (1) Using the blink frame as the center, find the nearest non-blink frame; (2) Use the signal value of the non-blinking frame as the replacement value, and mark the blinking frame as a normal frame; (3) Process in a loop until all blink frames have been processed.

Images