Blink identification and evaluation system and method

Abstract

The invention discloses a blink recognition evaluation system and method, and relates to the technical field of blink recognition, and the method comprises the steps: selecting upper and lower eyelid mark points, collecting a preprocessed video clip, located in front of a computer screen, of a subject, counting stable part trend features of interval periods between different time points of each blinking of the subject and every two adjacent different blinking time points, and extracting stable part trend features from change trends of upper and lower eyelid mark point movement data tracked under the condition of blinking time features in the preprocessed video; the method comprises the steps of preprocessing a video to select a first training image set and a second training image set from preprocessed segmented images obtained by segmenting the preprocessed video, selecting a test image set according to the first training image set, and counting the blinking frequency and the incomplete blinking proportion of a subject according to the first training image set, the second training image set and the test image set. The method can enhance real-time performance, automation and noninvasive performance, gets rid of dependence on special hardware, and can be applied to ophthalmic clinic.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of blink recognition, in particular to a blink recognition evaluation system and method. BACKGROUND

[0002] In the digital era, the widespread popularity of electronic devices such as smartphones, tablets and computers has greatly changed the way people communicate and access information. However, this change has also brought new challenges to eye health, especially the high incidence of dry eye disease (DED) worldwide. Dry eye is a multifactorial disease involving tear film homeostasis disruption and ocular surface structure dysfunction, usually manifested as ocular discomfort, visual fluctuation and tear film instability. In the context of electronic screen use, a key behavioral risk factor is the change in blink pattern, especially the occurrence of incomplete blinking.

[0003] Traditional blink detection mainly relies on manual frame-by-frame processing of video and manual analysis, which is tedious and time-consuming, and difficult to be widely used in actual clinical workflow. Although smartphone applications have higher accessibility, they have poor recognition effect in users with narrow eye slit or wearing masks, and inconsistent performance between different devices. More importantly, most existing systems can only quantify blink frequency, ignoring the clinical key dimension of "blink completeness". SUMMARY

[0004] In order to overcome the deficiencies of the prior art, the present application provides a blink recognition evaluation system and method.

[0005] In a first aspect, the present application provides a blink recognition evaluation method, which comprises:

[0006] Step S1, the height of the eyelid to which the subject of blink recognition belongs is counted, the eyelid region to be measured is selected according to the height of the eyelid, the upper eyelid marker point and the lower eyelid marker point are selected from the eyelid region to be measured, the pre-processing video segment of the subject in front of the computer screen is collected, the different time points of each blink of the subject and the interval period between each adjacent two different blink time points are counted according to the pre-processing video segment, and the frequency class trend feature to be measured is outputted;

[0007] Step S2, the variation trend of the tracked upper eyelid marker point movement data and the tracked lower eyelid marker point movement data in the pre-processing video under the blink time characteristic condition is extracted as the stable part trend feature, and the tracking class trend feature to be measured is outputted;

[0008] Step S3, according to the to-be-measured frequency class trend feature and the to-be-measured tracking class trend feature, selecting a sample training video segment from the preprocessed video, segmenting the sample training video segment into images frame by frame to obtain preprocessed segmented images, selecting a preset training amount of images from the preprocessed segmented images as a first training image set and a preset training amount of images in the case of complete blinking and incomplete blinking as a second training image set, selecting a remaining preset test amount of images as a test image set according to the first training image set, training a blinking test model according to the first training image set, training an incomplete blinking recognition model according to the second training image set, and according to the test image set, the blinking test model and the incomplete blinking recognition model, counting the blinking frequency and the incomplete blinking proportion of the subject, and outputting a blinking recognition evaluation result.

[0009] Preferably, the height of the eyelid to which the subject of the blinking recognition belongs is counted, and according to the height of the eyelid, a square region composed of the upper eyelid and the lower eyelid is selected to enclose the eye of the subject, and the to-be-measured eyelid region is outputted.

[0010] Three position points are selected from the upper eyelid of the to-be-measured eyelid region for marking, and upper eyelid marking points are outputted, the three position points of the upper eyelid marking points are equidistant, and the length from the center of the to-be-measured eyelid region is one quarter of the height of the eyelid, and the connecting line of the three position points of the upper eyelid marking points is in parallel relationship with the upper eyelid edge line of the to-be-measured eyelid region.

[0011] Three position points are selected from the lower eyelid of the to-be-measured eyelid region for marking, and lower eyelid marking points are outputted, the three position points of the lower eyelid marking points are equidistant, and the length from the center of the to-be-measured eyelid region is one quarter of the height of the eyelid, and the connecting line of the three position points of the lower eyelid marking points is in parallel relationship with the lower eyelid edge line of the to-be-measured eyelid region.

[0012] Preferably, the subject is arranged in front of a computer screen, it is ensured that the whole face of the subject is within the capture range of the camera, and a viewing video of neutral emotional content is provided to the subject, so as to collect the blinking condition of the subject in the viewing process, the movement data of the upper eyelid marking points and the movement data of the lower eyelid marking points, and output a subject video segment.

[0013] A preset initial viewing period is set, a video segment in the initial viewing period in the subject video segment is cut to obtain a preprocessed video segment, and the different time points of each blink of the blinking condition in the preprocessed video are counted to obtain a blinking time feature.

[0014] Preferably, the number of blinks between each adjacent two different blinking time points of the blinking condition in the preprocessed video is counted to obtain a blinking frequency feature.

[0015] According to the blink time feature and the blink frequency feature, a blink feature trend chart of the subject is drawn, and a blink time trend feature is output.

[0016] A stable partial trend feature is extracted from the blink time trend feature, and a to-be-tested frequency trend feature is output.

[0017] Preferably, the movement data of the upper eyelid marker points tracked in the pre-processed video under the blink time feature condition is counted to obtain a blink upper eyelid feature.

[0018] According to the blink upper eyelid feature and the blink time feature, a blink feature trend chart of the subject is drawn, and a first blink tracking trend feature is output.

[0019] The movement data of the lower eyelid marker points tracked in the pre-processed video under the blink time feature condition is counted to obtain a blink lower eyelid feature.

[0020] According to the blink lower eyelid feature and the blink time feature, a blink feature trend chart of the subject is drawn, and a second blink tracking trend feature is output.

[0021] A partial trend feature in which the marker point movement data is stable is extracted from the first blink tracking trend feature and the second blink tracking trend feature, and a to-be-tested tracking trend feature is output.

[0022] Preferably, a sampling training viewing period is preset, and a sampling training video segment is selected from the pre-processed video according to the same blink time point existing in the to-be-tested tracking trend feature and the to-be-tested frequency trend feature and the sampling training viewing period.

[0023] The sampling training video segment is segmented into images frame by frame to obtain pre-processed segmented images, a preset training amount of images is selected from the pre-processed segmented images as a first training image set, and the remaining pre-set test amount of images of the first training image set is removed from the pre-processed segmented images as a test image set.

[0024] According to the first training image set, a blink test model is trained, the test image set is input into the blink test model for testing whether the subject's eyes are in a fully open eye state, and a fully open eye probability value is output.

[0025] When the fully open eye probability value reaches a minimum value within a blink cycle, a corresponding key frame is obtained.

[0026] According to the key frame, the blink frequency of the subject is counted.

[0027] Preferably, a preset training amount of images in a fully blink and an incomplete blink condition is selected from the pre-processed segmented images as a second training image set.

[0028] According to the second training image set, an incomplete blinking recognition model is trained, the test image set and the key frame are input into the incomplete blinking recognition model for testing that the eyes of the subject are in an incomplete blinking state, and an incomplete blinking classification result is output;

[0029] According to the incomplete blinking classification result, the incomplete blinking proportion of the subject is counted, and the blinking frequency and the incomplete blinking proportion are combined to form a blinking recognition evaluation result.

[0030] In a second aspect, a state evaluation system of a photovoltaic power generation assembly comprises:

[0031] The blinking feature data statistical unit one is used for counting the eyelid height of the subject of the blinking recognition, selecting a to-be-measured eyelid region according to the eyelid height, selecting an upper eyelid marker point and a lower eyelid marker point from the to-be-measured eyelid region, collecting a preprocessing video segment of the subject in front of a computer screen, counting the stable part trend features between different time points of each blinking of the subject and the interval periods between each adjacent two different blinking time points according to the preprocessing video segment, and outputting the to-be-measured frequency trend features.

[0032] The blinking feature data statistical unit two is used for extracting the stable part trend features from the movement trend of the tracked upper eyelid marker point and the tracked lower eyelid marker point in the preprocessing video under the blinking time feature condition, and outputting the to-be-measured tracking trend features.

[0033] The model recognition evaluation unit is used for selecting a sample training video segment from the preprocessing video according to the to-be-measured frequency trend features and the to-be-measured tracking trend features, dividing the sample training video segment into images frame by frame to obtain preprocessing segmented images, selecting a preset training amount of images from the preprocessing segmented images as a first training image set and a preset training amount of images in the complete blinking and incomplete blinking conditions as a second training image set, selecting a remaining preset test amount of images as a test image set according to the first training image set, training a blinking test model according to the first training image set, training an incomplete blinking recognition model according to the second training image set, counting the blinking frequency and the incomplete blinking proportion of the subject according to the test image set, the blinking test model and the incomplete blinking recognition model, and outputting a blinking recognition evaluation result.

[0034] Compared with the prior art, the present application has the following characteristics and beneficial effects:

[0035] The position point is marked by selecting the peripheral eyelid region according to the eyelid height of the subject, so as to facilitate subsequent determination of whether the blinking state of the subject during watching of the neutral video content is in a natural condition, so as to remove various conditions of too frequent blinking or too large or too small blinking force of the subject in a state of emotional tension during the test, in order to determine the natural blinking condition of the subject during watching of the neutral video. Two types of feature data are counted, one is whether the interval period between each adjacent different time point blinking is stable, and the other is whether the movement data of the selected upper eyelid mark point and lower eyelid mark point is stable. When the feature data in the two cases is stable at the same blinking time point, it is determined that the blinking state of the subject at this time is natural, which is helpful to improve the accuracy of subsequent complete blinking and incomplete blinking recognition results. Two models are trained, one is a blinking test model, and the other is an incomplete blinking recognition model. The two models are used for identification to enhance the orderliness and rigor of the whole blinking recognition evaluation process. The final blinking recognition evaluation result is obtained by combining the two models, which further plays an information verification effect. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a step block diagram of a blinking recognition evaluation method mainly embodied by the embodiment.

[0037] Figure 2 is a structure block diagram of a blinking recognition evaluation system mainly embodied by the embodiment. DETAILED DESCRIPTION

[0038] The present application will be further described in detail below in combination with the following embodiments.

[0039] REFERENCE Figure 1 A blinking recognition evaluation method, the method comprising the following steps:

[0040] Step S1, the eyelid height of the subject of the blinking recognition is counted, the eyelid region to be tested is selected according to the eyelid height, the upper eyelid mark point and the lower eyelid mark point are selected from the eyelid region to be tested, the pre-processing video segment of the subject in front of the computer screen is collected, the stable part trend feature of the interval period between different time points of each blinking and each adjacent two different blinking time points of the subject is counted according to the pre-processing video segment, and the frequency to be tested is output.

[0041] Step S2, the variation trend of the upper eyelid mark point movement data and the lower eyelid mark point movement data tracked in the blinking time feature in the pre-processing video is extracted as a stable part trend feature, and a tracking to be tested is output.

[0042] Step S3, according to the to-be-measured frequency class trend characteristics and the to-be-measured tracking class trend characteristics, selecting a sample training video segment from the preprocessed video, segmenting the sample training video segment into images frame by frame to obtain preprocessed segmented images, selecting a preset training amount of images from the preprocessed segmented images as a first training image set and a preset training amount of images in the case of complete blinking and incomplete blinking as a second training image set, selecting a remaining preset test amount of images as a test image set according to the first training image set, training a blinking test model according to the first training image set, training an incomplete blinking recognition model according to the second training image set, and according to the test image set, the blinking test model and the incomplete blinking recognition model, counting the blinking frequency and the incomplete blinking proportion of the subject, and outputting a blinking recognition evaluation result.

[0043] Specifically, the peripheral eyelid region is selected according to the eyelid height of the subject to mark the position points, so as to facilitate subsequent determination of whether the blinking state of the subject during watching of the neutral video content is in a natural condition, so as to remove various conditions caused by the subject in a state of emotional tension during the test, such as too frequent blinking or too large or too small blinking force. In order to determine how the blinking feature of the subject appears during watching of the neutral video, two types of feature data are counted, one is whether the interval period between each adjacent different time point blinking is stable, and the other is whether the movement data of the selected upper eyelid mark point and lower eyelid mark point is stable. When the feature data in the two cases is stable at the same blinking time point, it can be determined that the blinking state of the subject at this time is in a natural state, which helps to improve the accuracy of subsequent complete blinking and incomplete blinking recognition results. By training two models, one is a blinking test model and the other is an incomplete blinking recognition model, the two models are used for identification to enhance the orderliness and rigor of the entire blinking recognition evaluation process, and the final blinking recognition evaluation result is obtained by combining the two models, which further plays an information verification effect.

[0044] The specific step S1 includes the following sub-steps:

[0045] The subject of the blinking recognition is counted according to the eyelid height, and a square region composed of the upper eyelid and the lower eyelid is selected to enclose the eyes of the subject, and the to-be-measured eyelid region is output.

[0046] Three position points are selected from the upper eyelid of the to-be-tested eyelid region for marking, and upper eyelid marking points are output, the three position points of the upper eyelid marking points are equidistant, and the length from the center of the to-be-tested eyelid region is one fourth of the eyelid height, and the connecting line of the three position points of the upper eyelid marking points is parallel to the upper eyelid edge line of the to-be-tested eyelid region.

[0047] Three position points are selected from the lower eyelid of the to-be-tested eyelid region for marking, and lower eyelid marking points are output, the three position points of the lower eyelid marking points are equidistant, and the length from the center of the to-be-tested eyelid region is one fourth of the eyelid height, and the connecting line of the three position points of the lower eyelid marking points is parallel to the lower eyelid edge line of the to-be-tested eyelid region.

[0048] The subject is arranged in front of a computer screen, ensuring that the entire face of the subject is within the capture range of the camera, and a video of neutral emotional content is provided to the subject for viewing, so as to collect the blinking condition of the subject during viewing, track the movement data of the upper eyelid marking points and the movement data of the lower eyelid marking points, and output a subject video segment.

[0049] An initial viewing period is preset, a video segment in the initial viewing period in the subject video segment is cut to obtain a pretreated video segment, and the different time points of each blink of the blinking condition in the pretreated video are counted to obtain a blinking time feature.

[0050] The number of blinks between each adjacent two different blinking time points in the pretreated video is counted to obtain a blinking frequency feature.

[0051] According to the blinking time feature and the blinking frequency feature, a blinking feature trend graph of the subject is drawn, and a blinking time trend feature is output.

[0052] A stable part trend feature is extracted from the blinking time trend feature, and a to-be-tested frequency trend feature is output.

[0053] Specifically, as the to-be-tested eyelid region (the face of the subject is positioned and the eyelid region is identified through (Open CV 68-point facial landmark detection), the height of the identified eyelid of the subject is determined. The height of the square region is selected as twice the height of the eyelid, and then the length of the square region containing the eyes of the subject is self-set. According to the height of the square region and the self-set length of the square region, the square region for blinking recognition is intercepted, that is, the to-be-tested eyelid region is obtained, and the subject is not less than one), the upper eyelid mark point (that is, three mark points are selected at the middle position in the upper eyelid region for subsequent displacement of the subject due to muscle pulling in the blinking state, that is, the three position points of the lower eyelid mark point are equidistant, and the length from the center of the to-be-tested eyelid region is half of the height of the eyelid. According to the displacement data value, it is determined whether the subject is in a natural state or an unnatural (such as nervous) state. The present application requires the blinking data of the subject in a natural state. Since the upper eyelid mark point is not vertically moved during normal blinking, but moves in an arc inward direction, the connecting line of the three position points of the lower eyelid mark point is parallel to the lower eyelid edge line of the to-be-tested eyelid region), the lower eyelid mark point (by analogy), and the movement of the upper eyelid mark point and the lower eyelid mark point is obtained through video recording by the built-in camera, the subject video segment (such as randomly using a notebook computer with a built-in camera (1920x1080 resolution) and a front camera (1080p) of a smart phone for blinking video recording, both of which have a frame rate greater than or equal to 30 frames / second, and are recorded under standardized lighting conditions (300-500 lux). The subject is 50 centimeters away from the computer screen to ensure that the entire face is within the capture range of the camera. In order to minimize the emotional variables that may affect the blinking pattern, the subject is instructed to watch a video with "neutral" emotional content (refers to works that do not have obvious positive or negative emotional tendencies, and objectively and simply convey information. For example, the lens of the natural documentary that shoots the migration of animals, only objectively records the behavior pattern without adding emotional hints of commentary; or the time-lapse photography works of urban scenery, simply showing the beauty of space-time change.Centralize the subject's line of sight to achieve the purpose of relaxing mood), preset initial viewing period (to avoid bias caused by the subject's awareness of being recorded, the first seven minutes of video are excluded from the analysis, which can be set by the user), blink time feature (refers to the time point of each blink: t1, t2, t3, tn), blink frequency feature (i.e. c1, c2, c3, cn), blink time class trend feature (such as taking the blink frequency feature as the y-axis and the blink time feature as the x-axis, drawing the variation curve between the two, i.e. the blink time class trend feature), and the measured frequency class trend feature (such as extracting the stable part of the blink frequency class trend feature, such as the blink frequency between each adjacent two time points from ti to tr is the same (here, the blink frequency between each adjacent two time points refers to the interval period of one blink), when the blink frequency in the same period is stable, it means that the subject is most likely to blink in a natural state, in order to ensure the accuracy of the data, further data verification analysis is required).

[0054] The specific step S2 includes the following sub-steps:

[0055] Statistical analysis of the upper eyelid marker point movement data tracked in the pre-processed video under the blink time feature condition is performed to obtain the blink upper eyelid feature.

[0056] According to the blink upper eyelid feature and the blink time feature, the blink feature trend graph of the subject is drawn, and the first blink tracking class trend feature is output.

[0057] Statistical analysis of the lower eyelid marker point movement data tracked in the pre-processed video under the blink time feature condition is performed to obtain the blink lower eyelid feature.

[0058] According to the blink lower eyelid feature and the blink time feature, the blink feature trend graph of the subject is drawn, and the second blink tracking class trend feature is output.

[0059] From the first blink tracking class trend feature and the second blink tracking class trend feature, the part of the trend feature in which the marker point movement data is stable is extracted, and the measured tracking class trend feature is output.

[0060] Specifically, the blinking upper eyelid feature (referring to the moving coordinate position data of the three marker points of the upper eyelid, for example, by capturing a video stream or image sequence using a camera, detecting facial key points through an algorithm. For example, the 68-point facial landmark model of the dlib library can accurately locate the feature points around the eyes; the FaceMeshModule of Mediapipe (a solution focusing on real-time 3D facial key point detection and analysis) can also provide similar functions. These tools can track the dynamic changes of the upper eyelid in real time and record the coordinate data of the selected marker points. The reference origin for statistical movement of the three position point coordinate position data of the upper eyelid marker points can be selected from one of the three position points of the upper eyelid marker points, or the reference origin can be set by the user), the first blinking tracking trend feature (such as taking the blinking time feature as the x-axis and the blinking upper eyelid feature (three marker points moving coordinate position data) as the y-axis, and drawing the variation curve between the two, that is, the first blinking tracking trend feature (the maximum moving coordinate position data between the three marker points at each time point of blinking)), the blinking lower eyelid feature, the second blinking tracking trend feature (by analogy), the to-be-measured tracking trend feature (the stable part of the first blinking tracking trend feature and the second blinking tracking trend feature is the normal moving coordinate data of the upper eyelid marker points and the lower eyelid marker points produced by the subject's blinking in a natural state (such as the upper eyelid marker point moving coordinate data at each time point from ti to tr being the same, and the lower eyelid marker point moving coordinate data at each time point from ti to tr being the same), and the to-be-measured tracking trend feature in this case corresponds to the most likely blinking data feature of the subject in a natural state.

[0061] The specific step S3 includes the following sub-steps:

[0062] A preset sampling training viewing period is selected from the preprocessed video according to the same blinking time point of the to-be-measured tracking trend feature and the to-be-measured frequency trend feature and the sampling training viewing period.

[0063] The sampling training video segment is segmented into images frame by frame to obtain preprocessed segmented images, and a preset training amount of images is selected from the preprocessed segmented images as a first training image set, and the remaining preset test amount of images of the first training image set is removed from the preprocessed segmented images as a test image set.

[0064] According to the first training image set, a blinking test model is trained, the test image set is input into the blinking test model to test whether the subject's eyes are in a fully open eye state, and a fully open eye probability value is output.

[0065] When the fully open eye probability value reaches the minimum value in one blink cycle, the corresponding key frame is obtained.

[0066] According to the key frame, the blink frequency of the subject is counted.

[0067] From the pre-processed segmented images, a preset training amount of images in the fully blinking and incomplete blinking conditions are selected as a second training image set.

[0068] According to the second training image set, an incomplete blinking recognition model is trained, the test image set and the key frame are input into the incomplete blinking recognition model for testing whether the subject's eyes are in the incomplete blinking state, and an incomplete blinking classification result is output.

[0069] According to the incomplete blinking classification result, the incomplete blinking proportion of the subject is counted, and the blink frequency and the incomplete blinking proportion are combined to form a blink recognition evaluation result.

[0070] Specifically, as preset sampling training viewing period (such as five minutes of video segments are selected from the remaining video for training purposes, which can be set by yourself), sample training video segments (if the to-be-tested tracking trend feature and the to-be-tested frequency trend feature have the same blink time point (such as ti to tr), five minutes of ti to tr are extracted, and the video segments of the five minutes of ti to tr in the preprocessed video are extracted accordingly, that is, the sample training video segments (which can be used for subsequent blink recognition processing) are extracted, if the to-be-tested tracking trend feature and the to-be-tested frequency trend feature do not have the same blink time point, the iris direction position of the subject to which the to-be-tested frequency trend feature belongs at each blink is detected (classify three situations: right view direction, forward view direction, and left view direction), and the iris direction feature Th in the two situations is extracted from the iris direction of the subject to which the to-be-tested frequency trend feature belongs at each blink (record the time points ta and tb corresponding to the two situations), according to ta, the moving coordinate position data of the upper eyelid marker point in the ta, Th (one of the two situations, such as right view direction) corresponding condition in the first blink tracking trend feature (if it is stable and changes): for example, the moving coordinate position data at ta1, ta2, and ta3 (which can be adjacent to each other or not adjacent) are the same, and similarly, the moving coordinate position data of the lower eyelid marker point in the tb, Th (the other of the two situations, such as forward view direction or left view direction) corresponding condition in the second blink tracking trend feature (if it is stable and changes): for example, the moving coordinate position data at tb1, tb2, and tb3 (which can be adjacent to each other or not adjacent) are the same, then it can be determined that the video segment corresponding to the to-be-tested frequency trend feature is the blink of the subject in the natural state (that is, the video segment corresponding to the to-be-tested frequency trend feature is used as the sample training video segment), if there is a deviation in the sample training video segment, in order to improve the data clarity and accuracy of subsequent analysis, information preprocessing is required: for example, 27 blink videos (48 eyes) are collected. Among the 12 subjects (18 eyes) included in the verification group, two subjects were excluded because their body inclination exceeded the camera field of view and could not be evaluated; in addition, four subjects were excluded because their hands were holding their faces, causing some eyes to be blocked and unable to be positioned. Finally, 24 blink videos (48 eyes) were included for analysis, with each eye serving as an independent analysis unit), the preprocessed segmented images (the video is segmented into images frame by frame (then the preset training amount (such as 80%, which can be set by yourself) is selected as the first training image set, and the preset test amount (such as 20%, which is adjusted accordingly from the preset training amount) is selected). For example, if 3153 images are extracted from 15 videos (30 eyes) initially, 200 images are excluded due to motion blur or poor contrast, and finally 2953 high-quality images are obtained.Finally, all the segmented images are converted to the format: (256x128x3). Subsequent analysis of the images requires extracting image frames for recognition, such as extracting video frames frame by frame through computer vision technology, VLC media player: key frames are captured through command line or API, and so on), blink test model (such as using a convolutional neural network (CNN) for a binary classification task, aiming to accurately distinguish between blinking and open eye states. This model is carefully trained on RGB images with a size of 256x128x3 pixels, covering images of complete blinking and complete open eyes. The output of the blink recognition model is a probability value between 0 and 1, representing the likelihood of the eye being completely open. A threshold of 0.5 is used for classification; frames with a probability value greater than or equal to 0.5 are classified as "open eyes", and those below 0.5 are designated as "blinking". This method not only ensures accurate detection and classification of eye states, but also uses advanced CNN technology to improve the practicality of the model in ophthalmic research and applications. The training process uses an image data generator to load and input images. For example, each batch consists of 16 randomly selected frames (i.e. from the first training image set), which are input into the CNN for training. The model uses the Adam optimizer (compared to traditional stochastic gradient descent, Adam can automatically balance the update amplitude of different parameters, especially suitable for handling sparse gradients or noisy scenarios. Its efficiency and robustness make it a popular choice in deep learning, such as image classification, natural language processing, etc.) and binary cross-entropy (with good gradient characteristics, suitable for gradient descent optimization; based on maximum likelihood estimation theory, with statistical rationality; compared with mean square error, it can better adapt to unbalanced data sets) as the loss function to optimize and improve training efficiency. Among them, complete blinking: upper and lower eyelids completely contact, complete open eyes: maximum eye slit, incomplete blinking: partial closure of eyelids and no contact), complete open eye probability value (test the test image set in the trained blink recognition model, output a probability value between 0 and 1, representing the likelihood of the eye being completely open (i.e. the likelihood of the current frame being completely open), i.e. the complete open eye probability value, frames with a probability value greater than or equal to 0.5 are classified as "open eyes", and those below 0.5 are designated as "blinking"), key frame (when the probability reaches a minimum value within a period (maximum closed frame), a blink event is recorded, and according to the minimum value, the frame corresponding to the minimum value is counted as the key frame), according to the key frame, automatically capture and count the blink frequency of the subject in the test image set (such as the number of blinks per minute), the second training image set (i.e. the classified image set can be directly distinguished, which may be completely overlapping with the first training image set, or partially overlapping, without affecting), incomplete blinking recognition model (trained on ImageNet (the model is initially trained on a large-scale data set containing 1000 natural images, with strong general feature extraction capability.The convolutional residual neural network (ResNet) pre-trained on image classification tasks or as the backbone network for other computer vision tasks (e.g., object detection, semantic segmentation) can be used for binary classification tasks to distinguish between complete and incomplete blinks. The model is trained using two classes of images: complete blinks (eyelids fully touching) and incomplete blinks (eyelids partially closed without touching). During classification, the DLM-2 generates a probability value between 0 and 1, with a threshold of 0.5; values greater than or equal to 0.5 indicate complete blinks, and values less than 0.5 indicate incomplete blinks. This approach effectively leverages the advanced capabilities of the ResNet architecture, enabling the detection of subtle differences in blink completeness and accurate classification based on the provided training data. For example, the training uses a batch size of 8 images (extracted from the second training image set), optimized using the Adam optimizer and binary cross-entropy as the loss function. The model architecture follows the ResNet, and the incomplete blink classification results are obtained by inputting the key frames and test image set into the incomplete blink recognition model (according to the key frames, the blink completeness of the corresponding blink event in the test image set to which the key frame belongs is identified, with values greater than or equal to 0.5 indicating complete blinks and values less than 0.5 indicating incomplete blinks) to identify and evaluate the completeness of blinks. The blink recognition evaluation results are obtained by identifying the blink completeness of several blink events in the test image set using the incomplete blink recognition model, classifying the incomplete blink results, and calculating the percentage of incomplete blinks (i.e., the proportion in the test image set). Based on the blink frequency and incomplete blink percentage calculated above, the blink recognition evaluation results are obtained by summarizing the blink frequency and incomplete blink percentage. Finally, based on the detected blink completeness and blink frequency, early identification of dry eye risk populations and development of personalized treatment strategies are achieved.

[0071] The blink recognition evaluation system comprises a blink feature data statistical unit one, a blink feature data statistical unit two and a model recognition evaluation unit, the blink feature data statistical unit one is used to statistically obtain the eyelid height of the subject, the eyelid height is used to select the eyelid region to be measured, the upper eyelid mark point and the lower eyelid mark point are selected from the eyelid region to be measured, the pre-processing video segment of the subject in front of the computer screen is collected, the stable part trend characteristics between the different time points of each blink and the interval period between each adjacent two different blink time points are statistically obtained according to the pre-processing video segment, and the frequency trend characteristics to be measured are output; the blink feature data statistical unit two is used to extract the stable part trend characteristics of the change trend of the upper eyelid mark point movement data and the lower eyelid mark point movement data tracked in the blink time feature in the pre-processing video, and the tracking trend characteristics to be measured are output; the model recognition evaluation unit is used to select the sampling training video segment from the pre-processing video, the sampling training video segment is segmented into images frame by frame to obtain the pre-processing segmented image, the first training image set of the preset training amount of images and the second training image set of the preset training amount of images in the complete blink and incomplete blink are selected from the pre-processing segmented image according to the frequency trend characteristics to be measured and the tracking trend characteristics to be measured, the remaining preset test amount of images are selected as the test image set according to the first training image set, the blink test model is trained according to the first training image set, the incomplete blink recognition model is trained according to the second training image set, the blink frequency and the incomplete blink proportion of the subject are statistically obtained according to the test image set, the blink test model and the incomplete blink recognition model, and the blink recognition evaluation result is output.

[0072] The above are preferred embodiments of the present application, and do not limit the protection scope of the present application, therefore: any equivalent changes made on the structure, shape, principle of the present application should be covered within the protection scope of the present application.

Claims

1. A blink recognition evaluation method, characterized by, The method comprises the following steps: Step S1, the height of the eyelid of the subject to be identified is counted, the eyelid region to be measured is selected according to the height of the eyelid, the upper eyelid marker point and the lower eyelid marker point are selected from the eyelid region to be measured, the pre-processing video segment of the subject in front of the computer screen is collected, the stable part trend characteristics of the different time points of each blink of the subject and the interval period between each adjacent two different blink time points are counted according to the pre-processing video segment, and the frequency trend characteristics to be measured are output; Step S2, the change trend of the upper eyelid marker point movement data and the lower eyelid marker point movement data tracked in the pre-processing video under the condition of the blink time characteristics is extracted as the stable part trend characteristics, and the tracking trend characteristics to be measured are output; Step S3, according to the frequency trend characteristics to be measured and the tracking trend characteristics to be measured, the sample training video segment is selected from the pre-processing video, the sample training video segment is segmented into images frame by frame to obtain pre-processing segmented images, the pre-processing segmented images are selected to obtain a first training image set of a preset training amount of images and a second training image set of a preset training amount of images under the conditions of complete blinking and incomplete blinking, the remaining preset test amount of images are selected as a test image set according to the first training image set, a blink test model is trained according to the first training image set, an incomplete blinking recognition model is trained according to the second training image set, the blinking frequency and the incomplete blinking proportion of the subject are counted according to the test image set, the blink test model and the incomplete blinking recognition model, and the blink recognition evaluation result is output.

2. The blink recognition evaluation method of claim 1, wherein, Step S1 comprises: The height of the eyelid of the subject to be identified is counted, and a square region composed of the upper eyelid and the lower eyelid which encloses the eyes of the subject is selected according to the height of the eyelid, and the eyelid region to be measured is output; Three position points are selected and marked from the upper eyelid of the eyelid region to be measured, and the upper eyelid marker point is output, the three position points of the upper eyelid marker point are equidistant, and the length from the center of the eyelid region to be measured is one quarter of the height of the eyelid, and the connecting line of the three position points of the upper eyelid marker point is parallel to the upper eyelid edge line of the eyelid region to be measured; Three position points are selected and marked from the lower eyelid of the eyelid region to be measured, and the lower eyelid marker point is output, the three position points of the lower eyelid marker point are equidistant, and the length from the center of the eyelid region to be measured is one quarter of the height of the eyelid, and the connecting line of the three position points of the lower eyelid marker point is parallel to the lower eyelid edge line of the eyelid region to be measured.

3. The blink recognition evaluation method of claim 2, wherein, Step S1 further comprises: The subject is arranged in front of the computer screen, it is ensured that the whole face of the subject is within the capture range of the camera, and a viewing video of neutral emotional content is provided to the subject, so as to collect the blinking condition of the subject during the viewing process, track the movement data of the upper eyelid marker point and the lower eyelid marker point, and output the subject video segment; The preset initial viewing period is set, the video segment in the subject video segment which is in the initial viewing period is cut, and a pretreated video segment is obtained, the different time points of each blink of the blink condition in the pretreated video are counted, and a blink time feature is obtained.

4. The blink recognition evaluation method of claim 3, wherein, Step S1 further includes: The number of blinks between each adjacent two different blink time points of the blink condition in the pretreated video is counted, and a blink frequency feature is obtained; According to the blink time feature and the blink frequency feature, a blink feature trend graph of the subject is drawn, and a blink time class trend feature is output; A stable partial trend feature is extracted from the blink time class trend feature, and a to-be-tested frequency class trend feature is output.

5. The blink recognition evaluation method of claim 4, wherein, Step S2 includes: The upper eyelid marker point movement data of the blink time feature in the pretreated video is counted, and a blink upper eyelid feature is obtained; According to the blink upper eyelid feature and the blink time feature, a blink feature trend graph of the subject is drawn, and a first blink tracking class trend feature is output; The lower eyelid marker point movement data of the blink time feature in the pretreated video is counted, and a blink lower eyelid feature is obtained; According to the blink lower eyelid feature and the blink time feature, a blink feature trend graph of the subject is drawn, and a second blink tracking class trend feature is output; The marker point movement data of the first blink tracking class trend feature and the second blink tracking class trend feature is extracted, and a partial trend feature in which the marker point movement data is stable is output, and a to-be-tested tracking class trend feature is output.

6. The blink recognition evaluation method of claim 5, wherein, Step S3 includes: A sampling training viewing period is preset, and a sampling training video segment is selected from the pretreated video according to the same blink time point existing in the to-be-tested tracking class trend feature and the to-be-tested frequency class trend feature and the sampling training viewing period; The sampling training video segment is segmented into images frame by frame to obtain pretreated segmented images, a preset training amount of images in the pretreated segmented images is selected as a first training image set, and the remaining preset test amount of images in the pretreated segmented images except the first training image set is removed as a test image set; According to the first training image set, a blink test model is trained, the test image set is input into the blink test model to test whether the eyes of the subject are in a fully open eye state, and a fully open eye probability value is output; When the fully open eye probability value reaches a minimum value within one blink cycle, a corresponding key frame is obtained; According to the key frame, the blink frequency of the subject is counted.

7. The blink recognition evaluation method of claim 6, wherein, Step S3 further includes: A preset training amount of images in the pretreated segmented images in the case of complete blink and incomplete blink is selected as a second training image set; According to the second training image set, an incomplete blink recognition model is trained, the test image set and the key frame are input into the incomplete blink recognition model to test whether the eyes of the subject are in an incomplete blink state, and an incomplete blink classification result is output; According to the incomplete blink classification result, the incomplete blink proportion of the subject is counted, and the blink frequency and the incomplete blink proportion are combined into a blink recognition evaluation result.

8. A blink recognition evaluation system, characterized by, The system is used to implement the blink recognition evaluation method in any one of claims 1-7, and includes: The blinking feature data statistical unit one is used for counting the eyelid height of the subject of the blinking recognition, selecting the to-be-tested eyelid region according to the eyelid height, selecting the upper eyelid mark point and the lower eyelid mark point from the to-be-tested eyelid region, collecting the pre-processing video segment of the subject in front of the computer screen, counting the stable part trend characteristics between different time points of each blinking of the subject and the interval period between each adjacent two different blinking time points according to the pre-processing video segment, and outputting the to-be-tested frequency trend characteristics. The blinking feature data statistical unit two is used for extracting the stable part trend characteristics from the change trend of the tracked upper eyelid mark point movement data and the tracked lower eyelid mark point movement data in the pre-processing video under the blinking time feature condition, and outputting the to-be-tested tracking trend characteristics. The model recognition evaluation unit is used for selecting the sampling training video segment from the pre-processing video according to the to-be-tested frequency trend characteristics and the to-be-tested tracking trend characteristics, segmenting the sampling training video segment into images frame by frame to obtain pre-processing segmented images, selecting the preset training amount of images from the pre-processing segmented images as a first training image set and the preset training amount of images under the complete blinking and incomplete blinking conditions as a second training image set, selecting the remaining preset test amount of images as a test image set according to the first training image set, training a blinking test model according to the first training image set, training an incomplete blinking recognition model according to the second training image set, counting the blinking frequency and the incomplete blinking proportion of the subject according to the test image set, the blinking test model and the incomplete blinking recognition model, and outputting the blinking recognition evaluation result.

Images