Method and device for detecting incomplete blink rate
Through the segmentation model of the convolution module that can be separated in depth by high frame rate acquisition and attention mechanism, the unclosed eyelid area in the blink image is extracted, and the robustness and resource utilization problems of incomplete blink rate detection in the prior art are solved, and the problem of resource utilization is realized. Fast and accurate detection results.
Patent Information
- Application Number
- CN202410931470.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-08
AI Technical Summary
When detecting incomplete blinking rates, the prior art is greatly affected by ambient light, eyelash occlusion and complex eye environment. The calculation results are low, and the robustness of motion blur caused by rapid closure of eyelids in the blinking moment is difficult to ensure. And normal eye-opening frames occupy the vast majority of computing resources and affect efficiency.
The high-frame rate acquisition module is used to capture blink images 40 frames per second, and the closed eye frame with the greatest degree of eyelid closure is automatically extracted from the frame sequence. The segmentation model of the convolution module can be separated into attention mechanism and depth can be used to extract the unclosed eyelid area, and calculate the calculation Incomplete blink rate.
It realizes rapid and accurate detection of incomplete blinking rates in subjects, improves the robustness of calculation results and resource utilization efficiency, and reduces the risk of misdiagnosis.
Smart Images

Figure CN118918133A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for detecting an incomplete blink rate. Background Art
[0002] In the absence of external stimulation and inflammation and pain, blinking is generally an involuntary physiological activity of the eyes opening and closing quickly. As a protective action, blinking can evenly spread tears on the cornea and conjunctiva, thereby keeping the eyeball moist. Correspondingly, incomplete blinking will cause the tear film to not be renewed in time after evaporation, resulting in dryness and stinging of the eyeball, which may then induce the onset of dry eye syndrome.
[0003] Detecting the incomplete blink rate of patients can effectively assist doctors in diagnosing dry eyes. Currently, high-speed cameras are generally used to capture the blinking of patients' eyes, and then doctors manually judge the degree of eyelid closure of each frame based on experience. This method is not only time-consuming and labor-intensive, but also in the face of a large number of repeated and feature-similar images, it is very easy to be confused and missed in naked eye detection.
[0004] Nowadays, there are two ways to automatically analyze incomplete blink rates: based on traditional image processing methods and based on deep learning methods. However, for a continuous frame sequence containing blinks, the traditional image processing scheme to calculate the degree of incomplete blinks is greatly affected by ambient lighting, eyelash occlusion and complex eye environment, and the calculation results are less robust. The scheme that uses deep learning methods to segment the unclosed eyelid area of each frame has relatively strong resistance to environmental interference, but it is still difficult to guarantee robustness against motion blur caused by rapid eyelid closure at the moment of blinking. In addition, blink frames generally only occupy a small part of the entire frame sequence, and the model's reasoning on normal open eye frames occupies the vast majority of processing time and running memory, affecting the effective use of computing resources. Summary of the invention
[0005] In view of this, an object of the present invention is to provide a method and device for detecting an incomplete blink rate, so as to quickly and accurately obtain the result of the unclosed eyelid area and determine the incomplete blink rate of the subject.
[0006] In a first aspect, an embodiment of the present invention provides a method for detecting an incomplete blink rate, the method comprising: acquiring multiple blink images of a subject based on an acquisition module; wherein the multiple blink images are a continuous frame sequence; determining multiple eye-closed frames from the frame sequence; wherein the eye-closed frames represent image frames in which the degree of eyelid closure is the greatest during each blink of the subject; inputting the multiple eye-closed frames into a pre-trained segmentation model, and outputting the eyelid unclosed areas of the multiple eye-closed frames; and determining the incomplete blink rate of the subject based on the eyelid unclosed areas of the multiple eye-closed frames.
[0007] In an optional embodiment of the present application, the above-mentioned step of acquiring multiple blinking images of the subject based on the acquisition module includes: the acquisition module is aimed at the eyes of the subject; the acquisition module captures the subject's normal open-eyes image, the upper eyelid closed downward image, the eyelid closed to the maximum degree image and the upper eyelid opened upward image as the subject's multiple blinking images at a rate of 40 frames per second.
[0008] In an optional embodiment of the present application, the above-mentioned step of determining multiple eye-closing frames from a frame sequence includes: determining a previous frame of a target frame, a target frame, and a subsequent frame of the target frame from the frame sequence; converting the previous frame, the target frame, and the subsequent frame into grayscale images, and calculating the grayscale values of the grayscale images corresponding to the previous frame, the target frame, and the subsequent frame, respectively; based on the grayscale value of the previous frame, the grayscale value of the target frame, the grayscale value of the subsequent frame, and the minimum grayscale value of the image frame of the frame sequence, determining whether the target frame is an alternative eye-closing frame.
[0009] In an optional embodiment of the present application, the step of determining whether the target frame is an alternative eye-closing frame based on the grayscale value of the previous frame, the grayscale value of the target frame, the grayscale value of the next frame and the minimum grayscale value of the image frame of the frame sequence includes: if the grayscale value of the previous frame is less than the grayscale value of the target frame, the grayscale value of the target frame is greater than or equal to the grayscale value of the next frame, and the difference between the grayscale value of the target frame and the minimum grayscale value of the image frame of the frame sequence is greater than a preset threshold, determining that the target frame is an alternative eye-closing frame.
[0010] In an optional embodiment of the present application, after the above-mentioned step of determining whether the target frame is an alternative closed eye frame, the method also includes: performing threshold segmentation and morphological processing on the alternative closed eye frame to obtain a contour image of the eye; determining the maximum contour of the contour image, and determining whether the alternative closed eye frame is a closed eye frame based on the width, height and area of the maximum contour.
[0011] In an optional embodiment of the present application, the above-mentioned step of determining whether the alternative closed eye frame is a closed eye frame based on the width, height and area of the maximum contour includes: if the area of the maximum contour is greater than a preset area threshold, and the ratio of the width to the height of the maximum contour is greater than a preset aspect ratio threshold, determining that the alternative closed eye frame is a closed eye frame; if the area of the maximum contour is less than or equal to the area threshold, determining that the alternative closed eye frame is a closed eye frame.
[0012] In an optional embodiment of the present application, the above-mentioned segmentation model includes: an attention mechanism module and a depth-separable convolution module.
[0013] In an optional embodiment of the present application, the step of determining the incomplete blink rate of the subject based on the unclosed eyelid area of multiple closed eye frames includes: determining the height values of multiple positions at preset horizontal intervals from the unclosed eyelid area of each closed eye frame; determining a non-zero height value from the height values of the multiple positions, and calculating the average of the non-zero height values; determining each closed eye frame as an incomplete closed eye frame or a completely closed eye frame based on the average value; taking the ratio of the number of incomplete closed eye frames to the total number of closed eye frames as the incomplete blink rate of the subject; wherein the total number of closed eye frames is the sum of the number of incomplete closed eye frames and the number of completely closed eye frames.
[0014] In an optional embodiment of the present application, the above-mentioned step of determining whether each eye-closing frame is an incomplete eye-closing frame or a complete eye-closing frame based on the average value includes: if the average value is greater than a preset height threshold, determining that the eye-closing frame is an incomplete eye-closing frame; if the average value is less than or equal to the height threshold, determining that the eye-closing frame is a complete eye-closing frame.
[0015] In a second aspect, an embodiment of the present invention further provides a device for detecting an incomplete blink rate, the device comprising: a blink image acquisition module, for acquiring multiple blink images of a subject based on the acquisition module; wherein the multiple blink images are a continuous frame sequence; a closed-eye frame determination module, for determining multiple closed-eye frames from the frame sequence; wherein the closed-eye frames represent image frames in which the eyelids are closed to the greatest extent during each blink of the subject; a segmentation model processing module, for inputting multiple closed-eye frames into a pre-trained segmentation model, and outputting eyelid unclosed areas of multiple closed-eye frames; an incomplete blink rate detection module, for determining the incomplete blink rate of the subject based on the eyelid unclosed areas of multiple closed-eye frames.
[0016] The embodiments of the present invention bring the following beneficial effects:
[0017] The embodiment of the present invention provides a method and device for detecting an incomplete blink rate, which collects multiple blink images of a subject based on an acquisition module; wherein the multiple blink images are a continuous frame sequence; multiple closed eye frames are determined from the frame sequence; wherein the closed eye frames represent image frames in which the subject's eyelids are most closed during each blink; the multiple closed eye frames are input into a pre-trained segmentation model, and the eyelid unclosed regions of the multiple closed eye frames are output; and the incomplete blink rate of the subject is determined based on the eyelid unclosed regions of the multiple closed eye frames. In this method, the closed eye frames in which the subject's eyelids are most closed during each blink can be automatically extracted from the entire frame sequence, the eyelid unclosed regions of the closed eye frames are extracted through the segmentation model, and then the incomplete blink judgment is performed through high-level calculation, thereby quickly and accurately determining the incomplete blink rate of the subject.
[0018] Other features and advantages of the present disclosure will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present disclosure.
[0019] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0021] Figure 1 A flow chart of a method for detecting an incomplete blink rate provided by an embodiment of the present invention;
[0022] Figure 2 A flowchart of another incomplete blink rate detection method provided by an embodiment of the present invention;
[0023] Figure 3 A schematic diagram of a grayscale image of an alternative eye-closing frame provided by an embodiment of the present invention;
[0024] Figure 4 A schematic diagram of a contour image provided by an embodiment of the present invention;
[0025] Figure 5 A schematic diagram of a segmentation network structure of a segmentation model provided by an embodiment of the present invention;
[0026] Figure 6 A schematic diagram of a segmentation result of a segmentation model provided by an embodiment of the present invention;
[0027] Figure 7 A schematic diagram of height detection of an unclosed area provided by an embodiment of the present invention;
[0028] Figure 8 A schematic structural diagram of an incomplete blink rate detection device provided by an embodiment of the present invention;
[0029] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0031] At present, there are two ways to automatically analyze incomplete blink rate: based on traditional image processing methods and based on deep learning methods. However, for a continuous frame sequence containing blinks, the traditional image processing scheme to calculate the degree of incomplete blinks is greatly affected by ambient lighting, eyelash occlusion and complex eye environment, and the calculation results are less robust. The scheme that uses deep learning methods to segment the unclosed eyelid area of each frame has relatively strong resistance to environmental interference, but it is still difficult to ensure robustness against motion blur caused by rapid eyelid closure at the moment of blinking. In addition, blink frames generally only occupy a small part of the entire frame sequence, and the model's reasoning on normal open eye frames occupies the vast majority of processing time and running memory, affecting the effective use of computing resources.
[0032] For example, in the incomplete blink rate detection method based on deep learning, the device captures data in the form of 8 frames per second (FPS) video, while normal blinking generally takes 0.2 to 0.4 seconds. At this frame rate, only 1 to 3 images of the upper eyelid from closing downward to opening upward can be captured for model training and result reasoning. It is difficult to characterize the true features of the eyelid and eyeball at each moment during rapid blinking and the feature differences at adjacent moments in terms of quantity and quality. The judged blink frame image cannot fully represent the true situation at the moment when the eyelid is most closed during blinking, which may cause misdiagnosis by doctors.
[0033] In addition, the above-mentioned incomplete blink rate detection method based on deep learning method does not divide normal eye-open frames and blink frames. The vast majority of normal eye-open frames used as input to participate in model reasoning cannot produce the target results, and occupy most of the computing time and running memory.
[0034] Based on this, an embodiment of the present invention provides a method and device for detecting an incomplete blink rate, which can provide an acquisition rate of 40 frames per second to capture more and more subtle real images of the blinking moment; it can also automatically remove normal open eye frames from the entire frame sequence, and only retain the closed eye frames in which the subject's eyelids are closed to the greatest extent during each blink, and extract the unclosed eyelid area of the closed eye frames by introducing a segmentation model that introduces an attention mechanism and a deeply separable convolution module, and then perform incomplete blink judgment through highly calculated results, thereby quickly and accurately determining the subject's incomplete blink rate.
[0035] To facilitate understanding of this embodiment, a method for detecting an incomplete blink rate disclosed in an embodiment of the present invention is first introduced in detail.
[0036] Embodiment 1:
[0037] The embodiment of the present invention provides a method for detecting an incomplete blink rate. Figure 1 A flow chart of a method for detecting an incomplete blink rate is shown, and the method for detecting an incomplete blink rate comprises the following steps:
[0038] Step S102: collecting multiple blink images of the subject based on the collection module; wherein the multiple blink images are continuous frame sequences.
[0039] In this embodiment, a high frame rate acquisition module can be used to ensure that the acquired frame sequence can include images of the moment when the eyelid is most closed during the blinking process. A high frame rate acquisition module can be used to obtain a blinking image with better representation.
[0040] Step S104, determining a plurality of eye-closing frames from the frame sequence; wherein the eye-closing frames represent image frames in which the eyelids are closed to the greatest extent during each blink of the subject.
[0041] In this embodiment, the image frames with the greatest degree of eyelid closure during each blinking process can be automatically extracted from the entire frame sequence, and the extracted image frames can be called eye-closing frames.
[0042] Step S106: input the multiple eye-closing frames into a pre-trained segmentation model, and output the eyelid open areas of the multiple eye-closing frames.
[0043] In this embodiment, only the extracted closed eye frames can be input into the pre-trained segmentation model, and the segmentation results output by the segmentation model can be the eyelid open areas of multiple closed eye frames. Therefore, in this embodiment, it is not necessary to use the vast majority of normal open eye frames as the input of the segmentation model, and only the closed eye frames with the largest degree of eyelid closure during each blink process are used as the input of the segmentation model, thereby saving computational time and running memory.
[0044] Step S108, determining the incomplete blink rate of the subject based on the eyelid open area of the multiple eye-closing frames.
[0045] After determining the eyelid unclosed area of multiple eye-closed frames, the present embodiment can also automatically analyze the unclosed height of the unclosed area based on the eyelid unclosed area of multiple eye-closed frames, thereby calculating the subject's incomplete blink rate, thereby ensuring the accuracy and efficiency of the incomplete blink rate detection results.
[0046] The embodiment of the present invention provides a method for detecting an incomplete blink rate, which collects multiple blink images of a subject based on an acquisition module; wherein the multiple blink images are a continuous frame sequence; multiple closed eye frames are determined from the frame sequence; wherein the closed eye frames represent image frames in which the subject's eyelids are most closed during each blink; the multiple closed eye frames are input into a pre-trained segmentation model, and the eyelid unclosed regions of the multiple closed eye frames are output; and the subject's incomplete blink rate is determined based on the eyelid unclosed regions of the multiple closed eye frames. In this method, the closed eye frames in which the subject's eyelids are most closed during each blink can be automatically extracted from the entire frame sequence, the eyelid unclosed regions of the closed eye frames are extracted through the segmentation model, and then the incomplete blink judgment is performed through high-level calculation, thereby quickly and accurately determining the subject's incomplete blink rate.
[0047] Embodiment 2:
[0048] This embodiment provides another method for detecting incomplete blink rate, which is implemented on the basis of the above embodiment. Figure 2 The flowchart of another incomplete blink rate detection method is shown, and the incomplete blink rate detection method includes the following steps:
[0049] Step S202: collecting multiple blink images of the subject based on the collection module; wherein the multiple blink images are continuous frame sequences.
[0050] In some embodiments, the acquisition module can be aimed at the eyes of the subject; the acquisition module captures the subject's normal open eye image, upper eyelid closed downward image, eyelid closure maximum degree image and upper eyelid open upward image at a rate of 40 frames per second as multiple blinking images of the subject.
[0051] In this embodiment, data collection of blinking images can be performed first. The acquisition module can collect a frame sequence of several blinks of the subject's eyes. The above-mentioned acquisition module can be a high frame rate acquisition module, which can shoot normal open eye images, upper eyelid closed images, maximum eyelid closed images and upper eyelid opened images in RGB (Red Green Blue) color space at a rate of 40 frames per second. The size of the image is 2072×1464. In order to maintain the efficiency of the processing, in this embodiment, each frame can be captured and the frame can be judged as a closed eye frame.
[0052] Step S204, determining candidate eye-closing frames from the frame sequence, and determining eye-closing frames from the candidate eye-closing frames.
[0053] After acquiring the frame sequence, this embodiment can also extract eye-closing frames. In this embodiment, whether each image frame in the frame sequence is an eye-closing frame can be determined in two steps: first determine whether the image frame is an alternative eye-closing frame, and then determine whether the alternative eye-closing frame is an eye-closing frame.
[0054] In some embodiments, the previous frame of the target frame, the target frame, and the next frame of the target frame can be determined from the frame sequence; the previous frame, the target frame, and the next frame are converted into grayscale images, and the grayscale values of the grayscale images corresponding to the previous frame, the target frame, and the next frame are calculated; based on the grayscale value of the previous frame, the grayscale value of the target frame, the grayscale value of the next frame, and the minimum grayscale value of the image frame of the frame sequence, it is determined whether the target frame is an alternative closed eye frame.
[0055] For a sequence of frames input continuously, this embodiment can determine whether each frame is the frame with the largest eyelid closure during a blink (referred to as a closed eye frame). In a blink process, to determine that the eyelid closure of a target frame is the largest, it is necessary to combine the eyelid closure of at least three frames, namely the target frame, the previous frame, and the next frame, to make a judgment.
[0056] In view of this, the specific calculation method adopted in this embodiment is as follows: first, the input current frame (the current frame is taken as the frame after the target frame, denoted as the cth frame) image is converted into a grayscale image and its grayscale value (denoted as gv_c) is calculated, and then the grayscale values of the previous two frames of the current frame are combined (the previous frame of the current frame is taken as the target frame, denoted as the bth frame; the previous two frames of the current frame are taken as the previous frames of the target frame, denoted as the ath frame, and the grayscale values are denoted as gv_a and gv_b respectively) and the minimum grayscale value of all the input frames (denoted as gv_min), and the eye closing condition of the alternative eye closing frame is judged for the bth frame (that is, the target frame).
[0057] In some embodiments, if the grayscale value of the previous frame is less than the grayscale value of the target frame, the grayscale value of the target frame is greater than or equal to the grayscale value of the next frame, and the difference between the grayscale value of the target frame and the minimum grayscale value of the image frame of the frame sequence is greater than a preset threshold, the target frame is determined to be an alternative closed-eye frame.
[0058] For example, the eye closing condition judgment of the candidate eye closing frame may be performed as follows: Among them, gv_a is the grayscale value of the previous frame, gv_b is the grayscale value of the target frame, gv_c is the grayscale value of the next frame, gv_min is the minimum grayscale value, and 5 is the preset threshold.
[0059] The constant 5 is a preset threshold for comparing the grayscale values of the image, which means that the grayscale value of the candidate closed-eye frame must be at least 5 greater than the grayscale value of the frame with the smallest grayscale value among all frames. If the bth frame (i.e., the target frame) meets the judgment condition, it can be determined as a candidate closed-eye frame. Figure 3A schematic diagram of a grayscale image of an alternative eye-closing frame is shown.
[0060] In some embodiments, the candidate closed eye frame may be subjected to threshold segmentation and morphological processing to obtain a contour image of the eye; the maximum contour of the contour image may be determined, and whether the candidate closed eye frame is a closed eye frame may be determined based on the width, height and area of the maximum contour.
[0061] For the candidate closed-eye frames, after excluding the slight eye movement during the normal eye-opening process, they can be determined as closed-eye frames. The specific operations are as follows: First, threshold segmentation and morphological processing are performed on the candidate closed-eye frames to obtain the contour image of the eye. Figure 4 A schematic diagram of a contour image is shown in FIG. Then, the largest contour in the contour image is extracted and the condition of the eye-closing frame is judged.
[0062] There may be two cases for the candidate closed eye frame: 1. The eyes are not closed and include the iris; 2. The eyes are completely closed and do not include the iris. Therefore, the above two cases can be judged based on the eye contour image.
[0063] In some embodiments, if the area of the maximum contour is greater than a preset area threshold, and the ratio of the width to the height of the maximum contour is greater than a preset aspect ratio threshold, the alternative closed eye frame is determined to be a closed eye frame; if the area of the maximum contour is less than or equal to the area threshold, the alternative closed eye frame is determined to be a closed eye frame.
[0064] Among them, "if the area of the largest contour is less than or equal to the area threshold, determine the candidate closed eye frame as the closed eye frame" implies the case where the eyes are completely closed and there is no iris (without considering the influence of image noise such as eyelashes, the area of the largest contour is 0, which means the eyes are completely closed and there is no iris). In addition, Figure 4 Shown in FIG. 1 is a case where the eye is not closed including the iris.
[0065] For example, the following eye-closing frame conditional judgment can be performed: Among them, area_b represents the area of the largest contour, w_b and h_b represent the width and height of the largest contour respectively, 80000 is the preset area threshold, and 2 is the preset aspect ratio threshold.
[0066] The constant 80000 is a preset threshold for judging the contour area, in units of pixels, and the constant 2 is a preset threshold for judging the contour aspect ratio. When the maximum contour meets one of the above conditions and the eyelid closure degree exceeds the iris midline, the candidate closed eye frame is confirmed as the closed eye frame.
[0067] According to the above judgment provided by this embodiment, after traversing and processing all input frames frame by frame, the closed-eye frame image with the maximum eyelid closure degree in each blinking process in the frame sequence is obtained.
[0068] Step S206: input the multiple eye-closing frames into the pre-trained segmentation model, and output the eyelid open areas of the multiple eye-closing frames.
[0069] After obtaining the closed eye frame image, the height of the eyelid open area in the closed eye frame image can be determined. The pre-trained semantic segmentation network model can be used to extract the eyelid open area.
[0070] In some embodiments, the above segmentation model includes: an attention mechanism module and a depth-wise separable convolution module.
[0071] Among them, the above-mentioned semantic segmentation network can use the U-Net network as the backbone network and introduce the attention mechanism to enhance the feature extraction capability. At the same time, in order to improve the model reasoning speed, all convolutional layers therein are replaced with depthwise separable convolutional modules.
[0072] See also Figure 5 The diagram shows a segmentation network structure diagram of a segmentation model. The semantic segmentation network adopts an encoder-decoder structure, and adds an attention module between each layer of the decoder. The encoder part contains five layers of depth-separable convolution blocks, each of which consists of two parts: channel-by-channel convolution and point-by-point convolution, which are used to improve the computational efficiency of feature extraction. The decoder part upsamples the output of the encoder and performs splicing and fusion. When performing splicing and fusion, the current layer of the encoder and the next layer of the decoder are first sent as input to the attention module to obtain the feature map of the enhanced region of interest, and then the channel dimension is fused to further improve the model's feature extraction capability. Finally, a 1×1 convolution is used to map the feature map to a two-dimensional mask output.
[0073] In addition, the loss function used in the training process of the above segmentation model can be BCELoss (Binary CrossEntropy Loss), and the evaluation index used can be IoU (Intersection over Union), and the segmentation model is obtained by training. Figure 3 As the input of the segmentation model, we can get Figure 6 Schematic diagram of the segmentation results of a segmentation model shown.
[0074] Step S208, determining the incomplete blink rate of the subject based on the eyelid open area of the multiple eye-closing frames.
[0075] In some embodiments, the height values of multiple positions at preset horizontal intervals can be determined from the eyelid unclosed area of each eye-closing frame; a non-zero height value can be determined from the height values of the multiple positions, and an average of the non-zero height values can be calculated; based on the average value, each eye-closing frame is determined to be an incomplete eye-closing frame or a complete eye-closing frame; the ratio of the number of incomplete eye-closing frames to the total number of eye-closing frames is taken as the incomplete blink rate of the subject; wherein the total number of eye-closing frames is the sum of the number of incomplete eye-closing frames and the number of complete eye-closing frames.
[0076] After obtaining the segmentation result of the unclosed area in the closed-eye frame, the present embodiment can traverse from top to bottom to detect the heights of multiple positions of the unclosed area at a certain horizontal interval, and after averaging all non-zero heights, obtain the average height result of the unclosed area (which can be called the height of the unclosed eyelid area), and then judge the degree of incomplete eye closure.
[0077] In some embodiments, if the average value is greater than a preset height threshold, the eye-closing frame is determined to be an incomplete eye-closing frame; if the average value is less than or equal to the height threshold, the eye-closing frame is determined to be a complete eye-closing frame.
[0078] See also Figure 7 A schematic diagram of height detection of an unclosed area is shown in FIG. Figure 7 5 positions are shown in FIG. 1 , and the degree of incomplete eye closure of the b-th frame (i.e., the target frame) can be determined as follows: Among them, ic_frame represents the incomplete eye closure frame, c_frame represents the complete eye closure frame, and h represents the height of the eyelid open area of the bth frame (i.e., the target frame). The constant 57 is the height threshold, which is the preset threshold for determining the height of the open area, and the unit is the number of pixels.
[0079] Therefore, after obtaining the eyelid unclosed height of all closed eye frames selected from the continuous frame sequence, the incomplete blink rate ic_rate can be calculated by the following formula: Among them, ∑ic_frame is the number of incomplete eye-closure frames, ∑c_frame is the number of complete eye-closure frames, and ∑c_frame is the total number of eye-closure frames.
[0080] The above method provided by the embodiment of the present invention can provide an acquisition rate of 40 frames per second to capture more and more subtle real images of the blinking moment; it can also automatically remove normal open eye frames from the entire frame sequence, and only retain the closed eye frames with the largest degree of eyelid closure during each blink of the subject. By introducing a segmentation model with an attention mechanism and a deep separable convolution module, the eyelid unclosed area of the closed eye frame is extracted, and then incomplete blink judgment is performed through high-level calculation, thereby quickly and accurately determining the subject's incomplete blink rate.
[0081] Embodiment three:
[0082] Corresponding to the above method embodiment, the present invention provides a device for detecting an incomplete blink rate, see Figure 8 A schematic structural diagram of an incomplete blink rate detection device is shown, and the incomplete blink rate detection device includes:
[0083] A blink image acquisition module 81 is used to acquire multiple blink images of a subject based on an acquisition module; wherein the multiple blink images are continuous frame sequences;
[0084] The eye-closing frame determination module 82 is used to determine a plurality of eye-closing frames from the frame sequence; wherein the eye-closing frame represents an image frame in which the eyelid is closed to the greatest extent during each blink of the subject;
[0085] The segmentation model processing module 83 is used to input a plurality of eye-closing frames into a pre-trained segmentation model, and output the eyelid open areas of the plurality of eye-closing frames;
[0086] The incomplete blink rate detection module 84 is used to determine the incomplete blink rate of the subject based on the unclosed eyelid area of multiple closed eye frames.
[0087] The embodiment of the present invention provides a device for detecting an incomplete blink rate, which collects multiple blink images of a subject based on an acquisition module; wherein the multiple blink images are a continuous frame sequence; multiple closed eye frames are determined from the frame sequence; wherein the closed eye frames represent image frames in which the subject's eyelids are most closed during each blink; the multiple closed eye frames are input into a pre-trained segmentation model, and the eyelid unclosed regions of the multiple closed eye frames are output; and the subject's incomplete blink rate is determined based on the eyelid unclosed regions of the multiple closed eye frames. In this method, the closed eye frames in which the subject's eyelids are most closed during each blink can be automatically extracted from the entire frame sequence, the eyelid unclosed regions of the closed eye frames are extracted through the segmentation model, and then the incomplete blink judgment is performed through high-level calculation, thereby quickly and accurately determining the subject's incomplete blink rate.
[0088] The above-mentioned blinking image acquisition module is used to aim the acquisition module at the subject's eyes; the acquisition module captures the subject's normal open eye image, upper eyelid closed downward image, eyelid closure maximum degree image and upper eyelid open upward image at a rate of 40 frames per second as multiple blinking images of the subject.
[0089] The above-mentioned closed-eye frame determination module is used to determine the previous frame of the target frame, the target frame and the next frame of the target frame from the frame sequence; convert the previous frame, the target frame and the next frame into grayscale images, and calculate the grayscale values of the grayscale images corresponding to the previous frame, the target frame and the next frame respectively; based on the grayscale value of the previous frame, the grayscale value of the target frame, the grayscale value of the next frame and the minimum grayscale value of the image frame of the frame sequence, determine whether the target frame is an alternative closed-eye frame.
[0090] The above-mentioned closed-eye frame determination module is used to determine that the target frame is an alternative closed-eye frame if the grayscale value of the previous frame is less than the grayscale value of the target frame, the grayscale value of the target frame is greater than or equal to the grayscale value of the next frame, and the difference between the grayscale value of the target frame and the minimum grayscale value of the image frame of the frame sequence is greater than a preset threshold.
[0091] The above-mentioned closed eye frame determination module is also used to perform threshold segmentation and morphological processing on the candidate closed eye frame to obtain a contour image of the eye; determine the maximum contour of the contour image, and determine whether the candidate closed eye frame is a closed eye frame based on the width, height and area of the maximum contour.
[0092] The above-mentioned closed-eye frame determination module is used to determine that the alternative closed-eye frame is the closed-eye frame if the area of the maximum contour is greater than a preset area threshold, and the ratio of the width to the height of the maximum contour is greater than a preset aspect ratio threshold; if the area of the maximum contour is less than or equal to the area threshold, determine that the alternative closed-eye frame is the closed-eye frame.
[0093] The above segmentation model includes: an attention mechanism module and a depth-wise separable convolution module.
[0094] The above-mentioned incomplete blink rate detection module is used to determine the height values of multiple positions at preset horizontal intervals from the eyelid unclosed area of each eye-closing frame; determine a non-zero height value from the height values of multiple positions, and calculate the average value of the non-zero height values; determine each eye-closing frame as an incomplete eye-closing frame or a complete eye-closing frame based on the average value; take the ratio of the number of incomplete eye-closing frames to the total number of eye-closing frames as the incomplete blink rate of the subject; wherein the total number of eye-closing frames is the sum of the number of incomplete eye-closing frames and the number of complete eye-closing frames.
[0095] The incomplete blink rate detection module is used to determine that the eye-closing frame is an incomplete eye-closing frame if the average value is greater than a preset height threshold; if the average value is less than or equal to the height threshold, determine that the eye-closing frame is a complete eye-closing frame.
[0096] Technicians in the relevant field can clearly understand that, for the convenience and simplicity of description, the specific working process of the incomplete blink rate detection system described above can refer to the corresponding process in the aforementioned embodiment of the incomplete blink rate detection method, and will not be repeated here.
[0097] Embodiment 4:
[0098] The embodiment of the present invention further provides an electronic device for executing the above-mentioned incomplete blink rate detection method; see Fig. 9A structural schematic diagram of an electronic device is shown, which includes a memory 100 and a processor 101, wherein the memory 100 is used to store one or more computer instructions, and the one or more computer instructions are executed by the processor 101 to implement the above-mentioned incomplete blink rate detection method.
[0099] Further, Fig. 9 The electronic device shown further includes a bus 102 and a communication interface 103 , and the processor 101 , the communication interface 103 and the memory 100 are connected via the bus 102 .
[0100] The memory 100 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0101] The processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 101. The above processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 100, and the processor 101 reads the information in the memory 100 and completes the steps of the method of the above embodiment in combination with its hardware.
[0102] An embodiment of the present invention also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned incomplete blink rate detection method. The specific implementation can be found in the method embodiment, which will not be repeated here.
[0103] The computer program product of the incomplete blink rate detection method and device provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method in the previous method embodiment. The specific implementation can be found in the method embodiment, which will not be repeated here.
[0104] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and / or device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0105] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0106] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0107] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.
[0108] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A method for detecting an incomplete blink rate, characterized in that: The method comprises: Collecting multiple blink images of the subject based on the acquisition module; wherein the multiple blink images are continuous frame sequences; Determine a plurality of eye-closing frames from the frame sequence; wherein the eye-closing frames represent image frames in which the eyelids of the subject are most closed during each blink; Inputting the plurality of eye-closing frames into a pre-trained segmentation model, and outputting the eyelid open areas of the plurality of eye-closing frames; An incomplete blink rate of the subject is determined based on the eyelid open area of the plurality of eye-closed frames.
2. The method according to claim 1, characterized in that The step of acquiring multiple blink images of a subject based on an acquisition module comprises: The acquisition module is aimed at the subject's eyes; The acquisition module captures the subject's normal open-eye image, the upper eyelid closed downward image, the eyelid closed to the maximum degree image and the upper eyelid opened upward image at a rate of 40 frames per second as multiple blinking images of the subject.
3. The method according to claim 1, characterized in that The step of determining a plurality of eye-closed frames from the frame sequence comprises: Determine a previous frame of a target frame, the target frame, and a subsequent frame of the target frame from the frame sequence; Converting the previous frame, the target frame and the next frame into grayscale images, and calculating the grayscale values of the grayscale images corresponding to the previous frame, the target frame and the next frame respectively; Based on the grayscale value of the previous frame, the grayscale value of the target frame, the grayscale value of the next frame and the minimum grayscale value of the image frames of the frame sequence, it is determined whether the target frame is a candidate eye-closing frame.
4. The method according to claim 3, characterized in that The step of determining whether the target frame is a candidate eye-closing frame based on the grayscale value of the previous frame, the grayscale value of the target frame, the grayscale value of the next frame, and the minimum grayscale value of the image frame of the frame sequence comprises: If the grayscale value of the previous frame is less than the grayscale value of the target frame, the grayscale value of the target frame is greater than or equal to the grayscale value of the subsequent frame, and the difference between the grayscale value of the target frame and the minimum grayscale value of the image frame of the frame sequence is greater than a preset threshold, the target frame is determined to be an alternative closed-eye frame.
5. The method according to claim 3, characterized in that: After the step of determining whether the target frame is a candidate eye-closing frame, the method further includes: Performing threshold segmentation and morphological processing on the candidate closed-eye frame to obtain an eye contour image; The maximum contour of the contour image is determined, and whether the candidate eye-closing frame is an eye-closing frame is determined based on the width, height and area of the maximum contour.
6. The method according to claim 5, characterized in that The step of determining whether the candidate eye-closing frame is an eye-closing frame based on the width, height and area of the maximum contour comprises: If the area of the maximum contour is greater than a preset area threshold, and the ratio of the width to the height of the maximum contour is greater than a preset aspect ratio threshold, determining that the candidate eye-closing frame is an eye-closing frame; If the area of the maximum contour is less than or equal to the area threshold, the candidate eye-closing frame is determined to be an eye-closing frame.
7. The method according to claim 1, characterized in that The segmentation model includes: an attention mechanism module and a depth-separable convolution module.
8. The method according to claim 1, characterized in that The step of determining the incomplete blink rate of the subject based on the eyelid open areas of the plurality of eye-closing frames comprises: Determine the height values of a plurality of positions at preset horizontal intervals from the eyelid open area of each of the eye-closing frames; Determine the non-zero height values from the height values of the plurality of positions, and calculate an average value of the non-zero height values; Determine, based on the average value, whether each of the eye-closing frames is an incomplete eye-closing frame or a complete eye-closing frame; The ratio of the number of the incomplete eye-closure frames to the total number of the eye-closure frames is taken as the incomplete blink rate of the subject; wherein the total number of the eye-closure frames is the sum of the number of the incomplete eye-closure frames and the number of the complete eye-closure frames.
9. The method according to claim 8, characterized in that The step of determining whether each of the eye-closing frames is an incomplete eye-closing frame or a complete eye-closing frame based on the average value comprises: If the average value is greater than a preset height threshold, determining that the eye-closing frame is an incomplete eye-closing frame; If the average value is less than or equal to the height threshold, it is determined that the eye-closed frame is a completely eye-closed frame.
10. A device for detecting an incomplete blink rate, characterized in that: The device comprises: A blink image acquisition module, used for acquiring multiple blink images of a subject based on the acquisition module; wherein the multiple blink images are a continuous frame sequence; An eye-closing frame determination module is used to determine a plurality of eye-closing frames from the frame sequence; wherein the eye-closing frame represents an image frame in which the eyelid closure degree of the subject is the largest during each blinking process; A segmentation model processing module, used for inputting the plurality of eye-closing frames into a pre-trained segmentation model, and outputting the eyelid unclosed areas of the plurality of eye-closing frames; The incomplete blink rate detection module is used to determine the incomplete blink rate of the subject based on the unclosed eyelid area of the multiple closed eye frames.
