Blink detection method, apparatus, device and medium

Abstract

The present application belongs to the technical field of face detection. Disclosed are a blink detection method, an apparatus, a device and a medium. The method comprises: performing face key point detection on each face image frame in a face image sequence; calculating eye aspect ratios of the corresponding frames on the basis of a face key point set, and enabling the eye aspect ratios of all the frames to constitute an eye aspect ratio sequence; filtering the eye aspect ratio sequence to obtain a denoised aspect ratio sequence; normalizing the denoised aspect ratio sequence to obtain a normalized aspect ratio sequence; performing trough detection on the normalized aspect ratio sequence to obtain a trough set; calculating the gradient of the normalized aspect ratio sequence, and, on the basis of the gradient of the normalized aspect ratio sequence, filtering the trough set to obtain a plurality of target troughs; and calculating a starting moment and an ending moment corresponding to each target trough, and determining blink actions on the basis of the starting moments and the ending moments. The described method can more accurately determine blink actions, reducing the probability of missed detection and false detection.

Description

Blink detection method, device, equipment and medium Technical Field

[0001] The present application relates to the field of face detection technology, for example, to a blink detection method, device, equipment and medium. Background Art

[0002] Currently, electronic devices are widely used. When people face electronic screens for a long time or use incorrect eye posture, it often causes eye fatigue. Changes in the number of blinks can reflect whether the eyes are tired.

[0003] Common blink detection methods mainly include vision-based methods and EEG-based methods. Currently, there are two types of vision-based methods: the first type inputs a certain number of eye images and predicts the probability that the corresponding time interval is a blink process, such as CN115761869A. This type of method is significantly affected by the number of input frames. The second type calculates blink-related features from eye key points and sets a fixed threshold. When the feature is less than the threshold and persists for a certain period of time, it is determined to be a blink, such as CN115019377A. This type of method uses a fixed threshold, resulting in different results between different individuals.

[0004] In summary, the existing technology has the problem of being greatly affected by the number of input frames or having different effects among different individuals. Technical issues

[0005] The purpose of the present application is to provide a blink detection method, apparatus, device and medium, which can solve the problem that the existing technology is greatly affected by the number of input frames or the effect varies among different individuals. Technical Solutions

[0006] This application provides a blink detection method, comprising:

[0007] Acquire a facial image sequence, perform facial key point detection on each facial image frame in the facial image sequence, and obtain a facial key point set;

[0008] Calculating eye aspect ratios of corresponding frames according to the facial key point set, and composing the eye aspect ratios of all frames into an eye aspect ratio sequence;

[0009] filtering the eye aspect ratio sequence to obtain a denoised aspect ratio sequence;

[0010] Normalizing the denoised aspect ratio sequence to obtain a normalized aspect ratio sequence;

[0011] Performing trough detection on the normalized aspect ratio sequence to obtain a trough set;

[0012] Calculating the gradient of the normalized aspect ratio sequence, and filtering the trough set according to the gradient of the normalized aspect ratio sequence to obtain a plurality of target troughs;

[0013] The start time and the end time corresponding to each target trough are calculated, and a blink action is determined according to the start time and the end time.

[0014] The present application provides a blink detection device, comprising:

[0015] A facial key point detection module is used to obtain a facial image sequence, perform facial key point detection on each facial image frame in the facial image sequence, and obtain a facial key point set;

[0016] an eye aspect ratio sequence composition module, configured to calculate the eye aspect ratio of the corresponding frame according to the facial key point set, and compose the eye aspect ratios of all frames into an eye aspect ratio sequence;

[0017] a filtering module, configured to filter the eye aspect ratio sequence to obtain a denoised aspect ratio sequence;

[0018] a normalization module, configured to normalize the denoised aspect ratio sequence to obtain a normalized aspect ratio sequence;

[0019] a trough detection module, configured to perform trough detection on the normalized aspect ratio sequence to obtain a trough set;

[0020] a trough set filtering module, configured to calculate the gradient of the normalized aspect ratio sequence, and filter the trough set according to the gradient of the normalized aspect ratio sequence to obtain a plurality of target troughs;

[0021] The blink action determination module is used to calculate the start time and the end time corresponding to each target trough, and determine the blink action according to the start time and the end time.

[0022] The present application also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements a blink detection method when executing the computer program;

[0023] The blink detection method includes:

[0024] Acquire a facial image sequence, perform facial key point detection on each facial image frame in the facial image sequence, and obtain a facial key point set;

[0025] Calculating eye aspect ratios of corresponding frames according to the facial key point set, and composing the eye aspect ratios of all frames into an eye aspect ratio sequence;

[0026] filtering the eye aspect ratio sequence to obtain a denoised aspect ratio sequence;

[0027] Normalizing the denoised aspect ratio sequence to obtain a normalized aspect ratio sequence;

[0028] Performing trough detection on the normalized aspect ratio sequence to obtain a trough set;

[0029] Calculating the gradient of the normalized aspect ratio sequence, and filtering the trough set according to the gradient of the normalized aspect ratio sequence to obtain a plurality of target troughs;

[0030] The start time and the end time corresponding to each target trough are calculated, and a blink action is determined according to the start time and the end time.

[0031] The present application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements a blink detection method when executed by a processor;

[0032] The blink detection method includes:

[0033] Acquire a facial image sequence, perform facial key point detection on each facial image frame in the facial image sequence, and obtain a facial key point set;

[0034] Calculating eye aspect ratios of corresponding frames according to the facial key point set, and composing the eye aspect ratios of all frames into an eye aspect ratio sequence;

[0035] filtering the eye aspect ratio sequence to obtain a denoised aspect ratio sequence;

[0036] Normalizing the denoised aspect ratio sequence to obtain a normalized aspect ratio sequence;

[0037] Performing trough detection on the normalized aspect ratio sequence to obtain a trough set;

[0038] Calculating the gradient of the normalized aspect ratio sequence, and filtering the trough set according to the gradient of the normalized aspect ratio sequence to obtain a plurality of target troughs;

[0039] The start time and the end time corresponding to each target trough are calculated, and a blink action is determined according to the start time and the end time. Beneficial effects

[0040] The present application discloses a blink detection method, comprising obtaining a facial image sequence, performing facial key point detection on each facial image frame in the facial image sequence to obtain a facial key point set; calculating the eye aspect ratio of the corresponding frame based on the facial key point set, and forming the eye aspect ratios of all frames into an eye aspect ratio sequence; filtering the eye aspect ratio sequence to obtain a denoised aspect ratio sequence; normalizing the denoised aspect ratio sequence to obtain a normalized aspect ratio sequence; performing trough detection on the normalized aspect ratio sequence to obtain a trough set; calculating the gradient of the normalized aspect ratio sequence, filtering the trough set based on the gradient of the normalized aspect ratio sequence to obtain multiple target troughs; calculating the start time and end time corresponding to each target trough, and determining the blink action based on the start time and the end time. The above method determines a gradient threshold based on the gradient of the normalized aspect ratio sequence of the person to be detected. The gradient threshold calculated for different persons to be detected is different and is less affected by the number of input frames. Compared to using a fixed gradient threshold for all subjects, this method can more accurately identify blinks, reducing the probability of missed and false detections. A larger gradient in the normalized aspect ratio sequence indicates faster eye movement. Blinks change more rapidly than other eye movements, and the gradient of the normalized aspect ratio sequence can filter out various non-blinking patterns, further reducing the probability of false detections. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] FIG1 is a schematic diagram of facial key points according to an embodiment;

[0042] FIG2 is a schematic diagram of a normalized aspect ratio sequence according to an embodiment;

[0043] FIG3 is a schematic diagram of a gradient of a normalized aspect ratio sequence according to an embodiment;

[0044] FIG4 is a schematic diagram of a flow chart of a blink detection method according to an embodiment;

[0045] FIG5 is a schematic diagram of a process for performing valley detection on a normalized aspect ratio sequence according to an embodiment;

[0046] FIG6 is a schematic diagram of a process for filtering a trough set according to a gradient of a normalized aspect ratio sequence according to an embodiment;

[0047] FIG7 is a schematic block diagram of the structure of a blink detection device according to an embodiment;

[0048] FIG8 is a schematic block diagram of the structure of a computer device according to an embodiment.

[0049] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. Best Mode for Carrying Out the Invention

[0050] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0051] Those skilled in the art will understand that, unless expressly stated otherwise, the singular forms "a", "an", "above", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of this application refers to the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any module and all combinations of one or more associated listed items.

[0052] Those skilled in the art will understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which this invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless specifically defined as such, will not be interpreted in an idealized or overly formal sense.

[0053] With the widespread use of electronic devices such as computers, smartphones, and tablets, people's work, entertainment, and daily lives are heavily reliant on screens. Prolonged screen time or incorrect eye posture often leads to eye fatigue, with faster or slower blinks being a sign of eye fatigue. Common blink detection methods include vision-based and EEG-based methods. Compared to EEG-based methods, vision-based methods require no additional hardware, are simpler to implement, and are less expensive. Currently, there are two types of vision-based methods. The first type inputs a certain number of eye images and predicts the probability that the corresponding time interval represents a blink. This method requires a specific number of frames to be fed into the model. However, in real-world scenarios, the duration of blinks can vary significantly. Using a small number of input frames may interrupt a complete blink, resulting in a prediction of a non-blink. Using a large number of input frames may include multiple blinks, and a single model may only predict a single blink. The second type uses eye key points to calculate features related to blinking. By setting a fixed threshold, if the feature is less than the threshold and persists for a certain period of time, the corresponding feature is counted as a blink. This type of method uses a fixed threshold, resulting in different results between different individuals.

[0054] Since the most noticeable change in a blink is the eye's closure, two keypoints on the upper eyelid and two corresponding keypoints on the lower eyelid can be used to calculate the distance between the upper and lower eyelids. As the distance between the face and the camera changes, the pixel positions in the facial image change accordingly, and the pixel distance also changes, necessitating normalization to obtain the eye aspect ratio. By calculating the magnitude of the decrease in the eye aspect ratio, we can identify possible blink intervals. During a blink, the eye aspect ratio decreases and then increases. When it reaches its minimum value, it corresponds to the moment of eye closure and is also a trough in the curve. Due to the fluctuation of keypoints, the normalized aspect ratio sequence will exhibit many troughs. However, the magnitude of the decrease in troughs caused by noise and those caused by real blinks differs. Troughs in real blink intervals are generally significantly lower than those caused by noise. By filtering the trough set according to the gradient of the normalized aspect ratio sequence, the noise troughs can be filtered out, and each remaining trough is regarded as a target trough. Since if the time interval corresponding to the trough is too long, the interval corresponding to the trough may be an interval of long-term eye closure, the blink action can be determined by calculating the start and end time corresponding to each target trough.

[0055] In one embodiment, referring to FIG. 1 to FIG. 4 , FIG. 1 is a flow chart of a blink detection method disclosed in the present application, the method comprising:

[0056] S1: Acquire a facial image sequence, perform facial key point detection on each facial image frame in the facial image sequence, and obtain a facial key point set.

[0057] The facial image sequence includes multiple facial images arranged in chronological order. The facial images can be in color or grayscale, which is not limited here.

[0058] Using an open source facial key point detection model to perform facial key point detection on each facial image frame in the facial image sequence to obtain a plurality of facial key points;

[0059] All the facial key points are taken as the facial key point set.

[0060] The open source facial key point detection model used in the embodiment of the present application is Mediapipe, which is used in the fields of image, video and audio processing. Facial key point detection is performed on each frame of facial image to obtain 68 facial key points, which are used as a facial key point set.

[0061] S2: Calculating the eye aspect ratio of the corresponding frame according to the facial key point set, and assembling the eye aspect ratios of all frames into an eye aspect ratio sequence.

[0062] The eye aspect ratio of the corresponding frame is calculated according to the following formula:

[0063] Since the most significant change in blinking is the eye closing, we select two key points of the upper eyelid and two key points of the lower eyelid. Taking the right eye as an example, the distance between the upper eyelid and the lower eyelid is , which is the pixel distance. When the distance between the face and the camera changes, the pixel position of the face image will change accordingly, and the pixel distance will also change, so standardization is required. In this embodiment of the application, a first reference key point and a second reference key point that are less affected by movements such as facial expressions and blinking are selected for standardization. One of the reference key points is the key point of the root of the nose, and the other is the key point of the base of the nose.

[0064] S3: Filtering the eye aspect ratio sequence to obtain a denoised aspect ratio sequence.

[0065] The eye aspect ratio sequence is filtered using SG filtering to obtain the denoised aspect ratio sequence.

[0066] SG filtering, also known as Savitzky-Golay filtering, is widely used for smoothing and denoising data streams. It is a filtering method based on local polynomial least squares fitting in the time domain. The advantage of SG filtering is that it can filter out noise while preserving the shape and width of the signal. Because facial landmarks detected during facial landmark detection often contain a lot of noise, such as jitter and / or errors, which can affect blink detection, SG filtering can filter out some of the noise in the eye aspect ratio sequence while ensuring that the shape of the eye aspect ratio sequence itself does not change significantly.

[0067] S4: Normalizing the denoised aspect ratio sequence to obtain a normalized aspect ratio sequence.

[0068] The denoised aspect ratio sequence is normalized by the following formula:

[0069] Since the eye sizes of different subjects are different when their eyes are open, the difference can be reduced to a certain extent by normalizing the maximum and minimum values ​​of the aspect ratio sequence of each subject.

[0070] S5: Perform trough detection on the normalized aspect ratio sequence to obtain a trough set.

[0071] The troughs in the trough set may correspond to real blinking movements or to noise. The troughs corresponding to real blinking movements are generally significantly lower than the troughs corresponding to noise.

[0072] S6: Calculate the gradient of the normalized aspect ratio sequence, and filter the trough set according to the gradient of the normalized aspect ratio sequence to obtain a plurality of target troughs.

[0073] The gradient of the normalized aspect ratio sequence reflects the speed of eye changes. The faster the eye changes, the larger the gradient; the slower the eye changes, the smaller the gradient.

[0074] Blinking and other eye movements may cause troughs. According to the gradient of the normalized aspect ratio sequence, the troughs corresponding to other movements in the trough set are filtered out to obtain the target trough corresponding to blinking.

[0075] S7: Calculate the start time and the end time corresponding to each target trough, and determine the blinking action according to the start time and the end time.

[0076] The horizontal axis of Figures 2 and 3 represents time. The vertical axis of Figure 2 represents the eye aspect ratio, and the vertical axis of Figure 3 represents the gradient. Referring to Figure 3, when a blink occurs, the gradient of the blink process will first increase in the positive direction until the eye is completely closed, and then increase in the negative direction. The moment the gradient reaches its maximum is recorded as the start of the blink, and the moment the gradient reaches its maximum in the negative direction is recorded as the end of the blink.

[0077] A blink detection method according to an embodiment of the present application includes obtaining a facial image sequence, performing facial key point detection on each facial image frame in the facial image sequence to obtain a facial key point set; calculating the eye aspect ratio of the corresponding frame based on the facial key point set, and forming the eye aspect ratios of all frames into an eye aspect ratio sequence; filtering the eye aspect ratio sequence to obtain a denoised aspect ratio sequence; normalizing the denoised aspect ratio sequence to obtain a normalized aspect ratio sequence; performing trough detection on the normalized aspect ratio sequence to obtain a trough set; calculating the gradient of the normalized aspect ratio sequence, filtering the trough set based on the gradient of the normalized aspect ratio sequence to obtain multiple target troughs; calculating the start time and end time corresponding to each target trough, and determining the blink action based on the start time and the end time. The above method determines a gradient threshold based on the gradient of the normalized aspect ratio sequence of the person to be detected. The gradient threshold calculated for different persons to be detected is different and is less affected by the number of input frames. Compared to using a fixed gradient threshold for all subjects, this method can more accurately identify blinks, reducing the probability of missed and false detections. A larger gradient in the normalized aspect ratio sequence indicates faster eye movement. Blinks change more rapidly than other eye movements, and the gradient of the normalized aspect ratio sequence can filter out various non-blinking patterns, further reducing the probability of false detections.

[0078] In one embodiment, referring to FIG. 5 , performing trough detection on the normalized aspect ratio sequence to obtain a trough set includes:

[0079] S51: taking the eye aspect ratio corresponding to the minimum peaks on both sides of the trough in the normalized aspect ratio sequence as the intermediate value.

[0080] The trough refers to the minimum value of the amplitude within a wavelength range. The highest point of the transverse wave is called the crest, and the lowest point is called the trough.

[0081] The range of a wavelength is set to 3 frames, that is, as long as a frame has a smaller eye aspect ratio than both the previous and next frames, the frame is a trough, otherwise it is a peak.

[0082] As shown in Figures 2-3, during a blink, the eye aspect ratio in the normalized aspect ratio sequence decreases and then increases. When it reaches its minimum value, it corresponds to the moment of eye closure and also marks a trough in the normalized aspect ratio sequence. Due to the fluctuation of key points, the normalized aspect ratio sequence exhibits many troughs. However, the troughs caused by noise and those caused by real blinks have different decreases. The troughs corresponding to real blinks are generally significantly lower than those caused by noise.

[0083] The formula for calculating the median value is:

[0084] S53: Detect whether the aspect ratio difference is greater than or equal to a difference threshold; if so, take the corresponding trough as a true trough.

[0085] Preferably, the difference threshold is set to 0.3. If the aspect ratio difference is greater than or equal to the difference threshold, it indicates that the trough may be in the blink process, and the corresponding trough is regarded as the true trough.

[0086] S54: All the real troughs are taken as the trough set.

[0087] The advantage of the above-mentioned search for the true trough is that larger troughs can be skipped during the search process. The difference between the eye aspect ratio of the peaks on both sides of the trough and the eye aspect ratio of the trough must be greater than or equal to the difference threshold, which can reduce the situation where troughs with smaller changes caused by fluctuations in the eye aspect ratio are mistakenly detected as blinks.

[0088] As described above, performing trough detection on the normalized aspect ratio sequence to obtain a trough set includes taking the eye aspect ratio corresponding to the minimum peaks on either side of a trough in the normalized aspect ratio sequence as an intermediate value. Subtracting the eye aspect ratio of the trough from the intermediate value to obtain an aspect ratio difference. Testing whether the aspect ratio difference is greater than or equal to a difference threshold; if so, determining the corresponding trough as a true trough. Using all true troughs as the trough set can reduce the possibility of falsely detecting troughs with smaller changes due to eye aspect ratio fluctuations as blinks.

[0089] In one embodiment, referring to FIG6 , filtering the trough set according to the gradient of the normalized aspect ratio sequence to obtain a plurality of target troughs includes:

[0090] S62: Calculate the gradient corresponding to the previous moment of each trough in the trough set to obtain a gradient set.

[0091] Before step S62 , the method further includes step S61 : calculating the gradient of the normalized aspect ratio sequence.

[0092] In real-world scenarios, blinking isn't the only factor that can cause a decrease in eye aspect ratio. Incomplete blinking and squinting can also cause aspect ratio differences greater than or equal to the difference threshold. To filter out these situations, the present embodiment utilizes the fact that blinking is faster than other processes like squinting. The gradient corresponding to the previous moment of each trough in the trough set is calculated to obtain a gradient set.

[0093] S63: Filter out a maximum gradient from the gradient set, and use half of the maximum gradient as a gradient threshold.

[0094] The calculation formula of the gradient threshold is:

[0095] S64: Utilize the trough in the trough set whose gradient is greater than the gradient threshold as the target trough.

[0096] When the gradient of a trough in the trough set exceeds a gradient threshold, that trough corresponds to a true blink. The gradient threshold is calculated and varies for different facial image sequences of different individuals. Compared to using a fixed gradient threshold, using a gradient threshold tailored to the individual can more accurately identify blinks, reducing the probability of missed and false detections.

[0097] As described above, filtering the trough set based on the gradient of the normalized aspect ratio sequence to obtain multiple target troughs includes calculating the gradient corresponding to the previous moment of each trough in the trough set to obtain a gradient set. The maximum gradient is selected from the gradient set, and half of the maximum gradient is used as the gradient threshold. The troughs in the trough set with gradients greater than the gradient threshold are used as the target troughs. Compared to using a fixed gradient threshold, using a gradient threshold tailored to the individual being tested can more accurately determine blinking, reducing the probability of missed detection and false detection.

[0098] In one embodiment, calculating the start time and end time corresponding to each target trough includes:

[0099] S71: Searching for a moment near each target trough where the gradient is greater than the gradient threshold; if so, recording the corresponding moment as the start moment.

[0100] Without exceeding the previous trough, search for the moment when the gradient is greater than the gradient threshold, and record this moment as the starting moment.

[0101] If there is no moment when the gradient is greater than the gradient threshold, the interval corresponding to the target trough is not a blink interval, that is, the target trough cannot be regarded as a blink action.

[0102] S72: Calculate the product of the gradient corresponding to the start time and the first multiple to obtain a gradient discrimination value.

[0103] The maximum speed that can be achieved when the eyes are open is smaller than the maximum speed that can be achieved when the eyes are closed. Preferably, the first multiple is set to -0.4.

[0104] S73: Record the time corresponding to the gradient discrimination value as the end time.

[0105] The calculation formula for the end time is:

[0106] Ending moment.

[0107] As described above, calculating the start and end times corresponding to each target trough includes searching for a time near each target trough where the gradient is greater than the gradient threshold. If so, the corresponding time is recorded as the start time. The product of the gradient corresponding to the start time and the first multiple is calculated to obtain a gradient discrimination value. The time corresponding to the gradient discrimination value is recorded as the end time. Based on the start and end times, a blink can be determined.

[0108] In one embodiment, determining the blinking action according to the start time and the end time includes:

[0109] S74: Calculating the time length according to the start time and the end time;

[0110] The difference between the end time and the start time is used as the time length. The larger the difference, the longer the time length.

[0111] S75: Counting the target trough corresponding to the interval whose time length is less than the time length threshold as a blink action.

[0112] If the duration is too long, the corresponding interval may be a period of prolonged eye closure. Therefore, intervals with a duration greater than or equal to the duration threshold are filtered out, and intervals with a duration less than the duration threshold are considered blink intervals. The target trough corresponding to this interval can be considered a blink action.

[0113] As described above, determining a blink based on the start and end times includes calculating a duration based on the start and end times; and identifying a target trough corresponding to an interval where the duration is less than a time duration threshold as a blink. The difference between the end and start times is defined as the duration. By comparing the duration with the time duration threshold, a blink interval can be determined, and the target trough corresponding to the interval can be identified as a blink.

[0114] 7 is a schematic block diagram of a blink detection device disclosed in the present application, the device comprising:

[0115] The facial key point detection module 10 is used to obtain a facial image sequence, perform facial key point detection on each facial image frame in the facial image sequence, and obtain a facial key point set;

[0116] an eye aspect ratio sequence forming module 20, configured to calculate the eye aspect ratios of corresponding frames according to the facial key point set, and form an eye aspect ratio sequence from the eye aspect ratios of all frames;

[0117] a filtering module 30, configured to filter the eye aspect ratio sequence to obtain a denoised aspect ratio sequence;

[0118] a normalization module 40, configured to normalize the denoised aspect ratio sequence to obtain a normalized aspect ratio sequence;

[0119] a trough detection module 50, configured to perform trough detection on the normalized aspect ratio sequence to obtain a trough set;

[0120] a trough set filtering module 60 for calculating the gradient of the normalized aspect ratio sequence, filtering the trough set according to the gradient of the normalized aspect ratio sequence, and obtaining a plurality of target troughs;

[0121] The blink action determination module 70 is configured to calculate the start time and the end time corresponding to each target trough, and determine the blink action according to the start time and the end time.

[0122] The above-mentioned blink detection device is used to implement the blink detection method.

[0123] In one embodiment, the eye aspect ratio sequence composition module 20 further includes:

[0124] The eye aspect ratio calculation unit is configured to calculate the eye aspect ratio of the corresponding frame according to the following formula:

[0125] In one embodiment, the normalization module 40 further includes:

[0126] A normalization unit is used to normalize the denoised aspect ratio sequence using the following formula:

[0127] In one embodiment, the trough detection module 50 further includes:

[0128] an intermediate value calculation unit, configured to take the eye aspect ratio corresponding to the minimum peaks on both sides of the trough in the normalized aspect ratio sequence as the intermediate value;

[0129] an aspect ratio difference calculation unit, configured to subtract the eye aspect ratio of the trough from the intermediate value to obtain an aspect ratio difference;

[0130] a true trough definition unit, configured to detect whether the aspect ratio difference is greater than or equal to a difference threshold, and if so, to define the corresponding trough as a true trough;

[0131] The trough set definition unit is used to take all the real troughs as the trough set.

[0132] In one embodiment, the trough set filtering module 60 further includes:

[0133] a gradient calculation unit, configured to calculate the gradient corresponding to the previous moment of each trough in the trough set to obtain a gradient set;

[0134] a gradient threshold definition unit, configured to screen out a maximum gradient from the gradient set and use half of the maximum gradient as a gradient threshold;

[0135] The target trough definition unit is configured to take a trough in the trough set whose gradient is greater than the gradient threshold as the target trough.

[0136] In one embodiment, the blink action determination module 70 further includes:

[0137] a start time definition unit, configured to search for a time near each target trough at which the gradient is greater than the gradient threshold, and if so, record the corresponding time as the start time;

[0138] a gradient discriminant value calculation unit, configured to calculate the product of the gradient corresponding to the start time and the first multiple to obtain a gradient discriminant value;

[0139] The end time definition unit is used to record the time corresponding to the gradient discrimination value as the end time.

[0140] In one embodiment, the facial key point detection module 10 further includes:

[0141] A facial key point detection unit, configured to perform facial key point detection on each face image frame in the face image sequence using an open source facial key point detection model to obtain a plurality of facial key points;

[0142] The facial key point set definition unit is configured to take all the facial key points as the facial key point set.

[0143] In one embodiment, the blink action determination module 70 further includes:

[0144] a time length determining unit, configured to calculate the time length according to the start time and the end time;

[0145] The blink action determination unit is configured to determine the target trough corresponding to the interval whose time length is less than the time length threshold as a blink action.

[0146] In one embodiment, the filtering module 30 further includes:

[0147] A filtering unit is configured to filter the eye aspect ratio sequence using SG filtering to obtain the denoised aspect ratio sequence.

[0148] 8 , a computer device is further provided in an embodiment of the present application, and the internal structure of the computer device may be as shown in FIG8 . The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. The processor designed for the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating device, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store target troughs, etc. The network interface of the computer device is used to communicate with an external terminal via a network connection. Furthermore, the above-mentioned computer device may also be provided with an input device, a display screen, etc. When the computer program is executed by a processor, it implements a blink detection method, comprising the following steps: obtaining a facial image sequence, performing facial key point detection on each facial image frame in the facial image sequence to obtain a facial key point set; calculating the eye aspect ratio of the corresponding frame based on the facial key point set, and forming the eye aspect ratios of all frames into an eye aspect ratio sequence; filtering the eye aspect ratio sequence to obtain a denoised aspect ratio sequence; normalizing the denoised aspect ratio sequence to obtain a normalized aspect ratio sequence; performing trough detection on the normalized aspect ratio sequence to obtain a trough set; calculating the gradient of the normalized aspect ratio sequence, filtering the trough set based on the gradient of the normalized aspect ratio sequence to obtain multiple target troughs; calculating the start time and end time corresponding to each target trough, and determining the blink action based on the start time and the end time. It will be understood by those skilled in the art that the structure shown in FIG8 is only a block diagram of a partial structure related to the present application scheme, and does not constitute a limitation on the computer device to which the present application scheme is applied.

[0149] In one embodiment, calculating the eye aspect ratio of the corresponding frame according to the set of facial key points includes:

[0150] The eye aspect ratio of the corresponding frame is calculated according to the following formula:

[0151] In one embodiment, normalizing the denoised aspect ratio sequence to obtain a normalized aspect ratio sequence includes:

[0152] The denoised aspect ratio sequence is normalized by the following formula:

[0153] In one embodiment, performing trough detection on the normalized aspect ratio sequence to obtain a trough set includes:

[0154] The eye aspect ratio corresponding to the minimum peak on both sides of the trough in the normalized aspect ratio sequence is taken as the median value;

[0155] Subtract the eye aspect ratio of the trough from the median value to obtain an aspect ratio difference;

[0156] detecting whether the aspect ratio difference is greater than or equal to a difference threshold, and if so, taking the corresponding trough as a true trough;

[0157] All the real troughs are regarded as the trough set.

[0158] In one embodiment, filtering the trough set according to the gradient of the normalized aspect ratio sequence to obtain a plurality of target troughs includes:

[0159] Calculating the gradient corresponding to the previous moment of each trough in the trough set to obtain a gradient set;

[0160] Screening out a maximum gradient from the gradient set, and using half of the maximum gradient as a gradient threshold;

[0161] A trough in the trough set whose gradient is greater than the gradient threshold is used as the target trough.

[0162] In one embodiment, calculating the start time and end time corresponding to each target trough includes:

[0163] Searching for a moment near each target trough where the gradient is greater than the gradient threshold, and if so, recording the corresponding moment as the start time;

[0164] Calculating the product of the gradient corresponding to the start time and the first multiple to obtain a gradient discrimination value;

[0165] The time corresponding to the gradient discriminant value is recorded as the end time.

[0166] In one embodiment, the performing facial key point detection on each facial image frame in the facial image sequence to obtain a facial key point set includes:

[0167] Using an open source facial key point detection model to perform facial key point detection on each facial image frame in the facial image sequence to obtain a plurality of facial key points;

[0168] All the facial key points are taken as the facial key point set.

[0169] In one embodiment, determining the blinking action according to the start time and the end time includes:

[0170] Calculate the time length according to the start time and the end time;

[0171] The target trough corresponding to the interval whose time length is less than the time length threshold is regarded as a blink action.

[0172] In one embodiment, filtering the eye aspect ratio sequence to obtain a denoised aspect ratio sequence includes:

[0173] The eye aspect ratio sequence is filtered using SG filtering to obtain the denoised aspect ratio sequence.

[0174] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements a blink detection method, comprising the following steps: obtaining a facial image sequence, performing facial key point detection on each facial image frame in the facial image sequence to obtain a facial key point set; calculating the eye aspect ratio of the corresponding frame based on the facial key point set, and forming the eye aspect ratios of all frames into an eye aspect ratio sequence; filtering the eye aspect ratio sequence to obtain a denoised aspect ratio sequence; normalizing the denoised aspect ratio sequence to obtain a normalized aspect ratio sequence; performing trough detection on the normalized aspect ratio sequence to obtain a trough set; calculating the gradient of the normalized aspect ratio sequence, filtering the trough set based on the gradient of the normalized aspect ratio sequence to obtain multiple target troughs; calculating the start time and end time corresponding to each target trough, and determining the blink action based on the start time and the end time.

[0175] In one embodiment, calculating the eye aspect ratio of the corresponding frame according to the set of facial key points includes:

[0176] The eye aspect ratio of the corresponding frame is calculated according to the following formula:

[0177] In one embodiment, normalizing the denoised aspect ratio sequence to obtain a normalized aspect ratio sequence includes:

[0178] The denoised aspect ratio sequence is normalized by the following formula:

[0179] In one embodiment, performing trough detection on the normalized aspect ratio sequence to obtain a trough set includes:

[0180] The eye aspect ratio corresponding to the minimum peak on both sides of the trough in the normalized aspect ratio sequence is taken as the median value;

[0181] Subtract the eye aspect ratio of the trough from the median value to obtain an aspect ratio difference;

[0182] detecting whether the aspect ratio difference is greater than or equal to a difference threshold, and if so, taking the corresponding trough as a true trough;

[0183] All the real troughs are regarded as the trough set.

[0184] In one embodiment, filtering the trough set according to the gradient of the normalized aspect ratio sequence to obtain a plurality of target troughs includes:

[0185] Calculating the gradient corresponding to the previous moment of each trough in the trough set to obtain a gradient set;

[0186] Screening out a maximum gradient from the gradient set, and using half of the maximum gradient as a gradient threshold;

[0187] A trough in the trough set whose gradient is greater than the gradient threshold is used as the target trough.

[0188] In one embodiment, calculating the start time and end time corresponding to each target trough includes:

[0189] Searching for a moment near each target trough where the gradient is greater than the gradient threshold, and if so, recording the corresponding moment as the start time;

[0190] Calculating the product of the gradient corresponding to the start time and the first multiple to obtain a gradient discrimination value;

[0191] The time corresponding to the gradient discriminant value is recorded as the end time.

[0192] In one embodiment, the performing facial key point detection on each facial image frame in the facial image sequence to obtain a facial key point set includes:

[0193] Using an open source facial key point detection model to perform facial key point detection on each facial image frame in the facial image sequence to obtain a plurality of facial key points;

[0194] All the facial key points are taken as the facial key point set.

[0195] In one embodiment, determining the blinking action according to the start time and the end time includes:

[0196] Calculate the time length according to the start time and the end time;

[0197] The target trough corresponding to the interval whose time length is less than the time length threshold is regarded as a blink action.

[0198] In one embodiment, filtering the eye aspect ratio sequence to obtain a denoised aspect ratio sequence includes:

[0199] The eye aspect ratio sequence is filtered using SG filtering to obtain the denoised aspect ratio sequence.

[0200] It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0201] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAM bus dynamic RAM (RDRAM).

[0202] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0203] The above description is only a preferred embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A blink detection method, wherein: include: Acquire a facial image sequence, perform facial key point detection on each facial image frame in the facial image sequence, and obtain a facial key point set; Calculating the eye aspect ratio of the corresponding frame according to the facial key point set, and composing the eye aspect ratios of all frames into an eye aspect ratio sequence; filtering the eye aspect ratio sequence to obtain a denoised aspect ratio sequence; Normalizing the denoised aspect ratio sequence to obtain a normalized aspect ratio sequence; Performing trough detection on the normalized aspect ratio sequence to obtain a trough set; Calculating the gradient of the normalized aspect ratio sequence, filtering the trough set according to the gradient of the normalized aspect ratio sequence, and obtaining a plurality of target troughs; The start time and the end time corresponding to each target trough are calculated, and the blinking action is determined according to the start time and the end time.

2. The blink detection method according to claim 1, wherein: The step of calculating the eye aspect ratio of the corresponding frame according to the facial key point set comprises: The eye aspect ratio of the corresponding frame is calculated according to the following formula:

3. The blink detection method according to claim 1, wherein: The step of normalizing the denoised aspect ratio sequence to obtain a normalized aspect ratio sequence comprises: The denoised aspect ratio sequence is normalized by the following formula:

4. The blink detection method according to claim 1, wherein: The performing trough detection on the normalized aspect ratio sequence to obtain a trough set includes: The eye aspect ratio corresponding to the minimum peak on both sides of the trough in the normalized aspect ratio sequence is taken as the intermediate value; Subtract the eye aspect ratio of the trough from the median value to obtain an aspect ratio difference; Detecting whether the aspect ratio difference is greater than or equal to a difference threshold, and if so, taking the corresponding trough as a true trough; All the real troughs are regarded as the trough set.

5. The blink detection method according to claim 1, wherein: The filtering of the trough set according to the gradient of the normalized aspect ratio sequence to obtain a plurality of target troughs includes: Calculate the gradient corresponding to the previous moment of each trough in the trough set to obtain a gradient set; Screening out a maximum gradient from the gradient set, and taking half of the maximum gradient as a gradient threshold; A trough in the trough set whose gradient is greater than the gradient threshold is taken as the target trough.

6. The blink detection method according to claim 5, wherein: The calculating the start time and the end time corresponding to each target trough includes: Searching for a moment when the gradient is greater than the gradient threshold near each target trough, and if so, recording the corresponding moment as the start moment; Calculate the product of the gradient corresponding to the start time and the first multiple to obtain a gradient discrimination value; The time corresponding to the gradient discrimination value is recorded as the end time.

7. The blink detection method according to claim 1, wherein: The step of performing facial key point detection on each frame of the facial image in the facial image sequence to obtain a facial key point set includes: Using an open source facial key point detection model to perform facial key point detection on each facial image frame in the facial image sequence to obtain a plurality of facial key points; All the facial key points are taken as the facial key point set.

8. The blink detection method according to claim 1, wherein: The step of determining the blinking action according to the start time and the end time includes: Calculate the time length according to the start time and the end time; The target trough corresponding to the interval whose time length is less than the time length threshold is regarded as a blink action.

9. The blink detection method according to claim 1, wherein: The filtering the eye aspect ratio sequence to obtain a denoised aspect ratio sequence comprises: The eye aspect ratio sequence is filtered using SG filtering to obtain the denoised aspect ratio sequence.

10. A blink detection device, wherein: include: A facial key point detection module is used to obtain a facial image sequence, perform facial key point detection on each facial image frame in the facial image sequence, and obtain a facial key point set; An eye aspect ratio sequence composition module, used for calculating the eye aspect ratio of the corresponding frame according to the facial key point set, and composing the eye aspect ratios of all frames into an eye aspect ratio sequence; A filtering module, used for filtering the eye aspect ratio sequence to obtain a denoised aspect ratio sequence; A normalization module, used for normalizing the denoised aspect ratio sequence to obtain a normalized aspect ratio sequence; A trough detection module, used for performing trough detection on the normalized aspect ratio sequence to obtain a trough set; A trough set filtering module, used to calculate the gradient of the normalized aspect ratio sequence, and filter the trough set according to the gradient of the normalized aspect ratio sequence to obtain a plurality of target troughs; The blink action determination module is used to calculate the start time and the end time corresponding to each of the target troughs, and determine the blink action according to the start time and the end time.

11. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, a blink detection method is implemented; Wherein, the blink detection method comprises: Acquire a facial image sequence, perform facial key point detection on each facial image frame in the facial image sequence, and obtain a facial key point set; Calculating the eye aspect ratio of the corresponding frame according to the facial key point set, and composing the eye aspect ratios of all frames into an eye aspect ratio sequence; filtering the eye aspect ratio sequence to obtain a denoised aspect ratio sequence; Normalizing the denoised aspect ratio sequence to obtain a normalized aspect ratio sequence; Performing trough detection on the normalized aspect ratio sequence to obtain a trough set; Calculating the gradient of the normalized aspect ratio sequence, filtering the trough set according to the gradient of the normalized aspect ratio sequence, and obtaining a plurality of target troughs; The start time and the end time corresponding to each target trough are calculated, and the blinking action is determined according to the start time and the end time.

12. The computer device according to claim 11, wherein: The step of calculating the eye aspect ratio of the corresponding frame according to the facial key point set comprises: The eye aspect ratio of the corresponding frame is calculated according to the following formula:

13. The computer device according to claim 11, wherein: The step of normalizing the denoised aspect ratio sequence to obtain a normalized aspect ratio sequence comprises: The denoised aspect ratio sequence is normalized by the following formula:

14. The computer device according to claim 11, wherein: The performing trough detection on the normalized aspect ratio sequence to obtain a trough set includes: The eye aspect ratio corresponding to the minimum peak on both sides of the trough in the normalized aspect ratio sequence is taken as the intermediate value; Subtract the eye aspect ratio of the trough from the median value to obtain an aspect ratio difference; Detecting whether the aspect ratio difference is greater than or equal to a difference threshold, and if so, taking the corresponding trough as a true trough; All the real troughs are regarded as the trough set.

15. The computer device of claim 11, wherein: The filtering of the trough set according to the gradient of the normalized aspect ratio sequence to obtain a plurality of target troughs includes: Calculate the gradient corresponding to the previous moment of each trough in the trough set to obtain a gradient set; Screening out a maximum gradient from the gradient set, and taking half of the maximum gradient as a gradient threshold; A trough in the trough set whose gradient is greater than the gradient threshold is taken as the target trough.

16. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, a blink detection method is implemented; Wherein, the blink detection method comprises: Acquire a facial image sequence, perform facial key point detection on each facial image frame in the facial image sequence, and obtain a facial key point set; Calculating the eye aspect ratio of the corresponding frame according to the facial key point set, and composing the eye aspect ratios of all frames into an eye aspect ratio sequence; filtering the eye aspect ratio sequence to obtain a denoised aspect ratio sequence; Normalizing the denoised aspect ratio sequence to obtain a normalized aspect ratio sequence; Performing trough detection on the normalized aspect ratio sequence to obtain a trough set; Calculating the gradient of the normalized aspect ratio sequence, filtering the trough set according to the gradient of the normalized aspect ratio sequence, and obtaining a plurality of target troughs; The start time and the end time corresponding to each target trough are calculated, and the blinking action is determined according to the start time and the end time.

17. The computer-readable storage medium of claim 16, wherein: The step of calculating the eye aspect ratio of the corresponding frame according to the facial key point set comprises: The eye aspect ratio of the corresponding frame is calculated according to the following formula:

18. The computer-readable storage medium of claim 16, wherein: The step of normalizing the denoised aspect ratio sequence to obtain a normalized aspect ratio sequence comprises: The denoised aspect ratio sequence is normalized by the following formula:

19. The computer-readable storage medium of claim 16, wherein: The performing trough detection on the normalized aspect ratio sequence to obtain a trough set includes: The eye aspect ratio corresponding to the minimum peak on both sides of the trough in the normalized aspect ratio sequence is taken as the intermediate value; Subtract the eye aspect ratio of the trough from the median value to obtain an aspect ratio difference; Detecting whether the aspect ratio difference is greater than or equal to a difference threshold, and if so, taking the corresponding trough as a true trough; All the real troughs are regarded as the trough set.

20. The computer-readable storage medium of claim 16, wherein: The filtering of the trough set according to the gradient of the normalized aspect ratio sequence to obtain a plurality of target troughs includes: Calculate the gradient corresponding to the previous moment of each trough in the trough set to obtain a gradient set; Screening out a maximum gradient from the gradient set, and taking half of the maximum gradient as a gradient threshold; A trough in the trough set whose gradient is greater than the gradient threshold is taken as the target trough.

Images