Blink detection method and device based on continuous frames and electronic equipment

Abstract

The invention relates to a blink detection method and device based on continuous frames and electronic equipment, and belongs to the technical field of intelligent cabin perception, and the detection method comprises the steps: obtaining an eye state value of each frame in a preset number of continuous frames through a single-frame eye state value algorithm, obtaining an eye state change curve according to the eye state value of each frame in the preset number of continuous frames, and determining whether to blink according to the eye state change curve; according to the method, whether blinking occurs or not is judged through the eye state change curve, the blinking process from eye closing to eye opening is concerned, false detection caused by the sizes of the eyes of the driver is effectively reduced, and the accuracy of blinking detection and fatigue driving detection is improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent cockpit perception technology, and in particular to a blink detection method, device and electronic equipment based on continuous frames. Background Art

[0002] In recent years, the intelligent cockpit has developed rapidly, and the driver status monitoring system is a hot development direction. The system mainly monitors the driver's status in real time and promptly reminds the driver when an abnormal state is detected, so as to ensure the life and property safety of the driver and passengers. Blink detection is one of the important technologies of the driver status monitoring system. This technology provides basic information for judging the driver's status by real-time monitoring of the occurrence and frequency of the driver's blinking action. Therefore, the accuracy of blink detection is one of the important indicators to measure the performance of the entire driver status monitoring system.

[0003] The technical solution with patent number "CN202410569303.9" is to detect whether the driver blinks based on the change of eye area, but this method is based on a simple judgment based on a fixed preset threshold. The judgment logic does not take into account the size of the driver's eyes themselves, which is prone to false detection problems. Summary of the invention

[0004] To this end, the present invention provides a blink detection method, device and electronic device based on continuous frames to solve the problem of false detection that is easy to occur in the prior art, which is based on simple judgment based on fixed preset thresholds and the judgment logic does not consider the size of the driver's eyes.

[0005] In a first aspect, a method for blink detection based on continuous frames is provided, the method comprising:

[0006] Obtaining the eye state value of each frame in a preset number of consecutive frames through a single-frame eye state value algorithm;

[0007] Obtaining an eye state change curve according to the eye state value of each frame in the preset number of consecutive frames;

[0008] Determine whether to blink according to the eye state change curve.

[0009] Furthermore, obtaining the eye state change curve according to the eye state value of each frame in the preset number of continuous frames includes:

[0010] storing the eye state value of each frame in the preset number of consecutive frames into an eye state queue; the queue length of the eye state queue is the preset number;

[0011] Performing curve fitting on the elements in the eye state queue based on a polynomial function to obtain the eye state change curve;

[0012] Each time a real-time frame is acquired, the eye state value of the real-time frame is stored in the eye state queue and then curve fitting is performed to obtain the continuously updated eye state change curve.

[0013] Further, determining whether to blink according to the eye state change curve includes:

[0014] Obtaining points with a curvature of 0 in the eye state change curve and arranging them in chronological order to obtain an extreme point queue;

[0015] The values ​​of two consecutive extreme value points in the extreme value point queue are obtained in chronological order. If the value difference between the two extreme value points is greater than a first preset value and the value of any one of the two extreme value points is 0, the number of blinks is added once.

[0016] Furthermore, the single-frame eye state value algorithm includes:

[0017] Identify the maximum eyelid distance between the upper and lower eyelids and the maximum eye canthus distance between the left and right eye corners in a single frame image;

[0018] The eye state value of the single frame image is obtained according to the maximum eyelid distance and the maximum eye corner distance.

[0019] Further, obtaining the eye state value of the single-frame image according to the maximum eyelid distance and the maximum eye canthus distance includes:

[0020] The eye status value is obtained by the status value formula, and the status value formula is:

[0021] S = H × W;

[0022] Among them, S is the eye state value, H is the maximum eyelid distance, and W is the maximum eye canthus distance.

[0023] Furthermore, the method further comprises:

[0024] If the values ​​of a preset number of consecutive extreme points in the extreme point queue are all less than a second preset value, the driver's status is determined to be fatigue driving, a fatigue driving reminder is issued in the car and a reminder message is sent to a reserved mobile phone number.

[0025] In a second aspect, a blink detection device based on continuous frames is provided, the device comprising:

[0026] A state value module, used to obtain the eye state value of each frame in a preset number of consecutive frames through a single-frame eye state value algorithm;

[0027] A change curve module, used for obtaining an eye state change curve according to the eye state value of each frame in the preset number of continuous frames;

[0028] The blink determination module is used to determine whether to blink according to the eye state change curve.

[0029] In a third aspect, an electronic device is provided, including:

[0030] at least one processor; and

[0031] a memory communicatively connected to the at least one processor; wherein,

[0032] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any of the above-mentioned blink detection methods based on continuous frames.

[0033] The present invention adopts the above technical solution and has at least the following beneficial effects:

[0034] Provided are a blink detection method, device and electronic device based on continuous frames. The eye state value of each frame in a preset number of continuous frames is obtained through a single-frame eye state value algorithm, an eye state change curve is obtained according to the eye state value of each frame in the preset number of continuous frames, and whether a blink occurs is determined according to the eye state change curve. The method determines whether a blink occurs through the eye state change curve, focusing on the blink process from closing eyes to opening eyes, effectively reducing false detections caused by the size of the driver's eyes, and improving the accuracy of blink detection and fatigue driving detection.

[0035] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 is a flow chart of a blink detection method based on continuous frames according to an exemplary embodiment of the present invention;

[0038] Figure 2 is a schematic block diagram of a blink detection device based on continuous frames according to an exemplary embodiment of the present invention;

[0039] Figure 3 It is a schematic block diagram of an electronic device shown in an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0041] At present, the proportion of traffic accidents caused by fatigue driving or distracted driving remains high. Strengthening the monitoring of driver status and reminding drivers to maintain safe driving status has become an effective means to reduce traffic accidents. In recent years, smart cockpits have developed rapidly, among which the driver status monitoring system is a popular development direction. The system mainly monitors the driver's status in real time and reminds the driver in time when an abnormal state is detected to ensure the life and property safety of the driver and passengers. Blink detection is one of the important technologies of the driver status monitoring system. This technology provides basic information for driver status judgment by real-time monitoring of the occurrence and frequency of the driver's blinking action. Therefore, the accuracy of blink detection is one of the important indicators to measure the performance of the entire driver status monitoring system. In the existing detection technology, a method based on the change of eye area is used to detect whether the driver has blinked. However, this method is based on a simple judgment based on a fixed preset threshold. The judgment logic is relatively simple and crude, and it may be easy to misdetect. At the same time, it cannot make good adaptive adjustments for different human eyes, which may lead to poor detection of small eyes.

[0042] The embodiments of the present application provide a blink detection method, device and electronic device based on continuous frames. An eye state change curve is obtained according to the eye state value of each frame in a preset number of continuous frames, and whether a blink occurs is determined according to the eye state change curve. The method pays more attention to the process of blinking action from closing eyes to opening eyes, and can achieve more accurate detection of blinking actions of different drivers.

[0043] The method and device in this application are described below through specific embodiments.

[0044] See also Figure 1 , Figure 1 is a flowchart of a blink detection method based on continuous frames according to an exemplary embodiment of the present invention. Figure 1 , the method comprising:

[0045] Step S11, obtaining the eye state value of each frame in a preset number of consecutive frames through a single-frame eye state value algorithm;

[0046] Step S12, obtaining an eye state change curve according to the eye state value of each frame in a preset number of consecutive frames;

[0047] Step S13: determine whether to blink according to the eye state change curve.

[0048] It should be noted that the technical solution provided in this embodiment can be added to the existing vehicle system in the form of a small program in specific practice, or it can also be in the form of an independent application to provide an interface to the outside world to complete the blink detection function; applicable scenarios include but are not limited to: fatigue driving detection reminders.

[0049] Specifically, the preset number can be set according to the image processing capability of the vehicle computer. If the image processing capability of the vehicle computer is weak, the preset number will be reduced; if the image processing capability of the vehicle computer is strong, it will be set according to the detection accuracy requirement. The higher the detection accuracy, the larger the preset number.

[0050] It can be understood that the method provided in this embodiment obtains the eye state value of each frame in a preset number of consecutive frames through a single-frame eye state value algorithm, obtains an eye state change curve based on the eye state value of each frame in a preset number of consecutive frames, and determines whether to blink based on the eye state change curve; this method uses the eye state change curve to determine whether to blink, and focuses on the blinking process from closing eyes to opening eyes, which effectively reduces false detections caused by the size of the driver's eyes and improves the accuracy of blink detection and fatigue driving detection.

[0051] In specific practice, the "single-frame eye state value calculation" in step S11 includes: identifying the maximum eyelid distance between the upper and lower eyelids and the maximum eye corner distance between the left and right eye corners in a single-frame image; and obtaining the eye state value of the single-frame image based on the maximum eyelid distance and the maximum eye corner distance.

[0052] Specifically, the eye state value of a single frame image is obtained according to the maximum eyelid distance and the maximum eye canthus distance, including: obtaining the eye state value through a state value formula, the state value formula is: S=H×W; wherein S is the eye state value, H is the maximum eyelid distance, and W is the maximum eye canthus distance.

[0053] It should be noted that when continuously acquiring image data collected by the camera, the frame rate FPS will change dynamically with the actual processing time of a single frame image. As the processing time of a single frame increases, the image processing speed decreases. By extracting frames, one frame of image can be collected every certain number of frames. The processing time is reduced and the image processing speed is accelerated, thus avoiding data accumulation due to unstable device performance.

[0054] Specifically, before recognizing a single-frame image, the single-frame image may be preprocessed for a face image, grayscale conversion and size transformation may be performed on the input image, and image data may be compressed.

[0055] In specific practice, the step S12 of "obtaining an eye state change curve based on the eye state value of each frame in a preset number of consecutive frames" includes: storing the eye state value of each frame in a preset number of consecutive frames into an eye state queue; the queue length of the eye state queue is a preset number; performing curve fitting on the elements in the eye state queue based on a polynomial function to obtain the eye state change curve; each time a real-time frame is obtained, the eye state value of the real-time frame is stored in the eye state queue and then curve fitting is performed to obtain a continuously updated eye state change curve.

[0056] It should be noted that based on all the values ​​in the eye state queue, the eye state transformation curve is obtained by curve fitting using a polynomial function. The relevant calculation formula is as follows: y = a n x n +a n-1 x n-1 +...+a 1 x+a 0 ; Where n is the preset number, that is, the number of elements in the eye state queue; a 0 ,a 1 ,……,a n As a constant, the eye state value in the eye state queue is substituted into the formula, and the value of all constants is solved to obtain the function of the fitting curve.

[0057] In specific practice, step S13 of "determining whether to blink based on the eye state change curve" includes: obtaining points with a curvature of 0 in the eye state change curve and arranging them in chronological order to obtain an extreme point queue; obtaining the values ​​of two consecutive extreme points in the extreme point queue in chronological order, if the value difference between the two extreme points is greater than a first preset value and the value of any one of the two extreme points is 0, the number of blinks is superimposed once.

[0058] It should be noted that, if the difference between the maximum value and the minimum value in the extreme point is set to m, the first preset value may be 60%-90% m.

[0059] In specific practice, the method also includes: if the values ​​of a preset number of consecutive extreme points in the extreme point queue are all less than a second preset value, the driver's status is determined to be fatigue driving, a fatigue driving reminder is issued in the car and a reminder message is sent to a reserved mobile phone number.

[0060] It should be noted that the preset number is generally set according to the duration, and the preset number can be set to 30%-50% of the number of continuous frames in 1 minute. The second preset value can be set to the minimum value of the extreme point minus 1.

[0061] In specific practice, the eye state value of each frame can also be adjusted based on the default standard face eyes fitted by big data; the relative proportion R of the driver's standard eye state S (eyes open state) relative to the preset threshold S0 is calculated and saved as the driver's unique attribute, and the preset threshold S0 is the relative proportion R of the default standard face eyes (eyes open state) fitted by big data. The calculation formula is as follows: R=S÷S0; after calculating the eye state value of a single frame, the final eye state value of the single frame is obtained by adjusting it through the formula Sn=S÷R, where Sn is the final eye state value.

[0062] See also Figure 2 , Figure 2 is a schematic block diagram of a blink detection device based on continuous frames according to an exemplary embodiment of the present invention. Figure 2 , the eye blink detection device 100 based on continuous frames includes:

[0063] A state value module 101 is used to obtain an eye state value of each frame in a preset number of consecutive frames by using a single-frame eye state value algorithm;

[0064] A change curve module 102, for obtaining an eye state change curve according to the eye state value of each frame in a preset number of consecutive frames;

[0065] The blink determination module 103 is used to determine whether a blink occurs according to the eye state change curve.

[0066] It should be noted that the technical solution provided in this embodiment can be applied in specific practice to scenarios including but not limited to: fatigue driving detection reminder.

[0067] Specifically, the preset number can be set according to the image processing capability of the vehicle computer. If the image processing capability of the vehicle computer is weak, the preset number will be reduced; if the image processing capability of the vehicle computer is strong, it will be set according to the detection accuracy requirement. The higher the detection accuracy, the larger the preset number.

[0068] It can be understood that the device provided in this embodiment obtains the eye state value of each frame in a preset number of consecutive frames through a single-frame eye state value algorithm, obtains an eye state change curve based on the eye state value of each frame in a preset number of consecutive frames, and determines whether to blink based on the eye state change curve; this method determines whether to blink through the eye state change curve, focusing on the blinking process from closing eyes to opening eyes, effectively reducing false detections caused by the size of the driver's eyes, and improving the accuracy of blink detection and fatigue driving detection.

[0069] See also Figure 3 , Figure 3 is a schematic block diagram of an electronic device according to an exemplary embodiment of the present invention, see Figure 3 , an electronic device 200, comprising:

[0070] at least one processor 202; and

[0071] A memory 201 is communicatively connected to at least one processor 202; wherein,

[0072] The memory 201 stores instructions that can be executed by at least one processor 202 . The instructions are executed by the at least one processor 202 so that the at least one processor 202 can perform any of the above-mentioned methods for blink detection based on continuous frames.

[0073] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

[0074] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0075] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0076] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A blink detection method based on continuous frames, characterized in that: The method comprises: Obtaining the eye state value of each frame in a preset number of consecutive frames through a single-frame eye state value algorithm; Obtaining an eye state change curve according to the eye state value of each frame in the preset number of consecutive frames; Determine whether to blink according to the eye state change curve.

2. The detection method according to claim 1, characterized in that: The step of obtaining the eye state change curve according to the eye state value of each frame in the preset number of consecutive frames includes: storing the eye state value of each frame in the preset number of consecutive frames into an eye state queue; the queue length of the eye state queue is the preset number; Performing curve fitting on the elements in the eye state queue based on a polynomial function to obtain the eye state change curve; Each time a real-time frame is acquired, the eye state value of the real-time frame is stored in the eye state queue and then curve fitting is performed to obtain the continuously updated eye state change curve.

3. The detection method according to claim 2, characterized in that: The determining whether to blink according to the eye state change curve includes: Obtaining points with a curvature of 0 in the eye state change curve and arranging them in chronological order to obtain an extreme point queue; The values ​​of two consecutive extreme value points in the extreme value point queue are obtained in chronological order. If the value difference between the two extreme value points is greater than a preset value and the value of any one of the two extreme value points is 0, the number of blinks is added once.

4. The detection method according to claim 1, characterized in that: The single-frame eye state value algorithm comprises: Identify the maximum eyelid distance between the upper and lower eyelids and the maximum eye canthus distance between the left and right eye corners in a single frame image; The eye state value of the single frame image is obtained according to the maximum eyelid distance and the maximum eye canthus distance.

5. The detection method according to claim 4, characterized in that: The step of obtaining the eye state value of the single-frame image according to the maximum eyelid distance and the maximum eye canthus distance includes: The eye status value is obtained by the status value formula, and the status value formula is: S = H × W; Among them, S is the eye state value, H is the maximum eyelid distance, and W is the maximum eye canthus distance.

6. The detection method according to claim 3, characterized in that: The method further comprises: If the values ​​of a preset number of consecutive extreme points in the extreme point queue are all less than a second preset value, the driver's status is determined to be fatigue driving, a fatigue driving reminder is issued in the car and a reminder message is sent to a reserved mobile phone number.

7. A blink detection device based on continuous frames, characterized in that: The device comprises: A state value module, used for obtaining the eye state value of each frame in a preset number of consecutive frames through a single-frame eye state value algorithm; A change curve module, used for obtaining an eye state change curve according to the eye state value of each frame in the preset number of continuous frames; The blink determination module is used to determine whether to blink according to the eye state change curve.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the blink detection method based on continuous frames as described in any one of claims 1-6.

Images