Human eye opening and closing detection device and human eye opening and closing detection method thereof

Abstract

The invention provides a human eye opening and closing detection device which comprises an image acquisition device and a processor. The image capturing device continuously captures a plurality of frames of face images of a user. The processor determines an eye region from the multi-frame face image, and calculates an eye opening average value based on the eye region. The processor continuously updates the open-eye threshold value in response to the open-eye average value. When the eye opening average value is smaller than the eye opening threshold value, the processor judges that the user is in an eye state.

Description

Technical Field

[0001] The present invention relates to a human eye opening and closing detection device and a human eye opening and closing detection method, and more particularly to a human eye opening and closing detection device and a human eye opening and closing detection method for continuously updating an eye opening threshold value. Background Art

[0002] The human eye opening and closing detection method has a very significant position in the automotive field and the medical field. In the driver monitoring system, the human eye opening and closing detection method is often needed to detect the driver's eye opening and closing state, and judge whether the driver is in a state of physiological fatigue based on the eye opening and closing state, and then issue a warning to maintain driving safety.

[0003] However, the human eye opening and closing detection method encounters many technical difficulties during the detection process. Since the size of the detected eyes varies with different people, different angles, different distances and other environmental factors, coupled with the interference of external factors such as whether glasses are worn, changes in light and dark in the picture, the traditional human eye opening and closing detection method cannot correctly judge the opening and closing state of the eyes. Therefore, a human eye opening and closing detection device and human eye opening and closing detection method that can accurately judge the opening and closing state of the human eye in any situation are needed to solve the above problems. Summary of the invention

[0004] The present invention proposes a human eye opening and closing detection device and a human eye opening and closing detection method, which continuously updates the eye opening threshold values ​​of the left eye and the right eye, so that a representative eye opening threshold value is used as a basis for judgment in the process of detecting the eye opening and closing state. In addition, by dynamically adjusting the eye opening threshold values ​​of the left eye and the right eye, it helps to greatly reduce the influence of the external environment. Regardless of whether the user's left and right eyes are the same size, whether the eyes are disturbed by the shadows of other things (such as hair), whether the user wears glasses, whether the lens has an elevation angle or a depression angle, and other individual factors and environmental factors, the human eye opening and closing detection device and the human eye opening and closing detection method proposed by the present invention can accurately judge the eye opening and closing state. Furthermore, the present invention determines whether the user is in a closed eye state by judging the opening and closing state of the left eye and the right eye, which helps to greatly reduce the probability of misjudgment and thus improve the user experience.

[0005] In view of this, the present invention proposes a human eye opening and closing detection device, comprising an image capture device and a processor. The image capture device continuously captures multiple frames of facial images of a user. The processor determines an eye region from the multiple frames of facial images, and calculates an eye opening average value based on the eye region. The processor continuously updates an eye opening threshold value in response to the eye opening average value. In response to the eye opening average value being less than the eye opening threshold value, the processor determines that the user is in an eye closing state.

[0006] According to an embodiment of the present invention, in response to the processor determining that an execution condition is satisfied, the processor calculates an eye opening value from the eye area of ​​each frame, and averages the eye opening values ​​of a number of frames at a first time to generate the eye opening average value. The eye opening average value generated at the first time is a first eye opening average value. The processor multiplies the first eye opening average value by a ratio to generate an initial eye opening threshold value, and uses the initial eye opening threshold value as the eye opening threshold value.

[0007] According to an embodiment of the present invention, after the processor uses the initial eye opening threshold as the eye opening threshold, the processor generates a real-time eye opening average value by averaging the eye opening values ​​of the number of frames at a second time to update the eye opening average value to a second eye opening average value. The eye opening threshold is the second eye opening average value multiplied by the ratio. The second eye opening average value is the average of the first eye opening average value and the real-time eye opening average value.

[0008] According to an embodiment of the present invention, the processor further averages a plurality of eye-closed values ​​within a given time to generate an eye-closed average value. Within the given time, the eye-opened value that is less than the eye-opened threshold value is the eye-closed value. The processor updates the eye-opened threshold value to a weighted average of the eye-opened average value and the eye-closed average value.

[0009] According to one embodiment of the present invention, the eye-opening threshold value is the eye-opening average multiplied by a first weighting factor and the eye-closed average multiplied by a second weighting factor. The sum of the first weighting factor and the second weighting factor is 1. In response to the processor operating at a first sensitivity, the first weighting factor is a first value. In response to the processor operating at a second sensitivity, the first weighting factor is a second value, wherein the first value and the second value are different. In response to the processor operating at the first sensitivity, the ratio is a first ratio value. In response to the processor operating at the second sensitivity, the ratio is a second ratio value, wherein the first ratio value and the second ratio value are different.

[0010] According to an embodiment of the present invention, the processor further determines whether the second eye opening average value is less than the initial eye opening threshold value. In response to the second eye opening average value being less than the initial eye opening threshold value, the processor adjusts the eye opening average value to the average of the first eye opening average value and the second eye opening average value.

[0011] According to an embodiment of the present invention, in response to the second eye-opening average value being not less than the initial eye-opening threshold value, the processor further determines whether the eye-opening threshold value is greater than the first eye-opening average value. In response to the eye-opening threshold value being greater than the first eye-opening average value, the processor adjusts the eye-opening average value to the average of the first eye-opening average value and the second eye-opening average value. In response to the eye-opening threshold value being not greater than the first eye-opening average value, the processor continuously updates the eye-opening average value in real time. The eye-opening threshold value changes with the eye-opening average value.

[0012] According to an embodiment of the present invention, the eye opening detection device further comprises: a detection device and a memory. The detection device detects a moving speed. The memory temporarily stores the eye opening average value, the first eye opening average value, the initial eye opening threshold value, and the eye opening threshold value. In response to the moving speed exceeding a predetermined speed, the processor determines that the execution condition is satisfied. In response to the execution condition not being satisfied, the processor clears the eye opening average value, the first eye opening average value, the initial eye opening threshold value, and the eye opening threshold value in the memory.

[0013] According to an embodiment of the present invention, the processor further determines whether the eye region is in an open eye state based on morphology. In response to the eye region being in the open eye state and the moving speed exceeding the predetermined speed, the processor determines that the execution condition is satisfied.

[0014] According to an embodiment of the present invention, the processor further determines a left eye region and a right eye region from the eye region, and generates the eye opening average value and the eye opening threshold value corresponding to the left eye region and the right eye region respectively based on the left eye region and the right eye region. In response to the processor determining that the eye opening average values ​​of the left eye region and the right eye region are both less than the corresponding eye opening threshold value, the processor determines that the user is in the eye-closing state.

[0015] The present invention further proposes a method for detecting eye opening and closing, comprising: continuously capturing multiple frames of facial images of a user; determining an eye area from the multiple frames of facial images, and calculating an eye opening average value based on the eye area; updating an eye opening threshold value in real time in response to the eye opening average value; and determining that the user is in an eye closing state in response to the eye opening average value being less than the eye opening threshold value.

[0016] According to one embodiment of the present invention, the above-mentioned human eye opening and closing detection method further includes: determining whether an execution condition is met; in response to determining that the above-mentioned execution condition is met, calculating an eye opening value from the above-mentioned eye area of ​​each frame, and averaging the above-mentioned eye opening values ​​of a number of frames at a first time to generate the above-mentioned eye opening average value; wherein the above-mentioned eye opening average value generated at the above-mentioned first time is a first eye opening average value; and multiplying the above-mentioned first eye opening average value by a ratio to generate an initial eye opening threshold value, and using the initial eye opening threshold value as the above-mentioned eye opening threshold value.

[0017] According to an embodiment of the present invention, the eye opening detection method further comprises: after the step of multiplying the first eye opening average value by the ratio to generate the initial eye opening threshold value, at a second time, using the average of the eye opening values ​​of the number of frames to generate a real-time eye opening average value to update the eye opening average value to a second eye opening average value. The eye opening threshold value is the second eye opening average value multiplied by the ratio. The second eye opening average value is the average of the first eye opening average value and the real-time eye opening average value.

[0018] According to one embodiment of the present invention, the above-mentioned human eye opening and closing detection method further includes: averaging multiple eye closing values ​​within a given time to generate an eye closing average value; wherein within the above-mentioned given time, the above-mentioned eye opening value that is less than the above-mentioned eye opening threshold value is the above-mentioned eye closing value; and updating the above-mentioned eye opening threshold value to a weighted average of the above-mentioned eye opening average value and the above-mentioned eye closing average value.

[0019] According to one embodiment of the present invention, the eye-opening threshold value is the eye-opening average multiplied by a first weighting factor and the eye-closing average multiplied by a second weighting factor, wherein the sum of the first weighting factor and the second weighting factor is 1. In response to the human eye opening and closing detection device operating at a first sensitivity, the first weighting factor is a first value. In response to the human eye opening and closing detection device operating at a second sensitivity, the first weighting factor is a second value, wherein the first value and the second value are different. In response to the processor operating at the first sensitivity, the ratio is a first ratio value. In response to the processor operating at the second sensitivity, the ratio is a second ratio value, wherein the first ratio value and the second ratio value are different.

[0020] According to one embodiment of the present invention, the above-mentioned human eye opening detection method further includes: determining whether the above-mentioned second eye opening average value is smaller than the above-mentioned initial eye opening threshold value; and in response to the above-mentioned second eye opening average value being smaller than the above-mentioned initial eye opening threshold value, adjusting the above-mentioned eye opening average value to the average of the above-mentioned first eye opening average value and the above-mentioned second eye opening average value.

[0021] According to an embodiment of the present invention, the human eye opening detection method further comprises: in response to the second eye opening average value being not less than the initial eye opening threshold value, further determining whether the eye opening threshold value is greater than the first eye opening average value; in response to the eye opening threshold value being greater than the first eye opening average value, adjusting the eye opening average value to the average of the first eye opening average value and the second eye opening average value; and in response to the eye opening threshold value being not greater than the first eye opening average value, the processor continuously updating the eye opening average value in real time. The eye opening threshold value changes with the eye opening average value.

[0022] According to one embodiment of the present invention, the above-mentioned human eye opening and closing detection method further includes: using a detection device to detect a moving speed to generate a moving signal; using a memory to temporarily store the above-mentioned eye opening average value, the above-mentioned first eye opening average value, the above-mentioned initial eye opening threshold value and the above-mentioned eye opening threshold value; in response to the above-mentioned moving speed exceeding a predetermined speed, determining that the above-mentioned execution condition is satisfied; and in response to the above-mentioned execution condition not being satisfied, clearing the temporarily stored above-mentioned eye opening average value, the above-mentioned first eye opening average value, the above-mentioned initial eye opening threshold value and the above-mentioned eye opening threshold value.

[0023] According to one embodiment of the present invention, the above-mentioned human eye opening and closing detection method further includes: judging whether the above-mentioned eye area is in an open eye state based on morphology; and when the above-mentioned eye area is in the above-mentioned open eye state and the above-mentioned moving speed exceeds the above-mentioned predetermined speed, judging that the above-mentioned execution condition is satisfied.

[0024] According to one embodiment of the present invention, the above-mentioned human eye opening and closing detection device further includes: determining a left eye area and a right eye area from the above-mentioned eye area; generating the above-mentioned eye opening average value and the above-mentioned eye opening threshold value corresponding to the above-mentioned left eye area and the above-mentioned right eye area respectively based on the above-mentioned left eye area and the above-mentioned right eye area; and in response to determining that the above-mentioned eye opening average values ​​corresponding to the above-mentioned left eye area and the above-mentioned right eye area are both smaller than the corresponding above-mentioned eye opening threshold values, determining that the above-mentioned user is in the above-mentioned eye closing state. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a block diagram showing a human eye opening and closing detection device according to an embodiment of the present invention;

[0026] Figure 2 is a schematic diagram showing an image signal according to an embodiment of the present invention;

[0027] Figure 3A is a flow chart showing the analysis process of an image signal according to an embodiment of the present invention;

[0028] Figure 3Bis a schematic diagram showing an analysis process of an image signal according to an embodiment of the present invention;

[0029] Figure 4 is a flow chart showing a method for detecting eye opening and closing according to an embodiment of the present invention;

[0030] Figure 5 is a flow chart showing an update procedure according to an embodiment of the present invention;

[0031] Figure 6 is a schematic diagram showing eye opening values ​​according to an embodiment of the present invention;

[0032] Figure 7 is a schematic diagram showing an average eye opening value and an eye opening threshold value according to an embodiment of the present invention;

[0033] Figure 8 is a schematic diagram showing an average eye opening value and an eye opening threshold value according to another embodiment of the present invention; and

[0034] Fig. 9 is a flow chart showing a determination procedure according to an embodiment of the present invention.

[0035] The reference numerals are described as follows:

[0036] 10: User

[0037] 100: Human eye opening and closing detection device

[0038] 110: Image capture device

[0039] 120: Processor

[0040] 130: Memory

[0041] 140: Detection Device

[0042] 150: Communication interface

[0043] 210: Face area

[0044] 220: Eye area

[0045] 221: Left eye area

[0046] 222: Right eye area

[0047] 321: First Schematic

[0048] 322: Second Schematic Diagram

[0049] 323: The third schematic diagram

[0050] 324: Fourth Schematic Diagram

[0051] 331: First largest profile

[0052] 332: Second largest profile

[0053] 400: Human eye opening and closing detection method

[0054] 500: Update Program

[0055] 600: Waveform

[0056] 900: Judgment procedure

[0057] SV: Speed ​​signal

[0058] OP: Eye opening value

[0059] OP1: First eye opening value

[0060] OP2: Second eye opening value

[0061] AV1: Average value of the first eye opening

[0062] AV2: Second eye opening average

[0063] AV3: Third eye opening average

[0064] AV4: Fourth eye average

[0065] AV5: Fifth eye opening average

[0066] AV6: Sixth average opening

[0067] AVOP: Average eye opening

[0068] AVCL: Average value with closed eyes

[0069] AVCL1: First closed eye average

[0070] T1: First time

[0071] T2: Second time

[0072] T3: Third Time

[0073] T4: The Fourth Time

[0074] TH: Eye opening threshold

[0075] TH1: First eye opening threshold

[0076] TH2: Second eye opening threshold

[0077] TD: Determined Time

[0078] IAVOP: Initial average eye opening

[0079] ITH: Initial eye opening threshold

[0080] S311~S315,S410~S460,S501~S508,S910~S920: Step flow DETAILED DESCRIPTION

[0081] The following description is an embodiment of the present disclosure, which is intended to illustrate the general principles of the present disclosure and should not be regarded as limiting the present disclosure, the scope of which shall be determined by the claims.

[0082] It is worth noting that the content disclosed below can provide multiple embodiments or examples for practicing different features of the present disclosure. The special component examples and arrangements described below are only used to briefly and concisely explain the spirit of the present disclosure and are not used to limit the scope of the present disclosure. In addition, the following description may reuse the same component symbols or words in multiple examples. However, the purpose of reusing is only to provide a simplified and clear description, and is not used to limit the relationship between the multiple embodiments and / or configurations discussed below.

[0083] In addition, the description of a feature being connected to, coupled to and / or formed on another feature described in the following specification may actually include multiple different embodiments, including that the features are directly in contact, or that other additional features are formed between the features, etc., so that the features are not in direct contact.

[0084] It is understood that, although the terms "first", "second", "third", etc. may be used herein to describe various elements, components, regions, layers, and / or parts, these elements, components, regions, layers, and / or parts should not be limited by these terms, and these terms are only used to distinguish different elements, components, regions, layers, and / or parts. Therefore, a first element, component, region, layer, and / or part discussed below may be referred to as a second element, component, region, layer, and / or part without departing from the teachings of some embodiments of the present disclosure.

[0085] Some embodiments of the present disclosure can be understood in conjunction with the drawings, and the drawings of the embodiments of the present disclosure are also considered as part of the description of the embodiments of the present disclosure. It should be understood that the drawings of the embodiments of the present disclosure are not drawn in proportion to the actual devices and components. The shapes and thicknesses of the embodiments may be exaggerated in the drawings to clearly show the features of the embodiments of the present disclosure. In addition, the structures and devices in the drawings are drawn in a schematic manner to clearly show the features of the embodiments of the present disclosure.

[0086] In some embodiments of the present disclosure, terms such as "connection" and "interconnection" may refer to two structures being in direct contact, or two structures not being in direct contact, with another structure disposed between the two structures, unless otherwise specified. Such terms may also include situations where both structures are movable or both structures are fixed.

[0087] In the drawings, similar components and / or features may have the same reference numerals. Various components of the same type may be distinguished by adding letters or numbers after the reference numerals to distinguish similar components and / or similar features.

[0088] Figure 1 FIG. 2 is a block diagram showing an eye opening and closing detection device according to an embodiment of the present invention. Figure 1 As shown, the eye opening detection device 100 includes an image capture device 110, a processor 120, and a memory 130. The image capture device 110 is used to continuously capture multiple frames of facial images of the user 10 to generate image signals. According to some embodiments of the present invention, the image capture device 110 can be a camera having a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) photosensitive element.

[0089] The processor 120 receives the image signal generated by the image capture device 110, analyzes the image signal, and then determines the eye opening and closing state of the user 10. The detailed eye opening and closing state determination process will be described in detail below. According to some embodiments of the present invention, the processor 120 can be implemented by an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other logic circuits, but the present invention is not limited thereto.

[0090] The memory 130 is used to temporarily store the image signal received by the processor 120, and temporarily store the data generated by the processor 120 in the process of determining the opening and closing state of the eyes of the user 10. According to some embodiments of the present invention, the memory 130 can be a static random access memory (SRAM), a dynamic random access memory (DRAM), a flash memory, an electrically-erasable programmable read-only memory (EEPROM), or any other machine-accessible memory.

[0091] Figure 2 2 is a schematic diagram showing an image signal according to an embodiment of the present invention. In an embodiment of the present invention, when the processor 120 receives the image signal generated by the image capture device 110, the processor 120 analyzes the image signal to determine the user's face region 210, and determines the eye region 220 based on the face region 210, wherein the eye region 220 is a region of interest (ROI).

[0092] It should be noted that the eye area 220 refers to the human eye area and the area around it, that is, it may include the frame, eyebrows, eyes, eyelids, etc. According to some embodiments of the present invention, the eye area 220 can be further divided into a left eye area 221 and a right eye area 222, wherein the left eye area 221 and the right eye area 222 are regions of interest.

[0093] Figure 3A is a flow chart showing an analysis process of an image signal according to an embodiment of the present invention. Figure 3B is a schematic diagram showing the analysis process of the image signal according to an embodiment of the present invention. In step S311, the processor 120 captures Figure 2 The eye area 220 of the face area 210 is used as the region of interest. Figure 3B In the first schematic diagram 321, the processor 120 captures Figure 2 The right eye region 222 is used as the region of interest, wherein the eye in the right eye region 222 may be in an open eye state or a closed eye state. In order to simplify the description, the right eye region 222 is used as an example for explanation, and is not limited to this in any form.

[0094] In step S312, the processor 120 converts the region of interest into a black and white binary image. Figure 3BAs shown in the second schematic diagram 322, the processor 120 converts the image of the first schematic diagram 321 into a black and white binary image. In step S313, the processor 120 searches for the contour in the region of interest. Figure 3B As shown in the third schematic diagram 323 , the processor 120 depicts all contours in the second schematic diagram 322 . In other words, the processor 120 searches for all contours in the right eye region 222 .

[0095] Next, in step S314, the processor 120 obtains the area with the largest contour in the region of interest. Figure 3B As shown in the fourth schematic diagram 324 , the processor 120 obtains the area of ​​the first maximum contour 331 or the second maximum contour 332 , wherein the first maximum contour 331 and the second maximum contour 332 correspond to the eye in the right eye region 222 in the open eye state and the closed eye state, respectively.

[0096] Finally, in step S315, the processor 120 calculates the ratio of the area of ​​the largest contour in the region of interest. Figure 3B In an embodiment, the processor 120 calculates the ratio of the area of ​​the first maximum contour 331 to the right eye region 222, or the ratio of the second maximum contour 332 to the right eye region 222, wherein the ratio is hereinafter referred to as the eye opening value, and the ratio is less than 1.

[0097] Back to Figure 1 , the eye opening detection device 100 further includes a detection device 140 and a communication interface 150. The detection device 140 is used to detect the moving speed of the eye opening detection device 100 to generate a speed signal SV, and provide the speed signal SV to the processor 120 through the communication interface 150. According to an embodiment of the present invention, the processor 120 determines whether the execution condition is satisfied according to the moving speed of the speed signal SV and decides whether to control the image capture device 110 to start capturing the facial image of the user 10.

[0098] According to one embodiment of the present invention, the eye opening detection device 100 is mounted on a mobile vehicle, and the detection device 140 is used to detect the moving speed of the mobile vehicle. According to some embodiments of the present invention, the communication interface 150 can be a controller area network (CAN) or other communication interfaces for data transmission.

[0099] Figure 4 is a flow chart showing a method for detecting eye opening and closing according to an embodiment of the present invention. Figure 4 The description of the human eye opening detection method 400 will be combined with Figure 1-Figure 2 Eli explained in detail.

[0100] In step S410, Figure 1 The processor 120 determines whether the execution condition is satisfied. According to one embodiment of the present invention, the processor 120 determines whether the moving speed represented by the speed signal SV exceeds a predetermined speed based on the speed signal SV generated by the detection device 140, and then determines whether to continue to execute the subsequent steps of the human eye opening and closing detection method 400. When the moving speed does not exceed the predetermined speed, the processor 120 does not execute the subsequent steps of the human eye opening and closing detection method 400, and continues to determine whether the execution condition is satisfied.

[0101] According to some embodiments of the present invention, the predetermined speed may be set to be less than 15 kilometers per hour, but the present invention is not limited thereto. According to other embodiments of the present invention, the predetermined speed may be approximately 10 kilometers per hour.

[0102] According to another embodiment of the present invention, the processor 120 may first use the image capture device 110 to capture the face region 210 of the user 10, and use morphology to determine whether the eye region 220 of the user 10 is in an open eye state. When the eye region 220 is in an open eye state, the processor 120 determines that the execution condition is satisfied, and then executes the subsequent steps of the human eye opening and closing detection method 400. When the eye region 220 is in a closed eye state, the processor 120 determines that the execution condition is not satisfied, and continues to determine whether the eye region 220 of the user 10 is in an open eye state.

[0103] According to other embodiments of the present invention, the processor 120 can determine whether to execute the subsequent steps of the eye opening and closing detection method 400 according to whether the moving speed exceeds a predetermined speed and whether the user 10 is in an eye opening state by using morphology. When the moving speed exceeds the predetermined speed and the user 10 is in an eye opening state by using morphology, the processor 120 determines that the execution condition is satisfied and starts to execute the subsequent steps of the eye opening and closing detection method 400. Otherwise, the processor 120 continues to execute step S410.

[0104] When it is determined in step S410 that the execution condition is satisfied, the processor 120 uses the image capturing device 110 to continuously capture multiple frames of facial images of the user 10 (step S420), wherein the facial images include Figure 2 Next, the processor 120 determines the eye region 220 based on the face region 210 (step S430), wherein the processor 120 uses the eye region 220 as a region of interest (ROI).

[0105] According to some embodiments of the present invention, the processor 120 may further determine a left eye region 221 and a right eye region 222 from the eye region 220, and use the left eye region 221 and the right eye region 222 as regions of interest. When the processor 120 uses both the left eye region 221 and the right eye region 222 as regions of interest, the processor 120 may execute the human eye opening and closing detection method 400 for the left eye region 221 and the right eye region 222, respectively.

[0106] Then, the processor 120 calculates the eye opening value in the region of interest (step S440). Figure 3A As shown, the processor 120 calculates the ratio of the maximum contour area to the region of interest in the left eye region 221 and the right eye region 222, respectively, and uses them as the eye opening value of the left eye and the eye opening value of the right eye, respectively. Figure 3A-3B As shown, no further repetition is given here.

[0107] After step S440, the processor 120 simultaneously executes the update procedure (step S450) and the determination procedure (step S460). According to some embodiments of the present invention, when executing steps S420 to S460, the processor 120 simultaneously continuously determines whether the execution condition is satisfied. When it is determined that the execution condition is not satisfied, the processor 120 may interrupt the eye opening and closing detection method 400 at any time and return to step S410 to continuously determine whether the execution condition is satisfied again.

[0108] Figure 5 is a flow chart showing an update procedure according to an embodiment of the present invention. According to some embodiments of the present invention, Figure 5 The updating program 500 is used to update the eye opening threshold values ​​of the left eye area 221 and the right eye area 222. In other words, the left eye area 221 and the right eye area 222 have different eye opening threshold values. For the convenience of explanation, the updating program 500 will be explained by taking the right eye area 222 as an example, and is not limited thereto in any form.

[0109] First, the processor 120 calculates the average of the eye opening values ​​of the first number of frames as the initial eye opening average value (step S501). According to some embodiments of the present invention, the first number may be 10 to 20. In other words, when the human eye opening detection method 400 just starts to capture the facial image of the user 10, the processor 120 averages the eye opening values ​​of the right eye region 222 of 10 to 20 frames to generate the initial eye opening average value.

[0110] Next, the processor 120 multiplies the initial eye-opening average value by the ratio to generate an initial eye-opening threshold value (step S502), and uses the initial eye-opening threshold value as the eye-opening threshold value. According to one embodiment of the present invention, when the processor 120 determines that the eye-opening average value is less than the eye-opening threshold value at any time, the processor 120 determines that the user 10 is in the eye-closed state. In other words, the eye-opening threshold value is a standard for determining whether the user 10 is in the eye-opening state or the eye-closed state.

[0111] According to some embodiments of the present invention, when the processor 120 operates in a low sensitivity mode, the initial eye opening average value is multiplied by a first proportional value to generate an initial eye opening threshold value. When the processor 120 operates in a medium sensitivity mode, the initial eye opening average value is multiplied by a second proportional value to generate an initial eye opening threshold value. When the processor 120 operates in a high sensitivity mode, the initial eye opening average value is multiplied by a third proportional value to generate an initial eye opening threshold value. The third proportional value is greater than the second proportional value, the second proportional value is greater than the first proportional value, and the first proportional value, the second proportional value, and the third proportional value are all not greater than 1.

[0112] After generating the initial eye opening threshold value (i.e., step S502), the processor 120 continues to calculate the average of the eye opening values ​​of the first number of frames to update the eye opening average value (step S503). Figure 6 Eli explained in detail.

[0113] Figure 6 is a schematic diagram showing the eye opening value according to an embodiment of the present invention. Figure 6 As shown, the vertical axis of the waveform 600 is the eye opening value of the region of interest, and the horizontal axis is the frame number. As the image capture device 110 continuously captures multiple frames of facial images, the processor 120 calculates the eye opening value OP corresponding to different frames, and generates the eye opening average value AVOP and the eye closing average value AVCL according to the eye opening value OP.

[0114] like Figure 6 As shown, the processor 120 averages the eye opening values ​​OP of the first number of frames at the first time T1 to generate a first eye opening average value AV1, and averages the eye opening values ​​OP of the first number of frames at the second time T2 to generate a real-time eye opening average value AVR, and uses the first eye opening average value AV1 and the real-time eye opening average value AVR to update the second eye opening average value AV2.

[0115] According to an embodiment of the present invention, when the processor 120 generates the initial eye-opening average value at the first time T1, the processor 120 averages the eye-opening values ​​of the first number of frames before the first time T1 to generate the first eye-opening average value AV1, wherein the first eye-opening average value AV1 is the initial eye-opening average value. In addition, the processor 120 further multiplies the first eye-opening average value AV1 by a ratio to generate the first eye-opening threshold value TH1, wherein the first eye-opening threshold value TH1 is the initial eye-opening threshold value.

[0116] According to another embodiment of the present invention, the processor 120 calculates the second eye opening average value AV2 again at the second time T2 after the first time T1. The average of the eye opening values ​​OP of the first number of frames at the second time T2 is the real-time eye opening average value AVR, and the second eye opening average value AV2 is the average of the real-time eye opening average value AVR and the first eye opening average value AV1, as shown in Formula 1:

[0117]

[0118] In other words, when the first eye-opening average value AV1 is the initial eye-opening average value, the second eye-opening average value AV2 is the average of the initial eye-opening average value and the real-time eye-opening average value AVR. When the first eye-opening average value AV1 is not the initial eye-opening average value, the second eye-opening average value AV2 is still the average of the first eye-opening average value AV1 and the real-time eye-opening average value AVR.

[0119] According to an embodiment of the present invention, since the processor 120 averages the real-time average eye opening value with the previous average eye opening value to generate an updated average eye opening value, the phenomenon that the average eye opening value changes too much due to misjudgment can be suppressed.

[0120] Back to Figure 5 The processor 120 averages the multiple eye-closing values ​​within a predetermined time to generate an eye-closing average value (step S504). In addition, the processor 120 further updates the eye-opening threshold value to a weighted average of the eye-opening average value and the eye-closing average value (step S505).

[0121] like Figure 6As shown, since the eye-closing average value AVCL is not generated at the second time T2, the second eye-opening threshold value TH2 is the second eye-opening average value AV2 multiplied by a ratio. In addition, the processor 120 averages the first eye-opening value OP1 and the second eye-opening value OP2 that are less than the eye-opening threshold value TH within a predetermined time TD to generate the first eye-closing average value AVCL1. In other words, before the first eye-closing average value AVCL1 is generated, the eye-closing average value AVCL is zero, and the processor 120 updates the eye-closing average value AVCL using the average of the eye-opening values ​​OP that are less than the eye-opening threshold value TH within each predetermined time TD. According to some embodiments of the present invention, the predetermined time is approximately 10 to 15 seconds.

[0122] Next, when the first closed eye average value AVCL1 is generated, the processor 120 generates a new open eye threshold value TH by weighted average of the first closed eye average value AVCL1 and the corresponding open eye average value AVOP. According to some embodiments of the present invention, the open eye threshold value TH is the sum of the open eye average value AVOP multiplied by the first weighting factor W1 and the closed eye average value multiplied by the second weighting factor W2, wherein the sum of the first weighting factor W1 and the second weighting factor W2 is 1. The open eye threshold value TH is shown in Formula 2:

[0123] TH=AVOP×W1+AVCL1×W2 (Formula 2)

[0124] For example, when the processor 120 operates in the low sensitivity mode, the first weighting factor W1 is less than the second weighting factor W2. When the processor 120 operates in the medium sensitivity mode, the first weighting factor W1 may be equal to the second weighting factor W2. When the processor 120 operates in the high sensitivity mode, the first weighting factor W1 is greater than the second weighting factor W2.

[0125] In other words, when operating in the high sensitivity mode, the eye-opening threshold value TH is closer to the eye-opening average value AVOP, so that the processor 120 can more easily determine that the user 10 is in the eye-closing state. When operating in the low sensitivity mode, the eye-opening threshold value TH is closer to the eye-closing average value AVCL1, so that the processor 120 can less easily determine that the user 10 is in the eye-closing state, thereby avoiding misjudgment.

[0126] Back to Figure 5 After step S505, the processor 120 determines whether the updated eye opening average value is less than the initial eye opening threshold value (step S506). When the updated eye opening average value is less than the initial eye opening threshold value, the processor 120 adjusts the eye opening average value to the average of the eye opening average value and the initial eye opening average value (step S507). When the updated eye opening average value is not less than the initial eye opening threshold value, the processor 120 further determines whether the eye opening threshold value is greater than the initial eye opening average value (step S508).

[0127] When the eye opening threshold value is greater than the initial eye opening average value, the processor 120 adjusts the eye opening average value to the average of the eye opening average value and the initial eye opening average value (step S507). When the eye opening threshold value is not greater than the initial eye opening average value, the processor 120 does not adjust the eye opening average value and returns to step S503 to continue updating the eye opening average value. After step S507, the processor 120 also returns to step S503 to continue updating the eye opening average value. The detailed operation method of steps S506 to S508 will be combined with Figure 7 And described in detail below.

[0128] Figure 7 is a schematic diagram showing the average eye opening value and the eye opening threshold value according to an embodiment of the present invention. Figure 7 As shown, the processor 120 generates an initial eye opening average value IAVOP in step S501 and generates an initial eye opening threshold value ITH in step S502. In addition, the eye opening average value AVOP and the eye opening threshold value TH are gradually reduced from the initial eye opening average value IAVOP and the initial eye opening threshold value ITH as the number of frames increases, and the eye opening average value AVOP is less than the initial eye opening threshold value ITH at the third time T3.

[0129] When the processor 102 determines in step S506 that the eye-opening average value AVOP is less than the initial eye-opening threshold value ITH at the third time T3, the processor 120 updates the eye-opening average value AVOP to a fourth eye-opening average value AV4, wherein the fourth eye-opening average value AV4 is an average of the third eye-opening average value AV3 at the third time T3 and the initial eye-opening average value IAVOP, as shown in Formula 3. According to some embodiments of the present invention, when the processor 120 adjusts the eye-opening average value AVOP, since the eye-opening threshold value TH is a weighted average of the eye-opening average value AVOP and the eye-closing average value AVCL, the eye-opening threshold value TH will also be adjusted accordingly.

[0130]

[0131] Figure 8 2 is a schematic diagram showing the average eye opening value and the eye opening threshold value according to another embodiment of the present invention. Figure 8 As shown, the average eye opening value AVOP and the eye opening threshold value TH gradually increase from the initial eye opening value IAVOP and the initial eye opening threshold value ITH respectively as the number of frames increases, and the eye opening threshold value TH is greater than the initial eye opening value IAVOP at the fourth time T4.

[0132] When the processor 120 determines in step S508 that the eye opening threshold value TH is greater than the initial eye opening average value IAVOP at the fourth time T4, the processor 120 updates the eye opening average value AVOP to the fifth eye opening average value AV5, wherein the fifth eye opening average value AV5 is the average of the sixth eye opening average value AV6 at the fourth time T4 and the initial eye opening average value IAVOP, as shown in Formula 4. According to some embodiments of the present invention, when the processor 120 adjusts the eye opening average value AVOP, since the eye opening threshold value TH is the weighted average of the eye opening average value AVOP and the eye closing average value AVCL, the eye opening threshold value TH is also adjusted accordingly.

[0133]

[0134] According to some embodiments of the present invention, Figure 7 , Figure 8 As shown in the figure, since the average eye opening value AVOP may become too large or too small over time, the eye opening threshold value TH may become an unreasonable value. By using the initial eye opening average value IAVOP and the initial eye opening threshold value ITH as the upper and lower limits, and maintaining the eye opening average value AVOP and the eye opening threshold value TH within a range for variation, it is helpful to maintain the rationality of the eye opening average value AVOP and the eye opening threshold value TH, thereby improving the accuracy of the human eye opening detection method.

[0135] According to one embodiment of the present invention, Figure 1 The memory 130 is further used to temporarily store the eye-opening threshold value TH, the eye-opening average value AVOP, the eye-closing average value AVCL, the initial eye-opening average value IAVOP and the initial eye-opening threshold value ITH corresponding to the left eye region 221 and the right eye region 222 respectively.

[0136] Fig. 9 is a flow chart showing a determination procedure according to an embodiment of the present invention. Fig. 9 As shown in the judgment procedure 900, the processor 120 judges whether the average eye opening values ​​of the left eye region 221 and the right eye region 222 are both less than the corresponding eye opening threshold value (step S910). When the processor 120 judges that the average eye opening values ​​of the left eye region 221 and the right eye region 222 are both less than the corresponding eye opening threshold value, the processor 120 judges that the user 10 is in the eye-closing state (step S920).

[0137] In other words, the processor 120 determines that the user 10 is in the eye-closed state only when the processor 120 determines that both eyes of the user 10 are in the eye-closed state. When the processor 120 determines that the eye-opening average value of either the left eye region 221 or the right eye region 222 is not less than the corresponding eye-opening threshold value, the processor 120 continues to execute step S910.

[0138] According to some embodiments of the present invention, when the processor 120 determines in step S920 that the user 10 is in the eyes-closed state, the user 120 may issue an alarm to notify the user 10. According to some embodiments of the present invention, when the processor 120 executes step S910, the processor 120 simultaneously executes the update program 500 to update the eye-opening average value AVOP, the eye-closed average value AVCL, and the eye-opening threshold value TH in real time, and determines whether the user 10 is in the eyes-closed state according to the eye-opening average value AVOP and the eye-opening threshold value TH at the time of executing step S910.

[0139] According to other embodiments of the present invention, when the processor 120 does not detect the face area 210 for more than a predetermined time and then detects the face area 210 again, the processor 120 re-executes the human eye opening and closing detection method 400 and recalculates the eye opening average value AVOP and the eye opening threshold value TH. According to other embodiments of the present invention, when the human eye opening and closing detection method 400 is paused and then restarted, the eye opening average value AVOP and the eye opening threshold value TH are recalculated.

[0140] According to other embodiments of the present invention, when executing steps S420 to S460, steps S501 to S508, and steps S910 to S920, the processor 120 may simultaneously continue to determine whether the execution condition is satisfied. When it is determined that the execution condition is not satisfied, the processor 120 may interrupt the eye opening detection method 400, the updating program 500, and the determination program 900 at any time and clear the eye opening average value AVOP, the initial eye opening average value IAVOP, the initial eye opening threshold value ITH, and the eye opening threshold value TH of the memory 130, and return to step S410 to continue to determine whether the execution condition is satisfied again and re-execute the eye opening detection method 400.

[0141] A human eye opening and closing detection device and a human eye opening and closing detection method are proposed here. By continuously updating the eye opening threshold values ​​of the left eye and the right eye, a representative eye opening threshold value is provided as a basis for judgment in the process of detecting the eye opening and closing state. In addition, by dynamically adjusting the eye opening threshold values ​​of the left eye and the right eye, it helps to greatly reduce the influence of the external environment. Regardless of whether the user's left and right eyes are the same size, whether the eyes are disturbed by the shadows of other things (such as hair), whether the user wears glasses, whether the lens has an elevation angle or a depression angle, and other individual factors and environmental factors, the eye opening and closing state can be accurately judged. Furthermore, by judging the opening and closing state of the left eye and the right eye to determine whether the user is in a closed eye state, it helps to greatly reduce the probability of misjudgment, thereby improving the user experience.

[0142] Although the embodiments and advantages of the present disclosure have been disclosed as above, it should be understood that any person with ordinary knowledge in the technical field can make changes, substitutions and modifications without departing from the spirit and scope of the present disclosure. In addition, the scope of protection of the present disclosure is not limited to the processes, machines, manufacturing, material compositions, devices, methods and steps in the specific embodiments described in the specification. Any person with ordinary knowledge in the technical field can understand the current or future developed processes, machines, manufacturing, material compositions, devices, methods and steps from the disclosure of some embodiments of the present disclosure. As long as the same functions can be implemented in the embodiments described herein or the same results can be obtained, they can be used according to some embodiments of the present disclosure. Therefore, the scope of protection of the present disclosure includes the above-mentioned processes, machines, manufacturing, material compositions, devices, methods and steps. In addition, each claim constitutes an individual embodiment, and the scope of protection of the present disclosure also includes the combination of each claim and embodiment.

Claims

1. A human eye opening and closing detection device, comprising: an image capturing device for continuously capturing multiple frames of facial images of a user; and a processor for determining an eye region from the plurality of frames of facial images and calculating an eye-opening average value based on the eye region; The processor continuously updates an eye opening threshold value in response to the eye opening average value; In response to the eye-opening average being less than the eye-opening threshold, the processor determines that the user is in an eye-closed state.

2. The eye opening detection device of claim 1 , wherein in response to the processor determining that an execution condition is satisfied, the processor calculates an eye opening value from the eye region of each frame, and averages the eye opening values ​​of a number of frames at a first time to generate the eye opening average value; The eye-opening average value generated at the first time is a first eye-opening average value; The processor multiplies the first eye-opening average value by a ratio to generate an initial eye-opening threshold value, and uses the initial eye-opening threshold value as the eye-opening threshold value.

3. The eye opening detection device of claim 2, wherein after the processor uses the initial eye opening threshold value as the eye opening threshold value, the processor generates a real-time eye opening average value by averaging the eye opening values ​​of the number of frames at a second time to update the eye opening average value to a second eye opening average value; The eye opening threshold value is the second eye opening average multiplied by the ratio; The second eye opening average value is the average of the first eye opening average value and the real-time eye opening average value.

4. The eye opening and closing detection device as claimed in claim 2, wherein the processor further averages a plurality of eye closing values ​​within a given time to generate an eye closing average value; Wherein, within the predetermined time, the eye-opening value that is less than the eye-opening threshold value is the eye-closing value; The processor updates the eye-opening threshold value to a weighted average of the eye-opening average value and the eye-closing average value.

5. The eye opening and closing detection device as claimed in claim 4, wherein the eye opening threshold value is the eye opening average value multiplied by a first weighting factor and the eye closing average value multiplied by a second weighting factor; The sum of the first weighting factor and the second weighting factor is 1; wherein in response to the processor operating at a first sensitivity, the first weighting factor is a first value; wherein in response to the processor operating at a second sensitivity, the first weighting factor is a second value; wherein the first value and the second value are different; wherein in response to the processor operating at the first sensitivity, the ratio is a first ratio value; wherein in response to the processor operating at the second sensitivity, the ratio is a second ratio value; The first ratio value and the second ratio value are different.

6. The eye opening detection device as claimed in claim 3, wherein the processor further determines whether the second eye opening average value is less than the initial eye opening threshold value; In response to the second eye-opening average value being less than the initial eye-opening threshold value, the processor adjusts the eye-opening average value to an average of the first eye-opening average value and the second eye-opening average value.

7. The human eye opening detection device as claimed in claim 6, wherein in response to the second eye opening average value being not less than the initial eye opening threshold value, the processor further determines whether the eye opening threshold value is greater than the first eye opening average value; In response to the eye opening threshold being greater than the first eye opening average value, the processor adjusts the eye opening average value to an average of the first eye opening average value and the second eye opening average value; In response to the eye opening threshold being not greater than the first eye opening average value, the processor continuously updates the eye opening average value in real time; The eye opening threshold value changes with the eye opening average value.

8. The eye opening and closing detection device as claimed in claim 2, further comprising: a detection device for detecting a moving speed; as well as a memory for temporarily storing the eye opening average value, the first eye opening average value, the initial eye opening threshold value, and the eye opening threshold value; wherein in response to the moving speed exceeding a predetermined speed, the processor determines that the execution condition is satisfied; In response to the execution condition not being met, the processor clears the eye opening average value, the first eye opening average value, the initial eye opening threshold value and the eye opening threshold value from the memory.

9. The eye opening and closing detection device as claimed in claim 8, wherein the processor further determines whether the eye region is in an open eye state based on morphology; In response to the eye area being in the eye-opening state and the moving speed exceeding the predetermined speed, the processor determines that the execution condition is satisfied.

10. The eye opening detection device as claimed in claim 1, wherein the processor further determines a left eye region and a right eye region from the eye region, and generates the eye opening average value and the eye opening threshold value corresponding to the left eye region and the right eye region respectively based on the left eye region and the right eye region; In response to the processor determining that the average eye-opening values ​​of the left eye area and the right eye area are both smaller than the corresponding eye-opening threshold values, the processor determines that the user is in the eye-closing state.

11. A human eye opening and closing detection method, used in a human eye opening and closing detection device, wherein the human eye opening and closing detection device comprises an image capture device, wherein the human eye opening and closing detection method comprises: Utilizing the image capture device to continuously capture multiple frames of facial images of a user; Determine an eye region from the plurality of frames of facial images, and calculate an open eye average value based on the eye region; In response to the above-mentioned eye opening average value, an eye opening threshold value is updated in real time; as well as In response to the eye-opening average being less than the eye-opening threshold, it is determined that the user is in an eye-closed state.

12. The method for detecting eye opening and closing as claimed in claim 11, further comprising: Determine whether an execution condition is met; In response to determining that the execution condition is satisfied, an eye opening value is calculated from the eye region of each frame, and the eye opening values ​​of a number of frames are averaged at a first time to generate the eye opening average value; The eye-opening average value generated at the first time is a first eye-opening average value; and The first eye-opening average value is multiplied by a ratio to generate an initial eye-opening threshold value, and the initial eye-opening threshold value is used as the eye-opening threshold value.

13. The method for detecting eye opening and closing as claimed in claim 12, further comprising: After the step of multiplying the first eye-opening average value by the ratio to generate the initial eye-opening threshold value, at a second time, averaging the eye-opening values ​​of the number of frames to generate a real-time eye-opening average value to update the eye-opening average value to a second eye-opening average value; The eye opening threshold value is the second eye opening average multiplied by the ratio; The second eye opening average value is the average of the first eye opening average value and the real-time eye opening average value.

14. The method for detecting eye opening and closing as claimed in claim 12, further comprising: Averaging multiple eye-closing values ​​within a given period of time to generate an eye-closing average value; wherein within the predetermined time, the eye-opening value that is less than the eye-opening threshold value is the eye-closing value; and The eye-opening threshold value is updated to a weighted average of the eye-opening average value and the eye-closed average value.

15. The human eye opening and closing detection method as claimed in claim 14, wherein the eye opening threshold value is the eye opening average value multiplied by a first weighting factor and the eye closing average value multiplied by a second weighting factor; The sum of the first weighting factor and the second weighting factor is 1; In response to the human eye opening and closing detection device operating at a first sensitivity, the first weighting factor is a first value; In response to the human eye opening and closing detection device operating at a second sensitivity, the first weighting factor is a second value; wherein the first value and the second value are different; wherein in response to a processor operating at the first sensitivity, the ratio is a first ratio value; wherein in response to the processor operating at the second sensitivity, the ratio is a second ratio value; The first ratio value and the second ratio value are different.

16. The method for detecting eye opening and closing as claimed in claim 13, further comprising: Determine whether the second eye opening average value is less than the initial eye opening threshold value; as well as In response to the second eye-opening average value being less than the initial eye-opening threshold value, the eye-opening average value is adjusted to be an average of the first eye-opening average value and the second eye-opening average value.

17. The method for detecting eye opening and closing as claimed in claim 16, further comprising: In response to the second eye-opening average value being not less than the initial eye-opening threshold value, further determining whether the eye-opening threshold value is greater than the first eye-opening average value; In response to the eye-opening threshold being greater than the first eye-opening average value, adjusting the eye-opening average value to an average of the first eye-opening average value and the second eye-opening average value; and In response to the eye opening threshold being not greater than the first eye opening average value, a processor continuously updates the eye opening average value in real time; The eye opening threshold value changes with the eye opening average value.

18. The method for detecting eye opening and closing as claimed in claim 12, further comprising: Using a detection device to detect a moving speed; Using a memory to temporarily store the eye opening average value, the first eye opening average value, the initial eye opening threshold value, and the eye opening threshold value; In response to the moving speed exceeding a predetermined speed, determining that the execution condition is satisfied; and In response to the execution condition being not satisfied, the temporarily stored average eye opening value, the first average eye opening value, the initial eye opening threshold value, and the eye opening threshold value are cleared.

19. The method for detecting eye opening and closing as claimed in claim 18, further comprising: Determining whether the eye region is in an open eye state according to morphology; as well as In response to the eye region being in the eye-opening state and the moving speed exceeding the predetermined speed, it is determined that the execution condition is satisfied.

20. The method for detecting eye opening and closing as claimed in claim 11, further comprising: Determine a left eye region and a right eye region from the eye region; Based on the left eye area and the right eye area, generating the eye opening average value and the eye opening threshold value corresponding to the left eye area and the right eye area respectively; as well as In response to determining that the eye-opening average values ​​corresponding to the left eye area and the right eye area are both smaller than the corresponding eye-opening threshold values, it is determined that the user is in the eye-closing state.

Images