Yawn detection system

Abstract

To provide a yawn detection system for appropriately detecting a yawn of a driver.SOLUTION: A yawn detection system 1 includes an opening determination section 11, a mouth covering determination section 12, a thinner eye determination section 13, and a yawn detection section 14. The opening determination section 11 determines whether an opening state where a driver opens the mouth continues for equal to or more than a first time on a face image. The mouth covering determination section 12 determines whether a mouth covering state where the driver covers a part of the mouth or the whole mouth by hand continues equal to or more than a second time on the face image when it is determined that the opening state continues for equal to or more than the first time. The thinner eye determination section 13 determines whether the driver becomes in a thinner eye state with a thinner vertical width on the face image compared with a state before the driver opens the mouth when it is determined that the mouth covering state continues for equal to or more than a second time. The yawn detection section 14 detects a yawn of the driver when it is determined that the driver becomes in the thinner eye state.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a yawn detection system. [Background technology]

[0002] As a technical document related to a conventional yawn detection device, JP 2021-152729 A is known. JP 2021-152729 A discloses a technology for detecting a waveform of the driver's opening degree based on a time series of facial images of the driver generated within a predetermined time, detecting a waveform of the driver's eye opening degree, and determining whether the driver is yawning. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-152729 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when the driver yawns by covering his / her mouth with his / her hand, the driver's mouth is hidden, so there is a risk that the driver's yawn may not be detected appropriately.

[0005] An object of the present disclosure is to provide a yawn detection system that can appropriately detect a driver's yawning. [Means for solving the problem]

[0006] The yawn detection system disclosed herein is a yawn detection system that detects a yawn by a vehicle driver from a facial image of the driver, and includes an opening determination unit that determines whether the driver's open mouth state on the facial image has continued for a first hour or more, a mouth covering determination unit that determines whether a mouth covering state in which part or all of the mouth is covered by the driver's hand on the facial image has continued for a second hour or more, a narrow eye determination unit that determines whether the driver's eyes have narrowed in vertical width on the facial image compared to before the driver's open eye state on the facial image when it is determined that the driver's eyes have narrowed, and a yawn detection unit that detects the driver's yawn when it is determined that the driver has narrowed his / her eyes. Effect of the Invention

[0007] According to the present disclosure, it is possible to provide a yawn detection system that can appropriately detect when a driver is yawning. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram of a yawn detection system according to one embodiment. [Diagram 2] 11 is a flowchart showing a yawn detection process performed by the yawn detection system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0010] Fig. 1 is a block diagram of a yawn detection system according to an embodiment. As shown in Fig. 1, the yawn detection system 1 includes a driver monitor camera 2, a microphone (sound detection unit) 3, and an ECU [Electronic Control Unit] 10. The yawn detection system 1 detects a yawn by a driver of a vehicle from a facial image of the driver.

[0011] The driver monitor camera 2 is disposed, for example, in the vehicle cabin. For example, the driver monitor camera 2 is disposed in front of the driver's seat so as to face the driver. The driver monitor camera 2 captures an image of the driver's face. The driver monitor camera 2 transmits the image of the driver's face to the ECU 10.

[0012] The microphone 3 is disposed, for example, in the vehicle cabin. For example, the microphone 3 is disposed in front of the driver's seat so as to face the driver. The microphone 3 is a part of a voice recognition device that recognizes the driver's voice and executes destination setting for a navigation function, etc. The microphone 3 can detect the driver's sneeze or laughter. The microphone 3 transmits a signal of the detected sound to the ECU 10.

[0013] The ECU 10 is an electronic control unit having a central processing unit (CPU) and a storage unit such as a read only memory (ROM) or a random access memory (RAM). In the ECU 10, for example, various functions are realized by the CPU executing a program stored in the storage unit. The ECU 10 is provided in, for example, a vehicle.

[0014] The ECU 10 includes, as its functional components, an opening determination unit 11, a mouth hiding determination unit 12, a small eyes determination unit 13, and a yawn detection unit 14.

[0015] The mouth opening determination unit 11 determines whether the driver's mouth is open on the face image of the driver for a first period or more. The mouth opening determination unit 11 determines the mouth opening state of the driver from the face image of the driver by using a known image processing method such as edge extraction, noise removal, pattern matching, etc. or a machine learning model learned by deep learning, etc. The first period is a value determined in advance based on the duration of a typical yawn.

[0016] When it is determined that the driver's open mouth state has continued for a first time or more, the mouth hiding determination unit 12 determines whether or not the mouth hiding state in which the driver's mouth is partially or entirely covered by the driver's hand on the face image of the driver has continued for a second time or more. The mouth hiding determination unit 12 determines the driver's mouth hiding state from the face image of the driver by using a known image processing method or machine learning model. The second time is a value determined in advance based on the duration of covering the mouth with a hand in a typical yawn.

[0017] When it is determined that the driver's mouth-hiding state has continued for a second time or more, the fine-eyes determination unit 13 determines whether the driver has entered a narrow-eyed state in which the vertical width of the driver's eyes in the face image is narrower than before the driver entered the open-mouthed state. The fine-eyes determination unit 13 compares the face image of the driver before the open-mouthed state with the face image of the driver in the mouth-hiding state. The fine-eyes determination unit 13 determines that the driver has entered a narrow-eyed state when the vertical width of the driver's eyes in the mouth-hiding state is narrower than the vertical width of the driver's eyes before the open-mouthed state. The fine-eyes determination unit 13 determines the driver's narrow-eyed state from the face image of the driver by using a well-known image processing method or machine learning model.

[0018] The yawn detection unit 14 detects the driver's yawn when it is determined that the driver has narrowed his / her eyes. However, even if it is determined that the driver's mouth-hiding state has continued for the second time or more, the yawn detection unit 14 may not detect the driver's yawn when the driver's sneezing sound or laughing sound is detected. This is because, even if the driver's mouth-hiding state has continued for the second time or more, if the driver is sneezing with his / her mouth covered by his / her hand or laughing with his / her mouth covered by his / her hand, it is highly likely that the driver is not yawning.

[0019] In addition, even if it is determined that the driver's mouth-hiding state has continued for more than the second time, the yawn detection unit 14 may not detect the driver's yawn when the driver suddenly turns his face down to sneeze. This is because, even if the driver's mouth-hiding state has continued for more than the second time, if the driver is sneezing, there is a high possibility that the driver is not yawning. The yawn detection unit 14 detects the driver's sneezing from the face image of the driver by using a known image processing method or machine learning model.

[0020] Next, the processing of the ECU 10 will be described. FIG. 2 is a flowchart showing the yawn detection processing by the yawn detection system 1. The yawn detection processing is executed, for example, while the vehicle is running. As shown in FIG. 2, in step S1, the ECU 10 determines whether the open state has continued for a first time or more. If the ECU 10 determines that the open state has continued for the first time or more (step S1: YES), the ECU 10 proceeds to step S2. If the ECU 10 determines that the open state has not continued for the first time or more (step S1: NO), the ECU 10 proceeds to step S7.

[0021] In step S2, the ECU 10 determines whether the mouth-hiding state has continued for a second time or more. If the ECU 10 determines that the mouth-hiding state has continued for the second time or more (step S2: YES), the ECU 10 proceeds to step S3. If the ECU 10 determines that the mouth-hiding state has not continued for the second time or more (step S2: NO), the ECU 10 proceeds to step S7.

[0022] In step S3, the ECU 10 determines whether the detailed state has been entered. If the ECU 10 determines that the detailed state has been entered (step S3: YES), the ECU 10 proceeds to step S4. If the ECU 10 determines that the detailed state has not been entered (step S3: NO), the ECU 10 proceeds to step S7.

[0023] In step S4, the ECU 10 judges whether or not the driver's sneeze or laughter has been detected. If the ECU 10 judges that the driver's sneeze or laughter has not been detected (step S4: YES), the ECU 10 proceeds to step S5. If the ECU 10 judges that the driver's sneeze or laughter has been detected (step S4: NO), the ECU 10 proceeds to step S7.

[0024] In step S5, the ECU 10 judges whether or not the driver's sneeze has been detected. If the ECU 10 judges that the driver's sneeze has not been detected (step S5: YES), the ECU 10 proceeds to step S6. If the ECU 10 judges that the driver's sneeze has been detected (step S5: NO), the ECU 10 proceeds to step S7.

[0025] The ECU 10 detects the driver's yawning in step S6. That is, the ECU 10 determines that the driver is yawning in step S6. When the ECU 10 detects the driver's yawning, the ECU 10 may transmit yawning detection information to a driving assistance system, for example. When the ECU 10 detects the driver's yawning frequently, the ECU 10 may alert the driver via a speaker or a display, for example. The ECU 10 does not detect the driver's yawning in step S7. That is, the ECU 10 does not determine that the driver is yawning in step S7.

[0026] As described above, according to the yawn detection system 1, when the driver's mouth is open for a first period of time or more and the mouth is covered for a second period of time or more, and when it is determined that the driver's eyes have narrowed in vertical width in the facial image compared to before the mouth was open, the system detects a yawn by the driver, thereby making it possible to properly detect yawning in which the mouth is covered with the hand.

[0027] Even if it is determined that the driver's mouth-hiding state has continued for the second time or more, the yawn detection unit 14 does not detect the driver's yawning when the driver's sneezing sound or laughter is detected. By not detecting the driver as yawning when the driver's sneezing sound or laughter is detected, it is possible to avoid erroneously detecting a yawn when the driver opens his / her mouth to sneeze or laugh and covers his / her mouth with his / her hand.

[0028] Even if it is determined that the mouth-hiding state has continued for more than the second time, the yawning detection unit 14 does not detect that the driver is yawning when it detects that the driver suddenly turns his face down to sneeze.Even if it is determined that the mouth-hiding state has continued for more than the second time, the yawning detection unit 14 does not detect that the driver is yawning when it detects that the driver suddenly turns his face down to sneeze, thereby making it possible to avoid erroneously detecting that the driver is yawning when he opens his mouth to sneeze and covers his mouth with his hand.

[0029] Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments. The present disclosure can be implemented in various forms including the above-described embodiments and various modifications and improvements based on the knowledge of those skilled in the art.

[0030] In the flowchart shown in FIG. 2, the ECU 10 may execute step S5 before step S4. The ECU 10 may execute step S4 and step S5 before step S3. The ECU 10 may not execute step S4, and may not execute step S5. The ECU 10 may detect the driver's yawning even if a sneezing sound or laughter of the driver is detected. The ECU 10 may detect the driver's yawning even if a sneezing action of the driver is detected.

[0031] The yawn detection unit 14 may increase the accuracy of determining a sneezing action based on the detection result of the temperature change in the vehicle cabin. For example, when the temperature in the vehicle cabin drops suddenly, the driver is more likely to sneeze, so the yawn detection unit 14 may increase the probability of determining a sneezing action as the temperature in the vehicle cabin drops. When the temperature drop in the vehicle cabin within a certain period of time is less than the drop determination threshold, the yawn detection unit 14 may determine a sneezing action even if the angle at which the face is lowered is smaller as the temperature in the vehicle cabin drops, compared to when the drop in temperature is not less than the drop determination threshold. The yawn detection unit 14 may increase the accuracy of determining a sneezing action based on the detection result of pollen, dust, etc. in the vehicle cabin. For example, when a large amount of pollen, dust, etc. is detected, the driver is more likely to sneeze, so the yawn detection unit 14 may increase the probability of determining a sneezing action as the amount of pollen, dust, etc. increases. [Explanation of symbols]

[0032] 1...yawn detection system, 11...open mouth determination unit, 12...mouth hiding determination unit, 13...small eyes determination unit, 14...yawn detection unit.

Claims

[Claim 1] A yawn detection system for detecting a yawn of a vehicle driver from a facial image of the driver, comprising: an opening determination unit that determines whether or not an open-mouth state of the driver on the face image continues for a first period or more; a mouth hiding determination unit that, when it is determined that the open mouth state has continued for the first time or more, determines whether a mouth hiding state in which a part of the mouth or the whole of the mouth is hidden by the driver's hand on the face image has continued for a second time or more; a narrow-eye determination unit that, when it is determined that the mouth-hiding state has continued for the second time period or longer, determines whether the driver has entered a narrow-eye state in which the vertical width of the driver's eyes is narrower in the face image than before the mouth-hiding state; a yawn detection unit that detects a yawn by the driver when it is determined that the driver is in the narrow-eyed state; A yawn detection system comprising:

Images