Warning device, warning method, and computer program

Abstract

The present invention provides a warning device, a warning method, and a computer program that can output a predetermined warning before the driver experiences a level of drowsiness exceeding a predetermined threshold. [Solution] A warning device 10 comprising a calculation unit 11 that outputs a warning to driver D, wherein the calculation unit acquires monitoring information that monitors the driver while driving the vehicle 1, recognizes predetermined actions to resist the driver's drowsiness based on the monitoring information, and outputs the warning from the output unit 3 if two or more of the predetermined actions exist within a predetermined time.

Description

Technical Field

[0001] The present invention relates to a warning device, a warning method, and a computer program for outputting a warning to a driver of a vehicle.

Background Art

[0002] Patent Document 1 discloses a drowsy driving prevention device that outputs a warning sound when it is determined that the user is drowsy driving. This prevention device determines the first stage, which is a precursor behavior of drowsiness, based on the recognition content of the driver's face and head while the vehicle is moving forward, and when determining the presence or absence of the second stage, which is the completed execution behavior of drowsiness, if it is determined that there is an execution behavior of drowsiness, it is configured to output a warning sound.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] According to the technique described in Patent Document 1, the first stage, which is a precursor behavior of the driver's drowsiness, is detected based on the state of the driver's head such as yawning, nodding (head movement), and blinking. Among the precursor behaviors of drowsiness, it is known that the nodding behavior appears when a person feels strong drowsiness. According to the technique described in Patent Document 1, when the driver is wearing a wearable such as sunglasses or a mask, the information of yawning and blinking is not recognized by the wearable, and the first stage, which is a precursor behavior of the driver's drowsiness, is determined based on the nodding behavior, so there is a possibility that a warning will be output after the driver feels strong drowsiness.

[0005] The present invention aims to provide a warning device, a warning method, and a computer program that can output a predetermined warning before the driver experiences a level of drowsiness exceeding a predetermined threshold. [Means for solving the problem]

[0006] One aspect of the present invention is a warning device comprising a calculation unit that outputs a warning to a driver, wherein the calculation unit acquires monitoring information of the driver while driving the vehicle, recognizes predetermined actions to resist the driver's drowsiness based on the monitoring information, and outputs a warning from the output unit if two or more of the predetermined actions exist within a predetermined time. [Effects of the Invention]

[0007] According to the present invention, a predetermined warning can be issued before the driver experiences a level of drowsiness exceeding a predetermined threshold. [Brief explanation of the drawing]

[0008] [Figure 1] Block diagram showing the configuration of a vehicle according to the embodiment. [Figure 2] This is a top view showing how to monitor the driver's condition. [Figure 3] This diagram classifies the content of prescribed behaviors and microsleep behaviors. [Figure 4] This is a flowchart showing the processing flow of the warning method executed in the warning device. [Modes for carrying out the invention]

[0009] As shown in Figure 1, the vehicle 1 is equipped with a detection unit 2 for monitoring the driver inside the vehicle. The detection unit 2 includes, for example, an in-vehicle camera 2A capable of capturing images of the driver. The in-vehicle camera 2A is positioned to capture images of the driver's face and actions. The in-vehicle camera 2A generates captured images of the driver as monitoring information. The detection unit 2 is equipped with a microphone 2B capable of recording audio inside the vehicle. The microphone 2B generates, for example, audio data spoken by the driver as monitoring information. The detection unit 2 may be configured as a functional part of a driver assistance device (not shown) that provides driving assistance in the vehicle 1.

[0010] Vehicle 1 is equipped with an output unit 3 capable of outputting information. The output unit 3 is equipped with, for example, a speaker 3A capable of outputting voice information. The speaker 3A outputs predetermined warnings to the driver based on voice, as described later. The output unit 3 is equipped with, for example, a display unit 3B capable of displaying information. The display unit 3B is composed of a display device such as a liquid crystal display. The display unit 3B may also be composed of a touch panel capable of inputting information. The output unit 3 may also be composed of an information processing device that provides various functions such as a navigation system or an audio system installed in Vehicle 1. In addition, the output unit 3 may be equipped with a device that stimulates the driver, such as a vibration generator (not shown) installed in the seat that provides vibration stimulation to the driver.

[0011] Vehicle 1 is equipped with a warning device 10 for outputting a predetermined warning to the driver. The warning device 10 is equipped with a calculation unit 11 that determines the driver's drowsiness based on monitoring information. The calculation unit 11 is composed of, for example, at least one hardware processor such as a CPU (Central Processing Unit). The calculation unit 11 may be implemented by hardware (including circuitry) such as an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or GPU (Graphics Processing Unit), or it may be implemented by the cooperation of software and hardware.

[0012] The calculation unit 11 is configured to perform machine learning, such as deep learning using training data, in advance, recognize the driver's state based on monitoring information acquired from the detection unit 2, and determine whether or not the driver is feeling drowsy. The calculation unit 11 recognizes the driver's state based on the image captured from the in-vehicle camera 2A. The calculation unit 11 recognizes the driver's state based on the audio data acquired from the microphone 2B. The processing of the calculation unit 11 will be described later.

[0013] The warning device 10 includes a storage unit 12 that stores data and computer programs necessary for calculations. The storage unit 12 is composed of a non-temporary storage medium such as a hard disk drive (HDD) or a solid-state disk (SSD). The computer program may be pre-stored in the storage unit 12, or it may be stored in a removable storage medium and installed in the storage unit 12 when the storage medium (non-temporary storage medium) is connected. The storage unit 12 constantly stores monitoring information. Unnecessary past monitoring information stored in the storage unit 12 is erased at a predetermined timing.

[0014] As shown in Figure 2, the in-vehicle camera 2A is positioned to capture images of the interior of vehicle 1, including the driver D, while the vehicle is in motion. The in-vehicle camera 2A captures images within an imaging range R that can capture the head region, including the face, and the body region of driver D. The in-vehicle camera 2A generates image data capturing the movement of driver D's head, the state of his face, and the state of his body, and outputs it to the warning device 10. The microphone 2B is positioned to input the voice spoken by driver D. The microphone 2B inputs the voice spoken by driver D and outputs it as audio data to the warning device 10.

[0015] Speaker 3A is positioned so that driver D can hear the audio information. There may be one or more speakers 3A in the vehicle. Speaker 3A may be shared with an audio device or other device that outputs sound. Speaker 3A outputs a warning based on the voice generated by the warning device 10, as described later, and makes it known to driver D. Display unit 3B is positioned so that driver D can see it. Display unit 3B outputs a warning based on the image generated by the warning device 10, as described later, and makes it known to driver D.

[0016] As shown in Figure 3, the behaviors of driver D when he feels drowsy are classified. Driver D is known to perform various microsleep behaviors as his drowsiness progresses. Microsleep behaviors include, for example, nodding while the body is relaxed, half-closed eyes, and fully closed eyes. Microsleep behaviors are in a state of advanced drowsiness and are difficult to detect if driver D is wearing headgear such as sunglasses.

[0017] s When the driver D driving the vehicle 1 feels drowsy, it is known that the driver performs a predetermined action to combat drowsiness at an initial stage before reaching the microsleep behavior. The predetermined action includes a first action in which the driver D touches the body including the head region. The first action is a contact action such as touching the facial region such as around the eyes, stimulating the head, or touching the body. The predetermined action includes a second action of stretching the body. The second action is a stretching action such as stretching the upper body or stretching the arm.

[0018] The predetermined action includes a third action in which the driver D yawns. The third action is a yawning action such as the driver D yawning or yawning including vocalization. The predetermined action includes a fourth action of blinking at a predetermined frequency or more. The fourth action is, for example, a rapid blinking action intentionally performed by the driver D at a predetermined frequency or more higher than normal blinking. The predetermined action includes a fifth action in which the driver D makes a vocalization at a predetermined level or more. The fifth action is, for example, a vocalization action of making a vocalization at a predetermined acoustic level or more to combat drowsiness. The predetermined action includes a sixth action in which the driver D pats the body. The predetermined action includes a seventh action in which the driver D loosens the legs.

[0019] The arithmetic unit 11 is configured to recognize various predetermined actions before reaching the microsleep behavior and output a warning to the driver D. The arithmetic unit 11 acquires monitoring information detected by the detection unit 2 for monitoring the driver D while the vehicle 1 is being driven. The arithmetic unit 11 determines whether there is a predetermined action for the driver to combat drowsiness based on the monitoring information. The arithmetic unit 11 analyzes the captured image acquired from the in-vehicle camera 2A and monitors the state of the driver D. The arithmetic unit 11 analyzes the voice information acquired from the microphone 2B and monitors the state of the driver D.

[0020] When the arithmetic unit 11 recognizes that the driver touches their body including the head area (the first action), it determines that the driver D is performing a predetermined action. When the arithmetic unit 11 recognizes that the driver stretches their body (the second action), it determines that the driver D is performing a predetermined action. When the arithmetic unit 11 recognizes that the driver D yawns (the third action), it determines that the driver D is performing a predetermined action. When the arithmetic unit 11 recognizes that the driver D blinks more than a predetermined frequency (the fourth action), it determines that the driver D is performing a predetermined action.

[0021] When the arithmetic unit 11 recognizes that the driver D makes a vocalization above a predetermined level (the fifth action), it determines that the driver D is performing a predetermined action. When the arithmetic unit 11 recognizes that the driver D pats their body (the sixth action), it determines that the driver D is performing a predetermined action. When the arithmetic unit 11 recognizes that the driver D loosens their legs (the seventh action), it determines that the driver D is performing a predetermined action.

[0022] When the arithmetic unit 11 determines that the driver D is performing two or more types of predetermined actions within a predetermined time, it causes a warning to be output from the output unit 3. For example, when the arithmetic unit 11 recognizes two or more types of predetermined actions within a predetermined time such as 10 minutes, it causes a warning to be output. The arithmetic unit 11 generates, for example, a warning based on voice information for alerting the driver D to drowsiness or voice information for prompting a break, and causes it to be output from the speaker 3A.

[0023] The arithmetic unit 11 generates a warning based on image information for alerting the driver D to drowsiness or image information for prompting a break, and causes it to be displayed on the display unit 3B. The arithmetic unit 11 may output the warning based on the voice information output from the speaker 3A and the warning based on the image information output from the display unit 3B in conjunction.

[0024] If the calculation unit 11 has issued a warning, it may, for example, wait a predetermined time, such as 30 minutes, before issuing the next warning. Based on the above process, it is possible to suppress the annoyance of excessive warnings being issued to the driver D. By having the calculation unit 11 recognize two or more predetermined actions, it is possible to prevent the driver D from immediately issuing a warning when the driver D simply touches their head instead of performing a predetermined action, thereby suppressing the annoyance the driver D may feel.

[0025] If the calculation unit 11 detects a microsleep behavior in the driver who is feeling drowsy, it determines that it is not a predetermined behavior. The calculation unit 11 determines that the driver is performing a microsleep behavior rather than a predetermined behavior if the driver nods, has half-closed eyes, or has closed eyes. If a microsleep behavior is detected, the driver D's drowsiness is progressing, and therefore, the calculation unit 1 may perform driving support control such as deceleration, stopping, or evasive maneuvers of the vehicle 1, which would ensure greater safety than outputting a warning.

[0026] Figure 4 shows the processing flow of the warning method executed by the warning device 10 to output a warning to the driver. The warning method is executed based on a computer program installed in the computer mounted on the warning device 10. The computer program causes the computer mounted on the warning device 10 to perform the following processes.

[0027] The calculation unit 11 of the warning device 10 acquires monitoring information from the detection unit 2 regarding the driver D while he is driving the vehicle 1 (S100). The calculation unit 11 analyzes the images captured by the in-vehicle camera 2A and the audio data from the microphone 2B to monitor the driver D's actions (S102). Based on the monitoring information, the calculation unit 11 determines whether the driver is performing a predetermined action to resist drowsiness (S104). If the calculation unit 11 determines that the driver D is performing one of the predetermined actions (S104: Yes), it determines whether a predetermined time has elapsed since the time the action was recognized (S106).

[0028] If a predetermined time has elapsed since the time the action was recognized (S106: Yes), the calculation unit 11 determines whether driver D is performing a predetermined action different from the predetermined action recognized last time (S108). If driver D is performing a predetermined action different from the predetermined action recognized last time (S108: Yes), that is, if there are two or more predetermined actions within the predetermined time, the calculation unit 11 outputs a warning from the output unit 3 (S110).

[0029] As described above, the warning device 10 can output a predetermined warning before the driver feels a certain level of drowsiness by recognizing predetermined actions taken by driver D to resist drowsiness while driving vehicle 1. The warning device 10 can output a predetermined warning even if driver D is wearing a device on their head by recognizing predetermined actions taken by driver D to resist drowsiness. The warning device 10 can output a predetermined warning when driver D is feeling mild drowsiness by recognizing predetermined actions taken by driver D before they perform microsleep behaviors that cause them to feel strong drowsiness. The warning device 10 can suppress the driver D from finding the warning bothersome by recognizing two or more predetermined actions before outputting a warning.

[0030] In the embodiments described above, the computer programs executed in each configuration of the warning device 10 may be provided in the form of being recorded on a computer-readable portable recording medium such as semiconductor memory, a magnetic recording medium, or an optical recording medium. The computer program product including the computer programs according to the above embodiments may be stored on a storage medium or provided via a communication line. [Explanation of Symbols]

[0031] 1 Vehicle, 2 Detection unit, 2A In-vehicle camera, 2B Microphone, 3 Output unit, 3A Speaker, 3B Display unit, 10 Warning device, 11 Calculation unit, 12 Memory unit, D Driver, R Imaging range

Claims

1. A warning device equipped with a calculation unit that outputs a warning to the driver, The aforementioned arithmetic unit, Obtain monitoring information that monitors the driver while the vehicle is in operation, Based on the aforementioned monitoring information, the system recognizes a predetermined action taken by the driver to resist drowsiness. If two or more of the above-mentioned predetermined actions occur within a predetermined time, the warning is output from the output unit. Warning device.

2. The aforementioned arithmetic unit, If the driver performs a first action of touching their body, including the head area, or a second action of stretching their body, it is determined that the driver is performing the predetermined action. The warning device according to claim 1.

3. The aforementioned arithmetic unit, If the driver performs a third action, such as yawning, or a fourth action, such as blinking at a predetermined frequency or more, it is determined that the driver is performing the predetermined action. The warning device according to claim 2.

4. The aforementioned arithmetic unit, If the driver performs a fifth action, in which they vocalize at a predetermined level or higher, it is determined that the driver is performing the predetermined action. The warning device according to claim 3.

5. The aforementioned arithmetic unit, If the driver performs the sixth action of hitting their body or the seventh action of shaking their legs, it is determined that the driver is performing the predetermined action. The warning device according to claim 4.

6. The aforementioned arithmetic unit, If the system recognizes that the driver is exhibiting microsleep behavior when feeling drowsy, it determines that the driver has not performed the predetermined action. The warning device according to claim 1.

7. The aforementioned arithmetic unit, If the driver performs any of the following actions: nodding, half-closing their eyes, or closing their eyes, it is determined that this is a microsleep behavior. The warning device according to claim 6.

8. A warning method that outputs a warning to the driver, Obtain monitoring information that monitors the driver while the vehicle is in operation, Based on the aforementioned monitoring information, the system recognizes a predetermined action taken by the driver to resist drowsiness. If two or more of the aforementioned predetermined actions occur within a predetermined time, the computer executes a process that causes the warning to be output from the output unit. Warning method.

9. A computer program that causes a computer to execute a warning method that outputs a warning to the driver, Obtain monitoring information that monitors the driver while the vehicle is in operation, Based on the aforementioned monitoring information, the system recognizes a predetermined action taken by the driver to resist drowsiness. If two or more of the above-mentioned predetermined actions occur within a predetermined time, the output unit will output the warning, and the computer will execute the processing. Computer program.

Images