Warning device, warning method and computer program product

Abstract

Provided is a warning device, a warning method, and a computer program product, capable of outputting a predetermined warning before a driver feels strong drowsiness of a predetermined level or more. A warning device (10) outputs a warning to a driver (D) and includes an arithmetic unit (11) that acquires monitoring information obtained by monitoring the driver who is driving a vehicle (1), identifies predetermined actions of the driver against drowsiness based on the monitoring information, and outputs the warning from an output unit (3) when two or more of the predetermined actions are present within a predetermined time.

Description

Technical Field

[0001] This invention relates to a warning device, warning method, and computer program product for outputting warnings to the driver of a vehicle. Background Technology

[0002] Patent Document 1 discloses a drowsy driving prevention device that outputs a warning sound when it is determined that the user is drowsy driving. The device is configured to determine, while the vehicle is moving forward, a first stage of actions that are precursors to drowsy driving is determined based on the recognition of the driver's face and head. If a second stage of actions that constitutes completed drowsy driving is determined to be present, a warning sound is output.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Publication No. 2016-539446 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] According to the technology described in Patent Document 1, the first stage of detecting premonitory actions of a driver's drowsiness is based on the driver's head state, such as yawning, nodding (head movements), and blinking. It is known that nodding, a premonitory action of drowsiness, occurs when a person feels intensely sleepy. According to the technology described in Patent Document 1, if the driver is wearing sunglasses, a mask, or other protective clothing, the information from yawning and blinking will not be recognized due to the clothing. Therefore, the first stage of determining premonitory actions of drowsiness based on nodding may only issue a warning after the driver feels intensely sleepy.

[0008] The purpose of this invention is to provide a warning device, warning method, and computer program product that can output a predetermined warning before the driver feels a strong drowsiness above a predetermined level.

[0009] Technical solutions for solving the problem

[0010] One aspect of the present invention is a warning device that outputs a warning to a driver and includes a calculation unit. The calculation unit acquires monitoring information obtained by monitoring the driver who is driving the vehicle, identifies predetermined actions by the driver to resist drowsiness based on the monitoring information, and outputs the warning from the output unit if two or more of the predetermined actions occur within a predetermined time.

[0011] Invention Effects

[0012] According to the present invention, a predetermined warning can be output before the driver feels a strong drowsiness above a predetermined level. Attached Figure Description

[0013] Figure 1 This is a block diagram illustrating the configuration of the vehicle involved in the embodiment.

[0014] Figure 2 This is a top view illustrating a method for monitoring the driver's condition.

[0015] Figure 3 This diagram categorizes the content of scheduled activities and microsleep activities.

[0016] Figure 4 This is a flowchart illustrating the process of a warning method executed in a warning device.

[0017] Explanation of reference numerals in the attached figures

[0018] 1 Vehicle, 2 Detection unit, 2A In-vehicle camera, 2B Microphone, 3 Output unit, 3A Speaker, 3B Display unit, 10 Warning device, 11 Calculation unit, 12 Storage unit, D Driver, R Camera range. Detailed Implementation

[0019] like Figure 1 As shown, vehicle 1 includes a detection unit 2 for monitoring the driver inside the vehicle. The detection unit 2 includes, for example, an in-vehicle camera 2A capable of capturing images of the driver. The in-vehicle camera 2A is positioned to capture images of the driver's face and movements. The in-vehicle camera 2A generates images of the driver as monitoring information. The detection unit 2 also includes a microphone 2B capable of recording sounds inside the vehicle. The microphone 2B generates data of the driver's voice as monitoring information. The detection unit 2 may also be a functional part of a driver assistance device (not shown) that provides driving assistance in vehicle 1.

[0020] Vehicle 1 includes an output unit 3 capable of outputting information. The output unit 3 may include, for example, a speaker 3A capable of outputting sound information. The speaker 3A outputs a predetermined warning to the driver based on sound, as described later. The output unit 3 may also include a display unit 3B capable of displaying information. The display unit 3B may be a display device such as a liquid crystal display. The display unit 3B may also be a touch panel capable of inputting information. The output unit 3 may also be an information processing device provided with various functions, such as a navigation device or an audio device installed in vehicle 1. In addition, the output unit 3 may also include a vibration generating device (not shown) installed in the seat and providing vibration stimulation to the driver, or other devices that stimulate the driver.

[0021] Vehicle 1 is equipped with a warning device 10 for outputting predetermined warnings to the driver. The warning device 10 is equipped with a computing unit 11 for determining the driver's drowsiness based on monitoring information. The computing unit 11 is, for example, composed of at least one hardware processor such as a CPU (Central Processing Unit). The computing unit 11 can be implemented using hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit), or it can be implemented through a combination of software and hardware.

[0022] The computing unit 11 is configured, for example, to perform machine learning such as deep learning using supervised data in advance, and to identify the driver's state based on the monitoring information obtained from the detection unit 2, determining whether the driver is drowsy. The computing unit 11 identifies the driver's state based on images captured by the in-vehicle camera 2A. The computing unit 11 also identifies the driver's state based on sound data obtained from the microphone 2B. The processing of the computing unit 11 will be described later.

[0023] The warning device 10 includes a storage unit 12 for storing data and computer programs required for computation. The storage unit 12 is composed of a non-transitory storage medium such as a hard disk drive (HDD) or a solid-state drive (SSD). The computer program can be pre-stored in the storage unit 12 or stored on a removable storage medium and installed in the storage unit 12 by connecting to the storage medium (a non-transitory storage medium). The storage unit 12 continuously stores monitoring information. Past, unwanted monitoring information stored in the storage unit 12 is erased at predetermined intervals.

[0024] like Figure 2 As shown, the in-vehicle camera 2A is positioned to capture images inside the vehicle 1, including the driver D who is driving the vehicle 1. The in-vehicle camera 2A captures images within a shooting range R that captures the head area, including the driver D's face, and the body area. The in-vehicle camera 2A generates image data of the driver D's head movements, facial state, and body state, and outputs this data to the warning device 10. The microphone 2B is positioned to input the sound emitted by the driver D. The microphone 2B inputs the sound emitted by the driver D and outputs it as sound data to the warning device 10.

[0025] Speaker 3A is positioned so that driver D can hear the sound information. More than one speaker 3A may be installed inside the vehicle. Speaker 3A may also function as a sound output device such as an audio device. Speaker 3A outputs an audible warning generated by warning device 10, as described later, so that driver D can recognize the warning. Display unit 3B is positioned so that driver D can see it. Display unit 3B outputs an image-based warning generated by warning device 10, as described later, so that driver D can visually confirm the warning.

[0026] like Figure 3 As shown, the actions of driver D when feeling drowsy were categorized. It is known that driver D engages in various microsleep actions during the deepening drowsiness phase. Microsleep actions include actions such as nodding in a relaxed state, half-opening the eyes with eyelids more than half closed, and completely closing the eyes. Microsleep actions occur during the deepening drowsiness phase and are difficult to detect when driver D is wearing head-facing clothing such as sunglasses.

[0027] It is known that when driver D, while driving vehicle 1, feels drowsy, in the initial stage before reaching a state of microsleep, he will perform predetermined actions to resist drowsiness. These predetermined actions include a first action where driver D touches his body, including the head area. The first action could be, for example, touching areas of the face such as around the eyes, stimulating the head, or touching the body. The predetermined actions also include a second action where the driver stretches his body. The second action could be, for example, stretching the upper body or the arms.

[0028] The planned actions include the third action: driver D yawning. The third action is, for example, driver D yawning, including vocalizing (making a sound) during the yawning. The planned actions include the fourth action: blinking at a predetermined frequency or higher. The fourth action is, for example, driver D intentionally blinking rapidly at a predetermined frequency or higher than normal blinking. The planned actions include the fifth action: driver D making a sound at a predetermined level or higher. The fifth action is, for example, making a sound at a predetermined level or higher to combat drowsiness. The planned actions include the sixth action: driver D patting his body. The planned actions include the seventh action: driver D shaking his legs.

[0029] The arithmetic unit 11 is configured to identify various predetermined actions prior to the onset of drowsiness and output a warning to the driver D. The arithmetic unit 11 obtains monitoring information from the detection unit 2, which monitors the driver D who is driving the vehicle 1. Based on the monitoring information, the arithmetic unit 11 determines whether there are predetermined actions by the driver to resist drowsiness. The arithmetic unit 11 analyzes images captured by the in-vehicle camera 2A to monitor the state of the driver D. The arithmetic unit 11 also analyzes audio information acquired by the microphone 2B to monitor the state of the driver D.

[0030] If the calculation unit 11 detects a first action (the driver touching the body, including the head area) as the driver D has performed a predetermined action, then the calculation unit 11 determines that the driver D has performed a predetermined action if the driver D has performed a second action (stretching the body). If the calculation unit 11 detects a third action (the driver yawning) as the driver D has performed a predetermined action, then the calculation unit 11 determines that the driver D has performed a predetermined action if the driver D has performed a fourth action (blinking at a predetermined frequency or higher).

[0031] If the calculation unit 11 detects that the driver D performs a fifth action of making a sound at or above a predetermined level, it determines that the driver D has performed a predetermined action. If the calculation unit 11 detects that the driver D performs a sixth action of patting his body, it determines that the driver D has performed a predetermined action. If the calculation unit 11 detects that the driver D performs a seventh action of shaking his leg, it determines that the driver D has performed a predetermined action.

[0032] If the calculation unit 11 determines that the driver D has performed two or more predetermined actions within a predetermined time, it outputs a warning from the output unit 3. For example, if the calculation unit 11 identifies two or more predetermined actions within a predetermined time such as 10 minutes, it outputs a warning. For example, the calculation unit 11 generates a warning based on sound information to arouse the driver D's attention to drowsiness and sound information to urge him to rest, and outputs it from the speaker 3A.

[0033] The arithmetic unit 11 generates warnings based on image information intended to arouse the driver D's attention to drowsiness and image information intended to urge rest, and displays them on the display unit 3B. The arithmetic unit 11 may also output warnings based on sound information output from the speaker 3A and warnings based on image information output from the display unit 3B in conjunction with each other.

[0034] The arithmetic unit 11 can also output the next warning after a predetermined time, such as 30 minutes, if a warning has already been output. Based on the above processing, the annoyance of outputting excessive warnings to the driver D can be suppressed. By enabling the arithmetic unit 11 to recognize two or more predetermined actions, it is possible to prevent the immediate output of a warning when the driver D performs an action that is merely touching his head instead of a predetermined action, thus suppressing situations that would annoy the driver D.

[0035] If the arithmetic unit 11 detects a micro-sleep action indicating drowsiness in the driver, it determines that the action is not a predetermined action. If the driver nods, becomes half-open-eyed, or closes their eyes, the arithmetic unit 11 determines that a micro-sleep action has occurred instead of a predetermined action. Upon detecting a micro-sleep action, since the driver's drowsiness has progressed, driver assistance controls such as deceleration, stopping, and evasive maneuvers of vehicle 1, which provide greater safety than issuing a warning, can be executed.

[0036] Figure 4 The diagram illustrates the processing flow of a warning method executed in the warning device 10 for outputting a warning to the driver. The warning method is executed based on a computer program installed in the computer mounted on the warning device 10. The computer program causes the computer mounted on the warning device 10 to perform the following processes.

[0037] The arithmetic unit 11 of the warning device 10 obtains monitoring information from the detection unit 2, which is used to monitor the driver D who is driving the vehicle 1 (S100). The arithmetic unit 11 analyzes the image captured by the in-vehicle camera 2A and the sound data from the microphone 2B to monitor the actions of the driver D (S102). Based on the monitoring information, the arithmetic unit 11 determines whether the driver has performed a predetermined action to resist drowsiness (S104). If the arithmetic unit 11 determines that the driver D has performed any of the predetermined actions (S104: Yes), it determines whether a predetermined time has elapsed since the moment the action was detected (S106).

[0038] If a predetermined time has elapsed since the moment the action was identified (S106: Yes), the arithmetic unit 11 determines whether the driver D has performed a predetermined action that is different from the previously identified predetermined action (S108). If the arithmetic unit 11 outputs a warning from the output unit 3 if the driver D has performed a predetermined action that is different from the previously identified predetermined action (S108: Yes), that is, if there are more than two predetermined actions within the predetermined time, the arithmetic unit 11 outputs a warning (S110).

[0039] As described above, according to the warning device 10, by recognizing a predetermined action by the driver D, who is driving the vehicle 1, to resist drowsiness, a predetermined warning can be output before the driver feels a predetermined level of drowsiness. According to the warning device 10, even if the driver D is wearing a head covering, a predetermined warning can be output by recognizing the driver D's predetermined action to resist drowsiness. According to the warning device 10, by recognizing a predetermined action performed before the driver D experiences a mild drowsiness, a predetermined warning can be output when the driver D feels a slight drowsiness. According to the warning device 10, by outputting a warning only after recognizing two or more predetermined actions, the driver D's aversion to warnings can be suppressed.

[0040] In the above embodiments, the computer programs executed in each component of the warning device 10 may also be provided in the form of computer-readable and portable recording media such as semiconductor memory, magnetic recording media, or optical recording media. The computer program product containing the computer programs described in the above embodiments may also be stored in a storage medium or provided via a communication line.

Claims

1. A warning device for outputting a warning to the driver, The warning device includes a computing unit. The arithmetic unit, Acquire surveillance information obtained by monitoring the driver who is driving the vehicle. Based on the surveillance information, the driver's planned actions to combat drowsiness can be identified. If two or more of the predetermined actions occur within a predetermined time period, the warning is output from the output unit.

2. The warning device according to claim 1, The arithmetic unit, If the driver is identified to have made a first action of touching the body, including the head area, or a second action of stretching the body, the driver is deemed to have performed the predetermined action.

3. The warning device according to claim 1 or 2, wherein, The arithmetic unit, If the driver is identified as performing a third action of yawning, or a fourth action of blinking at a predetermined frequency or higher, it is determined that the driver has performed the predetermined action.

4. The warning device according to any one of claims 1 to 3, The arithmetic unit, If the driver is identified to have performed a fifth action that involves vocalization at or above a predetermined level, it is determined that the driver has performed the predetermined action.

5. The warning device according to any one of claims 1 to 4, The arithmetic unit, If the driver's sixth action of slapping his body or the seventh action of shaking his legs is identified, it is determined that the driver has performed the predetermined action.

6. The warning device according to any one of claims 1 to 5, The arithmetic unit, If the driver is detected to be experiencing microsleep behavior when he feels drowsy, it is determined that the predetermined action was not performed.

7. The warning device according to claim 6, The arithmetic unit, If the driver performs any of the following actions: nodding, entering a half-open eye state, or entering a closed eye state, it is determined to be the microsleep action.

8. A warning method for issuing a warning to the driver. In the warning method, The computer performs the following processing: Acquire surveillance information obtained by monitoring the driver who is driving the vehicle. Based on the surveillance information, the driver's planned actions to combat drowsiness can be identified. If two or more of the predetermined actions occur within a predetermined time period, the warning is output from the output unit.

9. A computer program product comprising a computer program that causes a computer to execute a warning method for outputting a warning to a driver. The computer program causes the computer to perform the following processes: Acquire surveillance information obtained by monitoring the driver who is driving the vehicle. Based on the surveillance information, the driver's planned actions to combat drowsiness can be identified. If two or more of the predetermined actions occur within a predetermined time period, the warning is output from the output unit.

Images