Drowsiness sign notification device, drowsiness sign notification method, and computer readable medium

Abstract

The drowsiness symptom notification device detects the eye closing state of a driver from a plurality of face images which represent the face area of the driver of a vehicle and are generated at different times. A determination is made as to whether or not a first eye-closing ratio, which indicates the ratio of the number of face images in which the eye-closing state has been detected to the number of face images generated during the first period, exceeds a first threshold value, and after the determination is made that the first eye-closing ratio does not exceed the first threshold value, a second eye-closing ratio is determined. Determines whether or not a second eye closing ratio, which represents the ratio of the number of face images in which eye closing is detected to the number of a plurality of face images generated during a second period longer than the first period, exceeds a second threshold value that is equal to or less than the first threshold value; when it is determined that the first eye-closing ratio exceeds the first threshold value or the second eye-closing ratio exceeds the second threshold value, a notification device provided in the vehicle notifies the driver of a detected drowsiness symptom.

Description

Technical Field

[0001] The present disclosure relates to a drowsiness symptom notification device, a drowsiness symptom notification method, and a computer-readable medium for notifying that a drowsiness symptom of a vehicle driver is detected. Background Art

[0002] When a vehicle driver shows signs of drowsiness such as closing his eyes, the risk of a traffic accident increases. Patent Document 1 describes a drowsiness prevention device that monitors the state of the driver's eyes and outputs an alarm based on the length of time the eyes are closed. Prior art literature Patent Literature

[0003] Patent Document 1: Japanese Patent Application Publication No. 2008-181327 Summary of the invention Problems to be solved by the invention

[0004] It is known that PERCLOS (Percent of Eyelid Closure) which is one of the indicators for detecting a drowsiness sign has a predetermined correlation with a low wakefulness state (the pre-drowsiness stage).

[0005] When determining signs of drowsiness of a vehicle driver based on PERCLOS, it may be impossible to determine whether the driver's eyes are open or closed based on the driver's facial image due to changes in the driver's posture or changes in the amount of external light entering the vehicle cabin.

[0006] When calculating PERCLOS, the time when it is impossible to determine whether the eyes are open or closed is not considered as the time when the eyes are closed. Therefore, once the time when it is impossible to determine whether the eyes are open or closed occurs after PERCLOS exceeds the specified threshold and starts to notify the driver, the PERCLOS value will temporarily decrease even if the state at that time is actually the closed state. As a result, although the driver is in a low-awake state, the notification to the driver is interrupted, and the driver cannot be properly notified of the signs of drowsiness.

[0007] An object of the present disclosure is to provide a drowsiness sign notification device capable of appropriately notifying a driver of a vehicle of a drowsiness sign. Means for solving problems

[0008] The gist of the present disclosure is as follows.

[0009] (1) A sleepiness sign notification device comprising: a detection unit that detects the eye-closing state of the driver from a plurality of facial images representing a facial region of the driver of the vehicle and generated at different times; a first determination unit for determining whether a first eye-closing ratio indicating a ratio of the number of facial images in which the eye-closing state is detected relative to the number of the plurality of facial images generated during a first period exceeds a first threshold; a second determination unit that, after determining that the first eyes-closed ratio does not exceed the first threshold, determines whether a second eyes-closed ratio indicating a ratio of the number of facial images in which the eyes-closed state is detected to the number of the plurality of facial images generated in a second period longer than the first period exceeds a second threshold that is lower than the first threshold; and The notification unit notifies the driver of detection of a drowsiness sign via a notification device provided in the vehicle when it is determined that the first eyes-closed ratio exceeds the first threshold value or the second eyes-closed ratio exceeds the second threshold value.

[0010] (2) According to the drowsiness sign notification device described in (1) above, the second determination unit makes the second period when the automatic driving level at which the driving control device provided in the vehicle controls the driving of the vehicle is an automatic driving level that does not require monitoring by the driver longer than the second period when the automatic driving level is an automatic driving level that requires monitoring by the driver.

[0011] (3) According to the drowsiness sign notification device described in (1) or (2) above, the second determination unit makes the second period when the amount of external light incident on the vehicle compartment from the outside of the vehicle detected by an external light sensor provided on the vehicle exceeds an external light threshold value longer than the second period when the amount of external light does not exceed the external light threshold value.

[0012] (4) According to any one of (1) to (3) above, the second determination unit sets the second threshold value to be smaller than the first threshold value.

[0013] (5) According to the drowsiness sign notification device described in (4) above, the second judgment unit makes the second threshold value smaller when the automatic driving level at which the driving control device provided in the vehicle controls the driving of the vehicle is an automatic driving level that does not require the driver to monitor the surroundings of the vehicle than when the automatic driving level is an automatic driving level that requires the driver to monitor the surroundings of the vehicle.

[0014] (6) A drowsiness symptom notification method, comprising: a drowsiness symptom notification device for notifying a driver of a vehicle of detected drowsiness symptoms; detecting the eye-closing state of the driver from a plurality of facial images representing the facial region of the driver and generated at different times; determining whether a first closed-eye ratio indicating a ratio of the number of facial images in which the closed-eye state is detected relative to the number of the plurality of facial images generated during a first period exceeds a first threshold; After determining that the first eyes-closed ratio does not exceed the first threshold, determining whether a second eyes-closed ratio indicating a ratio of the number of facial images in which the eyes-closed state is detected relative to the number of the plurality of facial images generated in a second period longer than the first period exceeds a second threshold that is lower than the first threshold; and When it is determined that the first eyes-closed ratio exceeds the first threshold value or the second eyes-closed ratio exceeds the second threshold value, the driver is notified of detection of a drowsiness sign via a notification device provided in the vehicle.

[0015] (7) A computer-readable medium storing a computer program for drowsiness symptom notification, the program causing a computer to execute: detecting the eye-closing state of the driver from a plurality of facial images representing facial regions of the driver of the vehicle and generated at different times; determining whether a first closed-eye ratio indicating a ratio of the number of facial images in which the closed-eye state is detected relative to the number of the plurality of facial images generated during a first period exceeds a first threshold; After determining that the first eyes-closed ratio does not exceed the first threshold, determining whether a second eyes-closed ratio indicating a ratio of the number of facial images in which the eyes-closed state is detected relative to the number of the plurality of facial images generated in a second period longer than the first period exceeds a second threshold that is lower than the first threshold; and When it is determined that the first eyes-closed ratio exceeds the first threshold value or the second eyes-closed ratio exceeds the second threshold value, the driver is notified of detection of a drowsiness sign via a notification device provided in the vehicle.

[0016] According to the drowsiness sign notification device according to the present disclosure, it is possible to appropriately notify the driver of the vehicle of the drowsiness sign. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a schematic diagram of a vehicle equipped with a drowsiness sign notification device. Figure 2 is a hardware schematic diagram of a drowsiness symptom notification device. Figure 3 This is a functional block diagram of a processor included in the sleepiness sign notification device. Figure 4 This is a diagram showing an example of transition between the eye-opening and eye-closing states. Figure 5 This is a flowchart of sleepiness symptom notification processing. DETAILED DESCRIPTION

[0018] Hereinafter, with reference to the accompanying drawings, a sleepiness sign notification device capable of appropriately notifying a driver of a vehicle of a sleepiness sign is described in detail. The sleepiness sign notification device detects the closed eye state of the driver from a plurality of facial images representing the facial region of the driver of the vehicle and generated at different times. The sleepiness sign notification device first determines whether a first closed eye ratio, which indicates the ratio of the number of facial images in which the closed eye state is detected to the number of facial images generated in a first period, exceeds a first threshold. After determining that the first closed eye ratio does not exceed the first threshold, the sleepiness sign notification device determines whether a second closed eye ratio, which indicates the ratio of the number of facial images in which the closed eye state is detected to the number of facial images generated in a second period longer than the first period, exceeds a second threshold below the first threshold. Furthermore, when determining that the first closed eye ratio exceeds the first threshold or the second closed eye ratio exceeds the second threshold, the sleepiness sign notification device notifies the detection of the sleepiness sign via a notification device provided in the vehicle.

[0019] Figure 1 This is a schematic diagram of a vehicle equipped with a drowsiness sign notification device.

[0020] The vehicle 1 includes a driver monitoring camera 2 , an instrument display 3 , a peripheral camera 4 , an optical sensor 5 , a GNSS (Global Navigation Satellite System) receiver 6 , a storage device 7 , a driving control device 8 , and a drowsiness sign notification device 9 .

[0021] The driver monitoring camera 2, the instrument display 3, the peripheral camera 4, the optical sensor 5, the GNSS receiver 6, and the storage device 7 are connected to the driving control device 8 and the drowsiness sign notification device 9 via an in-vehicle network conforming to a standard such as a controller area network. In addition, the driving control device 8 and the drowsiness sign notification device 9 are connected to each other via the in-vehicle network.

[0022] The driver monitoring camera 2 is an example of a sensor for generating a facial image representing the facial area of ​​the driver of the vehicle in a time series. The driver monitoring camera 2 has a two-dimensional detector composed of an array of photoelectric conversion elements such as CCD or C-MOS that are sensitive to infrared light, and an imaging optical system that images the area to be photographed on the two-dimensional detector. In addition, the driver monitoring camera 2 has a light source that emits infrared light. The driver monitoring camera 2 is installed at the front of the vehicle compartment, for example, facing the face of the driver sitting in the driver's seat. The driver monitoring camera 2 irradiates the driver with infrared light at predetermined time intervals (for example, 1 / 30 second to 1 / 10 second), and generates a facial image representing the driver's face at predetermined time intervals.

[0023] The instrument display 3 is an example of a notification device, and includes, for example, a liquid crystal display. The instrument display 3 displays a screen notifying the driver of the detection of a drowsiness sign according to a signal received from the drowsiness sign notification device 9 via the in-vehicle network. The vehicle 1 may also include a speaker (not shown) that outputs a sound notifying the driver of the detection of a drowsiness sign as a notification device.

[0024] The surrounding camera 4 is an example of a surrounding sensor for generating surrounding data representing the surrounding conditions of the vehicle 1. The surrounding camera 4 has a two-dimensional detector composed of an array of photoelectric conversion elements such as CCD or C-MOS that are sensitive to visible light, and an imaging optical system that images the image of the area to be photographed on the two-dimensional detector. The surrounding camera 4 is arranged, for example, at the front upper part of the vehicle compartment facing forward. The surrounding camera 4 photographs the surrounding conditions of the vehicle 1 through the front window glass at each predetermined shooting cycle (for example, 1 / 30 second to 1 / 10 second). The surrounding camera 4 outputs a surrounding image representing the surrounding conditions as surrounding data. In addition, the vehicle 1 may also have a sensor other than the surrounding camera 4, such as a LiDAR (Light Detection And Ranging) sensor, as a surrounding sensor, which generates a distance image in which each pixel has a value corresponding to the distance to the object represented by the pixel based on the surrounding conditions of the vehicle 1 as surrounding data.

[0025] The light sensor 5 is an example of an external light sensor that detects the amount of external light. The light sensor 5 has a light receiving element including a photodiode, and outputs a light amount signal indicating the amount of external light received. The light sensor 5 is arranged in the vehicle cabin (for example, in the upper front part of the vehicle cabin) in such a manner that light outside the vehicle cabin is incident on the light receiving element via the front window glass.

[0026] The GNSS receiver 6 is an example of a positioning sensor, and receives GNSS signals from GNSS satellites at a predetermined period, and measures the own position of the vehicle 1 based on the received GNSS signals. The GNSS receiver 6 outputs a positioning signal indicating the positioning result of the own position of the vehicle 1 based on the GNSS signal to the driving control device 8 via the in-vehicle network at a predetermined period.

[0027] The storage device 7 is an example of a storage unit, and includes, for example, a hard disk device or a nonvolatile semiconductor memory. The storage device 7 stores a high-precision map. The high-precision map includes, for example, information indicating lane demarcation lines that demarcate lanes in a predetermined area indicated by the high-precision map.

[0028] The driving control device 8 is an example of an automatic driving system, and is an ECU (Electronic Control Unit) having a communication interface, a memory, and a processor. The driving control device 8 reads the information of the lane demarcation lines around the position of the own vehicle represented by the positioning signal received from the GNSS receiver 6 from the storage device 7 that stores the high-precision map. The driving control device 8 detects the surrounding lane demarcation lines by inputting the surrounding images received from the surrounding camera 4 to the identifier, and determines the lane in which the vehicle 1 is traveling by matching with the lane demarcation lines in the high-precision map. The driving control device 8 generates a driving path in a manner of maintaining or changing the lane according to the situation. Then, the driving control device 8 outputs a control signal to the driving mechanism (not shown) of the vehicle 1 so that it travels along the driving path. The driving mechanism includes, for example, an engine or motor that provides power to the vehicle 1, a brake that reduces the driving speed of the vehicle 1, and a steering mechanism that steers the vehicle 1.

[0029] The discriminator can be formed as a convolutional neural network (CNN) having a plurality of convolutional layers connected in series from the input side to the output side, such as YOLO (You Only Look Once) and SSD (Single Shot MultiBox Detector). By using a plurality of images showing lane markings to be detected as teacher data to input to the CNN for pre-learning, the CNN operates as a discriminator that detects lane markings and outputs the area showing the lane markings on the image.

[0030] When the prescribed conditions such as a detailed map of the driving location can be accessed and the driver is in a state where driving operations can be performed in an emergency are met, the driving control device 8 can control the driving of the vehicle 1 at an automatic driving level that does not require the driver to monitor the surroundings of the vehicle (for example, automatic driving level 3 defined by the Society of Automotive Engineers (SAE)). When such conditions are not met, the driving control device 8 controls the driving of the vehicle 1 at an automatic driving level that requires the driver to monitor the surroundings of the vehicle (for example, automatic driving level 2 defined by SAE). The vehicle 1 can also be driven by manual driving that outputs a control signal to the driving mechanism based on the driver's operation.

[0031] Figure 2 : is a hardware schematic diagram of the drowsiness sign notification device 9. The drowsiness sign notification device 9 is an ECU having a communication interface 91, a memory 92, and a processor 93. The drowsiness sign notification device 9 detects the closed eye state of the driver from the facial image generated by the driver monitoring camera 2, and when the drowsiness sign is detected based on the number of facial images in which the closed eye state is detected, the instrument display 3 displays an image indicating that the drowsiness sign is detected. The driving control device 8 and the drowsiness sign notification device 9 may also be installed in the same ECU.

[0032] The communication interface 91 is an example of a communication unit, and has a communication interface circuit for connecting the sleepiness sign notification device 9 to the in-vehicle network. The communication interface 91 provides the processor 93 with data received from, for example, the driver monitoring camera 2. In addition, the communication interface 91 outputs the data provided from the processor 93 to, for example, the meter display 3.

[0033] The memory 92 includes a volatile semiconductor memory and a nonvolatile semiconductor memory. The memory 92 stores various data used in the processing performed by the processor 93, such as a template representing a corneal reflection image of a pupil and a light source used to detect the closed eye state of the driver from a facial image. In addition, the memory 92 stores a first period and a second period for determining the generation time of a facial image to be determined, and a first threshold and a second threshold for detecting the presence or absence of a drowsiness sign based on the ratio of the number of facial images.

[0034] In addition, the memory 92 stores various application programs, for example, a sleepiness sign notification computer program for causing the sleepiness sign notification device 9 to execute a sleepiness sign notification method.

[0035] The processor 93 is an example of a control unit, and includes one or more processors and peripheral circuits thereof. The processor 93 may also include other arithmetic circuits such as a logical operation unit, a numerical operation unit, or a graphic processing unit.

[0036] Figure 3 This is a functional block diagram of the processor 93 included in the sleepiness sign notification device 9 .

[0037] The processor 93 of the sleepiness sign notification device 9 has a detection unit 931, a first determination unit 932, a second determination unit 933, and a notification unit 934 as functional blocks. These various units of the processor 93 are functional modules implemented by programs executed on the processor 93. The computer program that realizes the functions of the various units of the processor 93 can also be provided in the form of being recorded in a computer-readable removable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. Alternatively, these various units of the processor 93 can also be installed in the sleepiness sign notification device 9 as independent integrated circuits, microprocessors, or firmware.

[0038] The detection unit 931 detects the closed eye state of the driver from each of the plurality of facial images generated by the driver monitoring camera 2 at predetermined time intervals.

[0039] The detection unit 931 detects the face region by inputting the face image to the discriminator that detects the face region corresponding to the driver's face from the face image. Furthermore, the detection unit 931 determines the eye peripheral region that is assumed to represent the driver's eyes in the face region based on the structure of the face. Then, the detection unit 931 determines the positions of the upper and lower eyelids by performing template matching between the templates representing the upper and lower eyelids and the determined eye peripheral region image, and obtains the distance between the upper and lower eyelids.

[0040] The detection unit 931 detects the closed eye state when the distance between the upper and lower eyelids obtained from the facial image does not exceed a predetermined distance threshold, and stores the closed eye data (e.g., "2") indicating the closed eye state in association with the time when the facial image was generated in the memory 92. The detection unit 931 detects the open eye state when the distance between the upper and lower eyelids obtained from the facial image exceeds a predetermined distance threshold, and stores the open eye data (e.g., "1") indicating the open eye state in association with the time when the facial image was generated in the memory 92. When the facial area is not detected from the facial image, the peripheral area of ​​the eye is not determined in the facial area, and the positions of the upper and lower eyelids are not determined from the peripheral area of ​​the eye, the detection unit 931 stores the undeterminable data (e.g., "0") indicating that the determination cannot be made in association with the time when the facial image was generated in the memory 92.

[0041] The first determination unit 932 determines whether a first closed-eye ratio indicating a ratio of the number of facial images in which the closed-eye state is detected to the number of the plurality of facial images generated during the first period exceeds a first threshold value.

[0042] After determining that the first eyes-closed ratio does not exceed the first threshold, the second determination unit 933 determines whether a second eyes-closed ratio, which represents the ratio of the number of facial images in which the eyes-closed state is detected to the number of multiple facial images generated in a second period longer than the first period, exceeds a second threshold that is lower than the first threshold.

[0043] Figure 4 This is a diagram showing an example of transition between the eye-opening and eye-closing states.

[0044] The eye-opening and eye-closing state curve 100 represents the facial image FP generated by the driver monitoring camera 2 in time series at each time from time k-29 to time k. k-29 To face image FP k In the eye-opening / eye-closing state curve 100, the vertical axis represents the eye-opening / eye-closing state (2 for closed eyes, 1 for open eyes, and 0 for undetermined), and the horizontal axis represents the time. Figure 4 In the example, the facial image FP is generated at 1 second intervals. k-29 Is the face image FP k The face image was generated 30 seconds before the generation time.

[0045] exist Figure 4 In the example of , the first period is 10 seconds from the reference time to 10 seconds ago. k-10 Generated 10 face images FP k-19 To face image FP k-10 Therefore, the first determination unit 932 determines the first period T1 based on the time k-10 as the reference. k-10 The corresponding first eye-closing ratio is set to 10 / 10 = 1.0. In addition, the length of the first period may be different from 10 seconds, such as 8 seconds or 12 seconds.

[0046] exist Figure 4 In the example of , the first threshold is 0.6. Therefore, the first determination unit 932 determines that the first eye closure ratio corresponding to the time k-10 exceeds the first threshold. In addition, the first threshold may be a value other than 0.6, such as 0.5 or 0.7.

[0047] From the first period T1 when the time k after 10 seconds corresponding to the first period has elapsed from the time k-10 is used as the reference k Generated 10 face images FP k-9 To face image FP k The eye-closing state was detected in 5 of the 5 images, and the 5 images could not be determined. Therefore, the first determination unit 932 sets the first period T1 based on the time k as the reference. kThe corresponding first eye-closing ratio is set to 5 / 10=0.5, which is determined to be not more than the first threshold.

[0048] In the first period T1 when it is determined that the time k is used as the reference k After the first eye-closing ratio corresponding to the time k does not exceed the first threshold value, the second determination unit 933 determines that the second period T2 based on the time k is k Whether the corresponding second closed eye ratio exceeds the second threshold.

[0049] exist Figure 4 In the example of , the second period is from the reference time to 30 seconds before 30 seconds. k Generated 30 face images FP k-29 To FP k The closed eye state was detected in 21 of the images, the open eye state was detected in 2 of the images, and the determination could not be made in 7 of the images. k The corresponding second eye-closing ratio is set to 21 / 30=0.7.

[0050] In addition, the length of the second period may be different from 30 seconds, for example, 25 seconds or 35 seconds. In addition, the time that serves as the reference for the second period may be different from the time that serves as the reference for the first period. For example, the time when the determination on the end of the first period is completed may be set as the time that serves as the reference for the second period.

[0051] The second determination unit 933 may also change the length of the second period according to the automatic driving level at the time of reference. For example, the second determination unit 933 obtains the automatic driving level for controlling the driving of the vehicle 1 from the driving control device 8. When the driving control device 8 and the drowsiness sign notification device 9 are installed in different ECUs, the second determination unit 933 inquires the driving control device 8 for the automatic driving level via the communication interface 91. When the driving control device 8 and the drowsiness sign notification device 9 are installed in the same ECU, the second determination unit 933 reads the automatic driving level used in the driving control of the vehicle 1 recorded in the prescribed area of ​​the memory 92. In addition, the second determination unit 933 makes the second period when the automatic driving level is an automatic driving level that does not require the driver to monitor the surroundings of the vehicle longer than the second period when the automatic driving level is an automatic driving level that requires the driver to monitor the surroundings of the vehicle (for example, 60 seconds). In this way, the drowsiness sign notification device 9 can suppress the interruption of notification caused by the inability to determine accompanied by the change of the driver's posture, which is easy to occur during the period of driving control at the automatic driving level that does not require the driver to monitor.

[0052] The second determination unit 933 may also change the length of the second period according to the amount of external light. For example, the second determination unit 933 receives a light amount signal indicating the amount of external light incident on the cabin of the vehicle 1 from the outside of the vehicle 1 from the optical sensor 5 via the communication interface 91. The second determination unit 933 makes the second period when the amount of external light indicated by the received light amount signal exceeds the external light amount threshold stored in the memory 92 longer (for example, 60 seconds) than the second period when the amount of external light does not exceed the external light amount threshold. In this way, the sleepiness sign notification device 9 can suppress the interruption of notification caused by the inability to determine due to strong external light.

[0053] exist Figure 4 In the example of , the second threshold is 0.6 which is equal to the first threshold. Therefore, the second determination unit 933 determines that the second eye closing ratio corresponding to the time k exceeds the second threshold.

[0054] The second determination unit 933 may set the second threshold value to a value smaller than the first threshold value (for example, 0.5 with respect to the first threshold value of 0.6). In this way, the sleepiness sign notification device 9 can more appropriately suppress the interruption of notification due to the inability to determine.

[0055] For example, the second determination unit 933 may make the second threshold value at the automatic driving level that does not require the driver to monitor the vehicle smaller than the second threshold value at the automatic driving level that requires the driver to monitor the surroundings of the vehicle. In this way, the sleepiness sign notification device 9 can suppress the interruption of notification caused by the inability to determine due to the change of the driver's posture or the gaze in a direction other than the traveling direction (such as a navigation device), which is likely to occur during driving at the automatic driving level that does not require the driver to monitor the surroundings of the vehicle.

[0056] In addition, the second determination unit 933 can make the second threshold value when the vehicle 1 is traveling by manual driving based on the driver's operation smaller than the second threshold value when the vehicle 1 is traveling by automatic driving. In this way, the sleepiness sign notification device 9 can suppress the interruption of notification caused by the inability to determine that is easily caused by the change of sight line to a direction other than the traveling direction during manual driving.

[0057] In addition, the second determination unit 933 can make the second threshold value when the driver is wearing glasses or sunglasses smaller than the second threshold value when the driver is not wearing glasses or sunglasses. In this way, the sleepiness sign notification device 9 can suppress the interruption of notification caused by the inability to determine due to the overlap of the frames of glasses or sunglasses with the eyes. In addition, whether the driver is wearing glasses or sunglasses can be detected by, for example, performing template matching between a template representing the frames of glasses or sunglasses and an image of the peripheral area of ​​the eyes by the detection unit 931.

[0058] return Figure 3, when it is determined that the first eye-closing ratio exceeds the first threshold value, or the second eye-closing ratio exceeds the second threshold value, the notification unit 934 notifies the driver of the detection of drowsiness signs. The notification unit 934 sends a display signal to the instrument display 3 via the communication interface 91, which displays a screen that notifies the driver of the detection of drowsiness signs. The screen that notifies the driver of the detection of drowsiness signs may include, for example, a character string indicating that drowsiness signs have been detected, such as "Signs of drowsiness have been confirmed", or may include a character string such as "Do you want to take a break?" or an image indicating a coffee break, such as information urging the driver to take a break. In addition, in addition to the screen display on the instrument display 3, the notification unit 934 may also cause a speaker (not shown) to output a prescribed sound (such as the sound of a buzzer) as a notification to the driver that drowsiness signs have been detected.

[0059] Figure 5 2 is a flowchart of the drowsiness sign notification process. The processor 93 of the drowsiness sign notification device 9 executes the drowsiness sign notification process described below at predetermined time intervals (for example, every 1 second) while the vehicle 1 is traveling.

[0060] First, the detection unit 931 of the processor 93 of the sleepiness sign notification device 9 detects the closed eye state of the driver from the facial image obtained from the driver monitoring camera 2 (step S1). In step S1, the detection unit 931 detects the closed eye state of the driver from each of a plurality of facial images generated during a first period based on a predetermined time.

[0061] Next, the first determination unit 932 of the processor 93 determines whether a first closed eye ratio, which indicates the ratio of the number of facial images in which the closed eye state is detected to the number of the plurality of facial images generated during the first period, exceeds a first threshold value (step S2). If it is determined that the first closed eye ratio exceeds the first threshold value (step S2: yes), the processing of the processor 93 proceeds to step S4 described later.

[0062] When it is determined that the first eye-closing ratio does not exceed the first threshold value (step S2: No), the second determination unit 933 of the processor 93 determines whether the second eye-closing ratio, which indicates the ratio of the number of facial images in which the eye-closing state is detected to the number of multiple facial images generated in the second period longer than the first period, exceeds the second threshold value (step S3). When it is determined that the second eye-closing ratio does not exceed the second threshold value (step S3: No), the processing of the processor 93 returns to step S1, and the processing after step S1 is performed with the first period as the reference, which is the time after the specified time in the previous step S1 by the length of the first period.

[0063] When it is determined that the second eye-closing ratio exceeds the second threshold value (step S3: Yes), the notification unit 934 of the processor 93 notifies the driver of the detection of the drowsiness sign (step S4). After executing step S4, the processing of the processor 93 returns to step S1, and the processing after step S1 is performed with the first period based on the time after the predetermined time in the previous step S1 by the length of the first period as the object.

[0064] By executing the drowsiness sign notification process in this way, the drowsiness sign notification device 9 can appropriately notify the driver of the vehicle of the drowsiness sign.

[0065] It should be understood that those skilled in the art can make various changes, substitutions, and modifications without departing from the spirit and scope of the present disclosure. Description of Reference Numerals

[0066] 1 Vehicle 9Drowsiness Sign Notification Device 931 Detection Department 932 First Judgment Unit 933 Second Judgment Unit 934 Notification Department

Claims

1. A sleepiness sign notification device, comprising: a detection unit that detects the closed eye state of the driver from a plurality of facial images representing a facial region of the driver of the vehicle and generated at different times; a first determination unit for determining whether a first eye-closing ratio indicating a ratio of the number of facial images in which the eye-closing state is detected relative to the number of the plurality of facial images generated during a first period exceeds a first threshold; a second determination unit that, after determining that the first eyes-closed ratio does not exceed the first threshold, determines whether a second eyes-closed ratio, which indicates a ratio of the number of facial images in which the eyes-closed state is detected to the number of the plurality of facial images generated in a second period longer than the first period, exceeds a second threshold that is lower than the first threshold; as well as The notification unit notifies the driver of detection of a drowsiness sign via a notification device provided in the vehicle when it is determined that the first eyes-closed ratio exceeds the first threshold value or the second eyes-closed ratio exceeds the second threshold value.

2. The drowsiness sign notification device according to claim 1, wherein: The second determination unit makes the second period longer when the autonomous driving level at which a driving control device provided in the vehicle controls the driving of the vehicle is an autonomous driving level that does not require the driver to monitor the surroundings of the vehicle than when the autonomous driving level is an autonomous driving level that requires the driver to monitor the surroundings of the vehicle.

3. The drowsiness sign notification device according to claim 1 or 2, wherein: The second determination unit makes the second period longer when the amount of external light incident on the vehicle cabin from outside the vehicle detected by an external light amount sensor provided in the vehicle exceeds an external light amount threshold than when the amount of external light does not exceed the external light amount threshold.

4. The drowsiness sign notification device according to any one of claims 1 to 3, wherein: The second determination unit makes the second threshold smaller than the first threshold.

5. The drowsiness sign notification device according to claim 4, wherein: The second judgment unit makes the second threshold value smaller when the automatic driving level at which the driving control device provided in the vehicle controls the driving of the vehicle is an automatic driving level that does not require the driver to monitor the surroundings of the vehicle than when the automatic driving level is an automatic driving level that requires the driver to monitor the surroundings of the vehicle.

6. A drowsiness symptom notification method, comprising: detecting the eye-closing state of the driver from a plurality of facial images representing the facial region of the driver and generated at different times; determining whether a first closed-eye ratio indicating a ratio of the number of facial images in which the closed-eye state is detected relative to the number of the plurality of facial images generated during a first period exceeds a first threshold; After determining that the first eyes-closed ratio does not exceed the first threshold, determining whether a second eyes-closed ratio indicating a ratio of the number of facial images in which the eyes-closed state is detected relative to the number of the plurality of facial images generated in a second period longer than the first period exceeds a second threshold that is lower than the first threshold; as well as When it is determined that the first eyes-closed ratio exceeds the first threshold value or the second eyes-closed ratio exceeds the second threshold value, the driver is notified of detection of a drowsiness sign via a notification device provided in the vehicle.

7. A computer readable medium storing a computer program for drowsiness symptom notification, the program causing a computer to execute: detecting the eye-closing state of the driver from a plurality of facial images representing facial regions of the driver of the vehicle and generated at different times; determining whether a first closed-eye ratio indicating a ratio of the number of facial images in which the closed-eye state is detected relative to the number of the plurality of facial images generated during a first period exceeds a first threshold; After determining that the first eyes-closed ratio does not exceed the first threshold, determining whether a second eyes-closed ratio indicating a ratio of the number of facial images in which the eyes-closed state is detected relative to the number of the plurality of facial images generated in a second period longer than the first period exceeds a second threshold that is lower than the first threshold; as well as When it is determined that the first eyes-closed ratio exceeds the first threshold value or the second eyes-closed ratio exceeds the second threshold value, the driver is notified of detection of a drowsiness sign via a notification device provided in the vehicle.

Images