Drowsiness sign notification device, drowsiness sign notification method, and drowsiness sign notification computer program

Abstract

To appropriately notify a driver of a vehicle of a drowsiness sign.SOLUTION: A drowsiness sign notification device detects a closed eye state of a driver from each of a plurality of face images that represent a face region of the driver of a vehicle and that are generated at different times from each other; determines whether or not a first closed eye ratio exceeds a first threshold value, the first closed eye ratio indicating a ratio of the number of face images in which a closed eye state is detected to the number of a plurality of face images that are generated in a first period; after it is determined that the first closed eye ratio does not exceed the first threshold value, determines whether or not a second closed eye ratio exceeds a second threshold value equal to or less than the first threshold value, the second closed eye ratio indicating the ratio of the number of face images in which a closed eye state is detected to the number of a plurality of face images that are generated in a second period longer than the first period; and notifies the driver that a drowsiness sign has been detected via a notification device provided in the vehicle if it is determined that the first closed eye ratio exceeds the first threshold value or the second closed eye ratio exceeds the second threshold value.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to a drowsiness sign notification device, a drowsiness sign notification method, and a drowsiness sign notification computer program that notify that a drowsiness sign has been detected in a vehicle driver. [Background technology]

[0002] When a vehicle driver shows signs of drowsiness, such as closing the eyes, the risk of a traffic accident increases. Patent Document 1 describes a driver drowsiness prevention device that monitors the driver's eye state and outputs an alarm based on the length of time the eyes are closed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2008-181327 A Summary of the Invention [Problem to be solved by the invention]

[0004] PERCLOS (Percent of Eyelid Closure), which is one of the indices for detecting signs of drowsiness, is known to have a certain correlation with a low alertness state (the stage before falling asleep).

[0005] When determining the driver's drowsiness symptoms based on PERCLOS, it may not be possible to determine whether the driver's eyes are open or closed from the driver's facial image due to changes in the driver's posture, changes in the amount of external light entering the vehicle cabin, etc.

[0006] When calculating PERCLOS, the time when it is not possible to determine whether the eyes are open or closed is not treated as the time when the eyes are closed. Therefore, if a time when it is not possible to determine whether the eyes are open or closed occurs after PERCLOS exceeds a certain threshold and notification to the driver is started, the value of PERCLOS temporarily drops even if the state at that time is actually a closed state. As a result, notification to the driver is interrupted even if the driver is in a low alert state, and the driver cannot be properly notified of drowsiness symptoms.

[0007] An object of the present disclosure is to provide a drowsiness sign notification device that can appropriately notify a vehicle driver of drowsiness signs. [Means for solving the problem]

[0008] The gist of the present disclosure is as follows.

[0009] (1) a detection unit that detects a closed eye state of a vehicle driver from each of a plurality of face images that represent a face area of ​​the driver and are generated at different times; a first determination unit that determines whether a first eye closure ratio indicating a ratio of a number of face images in which the eye closure state is detected to a number of the plurality of face images generated in a first period exceeds a first threshold value; a second determination unit that, after it is determined that the first eye closure ratio does not exceed the first threshold, determines whether a second eye closure ratio indicating a ratio of a number of face images in which the eye closure state is detected to a number of the plurality of face images generated in a second period longer than the first period exceeds a second threshold that is equal to or less than the first threshold; a notification unit that notifies the driver that a sign of drowsiness has been detected via a notification device provided in the vehicle when it is determined that the first eye closure percentage exceeds the first threshold value or the second eye closure percentage exceeds the second threshold value; A drowsiness sign notification device comprising:

[0010] (2) The drowsiness sign notification device described in (1) above, wherein the second judgment unit makes the second period longer when the autonomous driving level at which a driving control device provided in the vehicle controls the driving of the vehicle is an autonomous driving level that does not require monitoring by the driver than the second period when the autonomous driving level is an autonomous driving level that requires monitoring by the driver.

[0011] (3) The drowsiness sign notification device described in (1) or (2) above, wherein the second judgment unit makes the second period when the amount of external light entering the vehicle cabin from outside the vehicle detected by an external light sensor installed in the vehicle exceeds an external light threshold longer than the second period when the amount of external light does not exceed the external light threshold.

[0012] (4) The drowsiness sign notification device according to any one of (1) to (3) above, wherein the second determination unit sets the second threshold to be smaller than the first threshold.

[0013] (5) The drowsiness sign notification device described in (4) above, wherein the second judgment unit sets the second threshold value when the autonomous driving level at which a driving control device provided in the vehicle controls the driving of the vehicle is an autonomous driving level that does not require the driver to monitor the surroundings of the vehicle to be smaller than the second threshold value when the autonomous driving level is an autonomous driving level that requires the driver to monitor the surroundings of the vehicle.

[0014] (6) A drowsiness sign notification device that notifies the driver that drowsiness signs have been detected in the vehicle, detecting a closed eye state of the driver from each of a plurality of face images representing a face area of ​​the driver and generated at different times; determining whether a first eye closure ratio indicating a ratio of the number of face images in which the eye closure state is detected to the number of the plurality of face images generated in a first period exceeds a first threshold value; after it is determined that the first eye closure ratio does not exceed the first threshold, it is determined whether a second eye closure ratio indicating a ratio of the number of face images in which the eye closed state is detected to the number of the plurality of face images generated in a second period longer than the first period exceeds a second threshold which is equal to or less than the first threshold; notifying the driver that a sign of drowsiness has been detected via a notification device provided in the vehicle when it is determined that the first eye closure percentage exceeds the first threshold value or the second eye closure percentage exceeds the second threshold value; A drowsiness symptom notification method comprising:

[0015] (7) detecting a closed eye state of a driver of a vehicle from each of a plurality of face images representing a face area of ​​the driver and generated at different times; determining whether a first eye closure ratio indicating a ratio of the number of face images in which the eye closure state is detected to the number of the plurality of face images generated in a first period exceeds a first threshold value; after it is determined that the first eye closure ratio does not exceed the first threshold, determining whether a second eye closure ratio indicating a ratio of a number of face images in which the eye closure state is detected to a number of the plurality of face images generated in a second period longer than the first period exceeds a second threshold which is equal to or less than the first threshold; notifying the driver that a sign of drowsiness has been detected via a notification device provided in the vehicle when it is determined that the first eye closure percentage exceeds the first threshold value or the second eye closure percentage exceeds the second threshold value; A computer program for notifying a user of signs of drowsiness, which causes a computer to execute the above.

[0016] According to the drowsiness sign notification device according to the present disclosure, it is possible to appropriately notify the driver of a vehicle of drowsiness signs. [Brief description of the drawings]

[0017] [Figure 1] 1 is a schematic configuration diagram of a vehicle in which a drowsiness sign notification device is implemented. [Diagram 2] 1 is a hardware schematic diagram of a drowsiness sign notification device. [Diagram 3] 3 is a functional block diagram of a processor included in the drowsiness sign notification device. FIG. [Figure 4] 11A and 11B are diagrams illustrating an example of a transition of an open / closed eye state. [Diagram 5] 13 is a flowchart of a drowsiness sign notification process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, a drowsiness sign notification device capable of appropriately notifying a driver of a drowsiness sign of a vehicle will be described in detail with reference to the drawings. The drowsiness sign notification device detects a closed eye state of the driver from each of a plurality of face images that represent a face area of ​​the driver of the vehicle and are generated at different times. The drowsiness sign notification device first determines whether or not a first closed eye ratio indicating a ratio of the number of face images in which a closed eye state is detected to the number of a plurality of face images generated in a first period exceeds a first threshold value. After it is determined that the first closed eye ratio does not exceed the first threshold value, the drowsiness sign notification device determines whether or not a second closed eye ratio indicating a ratio of the number of face images in which a closed eye state is detected to the number of a plurality of face images generated in a second period longer than the first period exceeds a second threshold value that is equal to or smaller than the first threshold value. Then, when it is determined that the first closed eye ratio exceeds the first threshold value or the second closed eye ratio exceeds the second threshold value, the drowsiness sign notification device notifies that a drowsiness sign has been detected via a notification device provided in the vehicle.

[0019] FIG. 1 is a schematic diagram of a vehicle in which a drowsiness sign notification device is installed.

[0020] The vehicle 1 has a driver monitor camera 2, a meter display 3, a surrounding camera 4, an optical sensor 5, a Global Navigation Satellite System (GNSS) receiver 6, a storage device 7, a driving control device 8, and a drowsiness sign notification device 9.

[0021] The driver monitor camera 2, the meter display 3, the peripheral camera 4, the optical sensor 5, the GNSS receiver 6, and the storage device 7 are communicatively connected to the driving control device 8 and the drowsiness sign notification device 9 via an in-vehicle network that complies with a standard such as a controller area network. The driving control device 8 and the drowsiness sign notification device 9 are also communicatively connected to each other via the in-vehicle network.

[0022] The driver monitor camera 2 is an example of a sensor for generating facial images representing the facial area of ​​the driver of the vehicle in a time series. The driver monitor camera 2 has a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to infrared light, such as CCD or C-MOS, and an imaging optical system for forming an image of the area to be photographed on the two-dimensional detector. The driver monitor camera 2 also has a light source that emits infrared light. The driver monitor camera 2 is attached, for example, at the front of the vehicle interior, facing the face of the driver seated in the driver's seat. The driver monitor camera 2 irradiates the driver with infrared light at predetermined time intervals (for example, 1 / 30 to 1 / 10 seconds) and generates facial images showing the driver's face at predetermined time intervals.

[0023] The meter display 3 is an example of a notification device, and has, for example, a liquid crystal display. The meter display 3 displays a screen notifying the driver that a drowsiness sign has been detected, according to a signal received from the drowsiness sign notification device 9 via the in-vehicle network. The vehicle 1 may have, as a notification device, a speaker (not shown) that outputs a sound notifying the driver that a drowsiness sign has been detected.

[0024] The surrounding camera 4 is an example of a surrounding sensor for generating surrounding data representing the surrounding conditions of the vehicle 1. The surrounding camera 4 has a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to visible light, such as a CCD or a C-MOS, and an imaging optical system for forming an image of an area to be photographed on the two-dimensional detector. The surrounding camera 4 is disposed, for example, at the front upper part of the vehicle interior, facing forward. The surrounding camera 4 photographs the surrounding conditions of the vehicle 1 through the windshield at a predetermined photographing period (for example, 1 / 30 seconds to 1 / 10 seconds). The surrounding camera 4 outputs a surrounding image representing the surrounding conditions as surrounding data. Note that the vehicle 1 may have, as a surrounding sensor, a sensor other than the surrounding camera 4, for example, a LiDAR (Light Detection And Ranging) sensor that generates, as surrounding data, a distance image in which each pixel has a value corresponding to the distance to an object represented in the pixel based on the surrounding conditions of the vehicle 1.

[0025] The optical sensor 5 is an example of an external light amount sensor that detects the amount of external light. The optical sensor 5 has a light receiving element including a photodiode, and outputs a light amount signal that indicates the amount of received external light. The optical sensor 5 is disposed inside the vehicle (for example, in the upper front part of the vehicle interior) so that light outside the vehicle interior is incident on the light receiving element through the windshield.

[0026] The GNSS receiver 6 is an example of a positioning sensor, which receives GNSS signals from GNSS satellites at predetermined intervals and determines the position of the vehicle 1 based on the received GNSS signals. The GNSS receiver 6 outputs a positioning signal representing the positioning result of the vehicle 1 based on the GNSS signals to the cruise control device 8 via the in-vehicle network at predetermined intervals.

[0027] The storage device 7 is an example of a storage unit, and includes, for example, a hard disk device or a non-volatile semiconductor memory. The storage device 7 stores a high-precision map. The high-precision map includes, for example, information representing lane markings that divide lanes in a predetermined area depicted on the high-precision map.

[0028] The driving control device 8 is an example of an automatic driving system, and is an ECU (Electronic Control Unit) equipped with a communication interface, a memory, and a processor. The driving control device 8 reads information on lane markings in the vicinity of the vehicle position represented in the positioning signal received from the GNSS receiver 6 from the storage device 7 that stores a high-precision map. The driving control device 8 detects surrounding lane markings by inputting the surrounding image received from the surrounding camera 4 to a classifier, and identifies the lane in which the vehicle 1 is traveling by matching the lane markings with the lane markings in the high-precision map. The driving control device 8 creates a driving route to maintain or change the lane depending on the situation. Then, the driving control device 8 outputs a control signal to a driving mechanism (not shown) of the vehicle 1 so that the vehicle 1 travels along the driving route. The driving mechanism includes, for example, an engine or motor that supplies power to the vehicle 1, a brake that reduces the traveling speed of the vehicle 1, and a steering mechanism that steers the vehicle 1.

[0029] The classifier can be, for example, a convolutional neural network (CNN) having multiple convolutional layers connected in series from the input side to the output side, such as YOLO (You Only Look Once) or SSD (Single Shot MultiBox Detector). A large number of images showing the lane markings to be detected are input to the CNN as training data for learning in advance, whereby the CNN operates as a classifier that detects the lane markings and outputs the area on the image where the lane markings are shown.

[0030] When certain conditions are met, such as the ability to access a detailed map of the driving location and the driver being able to operate the vehicle in an emergency, the driving control device 8 can control the driving of the vehicle 1 at an autonomous driving level that does not require the driver to monitor the vehicle's surroundings (e.g., autonomous driving level 3 defined by the Society of Automotive Engineers (SAE)). When such conditions are not met, the driving control device 8 controls the driving of the vehicle 1 at an autonomous driving level that requires the driver to monitor the vehicle's surroundings (e.g., autonomous driving level 2 defined by the SAE). The vehicle 1 may be driven manually, which outputs a control signal to the driving mechanism based on the driver's operation.

[0031] 2 is a hardware schematic diagram of the drowsiness sign notification device 9. The drowsiness sign notification device 9 is an ECU including a communication interface 91, a memory 92, and a processor 93. The drowsiness sign notification device 9 detects the driver's closed eye state from face images generated by the driver monitor camera 2, and when a drowsiness sign is detected based on the number of face images in which a closed eye state is detected, causes an image indicating that a drowsiness sign has been detected to be displayed on the meter display 3. The driving control device 8 and the drowsiness sign notification device 9 may be implemented in the same ECU.

[0032] The communication interface 91 is an example of a communication unit, and has a communication interface circuit for connecting the drowsiness sign notification device 9 to an in-vehicle network. The communication interface 91 supplies data received from, for example, the driver monitor camera 2 to the processor 93. In addition, the communication interface 91 outputs data supplied from the processor 93 to, for example, the meter display 3.

[0033] The memory 92 includes a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 92 stores various data used in processing by the processor 93, such as a template showing a pupil and a corneal reflection image of a light source used to detect a driver's closed eye state from a face image. The memory 92 also stores a first period and a second period for specifying the time of generation of a face image to be determined, and a first threshold value and a second threshold value for detecting the presence or absence of a drowsiness sign based on the ratio of the number of face images.

[0034] The memory 92 also stores various application programs, such as a computer program for drowsiness sign notification that causes the drowsiness sign notification device 9 to execute a drowsiness sign notification method.

[0035] The processor 93 is an example of a control unit, and includes one or more processors and their peripheral circuits. The processor 93 may further include other arithmetic circuits, such as a logic arithmetic unit, a numerical arithmetic unit, or a graphics processing unit.

[0036] FIG. 3 is a functional block diagram of the processor 93 included in the drowsiness sign notification device 9.

[0037] The processor 93 of the drowsiness sign notification device 9 has, as functional blocks, a detection unit 931, a first determination unit 932, a second determination unit 933, and a notification unit 934. Each of these units in the processor 93 is a functional module implemented by a program executed on the processor 93. A computer program that realizes the functions of each unit of the processor 93 may be provided in a form recorded on a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. Alternatively, each of these units in the processor 93 may be implemented in the drowsiness sign notification device 9 as an independent integrated circuit, a microprocessor, or firmware.

[0038] The detection unit 931 detects whether the driver's eyes are closed from each of a plurality of face images generated by the driver monitor camera 2 at a predetermined time interval.

[0039] The detection unit 931 detects the face area by inputting the face image to a classifier that detects a face area corresponding to the driver's face from the face image. Furthermore, the detection unit 931 specifies an eye peripheral area in the face area where the driver's eyes are assumed to be represented, based on the face structure. The detection unit 931 then performs template matching between templates representing the upper and lower eyelids and the specified eye peripheral area image to specify the positions of the upper and lower eyelids and obtain the distance between the upper and lower eyelids.

[0040] The detection unit 931 detects a closed eye state when the distance between the upper and lower eyelids obtained from the face image does not exceed a predetermined distance threshold, and associates closed eye data (e.g., "2") representing a closed eye state with the time when the face image was generated, and stores the data in the memory 92. The detection unit 931 detects an open eye state when the distance between the upper and lower eyelids obtained from the face image exceeds a predetermined distance threshold, and associates open eye data (e.g., "1") representing an open eye state with the time when the face image was generated, and stores the data in the memory 92. The detection unit 931 associates undeterminable data (e.g., "0") representing undeterminable with the time when the face image was generated, and stores the data in the memory 92 when a face area is not detected from the face image, an area around the eye is not identified in the face area, or the positions of the upper and lower eyelids are not identified from the area around the eye.

[0041] The first determination portion 932 determines whether a first eyes-closed ratio indicating the ratio of the number of face images in which the eyes are detected to the number of face images generated in a first period exceeds a first threshold value.

[0042] After it is determined that the first eye closure ratio exceeds the first threshold, the second judgment unit 933 judges whether a second eye closure ratio indicating the ratio of the number of face images in which a closed eye state is detected to the number of multiple face images generated in a second period longer than the first period exceeds a second threshold which is equal to or less than the first threshold.

[0043] FIG. 4 is a diagram showing an example of a transition between an open / closed eye state.

[0044] The eye open / closed state graph 100 is a graph of the facial image FP generated in time series by the driver monitor camera 2 at each of the times k-29-k. k-29 -Facial image FP k 4 shows the transition of the open / closed eye state detected from each of the graphs. In the open / closed eye state graph 100, the vertical axis represents the open / closed eye state (2 is closed eyes, 1 is open eyes, 0 is undeterminable), and the horizontal axis represents time. In the example of FIG. 4, face images are generated at one-second intervals. That is, the face image FP k-29 is face image FP k This is a face image generated 30 seconds before the generation time of the face image.

[0045] In the example of FIG. 4, the first period is 10 seconds from the reference time to 10 seconds before the reference time. k-10 10 facial images FP generated k-19 -Facial image FP k-10 Therefore, the first determination unit 932 determines that the eyes are closed in the first period T1 k-10 The first eye closure ratio corresponding to this is set to 10 / 10 = 1.0. Note that the length of the first period may be a length other than 10 seconds, for example, 8 seconds or 12 seconds.

[0046] 4, the first threshold is 0.6. Therefore, the first determination unit 932 determines that the first eye closure ratio corresponding to the time k-10 exceeds the first threshold. Note that the first threshold may be a value other than 0.6, for example, 0.5 or 0.7.

[0047] The first period T1 when the time k, which is 10 seconds corresponding to the first period, has elapsed from the time k-10, is used as the reference. k 10 facial images FP generated k-9 -Facial image FP k Of these, five images detect the eyes closed, and the remaining five images are deemed undetectable. Therefore, the first determination unit 932 determines whether the eyes are closed during the first period T1 k The first eye closure ratio corresponding to this is set to 5 / 10=0.5, and it is determined that this does not exceed the first threshold.

[0048] First period T1 based on time k k After it is determined that the first eye closure ratio corresponding to the time k does not exceed the first threshold, the second determination unit 933 determines whether the second period T2 k It is determined whether the second eye closure ratio corresponding to exceeds a second threshold value.

[0049] In the example of FIG. 4, the second period is 30 seconds from the reference time to 30 seconds before the reference time. k 30 facial images generated in FP k-29 -FP k Among them, 21 images detect the eye closed state, 2 images detect the eye open state, and 7 images are undetectable. Therefore, the second determination unit 933 determines whether the second period T2 k The corresponding second eye closure ratio is 21 / 30 = 0.7.

[0050] The length of the second period may be a length other than 30 seconds, for example, 25 seconds or 35 seconds. The reference time of the second period may be a time different from the reference time of the first period. For example, the reference time of the second period may be the time when the determination for the first period ends.

[0051] The second determination unit 933 may change the length of the second period depending on the autonomous driving level at the reference time. For example, the second determination unit 933 acquires the autonomous driving level for controlling the driving of the vehicle 1 from the driving control device 8. When the driving control device 8 and the drowsiness sign notification device 9 are implemented in different ECUs, the second determination unit 933 inquires of the autonomous driving level from the driving control device 8 via the communication interface 91. When the driving control device 8 and the drowsiness sign notification device 9 are implemented in the same ECU, the second determination unit 933 reads out the autonomous driving level used for the driving control of the vehicle 1 recorded in a predetermined area of ​​the memory 92. Then, the second determination unit 933 sets the second period when the autonomous driving level is an autonomous driving level that does not require the driver to monitor the surroundings of the vehicle to be longer (for example, 60 seconds) than the second period when the autonomous driving level is an autonomous driving level that requires the driver to monitor the surroundings of the vehicle. In this way, the drowsiness sign notification device 9 can suppress intermittent notifications due to an inability to make a determination when the driver's posture changes, which is likely to occur when driving is controlled at an autonomous driving level that does not require monitoring by the driver.

[0052] The second determination unit 933 may change the length of the second period depending on the amount of external light. For example, the second determination unit 933 receives a light amount signal representing the amount of external light entering the vehicle interior from outside the vehicle 1 from the light sensor 5 via the communication interface 91. The second determination unit 933 sets the second period when the amount of external light represented by the received light amount signal exceeds the external light amount threshold stored in the memory 92 to be longer (for example, 60 seconds) than the second period when the amount of external light does not exceed the external light amount threshold. In this way, the drowsiness sign notification device 9 can suppress intermittent notification due to inability to make a determination due to strong external light.

[0053] 4, the second threshold is equal to the first threshold, ie, 0.6, so the second determination unit 933 determines that the second eye closure proportion corresponding to the time k exceeds the second threshold.

[0054] The second determination unit 933 may set the second threshold to a value smaller than the first threshold (for example, 0.5 for the first threshold of 0.6). In this way, the drowsiness sign notification device 9 can more appropriately suppress discontinuance of notification due to inability to make a determination.

[0055] For example, the second determination unit 933 may set the second threshold value at an autonomous driving level that does not require monitoring by the driver to be smaller than the second threshold value at an autonomous driving level that requires monitoring of the surroundings of the vehicle by the driver. In this way, the drowsiness sign notification device 9 can suppress intermittent notification due to inability to make a determination caused by the driver's change in posture or gazing at a direction other than the traveling direction (e.g., a car navigation device), which is likely to occur when driving is controlled at an autonomous driving level that does not require monitoring of the surroundings of the vehicle by the driver.

[0056] Furthermore, the second determination unit 933 may set the second threshold value when the vehicle 1 is traveling by manual driving based on the driver's operation to be smaller than the second threshold value when the vehicle is traveling by automatic driving. In this way, the drowsiness sign notification device 9 can suppress intermittent notification due to inability to make a determination when the driver changes his or her line of sight to a direction other than the traveling direction, which is likely to occur during manual driving.

[0057] The second determination unit 933 may set the second threshold value when the driver is wearing glasses or sunglasses to be smaller than the second threshold value when the driver is not wearing glasses or sunglasses. In this way, the drowsiness sign notification device 9 can suppress intermittent notification due to the inability to make a determination caused by the frame of the glasses or sunglasses overlapping with the eyes. Note that whether the driver is wearing glasses or sunglasses can be detected by, for example, the detection unit 931 performing template matching between a template showing the frame of the glasses or sunglasses and an image of the peripheral area of ​​the eyes.

[0058] Returning to FIG. 3, when it is determined that the first eye closure ratio exceeds the first threshold value or the second eye closure ratio exceeds the second threshold value, the notification unit 934 notifies the driver that a drowsiness sign has been detected. The notification unit 934 transmits a display signal to the meter display 3 via the communication interface 91 to display a screen notifying the driver that a drowsiness sign has been detected. The screen notifying the driver that a drowsiness sign has been detected may include a character string indicating that a drowsiness sign has been detected, such as "Drowsiness sign has been confirmed," and may also include information encouraging a break, such as a character string such as "Would you like to take a break?" or an image representing a coffee break. In addition to displaying a screen on the meter display 3, the notification unit 934 may output a predetermined sound (such as a buzzer sound) from a speaker (not shown) as a notification to the driver that a drowsiness sign has been detected.

[0059] 5 is a flowchart of the drowsiness sign notification process. The processor 93 of the drowsiness sign notification device 9 executes the drowsiness sign notification process described below at predetermined time intervals (for example, every second) while the vehicle 1 is traveling.

[0060] First, the detection unit 931 of the processor 93 of the drowsiness sign notification device 9 detects whether the driver's eyes are closed from a face image acquired from the driver monitor camera 2 (step S1). In step S1, the detection unit 931 detects whether the driver's eyes are closed from each of a plurality of face images generated in a first period based on a predetermined time.

[0061] Next, the first determination unit 932 of the processor 93 determines whether or not a first eye closure ratio indicating the ratio of the number of face images in which the eye closure state is detected to the number of face images generated in the first period exceeds a first threshold value (step S2). If it is determined that the first eye closure ratio exceeds the first threshold value (step S2: Y), the process of the processor 93 proceeds to step S4 described later.

[0062] If it is determined that the first eye closure ratio does not exceed the first threshold (step S2: N), the second determination unit 933 of the processor 93 determines whether or not a second eye closure ratio indicating the ratio of the number of face images in which the eye closed state is detected to the number of face images generated in a second period longer than the first period exceeds a second threshold (step S3). If it is determined that the second eye closure ratio does not exceed the second threshold (step S3: N), the process of the processor 93 returns to step S1, and executes the processes from step S1 onwards for the first period based on a time that is the length of the first period after the predetermined time in the previous step S1.

[0063] If it is determined that the second eye closure ratio exceeds the second threshold (step S3: Y), the notification unit 934 of the processor 93 notifies the driver that a sign of drowsiness has been detected (step S4). After executing step S4, the process of the processor 93 returns to step S1, and executes the processes from step S1 onward for the first period based on a time that is the length of the first period after the predetermined time in the previous step S1.

[0064] By executing the drowsiness sign notification process in this manner, the drowsiness sign notification device 9 can appropriately notify the driver of the vehicle of a drowsiness sign.

[0065] It should be understood that those skilled in the art can make various changes, substitutions, and alterations thereto without departing from the spirit and scope of the present disclosure. [Explanation of symbols]

[0066] 1 vehicle 9. Drowsiness Signs Notification Device 931 Detection Unit 932 1st Judgment Department 933 Second Judgment Department 934 Notification Department

Claims

1. a detection unit that detects a closed eye state of a driver of a vehicle from each of a plurality of face images that represent a face area of ​​the driver and are generated at different times; a first determination unit that determines whether a first eye closure ratio indicating a ratio of a number of face images in which the eye closure state is detected to a number of the plurality of face images generated in a first period exceeds a first threshold; a second determination unit that, after it is determined that the first eye closure ratio does not exceed the first threshold, determines whether a second eye closure ratio indicating a ratio of a number of face images in which the eye closure state is detected to a number of the plurality of face images generated in a second period longer than the first period exceeds a second threshold that is equal to or smaller than the first threshold; a notification unit that notifies the driver that a sign of drowsiness has been detected via a notification device provided in the vehicle when it is determined that the first eye closure percentage exceeds the first threshold value or the second eye closure percentage exceeds the second threshold value; and A drowsiness sign notification device comprising:

2. The drowsiness sign notification device of claim 1, wherein the second determination unit makes the second period longer when the autonomous driving level at which a driving control device provided in the vehicle controls the driving of the vehicle is an autonomous driving level that does not require the driver to monitor the surroundings of the vehicle than the second period when the autonomous driving level is an autonomous driving level that requires the driver to monitor the surroundings of the vehicle.

3. 2. The drowsiness sign notification device according to claim 1, wherein the second determination unit sets the second period when an amount of external light entering the vehicle cabin from outside the vehicle detected by an external light sensor provided in the vehicle exceeds an external light threshold to be longer than the second period when the amount of external light does not exceed the external light threshold.

4. The drowsiness sign notification device according to claim 1 , wherein the second determination section sets the second threshold value to be smaller than the first threshold value.

5. The drowsiness sign notification device of claim 4, wherein the second judgment unit sets the second threshold value when the autonomous driving level at which a driving control device provided in the vehicle controls the driving of the vehicle is an autonomous driving level that does not require the driver to monitor the surroundings of the vehicle to be smaller than the second threshold value when the autonomous driving level is an autonomous driving level that requires the driver to monitor the surroundings of the vehicle.

6. A drowsiness sign notification device that notifies that a drowsiness sign of a driver of a vehicle has been detected, detecting a closed eye state of the driver from each of a plurality of face images representing a face area of ​​the driver and generated at different times; determining whether a first eye closure ratio indicating a ratio of the number of face images in which the eye closure state is detected to the number of the plurality of face images generated in a first period exceeds a first threshold value; after it is determined that the first eye closure ratio does not exceed the first threshold, it is determined whether a second eye closure ratio indicating a ratio of the number of face images in which the eye closure state is detected to the number of the plurality of face images generated in a second period longer than the first period exceeds a second threshold which is equal to or less than the first threshold, When it is determined that the first eye closure percentage exceeds the first threshold value or the second eye closure percentage exceeds the second threshold value, the driver is notified that a sign of drowsiness has been detected via a notification device provided in the vehicle. A drowsiness symptom notification method comprising:

7. detecting a closed eye state of a vehicle driver from each of a plurality of face images representing a face area of ​​the driver and generated at different times; determining whether a first eye closure ratio indicating a ratio of a number of face images in which the eye closure state is detected to a number of the plurality of face images generated in a first period exceeds a first threshold value; after it is determined that the first eye closure ratio does not exceed the first threshold, determining whether a second eye closure ratio indicating a ratio of a number of face images in which the eye closure state is detected to a number of the plurality of face images generated in a second period longer than the first period exceeds a second threshold which is equal to or less than the first threshold; notifying the driver that a sign of drowsiness has been detected via a notification device provided in the vehicle when it is determined that the first eye closure percentage exceeds the first threshold value or the second eye closure percentage exceeds the second threshold value; A computer program for notifying a user of signs of drowsiness, which causes a computer to execute the above.

Images