Driver monitoring device, driver monitoring method, and computer program for driver monitoring

The driver monitoring device uses facial image analysis to differentiate between dozing, talking, and yawning through eye and mouth movement thresholds, addressing erroneous determinations and reducing uncomfortable warnings.

JP7589614B2Active Publication Date: 2024-11-26TOYOTA JIDOSHA KK
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2021051353
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-03-25
Publication Date
2024-11-26
Estimated Expiration
2041-03-25

AI Technical Summary

Technical Problem

Existing driver monitoring systems erroneously determine that a driver is dozing off due to factors other than actual sleep, such as smiling or talking, leading to uncomfortable warnings.

Method used

A driver monitoring device that uses facial image analysis to differentiate between dozing, talking, and yawning by detecting eye and mouth movements, setting thresholds for mouth and eye opening frequencies, and considering conversation states to avoid erroneous determinations.

Benefits of technology

The system effectively suppresses false alarms by distinguishing between dozing, talking, and yawning, ensuring accurate driver condition assessment and reducing unnecessary warnings.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007589614000001
    Figure 0007589614000001
  • Figure 0007589614000002
    Figure 0007589614000002
  • Figure 0007589614000003
    Figure 0007589614000003
Patent Text Reader

Abstract

To provide a driver monitoring device, a driver monitoring method and a computer program for driver monitoring in which an erroneous determination of a driver's doze is suppressed.SOLUTION: A driver monitoring device is provided with: a doze determination part 51 that detects an open-close state of eyes of a driver based on a face image in which the face of the driver in a vehicle 10 is represented to determine whether or not the driver is dozing based on the open-close state of the eyes; and a conversation determination part 52 that determines whether or not the driver is having a conversation. The doze determination part does not determine that the driver is dozing while being determined that the driver is having a conversation.SELECTED DRAWING: Figure 6
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a driver monitoring device, a driver monitoring method, and a computer program for driver monitoring. [Background technology]

[0002] It is known that a driver's state is monitored using an imaging device provided in a vehicle (for example, Patent Documents 1 and 2). For example, when a state in which the driver's eyes are continuously closed is detected based on a facial image of the driver generated by the imaging device, the driver is determined to be dozing off, and a warning is given to the driver. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-149499 A [Patent Document 2] Patent No. 6689470 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there is a possibility that the driver's eyes are determined to be closed due to factors other than drowsiness. For example, typically, when a person laughs, their eyes narrow, resulting in a smaller degree of eye opening. Therefore, when the driver is laughing, the driver is determined to be dozing, and the warning at this time may cause the driver to feel uncomfortable.

[0005] In view of the above problem, an object of the present invention is to suppress erroneous determination that the driver is dozing off. [Means for solving the problem]

[0006] The gist of the present disclosure is as follows.

[0007] (1) A driver monitoring device comprising: a drowsing determination unit that detects whether the driver's eyes are open or closed based on a facial image showing the face of the driver of a vehicle, and determines whether the driver is dozing or not based on the eye opening or closing state; and a conversation determination unit that determines whether the driver is talking or not, wherein the drowsing determination unit does not determine that the driver is dozing while it is determined that the driver is talking.

[0008] (2) The driver monitoring device according to (1) above, wherein the drowsiness determination unit does not determine that the driver is drowsy until a predetermined time has elapsed since it was determined that the driver's conversation has ended.

[0009] (3) The driver monitoring device according to (1) or (2) above, wherein the conversation determination unit determines whether the driver is having a conversation based on the facial image.

[0010] (4) The driver monitoring device described in (3) above, wherein the conversation determination unit detects the driver's mouth opening degree based on the facial image, and determines that the driver is having a conversation when the frequency at which the change in the mouth opening degree becomes equal to or greater than a predetermined first threshold value is equal to or greater than a predetermined first reference frequency.

[0011] (5) A driver monitoring device as described in any one of (1) to (4) above, further comprising a yawning determination unit that determines whether the driver is yawning based on the facial image, wherein the yawning determination unit detects the degree of eye opening and mouth opening of the driver based on the facial image, and determines that the driver is yawning when the frequency at which the eye opening degree becomes equal to or lower than a predetermined lower threshold when the degree of eye opening is equal to or higher than a predetermined upper threshold and the change in the degree of eye opening becomes equal to or higher than a predetermined second threshold is less than a predetermined second standard frequency.

[0012] (6) A driver monitoring method including: detecting whether the eyes of a vehicle driver are open or closed based on a facial image showing the face of the driver; determining whether the driver is dozing off based on the eye open or closed state; determining whether the driver is talking; and not determining that the driver is dozing off while it is determined that the driver is talking.

[0013] (7) A computer program for driver monitoring that causes a computer to execute the following operations: detect whether the eyes of a vehicle driver are open or closed based on a facial image showing the face of the driver; determine whether the driver is dozing off based on the eye opening or closing state; determine whether the driver is talking; and not determine that the driver is dozing off while it is determined that the driver is talking. Effect of the Invention

[0014] According to the present invention, it is possible to suppress erroneous determination that the driver is dozing off. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a schematic configuration diagram of a vehicle control system including a driver monitoring device according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing a schematic view of the interior of a vehicle in which an imaging device is provided. [Diagram 3] FIG. 3 is a diagram illustrating a schematic configuration of the ECU. [Figure 4] FIG. 4 is a functional block diagram of a processor of the ECU in the first embodiment. [Diagram 5] FIG. 5 is a flowchart showing a control routine of the conversation determination process in the first embodiment of the present invention. [Figure 6] FIG. 6 is a flowchart showing a control routine for the drowsiness determination process in the first embodiment of the present invention. [Figure 7]FIG. 7 is a flowchart showing a control routine of the conversation determination process in the second embodiment of the present invention. [Figure 8] FIG. 8 is a functional block diagram of a processor of an ECU in the third embodiment. [Figure 9] FIG. 9 is a flowchart showing a control routine of the yawning determination process in the third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, like components are designated by like reference numerals.

[0017] First Embodiment A first embodiment of the present invention will be described below with reference to Fig. 1 to Fig. 6. Fig. 1 is a schematic configuration diagram of a vehicle control system 1 equipped with a driver monitoring device according to the first embodiment of the present invention. The vehicle control system 1 is mounted on a vehicle 10 and controls the vehicle 10. As shown in Fig. 1, the vehicle control system 1 includes an imaging device 2, a human machine interface (HMI) 3, and an electronic control unit (ECU) 4.

[0018] The imaging device 2 captures an image of the face of the driver of the vehicle 10 and generates a facial image showing the face of the driver. The imaging device 2 is electrically connected to the ECU 4, and the output of the imaging device 2, i.e., the facial image generated by the imaging device 2, is transmitted to the ECU 4. The imaging device 2 is also called a driver monitor camera. A specific configuration example of the imaging device 2 will be described below.

[0019] The imaging device 2 has a camera and a floodlight. The camera is composed of a lens and an imaging element, for example a CMOS (complementary metal oxide semiconductor) camera or a CCD (charge-coupled device) camera. The floodlight is an LED (light-emitting diode), for example two near-infrared LEDs arranged on both sides of the camera. By irradiating the driver with near-infrared light, the driver's face can be photographed without causing discomfort to the driver even in low-illumination conditions such as at night. In addition, a bandpass filter that removes light of wavelength components other than near-infrared is provided inside the camera, and a visible light cut filter that removes light of red wavelength components irradiated from the near-infrared LED is provided in front of the floodlight.

[0020] Fig. 2 is a diagram showing a schematic view of the interior of a vehicle 10 provided with an imaging device 2. The imaging device 2 is provided in the vehicle cabin so as to capture an image of the face of the driver of the vehicle 10. For example, as shown in Fig. 2, the imaging device 2 is provided on an upper portion of a steering column 11 of the vehicle 10. In Fig. 2, the projection range of the imaging device 2 is indicated by a dashed line. The imaging device 2 may be provided on the steering wheel 12, the room mirror, the meter panel, the meter hood, etc. of the vehicle 10. Furthermore, the vehicle control system 1 may include a plurality of imaging devices.

[0021] The HMI 3 inputs and outputs information between the driver and the vehicle 10. The HMI 3 includes, for example, a display that displays information, a speaker that generates sound, an operation button, an operation switch or a touch screen that allows the driver to perform an input operation, and a microphone that receives the driver's voice. The HMI 3 is electrically connected to the ECU 4. Therefore, the output of the ECU 4 is transmitted to the driver via the HMI 3, and the input from the driver is transmitted to the ECU 4 via the HMI 3. The HMI 3 is an example of an input device, an output device, or an input / output device.

[0022] Fig. 3 is a diagram showing a schematic configuration of the ECU 4. The ECU 4 executes various controls of the vehicle. As shown in Fig. 3, the ECU 4 includes a communication interface 41, a memory 42, and a processor 43. The communication interface 41 and the memory 42 are connected to the processor 43 via signal lines.

[0023] The communication interface 41 has an interface circuit for connecting the ECU 4 to an in-vehicle network that complies with a standard such as CAN (Controller Area Network). The ECU 4 communicates with in-vehicle devices connected to the in-vehicle network via the communication interface 41 and the in-vehicle network.

[0024] The memory 42 includes, for example, a volatile semiconductor memory (for example, a RAM) and a non-volatile semiconductor memory (for example, a ROM). The memory 42 stores computer programs executed by the processor 43, various data used when the processor 43 executes various processes, and the like.

[0025] The processor 43 has one or more CPUs (Central Processing Units) and their peripheral circuits, and executes various processes. The processor 43 may further have other arithmetic circuits such as a logic arithmetic unit, a numerical arithmetic unit, or a graphic processing unit.

[0026] In this embodiment, the ECU 4 functions as a driver monitoring device that monitors the state of the driver of the vehicle 10. FIG. 4 is a functional block diagram of the processor 43 of the ECU 4 in the first embodiment. In this embodiment, the processor 43 has a drowsiness determination unit 51 and a conversation determination unit 52. The drowsiness determination unit 51 and the conversation determination unit 52 are functional modules that are realized by the processor 43 of the ECU 4 executing a computer program stored in the memory 42 of the ECU 4. Note that these functional modules may be realized by dedicated arithmetic circuits provided in the processor 43. In addition, these functional modules may be realized by multiple ECUs connected to each other via an in-vehicle network. That is, multiple ECUs may function as a driver monitoring device.

[0027] As described above, the face image generated by the imaging device 2, i.e., the face image showing the face of the driver of the vehicle 10, is transmitted to the ECU 4. The drowsiness determination unit 51 detects whether the driver's eyes are open or closed based on the face image, and determines whether the driver is dozing or not based on the eye opening or closing state.

[0028] For example, the degree of eye opening of the driver is used as an index representing the open / closed state of the driver's eyes. In this case, the dozing-at-the-wheel determination unit 51 detects the degree of eye opening of the driver based on a face image. For example, the dozing-at-the-wheel determination unit 51 identifies a face area from the face image and detects face parts such as eyes, nose, and mouth. Then, the dozing-at-the-wheel determination unit 51 extracts feature points of the upper eyelid and the lower eyelid for each of the right eye and the left eye, and calculates the distance between the two feature points as the degree of eye opening. The degree of eye opening detected from one face image is calculated as, for example, the average value of the degree of eye opening of the right eye and the degree of eye opening of the left eye.

[0029] A classifier for image recognition may be used to detect the degree of eye opening of the driver. That is, the drowsiness determination unit 51 may detect the degree of eye opening of the driver by inputting a facial image to a classifier that has been trained in advance to output the degree of eye opening from a facial image.

[0030] The dozing determination unit 51 determines that the driver has his / her eyes closed when the degree of eye opening of the driver is equal to or less than a predetermined reference value. The reference value is determined in advance using an image of an unspecified number of people or a driver sleeping. The dozing determination unit 51 also determines that the driver is dozing when the eye closing state is maintained for a predetermined reference time or more, that is, when the degree of eye opening of the driver is maintained at a value equal to or less than the reference value for the reference time or more. The reference time is determined in advance to be longer than the time for closing the eyes by blinking, and is set to, for example, 1 to 7 seconds.

[0031] The dozing determination unit 51 may detect the open / closed state of the driver's eyes by inputting the face image to a classifier that has been trained in advance to classify the state of the driver's eyes into an open or closed state based on the face image. In this case, the dozing determination unit 51 also determines that the driver is dozing when the closed eye state is maintained for a reference time or longer. As a classifier that outputs the degree of eye opening or the state of the eyes, for example, a machine learning model such as a neural network (e.g., a Convolutional Neural Network (CNN)), a support vector machine, or a random forest is used.

[0032] When the drowsiness determination unit 51 determines that the driver is dozing, it warns the driver by outputting at least one of a warning display and a warning sound via the HMI 3. This makes it possible to prevent the driver from dozing.

[0033] On the other hand, the conversation determination unit 52 determines whether or not the driver of the vehicle 10 is talking. For example, the conversation determination unit 52 determines whether or not the driver is talking based on the face image generated by the imaging device 2, that is, the face image showing the face of the driver of the vehicle 10.

[0034] When a person is talking, the mouth frequently opens and closes according to the movement of the mouth, and the degree of mouth opening changes. Therefore, the conversation determination unit 52 detects the degree of mouth opening of the driver based on the face image, and determines whether the driver is talking or not based on the degree of mouth opening. For example, the conversation determination unit 52 identifies a face area from the face image, and detects face parts such as eyes, nose, and mouth. Then, the conversation determination unit 52 extracts feature points of the upper lip and the lower lip, and calculates the distance between the two feature points as the degree of mouth opening.

[0035] A classifier for image recognition may be used to detect the driver's mouth opening degree. That is, the conversation determination unit 52 may detect the driver's mouth opening degree by inputting a facial image to a classifier that has been trained in advance to output the mouth opening degree from a facial image.

[0036] The conversation determination unit 52 determines that the driver is talking when the frequency at which the change in the mouth opening degree is equal to or greater than a predetermined first threshold is equal to or greater than a predetermined first reference frequency. The first threshold and the first reference frequency are determined in advance using a series of facial images in time series when an unspecified number of people or the driver is talking, or the like.

[0037] The conversation determination unit 52 may determine whether the driver is talking by inputting a series of time-series facial images to a classifier that has been trained in advance to classify the driver's state into a talking state or a non-talking state based on the facial images. As a classifier that outputs the mouth opening degree or the driver's state, for example, a machine learning model such as a neural network (for example, a Convolutional Neural Network (CNN) or a Recurrent Neural Network (RNN)), a support vector machine, or a random forest is used.

[0038] However, even if it is detected that the driver's eyes are closed as described above, it does not necessarily mean that the driver is dozing. For example, typically, when a person laughs, their eyes narrow, which results in a smaller degree of eye opening. Therefore, when the driver is laughing, it is determined that the driver is dozing, and a warning at this time may cause the driver to feel uncomfortable.

[0039] Therefore, it is desirable to be able to distinguish between closed eyes due to drowsiness and closed eyes due to smiling. For example, a situation in which a driver smiles is assumed in which the driver is talking to someone. Therefore, a closed eye state detected during a conversation is likely to be caused by smiling.

[0040] Therefore, in this embodiment, the drowsiness determination unit 51 does not determine that the driver is dozing while the conversation determination unit 52 determines that the driver is talking. That is, even if the driver's eyes are kept closed, the drowsiness determination unit 51 does not determine that the driver is dozing as long as the driver is determined to be talking. In this case, even if the driver's smile due to conversation is determined to be in a closed eye state, the driver is not determined to be dozing. Therefore, it is possible to suppress erroneous determination that the driver is dozing.

[0041] <Conversation determination process> The above-mentioned control flow will be described below with reference to Figures 5 and 6. Figure 5 is a flowchart showing a control routine for conversation determination processing in the first embodiment of the present invention. This control routine is repeatedly executed by the ECU 4 at a predetermined execution interval.

[0042] First, in step S101, the conversation determination unit 52 acquires a series of chronological face images generated by the imaging device 2 during a predetermined period up to the present. This series of face images is stored, for example, in the memory 42 of the ECU 4. In the memory 42, the oldest face image is deleted every time a face image is transmitted from the imaging device 2 to the ECU 4, so that a predetermined number of the latest face images corresponding to the images generated during the predetermined period up to the present are stored in the memory 42.

[0043] Next, in step S102, conversation determination unit 52 determines whether the frequency at which the change in the driver's mouth degree becomes equal to or greater than the first threshold is equal to or greater than a first reference frequency, based on a series of chronological face images. For example, conversation determination unit 52 detects the mouth degree for each of the series of face images, and determines that the frequency at which the change in the mouth degree becomes equal to or greater than the first threshold is equal to or greater than the first reference frequency when the number of times that the mouth degree changes between a lower threshold and an upper threshold is equal to or greater than a predetermined number of times.

[0044] When it is determined in step S102 that the frequency at which the change in the mouth opening degree is equal to or greater than the first threshold value is equal to or greater than the first reference frequency, this control routine proceeds to step S103. In step S103, the conversation determination unit 52 determines that the driver is talking.

[0045] Next, in step S104, the conversation determination unit 52 sets the dozing determination prohibition flag F to 1. After step S104, this control routine ends.

[0046] On the other hand, if it is determined in step S102 that the frequency at which the change in the mouth opening degree is equal to or greater than the first threshold value is less than the first reference frequency, the control routine proceeds to step S105. In step S105, the conversation determination unit 52 sets the dozing determination prohibition flag F to zero. After step S105, the control routine ends.

[0047] In step S102, the conversation determination unit 52 may determine whether or not the driver is talking by inputting a series of time-series face images to a classifier, without detecting the mouth opening degree.

[0048] <Drowsy driving detection process> 6 is a flowchart showing a control routine for the drowsiness determination process in the first embodiment of the present invention. This control routine is repeatedly executed by the ECU 4 at predetermined execution intervals.

[0049] First, in step S201, the drowsiness determination unit 51 acquires a face image generated by the imaging device 2.

[0050] Next, in step S202, the dozing determination unit 51 determines whether the driver has his / her eyes closed based on the face image. Specifically, the dozing determination unit 51 determines that the driver has his / her eyes closed when the degree of eye opening of the driver is equal to or less than a reference value, and determines that the driver has his / her eyes open when the degree of eye opening of the driver is greater than the reference value. Note that the dozing determination unit 51 may determine whether the driver has his / her eyes closed by inputting the face image to a classifier without detecting the degree of eye opening.

[0051] If it is determined in step S202 that the driver's eyes are closed, that is, if it is determined that the driver's eyes are closed, this control routine proceeds to step S203. In step S203, the drowsiness determination unit 51 determines whether the eyes have been closed for a reference time or more. If it is determined in step S203 that the eyes have been closed for a reference time or more, this control routine proceeds to step S204.

[0052] In step S204, the doze determination unit 51 determines whether or not the doze determination prohibition flag F is set to 0. If it is determined that the doze determination prohibition flag F is set to 0, this control routine proceeds to step S205.

[0053] In step S205, the drowsiness determination unit 51 determines that the driver is dozing. Next, in step S206, the drowsiness determination unit 51 issues a warning to the driver by outputting at least one of a warning display and a warning sound by the HMI 3. Note that the drowsiness determination unit 51 may issue a warning to the driver by other means, for example, by vibrating the seat belt of the driver's seat. After step S206, this control routine ends.

[0054] On the other hand, if it is determined in step S202 that the driver has his / her eyes open, if it is determined in step S203 that the eyes have not been closed for the reference time or longer, or if it is determined in step S204 that the dozing-at-the-wheel determination prohibition flag F is set to 1, this control routine ends. In this case, the dozing-at-the-wheel determination unit 51 does not determine that the driver is dozing.

[0055] Note that step S204 may be executed before step S201. That is, the drowsiness determination unit 51 does not need to detect whether the driver's eyes are open or closed while it is determined that the driver is having a conversation.

[0056] Second Embodiment The driver monitoring device according to the second embodiment is basically similar in configuration and control to the driver monitoring device according to the first embodiment, except for the points described below. Therefore, the second embodiment of the present invention will be described below, focusing on the differences from the first embodiment.

[0057] In the first embodiment, the determination of whether the driver is dozing is prohibited while the driver is talking. However, even if the driver is not talking, the driver may laugh due to the conversation of the passenger. This situation is particularly likely to occur immediately after the driver has had a conversation. In addition, the driver is unlikely to doze off immediately after having a conversation.

[0058] Therefore, in the second embodiment, the drowsiness determination unit 51 does not determine that the driver is dozing not only while it is determined that the driver is talking, but also until a predetermined time has elapsed since it was determined that the driver has finished talking, thereby making it possible to further suppress erroneous determination that the driver is dozing.

[0059] <Conversation determination process> 7 is a flowchart showing a control routine of a conversation determination process in the second embodiment of the present invention. This control routine is repeatedly executed by the ECU 4 at a predetermined execution interval.

[0060] Steps S301 to S304 are executed in the same manner as steps S101 to S104 in Fig. 5. On the other hand, if it is determined in step S302 that the frequency at which the amount of change in the mouth opening degree becomes equal to or greater than the first threshold value is less than the first reference frequency, this control routine proceeds to step S305.

[0061] In step S305, the conversation determination unit 52 determines whether or not a predetermined time has elapsed since the end of the driver's conversation. The predetermined time is set to, for example, 5 to 20 seconds. Note that the conversation determination unit 52 determines that the driver's conversation has ended when the result of the determination in step S302 changes from positive to negative.

[0062] If it is determined in step S305 that the predetermined time has not elapsed since the end of the driver's conversation, this control routine proceeds to step S304. In step S304, the conversation determination unit 52 sets the drowsiness determination prohibition flag F to 1. After step S304, this control routine ends.

[0063] On the other hand, if it is determined in step S305 that the predetermined time has not elapsed since the end of the driver's conversation, this control routine proceeds to step S306. In step S306, the conversation determination unit 52 sets the drowsiness determination prohibition flag F to zero. After step S306, this control routine ends.

[0064] In the second embodiment, similarly to the first embodiment, the control routine of the drowsiness determination process in FIG. 6 is executed.

[0065] <Third embodiment> The driver monitoring device according to the third embodiment is basically similar in configuration and control to the driver monitoring device according to the first embodiment, except for the points described below. Therefore, the third embodiment of the present invention will be described below, focusing on the differences from the first embodiment.

[0066] 8 is a functional block diagram of the processor 43 of the ECU 4 in the third embodiment. In the third embodiment, the processor 43 has a yawning determination unit 53 in addition to the dozing determination unit 51 and the conversation determination unit 52. The dozing determination unit 51, the conversation determination unit 52, and the yawning determination unit 53 are functional modules that are realized by the processor 43 of the ECU 4 executing a computer program stored in the memory 42 of the ECU 4. Note that these functional modules may be realized by dedicated arithmetic circuits provided in the processor 43. These functional modules may also be realized by a plurality of ECUs connected to each other via an in-vehicle network. That is, a plurality of ECUs may function as a driver monitor device.

[0067] The yawn determination unit 53 determines whether the driver is yawning based on the face image generated by the imaging device 2, i.e., the face image showing the face of the driver of the vehicle 10. Usually, when a person yawns, the movement of the eyes is linked to the movement of the mouth, and the degree of opening of the eyes increases and decreases. Also, unlike conversation, when a person yawns, the mouth opens only once, and the mouth does not open and close continuously.

[0068] For this reason, the yawning determination unit 53 detects the degree of eye opening and the degree of mouth opening of the driver based on the face image, and determines that the driver is yawning when the degree of eye opening becomes equal to or smaller than a predetermined lower threshold when the degree of eye opening is equal to or larger than a predetermined upper threshold, and the frequency at which the change in the degree of eye opening becomes equal to or larger than a predetermined second threshold is less than a predetermined second reference frequency. This makes it possible to distinguish between yawning and dozing and to detect yawning with high accuracy. The degree of eye opening and the degree of mouth opening of the driver are detected by the method described above with respect to the dozing determination unit 51 and the conversation determination unit 52.

[0069] <Yawning detection process> In the third embodiment, in addition to the control routine of the conversation determination process in Fig. 5 and the control routine of the dozing determination process in Fig. 6, a control routine of the yawning determination process in Fig. 9 is executed. Fig. 9 is a flowchart showing the control routine of the yawning determination process in the third embodiment of the present invention. This control routine is repeatedly executed by the ECU 4 at a predetermined execution interval.

[0070] First, in step S401, similar to the process by the conversation determination section 52 in step S101 of FIG. 5, the yawn determination section 53 acquires a series of time-series face images generated by the imaging device 2 during a predetermined period up to the present.

[0071] Next, in step S402, the yawning determination unit 53 determines whether the driver's open-mouth degree has increased and the driver's eye-opening degree has decreased based on the series of chronological face images. Specifically, the yawning determination unit 53 detects the open-mouth degree and the eye-opening degree for each of the series of face images, and determines whether the eye-opening degree has become equal to or lower than the lower threshold when the open-mouth degree is equal to or higher than the upper threshold. If the determination in step S402 is negative, this control routine ends.

[0072] On the other hand, if it is determined in step S402 that the eye opening degree is equal to or greater than the upper threshold and the eye opening degree is equal to or less than the lower threshold, the control routine proceeds to step S403. In step S403, the yawning determination unit 53 determines whether or not the frequency at which the driver's eye opening degree changes to the second threshold or more is less than the second reference frequency based on a series of time-series face images. The second threshold and the second reference frequency are determined in advance using a series of time-series face images of an unspecified number of people or a driver yawning. The second threshold and the second reference frequency may be the same values ​​as the first threshold and the first reference frequency used for conversation determination. For example, the yawning determination unit 53 detects the eye opening degree for each of the series of face images, and determines that the frequency at which the eye opening degree changes to the second threshold or more is less than the second reference frequency when the number of times the eye opening degree changes between the lower threshold and the upper threshold is less than a predetermined number of times.

[0073] If it is determined in step S403 that the frequency at which the change in the mouth degree is equal to or greater than the second threshold is equal to or greater than the second reference frequency, this control routine ends. On the other hand, if it is determined in step S403 that the frequency at which the change in the mouth degree is equal to or greater than the second threshold is less than the second reference value, this control routine proceeds to step S404.

[0074] In step S404, the yawning determination unit 53 determines that the driver is yawning. Next, in step S405, the yawning determination unit 53 warns the driver by outputting at least one of a warning display and a warning sound by the HMI 3. The yawning determination unit 53 may warn the driver by other means, for example, by vibrating the seat belt of the driver's seat. The strength of the warning in the yawning determination may be set lower than the strength of the warning in the drowsiness determination. The yawning determination unit 53 may warn the driver when the number of times that the driver is determined to be yawning within a predetermined time reaches a predetermined number. After step S405, this control routine ends.

[0075] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the claims.

[0076] For example, the conversation determination unit 52 may determine whether the driver is talking based on an input signal to a microphone (e.g., a microphone of the HMI 3) provided in the vehicle cabin to capture the driver's voice. In this case, the conversation determination unit 52 determines that the driver is talking when, for example, the frequency of the input signal to the microphone is within a predetermined range of frequencies corresponding to a human voice.

[0077] The conversation determination unit 52 may also determine whether the driver is talking by inputting the input signal to the microphone to a classifier that has been trained in advance to classify the state of the driver into a talking state or a non-talking state based on the input signal to the microphone. In this case, a machine learning model such as a neural network (e.g., a recurrent neural network (RNN)), a support vector machine, or a random forest is used as the classifier.

[0078] Furthermore, a computer program for causing a computer to realize the functions of each unit of the processor of the driver monitoring device according to the above-described embodiment may be provided in a form stored in a computer-readable recording medium. The computer-readable recording medium is, for example, a magnetic recording medium, an optical recording medium, or a semiconductor memory.

[0079] In addition, the above-mentioned embodiments can be implemented in any combination. When the second embodiment and the third embodiment are combined, the control routine of the conversation determination process in FIG. 7, the control routine of the dozing determination process in FIG. 6, and the control routine of the yawn determination process in FIG. 9 are executed. [Explanation of symbols]

[0080] 10 Vehicles 4 Electronic Control Unit (ECU) 43 Processors 51 Drowsiness detection unit 52 Conversation determination unit

Claims

1. a dozing-off determination unit that detects an eye opening / closing state of a driver of a vehicle based on a facial image showing the face of the driver, and determines whether the driver is dozing off based on the eye opening / closing state; a conversation determination unit that determines whether the driver is having a conversation; Equipped with The driver monitoring device, wherein the drowsing determination unit does not determine that the driver is dozing while it is determined that the driver is having a conversation, regardless of whether the driver's eyes are opened or closed.

2. 2. The driver monitoring device according to claim 1, wherein the drowsiness determination unit does not determine that the driver is dozing off until a predetermined time has elapsed since it was determined that the driver's conversation has ended, regardless of whether the driver's eyes are open or closed.

3. The driver monitoring device according to claim 1 , wherein the conversation determination unit determines whether or not the driver is having a conversation based on the face image.

4. 4. The driver monitoring device according to claim 3, wherein the conversation determination unit detects the degree of mouth opening of the driver based on the facial image, and determines that the driver is talking when a frequency at which the amount of change in the degree of mouth opening becomes equal to or greater than a predetermined first threshold value is equal to or greater than a predetermined first reference frequency.

5. a yawn determination unit that determines whether the driver is yawning based on the face image, 5. The driver monitoring device according to claim 1, wherein the yawning determination unit detects the degree of eye opening and mouth opening of the driver based on the facial image, and determines that the driver is yawning if the frequency at which the eye opening degree becomes equal to or lower than a predetermined lower threshold when the eye opening degree is equal to or higher than a predetermined upper threshold and the amount of change in the eye opening degree becomes equal to or higher than a predetermined second threshold is less than a predetermined second reference frequency.

6. 1. A method of driver monitoring executed by a processor of a vehicle, comprising: Detecting an eye opening / closing state of a driver of a vehicle based on a facial image showing the face of the driver, and determining whether the driver is dozing off based on the eye opening / closing state; determining whether the driver is having a conversation; While it is determined that the driver is having a conversation, the driver is not determined to be dozing off, regardless of whether the driver's eyes are open or closed. A driver monitoring method comprising:

7. Detecting an eye opening / closing state of a driver of a vehicle based on a facial image showing the face of the driver, and determining whether the driver is dozing off based on the eye opening / closing state; determining whether the driver is having a conversation; While it is determined that the driver is having a conversation, the driver is not determined to be dozing off, regardless of whether the driver's eyes are open or closed. A computer program for driver monitoring that causes a computer to execute the above.

Citation Information

Patent Citations

  • Awakening support device

    JP2011118831A

  • Opened / closed eye determination device and opened / closed eye determination method

    JP2016115120A

  • Biological state determination apparatus, biological state determination system and biological state determination method

    JP2018143285A

  • Sleepiness estimation device

    JP2019111090A

  • Occupant observation device

    JP2020149499A