Driver monitoring device, driver monitoring method, and computer program for driver monitoring

Abstract

To increase safety for drivers who may be cognitively impaired.SOLUTION: A driver monitoring device includes: a facial expression determination unit 25 that determines the facial expression of a driver from an image generated by an image capturing device 2 that captures the face of a driver of a vehicle 30; and a processing unit 26 that calculates a frequency at which the driver's facial expression is determined to be an expression of surprise, and performs predetermined processing to enhance the driver's safety when the frequency is a predetermined threshold value or greater.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a driver monitoring device, a driver monitoring method, and a computer program for driver monitoring.

Background Art

[0002] Patent Document 1 describes detecting an involuntary reaction of a driver using at least one of imaging data obtained by imaging the driver's face before starting driving and speech sound data representing the driver's speech content, and diagnosing the risk level of driving by the driver based on the detection result.

[0003] Patent Document 2 describes determining the cognitive control ability of a driver based on the operation frequency of in-vehicle devices when the operation amount of a driving device exceeds a predetermined threshold value, and that the range of the line-of-sight scanning density becomes narrow when the driver's burden level is high and the driver's cognitive control ability is low.

[0004] Patent Document 3 describes calculating a risk score based on four judgment items related to the driving situation of a driver, and when the risk score exceeds a threshold value, prompting the driver to voluntarily return the driver's license and notifying a relevant person outside the vehicle of the decline in the driver's cognitive function.

[0005] Patent Document 4 describes specifying the recognizability of a driver with respect to a collision warning vehicle based on the driver's biological information, and performing a cognitive support output for the approach of the collision warning vehicle when the recognizability drops below a predetermined level.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Summary of the Invention

Problems to be Solved by the Invention

[0007] There is room for improvement in a method for accurately determining the cognitive function of a driver and enhancing the safety of a driver who may have a degraded cognitive function.

[0008] Therefore, in view of the above problems, an object of the present invention is to enhance the safety of a driver who may have a degraded cognitive function.

Means for Solving the Problems

[0009] The gist of the present disclosure is as follows.

[0010] (1) An expression determination unit that determines the expression of a driver from an image generated by an imaging device that captures the face of the driver of a vehicle, and a processing unit that calculates the frequency with which the expression of the driver is determined to be a surprised expression, and performs a predetermined process for enhancing the safety of the driver when the frequency is equal to or greater than a predetermined threshold value. A driver monitoring device.

[0011] (2) The driver monitoring device according to (1) above, wherein the processing unit sets the threshold value based on past data of the frequency.

[0012] (3) Further comprising an event detection unit that detects an event indicating inappropriate driving by the driver, and when the expression of the driver is determined to be a surprised expression and the event is detected, the processing unit reduces the threshold value, or increases the weight for the frequency when it is determined that the expression of the driver is a surprised expression when calculating the frequency. The driver monitoring device according to (1) or (2) above.

[0013] (4) The event detection unit of the driver monitoring device according to (3) above detects a dangerous driving scene as the event based on the surrounding environment information of the vehicle.

[0014] (5) The dangerous driving scene of the driver monitoring device according to (4) above includes a scene where the distance between the vehicle and a moving object around the vehicle is equal to or less than a predetermined distance.

[0015] (6) The event detection unit of the driver monitoring device according to any one of (3) to (5) above detects a sudden operation by the driver as the event based on the operation information of the vehicle by the driver.

[0016] (7) When it is determined that the driver's expression is a surprised expression and the frequency of the detected event reaches a predetermined value, the processing unit of the driver monitoring device according to any one of (3) to (6) above performs another process with a higher warning intensity than the predetermined process.

[0017] (8) The processing unit of the driver monitoring device according to any one of (1) to (7) above notifies the driver of a warning as the predetermined process.

[0018] (9) The processing unit of the driver monitoring device according to any one of (1) to (8) above notifies a person other than the driver of a warning as the predetermined process.

[0019] (10) The processing unit of the driver monitoring device according to any one of (1) to (9) above relaxes the implementation conditions of the driving support operation in the driving support function mounted on the vehicle as the predetermined process.

[0020] (11) The processing unit of the driver monitoring device according to any one of (1) to (10) above fixes the set vehicle-to-vehicle distance when the adaptive cruise control is activated in the vehicle to the maximum value as the predetermined process.

[0021] (12) A driver monitoring method executed by a computer, comprising: determining an expression of the driver from an image generated by an imaging device that captures an image of the face of the driver of a vehicle; calculating a frequency at which the expression of the driver is determined to be a surprised expression; and performing a predetermined process for enhancing the safety of the driver when the frequency is equal to or greater than a predetermined threshold value.

[0022] (13) A computer program for driver monitoring, which causes a computer to execute: determining an expression of the driver from an image generated by an imaging device that captures an image of the face of the driver of a vehicle; calculating a frequency at which the expression of the driver is determined to be a surprised expression; and performing a predetermined process for enhancing the safety of the driver when the frequency is equal to or greater than a predetermined threshold value.

Advantages of the Invention

[0023] According to the present invention, it is possible to enhance the safety of a driver who may have a degraded cognitive function.

Brief Description of the Drawings

[0024]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0025] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the following description, the same reference numerals are assigned to similar components.

[0026] <First Embodiment> Hereinafter, a first embodiment of the present invention will be described with reference to FIGS. 1 to 4. FIG. 1 is a schematic configuration diagram of a vehicle control system 1 including a driver monitoring device according to the first embodiment of the present invention. The vehicle control system 1 is mounted on a vehicle and executes various controls of the vehicle.

[0027] As shown in FIG. 1, the vehicle control system 1 includes an imaging device 2, a peripheral information detection device 3, a GNSS (Global Navigation Satellite System) receiver 4, a map database 5, a vehicle behavior detection device 6, a vehicle operation detection device 7, an actuator 8, an output device 9, a communication device 10, an occupant detection device 11, and an electronic control unit (ECU: Electronic Control Unit) 20. The imaging device 2, the peripheral information detection device 3, the GNSS receiver 4, the map database 5, the vehicle behavior detection device 6, the vehicle operation detection device 7, the actuator 8, the output device 9, the communication device 10, and the occupant detection device 11 are electrically connected to the ECU 20 via an in-vehicle network or the like conforming to a standard such as CAN (Controller Area Network).

[0028] The imaging device 2 captures an image of the face of the driver of the vehicle and generates an image representing the face of the driver. The output of the imaging device 2, that is, the image generated by the imaging device 2, is transmitted to the ECU 20. Note that the imaging device 2 is also referred to as a driver monitoring camera. Hereinafter, a specific example of the configuration of the imaging device 2 will be described.

[0029] The imaging device 2 includes a camera and a projector. The camera is composed of a lens and an imaging element, and is, for example, a CMOS (Complementary Metal Oxide Semiconductor) camera or a CCD (Charge Coupled Device) camera. The projector is an LED (Light Emitting Diode), and is, for example, two near-infrared LEDs arranged on both sides of the camera. By irradiating the driver with near-infrared light, it is possible to photograph the driver's face without causing discomfort to the driver even in low illumination conditions such as at night. Further, a band-pass filter for removing light of wavelength components other than near-infrared may be provided inside the camera, and a visible light cut filter for removing light of the red wavelength component irradiated from the near-infrared LED may be provided on the front surface of the projector.

[0030] Figure 2 is a diagram schematically showing the inside of the vehicle 30 in which the imaging device 2 is provided. The imaging device 2 is provided in the vehicle interior so as to photograph the face of the driver of the vehicle 30. For example, as shown in Figure 2, the imaging device 2 is provided at the upper part of the steering column 31 of the vehicle 30. In Figure 2, the projection range of the imaging device 2 is indicated by a broken line. Note that the imaging device 2 may be provided on the steering wheel 32, the rearview mirror, the meter panel, the meter hood, etc. of the vehicle 30. Further, the vehicle control system 1 may include a plurality of imaging devices.

[0031] The surrounding information detection device 3 acquires data (images, point cloud data, etc.) around the vehicle 30 and detects surrounding information of the vehicle 30 (for example, surrounding vehicles, lanes, pedestrians, bicycles, traffic lights, signs, etc.). For example, the surrounding information detection device 3 includes a camera (monocular camera or stereo camera), a millimeter wave radar, a lidar (Laser Imaging Detection And Ranging), or an ultrasonic sensor (sonar), or any combination thereof. Note that the surrounding information detection device 3 may further include an illuminance sensor, a rain sensor, etc. The output of the surrounding information detection device 3, that is, the surrounding information of the vehicle 30 detected by the surrounding information detection device 3 is transmitted to the ECU 20.

[0032] The GNSS receiver 4 detects the current position of the vehicle 30 (e.g., the latitude and longitude of the vehicle 30) based on positioning information obtained from a plurality (e.g., three or more) of positioning satellites. Specifically, the GNSS receiver 4 captures a plurality of positioning satellites and receives radio waves transmitted from the positioning satellites. Then, the GNSS receiver 4 calculates the distance to the positioning satellite based on the difference between the transmission time and reception time of the radio wave, and detects the current position of the vehicle 30 based on the distance to the positioning satellite and the position (orbital information) of the positioning satellite. The output of the GNSS receiver 4, i.e., the current position of the vehicle 30 detected by the GNSS receiver 4, is transmitted to the ECU 20. A GPS (Global Positioning System) receiver is an example of a GNSS receiver.

[0033] The map database 5 stores map information. The ECU 20 acquires map information from the map database 5. Note that the map database may be provided outside the vehicle 30 (e.g., a server, etc.), and the ECU 20 may acquire map information from outside the vehicle 30.

[0034] The vehicle behavior detection device 6 detects the behavior information of the vehicle 30. The vehicle behavior detection device 6 includes, for example, a vehicle speed sensor that detects the speed of the vehicle 30, a yaw rate sensor that detects the yaw rate of the vehicle 30, and the like. The output of the vehicle behavior detection device 6, i.e., the behavior information of the vehicle detected by the vehicle behavior detection device 6, is transmitted to the ECU 20.

[0035] The vehicle operation detection device 7 detects the operation information of the vehicle 30 by the driver. The vehicle operation detection device 7 includes, for example, an accelerator position sensor that detects the accelerator opening of the vehicle 30, a brake position sensor that detects the position of the brake pedal of the vehicle 30, a steering angle sensor that detects the steering amount of the vehicle 30, and the like. The output of the vehicle operation detection device 7, i.e., the operation information of the vehicle 30 detected by the vehicle operation detection device 7, is transmitted to the ECU 20.

[0036] The actuator 8 operates the vehicle 30. For example, the actuator 8 includes a drive device (e.g., at least one of an internal combustion engine and an electric motor) for accelerating the vehicle 30, a brake actuator for braking (decelerating) the vehicle 30, a steering actuator for steering the vehicle 30, and the like. The ECU 20 controls the actuator 8 to control the behavior of the vehicle 30.

[0037] The output device 9 outputs information to the occupants (e.g., the driver) of the vehicle 30. The output device 9 includes a display, a speaker, a light source, a vibration unit, and the like. The output of the ECU 20 is notified to the occupants of the vehicle 30 via the output device 9. Note that the output device 9 may be an input / output device (e.g., a human machine interface (HMI)) that further has an input unit (e.g., a touch panel, an operation button, an operation switch, a microphone, etc.) through which information is input by the occupants of the vehicle 30.

[0038] The communication device 10 is capable of communicating with the outside of the vehicle 30 and enables communication between the vehicle 30 and the outside of the vehicle 30. For example, the communication device 10 includes a wide-area communication device that enables wide-area communication between the vehicle 30 and the outside of the vehicle 30 (e.g., a server) via a communication network such as a carrier network and the Internet, a vehicle-to-vehicle communication device that enables vehicle-to-vehicle communication between the vehicle 30 and surrounding vehicles using a predetermined frequency band, and a roadside-to-vehicle communication device that enables roadside-to-vehicle communication between the vehicle 30 and a roadside unit using a predetermined frequency band. The ECU 10 communicates with the outside of the vehicle 30 via the communication device 10.

[0039] The occupant detection device 11 detects the occupants of the vehicle 30 other than the driver. The occupant detection device 11 includes an in-vehicle camera that photographs the interior of the vehicle, a seat belt sensor or a seating sensor provided on the passenger seat or the rear seat, and the like. The output of the occupant detection device 11, that is, the detection result of the occupants other than the driver, is transmitted to the ECU 20.

[0040] The ECU 20 executes various controls of the vehicle. As shown in FIG. 1, the ECU 20 includes a communication interface 21, a memory 22, and a processor 23. The communication interface 21 and the memory 22 are connected to the processor 23 via signal lines. In this embodiment, one ECU 20 is provided, but a plurality of ECUs may be provided for each function. The ECU 20 is an example of a driver monitoring device.

[0041] The communication interface 21 has an interface circuit for connecting the ECU 20 to the in-vehicle network. The ECU 20 is connected to other in-vehicle devices via the communication interface 21.

[0042] The memory 22 has, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores programs, data, etc. that are used when various processes are executed by the processor 23.

[0043] The processor 23 has one or more CPUs (Central Processing Units) and its peripheral circuits. Note that the processor 23 may further have an arithmetic circuit such as a logical arithmetic unit or a numerical arithmetic unit.

[0044] FIG. 3 is a functional block diagram of the processor 23 of the ECU 20 in the first embodiment. In this embodiment, the processor 23 has a facial expression determination unit 25 and a processing unit 26. The facial expression determination unit 25 and the processing unit 26 are functional modules realized by the processor 23 of the ECU 20 executing a computer program stored in the memory 22 of the ECU 20. Note that these functional modules may be realized by dedicated arithmetic circuits provided in the processor 23, respectively.

[0045] The expression determination unit 25 determines the driver's expression from an image (hereinafter referred to as the "driver's face image") generated by the imaging device 2 that captures the face of the driver of the vehicle 30. For example, the expression determination unit 25 determines the driver's expression by inputting the data of the face image into a discriminator that has been pre-trained to output the type of expression from the driver's face image. In the training of such a discriminator, as teacher data, for example, an image in which one of six expressions (anger, disgust, sadness, contempt, surprise, and fear) is tagged using a method of expression analysis based on the Facial Action Coding System (FACS) is used. Specific examples of discriminators include machine learning models such as neural networks (e.g., convolutional neural networks (CNNs), etc.), support vector machines, and random forests. Note that the expression determination unit 25 may determine the driver's expression using an image recognition method other than machine learning such as pattern matching.

[0046] The driver shows various expressions according to the surrounding situation of the vehicle during driving. As a result of recent research, it has been found that among drivers with mild cognitive impairment whose cognitive function has declined, the proportion of showing surprised expressions during driving is higher than that of healthy drivers without cognitive problems. Therefore, when the frequency of the driver showing a surprised expression during driving is high, a decline in the driver's cognitive function is suspected.

[0047] Therefore, in the present embodiment, the processing unit 26 calculates the frequency at which the driver's expression is determined to be a surprised expression, and when the frequency is equal to or higher than a predetermined threshold, performs predetermined processing to enhance the safety of the driver. By this, the safety of a driver who may have a decline in cognitive function can be enhanced.

[0048] Hereinafter, with reference to FIG. 4, the above-described control will be described in detail. FIG. 4 is a flowchart showing a control routine for driver monitoring processing in the first embodiment of the present invention. This control routine is repeatedly executed at a predetermined execution interval by the processor 23 of the ECU 20 having the expression determination unit 25 and the processing unit 26.

[0049] First, in step S101, the expression determination unit 25 acquires a face image of the driver. The face image of the driver is repeatedly generated by the imaging device 2 at a predetermined imaging period (for example, 1 / 30 second to 1 / 10 second), and the expression determination unit 25 acquires the face image of the driver from the imaging device 2. Next, in step S102, the expression determination unit 25 determines the expression of the driver based on the face image of the driver.

[0050] Next, in step S103, the processing unit 26 determines whether or not the expression determined by the expression determination unit 25 is a surprised expression. If it is determined that the expression determined by the expression determination unit 25 is not a surprised expression, this control routine ends. On the other hand, if it is determined that the expression determined by the expression determination unit 25 is a surprised expression, this control routine proceeds to step S104.

[0051] In step S104, the processing unit 26 calculates the frequency FR at which the driver's expression is determined to be a surprised expression. The frequency FR is calculated, for example, as the number of times the driver's expression is determined to be a surprised expression during a predetermined calculation time. Note that the frequency FR may be calculated as the number of times the driver's expression is determined to be a surprised expression during a predetermined driving distance. In this case, the driving distance of the vehicle 30 is calculated based on, for example, the output of the vehicle speed sensor of the vehicle behavior detection device 6.

[0052] Next, in step S105, the processing unit 26 determines whether or not the frequency FR is equal to or greater than a predetermined threshold value TH. The threshold value TH is, for example, a predetermined fixed value. Note that the threshold value TH may be set for each driver according to the pre-registered driver information (for example, age, gender, etc.).

[0053] For example, when the threshold TH is 10 times / hour, when the number of times the driver's expression is determined to be a surprised expression reaches 10 times within one hour, the frequency FR is determined to be equal to or higher than the threshold TH. As another example, when the threshold TH is 5 times / 10 km, when the number of times the driver's expression is determined to be a surprised expression reaches 5 times until the vehicle 30 travels 10 km, the frequency FR is determined to be equal to or higher than the threshold TH.

[0054] If it is determined in step S105 that the frequency FR is less than the threshold TH, this control routine ends. On the other hand, if it is determined in step S105 that the frequency FR is equal to or higher than the threshold TH, this control routine proceeds to step S106.

[0055] In step S106, the processing unit 26 determines that the driver's cognitive function has deteriorated, and performs a predetermined process to enhance the driver's safety. The processing unit 26 performs, for example, the following safety support processes. As a first safety support process, the processing unit 26 notifies the driver of a warning. In this case, the processing unit 26 notifies the driver of a visual, auditory, or tactile warning via the output device 9. Examples of visual warnings are warning messages (e.g., messages such as "Since the cognitive function has deteriorated, please drive carefully.") or warning images displayed on the display of the output device 9, warning lights emitted from the light source of the output device 9, etc. Examples of auditory warnings are warning voices or warning sounds output from the speaker of the output device 9. Examples of tactile warnings are vibrations output from the vibration unit of the output device 9 (e.g., vibrations of the steering wheel 32), etc. Note that the processing unit 26 may notify the driver of two or more types of warnings (e.g., visual warnings and auditory warnings).

[0056] Even if the driver is notified of a warning as described above, there is a risk that the driver may not accept the pointed out decrease in cognitive function. Therefore, as a second safety support process, the processing unit 26 notifies a warning to a person other than the driver. By doing so, with the support of a person other than the driver, it is possible to improve the safety of the driver who may have a decreased cognitive function. When a warning is notified to a person outside the vehicle 30 (for example, the driver's family member, the driver of a surrounding vehicle, a doctor, a police officer, etc.), the processing unit 26 transmits, via the communication device 10, a warning signal indicating that the cognitive function of the driver of the vehicle 30 is decreased to the outside of the vehicle 30 (a mobile terminal, a server, a surrounding vehicle, etc.). Further, when a passenger other than the driver is detected by the passenger detection device 11, the processing unit 26 may notify a visual warning to the passenger other than the driver. By notifying a visual warning, it is possible to notify a passenger other than the driver of the decrease in the cognitive function of the driver without the driver noticing. In this case, for example, the processing unit 26 displays a warning message indicating that the cognitive function of the driver of the vehicle 30 is decreased on a display provided in front of the passenger other than the driver or the like.

[0057] Also, when the cognitive function of the driver is decreased, it is desirable to support the driver's driving by vehicle control. Therefore, as a third safety support process, the processing unit 26 relaxes the implementation conditions of the driving support operation in the driving support function mounted on the vehicle 30. That is, the processing unit 26 makes it easier to perform the driving support operation in the driving support function mounted on the vehicle 30. For example, when the driving support operation (for example, warning or braking control) in the pre-crash safety (PCS) is performed when the time to collision (TTC) becomes equal to or less than a predetermined value, the processing unit 26 makes the predetermined value longer. Also, the implementation conditions of the driving support operation in the driving support function such as proactive driving assist (PDA) or lane departure alert (LDA) may be relaxed.

[0058] Also, as a fourth safety support process, when the adaptive cruise control (ACC) is activated as a driving support function, the processing unit 26 fixes the set inter-vehicle distance at the maximum value. For example, when the set inter-vehicle distance in ACC can be set in three levels (short distance, medium distance, and long distance), the processing unit 26 fixes the set inter-vehicle distance at the long distance. By doing this, it becomes easier to respond to unexpected actions of the preceding vehicle, and even when the driver's cognitive function is degraded, the safety of the driver when ACC is activated can be enhanced.

[0059] Note that when it is determined that the frequency FR is equal to or greater than the threshold value TH, the processing unit 26 can perform one or any combination of the first to fourth safety support processes. Also, as long as it is a process for enhancing the driver's safety, the processing unit 26 may perform processes other than the first to fourth safety support processes. After step S106, this control routine ends.

[0060] Note that when the frequency at which it is determined that the driver's expression is a surprised expression over a predetermined time or a predetermined driving distance is equal to or greater than a predetermined threshold value, the processing unit 26 may perform a predetermined process for enhancing the driver's safety. In this case, for example, when the threshold value is 10 times / hour, the processing unit 26 performs a predetermined process when a state where the number of times the driver's expression is determined to be a surprised expression reaches 10 times is maintained for 10 hours within 1 hour. As another example, when the threshold value TH is 5 times / 10 km, the processing unit 26 performs a predetermined process when a state where the number of times the driver's expression is determined to be a surprised expression reaches 5 times is maintained for 100 km until the vehicle 30 travels 10 km.

[0061] Further, the processing unit 26 may set a threshold value based on past data of the frequency at which the driver's expression is determined to be a surprised expression. In this case, the threshold value is set to a value higher than the past average value of the frequency. For example, the past average value of the frequency is stored in the memory 22 of the ECU 20 or the like, and when the ignition switch of the vehicle 30 is turned on, the threshold value is calculated by multiplying the past average value of the frequency by a coefficient (for example, 2 to 5). Further, the threshold value for the subsequent period of one trip may be calculated by multiplying the average value of the frequency in the first predetermined time (for example, 30 minutes) in one trip (the period from when the ignition switch of the vehicle 30 is turned on until it is turned off) by a coefficient (for example, 2 to 5).

[0062] <Second Embodiment> The driver monitoring device according to the second embodiment is basically the same as the configuration and control of the driver monitoring device according to the first embodiment, except for the points described below. Therefore, hereinafter, the second embodiment of the present invention will be described centering on the parts different from the first embodiment.

[0063] FIG. 5 is a functional block diagram of the processor 23 of the ECU 20 in the second embodiment. In the second embodiment, the processor 23 has an event detection unit 27 in addition to the expression determination unit 25 and the processing unit 26. The event detection unit 27 detects an event indicating inappropriate driving by the driver of the vehicle 30. The expression determination unit 25, the processing unit 26, and the event detection unit 27 are functional modules realized by the processor 23 of the ECU 20 executing a computer program stored in the memory 22 of the ECU 20. Note that these functional modules may each be realized by a dedicated arithmetic circuit provided in the processor 23.

[0064] The driver shows a surprised expression due to various factors during the driving of the vehicle 30. The surprised expression caused by the driver's driving has a strong correlation with the decline in the driver's cognitive ability compared to the surprised expression caused by factors other than the driver's driving (for example, conversation with other passengers, content of the radio, etc.). Therefore, when a surprised expression of the driver is detected when an event indicating inappropriate driving by the driver (hereinafter simply referred to as "event") occurs, it is desirable to perform processing to enhance the driver's safety at an earlier stage.

[0065] Therefore, in the second embodiment, when the processing unit 26 determines that the driver's expression is a surprised expression by the expression determination unit 25 and an event is detected by the event detection unit 27, the processing unit 26 lowers the threshold value when performing a predetermined process. That is, when an event is detected, the predetermined process is performed in a state where the frequency of determining that the driver's expression is a surprised expression is lower than when no event is detected. By this, it is possible to more appropriately detect the decline in the driver's cognitive function, and it is possible to support a driver who may have a decline in cognitive function at a more appropriate timing.

[0066] FIG. 6 is a flowchart showing a control routine of the driver monitoring process in the second embodiment of the present invention. This control routine is repeatedly executed at a predetermined execution interval by the processor 23 of the ECU 20 having the expression determination unit 25, the processing unit 26, and the event detection unit 27.

[0067] Steps S201 to S203 are executed in the same manner as steps S101 to S103 in FIG. 4. When it is determined in step S203 that the expression determined by the expression determination unit 25 is a surprised expression, this control routine proceeds to step S204.

[0068] In step S204, the processing unit 26 determines whether an event has been detected by the event detection unit 27 within a predetermined time until a face image of the driver including a surprised expression is acquired. The event detection unit 27 detects events such as a dangerous driving scene or a sudden operation by the driver. For example, the event detection unit 27 detects a dangerous driving scene based on the surrounding environment information of the vehicle 30. The surrounding environment information of the vehicle 30 is detected by the surrounding information detection device 3, and the event detection unit 27 acquires the surrounding environment information of the vehicle 30 from the surrounding information detection device 3.

[0069] The dangerous driving scene includes a scene where the distance between the vehicle 30 and moving objects around the vehicle 30 (for example, pedestrians, surrounding vehicles, bicycles, etc.) is equal to or less than a predetermined distance. Specific examples of the dangerous driving scene include a scene where a person jumps out from an intersection, a scene where a person jumps out from behind a surrounding vehicle, a scene where the vehicle 30 approaches a vehicle behind during a lane change, etc. In addition, the dangerous driving scene includes the vehicle 30 ignoring a signal, violating a stop, etc. The position information of the traffic signal and the stop sign may be acquired from the map database 5.

[0070] In addition, the event detection unit 27 detects a sudden operation (sudden acceleration, sudden deceleration, or sudden steering) by the driver based on the operation information of the vehicle 30 by the driver. The operation information of the vehicle 30 by the driver is detected by the vehicle operation detection device 7, and the event detection unit 27 acquires the operation information of the vehicle 30 by the driver from the vehicle operation detection device 7. The sudden acceleration by the driver is detected based on the output of the accelerator position sensor, the sudden deceleration by the driver is detected based on the output of the brake position sensor, and the sudden steering by the driver is detected based on the output of the steering angle sensor. Note that the event detection unit 27 may detect only one of the dangerous driving scene and the sudden operation by the driver as an event.

[0071] If it is determined in step S204 that an event has been detected, this control routine proceeds to step S205. In step S205, the event detection unit 27 changes the threshold value TH. For example, the event detection unit 27 reduces the threshold value TH by a predetermined value from a predetermined standard value. On the other hand, if it is determined in step S204 that no event has been detected, this control routine skips step S205 and proceeds to step S206. In this case, the standard value is used as the threshold value TH.

[0072] Steps S206 to S208 are executed in the same manner as steps S104 to S106 in FIG. 4. After step S208, this control routine ends. Note that the control routine in FIG. 6 can be modified in the same manner as the control routine in FIG. 4. For example, the standard value of the threshold value may be set based on the past data of the frequency at which the driver's expression is determined to be a surprised expression.

[0073] Also, when the processing unit 26 calculates the frequency at which the driver's expression is determined to be a surprised expression, if the expression determination unit 25 determines that the driver's expression is a surprised expression and the event detection unit 27 detects an event, the weight for the frequency at which the driver's expression is determined to be a surprised expression may be increased. For example, if no event is detected when the driver's expression is determined to be a surprised expression, 1 is added as the number of times the driver's expression is determined to be a surprised expression, and if an event is detected when the driver's expression is determined to be a surprised expression, a number larger than 1 (for example, 2 to 8) is added as the number of times the driver's expression is determined to be a surprised expression.

[0074] In addition, if it frequently occurs that the driver shows a surprised expression when an event occurs, there is a possibility that the driver's cognitive ability has significantly declined. Therefore, when the expression determination unit 25 determines that the driver's expression is a surprised expression and the frequency of event detection by the event detection unit 27 reaches a predetermined value, the processing unit 26 may perform another process (hereinafter referred to as "emergency process") with a higher warning intensity than a predetermined process. By doing so, it is possible to appropriately support the driver according to the degree of decline in the driver's cognitive ability.

[0075] In this case, for example, as an emergency process, the processing unit 26 always notifies a warning, forcibly stops the vehicle, requests rescue, limits the vehicle speed (for example, to a regulated speed defined by a speed sign), or always turns on the hazard lamp. The emergency process may be cancelable on an application operating on a mobile terminal (such as a smartphone or a tablet terminal) or at an automobile dealership by presenting the results of a cognitive ability test or a driving ability test.

[0076] <Other Embodiments> As described above, the preferred embodiments of the present invention have been described. However, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the claims.

[0077] For example, a part of the configuration of the vehicle control system 1 shown in FIG. 1 may be omitted. Further, the vehicle 30 may be a vehicle without a driving support function.

[0078] In addition, a computer program for causing a computer to realize the functions of each part of the processor 23 of the ECU 20 may be provided in a form stored in a computer-readable recording medium. The computer-readable recording medium is, for example, a magnetic recording medium, an optical recording medium, or a semiconductor memory.

Explanation of Reference Numerals

[0079] 2 Imaging device 20 Electronic control unit (ECU) 23 Processor 25 Facial Expression Judgment Unit 26 Processing Unit 30 Vehicle

Claims

1. An expression determination unit that determines the expression of the driver from an image generated by an imaging device that captures an image of the face of the driver of the vehicle; A processing unit that calculates the frequency at which the expression of the driver is determined to be a surprised expression, and performs a predetermined process for enhancing the safety of the driver when the frequency is equal to or higher than a predetermined threshold; An event detection unit that detects an event indicating inappropriate driving by the driver and has, When it is determined that the expression of the driver is a surprised expression and the event is detected, the processing unit reduces the threshold value, or increases the weight for the frequency when it is determined that the expression of the driver is a surprised expression when calculating the frequency, a driver monitoring device.

2. The driver monitoring device according to claim 1, wherein the processing unit sets the threshold value based on past data of the frequency.

3. The driver monitoring device according to claim 1, wherein the event detection unit detects a dangerous driving scene as the event based on the surrounding environment information of the vehicle.

4. The driver monitoring device according to claim 3, wherein the dangerous driving scene includes a scene in which the distance between the vehicle and a moving object around the vehicle is equal to or less than a predetermined distance.

5. The driver monitoring device according to claim 1, wherein the event detection unit detects a sudden operation by the driver as the event based on the operation information of the vehicle by the driver.

6. The driver monitoring device according to claim 1, wherein when the frequency at which it is determined that the expression of the driver is a surprised expression and the event is detected reaches a predetermined value, the processing unit performs another process with a higher warning intensity than the predetermined process.

7. The driver monitoring device according to any one of claims 1 to 6, wherein the processing unit notifies the driver of a warning as the predetermined process.

8. The driver monitoring device according to any one of claims 1 to 6, wherein the processing unit notifies a person other than the driver of a warning as the predetermined process.

9. The driver monitoring device according to any one of claims 1 to 6, wherein the processing unit relaxes the implementation conditions of a driving support operation in a driving support function mounted on the vehicle as the predetermined process.

10. The driver monitoring device according to any one of claims 1 to 6, wherein the processing unit fixes the set inter-vehicle distance when adaptive cruise control is activated in the vehicle to a maximum value as the predetermined processing.

11. A driver monitoring method executed by a computer, comprising: determining an expression of the driver from an image generated by an imaging device that captures an image of the face of the driver of a vehicle; calculating a frequency at which the expression of the driver is determined to be a surprised expression, and when the frequency is equal to or greater than a predetermined threshold, performing predetermined processing for enhancing safety of the driver; detecting an event indicating inappropriate driving by the driver; when it is determined that the expression of the driver is a surprised expression and the event is detected, reducing the threshold value or increasing a weight for the frequency at which it is determined that the expression of the driver is a surprised expression when calculating the frequency A driver monitoring method including the above.

12. determining an expression of the driver from an image generated by an imaging device that captures an image of the face of the driver of a vehicle; calculating a frequency at which the expression of the driver is determined to be a surprised expression, and when the frequency is equal to or greater than a predetermined threshold, performing predetermined processing for enhancing safety of the driver; detecting an event indicating inappropriate driving by the driver; when it is determined that the expression of the driver is a surprised expression and the event is detected, reducing the threshold value or increasing a weight for the frequency at which it is determined that the expression of the driver is a surprised expression when calculating the frequency A computer program for driver monitoring that causes a computer to execute the above.

Images