Driver monitoring device, driver monitoring method, and computer program for driver monitoring

Abstract

To suppress erroneous determination that a driver is not in a driving standby state capable of performing a driving operation.SOLUTION: A driver monitoring device includes: a determination section 25 for determining whether a driver of a vehicle 30 is in a driving standby state capable of performing a driving operation; and a notification section 26 for notifying the driver of warning when it is determined that the driver is not in the driving standby state. The determination section acquires an image generated by an imaging device arranged in a vehicle to image the driver and driver information acquired by a terminal being operable by the driver, and then determines whether the driver is in the driving standby state based on the image and the driver information.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a driver monitoring device, a driver monitoring method, and a computer program for driver monitoring.

Background Art

[0002] Patent Document 1 describes that in a vehicle equipped with an automatic driving function, when it is presumed that the driver is continuously performing an act other than an allowable act permitted to the driver during the operation of the automatic driving function, an alarm directed to the driver is caused to be implemented by an information presentation device.

[0003] Patent Document 2 describes that when determining whether a driver is in a driving difficulty state, if at least one of an abnormality in the driver's state, an abnormality in the driving operation by the driver, and an abnormality in the running state of the vehicle is detected in addition to the abnormality in the driver's state, it is determined that the driver is in a driving difficulty state.

[0004] Patent Document 3 describes detecting the state of a driver and changing the timing for notifying a change in the driving mode of a vehicle (for example, switching from an automatic driving mode to a manual driving mode) according to the state of the driver.

[0005] Patent Document 4 describes monitoring the driver's line of sight and state, and when detecting that the driver's line of sight is directed toward a device such as a smartphone when requesting the driver to take over from an automatic driving control mode to a manual driving mode, stopping the function of this device.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Summary of the Invention

Problems to be Solved by the Invention

[0007] When level 3 automated driving is being performed in a vehicle, if the driver can alternate driving operations in response to a request from the system, operation of a terminal such as a smartphone is permitted to the driver. On the other hand, actions such as dozing off that prevent the prompt takeover of driving operations are prohibited.

[0008] Therefore, even when level 3 automated driving is being performed, it is necessary to monitor the driver's state using an imaging device provided in the vehicle to photograph the driver. However, it is difficult to distinguish, based on an image generated by such an imaging device, between a state in which the driver is operating a terminal and an abnormal posture due to the driver's dozing off or sudden illness. For this reason, there is a risk of misjudging that the driver is not in a driving standby state in which the driver can perform a driving operation when the driver is operating a terminal.

[0009] Therefore, in view of the above problems, an object of the present invention is to suppress misjudging that the driver is not in a driving standby state in which the driver can perform a driving operation.

Means for Solving the Problems

[0010] The gist of the present disclosure is as follows.

[0011] (1) A determination unit that determines whether or not the driver of the vehicle is in a driving standby state in which a driving operation can be performed, and a notification unit that notifies the driver of a warning when it is determined that the driver is not in the driving standby state. The determination unit acquires an image generated by an imaging device provided in the vehicle so as to photograph the driver and driver information acquired by a terminal operable by the driver, and determines whether or not the driver is in the driving standby state based on the image and the driver information. A driver monitoring device.

[0012] (2) The driver monitoring device according to (1) above, wherein the driver information includes an image generated by a camera mounted on the terminal or input information to the terminal by the driver.

[0013] (3) The determination unit determines whether or not the driver is in the driving standby state based on the image according to a predetermined determination criterion, determines whether or not the driver is in a waking state based on the driver information, and relaxes the determination criterion when it is determined that the driver is in a waking state. The driver monitoring device according to (1) or (2) above.

[0014] (4) The determination unit determines whether or not the driver is in the driving standby state based on the image according to a predetermined determination criterion. When it is determined based on the image that the driver is not in the driving standby state, it determines whether or not the driver is in a waking state based on the driver information. When it is determined that the driver is in a waking state, the determination result based on the image is invalidated. The driver monitoring device according to (1) or (2) above.

[0015] (5) The determination unit acquires the driver information from the terminal when it is determined based on the image that the driver is not in the driving standby state, and does not acquire the driver information from the terminal when it is determined based on the image that the driver is in the driving standby state. The driver monitoring device according to any one of (1) to (4) above.

[0016] (6) When the vehicle is performing automated driving at level 2 or below, the driver monitoring device according to any one of (1) to (5) above, further comprising a terminal control unit that restricts or prohibits the driver from operating the terminal.

[0017] (7) The terminal is a smartphone, and the driver monitoring device according to any one of (1) to (6) above.

[0018] (8) A driver monitoring method executed by a computer, comprising: obtaining an image generated by an imaging device provided in the vehicle to photograph the driver of the vehicle; obtaining driver information obtained by a terminal operable by the driver; determining whether the driver is in a driving standby state in which a driving operation can be performed based on the image and the driver information; and when it is determined that the driver is not in the driving standby state, notifying the driver of a warning.

[0019] (9) A computer program for driver monitoring, which causes a computer to execute: obtaining an image generated by an imaging device provided in the vehicle to photograph the driver of the vehicle; obtaining driver information obtained by a terminal operable by the driver; determining whether the driver is in a driving standby state in which a driving operation can be performed based on the image and the driver information; and when it is determined that the driver is not in the driving standby state, notifying the driver of a warning.

Advantages of the Invention

[0020] According to the present invention, it is possible to suppress misjudgment that the driver is not in a driving standby state in which a driving operation can be performed.

Brief Description of the Drawings

[0021]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

[0022] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the following description, the same reference numerals are assigned to similar components.

[0023] <First Embodiment> Hereinafter, the first embodiment of the present invention will be described with reference to FIGS. 1 to 5. FIG. 1 is a schematic configuration diagram of a vehicle control system 1 including a driver monitoring device according to the first embodiment of the present invention. The vehicle control system 1 is mounted on a vehicle and executes various controls of the vehicle.

[0024] As shown in FIG. 1, the vehicle control system 1 includes a driver monitoring camera 2, a surrounding information detection device 3, a GNSS (Global Navigation Satellite System) receiver 4, a map database 5, a navigation device 6, a vehicle behavior detection device 7, an actuator 8, a human machine interface (HMI) 9, a communication device 10, and an electronic control unit (ECU) 20. The driver monitoring camera 2, the surrounding information detection device 3, the GNSS receiver 4, the map database 5, the navigation device 6, the vehicle behavior detection device 7, the actuator 8, the HMI 9, and the communication device 10 are electrically connected to the ECU 20 via an in-vehicle network or the like conforming to a standard such as CAN (Controller Area Network).

[0025] The driver monitoring camera 2 photographs the driver of the vehicle and generates an image representing the driver. The output of the driver monitoring camera 2, that is, the image generated by the driver monitoring camera 2, is transmitted to the ECU 20. Hereinafter, a specific example of the configuration of the driver monitoring camera 2 will be described.

[0026] The driver monitoring camera 2 has a camera and a projector. The camera is composed of a lens and an imaging element, and is, for example, a CMOS (Complementary Metal Oxide Semiconductor) camera or a CCD (Charge Coupled Device) camera. The projector is an LED (Light Emitting Diode), and is, for example, two near-infrared LEDs arranged on both sides of the camera. By irradiating the driver with near-infrared light, it is possible to photograph the driver's face without giving discomfort to the driver even in low illuminance situations such as at night. Also, a band-pass filter for removing light having wavelength components other than near-infrared may be provided inside the camera, and a visible light cut filter for removing the red wavelength component light irradiated from the near-infrared LED may be provided on the front surface of the projector.

[0027] FIG. 2 is a diagram schematically showing the interior of a vehicle 30 provided with a driver monitoring camera 2. The driver monitoring camera 2 is provided in the passenger compartment of the vehicle 30 so as to photograph the driver of the vehicle 30. For example, as shown in FIG. 2, the driver monitoring camera 2 is provided at the upper part of the steering column 31 of the vehicle 30. In FIG. 2, the projection range of the driver monitoring camera 2 is shown by a broken line. In the present embodiment, the driver monitoring camera 2 photographs the driver's face. Note that the driver monitoring camera 2 may be provided on the steering wheel 32, the rearview mirror, the meter panel, the meter hood, etc. of the vehicle 30. The driver monitoring camera 2 is an example of an imaging device provided on the vehicle so as to photograph the driver.

[0028] The surrounding information detection device 3 acquires data (images, point cloud data, etc.) around the vehicle 30 and detects the surrounding information of the vehicle 30 (for example, surrounding vehicles, lanes, pedestrians, bicycles, traffic lights, signs, etc.). For example, the surrounding information detection device 3 includes a camera (monocular camera or stereo camera), a millimeter wave radar, a lidar (laser imaging detection and ranging), or an ultrasonic sensor (sonar), or any combination thereof. Note that the surrounding information detection device 3 may further include an illuminance sensor, a rain sensor, etc. The output of the surrounding information detection device 3, that is, the surrounding information of the vehicle 30 detected by the surrounding information detection device 3 is transmitted to the ECU 20.

[0029] The GNSS receiver 4 detects the current position of the vehicle 30 (for example, the latitude and longitude of the vehicle 30) based on the positioning information obtained from a plurality (for example, three or more) of positioning satellites. Specifically, the GNSS receiver 4 captures a plurality of positioning satellites and receives the radio waves transmitted from the positioning satellites. Then, the GNSS receiver 4 calculates the distance to the positioning satellite based on the difference between the transmission time and the reception time of the radio wave, and detects the current position of the vehicle 30 based on the distance to the positioning satellite and the position (orbit information) of the positioning satellite. The output of the GNSS receiver 4, that is, the current position of the vehicle 30 detected by the GNSS receiver 4, is transmitted to the ECU 20. A GPS (Global Positioning System) receiver is an example of a GNSS receiver.

[0030] The map database 5 stores map information. The ECU 20 acquires map information from the map database 5. Note that the map database may be provided outside the vehicle 30 (for example, a server, etc.), and the ECU 20 may acquire map information from outside the vehicle 30.

[0031] The navigation device 6 sets the driving route of the vehicle 30 to the destination based on the current position of the vehicle 30 detected by the GNSS receiver 4, the map information of the map database 5, the input by the occupant (for example, the driver) of the vehicle 30, etc. The driving route set by the navigation device 6 is transmitted to the ECU 20.

[0032] The vehicle behavior detection device 7 detects the behavior information of the vehicle 30. The vehicle behavior detection device 7 includes, for example, a vehicle speed sensor that detects the speed of the vehicle 30, a yaw rate sensor that detects the yaw rate of the vehicle 30, etc. The output of the vehicle behavior detection device 7, that is, the behavior information of the vehicle detected by the vehicle behavior detection device 7, is transmitted to the ECU 20.

[0033] The actuator 8 operates the vehicle 30. For example, the actuator 8 includes a drive device (e.g., at least one of an internal combustion engine and an electric motor) for accelerating the vehicle 30, a brake actuator for braking (decelerating) the vehicle 30, a steering actuator for steering the vehicle 30, and the like. The ECU 20 controls the actuator 8 to control the behavior of the vehicle 30.

[0034] The HMI 9 transmits information between the vehicle 30 and the occupants (e.g., the driver) of the vehicle 30. The HMI 9 has an output unit (e.g., a display, a speaker, a light source, a vibration unit, etc.) for providing information to the occupants of the vehicle 30 and an input unit (e.g., a touch panel, an operation button, an operation switch, a microphone, etc.) through which information is input by the occupants of the vehicle 30. The output of the ECU 20 is notified to the occupants of the vehicle 30 via the HMI 9, and the input from the occupants of the vehicle 30 is transmitted to the ECU 20 via the HMI 9. The HMI 9 is an example of an input device, an output device, or an input / output device.

[0035] The communication device 10 is capable of communicating with the outside of the vehicle 30 and enables communication between the vehicle 30 and the outside of the vehicle 30. For example, the communication device 10 includes a wide-area wireless communication module that enables wide-area wireless communication between the vehicle 30 and the outside of the vehicle 30 (e.g., a server) via a communication network such as a carrier network and the Internet, and a short-range wireless communication module that enables short-range wireless communication based on a communication standard such as Bluetooth (registered trademark) or ZigBee (registered trademark).

[0036] On the other hand, the terminal 11 shown in FIG. 1 is possessed by the driver of the vehicle 30. The terminal 11 is brought into the vehicle 30 by the driver and can be operated by the driver. The terminal 11 is paired with the vehicle 30 in advance, and the ECU 20 can communicate with the terminal 11 via the short-range wireless communication module of the communication device 10. In this embodiment, the terminal 11 is a smartphone. Note that the terminal 11 may be electrically connected to the ECU 20 by wire.

[0037] The ECU 20 executes various controls of the vehicle. As shown in FIG. 1, the ECU 20 includes a communication interface 21, a memory 22, and a processor 23. The communication interface 21 and the memory 22 are connected to the processor 23 via signal lines. In this embodiment, one ECU 20 is provided, but a plurality of ECUs may be provided for each function.

[0038] The communication interface 21 has an interface circuit for connecting the ECU 20 to the in-vehicle network. The ECU 20 is connected to other in-vehicle devices via the communication interface 21.

[0039] The memory 22 has, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores programs, data, etc. that are used when various processes are executed by the processor 23.

[0040] The processor 23 has one or more CPUs (Central Processing Units) and its peripheral circuits. Note that the processor 23 may further have an arithmetic circuit such as a logical arithmetic unit or a numerical arithmetic unit.

[0041] The vehicle control system 1 functions as an autonomous driving system and realizes the autonomous driving of the vehicle 30. In this embodiment, the vehicle control system 1 can perform level 3 autonomous driving in a preset Operational Design Domain (ODD). Note that the autonomous driving level in this specification is based on the definition of SAE (Society of Automotive Engineers) J3016.

[0042] The vehicle control system 1 performs automated driving at level 2 or below under conditions outside the operation design domain. In level 1 or level 2 automated driving, the vehicle control system 1 activates driving assistance functions such as Adaptive Cruise Control (ACC) that automatically controls the speed of the vehicle according to the presence or absence of a preceding vehicle, Lane Keeping Assist (LKA) or Lane Tracing Assist (LTA) that automatically controls the steering of the vehicle so that the vehicle is maintained within the lane. Level 0 automated driving corresponds to manual driving in which all of the acceleration, deceleration (braking), and steering of the vehicle 30 are performed by the driver.

[0043] On the other hand, in level 3 automated driving, the system mainly executes the driving operations of the vehicle 30, and the driver is released from the obligation of peripheral monitoring. Therefore, when level 3 automated driving is performed in the vehicle 30, if the driver can switch the driving operation in response to a request from the system, the operation of the terminal 11 is permitted to the driver. On the other hand, actions such as dozing off that prevent the rapid takeover of driving operations are prohibited. For this reason, even when level 3 automated driving is being performed, it is necessary to monitor the state of the driver using an imaging device such as the driver monitoring camera 2.

[0044] In the present embodiment, the ECU 20 provided in the vehicle 30 functions as a driver monitoring device that monitors the driver of the vehicle 30. FIG. 3 is a functional block diagram of the processor 23 of the ECU 20 in the first embodiment. In the present embodiment, the processor 23 has a determination unit 25 and a notification unit 26. The determination unit 25 and the notification unit 26 are functional modules realized by the processor 23 of the ECU 20 executing a computer program stored in the memory 22 of the ECU 20. Note that these functional modules may each be realized by a dedicated arithmetic circuit provided in the processor 23.

[0045] The determination unit 25 determines whether or not the driver of the vehicle 30 is in a driving standby state in which a driving operation can be performed. In particular, in the present embodiment, the determination unit 25 determines whether or not the driver is in a driving standby state when the vehicle 30 is performing level 3 automated driving. For example, when an abnormal posture due to the driver's dozing off, leaving the seat, or sudden illness is detected, the determination unit 25 determines that the driver is not in a driving standby state. The notification unit 26 notifies the driver of a warning when the determination unit 25 determines that the driver is not in a driving standby state. By doing so, it is possible to prompt the driver to return to the driving standby state, and as a result, the safety of the autonomous driving of the vehicle 30 can be enhanced.

[0046] The determination unit 25 acquires an image (hereinafter referred to as a "monitoring image") generated by the driver monitoring camera 2, and determines whether or not the driver is in a driving standby state based on the monitoring image. As described above, in level 3 automated driving, the driver can operate the terminal 11 during the driving of the vehicle 30 as a legitimate act. However, it is difficult to distinguish between a state in which the driver is operating the terminal 11 and an abnormal posture due to the driver's dozing off or sudden illness based on the monitoring image. On the other hand, the terminal 11 can acquire information different from the monitoring image as information regarding the driver.

[0047] Therefore, the determination unit 25 acquires the driver information acquired by the terminal 11 in addition to the monitoring image, and determines whether or not the driver is in a driving standby state based on the monitoring image and the driver information. By doing so, it is possible to suppress misjudging that the driver is not in a driving standby state.

[0048] In the present embodiment, the determination unit 25 determines whether or not the driver is in a driving standby state based on the monitoring image according to a predetermined determination criterion. Further, the determination unit 25 determines whether or not the driver is in a waking state based on the driver information acquired by the terminal 11, and relaxes the determination criterion when it is determined that the driver is in a waking state. By doing so, it is possible to suppress misjudging that a driver who is operating the terminal 11 as a legitimate act is not in a driving standby state.

[0049] Hereinafter, with reference to FIGS. 4 and 5, the above-described control will be described in detail. FIG. 4 is a flowchart showing a control routine for determination criterion setting processing in the first embodiment of the present invention. This control routine is repeatedly executed by the processor 23 of the ECU 20 at a predetermined execution interval.

[0050] First, in step S101, the determination unit 25 of the processor 23 determines whether or not the level 3 automated driving is being performed in the vehicle 30. The level 3 automated driving is performed when the driver requests the operation of the automated driving in a preset operation design area. If it is determined that the level 3 automated driving is not being performed, this control routine ends. On the other hand, if it is determined that the level 3 automated driving is being performed, this control routine proceeds to step S102.

[0051] In step S102, the determination unit 25 acquires driver information from the terminal 11 by short-range wireless communication or the like. The driver information is information regarding the driver acquired by the terminal 11, and includes, for example, an image generated by a camera (for example, an in-camera) mounted on the terminal 11, or input information from the driver to the terminal 11 (for example, input information by finger or voice by the driver).

[0052] Next, in step S103, the determination unit 25 determines whether or not the driver is in a wakeful state based on the driver information. For example, the determination unit 25 determines whether or not the driver is in a wakeful state based on an image (hereinafter referred to as a "camera image") generated by the camera of the terminal 11. In this case, for example, the determination unit 25 calculates the eye opening degree of the driver based on the camera image, and determines that the driver is in a wakeful state when the eye opening degree is equal to or greater than a predetermined value, and determines that the driver is not in a wakeful state when the eye opening degree is less than the predetermined value. In addition, when the eyes of the driver are not detected from the camera image, the determination unit 25 determines that it is unknown whether or not the driver is in a wakeful state.

[0053] Note that the determination unit 25 may determine whether the driver is in a waking state based on the input information to the terminal 11 by the driver. In this case, when an input (such as a touch input, a button input, a voice input, etc.) to the terminal 11 is made by the driver within a predetermined time, the determination unit 25 determines that the driver is in a waking state. When no input to the terminal 11 is made by the driver within the predetermined time, the determination unit 25 determines that it is unknown whether the driver is in a waking state. Further, the determination unit 25 may determine whether the driver is in a waking state based on the camera image and the input information to the terminal 11 by the driver. In this case, for example, when the degree of eye opening of the driver is equal to or greater than a predetermined value and an input to the terminal 11 is made by the driver within a predetermined time, the determination unit 25 determines that the driver is in a waking state. Further, the determination unit 25 may determine whether the driver is in a waking state based on the output of the acceleration sensor mounted on the terminal 11 or the like.

[0054] If it is determined in step S103 that the driver is not in a waking state or if it is determined that it is unknown whether the driver is in a waking state, this control routine proceeds to step S104. In step S104, the determination unit 25 initializes the criteria for determining whether the driver is in a driving standby state. That is, the determination unit 25 sets the criteria to preset conditions. The criteria are, for example, the threshold range of the driver's face orientation (vertical and horizontal directions). After step S104, this control routine ends.

[0055] On the one hand, if it is determined in step S103 that the driver is in an awake state, this control routine proceeds to step S105. In step S105, the determination unit 25 relaxes the determination criteria for determining whether the driver is in a driving standby state. That is, the determination unit 25 changes the determination criteria so that the driver is more likely to be determined to be in a driving standby state. When the driver operates the terminal 11 on his / her knee, the driver's face orientation is downward. For this reason, for example, the determination unit 25 relaxes the determination criteria by increasing the threshold value in the downward direction of the face orientation. Also, in the surveillance image, the driver's eyes may be blocked by the driver's hand or arm holding the terminal 11 or the terminal 11. For this reason, the determination unit 25 may relax the determination criteria so that the driver is determined to be in a driving standby state as long as the driver's face is detected from the surveillance image. After step S105, this control routine ends.

[0056] Figure 5 is a flowchart showing a control routine for warning processing in the first embodiment of the present invention. This control routine is repeatedly executed by the processor 23 of the ECU 20 at a predetermined execution interval.

[0057] First, in step S201, similar to step S101 in FIG. 4, the determination unit 25 of the processor 23 determines whether the vehicle 30 is performing level 3 automated driving. If it is determined that level 3 automated driving is not being performed, this control routine ends. On the other hand, if it is determined that level 3 automated driving is being performed, this control routine proceeds to step S202.

[0058] In step S202, the determination unit 25 acquires a surveillance image. The surveillance image is repeatedly generated by the driver monitor camera 2 at a predetermined imaging cycle (for example, 1 / 30 second to 1 / 10 second), and the determination unit 25 acquires the surveillance image from the driver monitor camera 2.

[0059] Next, in step S203, the determination unit 25 determines whether the driver is in the driving standby state based on the monitoring image according to a predetermined determination criterion. For example, the determination unit 25 detects the face orientation of the driver from the monitoring image, and determines that the driver is in the driving standby state when the face orientation of the driver is within a predetermined threshold range, and determines that the driver is not in the driving standby state when the face orientation of the driver is outside the predetermined threshold range. At this time, the determination criterion set in step S104 or S105 is used. That is, when it is determined that the driver is in the awake state, the determination criterion is relaxed.

[0060] If it is determined in step S203 that the driver is in the driving standby state, this control routine ends. On the other hand, if it is determined in step S203 that the driver is not in the driving standby state, this control routine proceeds to step S204.

[0061] In step S204, the notification unit 26 of the processor 23 notifies the driver of a visual, auditory, or tactile warning via the HMI 9. Examples of visual warnings are warning lights emitted from the light source of the HMI 9 and the like. Examples of auditory warnings are warning voices or warning sounds output from the speaker of the HMI 9 and the like. Examples of tactile warnings are vibrations output from the vibration unit of the HMI 9 (for example, vibrations of the steering wheel 32 or the seat belt) and the like. Note that the notification unit 26 may notify the driver of two or more types of warnings (for example, visual warnings and auditory warnings). After step S204, this control routine ends.

[0062] Note that, if the control routine of FIG. 4 is omitted, steps S102 to S105 of FIG. 4 may be executed between steps S201 and S202 of FIG. 5. Further, the determination unit 25 may acquire driver information from the terminal 11 and determine whether the driver is in an awake state based on the driver information only when it is determined based on the monitoring image that the driver is not in the driving standby state. In this case, when the determination unit 25 determines that the driver is in the awake state, the determination criteria are relaxed, and using the relaxed determination criteria, it is determined again based on the monitoring image whether the driver is in the driving standby state. That is, if the control routine of FIG. 4 is omitted, steps S102, S103, and S105 of FIG. 4 and step S203 of FIG. 5 may be executed between steps S203 and S204 of FIG. 5. By acquiring the driver information from the terminal 11 only when it is determined that the driver is not in the driving standby state in this way, the power consumption in the terminal 11 and the ECU 20 can be reduced.

[0063] <Second Embodiment> The driver monitoring device according to the second embodiment is basically the same in configuration and control as the driver monitoring device according to the first embodiment, except for the points described below. Therefore, hereinafter, the second embodiment of the present invention will be described centering on the parts different from the first embodiment.

[0064] In the second embodiment, the determination unit 25 determines whether the driver is in the driving standby state based on a predetermined determination criterion based on the monitoring image, and when it is determined based on the monitoring image that the driver is not in the driving standby state, it determines whether the driver is in the awake state based on the driver information acquired by the terminal 11. Then, when the determination unit 25 determines that the driver is in the awake state, it invalidates the determination result based on the monitoring image and determines that the driver is in the driving standby state. By this, it is possible to suppress the driver who is operating the terminal 11 as a legitimate act from being erroneously determined not to be in the driving standby state.

[0065] Further, when the determination unit 25 determines based on the monitoring image that the driver is not in the driving standby state, it acquires driver information from the terminal 11, and when it determines based on the monitoring image that the driver is in the driving standby state, it does not acquire driver information from the terminal 11. By doing so, the power consumption in the terminal 11 and the ECU 20 can be reduced.

[0066] Hereinafter, with reference to FIG. 6, the above-described control will be described in detail. FIG. 6 is a flowchart showing a control routine for warning processing in the second embodiment of the present invention. This control routine is repeatedly executed at a predetermined execution interval by the processor 23 of the ECU 20.

[0067] Steps S301 to S303 are executed in the same manner as steps S201 to S203 in FIG. 5. In step S303, based on the initialized determination criteria, it is determined whether the driver is in the driving standby state based on the monitoring image. If it is determined in step S303 that the driver is not in the driving standby state, this control routine proceeds to step S304.

[0068] In step S304, in the same manner as step S102 in FIG. 4, the determination unit 25 acquires driver information from the terminal 11. Next, in step S305, in the same manner as step S103 in FIG. 4, the determination unit 25 determines whether the driver is in the awake state based on the driver information.

[0069] If it is determined in step S305 that the driver is in the awake state, this control routine proceeds to step S306. In step S306, the determination unit 25 invalidates the determination result based on the monitoring image. That is, the determination unit 25 determines that the driver is in the driving standby state. After step S306, this control routine ends.

[0070] On the other hand, if it is determined in step S305 that the driver is not in an awake state, this control routine proceeds to step S307. In this case, the determination result based on the monitoring image is maintained. Therefore, in step S307, similar to step S204 in FIG. 5, the notification unit 26 notifies the driver of a warning via the HMI 9. After step S307, this control routine ends. Note that step S304 may be executed between step S301 and step S303.

[0071] <Third Embodiment> The driver monitoring device according to the third embodiment is basically the same as the configuration and control of the driver monitoring device according to the first embodiment, except for the points described below. Therefore, hereinafter, the third embodiment of the present invention will be described centering on the differences from the first embodiment.

[0072] FIG. 7 is a functional block diagram of the processor 23 of the ECU 20 in the third embodiment. In the third embodiment, the processor 23 has a terminal control unit 27 in addition to the determination unit 25 and the notification unit 26. The determination unit 25, the notification unit 26, and the terminal control unit 27 are functional modules realized by the processor 23 of the ECU 20 executing a computer program stored in the memory 22 of the ECU 20. Note that these functional modules may be realized by dedicated arithmetic circuits provided in the processor 23, respectively.

[0073] The terminal control unit 27 controls the terminal 11 by transmitting a control signal to the terminal 11. In level 3 automated driving, operation of the terminal 11 by the driver is permitted, but in automated driving at level 2 or lower, operation of the terminal 11 by the driver is basically not permitted. However, in automated driving at level 2 or lower, there is a risk that the driver may be driven by an urge to operate the terminal 11. This is particularly prominent when the driver is operating the terminal 11 in level 3 automated driving.

[0074] Therefore, in the third embodiment, when the vehicle 30 is performing automated driving at level 2 or lower, the terminal control unit 27 restricts the operation of the terminal 11 by the driver. By doing so, it is possible to prevent the driver's attention from becoming distracted during automated driving at level 2 or lower.

[0075] In the third embodiment, in addition to the control routines of FIGS. 4 and 5, the control routine of FIG. 8 is executed. FIG. 8 is a flowchart showing the control routine of the terminal operation determination process in the third embodiment of the present invention. This control routine is repeatedly executed at a predetermined execution interval by the processor 23 of the ECU 20.

[0076] First, in step S401, the terminal control unit 27 of the processor 23 determines whether level 3 automated driving is being performed in the vehicle 30. If it is determined that level 3 automated driving is being performed, this control routine proceeds to step S402.

[0077] In step S402, the terminal control unit 27 permits the driver to operate the terminal 11. At this time, the terminal control unit 27 may transmit a control signal for permitting the operation of the terminal 11 to the terminal 11. After step S402, this control routine ends.

[0078] On the other hand, if it is determined in step S401 that level 3 automated driving is not being performed, that is, if automated driving at level 2 or lower is being performed, this control routine proceeds to step S403. In step S403, the determination unit 25 restricts the operation of the terminal 11 by the driver. Specifically, the determination unit 25 transmits a control signal for restricting the operation of the terminal 11 to the terminal 11. When the terminal 11 receives the control signal, the processor of the terminal 11 executes control for restricting the operation of the terminal 11. The operation restriction target of the terminal 11 is, for example, an operation using the driver's finger, and an operation by the driver's voice (for example, hands-free call, radio or music viewing, etc.) is excluded from the operation restriction target. After step S403, this control routine ends.

[0079] Note that the terminal control unit 27 may prohibit the driver from operating the terminal 11 when the vehicle 30 is performing automated driving at level 2 or lower. In this case, in step S403, the terminal control unit 27 transmits a control signal for prohibiting the operation of the terminal 11 to the terminal 11. As a result, the processor of the terminal 11 executes control for prohibiting the operation of the terminal 11.

[0080] <Other Embodiments> As described above, the preferred embodiments of the present invention have been described. However, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the claims. For example, a part of the configuration of the vehicle control system 1 shown in FIG. 1 may be omitted.

[0081] Also, the terminal 11 that can be operated by the driver is not limited to a smartphone, and may be other portable terminals such as a tablet terminal or a portable game machine, or a navigation device 6 provided in the vehicle 30. Further, in situations other than level 3 automated driving, the above-described control may be executed to monitor the driver's state.

[0082] Further, a server provided outside the vehicle 30 may function as a driver monitoring device. In this case, an image generated by an imaging device such as the driver monitor camera 2 and driver information acquired by the terminal 11 are transmitted from the vehicle 30 to the server via a communication network. When the determination unit of the server determines that the driver is not in the driving standby state, the notification unit of the server notifies the driver of a warning via the ECU 20 of the vehicle 30.

[0083] Further, a computer program for causing a computer to realize the functions of each part of the processor 23 of the ECU 20 or the processor of the server may be provided in a form stored in a computer-readable recording medium. The computer-readable recording medium is, for example, a magnetic recording medium, an optical recording medium, or a semiconductor memory.

[0084] In addition, the above-described embodiments can be implemented in any combination. For example, when the second embodiment and the third embodiment are combined, in the third embodiment, the control routine of FIG. 6 is executed instead of the control routines of FIGS. 4 and 5.

Description of Signs

[0085] 2 Driver monitoring camera 11 Terminal 20 Electronic control unit (ECU) 23 Processor 25 Judgment unit 26 Notification unit 30 Vehicle

Claims

1. A determination unit that determines whether or not the driver of the vehicle is in a driving standby state in which a driving operation can be performed; A notification unit that notifies the driver of a warning when it is determined that the driver is not in the driving standby state; It has, The determination unit acquires an image generated by an imaging device provided in the vehicle so as to photograph the driver and driver information acquired by a terminal operable by the driver, and determines whether or not the driver is in the driving standby state based on the image and the driver information. The determination unit acquires the driver information from the terminal when it is determined based on the image that the driver is not in the driving standby state, and does not acquire the driver information from the terminal when it is determined based on the image that the driver is in the driving standby state. A driver monitoring device.

2. A driver monitoring device, A determination unit that determines whether or not the driver of the vehicle is in a driving standby state in which a driving operation can be performed; A notification unit that notifies the driver of a warning when it is determined that the driver is not in the driving standby state; It has, The determination unit acquires an image generated by an imaging device provided in the vehicle so as to photograph the driver and driver information acquired by a terminal operable by the driver, and determines whether or not the driver is in the driving standby state based on the image and the driver information. The driver monitoring device further includes a terminal control unit that restricts or prohibits the operation of the terminal by the driver when the vehicle is performing level 2 or lower autonomous driving.

3. The driver monitoring device according to claim 1 or 2, wherein the driver information includes an image generated by a camera mounted on the terminal or input information to the terminal by the driver.

4. The determination unit determines whether or not the driver is in the driving standby state based on the image according to a predetermined determination criterion, determines whether or not the driver is awake based on the driver information, and relaxes the determination criterion when it is determined that the driver is awake. The driver monitoring device according to claim 1 or 2.

5. The determination unit determines whether or not the driver is in the driving standby state based on the image according to a predetermined determination criterion. When it is determined based on the image that the driver is not in the driving standby state, it determines whether or not the driver is in an awake state based on the driver information. When it is determined that the driver is in an awake state, the determination result based on the image is invalidated. The driver monitoring device according to claim 1 or 2.

6. The terminal is a smartphone. The driver monitoring device according to claim 1 or 2.

7. A driver monitoring method executed by a computer, comprising: acquiring an image generated by an imaging device provided in the vehicle so as to photograph a driver of the vehicle; acquiring driver information acquired by a terminal operable by the driver; determining whether or not the driver is in a driving standby state in which a driving operation can be performed based on the image and the driver information; when it is determined that the driver is not in the driving standby state, notifying the driver of a warning; when it is determined based on the image that the driver is not in the driving standby state, acquiring the driver information from the terminal, and when it is determined based on the image that the driver is in the driving standby state, not acquiring the driver information from the terminal A driver monitoring method including the above.

8. A driver monitoring method executed by a computer, comprising: acquiring an image generated by an imaging device provided in the vehicle so as to photograph a driver of the vehicle; acquiring driver information acquired by a terminal operable by the driver; determining whether or not the driver is in a driving standby state in which a driving operation can be performed based on the image and the driver information; when it is determined that the driver is not in the driving standby state, notifying the driver of a warning; when level 2 or lower automated driving is being performed in the vehicle, restricting or prohibiting the driver's operation of the terminal A driver monitoring method including the above.

9. acquiring an image generated by an imaging device provided in the vehicle so as to photograph a driver of the vehicle; acquiring driver information acquired by a terminal operable by the driver; Determining whether the driver is in a driving standby state in which the driver can perform a driving operation based on the image and the driver information; When it is determined that the driver is not in the driving standby state, notifying the driver of a warning; When it is determined based on the image that the driver is not in the driving standby state, acquiring the driver information from the terminal, and when it is determined based on the image that the driver is in the driving standby state, not acquiring the driver information from the terminal; A computer program for driver monitoring that causes a computer to execute the above.

10. Obtaining an image generated by an imaging device provided in the vehicle to photograph the driver of the vehicle; Obtaining driver information acquired by a terminal operable by the driver; Determining whether the driver is in a driving standby state in which the driver can perform a driving operation based on the image and the driver information; When it is determined that the driver is not in the driving standby state, notifying the driver of a warning; When level 2 or lower automated driving is being performed in the vehicle, restricting or prohibiting the driver from operating the terminal; A computer program for driver monitoring that causes a computer to execute the above.

Images