Driver monitoring device, driver monitoring method, and computer program for driver monitoring

Abstract

To provide a driver monitoring device capable of notifying a driver of a hands-on request at an appropriate timing.SOLUTION: A determination unit 31 configured to determine whether or not a hands-on request condition for requesting the driver to hold a steering wheel is satisfied based on a situation around the vehicle 10 or a state of the driver of the vehicle 10, a notification processing unit 32 configured to notify the driver of a hands-on request for requesting the driver to hold the steering wheel via a notification device 4 provided in the vehicle 10 when the hands-on request condition is satisfied and the driver does not hold the steering wheel continuously for a predetermined standby time, and a detection unit 33 configured to detect that the driver is changing the steering wheel; And a setting unit 34 configured to set the waiting time according to a frequency at which the hands-on request is notified when the switching of the steering is detected.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a driver monitoring device, a driver monitoring method, and a computer program for driver monitoring, which monitor a driver of a vehicle. [Background technology]

[0002] A technology has been proposed that requests the driver to hold the steering wheel under certain conditions when the vehicle's driving is automatically controlled with the driver's hands off the steering wheel (see Patent Document 1). In this technology, the driver monitoring system re-notifies the driver of the hands-on request or surroundings check request if the elapsed time since the notification of the hands-on request or surroundings check request to the driver exceeds a reference time. Furthermore, this driver monitoring system sets the reference time according to vehicle surroundings information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7409239 Summary of the Invention [Problem to be solved by the invention]

[0004] When the conditions for a hands-on request notification are met, if the driver changes his / her grip on the steering wheel, the system may determine that the driver has taken his / her hands off the steering wheel and notify the driver of the hands-on request. In such cases, the driver may find the notification annoying.

[0005] Therefore, an object of the present invention is to provide a driver monitor device that can notify the driver of a hands-on request at an appropriate timing. [Means for solving the problem]

[0006] According to one embodiment, a driver monitoring device is provided, which includes a determination unit that determines whether a hands-on request condition for requesting the driver to hold the steering wheel is satisfied based on the circumstances around the vehicle or the state of the driver of the vehicle, a notification processing unit that notifies the driver of a hands-on request requesting the driver to hold the steering wheel via a notification device provided in the vehicle when the hands-on request condition is satisfied and the driver does not hold the steering wheel continuously for a predetermined waiting time, a detection unit that detects when the driver is changing their grip on the steering wheel, and a setting unit that sets the waiting time depending on how often the hands-on request is notified when the driver's grip on the steering wheel is detected.

[0007] In one embodiment, the setting unit increases the waiting time by a first adjustment amount when the frequency at which the hands-on request is notified becomes equal to or greater than a first threshold.

[0008] In one embodiment, the setting unit shortens the waiting time by a second adjustment amount when the frequency at which the hands-on request is notified becomes equal to or less than a second threshold that is lower than the first threshold.

[0009] According to another embodiment, a driver monitoring device is provided, which includes a determination unit that determines whether a hands-on requirement condition that requests the driver to hold the steering wheel is satisfied based on the circumstances around the vehicle or the state of the driver of the vehicle, a detection unit that detects whether the driver is changing his / her grip on the steering wheel, and a notification processing unit that notifies the driver of a hands-on requirement requesting the driver to hold the steering wheel via a notification device provided in the vehicle when the hands-on requirement condition is satisfied and the driver has not continuously held the steering wheel for a predetermined waiting time and no change in the steering wheel grip is detected, but does not notify the hands-on requirement when a change in the steering wheel grip is detected.

[0010] According to another embodiment, a driver monitoring method is provided, which includes determining whether a hands-on request condition for requesting the driver to hold the steering wheel is satisfied based on a situation around the vehicle or a state of the driver of the vehicle, notifying the driver of a hands-on request requesting the driver to hold the steering wheel via a notification device provided in the vehicle when the hands-on request condition is satisfied and the driver does not hold the steering wheel continuously for a predetermined waiting time, detecting that the driver is changing his / her grip on the steering wheel, and setting the waiting time according to the frequency of the hands-on request notification when the driver's grip on the steering wheel is detected.

[0011] According to yet another embodiment, there is provided a computer program for driver monitoring, the computer program including instructions for causing a processor mounted on a vehicle to execute the following: determine whether a hands-on request condition for requesting the driver to hold the steering wheel is satisfied based on the circumstances around the vehicle or the state of the driver of the vehicle; if the hands-on request condition is satisfied and the driver does not hold the steering wheel continuously for a predetermined waiting time, notify the driver of a hands-on request requesting the driver to hold the steering wheel via a notification device provided in the vehicle; detect that the driver is changing his / her grip on the steering wheel; and set the waiting time according to the frequency of the hands-on request notification when the driver's grip on the steering wheel is detected. [Effects of the Invention]

[0012] The driver monitoring device according to the present disclosure has the effect of being able to notify the driver of a hands-on request at an appropriate time. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a schematic configuration diagram of a vehicle control system including a driver monitoring device. [Figure 2]FIG. 2 is a hardware configuration diagram of an ECU that is an example of a driver monitoring device. [Figure 3] FIG. 2 is a functional block diagram of a processor of the ECU. [Figure 4] FIG. 10 is a diagram illustrating an example of the relationship between the notification frequency of a hands-on request and a waiting time. [Figure 5] 10 is an operational flowchart of a driver monitoring process. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, a driver monitoring device, a driver monitoring method, and a computer program for driver monitoring executed by the driver monitoring device will be described with reference to the drawings. This driver monitoring device notifies the driver of a hands-on request when the driver does not hold the steering wheel continuously for a predetermined waiting time after a hands-on request condition, which requires the driver to hold the steering wheel, is satisfied. The driver monitoring device then adjusts the waiting time according to the frequency of hands-on requests when it is detected that the driver is changing his or her hands on the steering wheel. In this way, the driver monitoring device prevents the hands-on request from being notified when the driver changes his or her hands on the steering wheel.

[0015] FIG. 1 is a schematic diagram of a vehicle control system including a driver monitoring device. In this embodiment, the vehicle control system 1 is mounted on a vehicle 10 and controls the vehicle 10. The vehicle control system 1 includes an exterior camera 2, a driver monitoring camera 3, a notification device 4, and an electronic control unit (ECU) 5, which is an example of a driver monitoring device. The exterior camera 2, the driver monitoring camera 3, the notification device 4, and the ECU 5 are communicably connected to each other. The vehicle 10 may also be provided with a ranging sensor (not shown), such as a LiDAR or radar, that measures the distance to objects around the vehicle 10. The vehicle 10 may also be provided with a positioning device (not shown), such as a GPS receiver, that measures the position of the vehicle 10 using a satellite positioning system.

[0016] The exterior camera 2 is an example of an exterior imaging unit, and is attached to the vehicle 10 so as to face a predetermined area around the vehicle 10, such as the area ahead of the vehicle 10. Note that the vehicle 10 may be provided with a plurality of exterior cameras with different imaging directions or focal lengths. The exterior camera 2 captures an image of a predetermined area at each predetermined imaging cycle to generate an image of the predetermined area (hereinafter referred to as an exterior image), and outputs the generated exterior image to the ECU 5.

[0017] The driver monitor camera 3 is an example of an in-vehicle imaging unit, and is attached to the instrument panel or its vicinity, facing the driver, seated in the driver's seat of the vehicle 10, so that the head of the driver is included in the imaging target area. The driver monitor camera 3 may have a light source such as an infrared LED. The driver monitor camera 3 captures an image of the driver at each predetermined imaging period to generate an image of the driver (hereinafter referred to as a driver image), and outputs the generated driver image to the ECU 5.

[0018] The notification device 4 is provided in the passenger compartment of the vehicle 10 and is a device that provides a predetermined notification to the driver by light, sound, vibration, text display, or image display. To achieve this, the notification device 4 has at least one of a speaker, a light source, a vibrator, or a display device, for example. When the notification device 4 receives a notification signal representing a predetermined notification to the driver from the ECU 5, it notifies the driver by sound from the speaker, light emission or blinking of the light source, vibration of the vibrator, or display of a notification message or icon on the display device.

[0019] The ECU 5 controls the automatic driving of the vehicle 10 in accordance with the applied driving mode, or assists the driver of the vehicle 10 in driving. Furthermore, the ECU 5 monitors the driver of the vehicle 10 and notifies the driver of a hands-on request via the notification device 4 as necessary.

[0020] Fig. 2 is a hardware configuration diagram of the ECU 5. As shown in Fig. 2, the ECU 5 has a communication interface 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 may be configured as separate circuits, or may be configured integrally as a single integrated circuit.

[0021] The communication interface 21 has an interface circuit for connecting the ECU 5 to other devices in the vehicle. The communication interface 21 passes the outside image received from the outside camera 2 and the driver image received from the driver monitor camera 3 to the processor 23. The communication interface 21 also outputs the notification signal received from the processor 23 to the notification device 4.

[0022] The memory 22 is an example of a storage unit and includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores various data used in the driver monitoring process executed by the processor 23.

[0023] The processor 23 includes one or more central processing units (CPUs) and their peripheral circuits. The processor 23 may further include other arithmetic circuits such as a logic unit, a numerical calculation unit, or a graphics processing unit. The processor 23 executes driver monitoring processing.

[0024] 3 is a functional block diagram of the processor 23 relating to the driver monitoring process. The processor 23 has a determination unit 31, a notification processing unit 32, a detection unit 33, a setting unit 34, and a driving control unit 35. Each of these units included in the processor 23 is, for example, a functional module realized by a computer program running on the processor 23. Alternatively, each of these units included in the processor 23 may be a dedicated arithmetic circuit provided in the processor 23.

[0025] The determination unit 31 determines whether or not a hands-on requirement condition is satisfied based on the situation around the vehicle 10 or the situation of the driver of the vehicle 10. The hands-on requirement condition is a condition for requesting the driver to hold the steering wheel.

[0026] In one embodiment, the determination unit 31 determines that the hands-on requirement condition is satisfied when the situation around the vehicle 10 makes it difficult to continue autonomous driving control of the vehicle 10. For example, when the driving control unit 35 cannot detect lane markings from the outside-of-vehicle image, or when the driving control unit 35 notifies the determination unit 31 that the distance between the vehicle 10 and a lane marking that defines the lane in which the vehicle 10 is traveling is less than a predetermined lower limit threshold, the determination unit 31 determines that the hands-on requirement condition is satisfied.

[0027] Furthermore, the determination unit 31 determines that the hands-on requirement condition is satisfied when the driver is not ready to take over driving operation of the vehicle 10 from the ECU 5. For example, the determination unit 31 determines that the hands-on requirement condition is satisfied when the driver shows signs of drowsiness, continues to look away, or continues to not hold the steering wheel for a period exceeding the hands-off upper limit period.

[0028] The determination unit 31 estimates the driver's state based on the driver images to determine whether the hands-on requirement conditions are satisfied. For example, the determination unit 31 estimates the driver's drowsiness level at predetermined intervals based on a series of driver images acquired over a recent fixed period. If the driver's drowsiness level is equal to or higher than a predetermined level, the determination unit 31 determines that the driver is showing signs of drowsiness. To this end, the determination unit 31 detects the driver's gaze direction, eye opening degree (hereinafter referred to as eye opening degree), and mouth opening degree from each of the series of driver images. The determination unit 31 then estimates the driver's drowsiness level based on the detected gaze direction, eye opening degree, and mouth opening degree.

[0029] The determination unit 31 detects the area in the driver image where the driver's face appears (hereinafter referred to as the face area) by inputting the driver image into a classifier that has been trained in advance to detect the driver's face from the driver image. Such a classifier is configured as a deep neural network (DNN) with a convolutional neural network (CNN)-type architecture, such as the Single Shot MultiBox Detector (SSD) or Faster R-CNN, or a DNN with an attention mechanism, such as the Vision Transformer. Alternatively, such a classifier may be configured as a classifier based on a machine learning method other than a DNN, such as the AdaBoost classifier. The classifier is trained in advance using a large number of training images depicting human faces according to a predetermined learning method, such as backpropagation. The determination unit 31 further detects the driver's eyes and mouth from the face area. In this case, the determination unit 31 applies an edge detection filter, such as a Sobel filter, to detect edge pixels in the face area. The determination unit 31 then detects lines in which edge pixels are continuous in a substantially horizontal direction, and for each of the left and right eyes, detects two such lines aligned vertically within the range in the face region where the eye is expected to be located as the upper and lower eyelids of that eye. Similarly, the determination unit 31 detects the area surrounded by two such lines aligned vertically within the range in the face region where the mouth is expected to be located as the driver's mouth. Note that the determination unit 31 may also detect the upper and lower eyelids of each of the driver's left and right eyes from the driver image using other methods for detecting the upper and lower eyelids from an image. Similarly, the determination unit 31 may also detect the driver's mouth from the driver image using other methods for detecting the mouth from an image. For example, the above-mentioned face region detection classifier may be trained in advance to directly detect the upper eyelids, lower eyelids, and mouth from the driver image. In this case, the determination unit 31 can detect the upper eyelids, lower eyelids, and mouth by inputting the driver image into the classifier.

[0030] The determination unit 31 estimates the driver's eye opening degree based on the distance between the upper and lower eyelids of each of the left and right eyes in each of the most recent series of driver images. For example, the determination unit 31 may determine the eye opening degree by averaging the distance between the upper and lower eyelids of each of the left and right eyes. The determination unit 31 may also estimate the eye opening degree using other methods that calculate the eye opening degree from the upper and lower eyelids in the images. The determination unit 31 then calculates the time from a maximum eye opening degree to the next maximum eye opening degree as the duration of one blink of the driver based on the time-series change in the eye opening degree in each of the series of driver images. The determination unit 31 then counts the number of blinks during a recent fixed period and calculates the average time between blinks as the blink cycle. Furthermore, the determination unit 31 calculates the ratio of the vertical length to the horizontal length of the mouth in each of the head images during the most recent fixed period and calculates the average value as the degree of mouth opening. The determination unit 31 may calculate the degree to which the driver's mouth is open according to another method for calculating the degree to which the mouth is open from the area on the image where the mouth is represented.

[0031] Furthermore, the determination unit 31 detects the driver's gaze direction from each of the most recent series of driver images. For example, the determination unit 31 detects the corneal reflection image of the light source and the center of gravity of the pupil (hereinafter simply referred to as the pupil center of gravity) from the area surrounded by the upper eyelid and the lower eyelid (hereinafter referred to as the eye area) for at least one of the driver's left and right eyes displayed on the driver image. The corneal reflection image of the light source is also called a Purkinje image. In this case, the determination unit 31 detects the Purkinje image by, for example, template matching between a Purkinje image template and the eye area. Similarly, the determination unit 31 may detect the pupil by template matching between a pupil template and the eye area, and determine the center of gravity of the area in which the detected pupil is displayed as the pupil center of gravity. The determination unit 31 may also detect the Purkinje image and the pupil center of gravity using another method for detecting the Purkinje image and the pupil center of gravity from the eye area. The determination unit 31 then calculates the direction and distance from the Purkinje image to the pupil center, and detects the driver's gaze direction by referring to a table that shows the relationship between the direction and distance and the driver's gaze direction. Such a table may be stored in advance in the memory 22. The determination unit 31 then calculates the amount of movement in the gaze direction for each pair of two consecutive driver images, and calculates the average movement amount by dividing the acquisition interval of the driver images to determine the movement speed in the gaze direction.

[0032] The determination unit 31 estimates the driver's drowsiness level based on at least one of the number and frequency of blinks, the degree of mouth opening, and the gaze movement speed. If the drowsiness level is equal to or higher than a predetermined level, the determination unit 31 determines that the driver has signs of drowsiness. For example, the determination unit 31 determines that the driver has signs of drowsiness if the number of blinks in a recent fixed period is equal to or higher than a predetermined number, the blink period is longer than a predetermined time threshold, the gaze movement speed is equal to or lower than a predetermined speed threshold, and the mouth opening degree is greater than a predetermined opening degree. On the other hand, the determination unit 31 determines that the driver does not have signs of drowsiness if at least one of the number and frequency of blinks, the degree of mouth opening, and the gaze movement speed in a recent fixed period does not satisfy the above conditions. Alternatively, the determination unit 31 may set a drowsiness sign detection condition for a combination of up to three of the number and frequency of blinks, the degree of mouth opening, and the gaze movement speed. The determination unit 31 determines that the driver has signs of drowsiness when the detection conditions are met, and determines that the driver does not have signs of drowsiness when the detection conditions are not met. If the driver has signs of drowsiness, the determination unit 31 determines that the hands-on requirement conditions are met.

[0033] Furthermore, the determination unit 31 determines that the driver is not looking in the direction of travel of the vehicle if the line of sight is outside a predetermined angle range corresponding to the front direction of the vehicle 10. Therefore, if the driver's line of sight is continuously outside the predetermined angle range for a predetermined period of time, the determination unit 31 determines that the driver is continuously looking away, and as a result, the hands-on requirement condition is satisfied.

[0034] Furthermore, if the signal output from a touch sensor (not shown) on the steering wheel indicates that the driver is not touching the steering wheel for a period exceeding the hands-off upper limit, the determination unit 31 determines that the driver is not holding the steering wheel and that the hands-on requirement condition is met. Alternatively, the determination unit 31 may input a driver image to a classifier pre-trained to detect the driver's hands and the steering wheel, thereby detecting a hand region representing the driver's hands and a steering region representing the steering wheel on the driver image. The determination unit 31 may then determine that the driver is not holding the steering wheel if the hand region and the steering region on the driver image are separated by a predetermined distance or more. In this case, too, the determination unit 31 may determine that the hands-on requirement condition is met if the period during which the driver is determined not to be holding the steering wheel based on the driver image continues beyond the hands-off upper limit threshold. The classifier for detecting the hand region and the steering region is configured as a CNN or a DNN with an attention mechanism.

[0035] The determination unit 31 outputs the determination result to the notification processing unit 32 and the setting unit 34 every time the determination result as to whether or not the hands-on request condition is satisfied changes.

[0036] If the driver does not hold the steering wheel continuously for the waiting time after receiving the determination result that the hands-on request condition has been met from the determination unit 31, the notification processing unit 32 notifies the driver of a hands-on request via the notification device 4. As described above with respect to the determination unit 31, the notification processing unit 32 may determine whether the driver has held the steering wheel based on a signal from a touch sensor provided on the steering wheel or an image of the driver.

[0037] Furthermore, if the hands-on request condition is no longer satisfied or the driver holds the steering wheel before the waiting time has elapsed after the hands-on request condition is satisfied, the notification processing unit 32 does not notify the hands-on request. Furthermore, if the driver holds the steering wheel after starting to notify the hands-on request, the notification processing unit 32 stops notifying the hands-on request.

[0038] On the other hand, if the driver does not hold the steering wheel even after the first grace period has elapsed since the start of the hands-on request notification, the notification processing unit 32 may increase the intensity of the hands-on request notification. For example, the notification processing unit 32 may increase the volume of the sound emitted from the speaker of the notification device 4 or increase the light emission intensity of the light source of the notification device 4 to increase the intensity of the hands-on request notification. Alternatively, the notification processing unit 32 may increase the number of devices that notify the hands-on request.

[0039] When the notification processing unit 32 starts notifying the hands-on request, it notifies the setting unit 34 of that fact. Similarly, when the notification processing unit 32 stops notifying the hands-on request, it notifies the setting unit 34 of that fact. Furthermore, when a second grace period has elapsed since the start of the hands-on request notification without the driver holding the steering wheel, the notification processing unit 32 may notify the driving control unit 35 of that fact. Note that the second grace period is set to a period longer than the first grace period.

[0040] The detection unit 33 detects when the driver changes the way they hold the steering wheel. For example, the detection unit 33 sequentially inputs a plurality of driver images to a classifier that has been trained in advance to detect when the driver changes the way they hold the steering wheel, in the order in which the driver images were generated by the driver monitor camera 3. As a result, when the driver changes the way they hold the steering wheel, the classifier outputs a signal indicating that the change of hand is detected. Such a classifier is configured as a DNN with a recursive structure, such as a recurrent neural network (RNN) or LSTM.

[0041] Furthermore, when the steering angle of the steering wheel, which is represented by a signal received by the ECU 5 from a steering angle sensor (not shown) provided on the steering wheel, becomes equal to or greater than a predetermined angle threshold, the detection unit 33 may detect that the driver is changing his / her grip on the steering wheel. The predetermined angle threshold is set to a steering angle at which it becomes difficult to hold the steering wheel without changing his / her grip on the steering wheel.

[0042] Every time the detection unit 33 detects that the driver has changed the way he holds the steering wheel, it notifies the setting unit 34 of this.

[0043] The setting unit 34 sets the length of the waiting period from when the hands-on request condition is satisfied until the hands-on request is actually notified, depending on the frequency with which the hands-on request is notified when the driver's change of grip on the steering wheel is detected (hereinafter referred to as the notification frequency).

[0044] In one embodiment, the setting unit 34 calculates the notification frequency as the ratio of the number of times a hands-on request is notified when the driver changes his grip on the steering wheel to the driving time or driving distance of the vehicle 10 when the hands-on requirement condition is satisfied. When the setting unit 34 is notified by the determination unit 31 that the hands-on requirement condition is satisfied, the setting unit 34 starts measuring the driving time when the hands-on requirement condition is satisfied. Furthermore, the setting unit 34 stores the driving distance of the vehicle 10 at that time in the memory 22. When the setting unit 34 is notified by the determination unit 31 that the hands-on requirement condition is no longer satisfied, the setting unit 34 stops measuring the driving time when the hands-on requirement condition is satisfied. Furthermore, the setting unit 34 calculates the driving distance when the hands-on requirement condition is satisfied by subtracting the driving distance of the vehicle 10 when the hands-on requirement condition is notified, stored in the memory 22, from the driving distance of the vehicle 10 when the hands-on requirement condition is no longer satisfied. Each time a hands-on requirement condition is satisfied, the setting unit 34 adds the driving time calculated as described above to the total driving time under the condition that the hands-on requirement condition has been satisfied up to that point. Similarly, each time a hands-on requirement condition is satisfied, the setting unit 34 adds the driving distance calculated as described above to the total driving distance under the condition that the hands-on requirement condition has been satisfied up to that point.

[0045] Furthermore, if a hands-on request is notified when the detection unit 33 detects that the driver has changed his / her grip on the steering wheel, it is considered that the notification was caused by the driver changing his / her grip on the steering wheel. Therefore, the setting unit 34 counts the number of times a hands-on request is notified when the driver has detected that he / she has changed his / her grip on the steering wheel. The setting unit 34 then calculates the notification frequency by dividing the number of times by the total driving time or total driving distance when the hands-on request condition is satisfied.

[0046] The setting unit 34 compares the notification frequency with a first threshold. When the notification frequency is equal to or greater than the first threshold, the setting unit 34 lengthens the waiting time by a first adjustment amount. This allows the setting unit 34 to prevent a hands-on request from being notified when the driver changes their grip on the steering wheel. The setting unit 34 may lengthen the waiting time by the first adjustment amount each time the notification frequency is equal to or greater than the first threshold. However, it is preferable that the setting unit 34 does not lengthen the waiting time any further once the waiting time reaches an upper limit time length. This prevents the period from being excessively long between the satisfaction of the hands-on request condition and the actual notification of the hands-on request, and as a result, prevents the hands-on request from being excessively less likely to be notified.

[0047] Furthermore, the setting unit 34 compares the notification frequency with a second threshold. The second threshold is set to a value smaller than the first threshold. When the notification frequency becomes equal to or less than the second threshold, the setting unit 34 shortens the waiting time by the second adjustment amount. This allows the setting unit 34 to suppress a hands-on request from being notified while the driver is changing his / her grip on the steering wheel, while also preventing the hands-on request from being excessively notified. The setting unit 34 may shorten the waiting time by the second adjustment amount each time the notification frequency becomes equal to or less than the second threshold. However, it is preferable that the setting unit 34 does not shorten the waiting time below the lower limit time length. The second adjustment amount may be the same value as the first adjustment amount, or may be a value smaller than the first adjustment amount.

[0048] The setting unit 34 may reset the notification frequency once every time the length of the waiting time is changed. After resetting the notification frequency, the setting unit 34 may compare the notification frequency recalculated after the reset with the first threshold value and the second threshold value once the driving time or driving distance reaches a certain length or more.

[0049] Figure 4 shows an example of the relationship between the notification frequency of a hands-on request and waiting time. In each of the upper and lower charts in Figure 4, the horizontal axis represents elapsed time. The vertical axis in the upper chart represents the notification frequency, and graph 401 represents the change in notification frequency over time. Meanwhile, the vertical axis in the lower chart represents waiting time, and graph 402 represents the change in waiting time over time.

[0050] As shown in graphs 401 and 402, the waiting time is kept constant until the notification frequency reaches the first threshold Th1. When the notification frequency reaches the first threshold Th1 at time t1, the waiting time is adjusted to be longer by the first adjustment amount Δ1. After that, the notification frequency temporarily becomes smaller than the first threshold Th, but at time t2, the notification frequency reaches the first threshold Th1 again. Therefore, at time t2, the waiting time is readjusted to be longer by the first adjustment amount Δ1. Since the waiting time increases each time the notification frequency reaches the first threshold, the frequency at which the time required to change the steering wheel grip exceeds the waiting time decreases. As a result, the hands-on request is notified less frequently when the steering wheel grip is changed. After that, at time t3, the notification frequency becomes equal to or less than the second threshold Th2. Therefore, after time t3, the waiting time is adjusted to be shorter by the second adjustment amount Δ2. By shortening the waiting time when the notification frequency becomes equal to or less than the second threshold, the waiting time is prevented from becoming excessively long.

[0051] The driving control unit 35 controls the driving of the vehicle 10 in a driving assistance mode equivalent to Level 2 or Level 3 autonomous driving mode as defined by the Society of Automotive Engineers (SAE). To this end, the driving control unit 35 detects leading vehicles and lane markings from an exterior image. The driving control unit 35 detects other vehicles and lane markings by inputting the exterior image into a classifier that has been trained in advance to detect other vehicles and lane markings traveling around the vehicle 10 from the exterior image. Such a classifier is configured as a CNN or a DNN with an attention mechanism. The driving control unit 35 then identifies, as a leading vehicle, another vehicle located in the area between the two lane markings closest to the vehicle 10 in the exterior image. The position of the bottom edge of the object area representing the leading vehicle in the exterior image is assumed to represent the position where the leading vehicle contacts the road surface. Therefore, the driving control unit 35 estimates the inter-vehicle distance from the vehicle 10 to the preceding vehicle based on parameters of the exterior camera 2, such as the shooting direction and installation position of the exterior camera 2, and the position of the bottom edge of the object area on the vehicle exterior image. Furthermore, if the vehicle 10 is equipped with a distance measurement sensor (not shown), the driving control unit 35 may use the distance measured by the distance measurement sensor in the direction corresponding to the object area in which the preceding vehicle is displayed as the inter-vehicle distance from the vehicle 10 to the preceding vehicle.

[0052] When the distance between the preceding vehicle and vehicle 10 is less than a predetermined distance, the driving control unit 35 controls the powertrain or brakes of vehicle 10 so as to increase the distance between the preceding vehicle and vehicle 10. When the distance between the preceding vehicle and vehicle 10 is equal to or greater than a predetermined distance, the driving control unit 35 controls the powertrain of vehicle 10 so as to bring the speed of vehicle 10 closer to a target speed.

[0053] Furthermore, the driving control unit 35 sets a planned driving trajectory along the center of the two lane markings closest to the vehicle 10 and controls the steering device of the vehicle 10 so that the vehicle 10 drives along the set planned driving trajectory. If the driving control unit 35 cannot detect a lane marking from the exterior image for a certain period of time, it notifies the determination unit 31 that the detection of the lane marking has failed. The driving control unit 35 also calculates the distance between the vehicle 10 and each of the two lane markings that define the vehicle's own lane based on the positions of the lowermost ends of the two lane markings on the exterior image, the parameters of the exterior camera 2, and the lengths from the mounting position of the exterior camera 2 to the left and right ends of the vehicle 10. If the distance between either lane marking and the vehicle 10 falls below a predetermined lower threshold, the driving control unit 35 notifies the determination unit 31 of this fact.

[0054] Furthermore, when the notification processing unit 32 notifies that a second grace period has elapsed since the start of the hands-on request notification without the driver holding the steering wheel, the driving control unit 35 may control the powertrain and brakes to bring the vehicle 10 to an emergency stop.

[0055] 5 is an operational flowchart of the driver monitoring process executed by the processor 23. The determination unit 31 determines whether the hands-on request condition is satisfied (step S101). If the hands-on request condition is satisfied (step S101—Yes), the notification processing unit 32 determines whether a waiting time has elapsed with the driver not holding the steering wheel (step S102). If the waiting time has elapsed with the driver not holding the steering wheel (step S102—Yes), the notification processing unit 32 notifies the driver of a hands-on request via the notification device 4 (step S103). On the other hand, if the driver holds the steering wheel before the waiting time has elapsed (step S102—No) or if the hands-on request condition is not satisfied (step S101—No), the notification processing unit 32 does not notify the hands-on request, and the processor 23 repeats the processes from step S101 onwards.

[0056] The detection unit 33 detects the driver's change of grip on the steering wheel (step S104). If a change of grip on the steering wheel is detected (step S104—Yes), the setting unit 34 determines whether a hands-on request notification was made at the time of the detection of the change of grip on the steering wheel. That is, the setting unit 34 determines whether the hands-on request notification was made due to the driver's change of grip on the steering wheel (step S105). If the hands-on request notification was made due to the driver's change of grip on the steering wheel (step S105—Yes), the setting unit 34 corrects the notification frequency to increase the notification frequency (step S106). On the other hand, if a change of grip on the steering wheel is not detected (step S104—No) or if the hands-on request notification was not due to the driver's change of grip on the steering wheel (step S105—No), the setting unit 34 corrects the notification frequency to decrease the notification frequency (step S107).

[0057] The setting unit 34 determines whether the notification frequency is equal to or greater than a first threshold Th1 (step S108). If the notification frequency is equal to or greater than the first threshold Th1 (step S108—Yes), the setting unit 34 lengthens the waiting time by a first adjustment amount (step S109). On the other hand, if the notification frequency is less than the first threshold Th1 (step S109—No), the setting unit 34 determines whether the notification frequency is equal to or less than a second threshold Th2 (step S110). If the notification frequency is equal to or less than the second threshold Th2 (step S110—Yes), the setting unit 34 shortens the waiting time by a second adjustment amount (step S111). After step S109 or step S111, or if the notification frequency is greater than the second threshold Th2 in step S110 (step S110—No), the processor 23 ends the driver monitoring process. The order of the processes in steps S101 to S103 and the process in step S104 may be reversed, or they may be executed in parallel.

[0058] As explained above, this driver monitoring device adjusts the waiting time from when the hands-on request condition is satisfied until the hands-on request is actually notified, depending on the frequency with which the hands-on request is notified when it is detected that the driver is changing his / her grip on the steering wheel. This allows the driver monitoring device to prevent hands-on requests from being notified due to the driver changing his / her grip on the steering wheel, and as a result, it can notify the driver of the hands-on request at an appropriate time.

[0059] According to a modified example, the notification processing unit 32 may not notify the driver of a hands-on request when the detection unit 33 detects that the driver has changed his / her grip on the steering wheel even if the hands-on request condition is satisfied and a predetermined waiting time has elapsed since the hands-on request condition was satisfied. In other words, the notification processing unit 32 may notify the driver of a hands-on request via the notification device 4 only when the hands-on request condition is satisfied, a predetermined waiting time has elapsed since the hands-on request condition was satisfied, and the driver's changing his / her grip on the steering wheel has not been detected. In this case, the processing of the setting unit 34 may be omitted. According to this modified example, a hands-on request is prevented from being notified when the driver has changed his / her grip on the steering wheel even when the hands-on request condition is satisfied.

[0060] A computer program for realizing the driver monitoring process according to the above embodiment or modification may be provided in a form recorded on a computer-readable portable recording medium.

[0061] As described above, those skilled in the art can make various modifications to the embodiments within the scope of the present invention. [Explanation of symbols]

[0062] 1 Vehicle control system, 2 Exterior camera, 3 Driver monitor camera, 4 Notification device, 5 Electronic control device (driver monitor device), 10 Vehicle, 21 Communication interface, 22 Memory, 23 Processor, 31 Determination unit, 32 Notification processing unit, 33 Detection unit, 34 Setting unit, 35 Driving control unit

Claims

1. a determination unit that determines whether a hands-on requirement condition for requesting the driver to hold the steering wheel is satisfied based on the surrounding conditions of the vehicle or the state of the driver of the vehicle; a notification processing unit that notifies the driver of a hands-on request to hold the steering wheel via a notification device provided in the vehicle when the driver does not hold the steering wheel continuously for a predetermined waiting time when the hands-on request condition is satisfied; and a detection unit that detects whether the driver is changing the grip of the steering wheel; a setting unit that sets the waiting time in accordance with the frequency with which the hands-on request is notified when the steering wheel change is detected; A driver monitor device having:

2. The driver monitor device according to claim 1 , wherein the setting unit increases the standby time by a first adjustment amount when the frequency becomes equal to or greater than a first threshold value.

3. The driver monitor device according to claim 2 , wherein the setting unit shortens the standby time by a second adjustment amount when the frequency becomes equal to or less than a second threshold that is lower than the first threshold.

4. a determination unit that determines whether a hands-on requirement condition that requires the driver to hold the steering wheel is satisfied based on the surrounding conditions of the vehicle or the state of the driver of the vehicle; a detection unit that detects whether the driver is changing the grip of the steering wheel; a notification processing unit that notifies the driver of a hands-on request to hold the steering wheel via a notification device provided in the vehicle when the hands-on request condition is satisfied and the driver has not continuously held the steering wheel for a predetermined waiting time and when a change in the driver's grip on the steering wheel is not detected, and does not notify the driver of the hands-on request when a change in the driver's grip on the steering wheel is detected; A driver monitor device having:

5. Determine whether a hands-on requirement condition for requesting the driver to hold the steering wheel is satisfied based on the surrounding circumstances of the vehicle or the state of the driver of the vehicle; When the hands-on request condition is satisfied and the driver does not hold the steering wheel continuously for a predetermined waiting time, a hands-on request is notified to the driver via a notification device provided in the vehicle, requesting that the driver hold the steering wheel; Detecting that the driver is changing the grip of the steering wheel, The waiting time is set according to the frequency with which the hands-on request is notified when the change of the steering wheel grip is detected. A driver monitoring method comprising:

6. Determine whether a hands-on requirement condition for requesting the driver to hold the steering wheel is satisfied based on the surrounding circumstances of the vehicle or the state of the driver of the vehicle; When the hands-on request condition is satisfied and the driver does not hold the steering wheel continuously for a predetermined waiting time, a hands-on request is notified to the driver via a notification device provided in the vehicle, requesting that the driver hold the steering wheel; Detecting that the driver is changing the grip of the steering wheel, The waiting time is set according to the frequency with which the hands-on request is notified when the change of the steering wheel grip is detected. A computer program for a driver monitor that causes a processor mounted on the vehicle to execute the above.

Images