Driver monitoring device, driver monitoring method and computer program for driver monitoring

Abstract

To provide a driver monitoring device capable of appropriately executing a warning of inattentive driving to a driver.SOLUTION: A driver monitoring device comprises: an object detection section 31 for detecting an object which may exert any influence upon travel of a vehicle 10 on the basis of a vehicle outside sensor signal representing a situation around the vehicle 10; a direction detection section 32 for detecting at least a visual line direction between the visual line direction of a driver and a direction of the face of the driver on the basis of a driver image representing the driver; a determination section 33 for determining whether or not an object direction to the object detected from the driver is included in a peripheral view range corresponding to a peripheral field-of-view of the driver on the basis of the visual line direction; a setting section 34 for setting an inattentive driving warning condition in a case where the object direction is included in the peripheral view range based on a driving mode being applied to the vehicle 10 or a situation around the vehicle 10; and a notification processing section 36 which notifies the driver of an inattentive driving warning via a notification apparatus 4 in a case where the direction of the face or the visual line direction of the driver satisfies the inattentive driving warning condition.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a driver monitoring device, a driver monitoring method, and a computer program for driver monitoring, which monitor a driver of a vehicle. [Background technology]

[0002] A technology has been proposed for detecting when a driver is looking away from the road (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-15550 A Summary of the Invention [Problem to be solved by the invention]

[0004] In some cases, the driver may not be facing the direction of travel of the vehicle even after understanding the situation around the vehicle. If a warning about inattentive driving is issued even in such a case, the driver may feel annoyed.

[0005] SUMMARY OF THE PRESENT DISCLOSURE In view of the above, an object of the present invention is to provide a driver monitor device that can appropriately issue a warning to the driver about inattentive driving. [Means for solving the problem]

[0006] According to one embodiment, a driver monitor device is provided, which includes an object detection unit that detects an object that may affect vehicle driving based on an external sensor signal that indicates the situation around the vehicle, a direction detection unit that detects at least the line of sight direction of the driver and the direction of the driver's face based on a driver image showing the driver of the vehicle, a determination unit that determines whether the object direction to the detected object is included within a peripheral vision range corresponding to the driver's peripheral vision based on the line of sight direction, a setting unit that sets an inattentive warning condition when the object direction is included in the peripheral vision range based on a driving mode applied to the vehicle or the situation around the vehicle, and a notification processing unit that notifies the driver of an inattentive warning via a notification device when at least one of the detected direction of the driver's face and the line of sight direction satisfies the inattentive warning condition.

[0007] In this driver monitor device, it is preferable that the setting unit sets the inattentive warning condition when the driving mode is a first driving mode that does not require the driver to hold the steering wheel stricter than the inattentive warning condition when the driving mode is a second driving mode that requires the driver to hold the steering wheel.

[0008] In addition, when the driving mode is the first driving mode, it is preferable that the setting unit sets the inattentive warning condition when the object direction is within the peripheral vision range to be stricter than the inattentive warning condition when the object direction is outside the peripheral vision range.

[0009] Furthermore, it is preferable that the object detection unit further determines whether the detected object is moving or not, and when the driving mode is the first driving mode and the object direction is included in the peripheral vision range, the setting unit sets the inattentive warning condition when the detected object is stationary to be stricter than the inattentive warning condition when the detected object is moving.

[0010] According to another embodiment, a driver monitoring method is provided, which includes detecting an object that may affect vehicle driving based on an external sensor signal representing a situation around the vehicle, detecting at least a line of sight direction of the driver and a direction of the driver's face based on an in-vehicle image showing the driver of the vehicle, determining whether or not an object direction from the driver to the detected object is included within a peripheral vision range corresponding to the peripheral vision of the driver based on the line of sight direction, setting an inattentive warning condition when the object direction is included within a predetermined angle range based on a driving mode applied to the vehicle or the situation around the vehicle, and notifying the driver of an inattentive warning via a notification device when at least one of the detected direction of the driver's face and the line of sight direction satisfies the inattentive warning condition.

[0011] According to still another embodiment, a computer program for a driver monitor is provided, which includes instructions to cause a processor mounted on a vehicle to execute the following: detect an object that may affect vehicle driving based on an external sensor signal representing a situation around the vehicle, detect at least a line of sight direction of the driver and a direction of the driver's face based on an in-vehicle image showing the driver of the vehicle, determine whether or not an object direction from the driver to the detected object is included within a predetermined angle range corresponding to the peripheral vision of the driver based on the line of sight direction, set an inattentive warning condition when the object direction is included in the peripheral vision range based on a driving mode applied to the vehicle or the situation around the vehicle, and notify the driver of an inattentive warning via a notification device when at least one of the detected direction of the driver's face and the line of sight direction satisfies the inattentive warning condition. Effect of the Invention

[0012] The driver monitoring device according to the present disclosure has an effect of being able to appropriately issue a distracted driving warning to the driver. [Brief description of the drawings]

[0013] [Figure 1]1 is a schematic configuration diagram of a vehicle control system including a driver monitor device. [Diagram 2] FIG. 2 is a hardware configuration diagram of an ECU that is an example of a driver monitor device. [Diagram 3] FIG. 2 is a functional block diagram of a processor of the ECU. [Figure 4] 13A and 13B are diagrams showing examples of inattentive warning conditions. [Diagram 5] 13 is an operation flowchart of a driver monitor process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, a driver monitor device, a driver monitor method executed by the driver monitor device, and a computer program for the driver monitor will be described with reference to the drawings. The driver monitor device sets the conditions for an inattentive warning when an object that may affect the vehicle's running is included in the driver's peripheral vision, according to the driving mode or the situation around the vehicle while being applied to the vehicle. In this embodiment, inattentiveness refers to a state in which the driver is not monitoring the front direction of the vehicle, that is, the traveling direction of the vehicle when the vehicle is moving forward.

[0015] 1 is a schematic diagram of a vehicle control system including a driver monitoring device. In this embodiment, a vehicle control system 1 that is mounted on a vehicle 10 and controls the vehicle 10 has at least one outside sensor 2, a driver monitor camera 3, a notification device 4, and an electronic control unit (ECU) 5 that is an example of the driver monitoring device. The outside sensor 2, the driver monitor camera 3, the notification device 4, and the ECU 5 are connected to each other so as to be able to communicate with each other.

[0016] The external sensor 2 is a sensor for detecting the situation around the vehicle 10, and is, for example, a camera or a distance measuring sensor such as a radar or LiDAR. Note that the vehicle 10 may be provided with a plurality of external sensors with different detection ranges or types. The external sensor 2 generates an external sensor signal representing the situation of a predetermined area around the vehicle 10, such as an area in front of the vehicle 10, at predetermined intervals, and outputs the generated external sensor signal to the ECU 5.

[0017] The driver monitor camera 3 is an example of an in-vehicle sensor, and is attached to the instrument panel or its vicinity facing the driver so that the head of the driver seated in the driver's seat of the vehicle 10 is included in the imaging target area. The driver monitor camera 3 may have a light source such as an infrared LED. The driver monitor camera 3 captures an image of the driver (hereinafter referred to as a driver image) at each predetermined imaging period, and outputs the generated driver image to the ECU 5.

[0018] The notification device 4 is provided in the vehicle cabin of the vehicle 10, and is a device that provides a predetermined notification to the driver by light, sound, vibration, text display, or image display. For this purpose, the notification device 4 has at least one of a speaker, a light source, a vibrator, or a display device, for example. When the notification device 4 receives a notification indicating a warning to the driver from the ECU 5, it notifies the driver of the warning by sound from the speaker, light emission or blinking of the light source, vibration of the vibrator, or display of a warning message on the display device.

[0019] The ECU 5 controls the automatic driving of the vehicle 10 according to the applied driving mode, or assists the driving of the driver of the vehicle 10. Furthermore, the ECU 5 notifies the driver of an inattentive warning via the notification device 4 when an inattentive warning condition is satisfied.

[0020] Fig. 2 is a hardware configuration diagram of the ECU 5. As shown in Fig. 2, the ECU 5 has a communication interface 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 may each be configured as a separate circuit, or may be configured integrally as a single integrated circuit.

[0021] The communication interface 21 has an interface circuit for connecting the ECU 5 to other devices in the vehicle. The communication interface 21 passes the outside sensor signal or the driver image received from the outside sensor 2 or the driver monitor camera 3 to the processor 23 via the in-vehicle network. The communication interface 21 also outputs the notification signal received from the processor 23 to the notification device 4 via the in-vehicle network.

[0022] The memory 22 is an example of a storage unit and includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores various data used in the driver monitoring process executed by the processor 23.

[0023] The processor 23 includes one or more central processing units (CPUs) and their peripheral circuits. The processor 23 may further include other arithmetic circuits such as a logic arithmetic unit, a numerical arithmetic unit, or a graphic processing unit. The processor 23 executes the driver monitor process.

[0024] 3 is a functional block diagram of the processor 23 related to the driver monitor process. The processor 23 has an object detection unit 31, a direction detection unit 32, a determination unit 33, a setting unit 34, an inattentive driving determination unit 35, and a notification processing unit 36. Each of these units in the processor 23 is a functional module realized by a computer program running on the processor 23, for example. Alternatively, each of these units in the processor 23 may be a dedicated arithmetic circuit provided in the processor 23.

[0025] The object detection unit 31 detects objects that may affect the traveling of the vehicle 10 based on the external sensor signal generated by the external sensor 2. Objects that may affect the traveling of the vehicle 10 include, for example, other vehicles traveling around the vehicle 10, such as pedestrians and motorcycles, traffic lights, and road signs such as stop signs. In the following, an object that may affect the traveling of the vehicle 10 may be referred to as a target object.

[0026] The object detection unit 31 detects the target object represented by the external sensor signal by inputting the external sensor signal to a classifier that has been trained in advance to detect the target object, and classifies the type of the detected target object. As the classifier, a classifier based on a machine learning method such as a deep neural network (DNN) having a convolutional neural network (CNN) type architecture, AdaBoost, or a support vector machine is used.

[0027] Furthermore, the object detection unit 31 identifies the direction of the detected target object from the driver of the vehicle 10 (hereinafter, sometimes referred to as the object direction). When the external sensor 2 is a camera, the object detection unit 31 identifies, as the object direction, the direction corresponding to the position of the center of gravity of the object area in which the target object is displayed on the image, which is an example of an external sensor signal, generated by the camera. Although the position of the external sensor 2 and the position of the driver are not the same, the difference between their positions is sufficiently small compared to the expected distance from the vehicle 10 to the target object. Therefore, the object direction is obtained assuming that the difference between the direction from the external sensor 2 to the target object and the direction from the driver to the target object can be ignored. In this case, the object detection unit 31 may identify the direction corresponding to the position of the center of gravity of the object area based on the camera parameters such as the mounting position of the camera 2 with respect to the vehicle 10, the shooting direction, and the focal length. When the external sensor 2 is a distance measurement sensor, the object detection unit 31 may identify, as the object direction, the direction in which the target object is detected in the distance measurement signal, which is another example of an external sensor signal, generated by the distance measurement sensor.

[0028] Furthermore, the object detection unit 31 applies a predetermined tracking method such as KLT tracking to the target object detected in each of the multiple exterior sensor signals obtained in time series. As a result, the object detection unit 31 associates the same target object with each other in the multiple exterior sensor signals. Furthermore, the object detection unit 31 determines whether a target object that may move, such as a pedestrian, is moving based on the tracking result. In this case, the object detection unit 31 obtains the position of the target object at the time of generation of each exterior sensor signal, and if the amount of movement of the position is greater than a predetermined movement threshold, the object detection unit 31 determines that the target object is moving, and if the amount of movement of the position is equal to or less than the movement threshold, the object detection unit 31 determines that the target object is stationary. Note that, when the exterior sensor signal is a ranging signal, the object detection unit 31 may estimate the position of the target object based on the direction in which the target object is represented in the ranging signal and the distance in that direction, and the position and traveling direction of the vehicle 10 at the time of generation of the ranging signal. Furthermore, when the vehicle exterior sensor signal is an image, the object detection unit 31 may assume that the position of the bottom end of the area in which the target object is displayed is the position where the target object is in contact with the road surface, and estimate the position of the target object with respect to the camera based on the direction from the camera, which is the vehicle exterior sensor 2, corresponding to the position of the bottom end of the area on the image, and the installation height of the camera. The object detection unit 31 may estimate the position of the target object based on the position and traveling direction of the vehicle 10 at the time of image generation, and the position of the target object with respect to the camera. Note that, as the position of the vehicle 10 at the time of generation of the vehicle exterior sensor signal, the object detection unit 31 may use the position of the vehicle 10 at the time of generation of the vehicle exterior sensor signal, which is measured by a satellite positioning device (not shown) such as a GPS receiver mounted on the vehicle 10. Furthermore, the object detection unit 31 may use the traveling direction of the vehicle 10 at the time of generation of the vehicle exterior sensor signal, which is measured by a direction sensor (not shown) mounted on the vehicle 10, as the traveling direction of the vehicle 10 at the time of generation of the vehicle exterior sensor signal.

[0029] The object detection unit 31 notifies the determination unit 33 of the number of detected target objects and, for each detected target object, the type of the target object, the object direction, and whether or not it is moving.

[0030] The direction detection unit 32 detects at least the line of sight direction of the driver, out of the direction of the driver's face and the line of sight direction, based on the driver image.

[0031] The orientation detection unit 32, for example, detects an area in which the driver's face is captured on the driver image (hereinafter referred to as a face area) by inputting the driver image into a classifier that has been trained in advance to detect the driver's face from the image, and detects a plurality of feature points of the driver's face, such as the outer corners and inner corners of the eyes, the upper eyelids, the lower eyelids, the tip of the nose, and the corners of the mouth. In this case, the orientation detection unit 32 detects the face area and the feature points of the face by inputting the driver image into a classifier that has been trained in advance to detect the face area and the feature points of the face shown in the image. As such a classifier, the orientation detection unit 32 can use, for example, a DNN with a CNN-type architecture, a support vector machine, or an AdaBoost classifier. The orientation detection unit 32 may detect the face area and the feature points of the face from the driver image according to another method for detecting the face area and the feature points of the face, such as template matching.

[0032] The orientation detection unit 32 fits each of the detected facial feature points to a three-dimensional face model that represents the three-dimensional shape of the face. The orientation detection unit 32 then detects the facial orientation of the three-dimensional face model when each feature point is best fitted to the three-dimensional face model as the facial orientation of the driver. Note that the orientation detection unit 32 may detect the facial orientation of the driver based on the driver image according to another method for determining the facial orientation represented in the image.

[0033] In addition, in order to detect the driver's line of sight, the direction detection unit 32 detects a corneal reflection image of a light source (hereinafter referred to as a Purkinje image) and a center of a pupil (hereinafter simply referred to as a pupil center of gravity) from an area surrounded by an upper eyelid and a lower eyelid (hereinafter referred to as an eye area) for at least one of the driver's left and right eyes displayed on the driver image. In this case, the direction detection unit 32 detects the Purkinje image by template matching between a template of the Purkinje image and the eye area. Similarly, the direction detection unit 32 may detect the pupil by template matching between a template of the pupil and the eye area, and may set the center of gravity of the area in which the detected pupil is displayed as the pupil center of gravity. Then, the direction detection unit 32 detects the driver's line of sight by determining the positional relationship between the Purkinje image and the pupil center of gravity, and by referring to a table that indicates the relationship between the positional relationship and the driver's line of sight. Note that such a table may be stored in the memory 22 in advance.

[0034] The direction detection unit 32 notifies the judgment unit 33 and the inattentiveness judgment unit 35 of the detected direction of the driver's line of sight. When the direction detection unit 32 detects the direction of the driver's face, it notifies the inattentiveness judgment unit 35 of the detected direction of the driver's face.

[0035] The determination unit 33 sets the peripheral vision range based on the latest line of sight direction of the driver detected by the direction detection unit 32. The peripheral vision range is set as an angle range equivalent to the peripheral visual field of the driver, which is set on the outer periphery of the centrally visible range centered on the line of sight direction of the driver.

[0036] When the target object is an object that may move, such as a pedestrian, the determination unit 33 may adjust the peripheral vision range depending on whether the target object is moving or not. For example, the determination unit 33 may make the peripheral vision range narrower when the target object is moving than when the target object is stationary.

[0037] Furthermore, when multiple target objects are detected, the determination unit 33 may determine, for each target object, whether or not the object direction of the target object is included in the peripheral vision range. Furthermore, the determination unit 33 may set the peripheral vision range when multiple target objects are detected to be narrower than the peripheral vision range when only one target object is detected.

[0038] Furthermore, the determination unit 33 may narrow the peripheral vision range as the speed of the vehicle 10 measured by a vehicle speed sensor (not shown) mounted on the vehicle 10 increases.

[0039] The determination unit 33 compares, for each detected object, the peripheral vision range set for that object with the object direction for that object to determine whether or not the object direction for that object is included in the set peripheral vision range. The determination unit 33 then notifies the setting unit 34 of the determination result as to whether or not the object direction for the object is included in the peripheral vision range.

[0040] The setting unit 34 sets an inattentive warning condition for determining whether or not to notify an inattentive warning. In particular, when the object direction to the detected target object is included in the peripheral vision range, the setting unit 34 sets the inattentive warning condition based on the driving mode applied to the vehicle 10.

[0041] The inattentive warning is issued when it is assumed that the driver is not monitoring the situation around the vehicle 10 in the traveling direction of the vehicle 10. Therefore, the inattentive warning condition is set as a period during which the driver's face direction or line of sight direction is out of a predetermined angle range centered on the front direction of the vehicle 10 continues for a predetermined time threshold or more. Therefore, the narrower the predetermined angle range or the shorter the time threshold, the more relaxed the inattentive warning condition is, and the more likely the inattentive warning is issued.

[0042] For example, when the driving mode applied to the vehicle 10 is a first driving mode in which the driver is not required to hold the steering wheel, the setting unit 34 sets the first inattentive warning condition when the object direction of the target object is included in the peripheral vision range to be stricter than the second inattentive warning condition when the object direction is out of the peripheral vision range or when the target object is not detected. That is, the predetermined angle range in the first inattentive warning condition is set wider than the predetermined angle range in the second inattentive warning condition. Alternatively, the time threshold in the first inattentive warning condition is set to a value larger than the time threshold in the second inattentive warning condition. The first driving mode may be, for example, a driving mode conforming to the level 2 autonomous driving level defined by the Society of Automotive Engineers (SAE).

[0043] When the direction of the target object is included in the peripheral vision range, the driver may be viewing the target object in the peripheral vision. Therefore, by setting the first inattentive warning condition as described above, the inattentive warning is unlikely to be notified when the driver is viewing the target object, and the driver is prevented from feeling annoyed.

[0044] In addition, when the driving mode applied to the vehicle 10 is the second driving mode in which the driver is required to hold the steering wheel, the setting unit 34 sets the third inattentive warning condition when the object direction of the target object is included in the peripheral vision range to be more lenient than the second inattentive warning condition. That is, the predetermined angle range in the third inattentive warning condition is set to be narrower than the predetermined angle range in the second inattentive warning condition. Alternatively, the time threshold in the third inattentive warning condition is set to a value smaller than the time threshold in the second inattentive warning condition. The second driving mode may be a driving mode conforming to the level 0 or level 1 autonomous driving level defined by the SAE, for example, a driving mode to which adaptive cruise control or lane tracing assist is applied.

[0045] In this case, since the driver is likely to receive a distraction warning if he or she only views the target object in the peripheral vision, it is possible to guide the driver to focus on the target object.

[0046] 4(a) and 4(b) are diagrams showing examples of inattentive warning conditions. In the example shown in FIG. 4(a), a first driving mode is applied in which the driver is not required to hold the steering wheel. An object direction 401 toward a pedestrian 400, which is a target object, is included in a peripheral vision range 411 set based on the line of sight of a driver 410. Therefore, an angle range 421 that includes the traveling direction of the vehicle 10 and is not determined to be inattentive is set wider than an angle range 422 when a target object such as a pedestrian 400 is not detected. This makes it difficult to issue an inattentive warning.

[0047] 4(b), an object direction 401 to a pedestrian 400, which is a target object, is included in the peripheral vision range 411 of a driver 410. However, in this example, a second driving mode is applied in which the driver is required to hold the steering wheel. Therefore, an angle range 423 in which inattentive driving is not determined is set narrower than an angle range 422 in which no target object is detected. This makes it easier to issue an inattentive driving warning.

[0048] The setting unit 34 may also set the inattentive warning condition based on the situation around the vehicle 10. For example, the setting unit 34 may adjust the inattentive warning condition according to whether the detected target object is moving or not, or the number of detected target objects. In this case, the setting unit 34 may set the first inattentive warning condition when the target object is stationary to be stricter than the first inattentive warning condition when the target object is moving. This makes it easier for the inattentive warning to be notified when the target object is moving, even if the driver visually recognizes the target object in the peripheral vision, so that the possibility of notifying the inattentive warning is more appropriately determined. However, it is preferable that the first inattentive warning condition when the target object is moving is stricter than the second inattentive warning condition. Similarly, the setting unit 34 may set the third inattentive warning condition when the target object is moving to be more relaxed than the third inattentive warning condition when the target object is stationary. However, it is preferable that the third inattentive warning condition when the target object is stationary is more relaxed than the second inattentive warning condition.

[0049] Furthermore, the setting unit 34 may set the first inattentive warning condition when a plurality of target objects are detected more lenient than the first inattentive warning condition when a single target object is detected. However, the first inattentive warning condition when a plurality of target objects are detected is preferably stricter than the second inattentive warning condition. Similarly, the setting unit 34 may set the third inattentive warning condition when a plurality of target objects are detected more lenient than the third inattentive warning condition when a single target object is detected. However, the third inattentive warning condition when a single target object is detected is preferably lenient than the second inattentive warning condition. Note that, when a plurality of target objects are detected, if the object direction of at least one target object is included in the peripheral vision range, the setting unit 34 may apply the first inattentive warning condition or the third inattentive warning condition.

[0050] Furthermore, the setting unit 34 may set the second inattentive warning condition when the driving mode applied to the vehicle 10 is the second driving mode to be stricter than the second inattentive warning condition when the driving mode applied to the vehicle 10 is the first driving mode. For example, the setting unit 34 sets the time threshold value in the second inattentive warning condition when the driving mode applied to the vehicle 10 is the second driving mode to a value greater than the time threshold value in the second inattentive warning condition when the driving mode applied to the vehicle 10 is the first driving mode. This further reduces the annoyance to the driver when the second driving mode is applied.

[0051] The setting unit 34 notifies the inattentive driving determination unit 35 of the set inattentive driving warning condition.

[0052] The inattentiveness determination unit 35 determines whether the driver's facial orientation or line of sight satisfies the inattentiveness warning condition. As described above, the inattentiveness determination unit 35 determines that the inattentiveness warning condition is satisfied if the detected one of the driver's facial orientation or line of sight direction is out of a predetermined angle range centered on the front direction of the vehicle 10 for a period of time equal to or greater than a predetermined time threshold.

[0053] The inattentive driving determination unit 35 notifies the notification processing unit 36 ​​of the determination result as to whether or not the inattentive driving warning condition is satisfied.

[0054] When the notification processing unit 36 ​​is notified by the inattentiveness determination unit 35 that the inattentiveness warning condition has been satisfied, the notification processing unit 36 ​​notifies the driver of the inattentiveness warning via the notification device 4. For example, the notification processing unit 36 ​​causes a speaker of the notification device 4 to emit a voice signal or a warning sound to warn the driver of inattentiveness. Alternatively, the notification processing unit 36 ​​causes a display device of the notification device 4 to display a warning message or an icon to warn the driver of inattentiveness. Alternatively, the notification processing unit 36 ​​vibrates a vibrator of the notification device 4, or turns on or blinks a light source.

[0055] After notifying the driver of the inattentive driving warning via the notification device 4, the notification processing unit 36 ​​stops notifying the inattentive driving warning when it receives a determination result from the inattentive driving determination unit 35 that the inattentive driving warning condition is no longer satisfied.

[0056] 5 is an operational flowchart of the driver monitor process executed by the processor 23. The object detection unit 31 detects a target object based on an external sensor signal generated by the external sensor 2 (step S101). The direction detection unit 32 detects at least the line of sight direction of the driver based on the driver image generated by the driver monitor camera 3 (step S102).

[0057] The determination unit 33 sets a peripheral vision range based on the detected line of sight of the driver (step S103). Then, the determination unit 33 determines whether the object direction to the target object is included in the peripheral vision range (step S104).

[0058] If the object direction to the target object is included in the peripheral vision range (step S104-Yes), the setting unit 34 sets the first or third inattentive warning condition (step S105) depending on the driving mode applied to the vehicle 10 or the situation around the vehicle 10. On the other hand, if the object direction to the target object is not included in the peripheral vision range (step S104-No), the setting unit 34 sets the second inattentive warning condition (step S106).

[0059] The inattentive driving determination unit 35 determines whether the driver's face direction or line of sight direction satisfies the set inattentive driving warning condition (step S107). If the inattentive driving warning condition is satisfied (step S107-Yes), the notification processing unit 36 ​​notifies the driver of an inattentive driving warning via the notification device 4 (step S108).

[0060] After step S108, or if the inattentive warning condition is not satisfied in step S107 (step S107-No), the processor 23 ends the driver monitor process.

[0061] As described above, this driver monitoring device adjusts the inattentive warning conditions when a detected object is included in the driver's peripheral vision according to the driving mode, etc., so that it can properly issue an inattentive warning to the driver so that the driver does not feel bothered.

[0062] In addition, if the inattentive warning condition continues to be satisfied even after a first grace period has elapsed since the inattentive warning notification was started, the notification processing unit 36 ​​may increase the intensity of the warning. For example, the notification processing unit 36 ​​may increase the volume of the inattentive warning emitted from the speaker. Alternatively, the notification processing unit 36 ​​may increase the number of devices that issue the inattentive warning. Furthermore, if the inattentive warning condition continues to be satisfied even after a second grace period that is longer than the first grace period has elapsed since the inattentive warning notification was started, the processor 23 may control each unit of the vehicle 10 to decelerate or stop the vehicle 10.

[0063] In addition, it is considered that the longer the same object remains in the driver's peripheral vision, the more likely the driver is to perceive the object in the peripheral vision. For this reason, the setting unit 34 may adjust the inattentive warning condition so as to make the inattentive warning condition stricter as the duration during which the object direction to the same object is included in the peripheral vision range becomes longer, or when the duration exceeds a predetermined switching threshold.

[0064] According to another modification, the setting unit 34 may set the inattentive warning condition to be stricter as the lateral distance from the vehicle 10 to the edge of the road area on which the vehicle 10 can travel increases. In this case, the setting unit 34 may obtain the lateral distance by referring to the current position of the vehicle 10 measured by a satellite positioning device (not shown) mounted on the vehicle 10 and a so-called high-precision map including information on the road width of each road. Alternatively, the setting unit 34 may estimate the current position of the vehicle 10 by comparing an image of the surroundings of the vehicle 10 generated by a camera, which is an example of the exterior sensor 2, with the high-precision map, and obtain the lateral distance based on the estimated position.

[0065] A computer program for realizing the driver monitoring process according to the above embodiment or the modified example may be provided in a form recorded on a computer-readable portable recording medium.

[0066] As described above, those skilled in the art can make various modifications to the embodiments within the scope of the present invention. [Explanation of symbols]

[0067] 1 Vehicle Control System 2. External Sensor 3 Driver monitor camera 4 Notification device 5 Electronic control unit (driver monitor device) 10 Vehicles 21 Communication Interface 22 Memory 23 Processors 31 Object detection unit 32 Orientation detection unit 33 Judgment section 34 Setting section 35 Inattentiveness Judgment Department 36 Notification Processing Section

Claims

1. an object detection unit that detects an object that may affect the running of the vehicle based on an external sensor signal that indicates a situation around the vehicle; A direction detection unit that detects at least a line of sight direction of the driver and a face direction of the driver based on a driver image representing the driver of the vehicle; a determination unit that determines whether or not an object direction to an object detected based on the line of sight is included in a peripheral vision range corresponding to a peripheral vision of the driver; a setting unit that sets an inattentive warning condition when the object direction is included in the peripheral vision range based on a driving mode applied to the vehicle or a situation around the vehicle; a notification processing unit that notifies the driver of an inattentive warning via a notification device when the detected at least one of the face direction and the line of sight direction of the driver satisfies the inattentive warning condition; A driver monitor device having a

2. 2. The driver monitor device according to claim 1, wherein the setting unit sets the inattentive warning condition when the driving mode is a first driving mode that does not require the driver to hold the steering wheel stricter than the inattentive warning condition when the driving mode is a second driving mode that requires the driver to hold the steering wheel.

3. 2. The driver monitor device according to claim 1, wherein, when the driving mode is a first driving mode that does not require the driver to hold the steering wheel, the setting unit sets the inattentive warning condition when the object direction is included in the peripheral vision range to be stricter than the inattentive warning condition when the object direction is outside the peripheral vision range.

4. The object detection unit further determines whether the detected object is moving; 2. The driver monitor device according to claim 1, wherein, when the driving mode is a first driving mode that does not require the driver to hold the steering wheel and the object direction is included in the peripheral vision range, the setting unit sets the inattentive warning condition when the detected object is stationary to be stricter than the inattentive warning condition when the detected object is moving.

5. Detecting an object that may affect the running of the vehicle based on an external sensor signal that indicates the situation around the vehicle; Detecting at least a line of sight direction of the driver and a face direction of the driver based on a driver image showing the driver of the vehicle; determining whether or not an object direction to an object detected based on the line of sight is included in a peripheral vision range corresponding to a peripheral vision of the driver; setting an inattentive warning condition for when the object direction is included in the peripheral vision range based on a driving mode applied to the vehicle or a situation around the vehicle; notifying the driver of an inattentive warning via a notification device when the detected at least one of the face direction and the line of sight direction of the driver satisfies the inattentive warning condition; A driver monitoring method comprising:

6. Detecting an object that may affect the running of the vehicle based on an external sensor signal that indicates the situation around the vehicle; Detecting at least a line of sight direction of the driver and a face direction of the driver based on a driver image showing the driver of the vehicle; determining whether or not an object direction to an object detected based on the line of sight is included in a peripheral vision range corresponding to a peripheral vision of the driver; setting an inattentive warning condition for when the object direction is included in the peripheral vision range based on a driving mode applied to the vehicle or a situation around the vehicle; notifying the driver of an inattentive warning via a notification device when the detected at least one of the face direction and the line of sight direction of the driver satisfies the inattentive warning condition; A computer program for a driver monitor, which causes a processor mounted on the vehicle to execute the above steps.

Images