Driving assistance device and driving assistance computer program

Abstract

To provide a driving support device capable of setting an appropriate driving support level for the state of a driver before the state of the driver appears in the behavior of a vehicle.SOLUTION: A driving support device includes a detection unit 31 that detects a line-of-sight direction of a driver of a vehicle 1, a confirmation state specification unit 32 that obtains a confirmation state of a traffic signal around the vehicle 1 by the driver when the vehicle 1 is stopped based on movement of the line-of-sight direction, and a level change unit 33 that changes an application level of driving support for the driver based on the confirmation state.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a driving assistance device and a driving assistance computer program that assists a driver of a vehicle in driving. [Background technology]

[0002] A technology for detecting a driver's state has been proposed (see Patent Document 1). The detection device disclosed in Patent Document 1 detects driving state information indicating the driving state of at least one of a vehicle and its driver. The detection device then analyzes the detected driving state information to calculate hasty driving information indicating the degree of the driver's tendency to hasty driving, and detects that the driver is in a state of hasty driving based on the calculated hasty driving information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-120874 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above technology, the tendency to rush is determined based on the driver's actions or vehicle behavior while the vehicle is running. However, it is undesirable from a safety standpoint for a driver to drive a vehicle in an irritated state. Therefore, it is desirable to be able to determine the driver's state before the driver's irritation is reflected in the vehicle's behavior.

[0005] Therefore, an object of the present invention is to provide a driving assistance device that can set an appropriate driving assistance level for the driver's condition before the driver's condition is reflected in the vehicle behavior. [Means for solving the problem]

[0006] According to one embodiment, there is provided a driving assistance device including a detection unit that detects a gaze direction of a driver of a vehicle, a confirmation status determination unit that determines a confirmation status of traffic signals around the vehicle by the driver while the vehicle is stopped based on movement of the gaze direction, and a level change unit that changes an application level of driving assistance to the driver based on the confirmation status.

[0007] In one embodiment, the confirmation status identification unit determines the confirmation status as the frequency or the percentage of time during a specified period that the driver's line of sight is directed toward a traffic light indicating whether or not the vehicle can proceed in a direction that crosses the road in the vehicle's direction of travel, and the level change unit changes the application level of driving assistance to a higher level if the frequency exceeds a specified frequency threshold or the time percentage exceeds a specified percentage threshold.

[0008] In one embodiment, the confirmation status identification unit determines the duration during which the driver's line of sight is continuously directed toward a traffic light indicating whether the vehicle can proceed on the road in the vehicle's direction of travel as the confirmation status, and the level change unit changes the application level of driving assistance to a higher level if the duration exceeds a predetermined duration threshold.

[0009] In one embodiment, the driving assistance device further includes a driver state determination unit that determines the state of the driver based on the confirmation status.

[0010] According to another embodiment, there is provided a driving assistance computer program having instructions to cause a processor mounted on a vehicle to execute the following: detect a line of sight of a driver of a vehicle, determine a state of checking traffic signals around the vehicle by the driver while the vehicle is stopped based on movement of the line of sight, and change a level of application of driving assistance to the driver based on the checking state. [Effects of the Invention]

[0011] The driving assistance device according to the present disclosure has the effect of being able to set a driving assistance level appropriate for the driver's condition before the driver's condition is reflected in the vehicle's behavior. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a schematic configuration diagram of a vehicle in which a driving assistance device is implemented; [Figure 2] FIG. 2 is a functional block diagram of a processor of an ECU that is an example of a driving assistance device. [Figure 3] 1A and 1B are diagrams each showing an example of a situation in which a driver checks traffic signals around a vehicle. [Figure 4] 10 is an operation flowchart of a driving assistance process. [Figure 5] FIG. 10 is a functional block diagram of a processor of an ECU according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0013] The driving assistance device, the driving assistance method, and the driving assistance computer program executed by the driving assistance device will be described below with reference to the drawings. The driving assistance device determines the driver's confirmation status of traffic lights around the vehicle when the vehicle is stopped based on the driver's line of sight movement, and changes the application level of driving assistance to the driver based on the confirmation status.

[0014] FIG. 1 is a schematic configuration diagram of a vehicle in which a driving assistance device is implemented. In this embodiment, the vehicle 1 has a camera 2, a driver monitor camera 3, a notification device 4, and an electronic control unit (ECU) 5, which is an example of the driving assistance device. The camera 2, the driver monitor camera 3, the notification device 4, and the ECU 5 are communicably connected to each other. The vehicle 1 may also be provided with a ranging sensor (not shown), such as a LiDAR or radar, that measures the distance to objects around the vehicle 1. The vehicle 1 may also be provided with a positioning device (not shown), such as a GPS receiver, that measures the position of the vehicle 1 using a satellite positioning system.

[0015] Camera 2 is an example of an imaging unit, and is attached to vehicle 1 so as to face a predetermined area around vehicle 1, such as the area ahead of vehicle 1. Note that vehicle 1 may be provided with a plurality of cameras with different imaging directions or focal lengths. Camera 2 captures an image of the predetermined area at each predetermined imaging cycle to generate an image of the predetermined area (hereinafter referred to as an outside-vehicle image), and outputs the generated outside-vehicle image to ECU 5.

[0016] The driver monitor camera 3 is an example of an in-vehicle imaging unit, and is attached to the instrument panel or its vicinity, facing the driver, so that the head of the driver seated in the driver's seat of the vehicle 1 is included in the imaging target area. The driver monitor camera 3 may have a light source such as an infrared LED. The driver monitor camera 3 captures an image of the driver at each predetermined imaging period to generate an image of the driver (hereinafter referred to as a driver image), and outputs the generated driver image to the ECU 5.

[0017] The notification device 4 is provided in the passenger compartment of the vehicle 1 and is a device that provides predetermined notifications to the driver by light, sound, vibration, text display, or image display. To achieve this, the notification device 4 has at least one of a speaker, a light source, a vibrator, or a display device, for example. When the notification device 4 receives a notification signal representing a predetermined notification to the driver from the ECU 5, it notifies the driver by sound from the speaker, light emission or blinking of the light source, vibration of the vibrator, or display of a notification message or icon on the display device.

[0018] The ECU 5 executes driving assistance processing. To this end, the ECU 5 includes a communication interface 11, a memory 12, and a processor 13. The communication interface 11, the memory 12, and the processor 13 may be configured as separate circuits, or may be integrated into a single integrated circuit.

[0019] The communication interface 11 has an interface circuit for connecting the ECU 5 to other devices in the vehicle. The communication interface 11 passes the outside-of-vehicle image received from the camera 2 and the driver image received from the driver monitor camera 3 to the processor 13. The communication interface 11 also outputs the notification signal received from the processor 13 to the notification device 4.

[0020] The memory 12 is an example of a storage unit and includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 12 stores various data used in the driving assistance process executed by the processor 13 or generated during the driving assistance process.

[0021] The processor 13 includes one or more central processing units (CPUs) and their peripheral circuits. The processor 13 may further include other arithmetic circuits such as a logic unit, a numerical calculation unit, or a graphics processing unit. The processor 13 executes driving assistance processing.

[0022] 2 is a functional block diagram of the processor 13 related to the driving assistance process. The processor 13 has a detection unit 31, a confirmation status identification unit 32, a level change unit 33, and a driving control unit 34. Each of these units in the processor 13 is a functional module realized by a computer program running on the processor 13. Alternatively, each of these units in the processor 13 may be a dedicated arithmetic circuit provided in the processor 13.

[0023] The detection unit 31 detects the driver's gaze direction at predetermined intervals. To this end, the detection unit 31 inputs the latest driver image into a classifier that has been trained in advance to detect eyes, and detects an area in the driver image where the eyes are represented (hereinafter referred to as the eye area). Such a classifier is configured, for example, as a deep neural network (DNN) with a convolutional neural network (CNN) type architecture, a support vector machine, or an AdaBoost classifier. Note that the detection unit 31 may also detect the eye area from the driver image using other methods for detecting the eye area, such as template matching.

[0024] The detection unit 31 detects a corneal reflection image of a light source (hereinafter referred to as a Purkinje image) and a centroid of a pupil (hereinafter simply referred to as a pupil centroid) from the eye region for at least one of the driver's left and right eyes depicted on the driver image. The detection unit 31 detects the Purkinje image by template matching between a Purkinje image template and the eye region. Similarly, the detection unit 31 may detect the pupil by template matching between a pupil template and the eye region, and the centroid of the region in which the detected pupil is depicted may be set as the pupil centroid. The detection unit 31 then calculates the direction and distance from the Purkinje image to the pupil centroid and detects the driver's gaze direction by referring to a table that represents the relationship between the direction and distance and the driver's gaze direction. Note that such a table may be stored in the memory 12 in advance. Alternatively, the detection unit 31 may detect the driver's gaze direction from the driver image using another method for detecting the gaze direction of a person depicted in an image. The detection unit 31 notifies the confirmation situation identification unit 32 of the detected driver's gaze direction.

[0025] The confirmation status identification unit 32 determines the confirmation status of the driver of the traffic lights around the vehicle 1 when the vehicle 1 is stopped based on the movement of the driver's line of sight. The confirmation status identification unit 32 determines that the vehicle 1 is stopped when the vehicle speed of the vehicle 1 measured by a vehicle speed sensor (not shown) provided in the vehicle 1 is equal to or less than a stop determination threshold (for example, 0.1 to 1 km / h).

[0026] While the vehicle 1 is stopped, the confirmation status identification unit 32 determines, for example, the frequency (hereinafter referred to as gaze frequency) at which the driver's gaze direction turns toward a traffic light (hereinafter referred to as a crossing traffic light) that indicates whether or not the driver can proceed in a direction crossing the road in the vehicle's traveling direction, as an index representing the traffic light confirmation status. Alternatively, while the vehicle 1 is stopped, the confirmation status identification unit 32 may determine, as an index representing the traffic light confirmation status, the proportion of time the driver spends looking toward the crossing traffic light in a predetermined period (hereinafter referred to as a gaze time proportion). Here, to determine whether the driver is looking at the traffic light, the confirmation status identification unit 32 detects the crossing traffic light from the vehicle exterior image. Then, the confirmation status identification unit 32 compares the detected direction of the crossing traffic light with the driver's gaze direction detected from the driver image.

[0027] The confirmation situation identification unit 32 inputs the exterior image into a classifier that has been trained in advance to detect traffic lights from the exterior image, thereby detecting an area in the exterior image where a traffic light appears (hereinafter referred to as a signal area) and identifying the lighting state (green, yellow, red, etc.) of the traffic light represented in the detected signal area. Such a classifier is configured, for example, as a DNN with a recursive structure such as a CNN or a recurrent neural network (RNN) or a DNN with an attention mechanism. The confirmation situation identification unit 32 identifies a traffic light represented in a detected signal area that is located to the left or right of a partial area on the exterior image corresponding to the front direction of the vehicle 1 (hereinafter referred to as a front direction area) as a crossing traffic light. Note that the crossing traffic light may be a traffic light for pedestrians or vehicles. Furthermore, the crossing traffic light may be a traffic light with an auxiliary information indicator that displays the remaining time until the light changes to red. The position and size of the front direction area are pre-stored in the memory 12. The confirmation status specifying unit 32 then determines the direction corresponding to a reference point (for example, the center of gravity or center of the signal area) within the signal area in which the crossing traffic light is displayed as the direction from the driver to the crossing traffic light.

[0028] The confirmation status identification unit 32 determines that the driver's line of sight is directed toward the crossing traffic light when the angular difference between the identified direction from the driver to the crossing traffic light and the driver's line of sight detected by the detection unit 31 is within a predetermined tolerance range (for example, several degrees to several tens of degrees). On the other hand, when the angular difference between the direction from the driver to the crossing traffic light and the driver's line of sight is outside the predetermined tolerance range, the confirmation status identification unit 32 determines that the driver's line of sight is not directed toward the crossing traffic light.

[0029] The confirmation status identification unit 32 repeats the above process at predetermined intervals after the vehicle 1 stops. The confirmation status identification unit 32 counts the number of times the driver's gaze direction is determined to be directed toward the crossing traffic light as the gaze frequency. Alternatively, once the confirmation status identification unit 32 determines that the driver's gaze direction is directed toward the crossing traffic light, it calculates the duration until the next time it is determined that the driver's gaze direction is not directed toward the crossing traffic light. The confirmation status identification unit 32 then calculates the gaze time ratio as the ratio of the total duration of the predetermined period to the length of the predetermined period. The predetermined period may be a preset period, such as several tens of seconds, or may be the period from when the vehicle 1 stops until the crossing traffic light turns red. The confirmation status identification unit 32 may determine whether the crossing traffic light has turned red by referring to the identification result of the traffic light illumination state by the classifier for the signal area where the crossing traffic light is displayed.

[0030] Furthermore, while the vehicle 1 is stopped, the confirmation status identification unit 32 may determine, as an index representing the confirmation status of the traffic light, the duration (hereinafter referred to as gaze duration) that the driver's line of sight continues to be directed toward a traffic light indicating whether or not the vehicle 1 can proceed along the road in the direction of travel of the vehicle 1. In this case, the confirmation status identification unit 32 determines, among the detected signal areas, a traffic light displayed in a signal area included in the front direction area as the traffic light in the direction of travel. Note that, like crossing signals, the traffic light in the direction of travel may be a traffic light for pedestrians or for vehicles. Furthermore, the traffic light in the direction of travel may be a traffic light having an auxiliary information display that displays the remaining time until the light changes to green. Furthermore, if there are multiple signal areas included in the front direction area, the confirmation status identification unit 32 may determine, as the traffic light in the direction of travel, the largest signal area in the front direction area or the traffic light displayed in the signal area located at the top or the leftmost on the vehicle exterior image. The confirmation status identification unit 32 then determines that the direction corresponding to the reference point within the signal area in which the traffic light is displayed is the direction from the driver to the traffic light. If the angular difference between the direction from the driver to the traffic light and the driver's line of sight is within a predetermined tolerance range, the confirmation status identification unit 32 determines that the driver's line of sight is directed toward the traffic light. On the other hand, if the angular difference between the direction from the driver to the traffic light and the driver's line of sight is outside the predetermined tolerance range, the confirmation status identification unit 32 determines that the driver's line of sight is not directed toward the traffic light.

[0031] The confirmation status identification unit 32 repeats the above process at predetermined intervals after the vehicle 1 stops. The confirmation status identification unit 32 then determines the duration from when it is determined that the driver's line of sight is directed toward the traffic light in the direction of travel until it is determined that the line of sight is not directed toward the traffic light in the direction of travel next.

[0032] In addition, the confirmation status identification unit 32 may identify the direction from the driver to the cross traffic light and the direction from the driver to the forward traffic light based on map information showing the position and type of each traffic light, the stopped position of the vehicle 1 determined by a positioning device, and the forward direction of the vehicle 1 detected by a direction sensor (not shown) mounted on the vehicle 1.

[0033] According to a modified example, the confirmation status identification unit 32 may determine whether the driver is looking at a cross traffic light or a forward traffic light based only on the driver image acquired during a predetermined period. In this case, the gaze direction movement of the driver when gazing at a specific traffic light is learned in advance and stored in the memory 12 as a reference gaze behavior. For each sampling period shorter than the predetermined period, the confirmation status identification unit 32 compares the gaze direction movement detected during that sampling period with the reference gaze behavior. If the difference is within a predetermined tolerance, the confirmation status identification unit 32 may determine that the driver's gaze direction is directed toward the specific traffic light during that sampling period. Alternatively, each time the driver's gaze direction is detected, the confirmation status identification unit 32 may input the gaze direction into a gaze direction determination model to determine whether the driver is looking at a cross traffic light or a forward traffic light. The gaze direction determination model is configured as a DNN with a recursive structure such as an RNN.

[0034] The checking status specifying unit 32 passes an index value indicating the driver's checking status of the traffic light to the level changing unit 33 .

[0035] The level change unit 33 changes the application level of driving assistance to the driver depending on the driver's confirmation of the traffic light.

[0036] When the driver is irritated, the driver may gaze at a crossing traffic light (e.g., a pedestrian traffic light in the direction crossing the road on which the vehicle 1 is traveling) so that the vehicle 1 can start as soon as the traffic light on the vehicle's path changes to green. It is assumed that the driver tries to confirm the timing of the traffic light changing from green to red as soon as possible. In such a case, the gaze frequency or gaze time ratio on the crossing traffic light increases. Therefore, when the gaze frequency or gaze time ratio on the crossing traffic light is calculated as an index value representing the driver's traffic light confirmation status, the level change unit 33 changes the level of driving assistance to be applied so that the level of driving assistance is increased when the gaze frequency exceeds a predetermined frequency threshold (e.g., several tens of times) or the gaze time ratio exceeds a predetermined ratio threshold (e.g., 0.4 to 0.7). Conversely, when the gaze frequency falls below a reduction frequency threshold, which is lower than the predetermined frequency threshold, the level change unit 33 may change the level of driving assistance to be applied so that the level of driving assistance is decreased. Alternatively, when the gaze time ratio falls below a mitigation ratio threshold that is lower than a predetermined ratio threshold, the level change unit 33 may change the level of driving assistance to be applied so that the level of driving assistance to be applied becomes lower.

[0037] Furthermore, if the driver is irritated, it is expected that the driver will try to check as soon as possible when a traffic light along the vehicle 1's route will change from red to green so that the vehicle 1 can start as soon as the traffic light changes to green. In such a case, the driver's gaze duration on the traffic light in the vehicle's direction of travel will be longer. Therefore, when the gaze duration on the traffic light in the vehicle's direction of travel is required as an index value representing the driver's traffic light confirmation status, the level change unit 33 changes the level of driving assistance to be applied so that the level of driving assistance to be applied is increased if the gaze duration exceeds a predetermined duration threshold (e.g., several tens of seconds). Conversely, if the gaze duration falls below a mitigation duration threshold that is lower than the duration threshold, the level change unit 33 may change the level of driving assistance to be applied so that the level of driving assistance to be applied is decreased.

[0038] In addition, when multiple index values ​​representing the driver's confirmation status of traffic lights are obtained, the level change unit 33 changes the level of driving assistance to be applied so that the level of driving assistance to be applied becomes higher when any one of the multiple index values ​​exceeds the corresponding threshold value.

[0039] If the level changer 33 determines that the level of driving assistance should be increased and no driving assistance mode is currently being applied, the level changer 33 sets the changed driving assistance mode to a driving assistance mode that maintains a predetermined distance or greater between the vehicle 1 and a preceding vehicle traveling ahead of the vehicle 1. Alternatively, the level changer 33 may set the changed driving assistance mode to a driving assistance mode that limits the maximum acceleration value of the vehicle 1 to an upper limit acceleration or less. Furthermore, if the level changer 33 determines that the level of driving assistance should be increased and any driving assistance mode is currently being applied, the level changer 33 may set the changed driving assistance mode to an autonomous driving mode that automatically controls the driving of the vehicle 1.

[0040] In this way, if the driver exhibits behavior that is assumed to indicate irritation, the level of driving assistance being applied can be changed to a higher level, making it possible to prevent the driver from engaging in reckless driving in advance.

[0041] In addition, if it is determined that the level of driving assistance should be lowered and some driving assistance mode is currently being applied, the level change unit 33 may stop applying that driving mode and allow the driver to manually control the driving of the vehicle 1.

[0042] When the level change unit 33 changes the level of driving assistance currently being applied, it notifies the driving control unit 34 of a driving assistance mode corresponding to the changed level. Furthermore, when the level change unit 33 changes the level of driving assistance currently being applied, it may output a notification signal notifying that the level of driving assistance has been changed and the changed driving assistance mode to the notification device 4 via the communication interface 11. In this way, the driver is notified that the level of driving assistance has been changed and the changed driving assistance mode.

[0043] 3(a) and 3(b) are diagrams each showing an example of a driver's confirmation of traffic lights around vehicle 1. In the example shown in Fig. 3(a), driver 300 of vehicle 1 is gazing at traffic light 301 on a road that crosses the traveling direction of vehicle 1. As a result, if the frequency of gaze at traffic light 301 exceeds a predetermined frequency threshold or the proportion of gaze time exceeds a predetermined proportion threshold, the level of driving assistance to be applied is changed so that the level is increased.

[0044] 3(b), the driver 300 of the vehicle 1 is gazing at a traffic light 302 in the traveling direction of the vehicle 1. As a result, if the duration of gaze on the traffic light 302 exceeds the duration threshold, the level of driving assistance to be applied is changed so that the level of the driving assistance is increased.

[0045] The driving control unit 34 controls the driving of the vehicle 1 according to the driving assistance mode level applied to the vehicle 1. For example, when the applied driving assistance mode is a driving assistance mode that maintains a predetermined distance or greater between the preceding vehicle and the vehicle 1, the driving control unit 34 detects the preceding vehicle from the vehicle exterior image. To do so, the driving control unit 34 detects other vehicles and lane markings by inputting the vehicle exterior image into a classifier that has been trained in advance to detect other vehicles and lane markings traveling around the vehicle 1 from the vehicle exterior image. Such a classifier is configured as a CNN or a DNN with an attention mechanism. The driving control unit 34 then identifies the other vehicle located in the area between the two lane markings closest to the vehicle 1 in the vehicle exterior image as the preceding vehicle. The driving control unit 34 assumes that the position of the bottom edge of the object area representing the preceding vehicle represents the position where the preceding vehicle is in contact with the road surface, and estimates the inter-vehicle distance from the vehicle 1 to the preceding vehicle based on the parameters of the camera 2, such as the shooting direction and installation position of the camera 2, and the position of the bottom edge of the object area. In addition, if vehicle 1 is equipped with a ranging sensor (not shown), the driving control unit 34 may use the distance measured by the ranging sensor in the direction corresponding to the object area in which the preceding vehicle is represented as the inter-vehicle distance from vehicle 1 to the preceding vehicle.

[0046] When the distance between the preceding vehicle and vehicle 1 is less than a predetermined distance, the cruise control unit 34 controls the powertrain or brakes of vehicle 1 so as to increase the distance between the preceding vehicle and vehicle 1. When the distance between the preceding vehicle and vehicle 1 is equal to or greater than the predetermined distance, the cruise control unit 34 controls the powertrain of vehicle 1 so as to bring the speed of vehicle 1 closer to the target speed.

[0047] In addition, when a driving assistance mode is applied in which the maximum acceleration value of the vehicle 1 is limited to be equal to or less than the upper limit acceleration, the driving control unit 34 controls the powertrain so that the acceleration corresponding to the accelerator operation by the driver is equal to or less than the upper limit acceleration.

[0048] Furthermore, when the autonomous driving mode is applied, the driving control unit 34 detects lane markings from the outside-of-vehicle image as described above. The driving control unit 34 then sets a planned driving trajectory along the center of the two lane markings closest to the vehicle 1, and controls the steering device of the vehicle 1 so that the vehicle 1 drives along the set planned driving trajectory. Furthermore, the driving control unit 34 controls the powertrain and brake device so that the inter-vehicle distance between the preceding vehicle and vehicle 1 is maintained at or above a predetermined distance, just as when a driving assistance mode is applied that maintains the inter-vehicle distance between the preceding vehicle and vehicle 1 at or above a predetermined distance.

[0049] FIG. 4 is an operational flowchart of the driving assistance process executed by the processor 13.

[0050] The detection unit 31 detects the driver's line of sight (step S101). The confirmation status identification unit 32 determines the driver's confirmation status of traffic lights around the vehicle 1 when the vehicle 1 is stopped based on the movement of the driver's line of sight (step S102).

[0051] The level change unit 33 changes the application level of driving assistance to the driver depending on the driver's confirmation of the traffic light (step S103). The driving control unit 34 controls the driving of the vehicle 1 in accordance with the driving assistance mode corresponding to the changed level (step S104).

[0052] As described above, the driving assistance device changes the level of driving assistance depending on the driver's confirmation of traffic lights while the vehicle is stopped. The confirmation of traffic lights is one type of driver behavior that represents the driver's state. Therefore, the driving assistance device can set a driving assistance level appropriate for the driver's state before the driver's state is reflected in the vehicle's behavior.

[0053] According to a variant, the processor 13 may be configured to determine the state of the driver depending on how the driver checks the traffic lights when the vehicle 1 is stopped.

[0054] 5 is a functional block diagram of the processor 13 according to this modification. The processor 13 has a detection unit 31, a confirmation status identification unit 32, and a driver state determination unit 35. Each of these units of the processor 13 is a functional module realized by a computer program running on the processor 13, for example. Alternatively, each of these units of the processor 13 may be a dedicated arithmetic circuit provided in the processor 13. Compared to the above embodiment, this modification differs in that it has a driver state determination unit 35 instead of the level change unit 33 and the driving control unit 34. This difference will be described below.

[0055] The driver state determination unit 35 determines the driver state according to the driver's confirmation of traffic lights while the vehicle 1 is stopped. As described above, if the driver is irritated, the driver may gaze at a crossing traffic light so that the vehicle 1 can start moving as soon as the traffic light along the vehicle 1's route turns green. Therefore, when the frequency of gaze at a crossing traffic light or the proportion of gaze time is calculated as an index value representing the driver's confirmation of traffic lights, the driver state determination unit 35 determines that the driver is irritated if the gaze frequency exceeds a predetermined frequency threshold or the proportion of gaze time exceeds a predetermined proportion threshold.

[0056] Furthermore, if the driver is irritated, it is expected that the driver will try to check as soon as possible when a traffic light along the path of vehicle 1 changes from red to green so that vehicle 1 can start as soon as the traffic light changes to green. Therefore, if the duration of gaze on the traffic light in the direction of travel is obtained as an index value representing the driver's traffic light confirmation status, the driver state determination unit 35 may determine that the driver is irritated if the duration of gaze exceeds a predetermined duration threshold.

[0057] When the driver state determination unit 35 determines that the driver is irritated, it issues a warning to the driver that he is in an irritated state via the notification device 4. For example, the driver state determination unit 35 causes a speaker of the notification device 4 to output a voice message such as "Please calm down." In addition to or instead of the warning notification, the driver state determination unit 35 may also execute a notification to calm the driver. For example, the driver state determination unit 35 causes a speaker of the notification device 4 to output music to relax the driver or a voice message to calm the driver.

[0058] According to this variant, the driving assistance device determines the driver's state based on the movement of the driver's line of sight while the vehicle is stopped, so that the driver's state can be determined before it is reflected in the vehicle's behavior.

[0059] Furthermore, in the above embodiment or modified example, the confirmation status identification unit 32 may determine the duration of gaze on a cross traffic light or the frequency or percentage of gaze time on a traffic light in the direction of travel while the vehicle 1 is stopped as an index representing the driver's confirmation status of traffic lights around the vehicle 1. In this case, too, the level change unit 33 may change the level of driving assistance so that the level of applied driving assistance increases when any of the gaze frequency, gaze time percentage, and gaze duration exceeds a corresponding threshold. Similarly, the driver state determination unit 35 may determine that the driver is in an irritated state when any of the gaze frequency, gaze time percentage, and gaze duration exceeds a corresponding threshold.

[0060] Furthermore, in the above embodiment or modified example, the confirmation status identification unit 32 may obtain an index value representing the driver's state other than the driver's gaze direction movement. For example, the confirmation status identification unit 32 may identify the driver's facial expression by inputting a driver image to a classifier pre-trained to identify the driver's facial expression. In this case, the classifier is configured as a CNN or a DNN with an attention mechanism. The classifier then outputs an index value representing the driver's facial expression. If the index value related to the gaze direction movement satisfies the above-described conditions for determining that the driver is irritated and the index value representing the driver's facial expression indicates an irritated expression, the level change unit 33 may change the level of driving assistance to a higher level. According to this modified example, not only the driver's gaze direction movement when the vehicle 1 is stopped but also the driver's facial expression are referenced, so the driving assistance device can apply a level of driving assistance more appropriate for the driver's state.

[0061] 2 may be combined with the modified example shown in Fig. 5. That is, the processor 13 may have a detection unit 31, a confirmation status identification unit 32, a level change unit 33, a driving control unit 34, and a driver state determination unit 35. In this case, when the driver state determination unit 35 determines that the driver is irritated based on the driver's confirmation of traffic lights while the vehicle 1 is stopped, the level change unit 33 changes the level of driving assistance to be applied so that the level is higher. Then, the driving control unit 34 controls the driving of the vehicle 1 in accordance with the driving assistance mode of the applied level.

[0062] In addition, a computer program that realizes the functions of the processor 13 of the ECU 5 according to the above embodiment or variant may be provided in a form recorded on a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium or an optical recording medium.

[0063] As described above, those skilled in the art can make various modifications to the embodiments within the scope of the present invention. [Explanation of symbols]

[0064] 1 vehicle, 2 camera, 3 driver monitor camera, 4 notification device, 5 electronic control unit (ECU, driving assistance device), 11 communication interface, 12 memory, 13 processor, 31 detection unit, 32 confirmation status identification unit, 33 level change unit, 34 driving control unit, 35 driver state determination unit

Claims

1. a detection unit that detects the line of sight of a driver of the vehicle; a confirmation status determination unit that determines a confirmation status of traffic lights around the vehicle by the driver when the vehicle is stopped based on the movement of the line of sight; a level change unit that changes an application level of driving assistance to the driver based on the confirmation status; A driving assistance device having the above configuration.

2. the confirmation status specification unit determines, as the confirmation status, a frequency or a time ratio during a predetermined period in which the driver's line of sight is directed toward a traffic light indicating whether or not the vehicle can proceed in a direction crossing a road in the vehicle's traveling direction; 2. The driving assistance device according to claim 1, wherein the level changer changes the application level of the driving assistance so as to increase the application level when the frequency exceeds a predetermined frequency threshold or the time proportion exceeds a predetermined proportion threshold.

3. the confirmation status identification unit determines, as the confirmation status, a duration during which the driver's line of sight is continuously directed toward a traffic light indicating whether or not the vehicle can proceed on a road in a traveling direction of the vehicle; The driving assistance device according to claim 1 , wherein the level changer changes the application level of the driving assistance so as to increase the application level when the duration exceeds a predetermined duration threshold.

4. The driving support device according to claim 1 , further comprising a driver state determination unit that determines a state of the driver based on the confirmation status.

5. Detects the direction of the driver's gaze, determining a confirmation state of traffic lights around the vehicle by the driver when the vehicle is stopped based on the movement in the line of sight direction; changing the application level of driving assistance to the driver based on the confirmation status; A driving assistance computer program for causing a processor mounted on the vehicle to execute the above.

Images