Driving support device, vehicle, method for controlling driving support device, program, and storage medium

Abstract

To suppress excessive notification to a driver.SOLUTION: A driving support device comprises: detection means for detecting a notification object as a target about which notification is to be made to a driver of a vehicle on the basis of peripheral information of the vehicle; estimation means for estimating a line-of-sight direction of the driver; first control means for performing control to turn on a visual observation flag when the driver has visually observed the notification object on the basis of the line-of-sight direction, the visual observation flag indicating whether or not the driver has visually observed the notification object; second control means for performing control to turn on a recognition flag when the visual observation flag has been turned on, the recognition flag indicating whether or not the driver recognizes the notification object; and suppression means for suppressing the notification while the recognition flag is turned on. The first control means performs control to turn off the visual observation flag in response to the driver averting a line of sight from the notification object while the recognition flag is turned on. The second control means performs control to turn off the recognition flag when a predetermined condition is satisfied after the visual observation flag has been turned off.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a driving assistance device, a vehicle, a control method for a driving assistance device, a program, and a storage medium. [Background technology]

[0002] Patent document 1 discloses a system that detects an object of attention that an occupant should pay attention to, and if the occupant does not direct their gaze toward the object of attention, displays a marker to guide their gaze, but if the occupant directs their gaze toward the object of attention, no marker is displayed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-134352 Summary of the Invention [Problem to be solved by the invention]

[0004] However, drivers do not continue to look at an attention target once they have seen it while driving. Even if they do not continue to look at it, they may be mentally paying attention to it by capturing the attention target in their peripheral vision or guessing its approximate movement. The technology described in Patent Document 1 issues a notification when the driver is not looking at the target, which has the problem of potentially excessive notifications.

[0005] The present invention has been made in view of the above-mentioned problems, and has an object to provide a technique for suppressing excessive notifications to the driver. [Means for solving the problem]

[0006] A driving assistance device according to one aspect of the present invention that achieves the above object includes: a detection means for detecting a notification target object to be notified to a driver of the vehicle based on information about the vicinity of the vehicle; an estimation means for estimating the driver's line of sight; a first control means for controlling a visual observation flag indicating whether or not the driver has visually observed the notification object to be turned on when the driver has visually observed the notification object based on the line of sight direction; a second control means for controlling a recognition flag, which indicates whether the driver recognizes the object to be notified, to be on when the visual sight flag is turned on; suppression means for suppressing the notification while the recognition flag is on; the first control means controls the visual inspection flag to be turned off in response to the driver averting their gaze from the object to be notified while the recognition flag is on, The second control means controls the recognition flag to be turned off when a predetermined condition is satisfied after the visual inspection flag is turned off. [Effects of the Invention]

[0007] According to the present invention, excessive notifications to the driver can be suppressed. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram of a vehicle and a control device according to an embodiment. [Figure 2] 4 is a flowchart showing a procedure of a process performed by a driving assistance device according to an embodiment. [Figure 3] FIG. 10 is an explanatory diagram of a bounding box related to visual determination according to a modified example. [Figure 4] FIG. 10 is an explanatory diagram of a Gaussian distribution related to visual determination according to a modified example. [Figure 5] FIG. 10 is an explanatory diagram of notification conditions when a vehicle turns right or left according to an embodiment. [Figure 6] FIG. 6 is an explanatory diagram of a notification condition when a vehicle is ahead of a vehicle according to an embodiment. [Figure 7] FIG. 6 is an explanatory diagram of a notification condition when a vehicle is ahead of a vehicle according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be arbitrarily combined. Furthermore, the same reference numerals are used for the same or similar components, and redundant explanations will be omitted.

[0010] <Control device and its application examples> FIG. 1 is a block diagram of a control device CNT according to one embodiment of the present invention, and a schematic diagram of a vehicle V to which the control device CNT is applied. In FIG. 1, the vehicle V is shown in outline in plan view and side view. The vehicle V of this embodiment is, as an example, a four-wheeled sedan-type passenger vehicle, and may be, for example, a parallel hybrid vehicle. Note that the vehicle V is not limited to a four-wheeled passenger vehicle, and may be a saddle-type vehicle (motorcycle, motor tricycle), or a large vehicle such as a truck or bus.

[0011] The control device CNT includes a controller 1, which is an electronic circuit that controls the vehicle V, including driving assistance for the vehicle V. The controller 1 is equipped with multiple ECUs (Electronic Control Units). An ECU is provided, for example, for each function of the control device CNT. Each ECU includes a processor represented by a CPU (Central Processing Unit), a storage device such as a semiconductor memory, an interface with an external device, etc. The storage device stores programs executed by the processor and data used by the processor for processing, etc. The interface includes an input / output interface and a communication interface. Each ECU may be equipped with multiple processors, multiple storage devices, and multiple interfaces. The programs stored in the storage device may be stored in the storage device by being installed in the control device CNT using a storage medium such as a CD-ROM.

[0012] The controller 1 controls the drive (acceleration) of the vehicle V by controlling a power unit (power plant) 2. The power unit 2 is a traveling drive unit that outputs drive force to rotate the drive wheels of the vehicle V, and may include an internal combustion engine, a motor, and an automatic transmission. The motor can be used as a drive source to accelerate the vehicle V, and can also be used as a generator during deceleration, etc. (regenerative braking).

[0013] In this embodiment, the controller 1 controls the output of the internal combustion engine and the motor and controls the shifting of the gears of the automatic transmission in response to the driving operation of the driver detected by the operation detection sensor 2a provided on the accelerator pedal AP and the operation detection sensor 2b provided on the brake pedal BP, and the vehicle speed of the vehicle V detected by the rotation speed sensor 2c. The automatic transmission is provided with a rotation speed sensor 2c that detects the rotation speed of the output shaft of the automatic transmission as a sensor that detects the running state of the vehicle V. The vehicle speed of the vehicle V can be calculated from the detection result of the rotation speed sensor 2c.

[0014] The controller 1 controls the braking (deceleration) of the vehicle V by controlling the hydraulic device 3. The driver's braking operation on the brake pedal BP is converted into hydraulic pressure in the brake master cylinder BM and transmitted to the hydraulic device 3. The hydraulic device 3 is an actuator that can control the hydraulic pressure of the hydraulic oil supplied to the brake devices 3a (e.g., disc brake devices) provided on each of the four wheels based on the hydraulic pressure transmitted from the brake master cylinder BM.

[0015] The controller 1 can control the braking of the vehicle V by controlling the driving of the solenoid valves and the like provided in the hydraulic device 3. The controller 1 can also configure an electric servo brake system by controlling the distribution of braking force by the brake device 3a and braking force by regenerative braking of the motor provided in the power unit 2. The controller 1 may also turn on the brake lamps 3b during braking.

[0016] The controller 1 controls the steering of the vehicle V by controlling the electric power steering device 4. The electric power steering device 4 includes a mechanism for steering the front wheels in response to the driver's driving operation (steering operation) with respect to the steering wheel ST. The electric power steering device 4 includes a drive unit 4a that generates a driving force (sometimes referred to as steering assist torque) for assisting the steering operation or for automatically steering the front wheels of the vehicle V. The drive unit 4a includes a motor as a drive source. The electric power steering device 4 also includes a steering angle sensor 4b that detects the steering angle, a torque sensor 4c that detects the steering torque borne by the driver (called steering burden torque, to be distinguished from steering assist torque), and the like.

[0017] The controller 1 controls an electric parking brake device 3c provided on the rear wheels of the vehicle V. The electric parking brake device 3c has a mechanism for locking the rear wheels. The controller 1 can control the electric parking brake device 3c to lock and unlock the rear wheels.

[0018] The controller 1 controls an information output device 5 that notifies the driver of information inside the vehicle. The information output device 5 includes, for example, a display device 5a that notifies the driver of information by image and / or an audio output device 5b that notifies the driver of information by audio. The display device 5a includes, for example, a display device provided on the instrument panel or a display device provided on the steering wheel ST. The display device 5a may also include a head-up display. The information output device 5 may notify the occupants of information by vibration or light.

[0019] The controller 1 receives instruction inputs from a passenger (e.g., the driver) via the input device 6. The input device 6 is arranged in a position operable by the driver, and includes, for example, a group of switches 6a through which the driver issues instructions to the vehicle V, and / or a turn signal lever 6b that activates a turn signal (blinker).

[0020] The controller 1 recognizes and determines the current position and course (attitude) of the vehicle V. In this embodiment, the vehicle V is provided with a gyro sensor 7a, a GNSS (Global Navigation Satellite System) sensor 7b, and a communication device 7c. The gyro sensor 7a detects the rotational motion (yaw rate) of the vehicle V. The GNSS sensor 7b detects the current position of the vehicle V. The communication device 7c wirelessly communicates with a server that provides map information and traffic information to acquire this information. In this embodiment, the controller 1 determines the course of the vehicle V based on the detection results of the gyro sensor 7a and the GNSS sensor 7b, and sequentially acquires map information related to the course from the server via the communication device 7c and stores it in a database 7d (storage device). The vehicle V may be provided with other sensors for detecting the state of the vehicle V, such as an acceleration sensor that detects the acceleration of the vehicle V.

[0021] The controller 1 performs driving assistance for the vehicle V based on the detection results of various detection units provided in the vehicle V. The vehicle V is provided with surrounding detection units 8a to 8b, which are external sensors that detect the outside of the vehicle V (surrounding conditions), and interior detection units 9a to 9b, which are interior sensors that detect the conditions inside the vehicle (the conditions of the occupants (particularly the driver)). The controller 1 is able to grasp the surrounding conditions of the vehicle V based on the detection results of the surrounding detection units 8a to 8b, and perform driving assistance in accordance with the surrounding conditions. Furthermore, the controller 1 is able to determine, based on the detection results of the interior detection units 9a to 9b, whether the driver is performing the predetermined operational obligations imposed on the driver when driving assistance is performed.

[0022] The surroundings detection unit 8a is an imaging device that captures images in front of the vehicle V (hereinafter, sometimes referred to as the front camera 8a), and is attached, for example, to the inside of the passenger compartment of the windshield at the front of the roof of the vehicle V. The controller 1 can extract the contours of targets and lane markings (white lines, etc.) on the road by analyzing the images captured by the front camera 8a.

[0023] The surroundings detection unit 8b is a millimeter wave radar (hereinafter, may be referred to as radar 8b), and uses radio waves to detect targets around the vehicle V, and detect (measure) the distance to the target and the direction (azimuth) of the target relative to the vehicle V. In the example shown in FIG. 1, five radars 8b are provided: one in the center of the front of the vehicle V, one at each of the left and right corners of the front, and one at each of the left and right corners of the rear.

[0024] The surrounding detection unit installed in the vehicle V is not limited to the above configuration, and the number of cameras and the number of radars may be changed, or a lidar (Light Detection and Ranging: LIDAR) may be installed to detect targets around the vehicle V.

[0025] The in-vehicle detection unit 9a is an imaging device that captures images of the interior of the vehicle (hereinafter, sometimes referred to as in-vehicle camera 9a), and is attached, for example, to the inside of the vehicle cabin at the front of the roof of the vehicle interior V. In this embodiment, the in-vehicle camera 9a is a driver monitor camera that captures images of the driver (for example, the driver's eyes and face). The controller 1 can determine the driver's line of sight and facial direction by analyzing the image (image of the driver's face) captured by the in-vehicle camera 9a.

[0026] The in-vehicle detection unit 9b is a grip sensor that detects the driver's grip of the steering wheel ST (hereinafter, may be referred to as grip sensor 9b), and is provided, for example, on at least a part of the steering wheel ST. Note that the torque sensor 4c that detects the driver's steering torque may be used as the in-vehicle detection unit.

[0027] <Example of driving assistance control> Driving assistance for the driver of the vehicle V includes, for example, acceleration / deceleration assistance, lane keeping assistance, and lane change assistance. The acceleration / deceleration assistance is a driving assistance (ACC: Adaptive Cruise Control) in which the controller 1 automatically controls the power unit 2 and the hydraulic device 3 based on the detection results of the surroundings detection unit 8 and map information, thereby automatically controlling the acceleration / deceleration of the vehicle V within a predetermined vehicle speed. When there is a preceding vehicle, the ACC can also accelerate or decelerate the vehicle V so as to maintain a safe distance from the preceding vehicle. The ACC reduces the operational burden on the driver of acceleration / deceleration operations (operations of the accelerator pedal AP and brake pedal BP).

[0028] Lane keeping assist is a driving assistance system (LKAS) that keeps the vehicle V within the lane by the controller 1 automatically controlling the electric power steering device 4 based on the detection results of the surroundings detection unit 8 and map information. The LKAS reduces the driver's burden of steering (operation of the steering wheel ST) while the vehicle V is traveling straight.

[0029] Lane change assistance is a driving assistance system (ALC: Advanced Lane Change, ALCA: Active Lane Change Assist) in which the controller 1 automatically controls the power unit 2, hydraulic unit 3, and electric power steering unit 4 based on the detection results of the surroundings detection unit 8 and map information to change the lane of the vehicle V to an adjacent lane. ALC is lane change assistance based on a system request (a request from a control device), while ALCA is lane change assistance based on an occupant request. An example of a system request is a navigation system that guides the vehicle V to its destination, requesting that the vehicle V change lanes. When an occupant request is made, the driver commands a lane change by operating an input device (for example, the turn signal lever 6b). ALC or ALCA reduces the driver's operational burden of accelerating, decelerating, and steering the vehicle V when changing lanes.

[0030] Other examples of driving assistance control may include, for example, a collision mitigation brake, an ABS function, traction control, and / or attitude control of the vehicle V, which assists in avoiding a collision with an object on the road (e.g., a pedestrian, another vehicle, or an obstacle) by controlling the hydraulic device 3.

[0031] <Driving assistance> 2 is a flowchart showing an example of a driving assistance control process executed by an ECU included in the controller 1. In this embodiment, the controller 1 operates as a driving assistance device.

[0032] In S201, the ECU acquires information about the surroundings of the vehicle V using the surrounding detection units 8a to 8b, which are external sensors that detect the outside (surrounding conditions) of the vehicle V. The surrounding information is continuously acquired.

[0033] In S202, the ECU detects targets present around the vehicle V based on the surrounding information acquired in S201. Here, the targets may include various objects, such as traffic lights, road signs, vehicles ahead, vehicles to the side, bicycles, pedestrians, and people running (runners).

[0034] In S203, the ECU detects (identifies) a notification target from among the targets detected in S202, which is a target for notifying the driver. The notification to the driver is a notification to alert the driver, and can be made by voice and / or display via the information output device 5. The notification is made when the notification target suddenly approaches the vehicle V or when there is a significant change in the position or trajectory (movement trajectory) relative to the vehicle V. The notification target is, for example, a traffic participant present in the vicinity of the vehicle V, such as a four-wheeled vehicle, a two-wheeled vehicle, a pedestrian, a bicycle, or a runner in the vicinity of the vehicle V.

[0035] In S204, the ECU estimates the driver's line of sight direction by analyzing the image (image of the driver's face) captured by the in-vehicle camera 9a.

[0036] In S205, the ECU determines whether the driver has visually observed the notification object based on the estimated line of sight. Visual observation here is not limited to direct visual observation of the notification object, but may also include indirect visual observation of the notification object through a side mirror. Furthermore, if a display that displays an image of the area behind the vehicle is provided instead of a side mirror, it may also include indirect visual observation of the notification object through the display. If this step is Yes, proceed to S206. On the other hand, if this step is No, return to S204.

[0037] In S206, the ECU controls a visual inspection flag to be on, which indicates whether the driver has visually inspected the object to be notified. The visual inspection flag is information that can be either on or off. An object to be notified whose visual inspection flag is on indicates that the driver has visually inspected the object (seen it with their eyes). An object to be notified whose visual inspection flag is off indicates that the driver has not visually inspected the object (seen it with their eyes).

[0038] In S207, the ECU controls to turn on a recognition flag indicating whether the driver recognizes the notification object. In this embodiment, when the visual inspection flag is turned on in S206, the recognition flag for the notification object is also turned on at the same time. However, this is not limiting. The recognition flag may be controlled to be turned on when a predetermined time (e.g., 0.4 to 0.5 seconds) has elapsed after the visual inspection flag is turned on. Even if the driver visually inspects (sees) the notification object, there may be a time lag before the driver understands what the object is. In other words, considering that it may take a certain amount of time to recognize what the object is even after seeing it, a time lag may be provided before the recognition flag is turned on.

[0039] In S208, the ECU performs control to suppress notification to the driver. Even if the driver looks away from the notification object while the recognition flag is on, the driver may still be able to see the notification object in his or her peripheral vision, or may remember the approximate position and movement of the notification object even if it is outside the driver's peripheral vision. If notification is performed in such a case, the driver may be overwhelmed by the notification. Therefore, in this embodiment, control to suppress notification is performed when the recognition flag is on. Suppressing notification may include prohibiting notification, reducing the notification frequency, reducing the notification volume, or setting stricter notification conditions. Setting stricter notification conditions may include, for example, raising the threshold when a notification is to be performed when the speed at which the notification object rapidly approaches the vehicle V is equal to or exceeds a threshold.

[0040] In S209, the ECU determines whether the driver has averted their gaze from the object to be notified. The ECU continuously estimates the driver's gaze direction and determines whether the driver has averted their gaze from the object to be notified based on the estimated gaze direction. If this step is Yes, the ECU proceeds to S210. On the other hand, if this step is No, the ECU waits.

[0041] In S210, the ECU controls the visual check flag to be turned off, that is, changes the visual check flag from on to off.

[0042] In S211, the ECU determines whether or not a predetermined condition is satisfied. If the predetermined condition is satisfied, it may include a case where a predetermined time change and / or a predetermined situation change has occurred. If a predetermined time change has occurred, it may include a case where a predetermined time (for example, a preset time ranging from several seconds to several tens of seconds) has elapsed since the visual flag was turned off. If a predetermined situation change has occurred, it may include a case where the position and / or trajectory of the object to be notified relative to the vehicle V has changed beyond an allowable range.

[0043] In S212, the ECU controls the recognition flag to be turned off. That is, the ECU changes the recognition flag from on to off. For example, if a certain amount of time (30 seconds) has passed since the driver last visually checked the notified object, the information that the driver has recognized becomes outdated. Therefore, the ECU controls the recognition flag to be turned off in response to the elapse of a predetermined time (e.g., 30 seconds) since the visual check flag was turned off. Furthermore, if the position and / or trajectory of the notified object relative to the vehicle V has changed beyond an allowable range since the driver last visually checked the notified object, the information that the driver has recognized becomes outdated. Therefore, the ECU controls the recognition flag to be turned off in response to the change in the position and / or trajectory of the notified object beyond an allowable range since the visual check flag was turned off.

[0044] In S213, the ECU cancels the suppression of notification to the driver. That is, when the recognition flag is in the OFF state, notification to the driver is executed when the notification conditions are satisfied. On the other hand, when the recognition flag is in the ON state, notification to the driver is suppressed (prohibited, low frequency, stricter notification conditions, etc.) even if the notification conditions are satisfied. As a result, even if the driver looks away from the object to be notified, notification is suppressed as long as it is determined that the object to be notified is recognized, making it possible to prevent excessive notifications.

[0045] Here, the notification condition may be that, with the recognition flag in an off state, a notification object (e.g., a bicycle, pedestrian, or runner) is detected approaching vehicle V when vehicle V makes a right or left turn. For example, the notification condition may be that a notification object (e.g., a bicycle, pedestrian, or runner) is detected crossing a crosswalk when vehicle V makes a right turn. FIG. 5 is a diagram showing a situation in which vehicle V turns right at an intersection. Notification object 301 is about to cross a crosswalk in the direction of the right turn. At this time, a warning area 502 may be set outside a predetermined area 501 in the traveling direction of vehicle V, and the notification condition may be satisfied when it is detected that notification object 301 has entered warning area 502.

[0046] The notification condition may also be that the Time to Collision (TTC) is equal to or less than a threshold value when the recognition flag is off. That is, notification may be performed when the value obtained by dividing the distance between vehicle V and a notification target (e.g., a preceding vehicle) present within a predetermined area in the vehicle's traveling direction by the relative speed between the vehicle and the notification target is equal to or less than a threshold value. FIG. 6 is a diagram showing the relationship between vehicle V and a preceding vehicle traveling ahead of vehicle V. A preceding vehicle 601 is present within a predetermined area 602 in the traveling direction of vehicle V. At this time, the TTC may be continuously calculated, and the notification condition may be satisfied when the TTC is equal to or less than the threshold value. Alternatively, the moving average value of the TTC (the average value of TTC changes within a predetermined period) may be constantly calculated, and the notification condition may be satisfied when the second moving average value after a predetermined time (e.g., 1 second) becomes smaller than the first moving average value. Alternatively, notification may be performed when the time change of TTC indicates a decrease equal to or greater than a threshold value. When the TTC is observed to change over time and is found to have decreased by more than a certain amount, it is possible that the driver is approaching the vehicle without noticing. In such cases, it is possible to provide an appropriate warning to the driver. For example, if the TTC value decreases from 4 seconds to 2 seconds, a notification can be issued. On the other hand, even if the TTC value is small, if the TTC value does not change from 2 seconds to 2 seconds, it is possible that the driver is aware of the situation and is maintaining a certain distance between vehicles. In such cases, excessive notifications can be prevented.

[0047] Furthermore, the notification condition may be that an object to be notified (e.g., a bicycle, pedestrian, or runner) has entered a predetermined area in the traveling direction of the vehicle V (e.g., a predetermined area ahead of the vehicle) while the recognition flag is off. FIG. 7 is a diagram showing the relationship between the vehicle V and the object to be notified. FIG. 7 shows a state in which the object to be notified has entered a predetermined area 701 in the traveling direction of the vehicle V. A bounding box of the object to be notified 301, which will be described later with reference to FIG. 3, may be created, and it may be determined that the object has entered the area when at least one of the determination points constituting the corners of the bounding box has entered the predetermined area 701.

[0048] In this way, when the driver fails to recognize an object to be notified and overlooks it, a warning is sent to the driver.

[0049] In S214, the ECU determines whether to continue the process. For example, the process may be continued while the driver is starting the vehicle V (the engine is running, and the power supply of the electric vehicle is on). If this step is Yes, the process returns to S201. On the other hand, if this step is No, the process ends.

[0050] <Visual inspection flag and recognition flag transition> When the driver sees an object to be notified (for example, a runner running diagonally ahead and to the left of vehicle V), the visual flag for that object is turned on. Accordingly, the recognition flag is also turned on immediately, or after a predetermined time (0.4 to 0.5 seconds) has passed. If the driver then looks away from the object to be notified, the visual flag is turned off. At this time, the visual flag is set to off and the recognition flag is set to on. In this state, the driver is not looking at the object to be notified, but it is assumed that he or she has recognized it, so notification is suppressed even if the notification conditions are met. This makes it possible to suppress excessive notifications.

[0051] If a predetermined time has passed since the visual inspection flag was turned off, it can be assumed that the driver has not recognized the object to be notified, and therefore the recognition flag is changed to off. Alternatively, if the position and / or trajectory of the object to be notified relative to the vehicle V has changed beyond an allowable range since the visual inspection flag was turned off, it can be assumed that the driver has not recognized the object to be notified, and therefore the recognition flag is changed to off. At this time, the visual inspection flag = off and the recognition flag = off state is reached. In this state, notification will be made when the notification conditions are met.

[0052] As described above, in this embodiment, when the recognition flag is on, notification to the driver is suppressed (prohibited, low frequency, stricter notification conditions, etc.) even if the notification conditions are met. In other words, after the driver looks away from the object to be notified, notification is suppressed even if the notification conditions are met until a predetermined condition is met (a predetermined period has passed, or the position and / or trajectory of the object to be notified relative to the vehicle V has changed beyond an allowable range).

[0053] This means that even if the driver looks away from the object to be notified, notifications are suppressed as long as it is determined that the driver is able to recognize the object, thereby making it possible to prevent excessive notifications.

[0054] [Variations] In the above embodiment, an example has been described in which the driver's line of sight direction is estimated, and if an object is present in the line of sight, it is determined that the object has been visually observed. There may be various variations in the method of determining whether the driver has visually observed the object to be notified.

[0055] 3 and 4 are explanatory diagrams of visual judgment according to this modified example. Fig. 3 shows a rectangular parallelepiped bounding box 303 that surrounds the periphery of a notification target 301. The bounding box 303 includes a total of nine judgment points, including eight judgment points A to H that form the corners and one judgment point that is the center point I. A line of sight 302 indicates the direction of the driver's line of sight.

[0056] FIG. 4 shows an example of a model in which a predetermined range from the driver's central visual field is defined as the effective visual field. It is a Gaussian distribution diagram with the vertical axis representing the accuracy (0 to 1) of visually observing an object and the horizontal axis representing the distance (m) from the center of gaze. Among the judgment points A to I, judgment point A has the highest visual observation accuracy (it is located at the end of the gaze 302 and is the closest), so judgment point A is adopted. Then, it is determined whether the accuracy of this judgment point A is equal to or greater than a threshold value (e.g., 0.8). If the accuracy of the adopted judgment point is equal to or greater than the threshold, it is determined that the object has been visually observed. On the other hand, if the accuracy of the adopted judgment point is less than the threshold, it is determined that the object has not been visually observed. In the illustrated example, since the accuracy of judgment point A is equal to or greater than a threshold value (e.g., 0.8), it can be determined that the object has been visually observed by the gaze 302. Alternatively, it may be configured to integrate the accuracy of visually observing an object even once over time, and determine that the object has been visually observed if the integrated value is equal to or greater than a threshold. For example, the integrated value may be calculated using a formula such as "1.0 - (1.0 - accuracy t1) x (1.0 - accuracy t2) x (1.0 - accuracy t3)." Here, accuracy tX indicates the accuracy at time tX. This allows the accuracy of visually observing the object to be reflected in the processing with greater precision.

[0057] In the examples of FIGS. 3 and 4, the notification target 301 is a pedestrian, but the present invention is also applicable to vehicles such as four-wheeled vehicles, two-wheeled vehicles, bicycles, and runners.

[0058] In the above embodiment, the case where there is one notification object has been described as an example, but the present invention is not limited to this example. There may be multiple notification objects. In this case, it is sufficient to configure the system so that each notification object is distinguished and the visual inspection flag and recognition flag are controlled to be on / off for each notification object.

[0059] <Summary of the embodiment> 1. The driving assistance device (1) according to the above embodiment is A driving assistance device, A detection means (1, 8a to 8b) for detecting a notification target object that is a target for notifying a driver of a vehicle (V) based on peripheral information of the vehicle (V); An estimation means (1, 9a) for estimating the driver's line of sight; a first control means (1) for controlling a visual observation flag indicating whether or not the driver has visually observed the notified object based on the line of sight direction when the driver has visually observed the notified object, to be on; a second control means (1) for controlling a recognition flag, which indicates whether the driver recognizes the object to be notified, to be turned on when the visual flag is turned on; a suppression means (1) for suppressing the notification while the recognition flag is on; The first control means controls the visual inspection flag to be turned off in response to the driver averting their gaze from the object to be notified while the recognition flag is on, The second control means controls the recognition flag to be turned off when a predetermined condition is satisfied after the visual inspection flag is turned off.

[0060] In this way, when the recognition flag is on, even if the notification condition is met, notification to the driver is suppressed. Therefore, even if the driver looks away from the object to be notified, notification is suppressed as long as it is determined that the object to be notified is recognized. Therefore, it is possible to prevent excessive notifications from being sent to the driver.

[0061] 2. In the driving assistance device (1) according to the above embodiment, If the predetermined condition is satisfied, it includes that a first predetermined time has elapsed.

[0062] This allows the recognition flag to remain on until a predetermined time (for example, several to several tens of seconds) has passed since the visual flag was turned off, and until a predetermined time has passed since the last time the object to be notified was seen (while the memory is fresh), thereby preventing excessive notifications from being made during this time.

[0063] 3. In the driving assistance device (1) according to the above embodiment, If the predetermined condition is satisfied, it includes that the position and trajectory of the object to be notified have changed beyond an allowable range.

[0064] This keeps the recognition flag on until the object to be notified that the driver recognized changes significantly and the deviation from recognition becomes large, thereby preventing excessive notifications from being made during this period.

[0065] 4. In the driving assistance device (1) according to the above embodiment, The second control means controls the recognition flag to be on after a second predetermined time has elapsed since the visual inspection flag was turned on.

[0066] This makes it possible to realize control that takes into account the time lag between when the driver sees the object to be notified and when the driver recognizes what it is.

[0067] 5. In the driving assistance device (1) according to the above embodiment, The notification is made in response to the object to be notified entering a predetermined area in the traveling direction of the vehicle.

[0068] This makes it possible to give the driver an appropriate warning about sudden sudden jumps into the path of the vehicle.

[0069] 6. In the driving assistance device (1) according to the above embodiment, The notification is made when the value obtained by dividing the distance between the vehicle and the object to be notified, which is located within a specified area in the direction of travel of the vehicle, by the relative speed between the vehicle and the object to be notified becomes less than a threshold value.

[0070] This makes it possible to give the driver an appropriate warning when the TTC falls below the threshold.

[0071] 7. The driving assistance device (1) according to the above embodiment is Further, a notification means (5) is provided to notify the driver, The notification means issues the notification when a notification condition is satisfied.

[0072] This makes it possible to give the driver an appropriate warning.

[0073] 8. In the driving assistance device (1) according to the above embodiment, When the notification condition is satisfied, it includes the fact that the object to be notified has entered a predetermined area in the traveling direction of the vehicle.

[0074] This makes it possible to give the driver an appropriate warning about sudden sudden jumps into the path of the vehicle.

[0075] 9. In the driving assistance device (1) according to the above embodiment, The notification conditions are met when the value obtained by dividing the distance between the vehicle and the object to be notified that is located within a specified area in the direction of travel of the vehicle by the relative speed between the vehicle and the object to be notified is equal to or less than a threshold value.

[0076] This makes it possible to give the driver an appropriate warning when the TTC falls below the threshold.

[0077] 10. In the driving assistance device (1) according to the above embodiment, Cases where the notification conditions are met include cases where the change over time in the value obtained by dividing the distance between the vehicle and the object to be notified that is located within a specified area in the direction of travel of the vehicle by the relative speed between the vehicle and the object to be notified shows a decrease of more than a threshold value.

[0078] This allows the system to properly alert the driver when the TTC value decreases by more than a certain amount over time, indicating that the driver may be approaching the vehicle without noticing. For example, a notification can be issued when the TTC value decreases from 4 seconds to 2 seconds. On the other hand, even if the TTC value is small, if the TTC value does not change from 2 seconds to 2 seconds, the driver may be aware of the situation and may be maintaining a certain distance between vehicles. In such cases, excessive notifications can be prevented.

[0079] 11. In the driving assistance device (1) according to the above embodiment, The suppression means inhibits the notification while the recognition flag is on.

[0080] This makes it possible to prevent unnecessary notifications.

[0081] 12. In the driving assistance device (1) according to the above embodiment, The suppression means reduces the frequency of the notification while the recognition flag is on compared to while the recognition flag is off.

[0082] This makes it possible to suppress unnecessary notifications.

[0083] 13. In the driving assistance device (1) according to the above embodiment, The suppression means tightens the notification condition while the recognition flag is on so that the notification condition is less likely to be satisfied than while the recognition flag is off.

[0084] This makes it possible to suppress unnecessary notifications.

[0085] 14. The driving assistance device (1) according to the above embodiment is A driving assistance device, A detection means (1, 8a to 8b) for detecting a notification target object that is a target for notifying a driver of a vehicle (V) based on peripheral information of the vehicle (V); notification means (5) for issuing the notification when the notification conditions are met; An estimation means (1, 9a) for estimating the driver's line of sight; a determination means (1) for determining whether the driver has averted their gaze from the object to be notified based on the gaze direction; a suppression means (1) for suppressing notification by the notification means until a predetermined condition is satisfied after the driver turns away from the object to be notified, even if the notification condition is satisfied; Equipped with.

[0086] As a result, even if the driver looks away from the object to be notified, notification is suppressed as long as it is determined that the driver can recognize the object to be notified, making it possible to prevent excessive notification.

[0087] 15. The vehicle (V) according to the above embodiment is A vehicle is characterized by including the driving assistance device according to the above embodiment.

[0088] This allows the functions of the driving assistance device according to the above embodiment to be realized in the vehicle.

[0089] 16. The control method of the driving assistance device (1) according to the above embodiment is A control method for a driving assistance device, comprising: a detection step (S201, S202, S203) of detecting a notification target object that is a target for notifying a driver of a vehicle (V) based on peripheral information of the vehicle (V); an estimation step (S204) of estimating the driver's line of sight; a step of controlling a visual observation flag indicating whether or not the driver has visually observed the notified object based on the line of sight direction to be on (Yes in S205, S206); When the visual flag is turned on, a recognition flag indicating whether the driver recognizes the object to be notified is turned on (S207); a suppression step (S208) of suppressing the notification while the recognition flag is on; A step of turning off the visual flag in response to the driver averting his or her gaze from the object to be notified while the recognition flag is on (Yes in S209, S210); After the visual inspection flag is turned off, if a predetermined condition is satisfied, a step of controlling the recognition flag to be turned off (Yes in S211, S212); It has.

[0090] In this way, when the recognition flag is on, even if the notification condition is met, notification to the driver is suppressed. Therefore, even if the driver looks away from the object to be notified, notification is suppressed as long as it is determined that the object to be notified is recognized. Therefore, it is possible to prevent excessive notifications from being sent to the driver.

[0091] 17. The control method of the driving assistance device (1) according to the above embodiment is A control method for a driving assistance device, comprising: a detection step of detecting a notification target object to be notified to a driver of the vehicle based on information about the vicinity of the vehicle; a notification step of issuing the notification when a notification condition is satisfied; an estimation step of estimating the driver's line of sight; a determination step of determining whether the driver has averted their gaze from the object to be notified based on the gaze direction; a suppression step of suppressing the notification until a predetermined condition is satisfied after the driver turns away from the object to be notified, even if the notification condition is satisfied; It has.

[0092] As a result, even if the driver looks away from the object to be notified, notification is suppressed as long as it is determined that the driver can recognize the object to be notified, making it possible to prevent excessive notification.

[0093] 18. The program according to the above embodiment 1 is a program for causing a computer to execute a control method for a driving assistance device according to the above embodiment.

[0094] This makes it possible to realize the processing of the driving assistance device by a computer.

[0095] 19. The storage medium according to the above embodiment is A storage medium storing a program for causing a computer to execute the control method for the driving assistance device according to the above embodiment.

[0096] This makes it possible to realize the processing of the driving assistance device using the storage medium.

[0097] <Other embodiments> Furthermore, a program for realizing one or more functions described in each embodiment can be supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device can read and execute the program. The present invention can also be realized in such an embodiment.

[0098] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention. [Explanation of symbols]

[0099] CNT: control device, V: vehicle, 1: controller

Claims

1. A driving assistance device, a detection means for detecting a notification target object to be notified to a driver of the vehicle based on information about the vicinity of the vehicle; an estimation means for estimating the driver's line of sight; a first control means for controlling a visual observation flag, which indicates whether the driver has visually observed the notification object, to be on when the driver has visually observed the notification object based on the line of sight direction; and a second control means for controlling a recognition flag to be on when the visual flag is turned on, the recognition flag indicating whether the driver recognizes the object to be notified; a suppression means for suppressing the notification while the recognition flag is on, The first control means controls the visual inspection flag to be turned off in response to the driver averting their gaze from the object to be notified while the recognition flag is on, The driving assistance device is characterized in that the second control means controls the recognition flag to be turned off when a predetermined condition is satisfied after the visual inspection flag is turned off.

2. 2. The driving assistance device according to claim 1, wherein the satisfaction of the predetermined condition includes the lapse of a first predetermined time.

3. The driving assistance device according to claim 1 , wherein the satisfaction of the predetermined condition includes a change in the position and trajectory of the object to be notified exceeding an allowable range.

4. 2. The driving assistance device according to claim 1, wherein the second control means controls the recognition flag to be on after a second predetermined time has elapsed since the visual inspection flag was turned on.

5. 2. The driving assistance device according to claim 1, wherein the notification is made in response to the object to be notified entering a predetermined area in the traveling direction of the vehicle.

6. 2. The driving assistance device according to claim 1, wherein the notification is made when a value obtained by dividing the distance between the vehicle and the object to be notified that is present within a predetermined area in the direction of travel of the vehicle by the relative speed between the vehicle and the object to be notified becomes equal to or less than a threshold value.

7. Further, a notification means for notifying the driver is provided, 2. The driving assistance device according to claim 1, wherein the notification means issues the notification when a notification condition is satisfied.

8. The driving assistance device according to claim 7 , wherein the notification condition being satisfied includes the object to be notified entering a predetermined area in the traveling direction of the vehicle.

9. The driving assistance device according to claim 7, characterized in that the case where the notification condition is satisfied includes a case where a value obtained by dividing the distance between the vehicle and the object to be notified that is present within a predetermined area in the direction of travel of the vehicle by the relative speed between the vehicle and the object to be notified is equal to or less than a threshold value.

10. The driving assistance device according to claim 7, characterized in that when the notification conditions are met, the notification conditions include a case where a change over time in a value obtained by dividing the distance between the vehicle and the object to be notified that is present within a predetermined area in the traveling direction of the vehicle by a relative speed between the vehicle and the object to be notified shows a decrease of more than a threshold value.

11. The driving assistance device according to claim 1 , wherein the suppression means prohibits the notification while the recognition flag is on.

12. The driving assistance device according to claim 1 , wherein the suppression unit reduces the frequency of the notification while the recognition flag is on compared to while the recognition flag is off.

13. The driving assistance device according to claim 7, wherein the suppression means tightens the notification condition while the recognition flag is on so that the notification condition is less likely to be satisfied than while the recognition flag is off.

14. A driving assistance device, a detection means for detecting a notification target object to be notified to a driver of the vehicle based on information about the vicinity of the vehicle; a notification means for issuing the notification when a notification condition is satisfied; an estimation means for estimating the driver's line of sight; a determination means for determining whether the driver has averted their gaze from the object to be notified based on the gaze direction; a suppression means for suppressing notification by the notification means until a predetermined condition is satisfied after the driver turns away from the object to be notified, even if the notification condition is satisfied; A driving assistance device comprising:

15. A vehicle comprising the driving assistance device according to any one of claims 1 to 14.

16. A control method for a driving assistance device, comprising: a detection step of detecting a notification target object to be notified to a driver of the vehicle based on information about the vicinity of the vehicle; an estimation step of estimating the driver's line of sight; a step of controlling a visual observation flag indicating whether or not the driver has visually observed the notification object based on the line of sight direction to be on when the driver has visually observed the notification object; When the visual flag is turned on, controlling a recognition flag indicating whether the driver recognizes the object to be notified to be on; a suppression step of suppressing the notification while the recognition flag is on; a step of controlling the visual flag to be turned off in response to the driver averting his or her gaze from the object to be notified while the recognition flag is on; a step of turning off the recognition flag when a predetermined condition is satisfied after the visual inspection flag is turned off; A control method for a driving assistance device, comprising:

17. A control method for a driving assistance device, comprising: a detection step of detecting a notification target object to be notified to a driver of the vehicle based on information about the vicinity of the vehicle; a notification step of issuing the notification when a notification condition is satisfied; an estimation step of estimating the driver's line of sight; a determination step of determining whether the driver has averted their gaze from the object to be notified based on the gaze direction; a suppression step of suppressing the notification until a predetermined condition is satisfied after the driver turns away from the object to be notified, even if the notification condition is satisfied; A control method for a driving assistance device, comprising:

18. A program for causing a computer to execute the method for controlling a driving assistance device according to claim 16 or 17.

19. A storage medium storing a program for causing a computer to execute the method for controlling a driving assistance device according to claim 16 or 17.

Images