Driving support device, and driving support method

Abstract

To switch a level of giving notice of a collision risk between a vehicle and an obstacle in accordance with an attention degree of a driver to the obstacle outside the vehicle in consideration of a dead angle of the driver according to a peripheral situation of the vehicle.SOLUTION: A driving support device comprises: imaging means for acquiring an image in front of a vehicle; detection means for detecting an obstacle that is present around the vehicle; monitoring means for monitoring the condition of a driver of the vehicle; communication means for acquiring peripheral information concerning a peripheral situation including a dead angle of the vehicle by communicating with the other vehicle; and control means that switches a degree of driving support for the driver according to a collision risk for every dead angle area, by generating a watching point map based on the driver's condition, by generating a visual saliency map based on the image of the imaging means and the peripheral information of the vehicle, and by level-setting the collision risk between the vehicle and the obstacle in the dead angle area based on the watching point map and the visual saliency map.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a driving assistance device and a driving assistance method for a vehicle. [Background technology]

[0002] Technologies have been developed to support safe driving by providing audio and visual notifications (warnings, warnings, etc.) in response to road landmarks (obstacles, traffic signs, etc.) and the behavior of other vehicles (position, speed, direction of travel, etc.). To determine whether or not such notifications are necessary, vehicles acquire images of the vehicle's surroundings (front, left, right, rear, etc.) using on-board devices (sensing devices such as on-board cameras). Meanwhile, technologies have also been developed that use driver monitoring systems (DMSs) to detect information such as the driver's line of sight, head movement, and posture to provide notifications. Furthermore, it is also possible to estimate driving risks and provide notifications by detecting the vehicle's surroundings (presence or absence of oncoming vehicles, presence or absence of pedestrians, traffic conditions, weather conditions, etc.) based on information acquired through vehicle-to-vehicle (V2V) communications and vehicle-to-infrastructure (V2I) communications. Furthermore, it will be possible to detect the situation around the vehicle and provide safe driving assistance by communicating between multiple vehicles via the Internet, such as IoT (Internet of Things) or IoE (Internet of Everything), or by communicating with terminal devices carried by pedestrians.

[0003] Patent Document 1 discloses a driving assistance device mounted on a vehicle. The driving assistance device monitors the driver's line of sight using an on-board gaze sensor, and when it determines that the driver is aware of other vehicles, it excludes data transmitted from the other vehicles from the scope of vehicle-to-vehicle communication. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-191630 Summary of the Invention [Problem to be solved by the invention]

[0005] The driving assistance device of Patent Document 1 excludes data from other vehicles from the targets of vehicle-to-vehicle communication based on a determination of whether other vehicles have been detected from the driver's line of sight, thereby suppressing communication traffic unnecessary for safe driving assistance. Conventional driving assistance devices determine whether the driver can pay attention to targets outside the vehicle (obstacles, other vehicles, pedestrians, traffic signs, traffic lights, etc.), and do not take into account blind spots seen from the driver's perspective, such as blind spots caused by vehicle pillars, blind spots blocked by other vehicles, and blind spots that are difficult to focus attention on, such as those behind the driver. Conventional driving assistance devices make their determination regardless of whether there are targets requiring attention, such as other vehicles, in the driver's blind spots, and therefore may provide unnecessary notifications or control for targets that are difficult to predict through vehicle-to-vehicle communication, such as pedestrian movements.

[0006] Furthermore, even if the driver's line of sight is monitored and the situation around the vehicle detected by vehicle-to-vehicle communication (road-to-vehicle communication) is analyzed, there is a possibility that some object may be present in the driver's blind spot in an intersection environment where other vehicles and pedestrians are present. It is difficult to provide the necessary notification of the object present in the driver's blind spot in a manner that corresponds to the driver's attention level (attention level to objects outside the vehicle).

[0007] In addition to the method of monitoring the driver's gaze, there is also a known technology for estimating visual saliency. A visual saliency map is generated by performing image processing on an input image, such as an image ahead of the vehicle, using a predetermined algorithm that reflects human visibility, thereby emphasizing areas to which humans are likely to pay attention. Visually salient areas or areas that are likely to attract gaze are displayed with high brightness (high-brightness coordinates) to visualize them on the visual saliency map. When the gaze coordinates of the vehicle driver are superimposed on the visual saliency map, if the high-brightness coordinates and gaze coordinates are far apart, it can be determined that the driver's visual attention is distracted. On the other hand, if the high-brightness coordinates and gaze coordinates are close on the visual saliency map, it can be determined that the driver's visual attention is focused on a location where they should be visually focused.

[0008] As described above, a visual saliency map generated from the driver's gaze and the image of the area in front of the vehicle can be analyzed in a comprehensive manner to provide driver notification and vehicle control. Intersection environments involve a mixture of traffic lights, road signs, other vehicles, pedestrians, and other objects, placing a high level of caution on the driver. While visual saliency maps can extract highly conspicuous (salient) areas, they cannot include the vehicle's blind spots as targets that the driver should pay attention to. In particular, in intersection environments where attention must also be paid to the vehicle's blind spots, it is difficult to estimate the driver's visual attention concentration level and provide notification.

[0009] In particular, when there is an object (such as an obstacle, another vehicle, or a pedestrian) in the driver's blind spot, there is a risk that the vehicle will collide with the object due to a decrease in the driver's attention. Therefore, it is necessary to evaluate the collision risk between the vehicle and the object and determine whether or not to notify the driver depending on the collision risk.

[0010] In order to solve the above-mentioned problems, an embodiment of the present invention aims to provide a driving assistance device and a driving assistance method that can switch the level of notification of the risk of collision between the vehicle and a target according to the driver's level of attention to targets outside the vehicle, taking into account the driver's blind spots, depending on the situation around the vehicle. [Means for solving the problem]

[0011] A first aspect of the present invention is a driving assistance device comprising: an imaging means for acquiring an image of the area in front of the vehicle; a detection means for detecting obstacles present around the vehicle; a monitoring means for monitoring the driver's status; a communication means for communicating with other vehicles present outside the vehicle to acquire peripheral information relating to the surrounding conditions including the vehicle's blind spots; and a control means for generating a gaze point map displaying the driver's gaze area based on the driver's status, performing predetermined image processing on the image acquired by the imaging means and generating a visual saliency map reflecting the driver's blind spot area based on the information about the vehicle's surroundings, leveling the risk of collision between the vehicle and obstacles in the blind spot area based on the gaze point map and the visual saliency map, and switching the degree of driving assistance for the driver according to the level of collision risk for each blind spot area.

[0012] A second aspect of the present invention is a driving assistance method that acquires an image in front of the vehicle, detects obstacles present around the vehicle, monitors the driver's state, communicates with other vehicles outside the vehicle to acquire peripheral information related to the surrounding situation including the vehicle's blind spots, generates a gaze point map that displays the driver's gaze area based on the driver's state, performs predetermined image processing on the image, and generates a visual saliency map that reflects the driver's blind spot area based on the information about the vehicle's surroundings, levels the risk of collision between the vehicle and obstacles in the blind spot area based on the gaze point map and the visual saliency map, and switches the degree of driving assistance for the driver according to the level of collision risk for each blind spot area. [Effects of the Invention]

[0013] According to an embodiment of the present invention, the degree of notification of the risk of collision between the vehicle and a target can be switched according to the driver's level of attention to targets outside the vehicle, taking into account the driver's blind spots and the situation around the vehicle. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a block diagram showing a configuration of a driving assistance device according to an embodiment of the present invention; [Figure 2]3 is a flowchart showing the procedure of a driving assistance method according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0015] A driving assistance device and a driving assistance method according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0016] In a method that performs a combined analysis of the driver's line of sight and a visual saliency map generated from video footage outside the vehicle, in ever-changing traffic conditions such as in an intersection environment (traffic light conditions, the behavior of other vehicles relative to the vehicle, pedestrian behavior, etc.), there is a possibility that objects in the driver's blind spot may obstruct driving or cause a risk of contact or collision.

[0017] Blind spots of a vehicle based on the driver include (a) physical blind spots such as pillars at the four corners of the vehicle, (b) driving blind spots that the driver needs to consciously pay attention to while driving the vehicle (blind spots on the left, right, and rear of the vehicle, etc.), and (c) blind spots outside the field of view of the vehicle's front camera.

[0018] Although it is difficult for drivers to directly view physical blind spots, if image information from other vehicles can be acquired through vehicle-to-vehicle (or road-to-vehicle) communication, they can recognize targets in the blind spot on an in-vehicle monitor. Drivers can recognize blind spots by changing their posture and looking at the side or rearview mirror. For blind spots outside the camera's field of view, a visual saliency map can be created and analyzed based on images from the vehicle's front camera. However, while visual saliency maps created by incorporating human visual perception as an algorithm can identify salient areas in an image, it is difficult to estimate the driver's attention level. Therefore, a driver's gaze information detected by a driver monitoring system can be superimposed on the visual saliency map for analysis. However, because drivers' gaze movements are diverse while driving, it is difficult to predict the driver's ideal gaze depending on multiple scenarios (such as driving straight, turning right or left, stopping, parking, entering an intersection, the presence or absence of pedestrians, and the traffic light conditions at the intersection). It is possible to determine a driver's gaze information by collecting large amounts of gaze data from many drivers, collating statistics, and then performing machine learning or deep learning, but this is difficult to achieve in a real-world in-vehicle system.

[0019] In this embodiment, three types of blind spots are defined: (1) blind spots on the road (such as the shadows of other vehicles and parked vehicles), (2) blind spots requiring the driver's attention (such as blind spots to the left, right, or rear of the vehicle that the driver should pay attention to, and blind spots that occur when the driver pays too much attention to targets that are likely to attract the driver's attention), and (3) blind spots of the vehicle (such as blind spots due to the vehicle body structure). In this embodiment, the driver's attention level to the three types of blind spots is evaluated relatively, and driving assistance for the driver is activated in the blind spot area, thereby making it possible to prevent unnecessary driving assistance from being automatically provided regardless of the driver's attention level.

[0020] For example, a simple visual saliency map alone cannot address "road blind spots," such as behind parked vehicles, between lines of vehicles, or intersections with poor visibility. In this embodiment, a visual saliency map that takes road blind spots into account can be created and set to increase the saliency of the blind spots. A simple visual saliency map alone cannot address "attention blind spots," such as when a driver focuses too much attention on one area of ​​an image displayed in their field of view and is unable to pay attention to other areas. In this embodiment, the driving assistance device is linked to a driver monitoring system to determine whether the driver is paying attention to areas other than the area ahead of the vehicle and whether the driver's attention is appropriately dispersed. A simple visual saliency map alone cannot address "vehicle blind spots," such as areas that the driver cannot physically see from the driver's seat. When a blind spot that the driver cannot see occurs due to a vehicle pillar while the vehicle is stopped, the driver must change their posture to check for the presence of an obstacle or other object in the blind spot. In this embodiment, a driver monitoring system or the like can detect changes in the driver's line of sight, head (face) direction, posture, etc., and notify the driver of the presence of targets or the like in the blind spot area.

[0021] In this embodiment, for example, a driver's gaze map is generated based on information such as the driver's line of sight, head (face) direction, and posture detected by a driver monitoring system, and a visual saliency map that takes blind spots into account is generated based on images and videos captured by the vehicle's front camera.The gaze map and visual saliency map are combined to evaluate the driver's attention level, and the risk of contact or collision between the vehicle and targets such as obstacles (other vehicles, pedestrians, etc.) in the blind spots is leveled.In this way, the level of notification is switched depending on the level of contact or collision risk between the target and the vehicle (hereinafter referred to as "collision risk").

[0022] A gaze point map is a heat map obtained by detecting and quantifying the area in the field of view ahead of the vehicle to which the driver is paying attention based on the driver's gaze, head (face) direction, posture, etc. For example, when the driver's gaze is directed toward a portion of the scenery on both sides of the road in the driver's direction of travel while the vehicle is traveling straight, a heat map that emphasizes a portion of the scenery in the driver's gaze direction is expected. A method for emphasizing the driver's visual area may be to increase the brightness of a portion of the image or to use a different color. A heat map may be generated by varying the brightness or color depending on the number of times or duration that the driver directs his or her gaze toward a desired area within a predetermined period of time. For example, the heat map may be visualized by displaying areas where the driver is not looking in cool colors (e.g., blue) and areas where the driver is looking in warm colors (e.g., red, yellow).

[0023] A visual saliency map is an image processing method for images and videos captured by a vehicle's forward camera, using a predetermined algorithm that reflects human visibility. For example, lighting towers and buildings are more visible in the human field of view than backgrounds. For example, if the background is black, a map showing highly visible areas in white is used. Visual saliency maps are used to estimate human visual attention from images and videos, and visual saliency maps that take into account human head movement and facial features have been studied. However, it is difficult to detect areas corresponding to the driver's actual line of sight using a visual saliency map alone. In the present embodiment, a visual saliency map that takes into account the driver's blind spot area is generated. Specifically, a visual saliency map that emphasizes areas that are easily visible to the driver is generated, and the driver's blind spot area detected using the forward camera is projected onto the visual saliency map.

[0024] Next, the configuration of the driving assistance device according to this embodiment will be described. FIG. 1 is a block diagram of an in-vehicle system 1 according to this embodiment. The in-vehicle system 1 is configured with an input unit 100, a determination unit 200, and an output unit 300. The input unit 100, the determination unit 200, and the output unit 300 are connected via an in-vehicle network such as a Controller Area Network (CAN). The input unit 100 inputs predetermined information using a physical device. The determination unit 200 executes a driving assistance determination process based on the input information, and the output unit 300 notifies the driver of the presence or absence of a target (or performs control intervention) by voice, image, or the like. For this purpose, a processor (CPU, GPU, etc.) executes a program stored in a memory (semiconductor memory, etc.), thereby executing the driving assistance determination process of the determination unit 200.

[0025] The driving assistance device according to this embodiment is realized by the functions of the in-vehicle system 1, but is not limited to the information and functions shown in Fig. 1. For example, in the driving assistance device, an input unit 100 inputs a camera image or video, driver information such as the driver's line of sight, head (face) direction, and posture, and vehicle information related to the subject vehicle and other vehicles (or pedestrians). A determination unit 200 determines the need for a notification to the driver (or vehicle control intervention) based on a gaze point map and a visual saliency map that takes blind spots into account, and an output unit 300 uses audio, images, etc. to notify the driver to pay attention to the presence of a target (or executes vehicle control intervention).

[0026] The input unit 100 of the in-vehicle system 1 includes a front camera 101, a driver monitoring system 102, vehicle information 103, and IoE information 104 mounted on the vehicle.

[0027] The front camera 101 captures images and videos of the area in front of the vehicle. The driver monitoring system 102 includes an in-vehicle camera for detecting the driver's line of sight. While high accuracy in line of sight measurement is not necessarily required in an intersection environment, it is preferable to have a certain degree of resolution for the direction of the driver's head.

[0028] The driver monitoring system 102 may be provided with a sensing device such as a pressure sensor in the driver's seat. The driver's behavior can be detected by the sensing device in the driver's seat. For example, if the driver's line of sight detected by an onboard camera is facing the left side in front of the vehicle, and the sensing device in the driver's seat can detect that the driver has leaned their body to the left, it can be determined that the driver is paying attention to a pedestrian on the sidewalk on the left side of the road. Note that although the onboard camera is necessary for measuring the driver's line of sight in the driver monitoring system 102, the sensing device in the driver's seat may be used as an auxiliary device. In other words, the driver's line of sight may be used primarily, and the driver's posture in the driver's seat may be used as an auxiliary device.

[0029] The vehicle information 103 is a record of the driving status of the vehicle by the driver. For example, the vehicle information 103 may record not only information about the vehicle body but also information about the behavior of the vehicle while driving (speed, acceleration, deceleration, going straight, turning left, turning right, etc.).

[0030] In this embodiment, in order to evaluate the risk in the blind spot of the vehicle (driving risk depending on whether there is an obstacle or a target such as a pedestrian in the blind spot), it is necessary to provide the vehicle with an omnidirectional sensor (for example, a plurality of sensing devices attached to the vehicle body, a device that can be linked with GPS or GNSS, etc.) Alternatively, it is necessary to provide the vehicle with a mechanism for vehicle-to-vehicle communication or road-to-vehicle communication to acquire information from other vehicles and information from the traffic system (especially information related to the vehicle's surrounding environment).

[0031] The IoE information 104 includes information obtained from other vehicles via vehicle-to-vehicle communication (or road-to-vehicle communication) (such as the surrounding conditions of the vehicle as seen from other vehicles), information obtained from traffic systems (such as roadside devices, surveillance cameras, and traffic lights), and information obtained from mobile devices carried by pedestrians. Therefore, the IoE information 104 includes not only information obtained from a server via a base station connected to the Internet, but also information obtained via mobile communication (such as 4G, 5G, and WiFi).

[0032] For example, the driver can change his / her posture by tilting his / her head or body to view the blind spot that requires attention. Therefore, it is possible to determine whether the driver has recognized the blind spot that requires attention based on information from the driver monitoring system 102. It is difficult to detect blind spots on the road or blind spots of a vehicle using only the front camera 101. However, it is possible to determine whether the driver has recognized a blind spot on the road or blind spots of a vehicle by combining information from the surrounding situation including the blind spot of the vehicle as seen from other vehicles, which is included in the IoE information 104, and information from monitoring cameras installed at intersections, etc.

[0033] The judgment unit 200 includes a gaze point map 201, a visual saliency map 202, a driver state 203, a control condition 204, a blind spot area risk 205, a blind spot attention assessment 206, a blind spot risk assessment 207, a comprehensive risk assessment 208, and an output calculation unit 209.

[0034] The gaze point map 201 is generated based on the driver's gaze information acquired by the driver monitoring system 102. For example, it is a heat map generated to emphasize the area where the driver is gazing in an image in front of the vehicle (such as an image captured by the front camera 101).

[0035] The visual saliency map 202 is generated based on an image captured by the forward camera 101, etc. A typical saliency map uses image processing to indicate areas in an image or the like that are likely to attract human attention, and such image processing methods are well known. The visual saliency map 202 of this embodiment is characterized by reflecting the driver's blind spots, unlike typical saliency maps. For example, after generating a saliency map based on an image in front of the vehicle using image processing, an area corresponding to the driver's blind spots (blind spot areas) is additionally displayed. Note that if the processing load involved in generating the visual saliency map 202 is large, the calculation function of the processor installed in the on-board computer cannot handle it, so it is expected that the system will be linked to a server outside the vehicle (e.g., a roadside device installed along the road, an edge computer, etc.). In this case, the server can generate a saliency map based on an image in front of the vehicle, and then the on-board computer can generate the visual saliency map 202 by superimposing the driver's blind spot areas.

[0036] The driver state 203 indicates the driver's behavior (such as line of sight, head or face direction, and posture) detected by the driver monitoring system 102. The control condition 204 is set in advance to perform conditional branching of control intervention (such as deceleration or automatic braking) based on the vehicle's surrounding conditions (such as the vehicle information 103 and the IoE information 104) and the driver's behavior (such as the driver state 203). A plurality of conditions (for example, first to third conditions) may be set as the control condition 204. In the first condition, no vehicle control or notification may be performed; in the second condition, no vehicle control may be performed but the driver may be notified that there is a driving risk; and in the third condition, both vehicle control and notification may be performed.

[0037] Humans tend to selectively direct their attention to scenery and objects in their field of vision, and the cognitive resources to which they can direct their attention are limited. Therefore, if they pay too much attention to a specific part of the scenery or object in their field of vision, their awareness of other areas tends to decrease dramatically. In this embodiment, blind spots and risks within blind spots that occur in scenes when driving a vehicle (scenes related to driving conditions, road conditions, etc.) are determined based on information obtained from vehicle-to-vehicle communication and road-to-vehicle communication.

[0038] In the blind spot area risk 205, the collision risk due to the presence of an obstacle or target object in the driver's blind spot is set in advance. In a blind spot requiring attention, the driver can avoid the risk of collision with a target object by consciously directing their gaze. On the other hand, in a blind spot on the road or in a vehicle's blind spot, it is difficult to avoid the risk of collision with a target object with the driver's duty of care alone. For this reason, the driver needs to check the surrounding situation of the vehicle, including the blind spot, using vehicle-to-vehicle communication or road-to-vehicle communication to avoid the risk of collision with a target object. As the blind spot area risk 205, table data listing the risk for each blind spot area may be set, and the risk of the blind spot area may be estimated according to the table data.

[0039] The blind spot attention evaluation 206 evaluates whether the driver is paying attention to the blind spot. Even if the driver pays attention to other vehicles and pedestrians in an intersection environment, they may not be paying attention to objects in the blind spot. The blind spot attention evaluation 206 also changes depending on the vehicle driving situation. For example, a situation may be assumed in which the driver operates the turn signal and depresses the brake pedal to turn at an intersection. Furthermore, if a navigation device is installed in the vehicle, the driver may operate the turn signal or brake if the navigation device constantly instructs the driver to turn right or left. In this way, vehicle behavior can be predicted depending on the driver's operation. By tracking the blind spot, which changes depending on the vehicle behavior, the system evaluates whether the driver is paying attention to the blind spot based on the driver's gaze information and the vehicle's surrounding conditions. The blind spot attention evaluation 206 evaluates the driver's level of attention to the blind spot based on the correspondence between the gaze point map 201 and the visual saliency map 202.

[0040] Blind spot risk assessment 207 evaluates the risk of collision with objects in blind spots that the driver encounters while driving a vehicle. Blind spots for attention can be avoided by the driver consciously directing their gaze. On the other hand, blind spots on the road are difficult for the driver to see in an intersection environment, so it is necessary to obtain information about the surroundings of the vehicle from other vehicles, etc. The same applies to vehicle blind spots. For blind spots for attention and vehicle blind spots, it is necessary not only to combine gaze information with a normal saliency map, but also to obtain surrounding information including vehicle blind spots through vehicle-to-vehicle communication (road-to-vehicle communication). For this reason, blind spots on the road and vehicle blind spots are evaluated as having a higher risk than blind spots for attention.

[0041] The comprehensive risk assessment 208 comprehensively assesses the risk of the driver's vehicle coming into contact with or colliding with a target. Taking into consideration the visual saliency map 202 that takes blind spots into account, the gaze point map 201 is analyzed three-dimensionally to assess the driver's attention level. For example, it evaluates whether the driver is paying insufficient attention to the blind spot area detected in advance, or whether the driver is paying too much attention to a specific area.

[0042] A plurality of thresholds are set for the driver's attention level, and thresholds A, B, and C are set in descending order of attention level. If the driver's attention level is equal to or higher than threshold A and the risk to the vehicle in the blind spot area (risk of collision with an object) is equal to or lower than a certain value (certain level), driving assistance such as notifying the driver of the risk in the blind spot area is not provided. On the other hand, if the risk to the vehicle in the blind spot area is equal to or higher than a certain value (certain level), driving assistance such as notifying the driver of the risk in the blind spot area is provided. The strength of the notification to the driver may be switched depending on the driver's attention level and the level of risk in the blind spot area. Specific driving assistance methods for the driver will be described later.

[0043] The output calculation unit 209 generates a control signal to switch the notification mode according to the driver's attention to the blind spot area and the risk of the blind spot area, and sends it to the output unit 300. The control signal controls the strength (presence or absence) of the notification (control intervention) by the output unit 300. A specific control method (driving assistance method) will be described later.

[0044] The output unit 300 includes a speaker 301, a meter display 302, a HUD (Head Up Display) 303, an actuator 304, and a communication unit 305. The speaker 301 outputs sound in response to a control signal from the determination unit 200 (output calculation unit 209). The sound may be a warning sound or a pre-recorded human voice. The strength of the sound may be switched, or the pitch or volume of the sound may be changed, in response to the control condition 204 or the comprehensive risk assessment 208.

[0045] The meter display 302 displays various warning lights in addition to the vehicle speed meter. In this embodiment, the meter display 302 may turn on a predetermined warning light or display a predetermined message in response to a control signal from the determination unit 200.

[0046] The HUD 303 is a display that can be worn on the driver's head, and can display traffic information, warnings, and the like, superimposed on the image from the forward camera 101. In this embodiment, a warning light or a warning message may be displayed in response to a control signal from the determination unit 200. If the vehicle is equipped with a navigation device, a warning light may be flashed or a warning message may be displayed on the screen.

[0047] The actuator 304 operates an accelerator, a brake, etc. In this embodiment, the actuator 304 may be operated in response to a control signal from the determination unit 200 to decelerate or stop the vehicle. In the above description, the driver is notified of the occurrence of a risk of collision with a target (obstacle, other vehicle, etc.) in response to the driver's attention level to the blind spot. However, in addition to the notification, control intervention (such as deceleration or braking of the vehicle to reduce the risk of collision with a target in response to the driver's attention level to the blind spot) may also be performed.

[0048] The communication unit 305 performs vehicle-to-vehicle communication with other vehicles and road-to-vehicle communication with roadside devices installed on the roadside. The communication unit 305 may be provided with a mobile communication function to communicate with a terminal device carried by a pedestrian. In this embodiment, the driver's attention level is detected taking blind spots into consideration, and the collision risk between the host vehicle and an obstacle (another vehicle, a pedestrian, etc.) is evaluated to determine whether to issue a warning or intervene in the control. In this embodiment, in order to recognize targets and the like present in the blind spots of the host vehicle, the communication function is provided and peripheral information including the host vehicle's blind spots detected by other vehicles, etc., is required.

[0049] In this embodiment, driving assistance such as notification and control intervention is changed depending on the driver's attention to the blind spot and the risk to the vehicle of targets in the blind spot. Therefore, driving assistance is selectively provided at appropriate times, and driving assistance required by the driver in real time can be appropriately provided.

[0050] Although the IoE information 104 is used to determine blind spots and the strength (presence or absence) of notification (control intervention), the amount of IoE information 104 increases as the vehicle accelerates, potentially resulting in an overload of information for the driver. In this embodiment, the blind spot determination and the strength (presence or absence) of notification (control intervention) are determined by combining the driver's attention level based on the driver's line of sight (or the direction of the driver's head, etc.) with visual saliency that takes various blind spots into consideration. In this embodiment, information is presented to the driver (control intervention) when necessary while driving the vehicle, reducing the risk of collision with a target (such as an obstacle, another vehicle, or a pedestrian) due to driver complacency or habituation (e.g., the driver does not notice blind spots and believes they understand the vehicle's surroundings). Furthermore, this embodiment evaluates the driver's (or user's) attention to the target, making it less likely for the driver to feel annoyed even if an alarm sounds inside the vehicle.

[0051] The strength of the notification may be changed depending on the driver's level of attention and the risk of collision with an obstacle. The strength of the notification may be changed by varying the pitch or volume of the audio. For example, if there is a certain risk in the way the driver is paying attention, the driver may be notified that there is a risk of collision due to the presence of obstacles such as people (pedestrians, etc.) or other vehicles in front of or around the vehicle. A certain risk in the way the driver is paying attention means, for example, a risk that the driver does not see the area that should be visible while driving the vehicle or that the driver does not notice blind spots.

[0052] In the above, if an obstacle or the like is located far from the vehicle and the vehicle will not be in danger immediately, a weak warning ("Warning: Weak") is issued in this embodiment. On the other hand, if an obstacle or the like is located close to the vehicle and there is a possibility that the vehicle will soon be in danger, a strong warning ("Warning: Strong") is issued in this embodiment.

[0053] Regarding control intervention, depending on the driver's attention state and the distance between the obstacle and the vehicle, acceleration / deceleration is suppressed, slow deceleration is performed, or steering is used to steer away from the obstacle. Control intervention is performed when the vehicle is not likely to collide with the obstacle immediately. Emergency braking is performed when the vehicle is likely to collide with the obstacle immediately. For emergency braking, vehicles have collision damage mitigation brakes such as FCM (Forward Collision Mitigation). For this reason, in this embodiment, control intervention is performed in advance to avoid activation of FCM.

[0054] The area the driver is gazing at and the blind spot change in real time depending on the vehicle's surroundings (such as the intersection environment) and the vehicle's behavior (such as going straight, turning right, turning left, stopping, parking, etc.). In this embodiment, it is also necessary to consider the collision risk, which changes depending on the vehicle's behavior. The vehicle's traveling direction can be estimated from the turn signal operation status, information from a navigation device, etc. For example, if it is estimated that the vehicle will turn left at an intersection (or a three-way intersection), it is assumed that a pedestrian is present in the blind spot to the right rear of the vehicle. In this case, even if the driver is not behaving in a way that would draw attention to the blind spot to the right rear (behavior that would direct the driver's gaze or posture to the right rear), it may be determined that the collision risk between the vehicle and the pedestrian is low, and notification (control intervention) may be suppressed. On the other hand, if it is estimated that the vehicle will turn right and the driver is not paying attention to the blind spot, a collision risk exists in the blind spot, so it is necessary to notify the driver (control intervention).

[0055] Next, a driving assistance method according to this embodiment will be described with reference to the flowchart in FIG. 2. In conventional technology, for example, when a vehicle approaches an obstacle while traveling and becomes in a sufficiently dangerous state, an alarm is issued and braking control is implemented. In this embodiment, if the distance between the vehicle and the obstacle is relatively large and the driver is not paying attention to the obstacle, the vehicle will promptly notify the driver of the collision risk using audio or images, or will intervene in vehicle control to make the driver aware of the presence of the obstacle. This allows the driver to apply the brakes themselves when the vehicle approaches the obstacle and to be prompted to maintain a safe distance from the obstacle.

[0056] The flowchart in FIG. 2 is made up of processing steps (steps S1 to S5) relating to the blind spot determination and attention level evaluation of the driver, and three processing steps (steps S6 to S9, steps S10 to S13, and steps S14 to S17) according to the driver's attention level. The branching conditions in the flowchart are stored in advance as control conditions 204. It is not necessary to provide processing steps relating to three conditional branches in the flowchart, and processing steps relating to two conditional branches may be provided depending on the strength (presence or absence) of the notification (control intervention). In the following explanation, the terms for the steps will be omitted, and only the reference numerals (S1 to S17) will be provided.

[0057] First, the in-vehicle system 1 (driving assistance device) receives IoE information 104 via vehicle-to-vehicle communication (road-to-vehicle communication) or the like, and detects the risk of a blind spot for the driver (S1). In addition to the IoE information 104, the in-vehicle system 1 may receive video / images captured by a forward camera 101, information related to the driver's line of sight and posture detected by a driver monitoring system 102, and vehicle information 103 (e.g., information indicating the surrounding situation including the blind spot of the vehicle as seen from the other vehicle) received via vehicle-to-vehicle communication or the like from another vehicle. As described above, this embodiment assumes multiple blind spots (road blind spots, caution blind spots, vehicle blind spots), but is not limited thereto. In this embodiment, if an obstacle or the like is present in the blind spot area, the risk of a collision between the vehicle and the obstacle in the blind spot area is detected.

[0058] Next, the driver state 203 (such as the driver's line of sight and posture) is determined based on information from the driver monitoring system 102 (S2). In this embodiment (the determination unit 200 in FIG. 1), an attention point map 201 is generated based on the driver state 203 (S3). The attention point map 201 is a heat map that highlights areas in the driver's field of view (or an image ahead of the vehicle) that the driver is paying attention to. The heat map is created based on the number of times and duration that the driver directs their gaze to a specific area (gaze area) within a predetermined period of time. Road scenes such as intersections, merging points, and pedestrian crossings can be identified, and a heat map based on the driver's line of sight can be created while the FCM is recognizing the intersection.

[0059] Next, a visual saliency map 202 is generated by performing image processing using a predetermined algorithm based on the video and images captured by the forward camera 101 (S4). As described above, the visual saliency map 202 reflects blind spot areas related to a variety of blind spots. In other words, the visual saliency map 202 is generated by superimposing the blind spot areas on areas that are easily visible to the driver. Then, based on the gaze point map 201 and the visual saliency map 202, the driver's blind spot attention level (hereinafter referred to as "attention level") is calculated (S5). This corresponds to the blind spot attention evaluation 206 of the determination unit 200. For example, the driver's attention level is calculated and evaluated based on the degree of overlap between the highlighted area (driver's gaze area) in the attention point map 201 and the blind spot area in the visual saliency map 202. A numerical value selected from a certain numerical range (e.g., integer values ​​from 1 to 10) may be calculated as the attention level. Alternatively, the degree of overlap between the driver's gaze area and the blind spot area may be expressed as a percentage, and a numerical value such as 80%, 60%, etc. may be used as the attention level.

[0060] As described above, in this embodiment, three thresholds (A>B>C) are set for the attention level to distinguish the driver's attention level to the blind spot. However, it is not necessary to set three thresholds for the attention level, and two thresholds may be set to distinguish the strength (presence or absence) of notification (control intervention). If the attention level is equal to or greater than threshold A, it is determined that the driver's attention level to the blind spot is high (S6). If the attention level is less than threshold A and equal to or greater than threshold B, it is determined that the driver's attention level to the blind spot is medium (S10). If the attention level is less than threshold B and equal to or greater than threshold C, it is determined that the driver's attention level to the blind spot is low (S14). If the attention level is less than threshold C, it is determined that the driver is not paying attention to the blind spot.

[0061] In this embodiment, the risk to the vehicle of objects (obstacles, other vehicles, pedestrians, etc.) present in the blind spot is also evaluated in association with the driver's attention level (S7, S11, S15). This corresponds to the blind spot area risk 205 and blind spot risk evaluation 207 of the determination unit 200. The blind spot risk varies depending on the road environment on which the vehicle is traveling. For example, in an intersection environment, when the vehicle is turning left, the driver needs to pay attention to the blind spot of interest (particularly the blind spot on the left side of the vehicle) and the blind spot of the vehicle (particularly the blind spot obstructed by a pillar or the like at the left rear). In addition, the blind spot area risk 205 assumes the presence of an obstacle (pedestrian, bicycle, etc.) to the left rear of the vehicle. Thus, when the vehicle is turning left, attention needs to be paid to the blind spots on the left side and rear of the vehicle.

[0062] When a vehicle makes a right turn at an intersection, the driver must pay attention to the flashing traffic lights. As with turning left, the driver must pay attention to the blind spot (particularly the blind spot on the right side of the vehicle) and the vehicle's blind spot (particularly the blind spot blocked by the right rear pillar, etc.). In addition, the driver must pay attention to the presence of oncoming vehicles and pedestrians on the crosswalk on the right side of the road in the blind spot. If the oncoming vehicle is a large vehicle, there is a possibility that motorcycles, bicycles, etc. are hidden in the shadow of the oncoming vehicle and cannot be seen from the vehicle. In this case, it is also necessary to confirm the surrounding situation of the other vehicle using vehicle-to-vehicle communication between the vehicle and the other vehicle (or video from a surveillance camera at the intersection, etc.). When another vehicle makes a right turn, another vehicle behind the other vehicle may be traveling straight ahead, so the driver must also pay attention to other vehicles that may be in the blind spot behind the other vehicle. When a vehicle makes a right turn in this way, it is necessary to check not only the situation around the vehicle itself, but also the situation around the vehicle as seen by other vehicles, and the situation around other vehicles (especially other vehicles hidden to the side or rear of the other vehicle).

[0063] In this embodiment, a comprehensive risk assessment is performed by taking into account not only obstacles that may exist in multiple blind spots that are difficult for the driver to see from the vehicle, but also the surrounding conditions of other vehicles (comprehensive risk assessment 208 of determination unit 200). In this embodiment, the risk to the vehicle of targets present in blind spots that are difficult for the driver to see is classified into levels. Normally, drivers of four-wheeled vehicles and motorcycles are required to exercise caution in accordance with traffic laws, but pedestrians and cyclists may not be aware of traffic conditions (such as the risk of collision with a vehicle). For this reason, different levels may be set for the risk of targets present in blind spots, such as pedestrians (bicycles) > other vehicles (four-wheeled vehicles, motorcycles). For example, a certain value may be set as a level for distinguishing between pedestrians and other vehicles, and it may be determined whether the risk to the vehicle of targets present in blind spots is equal to or greater than the certain value.

[0064] (First treatment process) When the driver's attention level is equal to or higher than threshold A, if the risk to the vehicle from a target in the blind spot is equal to or lower than a certain value, in this embodiment, no notification (control intervention) is made to the driver (S6 → S7 → S8). In this case, the driver's attention level is the highest and the risk to the vehicle from a target in the blind spot is low, so there is no need to provide driving assistance such as notification (control intervention). On the other hand, if the risk to the vehicle from a target in the blind spot is not equal to or lower than the certain value, the flow proceeds from step S7 to step S9. In other words, if the risk to the vehicle from a target in the blind spot is relatively high, there is no need to perform control intervention, but a weak notification is made to draw the driver's attention (S9).

[0065] (Second treatment process) If the driver's attention level is less than threshold A and greater than or equal to threshold B, the flow proceeds to step S11 (S6 → S10 → S11). In step S11, if the risk to the vehicle from a target in the blind spot is equal to or less than a certain value, in this embodiment, no control intervention is performed, but a weak warning is issued to draw the driver's attention (S12). If the risk to the vehicle from a target in the blind spot is not equal to or less than the certain value, the flow proceeds from step S11 to step S13. In other words, if the risk to the vehicle from a target in the blind spot is relatively high, no control intervention is performed, but a strong warning is issued to draw the driver's considerable attention (S13). In this way, in the second processing step, the strength of the warning to the driver is switched depending on the risk to the vehicle from a target in the blind spot.

[0066] (Third treatment process) If the driver's attention level is less than threshold B and equal to or greater than threshold C, the flow proceeds to step S15 (S6 → S10 → S14 → S15). If the driver's attention level is less than threshold C, that is, if the driver is not paying any attention to the blind spot, the flow proceeds to step S17 (S14 → S17). In this case, in this embodiment, a strong warning is issued to alert the driver to the situation, and control intervention (such as deceleration or braking) is performed.

[0067] In step S15, if the risk to the vehicle from a target in the blind spot is equal to or less than a certain value, in this embodiment, no control intervention is performed, but a strong warning is issued to draw the driver's due attention (S16). On the other hand, if the risk to the vehicle from a target in the blind spot is not equal to or less than the certain value, the flow proceeds from step S15 to step S17. In step S15, if the risk to the vehicle from a target in the blind spot is relatively high, a strong warning is issued to draw the driver's due attention, and control intervention (such as deceleration or braking) is performed (S17). In this way, in the third processing step, a strong warning is issued to the driver regardless of the risk to the vehicle from a target in the blind spot. Note that the path from step S15 to S17 is not essential and may be set arbitrarily.

[0068] As a modification of this embodiment, in addition to a two-dimensional heat map based on the driver's line of sight measured by the driver monitoring system 102, a three-dimensional heat map may be used as the gaze point map 201. In this case, the three-dimensional gaze point map 201 may be linked to a vehicle driving scene and analyzed to detect the driver's individual attention habits (attention tendency). For example, if the driver tends to pay attention in the straight ahead direction and to the right, it can be detected that the driver is distracted to the left. By feeding back the driver's attention tendency to the in-vehicle system 1, it is also possible to provide driving assistance in a timely manner when the driver needs it.

[0069] The attention tendency of not only the driver but also the passenger sitting in the passenger seat may be detected and fed back to the in-vehicle system 1. For example, a blind spot of a vehicle hidden by a pillar of the vehicle body may be visible to the passenger even if it is not visible to the driver. If the passenger is paying attention to a blind spot of the vehicle that is difficult for the driver to see, the in-vehicle system 1 may stop issuing a notification (control intervention) for driving assistance. In this way, the presence or absence of driving assistance may be determined taking into account not only the driver's level of attention but also the passenger's level of attention to the blind spot.

[0070] It is also possible to collect statistics on the degree of attention drivers pay to blind spots and measure the correlations that appear in the statistics. In this case, to collect statistics on the degree of attention drivers pay to blind spots not only for one vehicle but for multiple vehicles, it is expected that statistical processing will be performed not only by combining an on-board computer with vehicle-to-vehicle communication but also by linking with an external server (such as a data center) via road-to-vehicle communication. For example, it is expected that the driver's attention tendencies for multiple blind spots will be accumulated and a certain correlation will be detected in the bias of attention. In this case, regardless of whether there is an inherent risk, such as the presence of an obstacle, in the blind spot area, which tends to be difficult for the driver to see, the on-board system 1 will issue a weak warning to alert the driver.

[0071] Next, technical features and effects of the driving assistance device and driving assistance method according to the present invention will be described. The driving assistance device and driving assistance method level the risk of collision between the vehicle and an obstacle in a blind spot area based on a gaze point map that displays the gaze area of ​​the vehicle driver and a visual saliency map that reflects the driver's blind spot area based on information about the vehicle's surroundings in an image ahead of the vehicle, and switch the level of driving assistance for the driver according to the collision risk for each blind spot area. The level of driving assistance may also be switched by switching the level of notification to the vehicle driver.

[0072] The gaze point map is generated as a heat map that highlights the driver's gaze area based on the driver's state, which indicates the direction of the driver's line of sight and head. The risk level of driving in a blind spot area varies depending on whether the driver is aware of the blind spot area and paying attention to it. Using the gaze point map, it is possible to accurately determine whether the driver is aware of the blind spot area and paying attention to it. In addition, it is possible to detect the risk of collision (including contact risk) with an obstacle (other vehicles, pedestrians, etc.) using vehicle-to-vehicle communication with other vehicles and roadside communication with roadside devices and change the level of notification to the driver. This makes it possible to avoid a situation where the driver is unnecessarily surprised by an unexpected audio or visual notification being output when a collision risk is detected in the vehicle.

[0073] As the gaze point map, for example, a heat map may be generated that quantifies the number of times and duration of viewing of the driver's gaze area within a predetermined period of time. This makes it possible to determine the gaze area to which the driver is particularly paying attention within a predetermined period of time (e.g., several minutes before the vehicle enters an intersection) and the number of gaze areas present in the driver's field of view (e.g., the image ahead of the vehicle). It is known that humans have a limited level of awareness to which they can pay attention, and that if they pay too much attention to a specific area, their awareness of other areas drops dramatically. Therefore, the gaze point map can distinguish between areas where the driver's awareness is declining and areas to which the driver is paying attention. This makes it possible to set an appropriate level of notification to the driver.

[0074] The collision risk level may be changed depending on the range or number of the driver's gaze areas in the gaze point map. If the driver's attention is focused over a wide range or the number of gaze areas is large, it can be determined that the driver's attention is scattered and that the driver is inadvertently distributing his or her attention over a wide range or many areas. In this case, it can be determined that the danger of obstacles in the blind spot area is high for the driver. Even if the number of driver's gaze areas is small, the collision risk in the blind spot direction can be leveled depending on whether the driver is paying attention to the blind spot area.

[0075] The level of driving assistance may be switched between whether or not to intervene in vehicle control. Possible control interventions include acceleration / deceleration control, braking control, steering control, etc. When the level of collision risk in the blind spot area is high, control such as acceleration suppression or deceleration can be intervened to prevent an accident from occurring by preventing the vehicle from colliding with an obstacle.

[0076] When the level of notification is set to two levels, high and low, the blind spot areas may be set as a first blind spot (a blind spot on the road) caused by a surrounding object other than the host vehicle that prevents the detection of an obstacle present around the host vehicle, and a second blind spot (a blind spot of the vehicle) caused by the host vehicle that prevents the driver from seeing it. In this case, the first blind spot and the second blind spot can be detected separately.

[0077] If a vehicle in the oncoming lane stops when the vehicle is entering an intersection, there may be an obstacle that the driver cannot detect because it is blocked by the other vehicle. By determining the collision risk including obstacles in the blind spot that the driver cannot directly see, the warning level can be set more appropriately. The collision risk can be leveled by taking into account the gaze point map depending on whether the driver and the vehicle's system can detect the obstacle. This allows for more detailed warning levels to be set.

[0078] The driver's attention level to the blind spot area may be calculated based on the gaze point map and the visual saliency map. Furthermore, a first threshold and a second threshold smaller than the first threshold may be set for the attention level, and a first notification and a second notification stronger than the first notification may be set as the notification level. For example, when the attention level is equal to or greater than the first threshold, the first notification is issued to the driver when the collision risk is at or above a predetermined level. When the attention level is less than the first threshold and equal to or greater than the second threshold, the second notification is issued to the driver regardless of the collision risk. In this way, the notification level may be switched based on a combination of the driver's attention level and the collision risk. When the driver's attention level is less than the first threshold, the driver can be constantly alerted to the presence of the blind spot area. By weighting the driver's attention level rather than the collision risk in the blind spot area, it becomes easier to take action when an obstacle in the blind spot moves unexpectedly.

[0079] Furthermore, if the driver's attention level is equal to or higher than a first threshold (i.e., the driver's attention level to the blind spot area is high) and the collision risk is below a predetermined level, the first notification is not made. In this case, a certain level of safety is predicted, so by not making a notification (control intervention), the driver can drive the vehicle safely and is freed from the annoyance of unexpected notifications (control intervention).

[0080] The past information of the gaze point map may be statistically processed to detect a tendency for the driver not to pay attention to blind spots in the gaze area, and if this tendency is maintained, the blind spots may be targeted for notification regardless of the risk of collision. The driver's attention can be alerted at an appropriate timing and manner depending on the driver's individual driving tendencies. The driver can also determine the notification tendency and correct their driving habits. Since the past information of the gaze point map is statistically evaluated, the driver's annoyance regarding the appearance of notifications in the future can be reduced. In this case, drivers with individual driving tendencies may be identified on the vehicle side. The level of notification can be switched not only for areas where the driver is not paying attention in a specific direction, such as blind spots, but also by combining the conditions of specific areas at intersections.

[0081] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0082] 1. In-vehicle systems (driving assistance devices) 100 Input section 101 Front camera 102 Driver Monitoring System 103 Vehicle Information 104 IoE Information 200 Judgment section 201 gaze map 202 Visual Saliency Map 203 Driver status 204 Control Conditions 205 Blind Spot Risk 206 Blind Spot Attention Evaluation 207 Blind Spot Risk Assessment 208 Comprehensive Risk Assessment 209 Output calculation unit 300 Output section 301 Speaker 302 Meter Display 303 HUD 304 Actuator 305 Communications Department

Claims

1. an imaging means for capturing an image of the area in front of the vehicle; a detection means for detecting an obstacle present around the vehicle; a monitoring means for monitoring the driver status of the driver of the vehicle; a communication means for communicating with other vehicles outside the vehicle to acquire surrounding information relating to the surrounding conditions including blind spots of the vehicle; a control means for generating a gaze point map that displays the driver's gaze area based on the driver's state, performing predetermined image processing on the image acquired by the imaging means and generating a visual saliency map that reflects the driver's blind spot area based on the surrounding information of the vehicle, leveling the collision risk between the vehicle and the obstacle in the blind spot area based on the gaze point map and the visual saliency map, and switching the degree of driving assistance for the driver in accordance with the level of the collision risk for each blind spot area.

2. the monitoring means monitors the driver state representing the line of sight and the direction of the head of the driver of the vehicle, 2. The driving support device according to claim 1, wherein the control means generates the gaze point map in which the gaze area of ​​the driver is emphasized based on the driver's state.

3. 3. The driving assistance device according to claim 2, wherein the control means generates the gaze point map quantified according to the number of times and duration of the driver's viewing of the gaze area within a predetermined period of time.

4. 3. The driving assistance device according to claim 2, wherein the control means changes the level of the collision risk depending on the range or number of the gaze areas of the driver in the gaze point map.

5. 2. The driving assistance device according to claim 1, wherein the control means switches the level of notification to the driver of the vehicle as the level of the driving assistance.

6. 2. The driving assistance device according to claim 1, wherein the control means switches between intervening in control of the vehicle and not intervening in control of the vehicle as the degree of driving assistance.

7. 2. The driving assistance device according to claim 1, wherein the blind spot areas are set to a first blind spot caused by a surrounding object other than the vehicle that prevents detection of the obstacle present around the vehicle, and a second blind spot caused by the vehicle that prevents the driver from seeing it, and the control means detects the blind spot areas individually for the first blind spot and the second blind spot.

8. the control means calculates the driver's attention level to the blind spot area based on the gaze point map and the visual saliency map, sets a first threshold value and a second threshold value smaller than the first threshold value for the attention level, and sets a first notification and a second notification stronger than the first notification as the level of the notification; When the attention level is equal to or greater than the first threshold, the control means issues the first notification to the driver when the collision risk is equal to or greater than a predetermined risk, The driving assistance device according to claim 5, characterized in that the control means issues the second notification to the driver regardless of the collision risk when the attention level is less than the first threshold and greater than or equal to the second threshold.

9. The driving assistance device according to claim 5, characterized in that past information of the gaze point map is statistically processed to detect a tendency for the driver not to pay attention to the blind spot area in relation to the gaze area by the driver, and when the tendency is maintained, the blind spot area is made the target of the notification regardless of the collision risk.

10. Captures video of the area in front of the vehicle, Detecting obstacles present around the vehicle; monitor the driver status of the driver of the vehicle; communicating with other vehicles present outside the vehicle to acquire surrounding information relating to a surrounding situation including a blind spot of the vehicle; generating a gaze point map that displays the gaze area of ​​the driver based on the driver state; performing predetermined image processing on the video and generating a visual saliency map reflecting a blind spot area of ​​the driver based on the surrounding information of the vehicle; leveling a collision risk between the vehicle and the obstacle in the blind spot area based on the gaze point map and the visual saliency map; A driving assistance method comprising: switching a degree of driving assistance for the driver according to a level of the collision risk for each of the blind spot areas.

Images