Driving assistance device, driving assistance method, and program

Abstract

To provide a driving support device, a driving support method, and a program capable of determining a target present in a direction where a line-of-sight direction of a driver is obstructed and making a notification of it.SOLUTION: A driving support device includes: a line-of-sight direction estimation unit that estimates a line-of-sight direction of a driver of a vehicle based on a rearward image including the driver; a target recognition unit that recognizes a target included in a forward image obtained by imaging a front side of the vehicle; a driver visual confirmation determination unit that determines whether or not the target is a notification object which is an object to make a notification to the driver, and determines a visual confirmation state indicating whether or not the driver visually confirms the notification object based on the light-of-sight direction; and a notification unit that notifies the driver of the presence of the notification object by a notification device when it is determined that the driver is in the visual confirmation state where the driver does not visually confirm the notification object.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This invention relates to a driver assistance device, a driver assistance method, and a program. [Background technology]

[0002] Conventionally, technologies have been disclosed regarding information processing devices that detect the direction of a vehicle driver's gaze and control the vehicle to a safe state if the driver is not looking in the direction they should be looking while driving (see, for example, Patent Document 1). [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] International Publication No. 2020 / 100585 [Overview of the project] [Problems that the invention aims to solve]

[0004] Conventional technologies have not taken into account cases where the driver's line of sight is obstructed by an opaque part of the vehicle body, such as the front pillar (the so-called A-pillar).

[0005] This invention is based on the recognition of the above-mentioned problems and aims to provide a driver assistance device, a driver assistance method, and a program that can determine and notify the driver of an object located in a direction that obstructs the driver's line of sight. [Means for solving the problem]

[0006] The driver assistance device, driver assistance method, and program according to this invention employ the following configuration. (1) The driving support device according to one aspect of the present invention includes a line-of-sight direction estimation unit that estimates the line-of-sight direction of the driver based on a rear image including the driver of the vehicle, an object recognition unit that recognizes an object included in a front image obtained by imaging the front of the vehicle, a driver visual recognition determination unit that determines whether the object is a notification target object that notifies the driver and determines a visual recognition state indicating whether the driver visually recognizes the notification target object based on the line-of-sight direction, and a notification unit that notifies the driver of the presence of the notification target object by a notification device when it is determined that the driver is in a visual recognition state of not visually recognizing the notification target object.

[0007] (2) In the aspect of (1) above, the driver visual recognition determination unit has a visual field obstruction area in which the visual field of the driver is obstructed by a non-transmissive area that is a part of the non-transmissive vehicle body provided in the vehicle, preset within the viewing angle of the front image. When the line-of-sight direction does not face the position where the notification target object exists for a first predetermined time or more, which is a first visual recognition state, or when the position where the notification target object exists is in the direction of the visual field obstruction area and the line-of-sight direction does not face the position where the notification target object exists for a second predetermined time or more, which is longer than the first predetermined time, which is a second visual recognition state, it is determined that the driver does not visually recognize the notification target object.

[0008] (3) In the aspect of (2) above, the visual field obstruction area is preset based on the positional relationship between the line-of-sight direction when the driver faces the direction of the non-transmissive area and the non-transmissive area.

[0009] (4) In the aspect of (3) above, the visual field obstruction area is preset based on the coordinates representing the position of the driver's head and the coordinates representing the center position of the non-transmissive area on a two-dimensional plane at the height of the driver's eyes.

[0010] (5): In the aspect of (3) above, the visual field obstruction area is preset based on the angle between the line of sight reference line representing the line of sight direction when the driver is facing forward and the end of the non-transmissive area on the two-dimensional plane at the eye height of the driver.

[0011] (6): In the aspect of (3) above, the visual field obstruction area is preset based on the width of the non-transmissive area on the two-dimensional plane at the eye height of the driver.

[0012] (7): In any one of the aspects from (2) to (6) above, the visual field obstruction area includes at least the area of the pillar in front of the vehicle where the driver's vision is obstructed in the driving state of driving the vehicle.

[0013] (8): In any one of the aspects from (2) to (7) above, the notification unit notifies the driver of the presence of the notification target object by causing the notification device to emit a sound representing the notification content.

[0014] (9): In any one of the aspects from (2) to (8) above, the notification unit includes a notification image generation unit that generates a notification image representing the notification content, and notifies the driver of the presence of the notification target object by causing the notification device to display the notification image.

[0015] (10): In any one of the aspects from (2) to (9) above, when in the second visual recognition state, the notification unit increases the notification intensity when notifying the driver of the presence of the notification target object compared to when in the first visual recognition state.

[0016] (11): In any one of the aspects from (1) to (10) above, the notification device is the terminal device used by the driver.

[0017] (12): A driving assistance method according to one aspect of the present invention is a driving assistance method in which a computer estimates the direction of the driver's gaze based on a rear image including the driver of a vehicle, recognizes a target object included in a front image taken in front of the vehicle, determines whether or not the target object is a notifiable object that should be notified to the driver, determines a viewing state based on the direction of the gaze indicating whether or not the driver is viewing the notifiable object, and if it is determined that the driver is not viewing the notifiable object, the notification device notifies the driver of the existence of the notifiable object.

[0018] (13): A program according to one aspect of the present invention causes a computer to estimate the direction of the driver's gaze based on a rear image including the driver of a vehicle, to recognize a target included in a front image taken in front of the vehicle, to determine whether the target is a notifiable object that should be notified to the driver, to determine a viewing state based on the direction of the gaze indicating whether the driver is viewing the notifiable object, and, if it is determined that the driver is not viewing the notifiable object, to notify the driver of the existence of the notifiable object via a notification device. [Effects of the Invention]

[0019] According to the embodiments described in (1) to (13) above, it is possible to determine and notify of an object located in a direction that obstructs the driver's line of sight. [Brief explanation of the drawing]

[0020] [Figure 1] This figure shows an example of the configuration and operating environment of a driver assistance device according to an embodiment. [Figure 2] This diagram illustrates an example of a visual field obstruction area that can be set in the visual field obstruction area setting unit. [Figure 3] This diagram shows an example of the relationship between the driver's gaze direction and the field of view obstruction area set in the field of view obstruction area setting unit. [Figure 4] This diagram shows an example of a scenario in which a driver assistance system notifies the driver of the presence of a target. [Figure 5] This flowchart shows an example of the processing flow performed in a driver assistance system. [Figure 6] This figure shows an example of a notification image displayed in a driver assistance system. [Modes for carrying out the invention]

[0021] The embodiments of the driver assistance device, driver assistance method, and program of the present invention will be described below with reference to the drawings. In the following description, the case in which left-hand traffic regulations apply will be described, but if right-hand traffic regulations apply, simply reverse the left and right directions.

[0022] [Configuration of driver assistance systems] Figure 1 shows an example of the configuration and operating environment of a driver assistance device according to an embodiment. The driver assistance device is mounted on a driver monitoring device 1 that is retrofitted to a vehicle (hereinafter referred to as "vehicle V"), such as a drive recorder. The driver monitoring device 1 is mounted on the front windshield, for example, near the rearview mirror at the top of the front windshield. The driver monitoring device 1 includes, for example, a front camera 10, a rear camera 12, and a driver assistance device 20. The driver monitoring device 1 stores images (moving images) captured by the front camera 10 and the rear camera 12 in a storage device (not shown), for example. The driver monitoring device 1 may also be configured to include a display device such as an LCD (Liquid Crystal Display) and display the images (moving images) stored in the storage device on the display device.

[0023] Each of the front camera 10 and rear camera 12 is a digital camera utilizing a solid-state image sensor, such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). The front camera 10 repeatedly (periodically) captures images of the area in front of the vehicle V from the position where the driving monitoring device 1 is mounted. The rear camera 12 repeatedly (periodically) captures images of the area behind the vehicle V from the position where the driving monitoring device 1 is mounted. The horizontal field of view captured by each of the front camera 10 and rear camera 12 is 180°. In other words, the driving monitoring device 1 captures images of a 360° range around the vehicle V from its mounting position.

[0024] The image captured by the front camera 10 (hereinafter referred to as the "front image") includes, as subjects, objects such as other vehicles, pedestrians, bicycles, and fixed objects that are visible in front of the vehicle V through the front windshield and to the left and right (front) of the vehicle V through the left and right side windows, as well as opaque parts of the vehicle body that obstruct the driver DR's visibility of objects while driving. Opaque parts of the vehicle body that obstruct the driver DR's visibility of objects include at least the right front pillar (so-called A-pillar) located between the front windshield and the front of the right side window, and the left front pillar (A-pillar) located between the front windshield and the front of the left side window. Opaque parts of the vehicle body that obstruct the driver DR's visibility of objects may also include side mirrors (so-called door mirrors) installed outside the vehicle in front of the left and right side windows. The front camera 10 also outputs the captured front image to the driver assistance device 20.

[0025] The rear camera 12 captures images (hereinafter referred to as "rear image") that show the interior of the vehicle V, the area behind the vehicle V as seen through the rear window, and objects located to the left and right (rear) of the vehicle V as seen through the left and right side windows. Therefore, the rear image captured by the rear camera 12 also includes, as a subject, the driver DR of the vehicle V to which the driving monitoring device 1 is installed. The rear camera 12 also outputs the captured rear image to the driving assistance device 20.

[0026] Based on the forward image output by the front camera 10 and the rear image output by the rear camera 12, the driver assistance system 20 notifies the driver DR via the terminal device T of the presence of an object that the driver DR has determined is not visible to the driver DR or has a low degree of visibility. At this time, the driver assistance system 20 determines whether there is an object that the driver DR cannot see due to the right front pillar or the left front pillar. If the driver assistance system 20 determines that there is an object that the driver DR cannot see (including cases where the degree of visibility is low), it notifies the driver DR by transmitting information indicating the presence of this object to the terminal device T.

[0027] Terminal device T is a portable terminal device used by the driver DR who drives the vehicle V to which the driving monitoring device 1 is installed, such as a smartphone or tablet. Terminal device T performs procedures to receive driving assistance from the driving assistance device 20 in response to operations by the driver DR. Terminal device T runs, for example, an application for receiving driving assistance from the driving assistance device 20. In response to operations by the driver DR, the application transmits information to the driving assistance device 20 to set areas that cannot be seen by the driver due to the right front pillar or the left front pillar (hereinafter referred to as the "visual field obstruction area"), for example. The application displays images on a display device or emits sounds through a speaker based on the information and notifications transmitted by the driving assistance device 20. Terminal device T is an example of a "notification device".

[0028] The driver assistance device 20 includes, for example, a target recognition unit 22, a gaze direction estimation unit 24, a driver visibility determination unit 26, and a notification unit 28. The driver visibility determination unit 26 includes, for example, a field of view obstruction area setting unit 262. The notification unit 28 includes, for example, a notification image generation unit 282.

[0029] The components of the driver assistance system 20 include, for example, a hardware processor such as a CPU (Central Processing Unit) and a memory device (a memory device with a non-transient storage medium) that stores a program (software), and the functions of each component are realized by the processor executing the program. Some or all of the components of the driver assistance system 20 may be realized by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit), or the functions of each component may be realized by the cooperation of software and hardware. Some or all of the components of the driver assistance system 20 may be realized by a dedicated LSI. Here, the program (software) may be stored in advance in a storage device (a storage device equipped with a non-transient storage medium) provided by the driving assistance device 20, such as ROM (Read Only Memory), RAM (Random Access Memory), or flash memory, or it may be stored in a removable storage medium (a non-transient storage medium) such as a memory card, and installed in the storage device when the storage medium is attached to the driving monitoring device 1. The program (software) may also be downloaded in advance from another computer device via short-range or wide-area communication by an application running on the terminal device T, and then installed in the storage device by being transmitted from the terminal device T.

[0030] The object recognition unit 22 recognizes the lane in which the vehicle V is traveling based on the forward image output by the forward camera 10. Furthermore, the object recognition unit 22 recognizes objects captured in the forward image output by the forward camera 10. Objects recognized by the object recognition unit 22 from the forward image include, for example, other vehicles, pedestrians, bicycles, and fixed objects that are around the vehicle V. Other vehicles include other vehicles traveling in the same lane or adjacent lanes, and oncoming vehicles traveling in the opposite lane. Fixed objects include stationary objects such as traffic lights installed on the road, and stationary objects such as parked vehicles that are moving but currently stationary, such as other vehicles on the lane. Object recognition in the object recognition unit 22 is performed by processing such as pattern matching using pre-set object patterns for recognizing vehicles, pedestrians, bicycles, and fixed objects. Object recognition in the object recognition unit 22 may be performed by processing such as deep learning. The object recognition unit 22 outputs information representing the recognized object (hereinafter referred to as "object information") to the driver visibility determination unit 26. The object recognition unit 22 recognizes the driving lane and objects each time a forward image is output from the forward camera 10 and outputs the object information to the driver visibility determination unit 26. The object information includes, for example, information representing the type of object recognized (e.g., vehicle, pedestrian, bicycle, fixed object, etc.) and information representing the location where the recognized object exists (including its positional relationship with the recognized driving lane). The object information may also include, for example, a forward image output by the forward camera 10 or an image showing the recognized object on the forward image. The object information may also include information indicating whether the object is, for example, a moving object such as another vehicle moving at high speed, or a moving object such as a pedestrian or bicycle moving at low speed.

[0031] The gaze direction estimation unit 24 estimates the driver's gaze direction based on the orientation of the driver's face as seen in the rear image output by the rear camera 12. The gaze direction estimation unit 24 has pre-set (registered) information such as the orientation of the driver's face when they are looking at the left and right front pillars. The setting (registration) of the face orientation in the gaze direction estimation unit 24 may be done, for example, by instructing the terminal device T to look at the left and right front pillars when the driving monitoring device 1 is attached to the vehicle V or before starting the current drive, and saving (memorizing) the rear image at that time as face orientation information. In this case, a marker of a predetermined size (e.g., a sticker) may be attached to the left and right front pillars at the height of the driver's line of sight, and the driver may be instructed to look at the marker, and the rear image at that time may be saved. The gaze direction estimation unit 24 estimates the driver's gaze direction by comparing the pre-set face orientation information with the current face orientation of the driver's face as seen in the rear image. The gaze direction estimation unit 24 outputs information representing the estimated gaze direction of the driver DR (hereinafter referred to as "gaze information") to the driver visibility determination unit 26. Each time a rear image is output from the rear camera 12, the gaze direction estimation unit 24 estimates the gaze direction of the driver DR and outputs the gaze information to the driver visibility determination unit 26.

[0032] The driver visibility determination unit 26 determines, based on the object information output by the object recognition unit 22, whether each object is an object that is present around the vehicle V and is subject to notification to the driver DR (hereinafter referred to as "notification target object"). The determination of whether or not an object is a notification target object in the driver visibility determination unit 26 is made based on whether or not it is present in the driving lane or is expected to enter the driving lane. Notification target objects present in the driving lane are determined, for example, based on the positional relationship between the location of the recognized object and the recognized driving lane, as included in the object information. Notification target objects that are expected to enter the driving lane are determined, for example, based on the change in the positional relationship between the location of the recognized object and the recognized driving lane, as included in the object information.

[0033] Furthermore, the driver visibility determination unit 26 determines whether the driver DR can see the notification object (hereinafter referred to as "visibility state") based on the gaze direction estimation unit 24. The driver visibility determination unit 26 determines the visibility state by determining whether the driver DR's gaze direction included in the gaze information is directed toward the notification object for a predetermined time threshold (hereinafter referred to as "first predetermined time") or longer, which is used to determine whether a person can see an object. Furthermore, if the notification object is located in a field of view obstruction area that is pre-set (registered) in the field of view obstruction area setting unit 262, where the driver DR cannot see the target, the driver visibility determination unit 26 determines the driver DR's visibility state of the notification object by determining whether the driver DR is directed toward the notification object for a predetermined time threshold (hereinafter referred to as "second predetermined time") that is longer than the first predetermined time. A field of view obstruction area is, for example, an area where the driver DR cannot see the notification object due to the left and right front pillars. The field of view obstruction area is set at the corresponding position within the field of view of the forward image captured by the forward camera 10.

[0034] Incidentally, when vehicle V is in motion, the position of the object to be notified and the direction of the driver DR's line of sight are constantly moving. For this reason, the driver visibility determination unit 26 accumulates the time that the driver DR's line of sight is directed towards the object to be notified from the time that object information including the object to be notified is output from the object recognition unit 22. The driver visibility determination unit 26 stores (stores) the accumulated time of the line of sight in a storage device (not shown), for example. The driver visibility determination unit 26 then determines that the driver DR is not seeing the object to be notified if the accumulated time is less than or equal to a first predetermined time. On the other hand, the driver visibility determination unit 26 determines that the driver DR is seeing the object to be notified if the accumulated time is greater than or equal to the first predetermined time. However, if the object to be notified comes into the field of view obstruction area, for example, due to the movement of vehicle V or the movement of the object to be notified itself, the driver visibility determination unit 26 will determine that the driver DR has not seen the object to be notified that is in the field of view obstruction area, if the cumulative time is not equal to or greater than the second predetermined time.

[0035] An example of a "first viewing state" is when the driver's gaze direction is not directed towards the notification object for a first predetermined time or longer (a state in which the driver's gaze direction is determined not to be directed towards the notification object). An example of a "second viewing state" is when the driver's gaze direction is not directed towards the notification object located in the field of view obstruction area for a second predetermined time or longer (a state in which the driver's gaze direction is determined not to be directed towards the notification object located in the field of view obstruction area).

[0036] As described above, the field of view obstruction area setting unit 262 has field of view obstruction areas pre-set (registered). The field of view obstruction area is at least an area where the driver DR cannot see a target due to the left and right front pillars. The field of view obstruction area may also include, for example, an area where the driver DR cannot see a target due to the left and right side mirrors. In the following description, the field of view obstruction area will be assumed to be an area where the driver DR cannot see a target due to the left and right front pillars. The field of view obstruction area may be set (registered) by, for example, the driver DR when the driving monitoring device 1 is attached to the vehicle V or before starting the current drive, by operating the terminal device T on which the application is running. In this case, the driving support device 20 will have the application display the forward image on the display device provided by the terminal device T, and the driver DR will operate the terminal device T to specify that an area is enclosed in the forward image but cannot actually be seen due to the left and right front pillars. The setting (registration) of the field of view obstruction area may be automatically set (registered) by the driver assistance device 20 based on, for example, the positional relationship between the position where the driver monitoring device 1 is installed and the positions of the left and right front pillars shown in the forward image. The left and right front pillars are an example of an "opaque area that is part of the vehicle's opaque body."

[0037] Here, an example of how the driver assistance device 20 automatically pre-sets (registers) field of view obstruction areas in the field of view obstruction area setting unit 262 will be described. Figure 2 is a diagram illustrating an example of a field of view obstruction area to be set in the field of view obstruction area setting unit 262. Figure 2 shows an example of the view seen from the driver DR while driving the vehicle V. As shown in Figure 2, the driver DR cannot see the object that is hidden in the shadow of the right front pillar AP-R and the left front pillar AP-L. For this reason, the driver assistance device 20 pre-sets (registers) in the field of view obstruction area setting unit 262 a field of view obstruction area AO-R of a predetermined width where the field of view is obstructed by the front pillar AP-R at the eye level of the driver DR, and a field of view obstruction area AO-L of a predetermined width where the field of view is obstructed by the front pillar AP-L. Therefore, the driver assistance device 20 uses the center line Lc of the vehicle V in the left-right direction, the center line Lr in the left-right direction of the field of view of the front camera 10, the offset distance Dd between the center line Lc and the center line Lr, and the distance Dp between the left and right front pillars to determine the area in the forward image corresponding to each field of view obstruction area, and sets the determined areas as field of view obstruction area AO-R and field of view obstruction area AO-L in the field of view obstruction area setting unit 262. The distance Dd may be input by the driver DR when the driver monitoring device 1 is attached to the vehicle V by operating the terminal device T on which the application is running, or it may be the standard distance when the range in which the driver monitoring device 1 is attached is predetermined and it is assumed to be attached within that predetermined range. Distance Dp may be input by the driver DR operating the terminal device T, for example, similar to distance Dd, or when the driving monitoring device 1 is installed on the vehicle V, markers of a predetermined size (e.g., stickers) may be attached to the left and right front pillars at the driver DR's line of sight, and the field of view obstruction area setting unit 262 may determine the obstruction area based on the size and position within the field of view of the left and right markers captured in the forward image. Field of view obstruction areas AO-R and AO-L are set at corresponding positions within the field of view of the forward image captured by the front camera 10. The size of field of view obstruction areas AO-R and AO-L may be larger than the size of the area in which the driver DR's field of view is actually obstructed.

[0038] Figure 3 is a diagram illustrating an example of the relationship between the driver DR's line of sight direction and the field of view obstruction area set in the field of view obstruction area setting unit 262. Figure 3 shows an example of the relationship between the driver DR's line of sight direction and the field of view obstruction area in a bird's-eye view. More specifically, Figure 3 shows an example of the positional relationship between the driver DR, front pillar AP-R, front pillar AP-L, and target OB (here, targets OB-1 to OB-3), where the position of the intersection of the y-direction reference line Ly and the center line Lc in the two-dimensional plane (yz plane) of the forward image captured by the front camera 10 is determined as the reference point Or of the driving monitoring device 1. Here, the driver DR is the subject captured in the rear image, and the front pillar AP-R, front pillar AP-L, and target OB-1 to OB-3 are the subjects captured in the front image.

[0039] The field of view obstruction area setting unit 262 represents the positions of the driver DR, front pillar AP-R, and front pillar AP-L in an xy coordinate system with the reference point Or as the origin, thereby determining the starting angle O of the field of view obstruction area AO-R relative to the center line Lr. R1 and end angle O R2 And, the starting angle of the visual field obstruction area AO-L O L1 and end angle O L2 The system then determines the following angles. Based on the determined angles, the field of view obstruction area setting unit 262 sets the field of view obstruction area AO-R and the field of view obstruction area AO-L, which are determined by the left and right front pillars.

[0040] At this time, the field of view obstruction area setting unit 262 first sets the coordinates (x) representing the position of the driver DR's head. d ,y d ) is determined. More specifically, the field of view obstruction area setting unit 262 determines, for example, the central position between the driver DR's eyes, that is, the coordinates (x) representing the driver DR's viewpoint, based on the difference in the positional positions of the front camera 10 and the rear camera 12 in the driving monitoring device 1 and the position of the driver DR's face captured in the rear image output by the rear camera 12. d ,y dFind d , y d ). A line extending parallel to the center line Lr from the coordinates (x

[0041] is set as a reference line (hereinafter referred to as "line of sight reference line") Ls representing the line of sight direction when the driver DR is facing forward. pR ), and the coordinates (0, y pL ) representing the position of the intersection of the reference line Ly and the center position of the front pillar AP-R, and the coordinates (0, y pR ) representing the position of the intersection of the reference line Ly and the center position of the front pillar AP-L are obtained. Here, the y-direction coordinates y at the coordinates (0, y pL ) and the coordinates (0, y pR can be obtained by calculating the yaw angle of the line of sight direction with respect to the coordinate system of the front image from the face orientation of the driver DR shown in the rear image (which may be the rear image obtained by the instruction when setting (registering) the face orientation in the line of sight direction estimation unit 24) when the terminal device T instructs the driver DR to look at the left and right front pillars. The coordinates y pL and the coordinates y pR can also be obtained by the following equations (1) and (2) when the distances Dp and Dd have already been obtained. pL

[0042]

Equation

[0043]

Equation

[0044] In equations (1) and (2) above, it is assumed that the y-coordinates of the center line Lr and the line of sight reference line Ls, which represents the direction of the driver DR's line of sight when facing forward, are the same, that is, the distance between the center line Lr and the line of sight reference line Ls is distance = 0. However, if the y-coordinates of the center line Lr and the line of sight reference line Ls are different, the distance between the center line Lr and the line of sight reference line Ls should be applied to equations (1) and (2) above. More specifically, the distance between the center line Lr and the line of sight reference line Ls should be subtracted from the result of equation (1), and the distance between the center line Lr and the line of sight reference line Ls should be added to the result of equation (2).

[0045] Then, the visual field obstruction area setting unit 262 sets the obtained coordinates (0, y pR ) and coordinates (0, y pL For each of the ) the coordinates (x) of the driver's viewpoint, including the thickness (width) of the corresponding front pillar. d ,y d ) is the area of ​​visual field obstruction AO-R (angle Φ pR ) and the range of the visual field obstruction area AO-L (angle Φ pL ) is calculated. For example, the width of the front pillar is width W p In this case, the field of view obstruction area setting unit 262 sets the angle Φ of the field of view obstruction area AO-R according to the following formula (3). pR The angle Φ of the visual field obstruction region AO-L is determined by the following equation (4). pL We seek.

[0046]

number

[0047]

number

[0048] Here, the width of the front pillar is width W pThis may be a standard value (fixed value) for a typical front pillar, or it may be a value input by the driver DR by operating the terminal device T, similar to distance Dd and distance Dp. Furthermore, the width of the front pillar is width W p is the coordinate y pR or coordinate y pL Similarly, the field of view obstruction area setting unit 262 may determine this by calculating the yaw angle of the line of sight relative to the coordinate system of the front image, using as a reference the direction of the driver's face captured in the rear image (which may be a rear image obtained by the instruction when setting (registering) the direction of the face with respect to coordinate system of the front image.

[0049] Here, for example, on the front pillar AP-R side shown in Figure 3, the angle Φ of the field of view obstruction region AO-R pR The angle θ is the angle between the line of sight reference line Ls and the far end of the front pillar AP-R (the front side of the vehicle V). R1 The angle θ between the line of sight reference line Ls and the front end of the front pillar AP-R (rear side of vehicle V) R2 Therefore, it can be expressed as shown in equation (5) below. Furthermore, the angle θ R1 and angle θ R2 The relationship is the coordinates (x) of the driver's viewpoint (DR). d ,y d ) and the coordinates of the front pillar AP-R (0, y pR ) can be expressed as shown in equation (6) below.

[0050]

number

[0051]

number

[0052] Therefore, the field of view obstruction area setting unit 262 sets the angle θ based on the forward image. R1 Or angle θ R2 We find the angle θ which can be expressed as shown in equation (7) below.R1 The angle θ can be calculated inversely from this, or expressed as shown in equation (8) below. R2 Calculating backwards, the angle Φ of the visual field obstruction region AO-R pR You may also try to find it.

[0053]

number

[0054]

number

[0055] On the front pillar AP-L side, the relationship between equations (5) and (8) above can be expressed similarly by reversing the left and right sides. Therefore, the field of view obstruction area setting unit 262 sets the angle Φ of the field of view obstruction area AO-L based on the same relationship between equations (5) and (8) above. pL It is possible to find this.

[0056] In this way, the field of view obstruction area setting unit 262 has field of view obstruction area AO-R and field of view obstruction area AO-L pre-set (registered). The driver visibility determination unit 26 then determines the visibility status of the notification target object in the driver DR using different predetermined time periods, depending on whether or not the notification target object is located in the field of view obstruction area. Specifically, for notification target objects that are not located in the field of view obstruction area, the driver visibility determination unit 26 determines the visibility status of the notification target object in the driver DR by determining whether or not the cumulative time the gaze direction has been directed toward the object is 1 predetermined time or longer. For notification target objects that are located in the field of view obstruction area, the driver visibility determination unit 26 determines the visibility status of the notification target object in the driver DR by determining whether or not the cumulative time the gaze direction has been directed toward the object is 2 predetermined time or longer. In the example shown in Figure 3, the visibility status of the notification target object in the driver DR is determined for both target OB-1 located in field of view obstruction area AO-R and target OB-2 located in field of view obstruction area AO-L by the 2 predetermined time period. On the other hand, in the example shown in Figure 3, for target OB-3 located in the area between field of view obstruction area AO-R and field of view obstruction area AO-L (the area in front of vehicle V), the visibility status of the notified object in the driver DR is determined by a first predetermined time.

[0057] The driver visibility determination unit 26 outputs information representing the driver DR's visibility status for each notified object (hereinafter referred to as "visibility information") to the notification unit 28. The visibility information includes, for example, information representing the type of notified object (e.g., vehicle, pedestrian, bicycle, fixed object, etc.), information representing the location where the notified object exists (which may also be information representing whether it is located within or outside the field of view obstruction area), and information representing the driver DR's visibility status of the notified object, for notified objects that the driver DR has not been able to see or has low visibility.

[0058] Returning to Figure 1, the notification unit 28 notifies the driver DR of the presence of a notifiable object that the driver DR has not been able to see or has a low degree of visibility, based on the visibility information output by the driver visibility determination unit 26. The notification unit 28 generates information representing the content of the notification to be sent to the driver DR. The notification unit 28 generates sound information that causes the audio output device, such as a speaker, on the terminal device T to pronounce a notification sound or voice message (for example, "There is a pedestrian hidden beyond the left front pillar"), and transmits the generated sound information to the terminal device T to pronounce or speak. As a result, the driver DR can know that there is a target (notifiable object) that needs to be recognized as a risk when continuing to drive the vehicle V.

[0059] The notification image generation unit 282 generates a notification image to inform the driver DR of the presence of a notification target object that the driver DR has not been able to see or has a low degree of visibility, based on the visibility information output by the driver visibility determination unit 26. The notification unit 28 transmits the notification image generated by the notification image generation unit 282 to the terminal device T, which displays it on the display device provided by the terminal device T. This allows the driver DR to visually know that there is a target object (notification target object) that needs to be recognized as a risk when continuing to drive the vehicle V.

[0060] If a notification target object that the driver DR cannot see or has low visibility of is located within the field of view obstruction area, the notification unit 28 will notify the driver DR with a stronger notification intensity (hereinafter referred to as "notification intensity") than when the notification target object is located outside the field of view obstruction area. "Strengthening the notification intensity" means enabling the driver DR to recognize the presence of the notification target object as a higher-risk target. In this case, the notification unit 28 may change the notification method to the driver DR in order to strengthen the notification intensity. For example, when strengthening the notification intensity, the notification unit 28 may raise the pitch of the notification sound that the audio output device of the terminal device T has pronounced than the pitch of the notification sound that is pronounced at the normal notification intensity when the notification target object is located outside the field of view obstruction area. For example, the notification unit 28 may have the notification image generation unit 282 generate a notification image in a color that indicates a higher risk. For example, when strengthening the notification intensity, the notification unit 28 may display the notification image along with the pronunciation or voice output by the audio output device. This allows the driver (DR) to know that there are objects (notification targets) that need to be recognized as posing a higher risk when continuing to drive the vehicle (V).

[0061] In the example shown in Figure 1, the driver assistance device 20 is shown to be equipped with a notification unit 28. However, the notification of the presence of a target (object to be notified) to the driver DR may be performed, for example, by an application running on a terminal device T. In this configuration, the driver visibility determination unit 26 transmits the visibility information to the terminal device T, and the application implements the functions of the notification unit 28 and the notification image generation unit 282. For this reason, the notification unit 28 and the notification image generation unit 282 may be omitted from the driver assistance device 20.

[0062] In this way, the driver monitoring device 1, based on the forward image output by the front camera 10 and the rear image output by the rear camera 12, determines whether or not there are any notifiable objects (targets) that the driver DR cannot see or have a low degree of visibility of. If the driver monitoring device 1 determines that there are notifiable objects that the driver DR cannot see (including cases where the degree of visibility is low), the driver monitoring device 1 notifies the driver DR of the existence of these notifiable objects via the terminal device T. As a result, even if the driver monitoring device 20 is not built into the vehicle V, that is, if it is retrofitted to the vehicle V, the driver monitoring device 1 can determine the visibility status of notifiable objects (targets) to the driver DR and notify the driver DR of the existence of notifiable objects (targets) that may pose a risk to the continued driving of the vehicle V. As a result, the driver DR of a vehicle V to which the driver monitoring device 1 has been retrofitted can perform operations (driving) of the vehicle V corresponding to the notified object or prepare for it in advance, and can continue driving more safely.

[0063] [Examples of situations where the existence of a target object is notified] Figure 4 shows an example of a scenario in which the driver assistance device 20 notifies the driver of the presence of a target (object to be notified). Figure 4 shows an example in a two-way road with one lane in each direction, where pedestrian H1 is walking on the sidewalk on the shoulder side of the lane in which vehicle V is traveling and is about to cross the road, and pedestrian H2 is walking on the sidewalk on the opposite lane side and may be about to cross the road. In this case, the target recognition unit 22 recognizes pedestrian H1 and pedestrian H2 as targets based on the forward image output by the front camera 10 and outputs target information corresponding to pedestrian H1 and pedestrian H2 to the driver visibility determination unit 26. On the other hand, the gaze direction estimation unit 24 estimates the direction of the driver DR's gaze based on the orientation of the driver DR's face shown in the rear image output by the rear camera 12 and outputs gaze information to the driver visibility determination unit 26.

[0064] Here, it is assumed that the driver visibility determination unit 26 has determined that pedestrian H1 and pedestrian H2 are both objects to be notified. The driver visibility determination unit 26 determines that the driver DR is in a state of seeing pedestrian H1 and pedestrian H2 if the cumulative time that the driver DR's gaze direction has been directed toward each of pedestrian H1 and pedestrian H2 is equal to or greater than the first predetermined time. On the other hand, the driver visibility determination unit 26 determines that the driver DR is not in a state of seeing pedestrian H if the cumulative time that the driver DR's gaze direction has been directed toward either one or both of pedestrian H1 and pedestrian H2 is less than the first predetermined time, and outputs the visibility information corresponding to that pedestrian H to the notification unit 28. As a result, the notification unit 28 notifies the driver DR of the presence of pedestrian H represented by the visibility information output by the driver visibility determination unit 26. In this case, the notification unit 28 notifies the driver DR of the presence of pedestrian H at the normal notification intensity when pedestrian H is outside the field of view obstruction area.

[0065] Subsequently, if either or both pedestrians H1 and H2 come into the field of view obstruction area due to the movement of vehicle V or the movement of pedestrians H1 or H2, the driver visibility determination unit 26 determines that the driver DR is in a state of seeing pedestrian H, which is now in the field of view obstruction area, if the cumulative time that the driver DR's gaze direction has been directed toward pedestrian H is 2 or more than the 2 predetermined time. On the other hand, if the cumulative time that the driver DR's gaze direction has been directed toward pedestrian H, which is now in the field of view obstruction area, is less than the 2 predetermined time, the driver visibility determination unit 26 determines that the driver DR is not in a state of seeing pedestrian H, and outputs the visibility information corresponding to pedestrian H to the notification unit 28. The visibility information output by the driver visibility determination unit 26 to the notification unit 28 at this time also includes information indicating that pedestrian H is in the field of view obstruction area. As a result, the notification unit 28 notifies the driver DR of the presence of pedestrian H, as indicated by the visibility information output by the driver visibility determination unit 26, with a stronger notification intensity than usual.

[0066] [Example of processing by driver assistance systems] Figure 5 is a flowchart showing an example of the processing flow performed in the driver assistance device 20. The processing in this flowchart is repeatedly performed at predetermined time intervals when the front camera 10 captures a forward image. In the processing in this flowchart, the rear camera 12 is assumed to capture a rear image at the same time intervals as the front camera 10 captures a forward image. For the sake of simplicity in the following explanation, it will be assumed that only one object is captured in the forward image captured by the front camera 10. For example, in the example scene shown in Figure 4, it will be assumed that only a pedestrian H1 is captured in the forward image. The driver visibility determination unit 26 will then determine the driver DR's visibility status of the pedestrian H1. Here, it will be assumed that a field of view obstruction area has already been set (registered) in the field of view obstruction area setting unit 262 of the driver visibility determination unit 26.

[0067] When the front camera 10 captures a forward image, the object recognition unit 22 recognizes the lane in which the vehicle V is traveling based on the forward image output by the front camera 10 (step S100). Furthermore, the object recognition unit 22 recognizes objects captured in the forward image output by the front camera 10 (step S110). The object recognition unit 22 outputs object information representing the recognized object to the driver visibility determination unit 26.

[0068] The gaze direction estimation unit 24 estimates the driver's gaze direction based on the orientation of the driver's face as seen in the rear image output by the rear camera 12 (step S120). The gaze direction estimation unit 24 outputs gaze information representing the estimated driver's gaze direction to the driver visibility determination unit 26.

[0069] The driver visibility determination unit 26 determines whether or not there is an object to notify the driver DR based on the object recognition information output by the object recognition unit 22 (step S130). If it is determined in step S130 that there is no object to notify (the object shown in the object information is not an object to notify), the driver visibility determination unit 26 returns to step S100.

[0070] On the other hand, if in step S130 it is determined that there is a notification object (that the target shown in the target information is the notification object), the driver visibility determination unit 26 acquires the cumulative time that the driver DR's line of sight is directed toward the notification object (step S140). Subsequently, the driver visibility determination unit 26 determines whether or not the notification object is located within the field of view obstruction area (step S150).

[0071] In step S150, if it is determined that the object to be notified is not within the field of view obstruction area, the driver visibility determination unit 26 checks whether the cumulative time is equal to or greater than the first predetermined time (step S160). In step S160, if it is confirmed that the cumulative time is equal to or greater than the first predetermined time, the driver visibility determination unit 26 determines that the driver DR is in a state of seeing the object to be notified, and returns the process to step S100. On the other hand, in step S160, if it is confirmed that the cumulative time is not equal to or greater than the first predetermined time, the driver visibility determination unit 26 determines that the driver DR is not in a state of seeing the object to be notified. The driver visibility determination unit 26 then outputs visibility information to the notification unit 28 indicating that the object to be notified is outside the field of view obstruction area. As a result, the notification unit 28 notifies the driver DR that the object to be notified (in this case, pedestrian H1) is present with normal notification intensity (step S162).

[0072] In step S150, if it is determined that the object to be notified is located within the field of view obstruction area, the driver visibility determination unit 26 checks whether the cumulative time is equal to or greater than the second predetermined time (step S170). In step S170, if it is confirmed that the cumulative time is equal to or greater than the second predetermined time, the driver visibility determination unit 26 determines that the driver DR is visually viewing the object to be notified located within the field of view obstruction area, and returns the process to step S100. On the other hand, in step S170, if it is confirmed that the cumulative time is not equal to or greater than the second predetermined time, the driver visibility determination unit 26 determines that the driver DR is not visually viewing the object to be notified located within the field of view obstruction area. The driver visibility determination unit 26 then outputs visibility information to the notification unit 28 indicating that the object to be notified is located within the field of view obstruction area. As a result, the notification unit 28 notifies the driver DR of the presence of the object to be notified (in this case, a pedestrian H1) with a notification intensity stronger than the normal notification intensity (step S172).

[0073] Subsequently, the driver visibility determination unit 26 updates the cumulative time that the driver DR's gaze direction was directed toward the notified object (step S180). Then, the driver visibility determination unit 26 finishes processing this flowchart for the forward image captured by the forward camera 10.

[0074] [Example of a notification in a driver assistance system] Figure 6 shows an example of a notification image displayed by the driver assistance device 20. Figure 6 shows an example of a notification image that visually notifies the driver DR of the presence of a notification object by displaying a vehicle icon IM representing vehicle V, a driving lane image LI representing the driving lane of the recognized vehicle V, a target presence direction image SI that simply shows the direction in which the notification object is located in front of the vehicle icon IM, and a notification object icon INO that represents the type of notification object on the display device of the terminal device T.

[0075] In the notification image shown in Figure 6(a), the direction image SI of the object presence in front of the vehicle icon IM is highlighted, and the notification object icon INO representing a pedestrian is shown, notifying the driver DR that there is a pedestrian who may be crossing in front of the vehicle V. In the notification image shown in Figure 6(b), the direction image SI of the object presence to the left of the vehicle icon IM is highlighted, and the notification object icon INO representing a pedestrian is shown, notifying the driver DR that there is a pedestrian to the left in front of the vehicle V, and therefore caution is needed to avoid hitting them when turning left.

[0076] Figures 6(a) and 6(b) show examples of notification images that include the following regions: the rearview mirror region AR, which indicates that the target can be seen by the rearview mirror; the right side mirror region ARR, which indicates that the target can be seen by the right side mirror; and the left side mirror region ARL, which indicates that the target can be seen by the left side mirror. The rearview mirror region AR, the right side mirror region ARR, and the left side mirror region ARL may each represent, for example, a target captured in the rear image taken by the rear camera 12. In this case, for example, the notification image generation unit 282 may generate a notification image by cutting out and superimposing the targets present in the rearview mirror region AR, the right side mirror region ARR, and the left side mirror region ARL from the rear image output by the rear camera 12. For example, the object recognition unit 22 may recognize an object located behind the vehicle V based on the rear image output by the rear camera 12, and the notification image generation unit 282 may generate a notification image in which the recognized object or information representing that object is displayed in the rearview mirror area AR, the right side mirror area ARR, and the left side mirror area ARL, respectively, based on the object information output by the object recognition unit 22.

[0077] As described above, according to the driving assistance device 20 of the embodiment, the field of view obstruction area setting unit 262 pre-sets (registers) field of view obstruction areas at corresponding positions within the field of view of the forward image captured by the front camera 10. The driving assistance device 20 of the embodiment then determines the driver DR's visibility status of a notification target (object) captured in the forward image output by the front camera 10 based on the gaze direction based on the orientation of the driver DR's face captured in the rear image output by the rear camera 12, the cumulative time that the gaze direction has been directed toward the notification target (object), and a predetermined time threshold (first predetermined time or second predetermined time) for determining the visibility status of the notification target. If the driving assistance device 20 of the embodiment determines that there is a notification target that the driver DR cannot see or has a low degree of visibility of, the terminal device T notifies the driver DR of the existence of this notification target. In this case, if the driver assistance device 20 of the embodiment has a visual field obstruction area, it will notify the presence of the object with a stronger notification intensity than the normal notification intensity when the object is outside the visual field obstruction area. As a result, the driver DR of a vehicle V equipped with the driver monitoring device 1 which includes the driver assistance device 20 of the embodiment can learn from the notification from the driver assistance device 20 of the embodiment that there is an object (target) that may pose a risk to the continued driving of the vehicle V. This allows the driver DR of the vehicle V to take action in advance to operate (drive) or prepare for the operation of the vehicle V corresponding to the notified object (target), and to continue driving more safely.

[0078] In the above-described embodiment of the driver assistance device 20, the driver visibility determination unit 26 determines the visibility status based on whether the object to be notified is within or outside the field of view obstruction area, and the predetermined time for this determination is set to either a first predetermined time or a second predetermined time; in other words, the visibility status is determined by two predetermined time stages. However, in the driver assistance device 20, the predetermined time for which the driver visibility determination unit 26 determines the visibility status may be divided into even more stages. For example, the length of the second predetermined time may be changed according to the type of object to be notified, thereby dividing it into multiple stages. In this case, the notification unit 28 may change the method of notifying the driver DR depending on the type of object to be notified.

[0079] In the above-described embodiment of the driver assistance device 20, the case was explained in which the field of view obstruction area setting unit 262 sets (registers) the field of view obstruction area based on the gaze direction estimation unit 24 estimated by the driver DR. However, the same functions as the driver assistance device 20 of the embodiment can also be realized by using an external recognition device, such as a camera that monitors the front of the vehicle V, and a monitoring camera that observes the driver DR, which are installed on the vehicle V in advance. In this case, if the orientation of the driver DR's face as captured by the monitoring camera is associated with the position and thickness (width) of the left and right front pillars, the field of view obstruction area can be set by estimating the position of the driver DR's head using the gaze direction estimation unit 24. In other words, the field of view obstruction area can be set in the field of view obstruction area setting unit 262 without attaching a marker of a predetermined size (e.g., a sticker) to the left and right front pillars at the height of the driver DR's line of sight, or without instructing the driver DR to look at the left and right front pillars. In this case as well, the operation and processing of the driving support device 20 after setting (registering) a visual field obstruction area in the visual field obstruction area setting unit 262 is the same as the operation and processing of the driving support device 20 in the embodiment described above.

[0080] In the above-described embodiment of the driver assistance device 20, the notification unit 28 notifies the driver DR of the presence of an object that the driver DR cannot see (including cases where the degree of visibility is low) via the terminal device T. However, the driver monitoring device 1 may also be configured to include, for example, a sound-producing device such as a buzzer or speaker, or a display device such as an LCD. In this case, the notification unit 28 may notify the driver DR of the presence of an object that the driver DR cannot see (including cases where the degree of visibility is low) via a notification device such as a sound-producing device or display device provided by the driver monitoring device 1, instead of or in addition to the terminal device T. In this case, the operation and processing of the driver assistance device 20 should be equivalent to the operation and processing of the driver assistance device 20 in the above-described embodiment.

[0081] According to the driver assistance device 20 of the embodiment described above, the driver assistance device 20 includes: a gaze direction estimation unit 24 that estimates the gaze direction of the driver DR based on a rear image that includes the driver DR of the vehicle V; an object recognition unit 22 that recognizes objects included in a forward image captured in front of the vehicle V; a driver visibility determination unit 26 that determines whether or not the object is a notification target object that should be notified to the driver DR, and determines a visibility state based on the gaze direction that indicates whether or not the driver DR is seeing the notification target object; and a notification unit 28 that, when it is determined that the driver DR is not seeing the notification target object, notifies the driver DR of the existence of the notification target object via a notification device (in this case, a terminal device T used by the driver DR). As a result, the driver assistance device 20 can determine and notify of notification target objects (objects) that are located in a direction that obstructs the driver DR's gaze direction. This allows the driver DR of vehicle V to perform the operations (driving) and preparations for vehicle V corresponding to the notified target object (object) in advance, enabling safer driving.

[0082] The embodiments described above can be expressed as follows. Hardware processor and It comprises a memory device that stores a program, The hardware processor reads and executes the program stored in the memory device, Based on a rear view image that includes the vehicle driver, the driver's line of sight is estimated. Recognizing a target included in the forward image captured in front of the aforementioned vehicle, Determine whether the aforementioned object is an object subject to notification to the driver, Based on the aforementioned line of sight, a viewing state is determined that indicates whether or not the driver is viewing the object to be notified. If it is determined that the driver is not in a viewing state where they have not seen the object to be notified, the notification device will notify the driver of the existence of the object to be notified. A driver assistance system configured in such a way.

[0083] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of symbols]

[0084] 1. Operation monitoring device 10. Front camera 12. Rear camera 20. Driving assistance devices 22...Target recognition unit 24... Gaze direction estimation unit 26. Driver Visibility Judgment Unit 262...Visual field obstruction area setting section 28... Notification Department 282...Notification Image Generation Unit T...terminal device OB, OB-1, OB-2, OB-3... Target H, H1, H2... Pedestrians (objects, objects subject to notification) V...vehicle AP-R, AP-L... Front pillar DR... Driver

Claims

1. A gaze direction estimation unit estimates the direction of the driver's gaze based on a rear image that includes the driver of the vehicle, A target recognition unit that recognizes targets included in a forward image captured in front of the vehicle, A driver visibility determination unit determines whether the object is a notification target object to be notified to the driver, and determines a visibility state indicating whether the driver is seeing the notification target object based on the direction of the line of sight, If it is determined that the driver is not in a viewing state and has not seen the object to be notified, the notification unit notifies the driver of the existence of the object to be notified by the notification device, Equipped with, The aforementioned driver visibility determination unit is: Within the field of view of the forward image, a field of view obstruction area is pre-defined where the driver's field of view is obstructed by an opaque region that is part of the opaque body of the vehicle. The system determines that the driver has not seen the object in the event of a first viewing state in which the line of sight is not directed toward the location of the object to be notified for a first predetermined time or longer, and a second viewing state in which the location of the object to be notified is in the direction of the field of view obstruction area and the line of sight is not directed toward the location of the object to be notified for a second predetermined time longer than the first predetermined time. Driving assistance system.

2. A gaze direction estimation unit estimates the direction of the driver's gaze based on a rear image that includes the driver of the vehicle, A target recognition unit that recognizes targets included in a forward image captured in front of the vehicle, A driver visibility determination unit determines whether the object is a notification target object to be notified to the driver, and determines a visibility state indicating whether the driver is seeing the notification target object based on the direction of the line of sight, If it is determined that the driver is not in a viewing state and has not seen the object to be notified, the notification unit notifies the driver of the existence of the object to be notified by the notification device, Equipped with, The aforementioned driver visibility determination unit is: Within the field of view of the forward image, a field of view obstruction area is pre-defined where the driver's field of view is obstructed by an opaque region that is part of the opaque body of the vehicle. If the driver is in a first viewing state in which the line of sight is not directed towards the location of the object to be notified for a first predetermined time or longer, or if the driver is in a second viewing state in which the line of sight is not directed towards the location of the object to be notified for a second predetermined time longer than the first predetermined time, then it is determined that the driver has not seen the object to be notified. The notification unit, when in the second visibility state, increases the notification intensity when notifying the driver of the presence of the object to be notified, compared to when in the first visibility state. Driving assistance system.

3. The field of view obstruction area is predetermined based on the positional relationship between the driver's line of sight when the driver is facing the direction of the opaque area and the opaque area. The driving support device according to claim 1 or claim 2.

4. The aforementioned field of view obstruction region is predetermined based on coordinates representing the position of the driver's head and coordinates representing the center of the opaque region on a two-dimensional plane at the driver's eye level. The driving support device according to claim 3.

5. The field of view obstruction area is predetermined based on the angle between the line of sight reference line, which represents the direction of the driver's line of sight when the driver is facing forward, and the edge of the opaque area, on a two-dimensional plane at the driver's eye level. The driving support device according to claim 3.

6. The field of view obstruction area is predetermined based on the width of the opaque area on a two-dimensional plane at the driver's eye level. The driving support device according to claim 3.

7. The aforementioned field of view obstruction region includes, at least, the area of ​​the front pillar of the vehicle in which the driver's field of view is obstructed while the driver is operating the vehicle. A driving support device according to any one of claims 1 to 6.

8. The notification unit notifies the driver of the presence of the object to be notified by having the notification device emit a sound representing the content of the notification. A driving support device according to any one of claims 1 to 7.

9. The aforementioned notification unit, It includes a notification image generation unit that generates a notification image representing the content of the notification, The notification device displays the notification image to notify the driver of the presence of the object to be notified. A driving support device according to any one of claims 1 to 8.

10. The notification unit, when in the second visibility state, increases the notification intensity when notifying the driver of the presence of the object to be notified, compared to when in the first visibility state. A driving support device according to any one of claims 1 to 9.

11. The notification device is a terminal device used by the driver. A driving support device according to any one of claims 1 to 10.

12. Computers Based on a rear view image that includes the vehicle driver, the driver's line of sight is estimated. Recognizing a target included in the forward image captured in front of the aforementioned vehicle, Determine whether the aforementioned object is an object to be notified to the driver, Within the field of view of the forward image, a field of view obstruction area is pre-defined where the driver's field of view is obstructed by an opaque region that is part of the opaque body of the vehicle. When determining the viewing state, which indicates whether the driver is viewing the object to be notified, based on the direction of gaze, if the first viewing state is when the direction of gaze is not directed towards the location where the object to be notified is located for a first predetermined time or longer, and if the second viewing state is when the location where the object to be notified is located is in the direction of the field of view obstruction area and the direction of gaze is not directed towards the location where the object to be notified is located for a second predetermined time longer than the first predetermined time, then it is determined that the driver is not viewing the object to be notified. If it is determined that the driver is not in a viewing state where they have not seen the object to be notified, the notification device will notify the driver of the existence of the object to be notified. Driving assistance methods.

13. Computers Based on a rear view image that includes the vehicle driver, the driver's line of sight is estimated. Recognizing a target included in the forward image captured in front of the aforementioned vehicle, Determine whether the aforementioned object is an object to be notified to the driver, Within the field of view of the forward image, a field of view obstruction area is pre-defined where the driver's field of view is obstructed by an opaque region that is part of the opaque body of the vehicle. When determining the viewing state, which indicates whether the driver is viewing the object to be notified, based on the aforementioned line of sight direction, if the first viewing state is one in which the line of sight direction is not directed towards the location where the object to be notified is located for a first predetermined time or longer, or if the second viewing state is one in which the location where the object to be notified is located is in the direction of the field of view obstruction area and the line of sight is not directed towards the location where the object to be notified is located for a second predetermined time longer than the first predetermined time, then it is determined that the driver is not viewing the object to be notified. When it is determined that the driver is not in a viewing state in which the object to be notified is not visible, the notification device notifies the driver of the presence of the object to be notified, and in the case of the second viewing state, the notification intensity when notifying the driver of the presence of the object to be notified is increased compared to the case of the first viewing state. Driving assistance methods.

14. On the computer, Based on a rear view image that includes the vehicle driver, the direction of the driver's gaze is estimated. The vehicle is made to recognize targets included in the forward image captured in front of the vehicle. The system will determine whether the aforementioned target is an object that should be notified to the driver. Within the field of view of the forward image, a field of view obstruction area is pre-defined where the driver's field of view is obstructed by an opaque region which is part of the opaque body of the vehicle. When determining a viewing state indicating whether the driver is viewing the object to be notified based on the aforementioned line of sight direction, if the first viewing state is when the line of sight direction is not directed towards the location where the object to be notified is located for a first predetermined time or longer, and if the second viewing state is when the location where the object to be notified is located is in the direction of the field of view obstruction area and the line of sight is not directed towards the location where the object to be notified is located for a second predetermined time longer than the first predetermined time, then it is determined that the driver is not viewing the object to be notified. If it is determined that the driver is not in a viewing state where he has seen the object to be notified, the notification device will notify the driver of the existence of the object to be notified. program.

15. On the computer, Based on a rear view image that includes the vehicle driver, the direction of the driver's gaze is estimated. The vehicle is made to recognize targets included in the forward image captured in front of the vehicle. The system will determine whether the aforementioned target is an object that should be notified to the driver. Within the field of view of the forward image, a field of view obstruction area is pre-defined where the driver's field of view is obstructed by an opaque region which is part of the opaque body of the vehicle. When determining a viewing state indicating whether the driver is viewing the object to be notified based on the aforementioned line of sight direction, if the first viewing state is when the line of sight direction is not directed towards the location where the object to be notified is located for a first predetermined time or longer, or if the second viewing state is when the location where the object to be notified is located is in the direction of the field of view obstruction area and the line of sight is not directed towards the location where the object to be notified is located for a second predetermined time longer than the first predetermined time, then it is determined that the driver is not viewing the object to be notified. When it is determined that the driver is not in a viewing state in which the object to be notified is not visible, the notification device notifies the driver of the presence of the object to be notified, and in the case of the second viewing state, the notification intensity when notifying the driver of the presence of the object to be notified is increased compared to the case of the first viewing state. program.

Images