Notification control device, notification control method, and program

Abstract

To more properly notify information about objects in the surrounding.SOLUTION: A control device comprises: a setting unit that sets a movement area that includes a trajectory along which a moving vehicle is expected to move and sets a surrounding area around the moving vehicle, in accordance with the movement of the moving vehicle when the moving vehicle advances straight, turns right, or turns left; and a notification control unit that controls an information provision device that provides information to a driver of the moving vehicle to notify the presence of a first kind of object to the driver when the first kind of object exists in the surrounding area and the driver does not visually recognize the first kind of object, and controls the information provision device to notify again regarding the first kind of object when the driver visually recognizes the first kind of object after said notification but the driver does not visually recognize the first kind of object when the first kind of object exists in the movement area thereafter.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a notification control device, a notification control method, and a program. [Background technology]

[0002] Conventionally, a driving assistance device has been disclosed that is characterized by comprising an object determination means that determines that an object to be avoided exists within a predetermined intersection area including the intersection and its surrounding area, and that the vehicle is approaching the object within the intersection area, and a warning implementation means that issues a warning to encourage the driver to visually check the object when the vehicle is determined to be in the predetermined low-speed state and the object determination means determines that the vehicle is approaching the object within the intersection area (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-87990 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned technologies may not always provide an appropriate notification to the driver. The present invention has been made in consideration of such circumstances, and an object of the present invention is to provide a notification control device, a notification control method, and a program that can more appropriately notify the driver of surrounding objects. [Means for solving the problem]

[0005] The notification control device, notification control method, and program according to the present invention employ the following configuration. (1): A notification control device according to one embodiment of the present invention includes a setting unit that sets a movement area including a trajectory along which a moving body is expected to move in accordance with the behavior of the moving body when the moving body moves straight, turns right, or turns left, and a surrounding area around the moving body; and a notification control unit that, when a first type of object is present in the surrounding area and the driver of the moving body does not visually recognize the first type of object, controls an information providing device that provides information to the driver to notify the driver of the presence of the first type of object, and, after the notification, when the driver visually recognizes the first type of object and then the first type of object is present in the movement area, controls the information providing device to re-notify the driver about the first type of object.

[0006] (2): In the above aspect (1), when a second type of object moving faster than the first type of object is predicted to be present in the surrounding area from another area after a first predetermined time, and the driver does not see the second type of object, the notification control unit controls the information providing device to notify the driver of the presence of the second type of object, and when the driver sees the second type of object after the notification, and then when the second type of object is predicted to be present in the movement area after a second predetermined time from the time the notification was made or the time it was seen, the driver does not see the second type of object, the notification control unit controls the information providing device to re-notify the driver about the second type of object.

[0007] (3): In the above-mentioned aspect (1) or (2), the notification control unit stops the notification if the driver sees the first type of object after the notification, and continues the notification if the driver does not see the first type of object after the notification.

[0008] (4): In the above aspect (4), when the moving body moves straight, the setting unit sets an area including the lane on which the moving body moves as the moving area, sets the length of the moving area based on the speed of the moving body, and the surrounding area includes lanes or sidewalks adjacent to the lane.

[0009] (5): In the above aspect (4), when the moving body turns right, the setting unit sets a moving area wider than the moving area when the moving body moves straight, based on the moving body, sets the length of the moving area based on the speed of the moving body, and sets surrounding areas to the right of the moving area, to the right of the moving body, and behind the right side of the moving body when based on the direction of travel of the moving body.

[0010] (6): In the above aspect (4), when the moving body turns left, the setting unit sets a moving area that is wider than the moving area when the moving body moves straight, based on the moving body, sets the length of the moving area based on the speed of the moving body, and sets surrounding areas to the left of the moving area, to the left of the moving body, and behind the left side of the moving body when based on the direction of travel of the moving body.

[0011] (7): In the aspect of (3) above, a recognition unit is provided that recognizes the object and tracks the recognized object, and the notification control unit assigns identification information to the first type of object recognized by the recognition unit, and when notifying the driver of the presence of the first type of object, assigns notification information indicating that a notification has been issued to the identification information of the first type of object, and generates correspondence information in which the identification information of the first type of object and the notification information are associated, and after the notification, if the driver visually recognizes the first type of object and the first type of object is then present in the movement area, but the driver does not visually recognize the first type of object and notification information has been assigned to the first type of object in the correspondence information, controls the information providing device to re-notify the first type of object.

[0012] (8): In the aspect of (3) above, a recognition unit is provided that recognizes the object and tracks the recognized object, and the notification control unit assigns identification information to a second type of object that is moving faster than the first type of object recognized by the recognition unit, and when the notification control unit notifies the driver of the presence of the second type of object, assigns notification information indicating that a notification has been issued to the identification information of the second type of object, and generates correspondence information in which the identification information of the second type of object and the notification information correspond to each other, and if, after the notification, the driver sees the second type of object and then the second type of object is predicted to move into the movement area, the driver does not see the second type of object and notification information has been assigned to the second type of object in the correspondence information, the information providing device is controlled to re-notify the second type of object.

[0013] (9): A control method according to one embodiment of the present invention is an alarm control method in which a computer sets a movement area including a trajectory along which a moving body is expected to move in accordance with the behavior of the moving body when the moving body moves straight, turns right, or turns left, and a surrounding area around the moving body; if a first type of object is present in the surrounding area and the driver of the moving body does not visually recognize the first type of object, the computer controls an information providing device that provides information to the driver to notify the driver of the presence of the first type of object; if, after the notification, the driver visually recognizes the first type of object, and then, when the first type of object is present in the movement area, the driver does not visually recognize the first type of object, the computer controls the information providing device to provide a second alarm regarding the first type of object.

[0014] (10): A program according to one aspect of the present invention is a program that causes a computer to execute the following processes: a process of setting a movement area including a trajectory along which a moving body is expected to move in accordance with the behavior of the moving body when the moving body moves straight, turns right, or turns left; a process of setting a surrounding area around the moving body; a process of controlling an information providing device that provides information to the driver to notify the driver of the presence of the first type of object when a first type of object is present in the surrounding area and the driver of the moving body does not visually recognize the first type of object; and a process of controlling the information providing device to re-notify the driver about the first type of object when the driver visually recognizes the first type of object after the notification and then the first type of object is present in the movement area but the driver does not visually recognize the first type of object. [Effects of the Invention]

[0015] According to the aspects (1) to (10), it is possible to more appropriately notify the driver of surrounding objects, because, for example, if the driver visually recognizes an object through a notification and the object is still in the moving area, the driver is notified again.

[0016] According to the aspect (2), an appropriate notification is given even to the second type of object that exists at a position distant from the moving body. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a configuration diagram of a vehicle M on which a driving assistance device 100 according to an embodiment is mounted. [Figure 2] FIG. 10 is a diagram for explaining the timing of notification when a risk object is present. [Figure 3] FIG. 10 is a diagram illustrating an example of a risk display. [Figure 4] FIG. 10 is a diagram showing an example of oversight notification. [Figure 5] FIG. 10 is a diagram illustrating an example of an action suggestion notification. [Figure 6] FIG. 1 is a diagram showing an example of the relationship between events, driver actions, and information provided in an HMI. [Figure 7] FIG. 10 is a diagram showing another example of the relationship between events, driver actions, and information provided in the HMI. [Figure 8] 10 is a flowchart showing an example of the flow of processing executed by a notification control unit 120. [Figure 9] FIG. 10 is a diagram for explaining braking control. [Figure 10] FIG. 10 is a diagram illustrating another example of a notification mode. [Figure 11] FIG. 10 is a diagram for explaining a notification area and a re-notification area for a straight road. [Figure 12] FIG. 10 is a diagram for explaining a notification area and a re-notification area of ​​an intersection. [Figure 13] FIG. 10 is a diagram for explaining a notification area and a re-notification area of ​​an intersection. [Figure 14] FIG. 10 is a diagram showing an example of a scene in which a notification is given. [Figure 15] FIG. 10 is a diagram showing an example of a scene in which a re-notification is made. [Figure 16] FIG. 10 is a diagram showing an example of a scene in which an alarm is output. [Figure 17] FIG. 10 is a diagram for explaining an outline of a process related to notification. [Figure 18] FIG. 10 is a diagram illustrating area setting processing. [Figure 19] FIG. 10 is a diagram illustrating a risk prediction process. [Figure 20] FIG. 1 is a diagram for explaining a fast object. [Figure 21] FIG. 10 is a diagram for explaining notification and re-notification filtering processing. [Figure 22] FIG. 10 is a diagram for explaining the process (S144) of selecting an object outside the notification area for notification of oversight. [Figure 23] FIG. 10 is a diagram for explaining the process of selecting an object to be re-notified (S146). [Figure 24] FIG. 10 is a diagram for explaining the inattentive driving determination process. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, an embodiment of a notification control device, a notification control method, and a program according to the present invention will be described with reference to the drawings. In the following, the mobile object will be described as a vehicle, but the mobile object may be an object other than a vehicle.

[0019] First Embodiment [Overall configuration] 1 is a configuration diagram of a vehicle M equipped with a driving assistance device 100 according to an embodiment. The vehicle M may be, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source may be an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination of these. The electric motor operates using power generated by a generator connected to the internal combustion engine, or discharged power from a secondary battery or a fuel cell.

[0020] The vehicle M is equipped with, for example, a camera 10, a radar device 12, a LIDAR (Light Detection and Ranging) 14, an object recognition device 16, an HMI (Human Machine Interface) 30, vehicle sensors 40, an in-vehicle camera 52, a driving operator 80, a driving assistance device 100, a driving force output device 200, a braking device 210, and a steering device 220. These devices and equipment are connected to each other via multiplexed communication lines such as a CAN (Controller Area Network) communication line, serial communication lines, a wireless communication network, etc. Note that the configuration shown in FIG. 1 is merely an example, and some of the configuration may be omitted, or other configurations may be added.

[0021] The camera 10 is, for example, a digital camera that uses a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is attached to any location of a vehicle (hereinafter referred to as vehicle M) in which the vehicle system 1 is installed. The camera 10 includes, for example, a camera that captures images in the front, a camera that captures images in the left and forward direction, a camera that captures images in the right and forward direction, a camera that captures images in the left and rearward direction, and a camera that captures images in the right and rearward direction. For example, the camera 10 that captures images in the front is attached to the top of the front windshield, the back of the rearview mirror, etc. For example, a camera that captures images to the left and forward is mounted on or near the left A-pillar or near the left side mirror, a camera that captures images to the right and forward is mounted on or near the right A-pillar or near the left side mirror, a camera that captures images to the left and rear is mounted on or near the left C-pillar, and a camera that captures images to the right and rear is mounted on or near the right C-pillar. The mounting positions of these cameras may be any positions that allow the camera to capture an image of the target imaging area. Camera 10, for example, periodically and repeatedly captures images of the periphery of vehicle M. Camera 10 may be a stereo camera.

[0022] The radar device 12 emits radio waves such as millimeter waves around the vehicle M and detects radio waves reflected by an object (reflected waves) to detect at least the position (distance and direction) of the object. The radar device 12 is attached to any location on the vehicle M. The radar device 12 may detect the position and speed of an object using an FM-CW (Frequency Modulated Continuous Wave) method. The radar device 12 may be attached, for example, near a position where a camera is attached. For example, the camera 10 and the radar device 12 may be attached as a pair.

[0023] The LIDAR 14 irradiates the surroundings of the vehicle M with light (or electromagnetic waves with wavelengths similar to light) and measures the scattered light. The LIDAR 14 detects the distance to the target based on the time between light emission and light reception. The irradiated light is, for example, pulsed laser light. The LIDAR 14 may be attached to any location on the vehicle M.

[0024] The object recognition device 16 performs sensor fusion processing on the detection results from some or all of the camera 10, the radar device 12, and the LIDAR 14 to recognize the position, type, speed, etc. of the object. The object recognition device 16 outputs the recognition results to the driving assistance device 100. The object recognition device 16 may output the detection results from the camera 10, the radar device 12, and the LIDAR 14 directly to the driving assistance device 100. The object recognition device 16 may be omitted from the vehicle system 1. Some or all of the camera 10, the radar device 12, the LIDAR 14, and the object recognition device 16 are examples of "detection devices."

[0025] The HMI 30 presents various information to the occupants of the vehicle M and accepts input operations by the occupants. The HMI 30 includes various display devices, speakers, buzzers, vibration generators (vibrators), touch panels, switches, keys, etc. The vibration generators, for example, vibrate the seat belt of the driver's seat or vibrate the steering wheel based on the control of the driving assistance device 100. The HMI is an example of an "information providing device" or an "information output device."

[0026] The vehicle sensor 40 includes a vehicle speed sensor that detects the speed of the vehicle M, an acceleration sensor that detects the acceleration, a yaw rate sensor that detects the angular velocity around a vertical axis, a direction sensor that detects the direction of the vehicle M, and the like.

[0027] The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver, a guidance control unit, and a storage unit storing map information. The GNSS receiver identifies the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M may be identified or supplemented by an INS (Inertial Navigation System) that uses the output of the vehicle sensors 40. The guidance control unit, for example, determines a route from the position of the vehicle M identified by the GNSS receiver (or any input position) to a destination input by the occupant by referring to map information, and causes the HMI 30 to output guidance information so that the vehicle M travels along the route. The map information is, for example, information that represents road shapes using links indicating roads and nodes connected by the links. The map information may include road curvature, POI (Point of Interest) information, and the like. The navigation device 50 may transmit the current position and destination of the vehicle M to a navigation server via a communication device and acquire the route from the navigation server.

[0028] The interior camera 52 is, for example, a digital camera using a solid-state imaging element such as a CCD or CMOS. The interior camera 52 is attached in a position where it can capture an image of an occupant of the vehicle M (for example, a driver seated in the driver's seat). The interior camera 52 captures an image of an area to be captured at a predetermined periodic interval, for example.

[0029] Driving operators 80 include, for example, an accelerator pedal, a brake pedal, a steering wheel, a shift lever, and other operators. Driving operators 80 are fitted with sensors that detect the amount of operation or the presence or absence of operation, and the detection results are output to some or all of driving force output device 200, braking device 210, and steering device 220.

[0030] Traveling drive force output device 200 outputs a traveling drive force (torque) to the driving wheels for the vehicle to travel. Traveling drive force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, a transmission, etc., and an ECU (Electronic Control Unit) that controls these. The ECU controls the above components according to information input from driving assistance device 100 or information input from driving operator 80.

[0031] Braking device 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and an ECU. The ECU controls the electric motor according to information input from driving assistance device 100 or information input from driving operator 80, so that a brake torque corresponding to the braking operation is output to each wheel. Braking device 210 may include a backup mechanism that transmits hydraulic pressure generated by operation of a brake pedal included in driving operator 80 to the cylinder via a master cylinder. Note that braking device 210 is not limited to the configuration described above, and may also be an electronically controlled hydraulic brake device that controls an actuator according to information input from driving assistance device 100 to transmit hydraulic pressure from a master cylinder to the cylinder.

[0032] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor applies force to, for example, a rack-and-pinion mechanism to change the direction of the steered wheels. The steering ECU drives the electric motor to change the direction of the steered wheels in accordance with information input from the driving assistance device 100 or information input from the driving operator 80. For example, the steering device 220 or the steering wheel may function as an "information providing device" or "information output device" that notifies or alerts the driver of information. For example, when the steering wheel is rotated or temporarily fixed in a position rotated a predetermined angle to suggest to the user how to avoid a risk object (described later), the steering device or steering wheel functions as an "information providing device" or "information output device."

[0033] [Driving assistance devices] The driving assistance device 100 includes, for example, a recognition unit 110, a notification control unit 120, a first control unit 140, and a second control unit 150. The notification control unit 120 includes, for example, a risk determination unit 122, an orientation determination unit 124, an object recognition determination unit 126, a behavior determination unit 128, a setting unit 130, an information processing unit 132, and an information provision unit 134. These functional units are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Furthermore, some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device having a non-transitory storage medium) such as the HDD or flash memory of the driving assistance device 100, or may be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the driving assistance device 100 by inserting the storage medium (non-transitory storage medium) into a drive device.

[0034] Settings are made within the driving force output device 200, the braking device 210, and the steering device 220 so that instructions from the driving support device 100 to the driving force output device 200, the braking device 210, and the steering device 220 are executed with priority over detection results from the driving operator 80. Regarding braking, if the braking force based on the operation amount of the brake pedal is greater than the instruction from the driving support device 100, the latter may be set to be executed with priority. Furthermore, communication priority on the in-vehicle LAN may be used as a mechanism for executing instructions from the driving support device 100 with priority.

[0035] [Processing when risk objects exist] When a risk object is present ahead of the vehicle M, the driving assistance device 100 executes various processes based on an index value indicating the degree of proximity of the vehicle M to the risk object and a threshold value.

[0036] The "degree of proximity" is expressed by various index values ​​indicating the degree of proximity between objects. For example, the "degree of proximity" is TTC (Time To Collision), which is an index value calculated by dividing the distance by the relative velocity (positive in the direction of approaching each other). If the relative velocity is negative (positive in the direction of moving away from each other), the TTC is temporarily set to infinity. The smaller the TTC value, the higher the "degree of proximity." Satisfying the "first condition" means, for example, that the TTC is less than a first threshold value Th1. The first threshold value Th1 is, for example, a value of about a few tenths of a second. Instead of the TTC, an index value having a similar property, such as time headway (THW), distance, or other index value, may be used as the "degree of proximity." Furthermore, the TTC adjusted to take into account acceleration and jerk may be used as the "degree of proximity." In the following description, the "degree of proximity" is assumed to be the TTC.

[0037] A risk object is, for example, an object that vehicle M must avoid. A risk object is, for example, an object that will interfere with vehicle M if it moves in its current state, or an object that has the potential to interfere. An object that has the potential to interfere is, for example, an object whose degree of proximity will reach a predetermined level or higher after a predetermined time when the object's position, movement direction, and movement speed are taken into account, along with the position, movement method, and movement speed of vehicle M. The object may be a traffic participant such as a pedestrian, bicycle, or vehicle, or may be an object other than a traffic participant, such as an object placed on the road or a fallen object.

[0038] The recognition unit 110 recognizes risk objects around the vehicle M based on the information input by the object recognition device 16. Recognizing means, for example, recognizing the presence of a risk object classified as described above and the relative position of the risk object with respect to the vehicle M. The recognition unit 110, for example, tracks the recognized object.

[0039] The risk determination unit 122 determines the direction in which a risk exists, for example, based on the position of the risk object. For example, when a risk object exists in front of the vehicle M, the risk determination unit 122 determines that a risk exists in the front direction.

[0040] The direction determining unit 124 analyzes, for example, an image captured by the interior camera 52, and determines the direction of the driver's face or the direction of the driver's line of sight based on the analysis result.

[0041] The object recognition determination unit 126 determines whether the driver has visually recognized a risk object. For example, information indicating a correlation between the driver's line of sight in an image (effective field of view in the line of sight direction) and the position of an object relative to the vehicle M is stored in advance in the storage device of the driving assistance device 100. The object recognition determination unit 126 refers to the information indicating the correlation and determines whether the line of sight in the identified image matches the position direction of the risk object. If they match, the object recognition determination unit 126 determines that the driver has visually recognized the risk object. If they do not match, the object recognition determination unit 126 determines that the driver has not visually recognized the risk object. For example, if the driver is looking at a side mirror or to the side (if their line of sight or face is facing the side mirror or to the side), the object recognition determination unit 126 determines that the driver has visually recognized an object that is expected to be reflected in the side mirror or an object located to the side. The object recognition determination unit 126 may also derive a score based on the number of matches or the time period, and if the score is equal to or greater than a threshold, the object recognition determination unit 126 may determine that the driver has visually recognized the risk object. For example, if the driver is looking at the rearview mirror (if the driver's line of sight is directed in the direction of the rearview mirror), it may be determined that the driver has seen an object that is assumed to be reflected in the rearview mirror. Instead of the above process, it may be determined using a predetermined algorithm whether the driver has seen a risk object.

[0042] The behavior determination unit 128 determines the behavior of the driver. Examples of behavior include the operation state of the accelerator pedal, the operation state of the steering wheel, the operation state of the brake pedal, etc. The behavior determination unit 128 determines the behavior of the driver based on the detection results of sensors that detect the amount of operation of the accelerator pedal, the brake pedal, and the steering wheel, or the presence or absence of operation.

[0043] The setting unit 130 sets an area for the moving object M in accordance with the behavior of the moving object M. Details of the area setting will be described later.

[0044] The information processing unit 132 assigns identification information to an object recognized by the recognition unit 110. The information processing unit 132 assigns information indicating whether the object has been visually recognized, information indicating whether a notification has been issued, and information indicating whether a re-notification has been issued to the object's identification information. The information processing unit 132 generates and manages information (an example of "associated information") that associates the object's identification information with information indicating the processing result of the driving assistance device 100 as described above. This information is stored in a storage device of the driving assistance device 100. The information processing unit 132 also predicts the future position of the object based on the position of the object recognized by the recognition unit 110 and changes in the position.

[0045] The information providing unit 134 provides information based on the determination results of the object recognition determination unit 126, the determination results of the behavior determination unit 128, the processing results of the setting unit 130, the processing results of the information management unit 132, etc. For example, a notification or an alarm is output. These will be described in detail later.

[0046] [Timing of notification] FIG. 2 is a diagram for explaining the timing of notification when a risk object is present. When a risk object is present, the information providing unit 134 causes the display unit to display a risk display indicating the direction in which the risk object is present. FIG. 3 is a diagram showing an example of a risk display. For example, as shown in FIG. 3, content is displayed that allows the driver to recognize the direction in which the risk object is present relative to the vehicle M (D1 in the figure). The information providing unit 134 provides the risk display to the driver, for example, until the TTC reaches a second threshold value Th2. The second threshold value Th2 is, for example, a value of about 3 [sec] or around 3 [sec]. The risk display, the oversight notification described below, and the action suggestion notification described below are examples of the "third process."

[0047] The information providing unit 134 issues an oversight notification if the driver does not visually recognize the risk object before the TTC reaches the third threshold Th3. The oversight notification is a notification that notifies the driver that the risk object that should be visually recognized has not been recognized. FIG. 4 is a diagram illustrating an example of the oversight notification. For example, as shown in FIG. 4, content that allows the driver to recognize the direction of the risk object relative to the vehicle M (D2 in the figure) is displayed. For example, the direction of the risk object is notified in a more emphasized manner than the manner illustrated in FIG. 3. The emphasized manner may be, for example, a manner that allows the driver to more easily visually recognize the direction of the risk object, such as by highlighting with color or by highlighting with a flashing or lit icon. The information providing unit 134 provides the oversight notification to the driver, for example, from when the TTC reaches the third threshold Th3 until the TTC reaches the second threshold Th2. For example, the oversight notification continues even if the risk object is visually recognized between the third threshold Th3 and the second threshold Th2. The third threshold value Th3 is, for example, a value of about 4 [sec] or about 4 [sec].

[0048] The information providing unit 134 performs an action suggestion notification (fourth process) if the driver visually recognizes a risk object and does not take a predetermined action before the TTC reaches the third threshold Th3. The action suggestion notification is a notification that suggests to the driver to take an action to avoid the risk object. For example, the information providing unit 134 provides the action suggestion notification to the driver from when the TTC reaches the third threshold Th3 until the TTC reaches the second threshold Th2. For example, the action suggestion notification continues even if an action is taken in response to the action suggestion notification when the TTC is between the third threshold Th3 and the second threshold Th2.

[0049] The predetermined action is, for example, an action corresponding to deceleration or turning. The predetermined action is, for example, the brake pedal being operated or the steering wheel (operator for steering) being operated in a direction to avoid a risk object. The predetermined action is when the operation amount of the accelerator pedal becomes smaller by a predetermined degree or more compared to the operation amount a predetermined time ago, or when the operation of the accelerator pedal is released. Release means that the driver stops operating the accelerator pedal and the operation amount becomes zero or close to zero.

[0050] FIG. 5 is a diagram showing an example of an action suggestion notification. For example, as shown in FIG. 5, content is displayed that allows the driver to recognize the direction (D3 in the figure) in which a risk object exists relative to vehicle M, similar to FIG. 3 (or FIG. 4). As shown in FIG. 5, content is displayed that allows the driver to recognize the direction (D4 in the figure) in which vehicle M should proceed to avoid the risk object. Furthermore, the content includes information indicating the action that the driver should take as an action suggestion notification. In FIG. 5, an icon (A1 in the figure) is displayed that encourages the driver to operate the brake pedal to slow down.

[0051] Instead of displaying the action suggestion notification, the notification may be made by voice, by automatically steering the steering wheel in the direction of travel to allow the driver to recognize the direction of travel (by rotating the steering wheel a predetermined angle in the direction corresponding to the direction of travel and fixing it in a position, or by rotating it a predetermined angle and returning it to its original position), by controlling the pretensioner mechanism of the driver's seat belt to increase the restraining force of the seat belt, or by vibrating the seat belt.The information providing unit 134 provides the action suggestion notification to the driver, for example, from when the TTC reaches the third threshold value Th3 until the TTC reaches the second threshold value Th2.

[0052] Here, the predetermined action is when the amount of accelerator pedal operation decreases by a predetermined amount or more compared to the amount of operation a predetermined time ago, or when the accelerator pedal operation is released. Even if the driver has not performed any action to control deceleration or steering, if the driver recognizes a risk object and wants to avoid the risk object, the driver tends to reduce the amount of accelerator pedal operation or stop operating the accelerator pedal, and then perform an action to avoid the risk object. This tendency was discovered by analyzing the results of various experiments and simulations. In this way, the information providing unit 134 determines whether to provide an action suggestion notification based on whether the predetermined action has been performed, thereby enabling the provision of information that better reflects the driver's intentions.

[0053] In the above example, if the driver visually recognizes a risk object while the oversight notification is being issued, the oversight notification may be stopped even before the TTC reaches the second threshold Th2. In the above example, if the driver takes an action in accordance with the suggestion or a predetermined action while the action suggestion notification is being issued, the action suggestion notification may be stopped even before the TTC reaches the second threshold Th2.

[0054] Furthermore, in the above example, if an event occurs in which a risk object is present when the TTC is between the third threshold Th3 and the second threshold Th2 (for example, when a pedestrian jumps out), the information providing unit 134 issues a notification along with a risk display based on the driver's behavior at that time. For example, if the driver does not visually recognize the risk object, the information providing unit 134 issues an oversight notification, and if the driver visually recognizes the risk object but does not perform a predetermined behavior, it issues an action suggestion notification.

[0055] [Example 1] FIG. 6 is a diagram showing an example of the relationship between events, driver actions, and information provided in the HMI. Assume that vehicle M is traveling straight on a road. At this time, a bicycle is traveling on the left side of vehicle M. At time T, the driver of vehicle M visually recognizes the bicycle and drives vehicle M straight while recognizing the bicycle. In this case, the information providing unit 134 displays a risk display on the display unit indicating the direction of the bicycle relative to vehicle M. Then, at time T+1, it is assumed that a pedestrian appears on the right side of vehicle M attempting to cross the road. At this time, if the driver does not visually recognize the pedestrian, the information providing unit 134 displays a pedestrian oversight notification on the display unit.

[0056] As described above, the information providing unit 134 notifies the driver of appropriate information based on the degree of awareness of the driver's surroundings, allowing the driver to control the vehicle M so as to avoid the risk object with ample time to spare.

[0057] [Example 2] Fig. 7 is a diagram showing another example of the relationship between an event, a driver's action, and information provided in an HMI. The following mainly explains the differences from Fig. 6. Assume that at time T+1, a pedestrian appears from the right side of vehicle M and is about to cross the road. At this time, if the driver visually recognizes the pedestrian but does not perform a predetermined action, the information providing unit 134 issues an action suggestion notification.

[0058] As described above, the information providing unit 134 notifies the driver of appropriate information based on the driver's awareness of the surroundings and the driver's behavior, allowing the driver to control the vehicle M so as to avoid the risk object with ample time to spare.

[0059] [flowchart] 8 is a flowchart showing an example of the flow of processing executed by the notification control unit 120. In this processing, it is assumed that a risk object exists at a predetermined distance from the vehicle M. First, the notification control unit 120 determines whether or not the driver has overlooked the risk object (step S10). If the driver has overlooked the risk object (if the driver has not visually recognized it), the notification control unit 120 issues an oversight notification (step S12).

[0060] If the driver has not overlooked the risk object (if the driver is viewing the object), the notification control unit 120 determines whether the driver is performing a predetermined action (step S14). If the driver is performing a predetermined action, one routine of this process ends. In this case, for example, a risk display indicating the direction of the position of the risk object relative to the vehicle M is displayed on the display unit.

[0061] If the predetermined behavior has not been performed, the notification control unit 120 performs a notification of a behavior suggestion (step S16), which ends one routine of this process.

[0062] As described above, the notification control unit 120 can notify the driver of appropriate information based on the driver's awareness of the surroundings and the driver's behavior. For example, when the driver is performing a predetermined behavior, the notification control unit 120 does not notify the driver of an action suggestion, thereby reducing the annoyance to the driver. In particular, when the driver's accelerator pedal operation amount decreases or the driver stops operating the accelerator pedal, it is estimated that the driver is performing an action to avoid a risk object, and the notification control unit 120 suppresses notification to the driver. As a result, the notification control unit 120 can provide information that reflects the driver's intention.

[0063] In the above example, the oversight notification or the action suggestion notification is displayed, but the oversight notification and the action suggestion notification may be displayed simultaneously. For example, the oversight notification may include a notification of information to decelerate, or the steering wheel may be automatically rotated or vibrated so that the driver can recognize the direction in which to avoid the risk object.

[0064] In addition, one or both of the oversight notification and the action suggestion notification may be omitted. For example, if the driver overlooks a risk object, the oversight notification may not be made. Even if the driver overlooks a risk object, the action suggestion notification may be made.

[0065] [Braking Control] Hereinafter, processing when the TTC becomes less than the second threshold value Th2 and processing when the TTC becomes less than the first threshold value Th1 will be described with reference to Fig. 9. Fig. 9 is a diagram for explaining braking control.

[0066] When the degree of proximity between the risk object and the vehicle M satisfies a predetermined condition (for example, when the TTC is less than the second threshold value Th2), the first control unit 140 performs a first operation to notify the driver of the vehicle M of the presence of the risk object. The first operation is, for example, an operation to instruct the brake device 210 and / or the traveling driving force output device 200 to output a braking force that decelerates the vehicle M at a second deceleration B2 during the period from when the TTC becomes less than the second threshold value Th2 to when the TTC becomes less than the first threshold value Th1. The second deceleration B2 is a deceleration that is smaller (closer to zero) than the first deceleration B1. The second threshold value Th2 is a value greater than the first threshold value Th1. The first operation is so-called CDC control. At this time, a warning (FCW) indicating approaching a risk object ahead may be output. Furthermore, deceleration by braking force may be performed after the warning (FCW) has been issued for a predetermined time.

[0067] If the TTC is less than the first threshold value Th1, the second control unit 150 performs a second operation. The second operation is an operation (collision mitigation braking; CMBS) for controlling the vehicle M to prevent the vehicle M from coming into contact with the risk object. The second control unit 150 instructs the brake device 210 and / or the driving force output device 200 to output a braking force that decelerates the vehicle M at a first deceleration B1. The first deceleration B1 is, for example, a deceleration of approximately 0.5 G (close to 1). As a result, the second control unit 150 quickly decelerates and stops the vehicle M, thereby avoiding contact with the risk object. The ECUs of the brake device 210 and the driving force output device 200 have the function of calculating the brake output, regenerative control amount, engine braking amount, etc. from the instructed deceleration, and the ECUs determine the respective control amounts based on the instructed deceleration and the speed of the vehicle M. This is a well-known technique, so a detailed description will be omitted.

[0068] The second control unit 150 may, for example, determine whether or not an object exists in a lateral area extending from slightly in front of to behind the risk object, and if no object exists, may cause the vehicle M to proceed onto a lane on the side of the risk object to avoid the risk object. For example, if it is difficult to avoid the risk object due to braking force, the second control unit 150 may cause the vehicle M to proceed onto a lane on the side.

[0069] In the above example, it has been described that CDC and CMBS are executed when the TTC is less than the second threshold Th2 or less than the first threshold Th1, but instead of (or in addition to) this, various controls may be executed by activating a traffic sign detection function (TSR), a traffic light detection function (TLI), a function for detecting or issuing an alarm about vehicles approaching from the left or right of the vehicle, an automatic steering avoidance function, an emergency braking system, etc. For example, when vehicle M passes through an intersection, a traffic sign detection function (TSR), a traffic light detection function (TLI), or a function for detecting or issuing an alarm about vehicles approaching from the left or right of the vehicle may be activated, or when vehicle M is approaching a pedestrian, an automatic steering avoidance function may be activated.

[0070] [Notification according to road structure] The notification control unit 120 may omit the vehicle M from issuing an action suggestion notification depending on the road structure. FIG. 10 is a diagram showing another example of a notification mode. For example, when the vehicle M enters an intersection and attempts to turn right or left, and the risk object is a specific type such as a pedestrian, the suggestion notification may be omitted. Furthermore, in this case, the warning (FCW) and CDC control may be omitted, and an oversight notification may be issued until the TTC reaches the first threshold value Th1. This is because the movement of pedestrians at intersections is difficult to predict, and action suggestions cannot be easily made.

[0071] As described above, the notification control unit 120 can notify the driver of appropriate information based on the driver's awareness of the surroundings and the driver's behavior. As a result, the notification control unit 120 can provide information that reflects the driver's intentions.

[0072] Furthermore, the above-described thresholds Th1-3 may be changed depending on the speed of the vehicle M and the performance and specifications of various devices for recognizing risk objects (specifications and functions of the camera 10, radar device 12, LIDAR 14, object recognition device 16, recognition unit 110, etc.). For example, they may be set based on the distance at which various devices can recognize an object.

[0073] [Another example of processing related to oversight notification] An oversight notification and an oversight re-notification may be made as follows: The notification control unit 120 issues a notification when a risk object enters a surrounding area (e.g., areas A and B when traveling straight, as described below, or areas C and D when turning right or left) and the driver does not visually recognize the risk object; and then issues a re-notification when the risk object enters a movement area (e.g., area C when traveling straight, as described below, or area A when turning right or left) and the driver does not visually recognize the risk object that has entered the re-notification area. The notification control unit 120 also issues a notification when a risk object moving faster than the above-mentioned risk object is predicted to be present (or move) in the surrounding area after a first predetermined time and the driver does not visually recognize the risk object; and then issues a re-notification when the risk object is predicted to enter the movement area when the driver visually recognizes the risk object or after a second predetermined time has passed since the notification was issued and the risk object predicted to enter the movement area is predicted to be entered by the driver. In the following description, it is assumed that the notification stops when the driver visually recognizes a risk object after the notification has been issued.

[0074] (Notification area and re-notification area on straight roads) 11 is a diagram for explaining the notification area and re-notification area for a straight road. It is assumed that a vehicle M is traveling in a direction D within a lane LA on a road including the lane LA. The direction D indicates the traveling direction of the vehicle M based on the center of gravity P of the vehicle M.

[0075] Area C is an area that encompasses the lane LA on which vehicle M is traveling, and is a region that connects a straight line that extends a first predetermined distance in the direction of travel from the center of gravity P in directions at predetermined angles to the left and right of direction D, a straight line that extends a second predetermined distance in the direction of travel from the tip of the straight line parallel to direction D, and the tips of the two straight lines that extend the second predetermined distance in the direction of travel. The length of area C in the direction of travel is the distance vehicle M travels in a predetermined time. The predetermined time is a time within a range of 4 to 5 seconds, for example, 4.5 seconds. However, the length is equal to or greater than the set minimum distance and less than the set maximum distance. Area C is a re-notification area.

[0076] Area A is an area that encompasses Area C, and is an area that connects a straight line that extends from the center of gravity P toward the direction of travel a third predetermined distance in directions at predetermined angles to the left and right of direction D, a straight line that extends from the tip of the line a fourth predetermined distance toward the direction of travel parallel to the direction of travel D, and the tips of two straight lines that extend the fourth predetermined distance parallel to the direction of travel D.

[0077] Area B is an area adjacent to Area A and Area C. Area B is an area connecting a straight line extending a predetermined distance in the direction of travel from the center of gravity P in directions at predetermined angles to the left and right of direction D, a straight line extending from the tip of the straight line parallel to direction D a fifth predetermined distance in the opposite direction to the direction of travel, and the tips of the two straight lines extending a fifth predetermined distance parallel to direction D. Areas A, B, and C are notification areas.

[0078] (Notification area and re-notification area at intersections) 12 is a diagram for explaining the notification area and re-notification area of ​​an intersection. It is assumed that a vehicle M is about to turn left at an intersection.

[0079] Area A is an area connecting a straight line extending from the center of gravity P as the starting point in the direction D at a predetermined angle (α) to the left and right in the direction of travel for a sixth predetermined distance, a straight line extending from the tip of the straight line parallel to direction D for a seventh predetermined distance in the direction of travel, and the tips of the two straight lines extending parallel to direction D for the seventh predetermined distance. The length L1 of Area A in the direction of travel is the distance that vehicle M travels in a predetermined time. The predetermined time is a time within a range of 4 to 5 seconds, for example, 4.5 seconds. However, the length is equal to or greater than the set minimum distance and less than the set maximum distance. Area A is a re-notification area.

[0080] Area D is an area adjacent to Area A. Area D is formed by a straight line extending from center of gravity P at a predetermined angle to the left of direction D for an eighth predetermined distance in the direction of travel, a straight line extending from the tip of the straight line for a ninth predetermined distance in the direction of travel parallel to direction D, a straight line connecting the tip of this straight line with the tip of the straight line for the seventh predetermined distance in Area A, and the straight line for the seventh predetermined distance.

[0081] Area C is, for example, a rectangular area set to the left of vehicle M. Area C is an area on the left side of the vehicle that includes the front and rear of the vehicle. The lateral length L2 of area C is, for example, within a range of 15 m to 20 m. Areas A, C, and D are notification areas.

[0082] 13 is a diagram for explaining the notification area and re-notification area of ​​an intersection. Assume that vehicle M is about to turn right at the intersection. The notification area includes areas A, C, and D. Areas A, C, and D when vehicle M turns right are the inverse of areas A, C, and D when vehicle M turns left, so detailed explanation will be omitted.

[0083] In the above example, the driving assistance device 100 was described as setting a predetermined area for the moving body M as the notification area and re-notification area, but instead (or in addition) the notification area or re-notification area may be set based on the edges of the road, lanes, road markings, structural parts, etc.

[0084] [Outline of notification processing] The following describes the process of issuing a notification when a risk object is present in the notification area or re-notification area, and the process of issuing a notification (or warning) related to the inattentive driving determination process.

[0085] For example, as shown in Fig. 14, if a risk object OB is present in area C at time T and the driver does not visually recognize this risk object, an alert is issued, and if the driver visually recognizes the risk object after the alert, the alert stops. Thereafter, if the risk object is present in area A at time T+1 as shown in Fig. 15, a re-alert is issued. In this way, if the risk object visually recognized by the driver due to the alert is found to be present in the re-alert area (area A), a re-alert is issued, so that the driver can be more appropriately notified about surrounding objects.

[0086] Furthermore, the notification control unit 120 causes the HMI 30 information output device to output a first warning when a first condition is satisfied. The first condition is that the driver's line of sight is outside a predetermined range in front of the moving object M (for example, outside a range corresponding to the front windshield), and the state in which the line of sight is outside the range continues for a first time (for example, 3 seconds). The notification control unit 120 causes the HMI 30 to output a second warning when a second condition is satisfied. The second condition is that the driver's line of sight is outside the above range, a risk object that should be watched is present in a reference area (re-notification area) set based on the moving object M as shown in FIG. 16, and the state in which the line of sight is outside the above range continues for a second time (for example, 1 second) that is shorter than the first time.

[0087] The first condition may be that the driver's face is facing out of a predetermined range for viewing the area ahead of the moving body M, and that the state in which the face is facing out of the range continues for a first time. The second condition may be that the driver's face is facing out of the range, a risk object that should be watched is present in a reference area set based on the moving body M, and that the state in which the face is facing out of the range continues for a second time that is shorter than the first time.

[0088] FIG. 17 is a diagram for explaining an outline of the notification process. (1. Mode switching) The notification control unit 120 sets a right turn mode, a left turn mode, or a straight ahead mode based on the state of the turn indicators of the vehicle M and the operation angle of the steering wheel by the driver. For example, if the turn indicators are not flashing, the straight ahead mode is selected, and if the turn indicators are flashing to indicate that the vehicle M is moving left, the left turn mode is selected. Also, if the steering wheel is operated to the right and the operation angle is equal to or greater than a threshold, the right turn mode is selected, and if the steering wheel is operated to the left and the operation angle is equal to or greater than a threshold, the left turn mode is selected, and the straight ahead mode is selected in all other cases. Note that if the flashing status of the turn indicators and the operation status of the steering wheel contradict each other, the steering wheel operation takes priority and the mode is determined.

[0089] (2. Area Settings) The notification control unit 120 sets an area according to the mode determined in (1. Mode switching) above, thereby setting a notification area and a re-notification area.

[0090] (3. Risk Prediction) The notification control unit 120 acquires the current position and predicted position of the object based on the detection result of the object (ID, position, speed, type of the object). The type of object may be a four-wheeled vehicle, a two-wheeled vehicle, a bicycle, a pedestrian, etc.

[0091] (4. Gaze Object Matching) The notification control unit 120 determines whether the driver's line of sight matches the direction in which the object exists.

[0092] (5. Notification and re-notification filtering processing) The notification control unit 120 performs notification and re-notification filtering processing based on the result of gaze object matching, which determines whether the gaze is viewing a risk object. The notification control unit 120 selects a risk object for which an oversight notification or an oversight re-notification is to be performed.

[0093] (6. Inattentiveness detection processing) The notification control unit 120 determines whether the driver is looking aside based on the driver's line of sight or the direction of the driver's face.

[0094] "(2. Area Settings) Details" 18 is a diagram for explaining the area setting process. In (1. Mode switching), when the right turn mode is selected, the notification control unit 120 sets a notification area and a re-notification area for a right turn (step S100). The notification control unit 120 sets area A as the re-notification area (step S102), and sets areas A, C, and D as notification areas (step S104).

[0095] In (1. Mode switching), if the left turn mode is selected, the notification control unit 120 sets a notification area and a re-notification area for a left turn (step S110). The notification control unit 120 sets area A as the re-notification area (step S112), and sets areas A, C, and D as notification areas (step S114).

[0096] In (1. Mode switching), if the mode is neither right turn mode nor left turn mode (other cases), the notification control unit 120 sets a notification area and a re-notification area for straight driving (step S120). The notification control unit 120 sets area C as the re-notification area (step S122), and sets areas A, B, and C as notification areas (step S124).

[0097] "Details of (3. Risk Prediction)" 19 is a diagram for explaining the risk prediction process. Based on the object detection result, the notification control unit 120 determines whether the object is present in a predetermined area and is moving at a predetermined speed or faster (step S130). The predetermined area includes the notification area and the re-notification area, and is an area larger than the notification area and the re-notification area. The predetermined area is an area that is set in advance based on the vehicle M. The predetermined area may be an area that differs depending on the mode of the vehicle M.

[0098] If the object is present in a predetermined area and is moving at a predetermined speed or faster, the notification control unit 120 predicts the position of the object at a predetermined time in the future (step S132). For example, the notification control unit 120 predicts the position of the object (a fast-moving object) at multiple future points in time based on the object's speed. This allows an object approaching vehicle M from a distance away from vehicle M to be recognized, as shown in FIG. 20.

[0099] "Details of (5. Notification and re-notification filtering process)" FIG. 21 is a diagram for explaining the notification and re-notification filtering process. The notification control unit 120 acquires objects that are the subject of oversight notification (objects that the driver has not visually recognized) and objects that are the subject of oversight re-notification (objects that the driver has not visually recognized) by referring to the object position, information indicating whether the object is visually recognized by the driver, the notification area, and the re-notification area (step S140). An object that has not been visually recognized refers to an object that has entered the re-notification area after an oversight notification has been issued and has not been visually recognized by the driver. The notification control unit 120 also has a forgetting function. For example, the forgetting function may be such that, when a predetermined time has passed since a risk object that has entered the notification area has left the notification area, information associated with the risk object (for example, information indicating that an oversight notification has been issued or information indicating that an oversight re-notification has been issued) may be forgotten. The same process is performed when an object that has entered the re-notification area leaves the notification area.

[0100] The notification control unit 120 selects an object in the notification area to be notified of if it has been overlooked, by referring to the information obtained in step S140 and information indicating the current position and predicted position of the object in the notification area (step S142). For example, if an object exists in the notification area, or if the predicted position of an object existing in the notification area has existed in the notification area for a predetermined period of time or longer, the object is selected as a notification target. As a result, an object that exists in the notification area but is not visually recognized is selected.

[0101] The notification control unit 120 selects an object outside the notification area for oversight notification by referring to the information obtained in step S140 and the information indicating the current position and predicted position of the object (step S144). As a result, an object that exists outside the oversight area but is predicted to enter the notification area in the near future and is not visually recognized (for example, a fast-moving object) is selected.

[0102] The notification control unit 120 selects objects in the re-notification area and objects predicted to enter the re-notification area by referring to the information obtained in step S140 and the information indicating the current position and predicted position of the object (step S146). As a result, objects that are present in the oversight re-notification area or predicted to be present in the near future and have not been re-seen are selected.

[0103] As described above, an object to be notified of oversight and an object to be notified of oversight again are selected.

[0104] [Details of the process of selecting an object outside the notification area for oversight notification (S144)] 22 is a diagram for explaining the process of selecting an object outside the notification area for missed notification (S144). The notification control unit 120 acquires information on the current position and predicted position of a fast-moving object and information on the notification area, and determines whether the current position or predicted position of the target object has been present within the notification area for a predetermined number of consecutive frames or more (a predetermined period of time or more) (step S150).

[0105] If the current position or predicted position of the target object is not within the notification area for a predetermined number of consecutive frames or more, the notification control unit 120 lists the object together with the judgment result as an object outside the overlooked area and stores it in the storage device (step S152).

[0106] If the current position or predicted position of the target object is within the notification area for a predetermined number of consecutive frames or more, the notification control unit 120 lists the object together with the judgment result as an object outside the overlooked area and stores it in the storage device (step S154).

[0107] The notification control unit 120 refers to the storage of oversight notification and re-notification for each object and the determination result of step S150, and determines whether or not oversight notification and re-notification have not been made for the object and the determination result of step S150 is affirmative "YES" (step S156). If this condition is met, the notification control unit 120 issues an oversight notification for the object. If this condition is not met, the notification control unit 120 lists the object together with the determination result as an object outside the oversight area and stores them in the storage device (step S158). For example, the object is listed as an object visually recognized by the driver.

[0108] As described above, even when a fast object outside the notification area is predicted to enter the notification area, an appropriate notification is given to the driver.

[0109] [Details of the process of selecting an object to be re-announced (S146)] 23 is a diagram for explaining the process of selecting an object to be re-notified (S146). The notification control unit 120 acquires information on each object that has been overlooked and notified (information on the object that exists in the notification area and has been overlooked and notified), information on the current position, and information indicating whether the driver has visually recognized the object, and by referring to this information, determines whether the driver has visually recognized (made eye contact with) the object that has been overlooked and notified again (step S160).

[0110] If the driver does not see the object again, the notification control unit 120 refers to the information on the history of oversight re-notification for each object and determines whether or not an oversight re-notification has been issued for the object (step S162).If an oversight re-notification has not been issued, the notification control unit 120 refers to the information on the re-notification area and determines whether or not the current position of the object has been included in the re-notification area for a predetermined number of consecutive frames or more (step S164).

[0111] If the current position of the object is included in the re-notification area for a predetermined number of consecutive frames or more, the notification control unit 120 performs a re-notification of the object being overlooked (step S164). If the current position of the object is not included in the re-notification area for a predetermined number of consecutive frames or more, the notification control unit 120 visually recognizes the object after the oversight notification, and lists the object as an object for which no oversight re-notification has been performed and which does not exist in the re-notification area, and stores the list in the storage device.

[0112] In the above example, if an oversight notification is not issued and an object is found to be present in the re-notification area, an oversight notification is issued. Also, in the above example, for example, an alert is issued when an object is present in the re-notification area of ​​the notification area, and thereafter, an oversight re-notification may not be issued for the object that continues to be present in the re-notification area, or an oversight re-notification may be issued. Also, in the above example, if an oversight notification is issued for a risk object in the notification area and the risk object continues to be present in the notification area for a predetermined period of time after the driver visually recognizes the risk object, an alert may not be issued, or an alert may be issued again after a preset period of time has elapsed.

[0113] For example, if an object remains outside the re-alert area for a predetermined period of time after an oversight notification has been issued, the information indicating that an oversight notification has been issued for the object is forgotten. In this case, if the object re-enters the re-alert area, the object will become the target of an oversight notification. For example, if an object remains outside the notification area for a predetermined period of time after an oversight notification has been issued, the information indicating that an oversight notification has been issued for the object is forgotten. In this case, if the object re-enters the notification area, the object will become the target of an oversight notification.

[0114] [Details of (6. Inattentiveness Detection Processing)] 24 is a diagram for explaining the inattentive driving determination process. The notification control unit 120 determines whether or not the driver's line of sight has been detected based on an image captured by the driver (step S170). If the line of sight has been detected in step S170, a determination is made using the line of sight direction, or a determination is made using the line of sight direction and the face direction. If the line of sight cannot be detected in step S170, a determination is made using the face direction.

[0115] The notification control unit 120 determines whether the face is facing forward based on the image (step S172). For example, the notification control unit 120 determines whether the yaw angle and pitch angle of the face orientation (reference direction) are within their respective angle ranges.

[0116] If the notification control unit 120 determines that the face is facing forward, it sets a flag indicating that the face is facing forward (step S174). If the notification control unit 120 determines that the face is not facing forward, it determines whether a predetermined time has passed since the determination (step S176). The predetermined time is, for example, 3 seconds.

[0117] The notification control unit 120 determines whether the line of sight is directed forward based on the image (step S178). For example, the notification control unit 120 determines whether the yaw angle and pitch angle of the line of sight direction (reference direction) are within their respective angle ranges.

[0118] If the notification control unit 120 determines that the line of sight is directed forward, it sets a flag indicating that the line of sight is directed forward (step S180). If the notification control unit 120 determines that the line of sight is not directed forward, it determines whether a predetermined time has elapsed since the determination (step S182). The predetermined time is, for example, three seconds.

[0119] The notification control unit 120 determines whether an object is present in a predetermined area and whether the object is in (1) or (2) by referring to the object detection result and the mode (step S184). The predetermined area is an area corresponding to each of the right turn mode, left turn mode, and straight ahead mode, and is an area that the driver should be aware of. For example, the predetermined area may be the same as the re-notification area, or may be an area that includes the re-notification area. The above (1) means that a negative determination result is obtained in step S176, and the above (2) means that a negative determination result is obtained in step S182.

[0120] If a positive determination result is obtained in step S184, the notification control unit 120 determines whether this state has continued for a predetermined time (step S186). If a positive determination result is obtained in step S184, the notification control unit 120 sets a flag indicating that an object is present in the predetermined area. The predetermined time in step S186 is shorter (for example, 1 second) than the predetermined time in step S176 or step S182.

[0121] If a positive determination result is obtained in step S186, if a positive determination result is obtained in step S176, or if a positive determination result is obtained in step S182, the notification control unit 120 determines whether or not the reset flag is set and the inattentive flag is not set (step S190). If the reset flag is set and the inattentive flag is not set, the notification control unit 120 determines whether or not the inattentive flag is set and the inattentive flag was not set in the previous step (step S192). If this condition is met, the notification control unit 120 sets the inattentive flag. If the inattentive flag is set, an inattentive notification is issued.

[0122] If the above conditions are not met, the notification control unit 120 sets a reset flag when a predetermined time has elapsed since the inattentiveness flag was set (step S194). As a result, even if the conditions for issuing an inattentiveness notification are met again after the inattentiveness notification has been issued (when the driver does not visually recognize the direction of the object), notification is suppressed until a predetermined time (e.g., 3 seconds) has elapsed since the inattentiveness notification. The predetermined time may be equal to or longer than the predetermined time of step S186.

[0123] For example, after the inattentive driving warning is issued, the driver may visually check the surroundings of the vehicle M in a direction different from the direction of the object's location and try to recognize the space to avoid the object. Therefore, as described above, the inattentive driving warning is suppressed, thereby reducing the annoyance felt by the driver.

[0124] As described above, the notification control unit 120 can provide an appropriate notification or warning against the driver's inattentive driving.

[0125] According to the embodiment described above, the notification control unit 120 can more appropriately notify the driver of a nearby object based on the current or predicted position of the object, the notification area, and the re-notification area. Furthermore, the notification control unit 120 can more appropriately issue a warning based on the driver's line of sight or facial direction and the position of the risk object.

[0126] The above-described embodiment can be expressed as follows. a storage medium for storing computer-readable instructions; a processor connected to the storage medium; The processor executes the computer-readable instructions to: A moving area including a trajectory along which the moving body is expected to move in accordance with the behavior of the moving body when the moving body moves straight, turns right, or turns left, and a surrounding area around the moving body are set; when a first type of object is present in the surrounding area and the first type of object is not visually recognized by the driver of the moving body, controlling an information providing device that provides information to the driver to notify the driver of the presence of the first type of object; A control device that controls the information providing device to issue a re-alert regarding the first type of object if the driver visually recognizes the first type of object after the notification and then the first type of object is present in the movement area but the driver does not visually recognize the first type of object.

[0127] The above-described embodiment can be expressed as follows. a storage medium for storing computer-readable instructions; a processor connected to the storage medium; The processor executes the computer-readable instructions to: Detecting the line of sight of the driver of a moving vehicle, When a first condition is satisfied, an information output device that outputs an alarm to the driver outputs a first alarm; when a second condition is satisfied, causing the information output device to output a second alarm; the first condition is that the driver's line of sight is outside a predetermined range in which the driver is supposed to be visually recognizing what is ahead of the moving body, and that the state in which the line of sight is outside the range continues for a first time; The second condition is that the driver's line of sight is outside the range, a risk object that should be watched is present in a reference area set based on the moving body, and the state in which the line of sight is outside the range continues for a second time period that is shorter than the first time period.

[0128] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0129] 1 Vehicle Systems 10 Camera 12 Radar equipment 16 Object recognition device 52 In-car camera 80 Driving controls 100 Driving assistance device 110 Recognition part 120 Notification control unit 122 Risk Assessment Department 124 Orientation determination unit 126 Object recognition judgment unit 128 Behavior Judgment Department 130 Setting section 132 Information Processing Department 134 Information Provision Department 140 First Control Section 150 Second Control Section

Claims

1. a setting unit that sets a movement area including a trajectory along which the moving body is expected to move in accordance with the behavior of the moving body when the moving body moves straight, turns right, or turns left, and a surrounding area around the moving body; when a first type of object is present in the surrounding area and the first type of object is not visually recognized by the driver of the moving body, controlling an information providing device that provides information to the driver to notify the driver of the presence of the first type of object; a notification control unit that controls the information providing device to issue a re-notification regarding the first type of object when the driver visually recognizes the first type of object after the notification and thereafter the first type of object is present in the movement area but the driver does not visually recognize the first type of object; An alarm control device comprising:

2. The notification control unit when a second type of object moving faster than the first type of object is predicted to be present in the surrounding area from another area after a first predetermined time, and the driver has not visually recognized the second type of object, controlling the information providing device to notify the driver of the presence of the second type of object; after the notification, if the driver visually recognizes the second type of object, and thereafter, when the second type of object is predicted to be present in the movement area at the time the notification was made or a second predetermined time after the second type of object was recognized, the information providing device is controlled to re-notify the driver about the second type of object. The notification control device according to claim 1 .

3. The notification control unit stops the notification when the driver visually recognizes the first type of object after the notification, and continues the notification when the driver does not visually recognize the first type of object after the notification. The notification control device according to claim 1 or 2.

4. The setting unit When the moving object moves straight, an area including a lane on which the moving object moves is set as a movement area; setting the length of the moving area based on the speed of the moving object; The surrounding area includes a lane or a sidewalk adjacent to the lane. The notification control device according to claim 1 or 2.

5. The setting unit When the moving body turns right, a movement area wider than the movement area when the moving body moves straight is set based on the moving body; setting the length of the moving area based on the speed of the moving object; a surrounding area is set to the right of the moving area, to the right of the moving body, and behind the right of the moving body when the moving direction of the moving body is used as a reference; The notification control device according to claim 4.

6. The setting unit When the moving body turns left, a movement area wider than the movement area when the moving body moves straight is set based on the moving body; setting the length of the moving area based on the speed of the moving object; a surrounding area is set to the left of the moving area, to the left of the moving body, and behind the left of the moving body when the moving direction of the moving body is used as a reference; The notification control device according to claim 4.

7. a recognition unit that recognizes the object and tracks the recognized object; the notification control unit assigns identification information to the first type of object recognized by the recognition unit, and when notifying the driver of the presence of the first type of object, assigns notification information indicating that a notification has been made to the identification information of the first type of object, and generates correspondence information in which the identification information of the first type of object and the notification information are associated with each other; after the notification, if the driver visually recognizes the first type of object, and thereafter, when the first type of object is present in the movement area, the driver does not visually recognize the first type of object, and notification information is assigned to the first type of object in the correspondence information, the information providing device is controlled to re-notify the first type of object. The notification control device according to claim 3 .

8. a recognition unit that recognizes the object and tracks the recognized object; the notification control unit assigns identification information to a second type of object that is moving faster than the first type of object recognized by the recognition unit, and when notifying the driver of the presence of the second type of object, assigns notification information indicating that a notification has been made to the identification information of the second type of object, and generates correspondence information in which the identification information of the second type of object and the notification information are associated with each other; after the notification, if the driver visually recognizes the second type of object, and thereafter, when it is predicted that the second type of object will move into the movement area, if the driver does not visually recognize the second type of object and notification information is assigned to the second type of object in the correspondence information, the information providing device is controlled to re-notify the second type of object. The notification control device according to claim 3 .

9. The computer A moving area including a trajectory along which the moving body is expected to move in accordance with the behavior of the moving body when the moving body moves straight, turns right, or turns left, and a surrounding area around the moving body are set; when a first type of object is present in the surrounding area and the first type of object is not visually recognized by the driver of the moving body, controlling an information providing device that provides information to the driver to notify the driver of the presence of the first type of object; after the notification, if the driver visually recognizes the first type of object, and thereafter, when the first type of object is present in the movement area, the driver does not visually recognize the first type of object, the information providing device is controlled to re-notify the driver about the first type of object. Notification control method.

10. On the computer, A process of setting a movement area including a trajectory along which the moving body is expected to move in accordance with the behavior of the moving body when the moving body moves straight, turns right, or turns left, and a surrounding area around the moving body; a process of controlling an information providing device that provides information to the driver of the moving body to notify the driver of the presence of the first type of object when a first type of object is present in the surrounding area and the driver of the moving body does not visually recognize the first type of object; a process of controlling the information providing device to re-notify the driver about the first type of object when the driver visually recognizes the first type of object after the notification and the first type of object is present in the movement area thereafter but the driver does not visually recognize the first type of object; A program that executes the following.

Images