Notification device, vehicle, and notification control method

Abstract

This invention relates to a notification device, a vehicle, and a notification control method. A notification technology is provided capable of notifying the driver of hazards they are unaware of. The notification device mounted on a vehicle includes: a recognition processing unit that recognizes obstacles present in front of the vehicle based on information detected by a forward detection unit, and recognizes potential hazards present at least diagonally behind the vehicle based on information detected by a rear detection unit; a line-of-sight monitoring unit that determines whether the driver is aware of an obstacle or hazard based on the overlap between the driver's line of sight and the obstacle or hazard, obtained from an image captured of the driver; and a notification control unit that notifies the presence of an obstacle or hazard. If the line-of-sight monitoring unit determines that the driver is unaware of an obstacle or hazard, the notification control unit issues a notification.

Description

Technical Field

[0001] This invention relates to a notification device, a vehicle, and a notification control method. Background Technology

[0002] Patent Document 1 discloses a technology for detecting other vehicles and obstacles around a vehicle and determining the level of danger to output an alarm signal. Patent Document 2 discloses a technology that outputs an alarm sound from the direction of the highest level of danger when other vehicles traveling in adjacent lanes or following vehicles are approaching the vehicle and the danger is high based on their relative position.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2003-291689

[0006] Patent Document 2: Japanese Patent Application Publication No. 11-042988 Summary of the Invention

[0007] The problem that the invention aims to solve

[0008] However, in the structures of Patent Documents 1 and 2, the notification of the presence of other vehicles approaching from the rear or side may increase the frequency of notifications due to traffic conditions, thus annoying the driver and, in particular, affecting the driver's awareness of important dangers (risks).

[0009] In view of the above-mentioned problems, the present invention aims to provide a notification technology that, from the viewpoint of improving traffic safety, can notify drivers of dangers they are unaware of.

[0010] Solution for solving the problem

[0011] A notification device according to one aspect of the present invention, mounted on a vehicle, the notification device having:

[0012] The recognition processing unit recognizes obstacles existing in front of the vehicle based on information detected by the front detection unit, and recognizes risk objects existing at least diagonally behind the vehicle based on information detected by the rear detection unit.

[0013] A line-of-sight monitoring unit determines whether the driver is aware of the obstacle or the potential hazard based on the overlap between the driver's line of sight and the obstacle or the potential hazard, obtained from images captured of the driver; and

[0014] The notification control department informs the existence of the aforementioned obstacle or the aforementioned risk object.

[0015] If the visual monitoring unit determines that the driver is unaware of the obstacle or the risky object, the notification control unit will issue the notification.

[0016] Other aspects of the notification control method of the present invention are notification control methods for notification devices mounted in vehicles, characterized in that they include:

[0017] The recognition processing step involves recognizing obstacles in front of the vehicle based on information detected by the front detection unit, and recognizing potential hazards at least diagonally behind the vehicle based on information detected by the rear detection unit.

[0018] In the line-of-sight monitoring step, the line-of-sight monitoring unit determines whether the driver is aware of the obstacle or the potential hazard based on the overlap between the driver's line of sight and the obstacle or the potential hazard, obtained from images captured of the driver; and

[0019] The notification control step involves the notification control unit informing the other party of the existence of the obstacle or the risky object.

[0020] In the notification control step, the notification is made if it is determined, based on the determination in the line-of-sight monitoring step, that the driver is unaware of the obstacle or the risky object.

[0021] The effects of the invention

[0022] According to the present invention, a notification technology can be provided that, from the perspective of improving traffic safety, can notify drivers of dangers they are unaware of. Attached Figure Description

[0023] Figure 1 This is a diagram illustrating the basic structure of a notification device.

[0024] Figure 2 This is a diagram illustrating an example configuration of a notification device mounted on a vehicle.

[0025] Figure 3 This is a flowchart illustrating the processing flow of the notification device.

[0026] Figure 4 This is a diagram illustrating an example of the processing of a notification device.

[0027] Figure 5 This is a flowchart illustrating the processing flow of the notification device according to the second embodiment.

[0028] Figure 6 This is a diagram illustrating a variation of notification processing.

[0029] Explanation of reference numerals in the attached figures

[0030] 100: Notification device; 200: Vehicle; CNT: Control unit; C11: Recognition processing unit; C12: Line of sight monitoring unit; C13: Notification control unit; SN1: Forward detection unit; SN2: Rear detection unit; DMC: Driver monitoring camera; SPK: Output unit. Detailed Implementation

[0031] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Furthermore, the following embodiments are not intended to limit the scope of the claims, and the present invention does not require a combination of all the features described in the embodiments. Two or more features from the plurality of features described in the embodiments may be arbitrarily combined. Additionally, the same reference numerals are used for the same or identical structures, and repeated descriptions are omitted.

[0032] [First Implementation Method]

[0033] (Basic structure of a notification device)

[0034] Figure 1 This diagram illustrates the basic structure of a notification device 100 mounted on a vehicle. The notification device 100 includes: a forward detection unit SN1, a rear detection unit SN2, a driver monitoring camera DMC, a control unit CNT, and an output unit SPK. The forward detection unit SN1 detects obstacles present at least in front of the vehicle (its own vehicle). The rear detection unit SN2 detects potential hazards present at least diagonally behind or to the side of the vehicle. Here, obstacles include, for example, stopped vehicles, pedestrians, lane reduction, and other major factors that may cause a moving vehicle (its own vehicle) to change lanes. In addition, potential hazards include dynamic objects that move over time (e.g., various moving objects such as two-wheeled vehicles, four-wheeled vehicles, etc., traveling at least diagonally behind or to the side in the adjacent lane to which the vehicle is traveling).

[0035] The Driver Monitoring Camera (DMC) is installed inside the vehicle to capture images of the driver's appearance. The DMC captures images of the driver's face and inputs the image information related to the captured facial images into the Control Unit (CNT).

[0036] The control unit CNT includes: a CPU (C1: Central Processing Unit) responsible for processing related to the notification device 100, a storage unit C2 for the hard disk drive (HDD), and an interface unit (I / F unit C3). The storage unit C2 stores the program executed by the CPU (C1) of the control unit CNT, and the data used by the CPU (C1) during processing. Additionally, the storage unit C2 stores stereo audio files used during notification processing. The storage unit C2 can store multiple stereo audio files corresponding to the direction of the notification.

[0037] The interface section (I / F section C3) can acquire information through communication with external devices, or acquire sensor information recognized by various sensors installed in the vehicle. Here, sensor information includes, for example, information related to the driver's braking operation detected by the operation detection sensor installed on the brake pedal, information related to traffic light operation during lane changes, and information related to the driver's steering wheel operation.

[0038] The CPU (C1) executes the notification control program stored in the storage unit C2, thereby serving as the functional structure for notification processing in this embodiment, and functions as the recognition processing unit C11, the line-of-sight monitoring unit C12, and the notification control unit C13. The CPU (C1) determines the vehicle's operating state (e.g., driver's braking operation, wheel angle operation, lane change operation, etc.) based on sensor information obtained from the interface unit (I / F unit C3), and controls the notification processing based on the determination result.

[0039] The recognition processing unit C11 performs image processing based on information detected by the front detection unit SN1 and the rear detection unit SN2, thereby recognizing (acquiring) obstacles and potential hazards existing around its own vehicle. Furthermore, the recognition processing unit C11 acquires the relative distances between the acquired obstacles and potential hazards and the vehicle (its own vehicle).

[0040] In addition, the line-of-sight monitoring unit C12 performs image processing on the driver's face image (the image captured of the driver) input from the driver monitoring camera DMC, thereby obtaining the driver's appearance information such as the direction of the driver's face, line of sight, and driving posture.

[0041] When the line-of-sight monitoring unit C12 detects that the driver's line of sight overlaps with an obstacle perceived (acquired) based on information from the forward detection unit SN1, it determines that the driver is aware of the obstacle. On the other hand, when the line-of-sight monitoring unit C12 does not overlap with an obstacle perceived (acquired) based on information from the forward detection unit SN1, it determines that the driver is not aware of the obstacle.

[0042] The line-of-sight monitoring unit C12 can determine the driver's line of sight to the side or rear of the vehicle 200 (its own vehicle) based on the driver's appearance information, such as the direction of the driver's face and the line of sight towards the door mirror and the rearview mirror, which are obtained through image processing.

[0043] The line-of-sight monitoring unit C12 uses the driver's visual information for judgment processing. If it detects that the driver's line of sight overlaps with a risk object perceived (acquired) based on information from the rear detection unit SN2, it determines that the driver is aware of the risk object. Conversely, if the driver's line of sight does not overlap with a risk object perceived (acquired) based on information from the rear detection unit SN2, the line-of-sight monitoring unit C12 determines that the driver is not aware of the risk object.

[0044] Based on the determination result of the line-of-sight monitoring unit C12, the notification control unit C13 outputs a notification tone from the output unit SPK to notify the driver of the presence of an obstacle or risky object. (See reference...) Figure 3 This will explain the specific processing flow of the notification device 100.

[0045] (Configuration example of notification device 100)

[0046] Figure 2 This diagram illustrates an example configuration of the notification device 100 mounted on a vehicle 200. The front detection unit SN1 includes, for example, a camera unit 21a, a lidar detection unit 22a, and a radar detection unit 23a. Similarly, the rear detection unit SN2 includes, for example, a camera unit 21b, a lidar detection unit 22b, and a radar detection unit 23b. Furthermore, while the structural examples of the front detection unit SN1 and the rear detection unit SN2 are illustrative, it is also possible for both the front detection unit SN1 and the rear detection unit SN2 to have at least a camera unit 21a and a camera unit 21b.

[0047] Camera units 21a and 21b are imaging devices used to detect objects around the vehicle 200 by capturing images. Camera unit 21a, constituting the front detection unit SN1, is mounted on the interior side of the front window of the vehicle 200 at the front of the roof, enabling it to capture images of the front of the vehicle 200. Similarly, camera unit 21b, constituting the rear detection unit SN2, is mounted on the interior side of the rear window at the rear of the roof of the vehicle 200, enabling it to capture images of the rear of the vehicle 200. Furthermore, the mounting positions of camera units 21a and 21b are illustrative and not limited to any particular location. Figure 2 The location is shown. Based on the analysis of images captured by camera units 21a and 21b, the outline of obstacles existing in front of the vehicle 200 and the lane dividing lines (white lines, etc.) on the road can be obtained.

[0048] The lidar detection units 22a and 22b are, for example, Light Detection and Ranging (LIDAR), which use light to detect objects (obstacles, potential hazards) around the vehicle 200. Based on the analysis of the information detected by the lidar detection units 22a and 22b, the distance to the object (obstacle, potential hazard) can be measured. For example, one lidar detection unit 22a constituting the front detection unit SN1 is installed at each corner of the front of the vehicle 200, and one lidar detection unit 22b constituting the rear detection unit SN2 is installed at the center of the rear and one on each side of the rear.

[0049] Radar detection units 23a and 23b are, for example, millimeter-wave radars, which use radio waves to detect objects (obstacles, potential hazards) around the vehicle 200. Based on the analysis of the information detected by the radar detection units 23a and 23b, the distance to the objects (obstacles, potential hazards) can be measured. For example, one radar detection unit 23a constituting the front detection unit SN1 is installed in the center of the front of the vehicle 200, and one is installed at each of the front corners; similarly, one radar detection unit 23b constituting the rear detection unit SN2 is installed at each of the rear corners.

[0050] The output unit SPK can be installed, for example, in the headrest 27 of the driver's seat 25. The output unit SPK may have a speaker system capable of outputting stereo sound. At a predetermined notification time, the notification control unit C13 outputs stereo sound (audio source file) stored in the storage unit C2 to the output unit SPK. The output unit SPK plays the stereo sound (audio source file) input from the notification control unit C13, outputting the sound in stereo to notify the driver of the direction of the obstacle or potential hazard. Furthermore, the stereo sound is not limited to the audio source file stored in the storage unit C2; for example, it could be external ambient sound collected by a microphone (not shown) mounted on the vehicle 200, including the sound of vehicles moving around obstacles or potential hazard, and outputting this ambient sound in stereo. That is, the notification control unit C13 may output the ambient sound collected by the microphone, including the sound of vehicles moving around, to the output unit SPK, and the output unit SPK may output the ambient sound input from the notification control unit C13 in stereo.

[0051] Furthermore, the location of the output unit SPK is not limited to the headrest 27; for example, it can also be located on the backrest 26 of the driver's seat 25 (near the driver's shoulder).

[0052] Alternatively, the output unit SPK can be an audio device with a communication unit that can communicate with the control unit CNT. Alternatively, both the control unit CNT and the audio device's communication unit can support short-range wireless communication such as Bluetooth (registered trademark).

[0053] Audio equipment could be, for example, a portable audio player for the driver, or a wearable terminal device for the driver (e.g., glasses with built-in headphones and speakers). Alternatively, the output unit SPK located within the headrest 27 could be combined with the wearable terminal device for the driver to indicate the direction of obstacles or potential hazards.

[0054] Alternatively, the notification control unit C13 may use a vibration generating unit 29, such as a vibrator, to vibrate the seat belt 28 of the vehicle seat 25 (driver's seat) to provide the driver with a notification-related attention reminder.

[0055] (Processing of notification device 100)

[0056] The processing flow of the notification device 100 in this embodiment will then be explained. Figure 3 This is a flowchart illustrating the processing flow of the notification device 100. Figure 4 This is a schematic diagram illustrating an example of the processing of the notification device 100. Figure 4 ST41 shows the state of vehicle 200 (its own vehicle) traveling in the left lane L1 of a single-sided two-lane road. The state is shown as follows: An obstacle 410, detected (acquired) in front of vehicle 200 based on information from the forward detection unit SN1, is a stopped vehicle, as shown by the dashed line 415. Vehicle 200 needs to change lanes from the left lane L1 to the right lane L2. Additionally, in ST41, a two-wheeled vehicle is detected (acquired) as a potential danger object 450 diagonally behind vehicle 200 based on information from the rear detection unit SN2.

[0057] Figure 4 ST42 shows a variation of ST41, illustrating a state where, as an obstacle 420 recognized based on information from the forward detection unit SN1, lanes L1 and L2 merge, reducing the number of lanes from two lanes to one. In ST42, as shown by dashed line 425, vehicle 200 needs to change lanes from left lane L1 to right lane L2. Furthermore, in ST42, a four-wheeled vehicle is present diagonally behind vehicle 200 as a potential danger object 460 recognized based on information from the rear detection unit SN2. (See reference...) Figure 4 The status of ST41 and ST42, using Figure 3 The flowchart below illustrates the processing flow of the notification device 100.

[0058] In step S310, the forward detection unit SN1 detects an obstacle located in front of the vehicle 200 (e.g., Figure 4The obstacle 410 (stopped vehicle) and obstacle 420 (lane reduction unit) are detected by the recognition processing unit C11. Based on the information detected by the forward detection unit SN1, the recognition processing unit C11 performs image processing to recognize (acquire) the obstacles existing around the vehicle 200.

[0059] In step S320, the rear detection unit SN2 detects a potential danger object located to the side or rear of the vehicle 200 (e.g., Figure 4 Risk objects 450 (two-wheeled vehicles) and risk objects 460 (four-wheeled vehicles) are identified by the recognition and processing unit C11, which performs image processing based on the information detected by the rear detection unit SN2 to recognize (acquire) the risk objects existing around the vehicle 200.

[0060] In step S330, the driver monitoring camera (DMC) captures an image of the driver's face and inputs image information related to the captured image to the control unit (CNT). The gaze monitoring unit (C12) performs image processing on the driver's face image input from the driver monitoring camera (DMC) to obtain the driver's appearance information, such as the direction of the driver's face, gaze, and driving posture.

[0061] In step S340, the line-of-sight monitoring unit C12 determines whether it detects an overlap between the driver's line of sight and an obstacle perceived by the forward detection unit SN1. If it detects an overlap between the driver's line of sight and an obstacle, the line-of-sight monitoring unit C12 determines that the driver has perceived an obstacle (S340: "Yes"), and the process proceeds to step S350.

[0062] On the other hand, in the determination process of step S340, if the driver's line of sight does not overlap with the obstacle, the line of sight monitoring unit C12 determines that the driver does not recognize the obstacle (S340: "No"), and the process proceeds to step S370.

[0063] In this step, if the driver is determined by the line-of-sight monitoring unit C12 to be unaware of an obstacle, the control unit C13 is notified to issue a notification at a higher level than the notification (first notification) in step S360 (second notification). In this case, for example, as... Figure 4As shown in ST41 and ST42, vehicle 200 (its own vehicle) is approaching obstacles 410 and 420 in front of it in lane L1, and needs to issue a notification of higher urgency than the notification of the presence of risk objects 450 and 460 diagonally behind it (the first notification in step S360). In the notification (second notification) in step S370, the notification control unit C13 controls the notification to be issued at a higher notification level (notification intensity) than the notification level of the notification (first notification) in step S360. For example, the volume of the notification tone output from the output unit SPK can be increased, or the vibration level of the vibration generator 29 can be increased. Alternatively, the notification tone from the output unit SPK can be combined with the vibration of the vibration generator 29 to issue the notification.

[0064] In step S350, the line-of-sight monitoring unit C12 uses driver appearance information, such as the direction of the driver's face and the driver's gaze toward the door mirror and rearview mirror, to determine whether the driver's line of sight overlaps with a risk object recognized (acquired) based on information from the rear detection unit SN2. If the driver's line of sight does not overlap with a risk object recognized (acquired) based on information from the rear detection unit SN2, the line-of-sight monitoring unit C12 determines that the driver has not recognized the risk object (S350: "No"), and the process proceeds to step S360.

[0065] In step S360, the notification control unit C13, based on the determination result of step S350, outputs a notification tone from the output unit SPK to notify the driver (first notification) of the presence of the risk objects 450 and 460. That is, if the driver is aware of the obstacle but not the risk object based on the determination of the line-of-sight monitoring unit C12, the notification control unit C13 outputs a notification tone from the output unit SPK to notify the driver.

[0066] On the other hand, in the determination of step S350, if the driver's line of sight overlaps with the risk object, the line of sight monitoring unit C12 determines that the driver is aware of the risk object (S350: "Yes"), and the process proceeds to step S380.

[0067] In step S380, the driver recognizes obstacles 410 and 420 (S340: "Yes") and the driver recognizes hazardous objects 450 and 460 (S350: "Yes"). In this state, to suppress unnecessary notifications, the notification control unit C13 does not issue a notification. That is, if the driver recognizes both obstacles and hazardous objects based on the determination of the line-of-sight monitoring unit C12, the notification control unit C13 does not issue a notification.

[0068] According to this embodiment, a notification technology can be provided that can suppress notifications of hazards perceived by the driver, and, from the viewpoint of improving traffic safety, can notify the driver of hazards not perceived by the driver.

[0069] [Second Implementation]

[0070] The processing of the notification device 100 in the second embodiment will then be described. Figure 5 This is a flowchart illustrating the processing flow of the notification device 100, for example, in... Figure 3 The flowcharts described herein use the same processing steps, with the same step numbers appended and descriptions omitted. Figure 5 In the flowchart, after the "No" processing in step S350 and before the notification in step S360, a processing step S510 is added, which is different from the first embodiment. Figure 3 The processing methods differ.

[0071] In step S510, the control unit C13 is notified to obtain the position P2 of the vehicle 200 (its own vehicle) and the obstacles in front (e.g., ...) from the recognition processing unit C11. Figure 4 The relative distance between obstacles 410 and 420 (P1) is determined. Then, the notification control unit C13 determines whether the acquired relative distance is below a threshold distance. If the relative distance is greater than the threshold distance (S510: "No"), the process returns to step S350, and the determination process of steps S350 and S510 is repeated in the same way. Here, the state where the relative distance is greater than the threshold distance means that there is still a margin in distance from the perspective of colliding with the obstacle in front. In this state, the driver will be aware of the existence of the risk object before the notification device 100 issues a notification (S350: "Yes"), and the operability to avoid collision with the risk object is high. Therefore, before the relative distance becomes below the threshold distance (S510: "Yes"), the notification control unit C13 controls the output unit SPK not to output a notification tone.

[0072] Then, in the determination process of step S510, if the relative distance becomes below the threshold distance (S510: "Yes"), the process proceeds to step S360, and the control unit C13 is notified to output a notification tone from the output unit SPK based on the determination result of step S350 to notify the driver of the presence of risk objects 450 and 460.

[0073] According to this embodiment, a notification technology can be provided that can suppress notifications about hazards perceived by the driver, and, from the viewpoint of improving traffic safety, can notify the driver of hazards not perceived by the driver.

[0074] (Variation Example 1)

[0075] Alternatively, after the notification (S360) described in the first and second embodiments, if the driver's line of sight overlaps with a risky object, the line-of-sight monitoring unit C12 determines that the driver is aware of the risky object, and the notification control unit C13 controls the notification based on the determination result of the line-of-sight monitoring unit C12 to issue a notification at a lower level (e.g., notification intensity, such as the volume of the notification tone) than the notification level of the notification in step S360 (first notification) (third notification). Furthermore, this example is not limited to issuing a notification at a lower level (notification intensity) (third notification); if the line-of-sight monitoring unit C12 determines that the driver is aware of the risky object, the notification control unit C13 controls the output of the notification tone from the output unit SPK to stop. According to this example, the notification level regarding hazards that the driver is aware of can be suppressed, and from the viewpoint of improving traffic safety, hazards that the driver is not aware of can also be notified.

[0076] (Variation Example 2)

[0077] Alternatively, after the notification (S360) described in the first and second embodiments, if the interface unit C3 (I / F unit) obtains information related to the driver's braking operation, the notification control unit C13 controls the notification to be delivered at a lower notification level (e.g., notification intensity, such as the volume of the notification tone) than the notification level of the notification in step S360 (first notification) (e.g., notification intensity, such as the volume of the notification tone) (third notification). Furthermore, this example is not limited to delivering the notification at a lower notification level (notification intensity) (third notification); it is also possible that, upon obtaining information related to the driver's braking operation, the notification control unit C13 controls the output of the notification tone from the output unit SPK to stop.

[0078] Alternatively, as described in Modification 1, after a notification (third notification) is given, if the interface unit C3 (I / F unit) obtains information related to the driver's braking operation, the notification control unit C13 controls the notification to give a fourth notification at a lower notification level (notification intensity) than the notification level of the third notification. According to this example, the notification level regarding hazards perceived by the driver is suppressed, and from the viewpoint of improving traffic safety, hazards not perceived by the driver can be notified.

[0079] (Variation Example 3)

[0080] Figure 6This diagram schematically illustrates a variation of the notification processing, showing vehicle 200 (its own vehicle) traveling in the central lane L1 of a three-lane road on one side. An obstacle 430 (road debris) is detected (acquired) in front of vehicle 200 based on information from the forward detection unit SN1. Furthermore, a two-wheeled vehicle is present as a risk object 450 in lane L2 to the right of vehicle 200's (its own vehicle's) driving lane L1, and a four-wheeled vehicle is present as a risk object 460 in lane L3 to the left of driving lane L1. Figure 6 In the illustrated states, even when lane changes have been performed, as shown by dashed lines 615 and 625, risky objects 450 and 460 still exist. In such states, the notification control unit C13 uses stereo communication from the output unit SPK to notify the driver of the direction in which the risky object exists (diagonally behind to the right and diagonally behind to the left). Based on this example, from the perspective of improving traffic safety, it is possible to notify the driver of potential hazards from multiple directions where risky objects may exist.

[0081] [Summary of Implementation Methods]

[0082] Structure 1. The notification device in the above embodiment is a notification device 100 mounted on a vehicle 200, and the notification device includes:

[0083] The recognition processing unit C11 recognizes obstacles existing in front of the vehicle based on information detected by the front detection unit SN1, and recognizes risk objects existing at least diagonally behind the vehicle based on information detected by the rear detection unit SN2.

[0084] The line-of-sight monitoring unit C12 determines whether the driver is aware of the obstacle or the potential hazard based on the overlap between the driver's line of sight and the obstacle or the potential hazard, obtained from images captured of the driver; and

[0085] The notification control unit C13 informs the other party of the existence of the aforementioned obstacle or the aforementioned risky object.

[0086] If the driver is determined by the line-of-sight monitoring unit C12 to be unaware of the obstacle or the risky object, the notification control unit C13 will issue the notification.

[0087] According to the notification device of structure 1, based on the viewpoint of improving traffic safety, it is able to notify the driver of dangers that they are unaware of.

[0088] Structure 2. If, based on the determination of the line-of-sight monitoring unit C12, it is determined that the driver is aware of the obstacle and the driver is not aware of the risk object, the notification control unit C13 makes the notification.

[0089] The notification device according to structure 2 can notify the driver of dangers that they are unaware of, and can only notify the driver when there is a high probability that the driver will change the way they are going, thus reducing annoyance caused by excessive notification.

[0090] Structure 3. When the relative distance between the obstacle and the vehicle is below a threshold distance, the notification control unit C13 issues the notification.

[0091] The notification device according to structure 3 can suppress notifications before the relative distance becomes below a threshold distance, and only notify when the relative distance becomes below the threshold distance and the driver is likely to change the route, thus reducing annoyance caused by excessive notifications.

[0092] Structure 4. After making the notification, if the notification control unit C13 determines, based on the determination of the line-of-sight monitoring unit C12, that the driver is aware of the risk object, the notification control unit C13 may make a notification at a lower notification level than the notification level of the original notification, or stop the notification.

[0093] According to the notification device of structure 4, the driver will change the route when he recognizes a risky object, thereby reducing the possibility of contact with the risky object and thus reducing the annoyance of notification.

[0094] Structure 5. After making the notification, if the driver's braking operation is detected, the notification control unit C13 may make a notification at a lower notification level than the notification level, or stop the notification.

[0095] According to the notification device of structure 5, the possibility of stopping without lane changing is high, thus reducing the annoyance of notification.

[0096] Structure 6. If the driver is determined not to be aware of the obstacle based on the determination of the line-of-sight monitoring unit, the notification control unit C13 issues a notification at a higher notification level than the notification level of the previous notification.

[0097] According to the notification device of structure 6, when a notification of greater urgency is required, a higher notification level is provided, thereby enabling the driver to be notified of hazards that they have not yet recognized.

[0098] Structure 7. The notification control unit C13 outputs stereo sound from the output unit, thereby notifying the direction of the presence of the risk object.

[0099] According to the notification device of structure 7, a stereo sound is used as the notification tone, so that the driver can intuitively grasp the direction of the risk object.

[0100] Structure 8. The notification control unit C13 uses stereo sound from the output unit provided in the headrest of the driver's seat to notify the direction of the presence of the risk object.

[0101] According to the notification device of structure 8, an output unit is provided in the headrest near the driver's ear, which can transmit a clearer notification tone to the driver, so that the driver can more clearly grasp the direction of the risk object.

[0102] Structure 9. If the driver is determined to be aware of the obstacle and the risk object based on the determination of the line-of-sight monitoring unit C12, the notification control unit C13 does not issue the notification.

[0103] The notification device according to structure 9 can suppress notifications about hazards perceived by the driver.

[0104] Structure 10. The vehicle 200 of the above embodiment has a notification device 100 with any one of the structures of Structure 1 to Structure 9.

[0105] According to the vehicle of structure 10, based on the viewpoint of improving traffic safety, a vehicle can be provided that is equipped with a notification device that can notify the driver of dangers that they are unaware of.

[0106] Structure 11. The notification control method of the above embodiment is a working method of the notification device 100 mounted on a vehicle, including:

[0107] In the recognition processing steps S310 and S320, the recognition processing unit C11 recognizes obstacles existing in front of the vehicle based on information detected by the front detection unit SN1, and recognizes risk objects existing at least diagonally behind the vehicle based on information detected by the rear detection unit SN2.

[0108] In the line-of-sight monitoring step S330, the line-of-sight monitoring unit C12 determines, based on the overlap between the driver's line of sight and the obstacle or the potential hazard obtained from an image captured of the driver, whether the driver is aware of the obstacle or the potential hazard; and

[0109] In notification control steps S340-S370, the notification control unit C13 notifies the existence of the obstacle or the risky object.

[0110] In the notification control step, the notification is made if it is determined, based on the determination in the line-of-sight monitoring step, that the driver is unaware of the obstacle or the risky object.

[0111] According to the notification control method of Structure 11, based on the viewpoint of improving traffic safety, it is possible to notify drivers of dangers they are unaware of.

[0112] Structure 12. In the notification control step, the notification is made if it is determined, based on the determination in the line-of-sight monitoring step, that the driver is aware of the obstacle and the driver is not aware of the risk object.

[0113] According to the notification control method of structure 12, it is possible to notify the driver of dangers that they are unaware of, and to notify only when the driver is highly likely to change the direction of travel, thus reducing annoyance caused by excessive notification.

[0114] (Other implementation methods)

[0115] The present invention can also provide a program that implements the functions of the above embodiments to a system or a notification device constituting the system via a network or storage medium, wherein one or more processors of the computer of the notification device read the program and execute the processing of the notification device.

[0116] The invention is not limited to the above-described embodiments, and various modifications and alterations can be made within the scope of the spirit of the invention.

Claims

1. A notification device mounted on a vehicle, the notification device characterized by comprising: a recognition processing portion that recognizes an obstacle present in front of the vehicle based on information detected by a front detection portion, and recognizes a risk object present at least diagonally rearward of the vehicle based on information detected by a rear detection portion; a line-of-sight monitoring portion that determines whether or not a driver recognizes the obstacle or the risk object, based on overlap of a line of sight of the driver acquired based on a captured image of the driver with the obstacle or the risk object; and a notification control portion that notifies of the presence of the obstacle or the risk object, the notification control portion performing the notification at a high notification level in a case where the driver is determined not to recognize the obstacle based on the determination of the line-of-sight monitoring portion, as compared to a case where the driver is determined to recognize the obstacle and not to recognize the risk object based on the determination of the line-of-sight monitoring portion.

2. The notification device according to claim 1, characterized in that the notification control portion performs the notification in a case where a relative distance between the obstacle and the vehicle becomes equal to or less than a threshold distance.

3. The notification device according to claim 1 or 2, characterized in that the notification control portion, after performing the notification, performs notification at a lower notification level than the notification level of the notification, or stops the notification, in a case where the driver is determined to recognize the risk object based on the determination of the line-of-sight monitoring portion.

4. The notification device according to claim 1 or 2, characterized in that the notification control portion, after performing the notification, performs notification at a lower notification level than the notification level of the notification, or stops the notification, in a case where a brake operation of the driver is detected.

5. The notification device according to claim 1 or 2, characterized in that the notification control portion outputs stereo sound from an output portion, thereby notifying of a direction in which the risk object is present.

6. The notification device according to claim 5, characterized in that the notification control portion notifies of the direction in which the risk object is present using stereo sound from the output portion provided to a headrest of a driver's seat on which the driver is seated.

7. The notification device according to claim 1, characterized in that the notification control portion does not perform the notification in a case where the driver is determined to recognize the obstacle and to recognize the risk object based on the determination of the line-of-sight monitoring portion.

8. A vehicle in which the notification device according to claim 1 is provided.

9. A notification control method of a notification device mounted on a vehicle, the notification control method characterized by comprising: a recognition processing step of recognizing an obstacle present in front of the vehicle based on information detected by a front detection portion, and recognizing a risk object present at least diagonally rearward of the vehicle based on information detected by a rear detection portion. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ a line-of-sight monitoring step of determining whether the driver recognizes the obstacle or the risk object based on overlap of the driver's line of sight, which is acquired based on a captured image of the driver, and the obstacle or the risk object; and a notification control step of notifying that the obstacle or the risk object exists, in the notification control step, the notification is performed at a high notification level in a case where it is determined based on the determination of the line-of-sight monitoring step that the driver does not recognize the obstacle, as compared with the notification performed in a case where it is determined based on the determination of the line-of-sight monitoring step that the driver recognizes the obstacle and does not recognize the risk object.

Images