Driving assistance device, driving assistance method, and recording medium

Abstract

The present invention provides a driving assistance device, a driving assistance method and a recording medium. The driving assistance device comprises: a storage unit (22) for storing information indicating the degree of familiarity of a driver of a vehicle (2) with an environment; a detection unit (32) for detecting an object existing around the vehicle (2) based on a sensor signal indicating the condition around the vehicle (2) obtained by a sensor (12) mounted on the vehicle (2); an approach determination unit (33) for determining whether an object is approaching to a degree that has the possibility of colliding with the vehicle (2); and a notification unit (34) for notifying the driver of the approach via a notification device (14) provided on the vehicle (2) at a timing corresponding to the degree of familiarity of the driver of the vehicle (2) with the environment when it is determined that the object is approaching to a degree that has the possibility of colliding with the vehicle (2).

Description

Technical Field

[0001] The present invention relates to a driving assistance device, a driving assistance method and a driving assistance computer program for calling a driver's attention to a certain danger when a certain danger is detected during vehicle driving. Background Art

[0002] There has been proposed a technology for calling the driver's attention to a certain danger when the danger is detected during driving of a vehicle (see Japanese Patent Application Laid-Open No. 2015-219531).

[0003] In the technology disclosed in Japanese Patent Application Laid-Open No. 2015-219531, based on the detection results obtained by the surrounding sensor that detects the distance between the vehicle and the object existing around the vehicle, it is determined whether the vehicle and the object are in a proximity state of being close or in contact with each other at a distance less than a predetermined distance. And, if it is determined to be in a proximity state, the main point of the situation is notified to the driver of the vehicle. Summary of the invention

[0004] If the timing of notifying the approach of an object to the vehicle is too early, the driver may feel annoyed. On the contrary, if the timing of notifying the approach of an object to the vehicle is too late, the driver may become anxious and may not be able to properly perform vehicle operations to avoid collision with the approaching object.

[0005] Therefore, an object of the present invention is to provide a driving assistance device that can notify a driver of a vehicle of an object approaching the vehicle at an appropriate timing.

[0006] According to one embodiment, a driving assistance device is provided. The driving assistance device includes: a storage unit that stores information indicating the degree of familiarity of a driver of a vehicle with an environment; a detection unit that detects an object existing around the vehicle based on a sensor signal indicating the state of the surroundings of the vehicle obtained by a sensor mounted on the vehicle; an approach determination unit that determines whether the object is close to the extent that there is a possibility of collision with the vehicle; and a notification unit that, when it is determined that the object is close to the extent that there is a possibility of collision with the vehicle, notifies the driver of the approach via a notification device installed on the vehicle at a timing corresponding to the degree of familiarity of the driver of the vehicle with the environment.

[0007] In the driving assistance device, preferably, when the approach determination unit determines that an object is approaching to a degree that has the possibility of colliding with the vehicle, the approach determination unit calculates a predicted time until the collision, and when the predicted time becomes below a time threshold indicating the timing of notification, which is set shorter as the driver's familiarity with the environment increases, the notification unit notifies the driver of the approach.

[0008] In this case, preferably, the information indicating the driver's familiarity with the environment includes the driver's place of residence and the number of years the driver has lived at the place of residence, and when the notification unit determines that the position of the vehicle when an object is close to a degree that has the possibility of colliding with the vehicle is the driver's place of residence, the longer the driver's number of years of residence, the shorter the time threshold is set.

[0009] Furthermore, preferably, the notification unit sets a time threshold shorter than the time threshold when the vehicle's position when it is determined that an object is approaching to a degree that has the possibility of colliding with the vehicle is the driver's residence.

[0010] Furthermore, in the driving assistance device, preferably, information indicating the driver's familiarity with the environment includes the number of times the driver has traveled in each predetermined area or predetermined road section, and the notification unit sets the time threshold to be shorter the greater the number of times the driver has traveled in an area or road section that includes the position of the vehicle when an object is determined to be close to a degree that has the possibility of colliding with the vehicle.

[0011] Preferably, the driving assistance device also has a learning unit, which records the reaction time from the time the driver is notified of the approach of an object that is approaching to an extent that has the possibility of colliding with the vehicle until the driver performs an operation of the vehicle to avoid the collision, and learns the timing of the notification based on the reaction time and the driver's familiarity with the environment at the position of the vehicle when it is determined that the object is approaching to an extent that has the possibility of colliding with the vehicle.

[0012] According to another embodiment, a driving assistance method is provided. The driving assistance method includes the following contents: based on a sensor signal indicating a condition around the vehicle obtained by a sensor mounted on the vehicle, an object existing around the vehicle is detected, and whether the object is close to the extent that there is a possibility of collision with the vehicle is determined, and when it is determined that the object is close to the extent that there is a possibility of collision with the vehicle, the driver is notified of the approach through a notification device installed on the vehicle at a timing corresponding to the degree of familiarity of the driver of the vehicle with the environment stored in a storage unit.

[0013] According to another embodiment, a driving assistance computer program for being executed by a processor mounted on a vehicle is provided. The driving assistance computer program includes a command for causing the processor mounted on the vehicle to execute the following processing, wherein the processing is to detect objects existing around the vehicle based on sensor signals indicating the surrounding conditions of the vehicle obtained by sensors mounted on the vehicle, determine whether the objects are close to the extent that there is a possibility of collision with the vehicle, and if it is determined that the objects are close to the extent that there is a possibility of collision with the vehicle, at a timing corresponding to the degree of familiarity of the driver of the vehicle with the environment stored in a storage unit, notify the driver of the approach via a notification device provided on the vehicle.

[0014] The driving assistance device according to the present invention has the effect of being able to notify the driver of the vehicle of the approach of an object to the vehicle at an appropriate timing. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a schematic diagram of the driving assistance system including the driving assistance device.

[0016] Figure 2 This is a schematic diagram of the structure of a vehicle equipped with a driving assistance device.

[0017] Figure 3 FIG. 4 is a hardware structure diagram of an electronic control device as one embodiment of a driving assistance device.

[0018] Figure 4 The present invention is a functional block diagram of a processor of an electronic control device related to driving assistance processing.

[0019] Figure 5A This is a diagram showing an example of the relationship between the driver's familiarity with the environment and the timing of notifying the approach of an object.

[0020] Figure 5B This is a diagram showing an example of the relationship between the driver's familiarity with the environment and the timing of notifying the approach of an object.

[0021] Figure 6 This is a flowchart of the driving assistance process in the electronic control device.

[0022] Figure 7 This is the hardware structure diagram of the server.

[0023] Figure 8 A functional block diagram of a processor of a server associated with learning processing. DETAILED DESCRIPTION

[0024] Hereinafter, a driving assistance device, a driving assistance method executed in the driving assistance device, a driving assistance computer program, and a driving assistance system including the driving assistance device will be described with reference to the accompanying drawings. The driving assistance device notifies the driver of the approach when detecting an object approaching the vehicle. At this time, the driving assistance device determines the timing of notifying the driver of the approach based on information indicating the driver's familiarity with the environment, thereby being able to notify the driver of the approach of the object to the vehicle at an appropriate timing.

[0025] Figure 1 1 is a schematic diagram of a driving assistance system including a driving assistance device. In the present embodiment, a driving assistance system 1 is mounted on a vehicle 2 and includes an electronic control unit (ECU) 3 as an example of a driving assistance device, and a server 4. The ECU 3 is connected to the server 4 in a manner capable of communicating via the wireless base station 6 and the communication network 5, for example, by accessing a wireless base station 6 connected to a communication network 5 connected to the server 4 via a gateway (not shown). In addition, although Figure 1 Although only one vehicle 2 and one ECU 3 are shown in the figure, the driving assistance system 1 may also include a plurality of vehicles 2 and an ECU 3 installed in each of the plurality of vehicles 2. Similarly, a plurality of wireless base stations 6 may be connected to the communication network 5. In addition, the server 4 may be, for example, a server used for traffic management in a smart city or a networked city that effectively utilizes advanced technologies such as big data.

[0026] When the ECU 3 detects an object (e.g., a pedestrian or another vehicle) approaching the vehicle 2 and having a risk of colliding with the vehicle 2, the ECU 3 notifies the driver of the approach at a timing set according to the familiarity of the driver of the vehicle 2 with the environment. In the present embodiment, the relationship between the familiarity of the driver with the environment and the timing of notifying the approach of the object to the vehicle 2 is learned by the server 4 and updated by being notified to the ECU 3 of the vehicle 2. Therefore, the server 4 obtains the reaction time from the notification of the approach of the object to the vehicle 2 to the start of the operation of the vehicle 2 to avoid the collision from the ECU 3 of the vehicle 2 together with the information indicating the familiarity of the driver with the environment. When the server 4 receives a predetermined number of sets of information indicating the reaction time and the familiarity, the server 4 learns the timing of notifying the approach of the object to the vehicle 2 according to the familiarity based on these sets.

[0027] Figure 21 is a schematic structural diagram of a vehicle 2 equipped with a driving assistance device. In the present embodiment, the vehicle 2 includes a GPS receiver 11, a camera 12 for photographing the surroundings of the vehicle 2, a driver monitoring camera 13 for photographing the driver of the vehicle 2, a notification device 14, a wireless communication terminal 15, and an ECU 3 as an example of a driving assistance device. The GPS receiver 11, the camera 12, the driver monitoring camera 13, the notification device 14, and the wireless communication terminal 15 are connected to the ECU 3 via an in-vehicle network conforming to a standard such as a controller area network so as to be able to communicate with each other.

[0028] The GPS receiver 11 is an example of a position measurement unit, which receives GPS signals from GPS satellites at predetermined intervals and locates the position of the vehicle 2 based on the received GPS signals. Furthermore, the GPS receiver 11 outputs positioning information indicating the positioning result of the vehicle 2 based on the GPS signals to the ECU 3 via the in-vehicle network at predetermined intervals. In addition, the vehicle 2 may also have a receiver in accordance with the satellite positioning system other than the GPS receiver 11. In this case, the receiver only needs to locate the position of the vehicle 2.

[0029] The camera 12 is an example of a sensor unit that obtains a sensor signal indicating the conditions around the vehicle 2. The camera 12 has a two-dimensional detector composed of an array of photoelectric conversion elements that are sensitive to visible light, such as CCD or C-MOS, and an imaging optical system that forms an image of the area that becomes the shooting object on the two-dimensional detector. Moreover, the camera 12 is installed on the vehicle 2 in a manner to shoot a predetermined shooting area around the vehicle 2, such as the front area or the rear area of ​​the vehicle 2. The camera 12 shoots the shooting area at each predetermined shooting cycle (for example, 1 / 30 second to 1 / 10 second) and generates an image showing the shooting area. The generated image is an example of a sensor signal. In addition, a plurality of cameras 12 with different shooting directions or focal distances may be provided on the vehicle 2.

[0030] Whenever the camera 12 generates an image, it outputs the generated image to the ECU 3 via the in-vehicle network.

[0031] The driver monitoring camera 13, like the camera 12, has a two-dimensional detector composed of an array of photoelectric conversion elements such as CCD or C-MOS that are sensitive to visible light or infrared light, and an imaging optical system that forms an image of the area to be photographed on the two-dimensional detector. The driver monitoring camera 13 may further have a light source such as an infrared LED for illuminating the driver. Moreover, the driver monitoring camera 13 is installed, for example, on the dashboard or its vicinity, facing the driver in such a manner that the head of the driver sitting on the driver's seat of the vehicle 2 is included in its photographed area, that is, in such a manner that the driver's head can be photographed. Moreover, the driver monitoring camera 13 photographs the driver at each predetermined photographing cycle (for example, 1 / 30 second to 1 / 10 second) and generates an image showing the driver (hereinafter referred to as a driver image). The driver image obtained by the driver monitoring camera 13 may be a color image or a gray image. Whenever the driver monitoring camera 13 generates a driver image, it outputs the generated driver image to the ECU 3 via the in-vehicle network.

[0032] The notification device 14 is a device that is installed in the vehicle compartment of the vehicle 2 and implements a predetermined notification to the driver by light, voice, text display or image display. Therefore, the notification device 14 has at least one of, for example, a speaker, a light source or a display device. Moreover, when the notification device 14 receives a notification from the ECU 3 to warn of the approach of an object to the vehicle 2, it will notify the driver of the approach through voice from the speaker, light emission or flashing of the light source, or display of a warning message or warning icon implemented by the display device.

[0033] The wireless communication terminal 15 is an example of a communication unit, and performs wireless communication processing in accordance with a predetermined wireless communication standard. In the present embodiment, when the wireless communication terminal 15 receives information indicating the driver's familiarity with the environment and information indicating the driver's reaction time from the ECU 3, it generates an uplink wireless signal including these information, and transmits the generated wireless signal to the wireless base station 6. In addition, the wireless communication terminal 15 receives a downlink wireless signal including information indicating the relationship between the driver's familiarity with the environment and the timing of notifying the approach of an object to the vehicle 2 from the wireless base station 6, and transmits the information to the ECU 3.

[0034] The ECU 3 assists the driver in driving. In the present embodiment, the ECU 3 detects an object approaching the vehicle 2 based on an image obtained by the camera 12 , and notifies the driver of the approach via the notification device 14 .

[0035] Figure 3 As shown in Figure 2, the hardware structure diagram of ECU3 is shown in Figure 3. Figure 3 As shown, the ECU 3 includes a communication interface 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 may be configured as separate circuits, or may be integrally configured as one integrated circuit.

[0036] The communication interface 21 has an interface circuit for connecting the ECU 3 to the in-vehicle network. Moreover, whenever the communication interface 21 receives positioning information from the GPS receiver 11, the positioning information is sent to the processor 23. In addition, whenever the communication interface 21 receives an image from the camera 12, the image is sent to the processor 23. Similarly, whenever the communication interface 21 receives a driver image from the driver monitoring camera 13, the driver image is sent to the processor 23. Further, when the communication interface 21 receives information indicating the relationship between the driver's familiarity with the environment and the timing of notifying the approach of an object to the vehicle 2 from the wireless communication terminal 15, the communication interface 21 sends the information to the processor 23. In addition, when the communication interface 21 receives a notification of warning the approach of an object to the vehicle 2 from the processor 23, the communication interface 21 outputs the notification to the notification device 14. In addition, when the communication interface 21 receives information indicating the reaction time generated by the driver from the notification of the approach of the object to the vehicle 2 and the driver's familiarity with the environment from the processor 23, the communication interface 21 outputs the information to the wireless communication terminal 15.

[0037] The memory 22 is an example of a storage unit, and has, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. Moreover, the memory 22 stores various algorithms and various data used in the driving assistance processing executed by the processor 23 of the ECU 3. For example, the memory 22 stores various parameters used in the detection of objects based on the image received from the camera 12 and the determination of whether the detected object is approaching the vehicle 2. In addition, the memory 22 stores identification information such as the driver's identification number and facial image for identifying the driver, familiarity information indicating the driver's familiarity with the environment, and information indicating the relationship between the driver's familiarity with the environment and the timing of notifying the approach of the object. In addition, the driver's identification information and familiarity information are stored in the memory 22 for each driver registered for the vehicle 2. Further, the memory 22 temporarily stores positioning information, images, driver images, and various data generated in the middle of the driving assistance processing.

[0038] In the present embodiment, the familiarity information can be set to, for example, the number of years the driver has lived in the area where he currently lives. This is because it can be assumed that the longer the number of years of residence, the more familiar the driver is with driving in the area. In this case, the information indicating the driver's place of residence and the number of years of residence is input, for example, via a user interface (not shown) provided in the cabin of the vehicle 2 when the driver is registered in the vehicle 2, and is stored in the memory 22. In addition, the driver's place of residence is defined as, for example, a predetermined area (for example, a circular area with a radius of several km) centered on the address where the driver lives or a predetermined administrative division (for example, any one of a city, a street, and a village) including the address. In addition, the familiarity information is not limited to the above examples, and can also be the number of trips in each predetermined area or a predetermined interval of each road. In this case, in the memory 22, a counter indicating the number of trips in each predetermined area or a predetermined interval of each road is stored for each registered driver. Moreover, the processor 23 only needs to refer to the positioning information and the map information to specify the area or road section in which the vehicle 2 is traveling, and increase the value of the counter of the specified area or road section by 1 for the driver in driving. Furthermore, when the driver is registered in the vehicle 2, the driver image representing the driver's face captured by the driver monitoring camera 13 can be stored in the memory 22 as the driver's facial image. In the following, the driver registered in the vehicle 2 is sometimes referred to as a registered driver.

[0039] In addition, the initial information indicating the relationship between the driver's familiarity with the environment and the timing of notifying the approach of an object is stored in the memory 22, for example, when the vehicle 2 leaves the factory. Before receiving the information indicating the relationship between the driver's familiarity with the environment and the timing of notifying the approach of an object from the server 4, it is only necessary to set the timing of notifying according to the initial information. Moreover, when the ECU 3 receives the information indicating the relationship between the driver's familiarity with the environment and the timing of notifying the approach of an object from the server 4, it is only necessary to update the above initial information with the received information. Thereafter, it is only necessary to set the timing of notifying according to the updated information indicating the relationship between the driver's familiarity with the environment and the timing of notifying the approach of an object.

[0040] The processor 23 includes one or more CPUs (Central Processing Units) and peripheral circuits thereof. The processor 23 may further include other arithmetic circuits such as a logic operation unit, a numerical operation unit, or a graphics processing unit. Furthermore, the processor 23 executes driving assistance processing.

[0041] Figure 4 2 is a functional block diagram of the processor 23 related to the driving assistance process. The processor 23 includes a recognition unit 31, a detection unit 32, a proximity determination unit 33, a notification unit 34, and a reaction time notification unit 35. These parts of the processor 23 are, for example, functional modules implemented by a computer program running on the processor 23. Alternatively, these parts of the processor 23 may be dedicated operation circuits provided in the processor 23.

[0042] The recognition unit 31 recognizes the driver based on the driver image. Therefore, the recognition unit 31 compares, for example, the driver image obtained from the driver monitoring camera 13 after the ignition switch of the vehicle 2 is turned on, and the facial images of each registered driver stored in the memory 22. The recognition unit 31 only needs to compare the driver image and the facial images of each registered driver according to any one of the facial authentication methods used to compare the face represented in the image. For example, the recognition unit 31 uses template matching or a recognizer to detect each feature point of the face (for example, the inner corner of the eye, the outer corner of the eye, the tip of the nose, and the corner of the mouth) from the driver image, and calculates information representing the positional relationship between the detected feature points. Then, the recognition unit 31 only needs to compare the information representing the calculated positional relationship with the information representing the positional relationship between the feature points also detected from the facial images of each registered driver, so as to determine the facial image that is most similar to the face represented in the driver image. Then, the recognition section 31 recognizes the registered driver corresponding to the facial image determined to be most similar to the face represented in the driver monitoring image as the driver who is driving the vehicle 2 .

[0043] The identification unit 31 notifies the notification unit 34 of the identification number of the driver who is driving. In addition, the driver who is driving only needs to be identified once. Therefore, when the identification number of the driver who is driving is notified to the notification unit 34, the processing of the identification unit 31 may not be performed again until the ignition switch of the vehicle 2 is turned on again after being turned off.

[0044] The detection unit 32 detects objects to be detected that are located around the vehicle 2 based on the image whenever the ECU 3 receives an image from the camera 12. In the present embodiment, the objects to be detected are objects that are at risk of colliding with the vehicle 2, such as moving objects such as other vehicles or pedestrians, and structures such as guardrails, electric poles, or fallen objects on the road on which the vehicle 2 is traveling or around it. Hereinafter, the objects to be detected may be referred to simply as objects or target objects.

[0045] The detection unit 32 detects the target object represented in the image, for example, by inputting the image into the recognizer. As such a recognizer, the detection unit 32 can use a recognizer based on AI or artificial intelligence technology, in particular, a recognizer based on machine learning or deep learning. More specifically, in the detection unit 32, a recognizer based on a so-called neural network, such as a deep neural network (DNN) having a convolutional neural network (CNN) architecture such as SingleShotMultiBoxDetector or FasterR-CNN can be used as a recognizer. Alternatively, the detection unit 32 can also use a recognizer based on other machine learning methods such as an AdaBoost recognizer. Such a recognizer uses big data such as a large number of teaching images to implement pre-learning in order to detect the object to be detected from the image.

[0046] The recognizer outputs information indicating the area indicated by the detected object (hereinafter referred to as the object area). For example, the recognizer outputs a circumscribed rectangle that surrounds the object area as such information. Therefore, the detection unit 32 sends information indicating the object area to the proximity determination unit 33 for each of the detected objects.

[0047] According to a modified example, a distance sensor (not shown) may be provided on the vehicle 2 for measuring the distance to an object existing around the vehicle 2 for each direction and outputting a distance measurement signal indicating the measurement result. In this case, the detection unit 32 may also detect the object existing around the vehicle 2 based on the distance measurement signal from the distance sensor. In addition, the distance sensor becomes another example of a sensor that obtains a sensor signal indicating the condition around the vehicle, and the distance measurement signal becomes another example of a sensor signal. In this case, the detection unit 32 only needs to input the distance measurement signal into the identifier to which pre-learning is implemented in a manner of detecting the object around the vehicle 2 according to the distance measurement signal, thereby detecting the object around the vehicle 2. Alternatively, the detection unit 32 may also detect the object around the vehicle 2 according to other methods of detecting the object according to the distance measurement signal. In this case, the detection unit 32 sends information indicating the direction and distance of the object to the proximity determination unit 33 for each of the detected objects.

[0048] The approach determination unit 33 determines whether each object detected by the detection unit 32 is close to the extent that there is a risk of collision with the vehicle 2. To this end, the approach determination unit 33 tracks each object detected by the detection unit 32 to predict the trajectory of the object until a predetermined time. The approach determination unit 33 calculates the distance between the object at the position on the predicted trajectory at each future time point (hereinafter, sometimes referred to as the predicted position) and the predicted position of the vehicle 2 for each object being tracked. Then, the approach determination unit 33 determines that the object being tracked is close to the extent that there is a risk of collision with the vehicle 2 when the distance from the predicted position of the vehicle 2 at any time point for any object being tracked becomes less than a predetermined threshold. Hereinafter, the predicted time from the current time until the distance between the predicted position of the object being tracked and the predicted position of the vehicle 2 becomes less than a predetermined threshold is referred to as the collision prediction time.

[0049] In this case, the proximity determination unit 33 tracks the object represented in the object area by applying an optical flow-based tracking process such as the Lucas-Kanade method to the object area of ​​interest in the latest image obtained by the camera 12 and the object area in the past image. Therefore, the proximity determination unit 33 extracts multiple feature points from the object area by applying a filter for feature point extraction such as SIFT or Harris operator to the object area of ​​interest. Then, the proximity determination unit 33 only needs to specifically specify the corresponding point in the object area in the past image according to the applied tracking method for each of the multiple feature points to calculate the optical flow. Alternatively, the proximity determination unit 33 may also track the object represented in the object area by applying other tracking methods that are applied to the tracking of moving objects detected from the image to the object area of ​​interest in the latest image and the object area in the past image.

[0050] The approach determination unit 33 performs viewpoint conversion processing for each object being tracked using information such as the optical axis direction, focal length, and installation height of the camera 12, thereby converting the coordinates in the image of the object into coordinates on the bird's-eye view image based on the position of the camera 12 (bird's-eye coordinates). Then, the approach determination unit 33 performs prediction processing using Kalman filter (Kalman filter) or particle filter (particle filter) for each bird's-eye coordinate obtained from a series of images obtained during the tracking process, using the position of the vehicle 2 when the series of images are obtained. Thus, the approach determination unit 33 can infer the predicted trajectory of the object until a predetermined time ago. In addition, the position of the vehicle 2 when each image is obtained can be set to, for example, the position of the vehicle 2 indicated in the positioning information when the image is obtained. Alternatively, the position of the vehicle 2 when each image is obtained can also be inferred by comparing the image with the map information and obtaining the position of the vehicle 2 that is most consistent with each ground object indicated in the image and the corresponding ground object indicated in the map information.

[0051] In addition, when an object is detected based on the ranging signal, the approach determination unit 33 only needs to perform prediction processing based on the position of the vehicle 2 when each ranging signal is obtained and the direction and distance from the vehicle 2 to the object in each ranging signal, thereby inferring the predicted trajectory of the object.

[0052] Furthermore, the approach determination unit 33 only needs to calculate the predicted position of the vehicle 2 at each time point until the predetermined time based on the yaw rate, vehicle speed and acceleration of the vehicle 2 in the most recent predetermined period. In addition, the ECU 3 only needs to obtain the yaw rate, vehicle speed and acceleration of the vehicle 2 from a yaw rate sensor (not shown), a vehicle speed sensor (not shown) and an acceleration sensor (not shown) mounted on the vehicle 2.

[0053] According to the modified example, the ECU 3 may control various parts of the vehicle 2 so that the vehicle 2 travels along the lane (hereinafter referred to as the own lane) in which the vehicle 2 is located. In such a case, the approach determination unit 33 may determine that any object being tracked is close to the extent that there is a risk of collision with the vehicle 2 when the object is located in the own lane and in front of the vehicle 2. In this case, the approach determination unit 33 only needs to divide the estimated value of the distance to the object by the current speed of the vehicle 2 to calculate the collision prediction time.

[0054] When the approach determination unit 33 determines that any object being tracked is approaching to a degree that there is a risk of collision with the vehicle 2, the approach determination unit 33 notifies the notification unit 34 of the determination result. Furthermore, the approach determination unit 33 notifies the notification unit 34 of the predicted collision time.

[0055] When the notification unit 34 receives the determination result that any object being tracked is approaching to a degree that is dangerous to collide with the vehicle 2 from the approach determination unit 33, it reads the time threshold representing the timing of notifying the approach of the object to the vehicle 2 according to the driver's familiarity with the environment from the memory 22. In addition, the information representing the relationship between the driver's familiarity with the environment and the time threshold is an example of the above-mentioned information representing the relationship between the driver's familiarity with the environment and the timing of notifying the approach of the object. At this time, the notification unit 34 reads the information representing the driver's familiarity with the environment based on the identification information of the driver notified from the identification unit 31.

[0056] As described above, when the information indicating the driver's familiarity with the environment is the driver's residence and the number of years of residence, the notification unit 34 determines whether the current position of the vehicle 2 specified by the positioning information received from the GPS receiver 11 is the driver's residence. Then, the notification unit 34 reads the time threshold corresponding to the driver's number of years of residence from the memory 22 when the current position of the vehicle 2 is the driver's residence. On the other hand, when the current position of the vehicle 2 is not the driver's residence, the notification unit 34 reads the time threshold equivalent to the driver's residence from the memory 22. In addition, it is assumed that the longer the number of years of residence, the more familiar the driver is with the driving of the vehicle 2 around his residence, and the faster his reaction to the notification of the approach of the object to the vehicle 2. Therefore, in the initial information indicating the relationship between the driver's familiarity with the environment and the timing of the notification of the approach of the object, the longer the number of years of residence, the shorter the time threshold is set. Furthermore, when the current position of the vehicle 2 is not the driver's residence, there is a possibility that the driver is not familiar with driving the vehicle 2 near the current position of the vehicle 2. Therefore, the time threshold when the current position of the vehicle 2 is not the driver's residence is set longer than the time threshold when the current position of the vehicle 2 is the driver's residence.

[0057] Furthermore, in the case where the information indicating the degree of familiarity of the driver with the environment is the number of times the driver has traveled for each predetermined area or for each predetermined section of the road, the notification unit 34 reads the number of times the driver has traveled with respect to the area or road section including the current position of the vehicle 2 from the memory 22. Then, the notification unit 34 reads the time threshold value corresponding to the number of times the driver has traveled from the memory 22. In this case, it is assumed that the greater the number of times the driver has traveled with respect to the area or road section including the current position of the vehicle 2, the more familiar the driver is with driving the vehicle 2 near the current position of the vehicle 2, and the faster the driver's reaction to the notification of the approach of the object to the vehicle 2 is. Therefore, in the initial information indicating the relationship between the degree of familiarity of the driver with the environment and the timing of the notification of the approach of the object, the greater the number of times the driver has traveled, the shorter the time threshold value is set.

[0058] The notification unit 34 compares the collision prediction time with the read time threshold. Then, when the collision prediction time is less than the time threshold, the notification unit 34 outputs a notification warning of the object approaching the vehicle 2 to the notification device 14, thereby notifying the driver of the approach. On the other hand, when the collision prediction time is longer than the time threshold, the notification unit 34 does not notify the driver of the object approaching the vehicle 2. Thus, the notification unit 34 can ensure that the driver has sufficient time to deal with the object approaching the vehicle 2, while preventing the notification of the approach from being too early and annoying the driver.

[0059] Figure 5A as well as Figure 5B , respectively, are diagrams showing an example of the relationship between the driver's familiarity with the environment and the timing of notifying the approach of an object. Figure 5A as well as Figure 5B In , the horizontal axis represents time. In addition, suppose that Figure 5A The length of the collision prediction time Tp shown in Figure 5B The lengths of the collision prediction times Tp shown in are equal.

[0060] exist Figure 5A, the figure shows a situation where the driver is relatively familiar with the environment. In this example, since the driver is relatively familiar with the environment, the time threshold Thm is set relatively short, so that the collision prediction time Tp is longer than the time threshold Thm. Therefore, at the current time t, the approach of the object will not be notified. Then, based on the behavior of the object after the current time, if the collision prediction time Tp is shortened to become less than the time threshold Thm, the approach of the object will be notified to the driver at this time point. On the other hand, based on the behavior of the object after the current time, if the collision between the object and the vehicle 2 is not predicted, the approach of the object will remain unchanged without being notified to the driver, so that there is no need to implement useless notifications for the driver.

[0061] exist Figure 5B , the figure shows a case where the driver is not familiar with the environment. In this example, since the driver is not familiar with the environment, the time threshold Thm is set relatively long, so that the collision prediction time Tp is shorter than the time threshold Thm. Therefore, at the current time t, the driver is informed of the approach of the object. In this example, since the driver is informed of the approach of the object even if the collision prediction time Tp is relatively long, even if the driver is not familiar with the environment, sufficient time is ensured for the driver to deal with the approach of the object.

[0062] The reaction time notification unit 35 measures the reaction time (hereinafter sometimes simply referred to as the reaction time) from the notification of the approach of the object to the vehicle 2 to the driver's operation for avoiding the collision with the object. The operation for avoiding the collision is, for example, the operation of the brake for decelerating the vehicle 2 or the operation of the steering device for changing the forward route of the vehicle 2. Then, the reaction time notification unit 35 transmits information indicating the reaction time and the driver's familiarity with the environment at the position of the vehicle 2 when the approach of the object to the vehicle 2 is notified to the server 4 via the wireless communication terminal 15.

[0063] Figure 6 1 is an operation flow chart of the driving assistance process in the ECU 3. Whenever the processor 23 of the ECU 3 receives an image from the camera 12, it executes the driving assistance process according to the following operation flow chart.

[0064] Every time the ECU 3 receives an image from the camera 12 , the detection unit 32 of the processor 23 detects an object to be detected that is located around the vehicle 2 from the image (step S101 ).

[0065] The approach determination unit 33 of the processor 23 tracks each object detected by the detection unit 32, thereby obtaining the predicted trajectory of the object until a predetermined time (step S102). Then, the approach determination unit 33 determines whether the distance between the predicted position on the predicted trajectory and the predicted position of the vehicle 2 at each time point until a predetermined time is less than a predetermined threshold value Thd for each object being tracked (step S103). When the distance between the predicted position on the predicted trajectory and the predicted position of the vehicle 2 at any time point is less than the threshold value Thd for any object (step S103-yes), the approach determination unit 33 determines that the object is close to the extent that there is a risk of collision with the vehicle 2 (step S104).

[0066] The notification unit 34 of the processor 23 reads the time threshold Thm corresponding to the driver's familiarity with the environment from the memory 22 (step S105). Then, the notification unit 34 determines whether the collision prediction time until the predicted time when the distance between the predicted position of the object and the predicted position of the vehicle 2 becomes less than the predetermined threshold Thd becomes less than the time threshold Thm for the object determined to be close to the vehicle 2 (step S106). When the collision prediction time is less than the time threshold Thm (step S106-yes), the notification unit 34 notifies the driver of the object's approach to the vehicle 2 via the notification device 14 (step S107). Thereafter, the reaction time notification unit 35 of the processor 23 measures the reaction time from the notification of the object's approach to the vehicle 2 to the driver's operation for avoiding the collision with the object. Then, the reaction time notification unit 35 sends information indicating the reaction time and the driver's familiarity with the environment to the server 4 via the wireless communication terminal 15 (step S108). Then, the processor 23 ends the driving assistance process.

[0067] On the other hand, when the distance between the predicted position on the predicted trajectory and the predicted position of the vehicle 2 is not less than the predetermined threshold value Thd for any object being tracked (step S103-No), the approach determination unit 33 determines that there is no object approaching to the extent that there is a risk of collision with the vehicle 2 (step S109). Then, the processor 23 ends the driving assistance process. In addition, when the collision prediction time is longer than the time threshold value Thm in step S106 (step S106-No), the processor 23 ends the driving assistance process without notifying the driver of the approach of the object.

[0068] Next, the server 4 will be described. Figure 74 is a hardware structure diagram of the server 4. The server 4 has a communication interface 41, a storage device 42, a memory 43, and a processor 44. The communication interface 41, the storage device 42, the memory 43, and the processor 44 are connected together via a signal line. The server 4 may further have an input device such as a keyboard and a mouse, and a display device such as a liquid crystal display.

[0069] The communication interface 41 is an example of a communication unit, and has an interface circuit for connecting the server 4 to the communication network 5. Furthermore, the communication interface 41 is configured to be able to communicate with the vehicle 2 via the communication network 5 and the wireless base station 6. That is, the communication interface 41 sends information indicating the driver's reaction time and the driver's familiarity with the environment, etc., received from the vehicle 2 via the wireless base station 6 and the communication network 5 to the processor 44. In addition, the communication interface 41 sends information indicating the relationship between the driver's familiarity with the environment and the timing of notifying the approach of an object to the vehicle 2, etc., received from the processor 44, to the vehicle 2 via the communication network 5 and the wireless base station 6.

[0070] The storage device 42 is an example of a storage unit, and has, for example, a hard disk device or an optical recording medium and an access device thereof. Furthermore, the storage device 42 stores information indicating the driver's reaction time and the driver's familiarity with the environment. The storage device 42 may further store identification information of the vehicle 2. Furthermore, the storage device 42 may also store a computer program executed on the processor 44 for performing a learning process, wherein the learning process is used to learn the relationship between the driver's familiarity with the environment and the timing of notifying the approach of an object to the vehicle 2. Further, the storage device 42 may also store information indicating the relationship between the driver's familiarity with the environment and the timing of notifying the approach of an object to the vehicle 2.

[0071] The memory 43 is another example of a storage unit, and includes, for example, a nonvolatile semiconductor memory and a volatile semiconductor memory, and temporarily stores various data generated during the execution of the learning process and various data acquired through communication with the vehicle 2 .

[0072] The processor 44 is an example of a control unit and has one or more CPUs (Central Processing Units) and peripheral circuits thereof. The processor 44 may further have other operation circuits such as a logic operation unit or a numerical operation unit. Furthermore, the processor 44 performs a learning process when the information indicating the driver's reaction time and the driver's familiarity with the environment accumulates to a predetermined amount or more.

[0073] Figure 8 4 is a functional block diagram of the processor 44 related to the learning process. The processor 44 has a learning unit 51 and a notification unit 52. These various units of the processor 44 are functional modules implemented by a computer program executed on the processor 44. Alternatively, these various units of the processor 44 may also be dedicated operation circuits provided in the processor 44.

[0074] The learning unit 51 calculates the timing of notifying the approach of an object to the vehicle 2 according to the driver's familiarity with the environment and the reaction time. For example, the learning unit 51 calculates the average value of the reaction time for each driver's familiarity with the environment, for example, for each number of years of residence or for each number of times of driving. Then, the learning unit 51 sets the value obtained by adding a predetermined offset time to the average value of the reaction time related to the familiarity of each driver as the timing of notifying the approach of an object to the vehicle 2 related to the familiarity (the above-mentioned time threshold). Thus, the learning unit 51 can set the appropriate timing of notification according to the driver's familiarity with the environment. Alternatively, the learning unit 51 uses the combination of the familiarity of each driver and the reaction time of each driver as each data included in the big data, and uses it in the learning model for calculating the timing of notifying the approach of an object to the vehicle 2 according to the driver's familiarity with the environment. Such a learning model can be set as a learning model based on AI or artificial intelligence. More specifically, such a learning model can be a learning model based on machine learning or deep learning such as a neural network. The learning unit 51 may calculate the timing of reporting the degree of familiarity with the environment for each driver using such a learning model.

[0075] When the learning unit 51 calculates the timing of notification for each driver's familiarity with the environment, the notification unit 52 generates notification timing information including the timing of notification of the approach of an object to the vehicle 2 according to each driver's familiarity with the environment. Then, the notification unit 52 notifies the vehicle 2 of the generated notification timing information via the communication interface 41, the communication network 5, and the wireless base station 6.

[0076] As described above, the driving assistance device determines the timing to notify the driver of the object approaching the vehicle based on the information indicating the driver's familiarity with the environment. Therefore, the driver can be notified of the object approaching the vehicle at an appropriate timing.

[0077] According to a modified example, the timing of notifying the approach of an object to the vehicle 2 may be determined not only based on the driver's familiarity with the environment but also based on the driver's driving proficiency for the vehicle 2 or the environment itself around the vehicle 2. For example, it is assumed that the lower the driver's driving proficiency for the vehicle 2, the longer the driver's reaction time from the notification of the approach of the object to the vehicle 2. In addition, it is preferred that the environment around the vehicle 2 is more difficult for the driver to drive the vehicle 2, and the driver is notified of the approach of the object to the vehicle 2 as early as possible. Therefore, the notification unit 34 may also correct the timing of notification determined based on the driver's familiarity with the environment in the following manner: the lower the driver's driving proficiency for the vehicle 2 or the more difficult the environment around the vehicle 2 is to drive, the earlier the notification is implemented. For example, the notification unit 34 only needs to correct the time threshold in such a way that the above-mentioned time threshold becomes longer as the driver's driving proficiency for the vehicle 2 is lower or the more difficult the environment around the vehicle 2 is to drive. In addition, the driver's driving proficiency for the vehicle 2 can be set, for example, as the number of years counted from the time the driver purchased the vehicle 2, or the number of years counted from the time the driver obtained a driver's license. In this case, the shorter these years, the lower the driver's driving proficiency for the vehicle 2. In addition, these years, like the information indicating the driver's place of residence and years of residence, can be input through a user interface (not shown) provided in the cabin of the vehicle 2 when the driver is registered in the vehicle 2 and stored in the memory 22. In addition, regarding the environment around the vehicle 2, for example, it is assumed that it is difficult to drive at night compared to daytime, it is difficult to drive on rainy days compared to sunny days, it is difficult to drive on narrow roads compared to wide roads, or it is difficult to drive at places with confluence compared to places without confluence. Therefore, when the current time obtained from a clock (not shown) provided on the vehicle 2 is included in the time period corresponding to the night, the notification unit 34 corrects the timing of notification determined according to the driver's familiarity with the environment to implement notification more in advance. Alternatively, the notification unit 34 corrects the timing of notification determined based on the driver's familiarity with the environment by advancing the timing of notification as the amount of rain detected by the rain sensor (not shown) provided on the vehicle 2 increases. Alternatively, the notification unit 34 refers to the current position of the vehicle 2 indicated in the positioning information and the map information to determine whether the current position of the vehicle 2 is a confluence point or whether the width of the road at the current position of the vehicle 2 is less than a predetermined width.Then, when the current position of the vehicle 2 is the merging point or the width of the road at the current position of the vehicle 2 is less than the predetermined width, the notification unit 34 corrects the timing of notification determined based on the driver's familiarity with the environment to perform notification more early. In this way, the notification unit 34 corrects the timing of notification of the approach of the object based on the driver's driving proficiency for the vehicle 2 or the environment around the vehicle 2, thereby being able to notify the driver of the approach at a more appropriate timing.

[0078] In addition, the learning unit 51 of the processor 44 of the server 4 can also learn the timing of notification of each combination based on the driver's familiarity with the environment, the driver's driving proficiency, and the reaction time of each combination of the environment around the vehicle 2, similarly to the above-mentioned embodiment. In this case, the reaction time notification unit 35 of the processor 23 of the ECU 3 only needs to send the information indicating the driver's driving proficiency and the environment around the vehicle 2 to the server 4 together with the information indicating the reaction time and the familiarity.

[0079] According to another modification, the processor 23 of the ECU 3 may also perform the same learning process as the learning unit 51 of the processor 44 of the server 4. In this case, the reaction time notification unit 35 may also store information indicating the reaction time and the driver's familiarity with the environment in the memory 22 so that the learning unit 51 provided in the processor 23 can use it. In addition, the server 4 may be omitted. Furthermore, in the case where the driver of the vehicle 2 is limited to a person specifically designated in advance, the processing of the recognition unit 31 may be omitted.

[0080] The ECU3 based on the above-mentioned embodiment or modified example can also be installed in a vehicle that can apply not only manual driving control but also automatic driving control. In addition, the computer program that realizes the functions of the processor 23 of the ECU3 based on the above-mentioned embodiment or modified example can also be provided in the form of a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium or an optical recording medium.

[0081] As described above, those skilled in the art can implement various modifications within the scope of the present invention in a manner matching the implemented embodiments.

Claims

1. A driving assistance device, comprising: a storage unit that stores information indicating the driver's familiarity with the environment, including the driver's residence and the number of years the driver has lived at the residence; a detection unit that detects an object existing around the vehicle based on a sensor signal indicating a condition around the vehicle obtained by a sensor mounted on the vehicle; an approach determination unit that determines whether the object is approaching to such an extent that there is a possibility of a collision with the vehicle, and, if it is determined that the object is approaching to such an extent that there is a possibility of a collision with the vehicle, calculates a predicted time until the collision; a notification unit that, when it is determined that the object is approaching to such an extent that there is a possibility of a collision with the vehicle, notifies the driver of the vehicle of the approach via a notification device provided on the vehicle at a timing corresponding to the driver's familiarity with the environment, The notification unit notifies the driver of the approach when the predicted time becomes below a time threshold indicating the timing, which is set shorter as the driver's familiarity with the environment increases, and when the position of the vehicle when it is determined that the object is approaching to a degree that there is a possibility of a collision with the vehicle is the driver's residence, the time threshold is set shorter the longer the driver has lived there.

2. The driving assistance device according to claim 1, wherein: The notification unit sets the time threshold shorter when the position of the vehicle is the driver's residence when it is determined that the object is close enough to have a possibility of colliding with the vehicle than when the position is not the driver's residence.

3. The driving assistance device according to claim 1, wherein: The information indicating the driver's familiarity with the environment includes the number of times the driver has traveled in each predetermined area or predetermined road section. The notification unit sets the time threshold to be shorter the more times the driver travels in the area or road section including the position of the vehicle when it is determined that the object approaches to such an extent that there is a possibility of collision with the vehicle.

4. The driving assistance device according to any one of claims 1 to 3, wherein: There is also a learning unit that records the reaction time from the time the driver is notified of the approach of an object that is close to the extent that there is a possibility of collision with the vehicle until the driver performs an operation of the vehicle to avoid the collision, and learns the timing based on the reaction time and the driver's familiarity with the environment at the position of the vehicle when it is determined that the object is close to the extent that there is a possibility of collision with the vehicle.

5. A driving assistance method, the method comprising the following contents, namely: Based on a sensor signal indicating a condition around the vehicle obtained by a sensor mounted on the vehicle, an object existing around the vehicle is detected. determining whether the object is close enough to cause a collision with the vehicle, When it is determined that the object is approaching to such an extent that there is a possibility of a collision with the vehicle, a predicted time until the collision is calculated; When it is determined that the object is approaching to such an extent that there is a possibility of a collision with the vehicle, the driver is notified of the approach via a notification device provided on the vehicle at a timing corresponding to the driver's familiarity with the environment, including the driver's residence and the number of years the driver has lived at the residence, which are stored in a storage unit, Notifying the driver of the approach includes the following: when the predicted time becomes below a time threshold representing the timing, which is set shorter as the driver's familiarity with the environment increases, the driver is notified of the approach; and when the position of the vehicle when it is determined that the object is approaching to a degree that has the possibility of colliding with the vehicle is the driver's residence, the time threshold is set shorter the longer the driver has lived there.

6. A recording medium having recorded thereon a driving assistance computer program for causing a processor mounted on a vehicle to execute the following processing, wherein: Based on a sensor signal indicating a condition around the vehicle obtained by a sensor mounted on the vehicle, an object existing around the vehicle is detected. determining whether the object is close enough to cause a collision with the vehicle, When it is determined that the object is approaching to such an extent that there is a possibility of a collision with the vehicle, a predicted time until the collision is calculated; When it is determined that the object is approaching to such an extent that there is a possibility of a collision with the vehicle, the driver is notified of the approach via a notification device provided on the vehicle at a timing corresponding to the driver's familiarity with the environment, including the driver's residence and the number of years the driver has lived at the residence, which are stored in a storage unit, Notifying the driver of the approach includes the following: when the predicted time becomes below a time threshold representing the timing, which is set shorter as the driver's familiarity with the environment increases, the driver is notified of the approach; and when the position of the vehicle when it is determined that the object is approaching to a degree that has the possibility of colliding with the vehicle is the driver's residence, the time threshold is set shorter the longer the driver has lived there.

Images