Driving assistance device and driving assistance method

Abstract

The purpose of the present invention is to provide technology capable of appropriately calling the attention of a driver. This driving assistance device comprises: a recognition difficulty computation unit for determining the degree of recognition difficulty, which is the degree of difficulty that a driver has with recognition pertaining to a region in the surroundings of a vehicle; a latent risk extraction unit for extracting a latent risk on the basis of a risk value and the degree of recognition difficulty; a region identification unit for identifying, on the basis of the latent risk, an attention-calling region, which is a region to which the attention of the driver is to be called; a light emission method determination unit for determining, on the basis of the latent risk or behavior information, a light emission method relating to the attention-calling region; and a light emission control unit for controlling, on the basis of the light emission method, light emission by light emission devices of the vehicle.

Description

Driving assistance device and driving assistance method

[0001] The present disclosure relates to a driving assistance device and a driving assistance method.

[0002] In recent years, driving assistance devices that assist vehicle driving have been proposed. For example, Patent Literature 1 proposes a technology that calculates a potential risk value indicating the likelihood that obstacles such as pedestrians around the vehicle will jump out in front of the vehicle based on obstacle information around the vehicle and the vehicle's movement state, and illuminates obstacles whose potential risk value exceeds a threshold. This technology can alert the driver to obstacles whose potential risk value exceeds the threshold.

[0003] Japanese Patent Application Laid-Open No. 2022-100852

[0004] In conventional technology, even if a driver can easily notice and recognize an obstacle, if the potential risk value of the obstacle exceeds a threshold, the obstacle is illuminated. This can attract the driver's attention more than necessary, which can cause the driver to be unable to concentrate on driving.

[0005] Therefore, the present disclosure has been made in consideration of the above-mentioned problems, and aims to provide a technology that can appropriately attract the driver's attention.

[0006] The driving assistance device according to the present disclosure includes an acquisition unit that acquires vehicle surroundings information relating to at least the area around the vehicle; a risk candidate value calculation unit that calculates a risk value for an area of ​​a potential risk candidate based on the vehicle surroundings information or vehicle information including the state and position of the vehicle; a recognition difficulty calculation unit that calculates a recognition difficulty level, which indicates the degree of difficulty that the driver has in recognizing the area around the vehicle, based on the vehicle surroundings information or behavior information of the driver of the vehicle; a potential risk extraction unit that extracts potential risk candidates whose risk value is equal to or greater than a threshold and whose recognition difficulty level is equal to or greater than a threshold as potential risks based on the risk value and the recognition difficulty level; an area identification unit that identifies an attention area that is an area in which the driver should be alerted based on the potential risk; an illumination method determination unit that determines an illumination method for the attention area based on the potential risk or behavior information; and an illumination control unit that controls the illumination of the vehicle's illumination device based on the illumination method.

[0007] According to the present disclosure, a recognition difficulty level, which is the degree of difficulty for a vehicle driver to recognize an object, is calculated, a potential risk is extracted based on the potential risk candidate value and the recognition difficulty level, a warning area is identified based on the potential risk, a lighting method is determined based on the potential risk or behavior information, and lighting control of a lighting device is performed based on the determined lighting method. With this configuration, it is possible to appropriately draw the driver's attention.

[0008] The objects, features, aspects and advantages of the present disclosure will become more apparent from the following detailed description and the accompanying drawings.

[0009] 1 is a block diagram showing the configuration of a driving assistance system including a driving assistance device according to embodiment 1. FIG. 1 is a top view showing an example of a situation around a vehicle. FIG. 2 is a diagram showing an example of road environment factors. FIG. 3 is a diagram for explaining a light emitting method determination unit according to embodiment 1. FIG. 4 is a diagram showing the situation around a vehicle as seen from an onboard camera of the vehicle. FIG. 5 is a diagram showing an example of illumination from an exterior light emitting device. FIG. 6 is a diagram showing an example of lighting from an interior light emitting device. FIG. 7 is a flowchart showing the operation of the driving assistance device according to embodiment 1. FIG. 8 is a flowchart showing the operation of the driving assistance device according to embodiment 1. FIG. 9 is a flowchart showing the operation of the driving assistance device according to embodiment 1. FIG. 10 is a flowchart showing the operation of the driving assistance device according to embodiment 1. FIG. 11 is a block diagram showing the configuration of a driving assistance system including a driving assistance device according to embodiment 2. FIG. 12 is a diagram showing an example of lighting from a driving assistance device according to embodiment 2. FIG. 13 is a flowchart showing the operation of a driving assistance device according to embodiment 2. FIG. 14 is a block diagram showing the configuration of a driving assistance system including a driving assistance device according to embodiment 3. FIG. 15 is a block diagram showing the hardware configuration of a driving assistance device according to another modification. FIG. 16 is a block diagram showing the hardware configuration of a driving assistance device according to another modification.

[0010] <First Embodiment> Fig. 1 is a block diagram showing the configuration of a driving assistance system according to the first embodiment. The driving assistance system shown in Fig. 1 is capable of increasing the driver's awareness of potential risks that may exist around the vehicle by, for example, calling the driver's attention based on vehicle surroundings information relating to the area around the vehicle. In the following description, as a general rule, a vehicle equipped with a driving assistance system that calls the driver's attention will be referred to as a "vehicle," and other vehicles will be referred to as "other vehicles."

[0011] In order to alert the driver, the driving assistance system of this embodiment 1 performs at least one of illuminating the vicinity of a potential risk outside the vehicle from the vehicle's light-emitting device, and turning on or illuminating a portion of the light-emitting device inside the vehicle's cabin that is close to the potential risk.

[0012] The risk of coming into contact with a vehicle includes both obvious risks and potential risks. Obvious risks are objects that are detected by the surrounding environment sensors attached to the vehicle and may come into contact with the vehicle. Potential risks are objects that cannot be detected by the surrounding environment sensors and may come into contact with the vehicle.

[0013] 2(a) is a top view showing an example of the situation of a straight road around a vehicle, and FIG. 2(b) is a top view showing an example of the situation of an intersection around a vehicle. FIG. 2(a) shows a situation in which a brightly lit store 52, pedestrians 53 and 54, and a utility pole 55 are present beside the straight road on which a vehicle 51 is entering. The surrounding environment sensor of the vehicle 51 cannot detect the pedestrian 54 due to the obstruction of the utility pole 55, so the pedestrian 54 is a candidate for a potential risk. On the other hand, there is no object between the vehicle 51 and the pedestrian 53, and the surrounding environment sensor of the vehicle 51 can detect the pedestrian 53, so the pedestrian 53 is an apparent risk.

[0014] 2(b) shows a situation in which there is another vehicle 56 waiting to turn right at the intersection where vehicle 51 is entering, and there are also a motorcycle 57, another vehicle 58 waiting to turn right, a pedestrian 59, and a building wall 60 near the intersection. Because the surrounding environment sensor of vehicle 51 is unable to detect motorcycle 57 entering the intersection from the left side due to the obstruction of building wall 60, motorcycle 57 is a candidate for a potential risk. Because the surrounding environment sensor of vehicle 51 is unable to detect other vehicle 58 waiting to turn right due to the obstruction of other vehicle 56 waiting to turn right, other vehicle 58 waiting to turn right is a candidate for a potential risk.

[0015] On the other hand, there is no object between the vehicle 51 and the other vehicle 56 waiting to turn right, and the surrounding environment sensor of the vehicle 51 can detect the other vehicle 56 waiting to turn right, so the other vehicle 56 waiting to turn right is an apparent risk. Also, there is no object between the vehicle 51 and the pedestrian 59, and the surrounding environment sensor of the vehicle 51 can detect the pedestrian 59, so the pedestrian 59 is an apparent risk.

[0016] The driving assistance system of FIG. 1 includes a driving assistance device 1, a surrounding environment sensor 31, a vehicle travel sensor 32, a high-precision locator 33, a map database 34, an exterior light-emitting device 38, and an interior light-emitting device 39.

[0017] The driving assistance device 1 extracts potential risks around the vehicle based on information acquired by a surrounding environment sensor 31, a vehicle travel sensor 32, a high-precision locator 33, and a map database 34. Based on the potential risks, the driving assistance device 1 then controls light-emitting devices including at least one of an exterior light-emitting device 38 and an interior light-emitting device 39 to alert the driver. The components of the driving assistance device 1 will be described in detail later.

[0018] The surrounding environment sensor 31 is one or more sensors that detect the situation around the vehicle, and includes, for example, an on-board camera and a radar device. The on-board camera is a device that captures an image of the area ahead of the vehicle and generates an image of the area ahead of the vehicle, and outputs the image results including the image to the driving assistance device 1. The on-board camera includes, for example, a visible light camera and an infrared camera. The radar device is a device that detects objects around the vehicle using millimeter waves, and includes, for example, millimeter-wave radar and LiDAR (Light Detection and Ranging). The radar device outputs measurement results, such as the distance between the vehicle and an object present around the vehicle and the direction from the vehicle to the object, to the driving assistance device 1.

[0019] The vehicle travel sensor 32 is one or more sensors that detect the travel state of the vehicle, and includes, for example, a vehicle speed sensor and a steering angle sensor. The vehicle speed sensor measures the speed of the vehicle and outputs the result to the driving assistance device 1. The steering angle sensor measures the steering direction of the vehicle and outputs the result to the driving assistance device 1.

[0020] The high-precision locator 33 calculates the current position (e.g., latitude and longitude) of the vehicle with high precision based on positioning signals from GNSS (Global Navigation Satellite System) satellites, and outputs the calculated position to the driving assistance device 1 .

[0021] The map database 34 is a medium for storing map information. The map information stored in the map database 34 includes, for example, the positions of road lanes, shoulders, sidewalks, lane attributes (such as right-turn lanes), signs installed on the road, and information about buildings around the road.

[0022] The exterior light-emitting device 38 is a device that illuminates the area ahead of the vehicle and is capable of irradiating light onto at least one of a plurality of divided areas that divide the area ahead of the vehicle. The exterior light-emitting device 38 includes, for example, a light distribution illumination device such as an LED array type in which a plurality of light-emitting diodes (LEDs) are arranged in an array, or a scanning type using micro-electromechanical systems (MEMS). Note that the LED array type and the MEMS scanning type are known technologies, and therefore detailed description thereof will be omitted in this specification.

[0023] The interior light-emitting device 39 is a device capable of presenting various types of information to the driver inside the vehicle cabin. The interior light-emitting device 39 includes, for example, a linear light-emitting area on an instrument panel in which a plurality of LEDs are arranged in an array, and draws the driver's attention by partially lighting up the linear light-emitting area. Alternatively, the interior light-emitting device 39 irradiates light onto a windshield, also known as a windshield, causing the windshield to partially emit light and overlap with the view ahead, thereby drawing the driver's attention.

[0024] The light-emitting device including at least one of the exterior light-emitting device 38 and the interior light-emitting device 39 as described above may simply emit light, and the term "light-emitting" as used herein includes at least one of, for example, light irradiation, light distribution, light display, and light illumination.

[0025] <Driving Assistance Device 1> Next, a description will be given of the components of the driving assistance device 1. The driving assistance device 1 includes an information acquisition unit 11, a risk candidate value calculation unit 12, a recognition difficulty calculation unit 13, a potential risk extraction unit 14, an area identification unit 15, a light emission method determination unit 16, and a light emission control unit 17. The driving assistance device 1 may also include at least one of a surrounding environment sensor 31, a vehicle travel sensor 32, a high-precision locator 33, a map database 34, an exterior light-emitting device 38, and an interior light-emitting device 39.

[0026] <Information Acquisition Unit 11> The concept of an acquisition unit includes the information acquisition unit 11. The information acquisition unit 11 in Fig. 1 includes a vehicle state acquisition unit 11a, a vehicle position acquisition unit 11b, and a surrounding environment acquisition unit 11c.

[0027] The vehicle state acquisition unit 11a generates (acquires) vehicle state information indicating the vehicle's running state by recognizing the running state of the vehicle based on the measurement results of the vehicle running sensor 32. The vehicle state information includes, for example, information such as the vehicle's speed, acceleration, and moving direction (steering angle).

[0028] The vehicle position acquisition unit 11b generates (acquires) vehicle position information indicating the position of the vehicle on the map by recognizing the position of the vehicle on the map based on the current position of the vehicle calculated by the high-precision locator 33 and the map information in the map database 34. Note that the vehicle position acquisition unit 11b may also recognize the position of the vehicle on the map by other known methods. Note that, hereinafter, information including the vehicle state information and the vehicle position information may also be referred to as vehicle information.

[0029] The surrounding environment acquisition unit 11c generates (acquires) vehicle surrounding information relating to the surroundings of the vehicle by recognizing the situation around the vehicle based on the vehicle position information generated by the vehicle position acquisition unit 11b and the output results of the surrounding environment sensor 31.

[0030] The vehicle surroundings information includes driving environment information, obstacle information, and the image capture results of the on-board camera. The driving environment information is information that indicates the driving environment of the vehicle, such as weather, time of day, brightness around the vehicle, road signs, buildings, walls and roadside installations (buildings, utility poles, etc.), road shape, driving area (urban area, residential area, etc.), and the degree of vehicle congestion.

[0031] The obstacle information is information indicating obstacles present around the vehicle, and includes, for example, information on the type, position, size, and driving state of the obstacle. The type of obstacle includes, for example, information on vehicles, trucks, motorcycles, bicycles, pedestrians, etc. The driving state of the obstacle includes, for example, the driving state of other vehicles, such as a preceding vehicle, an oncoming vehicle, and a parked vehicle.

[0032] In the following description, the information acquisition unit 11 is assumed to acquire vehicle information including vehicle state information and vehicle position information, and vehicle surroundings information. However, the information acquisition unit 11 only needs to acquire information used by the risk candidate value calculation unit 12 and the recognition difficulty calculation unit 13, which will be described below, and is not necessarily required to acquire both the vehicle information and the vehicle surroundings information.

[0033] <Risk candidate value calculation unit 12> The risk candidate value calculation unit 12 obtains information on potential risk candidates based on the vehicle surroundings information or vehicle information acquired by the information acquisition unit 11. As described above, potential risks are objects that cannot be detected by the surrounding environment sensor 31, and potential risk candidates are candidates for potential risks. The information on potential risk candidates includes a potential risk candidate type, a potential risk candidate area, and a potential risk candidate value, which is a risk value, so obtaining information on potential risk candidates means obtaining a potential risk candidate value.

[0034] The potential risk candidate type is the type of potential risk candidate and indicates the type of moving object, such as a pedestrian or vehicle, that may appear from the driver's blind spot. The potential risk candidate area is an area where a potential risk candidate may exist and corresponds to the driver's blind spot area formed by structures on the road, such as walls and utility poles, and obstacles, such as other vehicles. The potential risk candidate value corresponds to the occurrence probability of the potential risk candidate in the potential risk candidate area.

[0035] Below, an example will be described in which the risk candidate value calculation unit 12 obtains information on potential risk candidates. As a first example, the risk candidate value calculation unit 12 obtains road environment factors as shown in Fig. 3 by extracting structures such as walls and utility poles on the road, and obstructions that may form blind spots for the driver, such as parked vehicles, from the driving environment information and obstacle information included in the vehicle surroundings information. Then, the risk candidate value calculation unit 12 obtains information on potential risk candidates by inputting the road environment factors into a potential risk candidate learning model.

[0036] The potential risk candidate learning model is a learning model that infers information about potential risk candidates from road environmental factors. The potential risk candidate learning model is generated by the risk candidate value calculation unit 12 learning (training) the relationship between road environmental factors and the presence or absence of potential risk candidates for scenes in which traffic accidents and near misses have occurred, extracted from, for example, traffic accident information and dashcam footage. This learning may use, for example, a statistical method such as logistic regression that uses training data including road environmental factors as explanatory variables and the presence or absence of potential risk candidates as a response variable, or deep learning that uses the training data.

[0037] The potential risk candidate learning model may be any learning model capable of inferring information about potential risk candidates from vehicle surrounding information. For example, the potential risk candidate learning model may be created for each type of moving object that poses a risk of jumping out, such as a vehicle, a motorcycle, a bicycle, or a pedestrian. Furthermore, for example, the potential risk candidate learning model may include a model that learns the relationship between whether or not a moving object will jump out, and a model that learns the type of moving object that will jump out.

[0038] In the first example above, a method has been described in which the risk candidate value calculation unit 12 obtains information on potential risk candidates while the vehicle is traveling, but the present invention is not limited to this. As a second example, information on potential risk candidates may be registered in advance at points in the map database 34 where traffic accidents and near misses have occurred. Then, when the point is located within a certain range ahead of the vehicle from the vehicle position indicated by the vehicle information, the risk candidate value calculation unit 12 may read the information on the potential risk candidate registered at the point.

[0039] Furthermore, the risk candidate value calculation unit 12 may calculate the potential risk candidate value using the method described above or another known method. Furthermore, the risk candidate value calculation unit 12 may correct the potential risk candidate value based on information about parked vehicles calculated from obstacle information. Furthermore, in the above description, the risk candidate value calculation unit 12 calculates the information about the potential risk candidate from road environment factors based on information about the vehicle's surroundings. However, the risk candidate value calculation unit 12 may also calculate the information about the potential risk candidate by taking vehicle information into consideration. For example, the risk candidate value calculation unit 12 may increase the potential risk candidate value the higher the vehicle speed, or may increase the potential risk candidate value of a potential risk candidate region in the direction indicated by the steering angle of the steering wheel.

[0040] <Recognition Difficulty Calculation Unit 13> The recognition difficulty calculation unit 13 calculates a recognition difficulty level for the area around the vehicle based on the image (i.e., the image capture result captured by the on-board camera) included in the vehicle surroundings information acquired by the information acquisition unit 11. This recognition difficulty level is the degree of difficulty for the driver to recognize the area, independent of the presence or absence of an obstruction that forms a blind spot area.

[0041] Below, an example of how the recognition difficulty calculation unit 13 calculates the recognition difficulty will be described. As a first example, the recognition difficulty calculation unit 13 generates a saliency map based on an image included in the vehicle surroundings information. The saliency map is a map that calculates the ease with which a person will gaze at an image for each pixel, and in the case of the present disclosure, it is a map that reflects the distribution pattern of areas that are easy for a driver to gaze at. Various methods have been proposed for calculating the saliency map, and since these are well known, their explanation will be omitted.

[0042] The recognition difficulty calculation unit 13 calculates a numerical value indicating the difficulty of the driver's attention as the recognition difficulty by reversing the magnitude relationship of the numerical value indicating the ease of attention in the saliency map, for example, by calculating the reciprocal of the numerical value. In other words, the recognition difficulty calculation unit 13 increases the recognition difficulty for areas with lower saliency in the saliency map.

[0043] As a second example, the recognition difficulty calculation unit 13 calculates the luminance, which is the brightness of an area in the image included in the vehicle surrounding information where the potential risk candidate value is equal to or greater than a threshold, and calculates the recognition difficulty level based on the luminance. For example, the recognition difficulty calculation unit 13 may divide the image area into partial areas of 16 x 16 pixels, calculate the average luminance values ​​of the pixels in the partial areas included in the potential risk candidate areas, and increase the recognition difficulty level of the divided area as the average value decreases. Alternatively, for example, the recognition difficulty calculation unit 13 may calculate the difference in luminance between adjacent divided areas in the horizontal and vertical directions. Then, since the greater the difference in luminance, the more noticeable the area is to the driver, the recognition difficulty calculation unit 13 may increase the recognition difficulty level of the divided area as the difference in luminance decreases.

[0044] <Potential Risk Extraction Unit 14> The potential risk extraction unit 14 extracts as potential risks potential risk candidates whose potential risk candidate value is equal to or greater than a threshold and whose recognition difficulty level is equal to or greater than a threshold, based on the potential risk candidate value calculated by the risk candidate value calculation unit 12 and the recognition difficulty level calculated by the recognition difficulty level calculation unit 13. For example, when the potential risk candidate value of a potential risk candidate area included in the information of the potential risk candidate is equal to or greater than a threshold and the recognition difficulty level of an area corresponding to the potential risk candidate area is equal to or greater than a threshold, the potential risk extraction unit 14 extracts the potential risk candidate as a potential risk.

[0045] <Area Identification Unit 15> The area identification unit 15 identifies an attention-calling area, which is an area in which the driver's attention should be called, based on the potential risk extracted by the potential risk extraction unit 14.

[0046] An example of how the area identification unit 15 identifies an attention-warning area will be described below. First, the area identification unit 15 determines a movement area of ​​the potential risk based on the potential risk candidate type and potential risk candidate area included in the information on the potential risk candidate extracted as a potential risk. For example, the area identification unit 15 determines an existence probability distribution, which is a probability distribution of the presence position of the potential risk after a certain time, based on the potential risk candidate type, and determines a movement area of ​​the potential risk by reflecting the existence probability distribution in the potential risk candidate area. The certain time here is the reaction time from when the driver is notified of a potential risk warning until he or she turns his or her gaze toward the potential risk, and is, for example, one second.

[0047] For example, the region identification unit 15 may use a distribution uniquely determined in advance for each potential risk candidate type as the presence probability distribution. Alternatively, for example, the region identification unit 15 may obtain the presence probability distribution by inputting the potential risk candidate type into a distribution pattern learning model. The distribution pattern learning model is a learning model that infers the presence probability distribution from the potential risk candidate type, and is generated by the region identification unit 15 learning (training) the relationship between the potential risk candidate type and the presence probability distribution. This learning uses training data including, for example, a presence probability distribution pattern based on possible actions of a mobile object, such as going straight, turning left, turning right, and changing lanes, the type of the mobile object, and the road environment factors shown in FIG. 3 . The presence probability distribution pattern is obtained based on, for example, trajectory data of a mobile object observed at fixed points by a camera installed on a road.

[0048] After determining the movement area of ​​the potential risk, the area identification unit 15 extracts an actual risk based on the obstacle information included in the vehicle surroundings information and the vehicle information. For example, the area identification unit 15 determines whether the obstacle will come into contact with the vehicle based on the positions of obstacles detected by the surrounding environment sensor 31, such as other vehicles (e.g., preceding vehicles and oncoming vehicles) and pedestrians, and the position, speed, and movement direction of the vehicle. Then, the area identification unit 15 extracts the obstacle determined to be in contact as an actual risk.

[0049] Then, if the actual risk does not exist within the movement region of the potential risk, the region specifying unit 15 sets the movement region as a potential risk region.

[0050] The area identification unit 15 then identifies an attention area based on the positional relationship between the vehicle and the potential risk area. For example, the area identification unit 15 calculates, as the notification target area, an area that the vehicle will reach in 1 to 6 seconds, preferably 2 to 6 seconds, from its current location, based on the vehicle's position, speed, and moving direction included in the vehicle information. The area identification unit 15 then identifies, as the attention area, a potential risk area that exists within the notification target area. By excluding potential risk areas outside the notification target area from the attention area in this way, it is possible to suppress attention alerts to attention areas that are highly likely to require a driver assistance response, such as a collision mitigation brake, and that cannot be addressed even if the driver is notified.

[0051] Finally, the area identification unit 15 determines whether or not there is an obstruction, such as a structure or an obstacle, on the line connecting the vehicle and the attention area. If the area identification unit 15 determines that there is no obstruction, it maintains the attention area as it is, but if it determines that there is an obstruction, it moves the attention area to near the edge of the obstacle.

[0052] <Light Emission Method Determining Unit 16 > The light emission method determining unit 16 determines a light emission method for the light emitting device of the vehicle in relation to the attention-attention area, based on the potential risk for which the attention-attention area has been identified by the area identifying unit 15 .

[0053] As a first example, the light emission method determination unit 16 determines the light emission control amount, such as the range, brightness, and mode (on or flashing) of the light emission (irradiation) of the light emitting device, as the light emission method based on the potential risk candidate value and the recognition difficulty level of the potential risk for which the warning area has been identified.

[0054] For example, if the light emission method determination unit 16 determines a light emission method in which the blinking rate of the light-emitting device increases as the potential risk candidate value increases, the driver's attention can be more effectively drawn to potential risks with large potential risk candidate values. Furthermore, if the light emission method determination unit 16 determines a light emission method in which at least one of the range, brightness, and color of the light emission (illumination) of the light-emitting device changes as the recognition difficulty increases, potential risks that are difficult to recognize can be made easier to recognize. Furthermore, for example, the light emission method determination unit 16 may determine a light emission method in which at least one of the range, brightness, and color of the light emission (illumination) of the light-emitting device changes based on the relative position between the potential risk and the vehicle, the speed of the vehicle, and the weather and brightness around the road on which the vehicle is traveling.

[0055] As a second example, the lighting method determination unit 16 may obtain road environment factors as shown in Fig. 3 from the driving environment information and obstacle information included in the vehicle surroundings information, and obtain the vehicle speed from the vehicle state information. The lighting method determination unit 16 may then input the determination information, including the road environment factors, the vehicle speed, the potential risk candidate value of the potential risk, and the recognition difficulty level, into a lighting method learning model to determine the lighting method.

[0056] The light emission method learning model is a learning model that infers the light emission method from the decision information, and is generated by the light emission method determination unit 16 learning (training) the relationship between the decision information and the light emission method for scenes where, for example, a traffic accident or a near miss has occurred. For this learning, for example, a statistical method such as regression using training data including the decision information as an explanatory variable and the light emission method as a target variable may be used, or deep learning using the training data may be used.

[0057] As a third example, as shown in Fig. 4, the light emission method determination unit 16 determines a light emission method for selectively emitting light from the vehicle exterior light emission device 38 and the vehicle interior light emission device 39 based on the comparison result between the potential risk candidate value and the light emission method threshold Thr and the comparison result between the recognition difficulty level and the light emission method threshold Thp. As a light emission method, it is preferable to perform notification by at least the vehicle exterior light emission device 38 when the potential risk candidate value and the recognition difficulty level are equal to or greater than the light emission method threshold Thr and the light emission method threshold Thp, respectively. With this configuration, the attention alert area or its vicinity is illuminated, thereby alerting the driver to the attention alert area, and the illumination of an attention alert area that is difficult for the driver to recognize allows the driver to easily recognize the situation in the attention alert area.

[0058] 4, when the potential risk candidate value and the recognition difficulty level are equal to or greater than the light-emitting method threshold Thr and the light-emitting method threshold Thp, respectively, only the vehicle exterior light-emitting device 38 issues a notification, but the vehicle interior light-emitting device 39 may also issue a notification. Also, in the light-emitting method of FIG. 4, when the potential risk candidate value is smaller than the light-emitting method threshold Thr, neither the vehicle exterior light-emitting device 38 nor the vehicle interior light-emitting device 39 issues a notification. However, this is not limited to this, and for example, even if the potential risk candidate value is smaller than the light-emitting method threshold Thr, the vehicle exterior light-emitting device 38 may issue a notification when the recognition difficulty level is equal to or greater than the light-emitting method threshold Thp.

[0059] <Light Emission Control Unit 17> The light emission control unit 17 controls the light emission of the light-emitting device based on the attention-attracting area. In the first embodiment, the light emission control unit 17 controls the light emission of the light-emitting device based on the attention-attracting area using the light emission method determined by the light emission method determination unit 16. This makes it possible to alert the driver to the presence of a potential risk.

[0060] Figures 5(a) and 5(b) are views of the situations shown in Figures 2(a) and 2(b) as viewed from an on-board camera of a vehicle. Figures 6(a) and 6(b) are views of an example of illumination by the exterior light-emitting device 38 in the situations shown in Figures 2(a) and 2(b), respectively. Figures 7(a) and 7(b) are views of an example of illumination by the interior light-emitting device 39 in the situations shown in Figures 2(a) and 2(b), respectively.

[0061] 5(a) and 6(a), the risk candidate value calculation unit 12 detects a pedestrian 54 who may be in the shadow of a utility pole 55 as a potential risk candidate. As shown in FIG. 5(a), the recognition difficulty calculation unit 13 increases the recognition difficulty of a dark area 61 in the distance that is not illuminated by vehicle lights or store lights.

[0062] The potential risk extraction unit 14 extracts, as a potential risk, a pedestrian 54 included in an area 61 with a high degree of difficulty in recognition based on the potential risk candidate value and the degree of difficulty in recognition of the pedestrian 54. The area identification unit 15 determines the movement area of ​​the pedestrian 54 extracted as a potential risk, and determines the movement area of ​​the pedestrian 54 where no apparent risk exists as a potential risk area 62 as shown in FIG.

[0063] The area identification unit 15 identifies a potential risk area 62 for a pedestrian 54 present within the notification target area around the vehicle as an attention area 63, as shown in Figure 6(a). Because there is no obstruction on the line connecting the attention area 63 and the vehicle 51, the area identification unit 15 maintains the attention area 63 as it is.

[0064] The exterior light-emitting device 38 includes a first illumination means, which is a passing headlight, and a second illumination means provided separately from the first illumination means. In the example of FIG. 6A , an illumination range 66 of the first illumination means and an illumination range 67 of the second illumination means are illustrated. The second illumination means may be a variable light distribution running light, or may be an illumination device provided separately for notifying the pedestrian of the attention-calling area. When the illumination range 67 of the second illumination means of the exterior light-emitting device 38 includes at least a portion of the attention-calling area 63 of the pedestrian 54, the light-emission control unit 17 controls the second illumination means of the exterior light-emitting device 38 to emit light in the attention-calling area 63 of the pedestrian 54.

[0065] In the situation shown in Figure 5(b), the risk candidate value calculation unit 12 detects, as potential risk candidates, another vehicle 58 waiting to turn right that may be located behind another vehicle 56 waiting to turn right, and a motorcycle 57 that may be located behind a building wall 60. As shown in Figure 5(b), the recognition difficulty calculation unit 13 increases the recognition difficulty of a dark area 61 that is not illuminated by light such as vehicle headlights. On the other hand, the recognition difficulty calculation unit 13 decreases the recognition difficulty of the area surrounding the other vehicle 58 waiting to turn right, because the area around the other vehicle 58 waiting to turn right is bright due to the lights of the other vehicle 58 waiting to turn right.

[0066] Based on the potential risk candidate value and the degree of difficulty of recognition, the potential risk extraction unit 14 extracts the two-wheeled vehicle 57 included in the area 61 with a high degree of difficulty of recognition as a potential risk, but does not extract the other vehicle 58 waiting to turn right as a potential risk.

[0067] The area identification unit 15 determines the movement area of ​​the two-wheeled vehicle 57 extracted as a potential risk, and determines the movement area of ​​the two-wheeled vehicle 57 where there is no apparent risk as a potential risk area 62, as shown in Figure 5(b). Note that even if another vehicle 58 waiting to turn right had been extracted as a potential risk in the situation of Figure 5(b), the area identification unit 15 would not determine the movement area of ​​the other vehicle 58 waiting to turn right as a potential risk area 62, because another vehicle 56 waiting to turn right, which is an apparent risk, is present within the movement area of ​​the other vehicle 58 waiting to turn right.

[0068] The area identification unit 15 identifies a potential risk area 62 for the motorcycle 57 that exists within the notification area around the vehicle as an attention area 63, as shown in Figure 6(b). Because a wall 60, which is an obstruction, exists on the line connecting the attention area 63 and the vehicle 51, the area identification unit 15 moves the position of the attention area 63 to the position of an attention area 63a near the end of the wall 60.

[0069] When the illumination range 67 of the second illumination means of the exterior light-emitting device 38 includes at least a portion of the attention-warning area 63a of the two-wheeled vehicle 57, the light-emitting control unit 17 controls the second illumination means of the exterior light-emitting device 38 to control the illumination of the attention-warning area 63a of the two-wheeled vehicle 57.

[0070] Although one attention warning area 63 is present in each of the examples of Figures 6(a) and 6(b), one or more attention warning areas 63 may be present. When multiple attention warning areas 63 are present, the multiple attention warning areas 63 may be illuminated. When illuminating all of the multiple attention warning areas 63 may hinder the driver's concentration on driving, only one attention warning area 63 may be illuminated. When one attention warning area 63 is illuminated from multiple attention warning areas 63, only the attention warning area 63 with the highest potential risk candidate value may be illuminated, or only the attention warning area 63 with the highest product of the potential risk candidate value and the recognition difficulty level may be illuminated. In this way, when multiple attention warning areas 63 exist, limiting the attention warning area 63 to be notified to the driver can prevent the driver from being distracted from driving. Below, an example will be described in which only the attention warning area 63 with the highest product of the potential risk candidate value and the recognition difficulty level is illuminated.

[0071] 7A is a diagram showing an example of the configuration of the interior light-emitting device 39, and FIG. 7B is a diagram showing an example of lighting of the interior light-emitting device 39. The interior light-emitting device 39 in FIG. 7A includes a linear light-emitting area 39a provided on the instrument panel of the vehicle. The interior light-emitting device 39 notifies the driver of the location of the attention-calling area 63 by partially lighting up the light-emitting area 39a. For example, as shown in FIG. 7B, the light-emission control unit 17 controls the light emission of the interior light-emitting device 39 so as to light up a portion 39b of the light-emitting area 39a that intersects with a line connecting the driver 71 and the attention-calling area 63, thereby causing the interior light-emitting device 39 to notify the driver of the location of the attention-calling area 63.

[0072] <Operation> Fig. 8 is a flowchart showing the overall operation of the driving assistance device 1 according to the present embodiment 1. The operation in Fig. 8 is repeatedly performed as needed.

[0073] In step S1, the information acquisition unit 11 acquires vehicle state information, vehicle position information, and vehicle surroundings information from information obtained from the surrounding environment sensor 31, vehicle travel sensor 32, high-precision locator 33, map database 34, and the like.

[0074] In step S2, the risk candidate value calculation unit 12 obtains information on potential risk candidates including potential risk candidate values ​​based on the vehicle surroundings information or vehicle information acquired by the information acquisition unit 11.

[0075] In step S3 , the recognition difficulty calculation unit 13 calculates the recognition difficulty level based on the image included in the vehicle surroundings information acquired by the information acquisition unit 11 .

[0076] In step S4 , the potential risk extraction unit 14 extracts a potential risk based on the potential risk candidate value calculated by the risk candidate value calculation unit 12 and the recognition difficulty calculated by the recognition difficulty calculation unit 13 .

[0077] In step S5, the area specifying unit 15 specifies an attention-request area in which the driver's attention should be called, based on the potential risks extracted by the potential risk extracting unit 14.

[0078] In step S6 , the light emission method determination unit 16 determines a light emission method for the attention area of ​​the light-emitting device based on the potential risk for which the attention area has been identified by the area identification unit 15 .

[0079] In step S7, the light emission control unit 17 notifies the driver of the attention area by controlling the light emission of the light emitting device based on the attention area using the light emission method determined by the light emission method determination unit 16. Some of the processes in FIG. 8 will be described in detail below.

[0080] <Recognition Difficulty Level Calculation Process> FIG. 9 is a flowchart showing the recognition difficulty level calculation process according to the first embodiment, that is, the process in step S3 of the recognition difficulty level calculation unit 13 of the driving assistance device 1 according to the first embodiment.

[0081] In step S11, the recognition difficulty calculation unit 13 generates a saliency map based on the image included in the vehicle surroundings information. In step S12, the recognition difficulty calculation unit 13 reverses the magnitude relationship of the numerical values ​​indicating the ease of gaze in the saliency map to obtain a numerical value indicating the difficulty of gaze by the driver as the recognition difficulty. Then, the processing of FIG. 9 ends.

[0082] 10 is a flowchart showing the potential risk extraction process according to the present embodiment 1, that is, the process in step S4 of the potential risk extraction unit 14 of the driving assistance device 1 according to the present embodiment 1. The potential risk extraction unit 14 performs the process of FIG. 10 for each potential risk candidate determined by the risk candidate value calculation unit 12.

[0083] In step S21, the potential risk extraction unit 14 determines whether the potential risk candidate value included in the information on the potential risk candidate calculated by the risk candidate value calculation unit 12 is equal to or greater than a first threshold. Note that the first threshold may be the same as the light emission method threshold Thr in Fig. 4. If the potential risk candidate value is equal to or greater than the first threshold, the process proceeds to step S22. If the potential risk candidate value is smaller than the first threshold, the potential risk candidate is not extracted as a potential risk, and the process in Fig. 10 ends.

[0084] In step S22, the potential risk extraction unit 14 extracts the recognition difficulty level of an area corresponding to the potential risk candidate area included in the information on the potential risk candidate from the recognition difficulty levels calculated by the recognition difficulty level calculation unit 13.

[0085] In step S23, the potential risk extraction unit 14 determines whether the extracted recognition difficulty level is equal to or greater than a second threshold value. Note that the second threshold value may be the same as the light emission method threshold value Thp in FIG. 4. If the recognition difficulty level is equal to or greater than the second threshold value, the process proceeds to step S24. If the recognition difficulty level is less than the second threshold value, the potential risk candidate is not extracted as a potential risk, and the process in FIG. 10 ends.

[0086] In step S24, the potential risk extraction unit 14 extracts the potential risk candidates as potential risks, and then the process of FIG. 10 ends.

[0087] <Area Identification Processing> FIG. 11 is a flowchart showing the area identification processing according to the first embodiment, that is, the processing in step S5 of the area identification unit 15 of the driving assistance device 1 according to the first embodiment.

[0088] In step S31, the area specifying unit 15 determines a movement area of ​​the potential risk based on the potential risk candidate type and potential risk candidate area included in the information of the potential risk candidate extracted as the potential risk.

[0089] In step S32, the area specifying unit 15 determines the apparent risk based on the obstacle information included in the vehicle surroundings information and the vehicle information.

[0090] In step S33, it is determined whether the actual risk exists within the movement range of the potential risk. If it is determined that the actual risk exists within the movement range of the potential risk, the process proceeds to step S39, and if it is determined that the actual risk does not exist within the movement range of the potential risk, the process proceeds to step S34.

[0091] In step S34, the area specifying unit 15 calculates, based on the vehicle information, an area that the vehicle will reach within a certain period from the current position as a notification target area. The certain period is, for example, 1 to 6 seconds, but may be changed based on the characteristics of the driver's visual behavior and driving behavior.

[0092] In step S35, the area identification unit 15 sets the movement area of ​​the potential risk for which it has been determined that no actual risk exists as a potential risk area, and determines whether the potential risk area exists within the notification target area. If it is determined that the potential risk area exists within the notification target area, the process proceeds to step S36, and if it is determined that the potential risk area does not exist within the notification target area, the process proceeds to step S39.

[0093] In step S36, the area identification unit 15 sets the potential risk area determined to be within the notification target area as a warning area, and determines whether an obstruction exists on the line connecting the warning area and the vehicle. If it is determined that an obstruction exists, the process proceeds to step S37, and if it is determined that an obstruction does not exist, the process proceeds to step S38.

[0094] In step S37, the area identification unit 15 adjusts the position of the attention area to the vicinity of the edge of the obstacle, and the process proceeds to step S39. In step S38, the area identification unit 15 maintains the attention area as it is, and the process proceeds to step S39. Note that the area identification unit 15 performs the processes of steps S31 to S38 described above for each potential risk extracted by the potential risk extraction unit 14.

[0095] In step S39, the area identification unit 15 determines whether or not multiple attention areas exist. If it is determined that multiple attention areas exist, the process proceeds to step S40. If it is determined that multiple attention areas do not exist, the process of FIG. 11 ends.

[0096] In step S40, the region specifying unit 15 specifies the attention region having the highest product of the potential risk candidate value and the recognition difficulty level from among the plurality of attention regions. Then, the process of FIG. 11 ends.

[0097] 12 is a flowchart showing the light emission method determination process and the light emission control process according to the present embodiment 1, that is, the processes in steps S6 and S7 of the light emission method determination unit 16 and the light emission control unit 17 of the driving assistance device 1 according to the present embodiment 1. The light emission method determination unit 16 and the light emission control unit 17 perform the processes of FIG. 12 for the light-emitting devices, that is, for at least one of the vehicle exterior light-emitting device 38 and the vehicle interior light-emitting device 39.

[0098] In step S51, the light emission method determination unit 16 determines whether or not an attention area has been identified by the area identification unit 15. If it is determined that an attention area has been identified, the process proceeds to step S52, and if it is determined that an attention area has not been identified, the process proceeds to step S58.

[0099] In step S52, the light emission method determination unit 16 determines whether the light emission control unit 17 is currently executing light emission control of the light emitting device. If it is determined that light emission control is currently being executed, the process proceeds to step S53, and if it is determined that light emission control is not currently being executed, the process proceeds to step S55.

[0100] In step S53, the light emission method determination unit 16 determines whether or not there has been a change in the attention area. If it is determined that there has been a change in the attention area, the process proceeds to step S54, and if it is determined that there has not been a change in the attention area, the process proceeds to step S57.

[0101] In step S54, the light emission control unit 17 temporarily stops the light emission control based on the attention area, thereby temporarily turning off the light emitting device. Thereafter, the process proceeds to step S55.

[0102] In step S55 , the light emission method determination unit 16 determines a light emission method for the attention area of ​​the light-emitting device based on the potential risk for which the attention area has been identified by the area identification unit 15 .

[0103] In step S56, the light-emitting control unit 17 uses the determined light-emitting method to control the light-emitting device based on the attention-attracting area, thereby turning on or illuminating the light-emitting device. If the light-emitting device includes the exterior light-emitting device 38 and the illumination range of the exterior light-emitting device 38 includes at least a portion of the attention-attracting area 63a, the exterior light-emitting device 38 illuminates the attention-attracting area 63a through the light-emitting control. If the light-emitting device includes the interior light-emitting device 39, the interior light-emitting device 39 illuminates or illuminates a portion of the light-emitting area that intersects with a line connecting the driver and the attention-attracting area through the light-emitting control. The interior light-emitting device 39 here includes a so-called head-up display function. Then, the processing of FIG. 12 ends.

[0104] In step S57, the light emission control unit 17 continues the light emission control based on the attention area, thereby continuing to turn on or irradiate the light emitting device. Thereafter, the process of FIG. 12 ends.

[0105] In step S58, the light-emission control unit 17 continuously stops the light-emission control based on the attention area, thereby continuously turning off the light-emitting device. After that, the processing in FIG. 12 ends.

[0106] Summary of First Embodiment The driving assistance device 1 according to the first embodiment calculates a recognition difficulty level, which is the degree of difficulty for the driver of the vehicle to recognize a potential risk, extracts a potential risk based on the potential risk candidate value and the recognition difficulty level, identifies an attention-drawing area based on the potential risk, determines a light-emitting method for the attention-drawing area based on the potential risk, and controls light emission of the light-emitting device based on the light-emitting method. This configuration can alert the driver to potential risks that are difficult for the driver to recognize without drawing the driver's attention more than necessary. This allows the driver to concentrate on driving while appropriately paying attention to potential risks.

[0107] In addition, in the first embodiment, a saliency map is generated based on an image of the area in front of the vehicle, and the lower the saliency of an area in the saliency map, the higher the recognition difficulty level is set. With this configuration, an appropriate recognition difficulty level can be determined by using the saliency map.

[0108] In the first embodiment, the warning area is identified using a notification target area that the vehicle will reach after a certain time from the current location. With this configuration, it is possible to suppress warnings about potential risks that are highly likely to require a response by driving assistance such as a collision mitigation brake, and that cannot be addressed even if the driver is notified.

[0109] <Embodiment 2> Fig. 13 is a block diagram showing the configuration of a driving assistance system according to Embodiment 2. The configuration of the driving assistance system shown in Fig. 13 is similar to the configuration of the driving assistance system shown in Fig. 1 except that a driver monitoring device 35 is added to the outside of the driving assistance device 1, and the driving assistance device 1 further includes a monitoring information acquisition unit 18 and an information storage unit 19. The driving assistance device 1 may further include the driver monitoring device 35. Hereinafter, of the components according to Embodiment 2, components that are the same as or similar to the components described above will be assigned the same or similar reference numerals, and different components will be mainly described.

[0110] The driver monitoring device 35 monitors the state of the driver of the vehicle. The driver monitoring device 35 includes, for example, an in-vehicle camera and a biometric sensor. The in-vehicle camera detects the driver's line of sight and facial direction. The biometric sensor measures the driver's electrocardiogram and pulse wave. The driving assistance device 1 according to the second embodiment is capable of determining the recognition difficulty level and determining the light emission method based on the monitoring results of the driver monitoring device 35.

[0111] <Monitoring Information Acquisition Unit 18> The monitoring information acquisition unit 18 is included in the concept of an acquisition unit. The monitoring information acquisition unit 18 generates (acquires) driver behavior information based on the monitoring results of the driver monitoring device 35. The driver behavior information includes the driver's visual behavior such as the driver's gaze position, and in the second embodiment, further includes the driving state such as the driver's alertness and concentration level. Various known methods can be used for determining the driver's gaze position based on the driver's line of sight and facial direction, and for determining the driver's alertness and concentration level based on the driver's electrocardiogram and pulse wave.

[0112] The monitoring information acquisition unit 18 may acquire vehicle state information related to the driving operation of the driver from the information acquisition unit 11. The monitoring information acquisition unit 18 may generate (acquire) driver behavior information based on the monitoring results of the driver monitoring device 35 and the vehicle state information from the information acquisition unit 11.

[0113] <Information Storage Unit 19> The information storage unit 19 stores information such as vehicle state information and behavior information acquired by the information acquisition unit 11 and the monitoring information acquisition unit 18 together with the time of acquisition.

[0114] <Recognition Difficulty Level Calculation Unit 13 > The recognition difficulty level calculation unit 13 calculates the recognition difficulty level based on the visual recognition behavior included in the behavior information acquired by the monitoring information acquisition unit 18 .

[0115] As a first example, the recognition difficulty calculation unit 13 acquires the driver's gaze position during a certain period based on the viewing behavior during the certain period from the present to the past, including the viewing behavior acquired by the information acquisition unit 11 and the past viewing behavior stored in the information storage unit 19. The certain period is preferably a period from the current time to a time two seconds prior. Areas other than the gaze position and the vicinity of the gaze position are areas where the driver is not looking, making it difficult for the driver to grasp the situation. Therefore, the recognition difficulty of these areas is considered to be high. Therefore, the recognition difficulty calculation unit 13 increases the recognition difficulty of areas other than the driver's gaze position and the vicinity of the gaze position during the certain period.

[0116] As a second example, the recognition difficulty calculation unit 13 calculates the recognition difficulty by inputting the viewing behavior into a recognition difficulty learning model. The recognition difficulty learning model is a learning model that infers the recognition difficulty from the viewing behavior. The recognition difficulty learning model is generated by the recognition difficulty calculation unit 13 learning (training) the relationship between the viewing behavior and the recognition difficulty. For this learning, for example, a statistical method such as logistic regression using training data including the viewing behavior as an explanatory variable and the recognition difficulty as a target variable may be used, or deep learning using the training data may be used.

[0117] The teacher data for the recognition difficulty learning model may further include behavior information of an experienced driver, and the recognition difficulty calculation unit 13 may increase the recognition difficulty as the difference between the gaze position of the experienced driver and the gaze position of the vehicle driver increases. Experienced drivers are excellent at checking the surrounding traffic environment and tend to focus their gaze on many important areas for safe driving. Therefore, when the above-mentioned teacher data is used in the recognition difficulty learning model, the recognition difficulty calculation unit 13 can increase the recognition difficulty of areas that experienced drivers gaze at but the vehicle driver does not gaze at. The recognition difficulty calculation unit 13 may also increase the recognition difficulty of areas that vehicle drivers gaze at but the experienced driver does not gaze at. In this case, the driver's attention can be alerted to areas that are difficult for even experienced drivers to recognize.

[0118] In addition, the recognition difficulty calculation unit 13 may calculate the final recognition difficulty by, for example, taking a weighted average of the recognition difficulty calculated based on the image included in the vehicle surroundings information described in embodiment 1 and the recognition difficulty calculated based on the behavior information described in embodiment 2.

[0119] <Light Emission Method Determination Unit 16> Fig. 14 is a diagram showing an example of lighting of the interior light emitting device 39. In the example of Fig. 14, a position in the line of sight of the driver's eyes, that is, a general range of about 30 degrees to the left and right from the center position where the driver 71 is gazing, is shown as an effective visual field 71a.

[0120] 7B, the interior light-emitting device 39 illuminates, for example, a portion 39b of the light-emitting region 39a that intersects with a line connecting the driver 71 and the attention-calling region 63. In contrast, in the present embodiment 2, the light-emitting method determination unit 16 determines the light-emitting method for the attention-calling region of the light-emitting device based on the behavior information acquired by the monitoring information acquisition unit 18. For example, the interior light-emitting device 39 changes the illumination position of the illuminated portion 39b depending on whether the attention-calling region 63 is included in the effective visual field 71a of the driver 71, based on the attention-calling region 63 and the behavior information.

[0121] When the attention-attracting region 63 is included within the visual field 71a, the light-emitting method determination unit 16 sets the illumination position to a portion 39b of the light-emitting region 39a that intersects with a line connecting the driver 71 and the attention-attracting region 63, as in the first embodiment. On the other hand, when the attention-attracting region 63 is not included within the visual field 71a, the light-emitting method determination unit 16 sets the light-emission start position to a portion 39c of the light-emitting region 39a that corresponds to the edge of the visual field 71a, and sets the light-emission end position to a portion 39b of the light-emitting region 39a that intersects with a line connecting the driver 71 and the attention-attracting region 63. The light-emitting method determination unit 16 then determines a light-emitting method for the interior light-emitting device 39 to move the illuminated portion from the light-emission start position to the light-emission end position. Because human vision generally responds quickly to moving or changing objects, the above-described illumination can alert the driver to the attention-attracting region. Note that a similar illumination, or so-called line-of-sight guidance, may also be performed on the exterior light-emitting device 38.

[0122] Furthermore, the light emission method determination unit 16 may determine the light emission method by combining the light emission method determined in the first embodiment with the light emission method determined based on the behavior information in the second embodiment. For example, the light emission method determination unit 16 may determine the light emission method based on the vehicle state information, the vehicle surroundings information, the potential risk candidate value, and the recognition difficulty level, and correct the light emission method based on the driver's alertness and concentration level.

[0123] As an example, when the driver's concentration level is low and the driver's attention is highly distracted, or when the driver's alertness level is low and the driver is very drowsy, the light emission method determination unit 16 may flash the exterior light-emitting device 38 and the interior light-emitting device 39 or change the brightness of their light emission. This configuration can increase the driver's attention. It is also possible to reduce the brightness so that a strong stimulus such as flashing does not overly startle the driver and cause dangerous driving, such as lane departure. The light emission method determination unit 16 may also turn on or illuminate the exterior light-emitting device 38 and the interior light-emitting device 39 to brighten the area around the lane. This configuration can increase the driver's awareness of the lane, thereby preventing lane departure.

[0124] <Operation> <Recognition Difficulty Level Calculation Processing> In the first embodiment, the recognition difficulty level calculation unit 13 generates a saliency map based on an image included in the vehicle surroundings information, and calculates the recognition difficulty level based on the saliency map, as shown in Fig. 9. In contrast, in the second embodiment, the recognition difficulty level calculation unit 13 calculates the recognition difficulty level based on driver behavior information.

[0125] 15 is a flowchart showing a light emitting method determination process and a light emitting control process according to the present embodiment 2. The light emitting method determination unit 16 and the light emission control unit 17 perform the process of FIG. 15 for the light emitting devices, that is, for at least one of the vehicle exterior light emitting device 38 and the vehicle interior light emitting device 39.

[0126] In step S61, the light emission method determination unit 16 calculates the driver's effective field of view based on the driver's behavior information.

[0127] In step S62, the light emission method determination unit 16 determines whether or not an attention area has been identified by the area identification unit 15. If it is determined that an attention area has been identified, the process proceeds to step S63, and if it is determined that an attention area has not been identified, the process of FIG. 15 ends.

[0128] In step S63, the light emission method determination unit 16 determines whether the attention area is included in the effective field of view. If it is determined that the attention area is included in the effective field of view, the process proceeds to step S64. If it is determined that the attention area is not included in the effective field of view, the process proceeds to step S65.

[0129] In step S64, the light emission method determination unit 16 sets the light emission position of the light emitting device to the attention drawing area. Then, the process proceeds to step S66.

[0130] In step S65, the light emission method determination unit 16 sets the light emission start position of the light emitting device to the end of the effective field of view that is closest to the attention drawing area, and sets the light emission end position of the light emitting device to the attention drawing area. Then, the process proceeds to step S66.

[0131] In step S66, the light emission control unit 17 controls the light emission of the light emitting device based on the light emission position or based on the light emission start position and light emission end position, thereby turning on the light emitting device. After that, the operation of FIG. 15 ends.

[0132] Summary of Second Embodiment According to the driving assistance device 1 according to the second embodiment as described above, the recognition difficulty level is calculated based on the behavior information. With this configuration, the recognition difficulty level can be calculated appropriately.

[0133] In addition, in the second embodiment, the light emitting method for the attention drawing area of ​​the light emitting device is determined based on the behavior information. With this configuration, it is possible to increase the driver's attention to potential risks and to suppress dangerous driving.

[0134] <Embodiment 3> Figure 16 is a block diagram showing the configuration of a driving assistance system according to Embodiment 3. The configuration of the driving assistance system shown in Figure 16 is similar to the configuration of the driving assistance system shown in Figure 13 in which the driving assistance device 1 further includes an evaluation unit 21. Hereinafter, of the components according to Embodiment 3, components that are the same as or similar to the components described above will be assigned the same or similar reference numerals, and different components will mainly be described.

[0135] The monitoring information acquisition unit 18 not only acquires the driver's behavior information described in the second embodiment, but also acquires obstacle information from the vehicle surroundings information from the information acquisition unit 11. The information storage unit 19 stores the vehicle state information, behavior information, and obstacle information acquired by the information acquisition unit 11 and the monitoring information acquisition unit 18, the light emission method determined by the light emission method determination unit 16, and the attention calling area identified by the area identification unit 15, together with the time of acquisition. Note that the behavior information according to the third embodiment further includes the driver's driving operation.

[0136] <Evaluation unit 21> The evaluation unit 21 evaluates the light emission method determined by the light emission method determination unit 16 and the attention area identified by the area identification unit 15 based on behavior information acquired after the light emission control unit 17 has performed light emission control. In the third embodiment, the evaluation unit 21 evaluates the light emission method and the attention area based on stored information including the behavior information. The stored information here is information including the behavior information stored in the information storage unit 19.

[0137] The evaluation unit 21 obtains an evaluation of the light-emitting method and the attention area by inputting memory information including behavior information into an evaluation learning model. The evaluation learning model is a learning model that infers the evaluation of the light-emitting method and the attention area from memory information including behavior information. The evaluation learning model is generated by the evaluation unit 21 learning (training) the relationship between the memory information including behavior information and the evaluation of the light-emitting method and the attention area. This learning may use a statistical method such as logistic regression that uses training data including memory information as an explanatory variable and evaluations of the light-emitting method and the attention area as a target variable, or deep learning that uses the training data. Note that the evaluation of the light-emitting method and the attention area used by the evaluation unit 21 to train the evaluation learning model uses the evaluation results of the appropriateness of the light-emitting method and the attention area by the user (the driver of the vehicle).

[0138] The stored information may include behavior information and attention-warning areas. Furthermore, if the driver monitoring device 35 is capable of identifying drivers, the evaluation unit 21 may generate an evaluation learning model for each driver. The learning of the evaluation learning model by the evaluation unit 21 and the learning of the potential risk candidate learning model by the risk candidate value calculation unit 12 may be performed serially or in parallel. Furthermore, the learning model for inferring the evaluation of the light-emitting method and the learning model for inferring the evaluation of the attention-warning areas may be provided separately.

[0139] By the above learning, when the gaze position of the driver included in the behavior information is not directed toward the attention area, the evaluation unit 21 can request a low evaluation for the light-emitting method and the attention area. On the other hand, when the gaze position of the driver included in the behavior information is directed toward the attention area, the evaluation unit 21 can request a high evaluation for the light-emitting method and the attention area.

[0140] Furthermore, when the behavior information includes large changes in vehicle speed, acceleration, steering angular velocity, etc. due to a sudden driving operation such as abrupt steering or braking, the evaluation unit 21 can assign a low evaluation to the light emission method and the attention-drawing area. On the other hand, when the behavior information does not include large changes in vehicle speed, acceleration, steering angular velocity, etc., the evaluation unit 21 can assign a high evaluation to the light emission method and the attention-drawing area.

[0141] <Light Emission Method Determination Unit 16> The light emission method determination unit 16 corrects the light emission method based on the evaluation of the light emission method obtained by the evaluation unit 21. For example, when the evaluation of the light emission method is equal to or lower than a threshold, the light emission method determination unit 16 may increase the brightness of the light emitted by the light emitting device or change the light emission of the light emitting device from a steady light to a flashing light. By changing the light emission method in this way, it is possible to more effectively alert the driver to potential risks.

[0142] The evaluation of the light emitting method obtained by the evaluation unit 21 may be used for learning the light emitting method learning model described in embodiment 1, or the road environment factors described in embodiment 1 may be used for learning the evaluation learning model. Furthermore, instead of the light emitting method determination unit 16 correcting the light emitting method, the evaluation of the light emitting method obtained by the evaluation unit 21 may be presented to the user, allowing the user to correct the light emitting method.

[0143] <Area Identification Unit 15> The area identification unit 15 corrects the attention area based on the evaluation of the attention area obtained by the evaluation unit 21. For example, when the evaluation of the attention area is equal to or lower than a threshold, the area identification unit 15 may expand the attention area by changing the fixed time period of the movement area of ​​the potential risk that is the source of the attention area from 1 second to 2 seconds. Furthermore, when the evaluation of the attention area is equal to or lower than a threshold, the area identification unit 15 may change the attention area by changing the fixed time period of the notification target area used to identify the attention area from 1 to 6 seconds to 1 to 7 seconds. Note that instead of the area identification unit 15 correcting the attention area, the user may correct the attention area by presenting the evaluation of the attention area obtained by the evaluation unit 21 to the user.

[0144] Summary of Third Embodiment According to the driving assistance device 1 of the third embodiment described above, the light emission method and the attention calling area are evaluated based on the behavior information acquired after the light emission control. This configuration makes it possible to determine the light emission method suitable for the driver of the vehicle and identify the attention calling area, thereby realizing the attention calling suitable for the driver.

[0145] <Other Modifications> The information acquisition unit 11, risk candidate value calculation unit 12, recognition difficulty level calculation unit 13, potential risk extraction unit 14, area specification unit 15, light emission method determination unit 16, and light emission control unit 17 shown in Fig. 1 will be hereinafter referred to as the "information acquisition unit 11, etc." The information acquisition unit 11, etc. is realized by a processing circuit 81 shown in Fig. 17. That is, the processing circuit 81 includes an information acquisition unit 11 that acquires vehicle surroundings information related to at least the vehicle's surroundings, a risk candidate value calculation unit 12 that calculates a risk value for a potential risk candidate area based on the vehicle surroundings information or vehicle information including the vehicle's state and position, a recognition difficulty calculation unit 13 that calculates a recognition difficulty level, which indicates the degree of difficulty for the driver to recognize the area around the vehicle, based on the vehicle surroundings information or driver behavior information, a potential risk extraction unit 14 that extracts potential risk candidates whose risk value is equal to or greater than a threshold and whose recognition difficulty level is equal to or greater than a threshold based on the risk value and the recognition difficulty level, an area identification unit 15 that identifies a warning area that requires the driver's attention based on the potential risk, a lighting method determination unit 16 that determines a lighting method for the warning area based on the potential risk or behavior information, and a lighting control unit 17 that controls lighting of the vehicle's lighting device based on the lighting method. The processing circuit 81 may be implemented using dedicated hardware or a processor that executes a program stored in memory. Examples of processors include central processing units, processing units, arithmetic units, microprocessors, microcomputers, and DSPs (Digital Signal Processors).

[0146] When the processing circuitry 81 is dedicated hardware, the processing circuitry 81 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The functions of each unit such as the information acquisition unit 11 may be realized by a circuit in which processing circuits are distributed, or the functions of each unit may be realized together by a single processing circuit.

[0147] When the processing circuit 81 is a processor, the functions of the information acquisition unit 11 and the like are realized in combination with software and the like. Note that software and the like includes, for example, software, firmware, or software and firmware. The software and the like is written as a program and stored in memory. As shown in FIG. 18 , a processor 82 applied to the processing circuit 81 realizes the functions of each unit by reading and executing a program stored in a memory 83. That is, the driving assistance device 1 includes a memory 83 for storing a program that, when executed by the processing circuit 81, results in the following steps: acquiring vehicle surroundings information related to at least the vehicle's surroundings, calculating a risk value for a potential risk candidate area based on the vehicle surroundings information or vehicle information including the vehicle's state and position, calculating a recognition difficulty level that indicates the degree to which the area around the vehicle is difficult for the driver to recognize based on the vehicle surroundings information or behavior information of the vehicle's driver, extracting, as potential risks, potential risk candidates whose risk value is equal to or greater than a threshold and whose recognition difficulty level is equal to or greater than a threshold based on the risk value and the recognition difficulty level, identifying, based on the potential risk, a warning area that is an area in which the driver should be warned, determining a light emission method for the warning area based on the potential risk or the behavior information, and controlling the light emission of the vehicle's light-emitting device based on the light emission method. In other words, this program can be said to cause a computer to execute the procedures and methods of the information acquisition unit 11, etc.Here, the memory 83 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read Only Memory), a HDD (Hard Disk Drive), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a DVD (Digital Versatile Disc), a drive device for any of these, or any storage medium to be used in the future.

[0148] The above describes a configuration in which each function of the information acquisition unit 11 and the like is realized either by hardware or software, etc. However, this is not limited to this, and a configuration in which part of the information acquisition unit 11 and the like is realized by dedicated hardware and another part is realized by software, etc. For example, the function of the information acquisition unit 11 can be realized by a processing circuit 81 as dedicated hardware, and the other functions can be realized by the processing circuit 81 as a processor 82 reading and executing a program stored in a memory 83.

[0149] As described above, the processing circuitry 81 can realize the above-mentioned functions by hardware, software, or a combination of these. The same applies to the light emission method determination unit 16 and the like.

[0150] The driving assistance device described above can also be applied to a system constructed by appropriately combining a vehicle device, a communication terminal, the functions of an application installed on at least one of the vehicle device and the communication terminal, and a server. Communication terminals include, for example, mobile phones, smartphones, and tablets. The functions or components of the driving assistance device described above may be distributed among the devices that constitute the system, or may be centrally located in one of the devices.

[0151] In this disclosure, 'a' and 'an' mean one or more. Therefore, 'a', 'an', 'one or more', and 'at least one' can be used interchangeably.

[0152] It should be noted that the embodiments and modifications may be freely combined, and the embodiments and modifications may be modified or omitted as appropriate.

[0153] The above description is illustrative in all respects and is not restrictive. It is understood that countless variations not illustrated can be envisioned.

[0154] REFERENCE SIGNS LIST 1 Driving assistance device, 11 Information acquisition unit, 12 Risk candidate value calculation unit, 13 Recognition difficulty calculation unit, 14 Potential risk extraction unit, 15 Area identification unit, 16 Light emission method determination unit, 17 Light emission control unit, 18 Monitoring information acquisition unit, 21 Evaluation unit, 38 Vehicle exterior light emission device, 39 Vehicle interior light emission device, 51 Vehicle.

Claims

1. A driving assistance device comprising: an acquisition unit that acquires vehicle surroundings information related to at least the area around the vehicle; a risk candidate value calculation unit that calculates a risk value for an area of ​​a potential risk candidate based on the vehicle surroundings information or vehicle information including the state and position of the vehicle; a recognition difficulty calculation unit that calculates a recognition difficulty level, which indicates the degree of difficulty that the area around the vehicle is difficult for the driver to recognize, based on the vehicle surroundings information or behavior information of the driver of the vehicle; a potential risk extraction unit that extracts, as potential risks, potential risk candidates whose risk value is equal to or greater than a threshold and whose recognition difficulty level is equal to or greater than a threshold based on the risk value and the recognition difficulty level; an area identification unit that identifies, based on the potential risk, an attention warning area, which is an area in which the driver should be warned; a light emission method determination unit that determines a light emission method for the attention warning area based on the potential risk or the behavior information; and a light emission control unit that controls the light emission of a light emitting device of the vehicle based on the light emission method.

2. A driving assistance device according to claim 1, wherein the vehicle surroundings information includes an image in front of the vehicle, and the recognition difficulty calculation unit calculates the recognition difficulty based on the image included in the vehicle surroundings information.

3. A driving assistance device according to claim 2, wherein the recognition difficulty calculation unit generates a saliency map based on the image, and increases the recognition difficulty in areas of the saliency map that have lower saliency.

4. A driving assistance device according to claim 2, wherein the recognition difficulty calculation unit calculates the brightness of an area in the image where the risk value is equal to or greater than a threshold, and calculates the recognition difficulty based on the brightness.

5. A driving assistance device as described in claim 1, wherein the behavior information includes the driver's visual behavior, and the recognition difficulty calculation unit calculates the recognition difficulty based on the visual behavior included in the behavior information.

6. A driving assistance device according to claim 1, wherein the light emission method determination unit determines the light emission method based on the behavior information.

7. A driving assistance device as described in claim 1, wherein the light emission method determination unit determines the light emission method based on the potential risk or the behavior information, and further comprises an evaluation unit that evaluates the light emission method based on the behavior information acquired after the light emission control.

8. A driving assistance device according to claim 7, wherein the light emission method determination unit corrects the light emission method based on the evaluation.

9. A driving assistance device according to claim 1, further comprising an evaluation unit that evaluates the attention-drawing area based on the behavior information acquired after the light emission control.

10. A driving assistance device according to claim 9, wherein the area identification unit corrects the attention-drawing area based on the evaluation.

11. A driving assistance method comprising: acquiring vehicle surroundings information relating to at least the area around the vehicle; calculating a risk value for an area of ​​a potential risk candidate based on the vehicle surroundings information or vehicle information including the state and position of the vehicle; calculating a recognition difficulty level, which indicates the degree of difficulty for the driver to recognize the area around the vehicle, based on the vehicle surroundings information or behavior information of the driver of the vehicle; extracting, as potential risks, potential risk candidates whose risk value is equal to or greater than a threshold and whose recognition difficulty level is equal to or greater than a threshold, based on the risk value and the recognition difficulty level; identifying, based on the potential risks, an attention-warning area, which is an area in which the driver should be warned; determining a lighting method for the attention-warning area based on the potential risks or the behavior information; and controlling lighting of a lighting device of the vehicle based on the lighting method.

Images