Driver assistance systems and vehicles

Abstract

The present invention provides a driver assistance system and vehicle that can enable drivers to take actions that contribute to safety against multiple hazards by providing notifications that prevent them from becoming preoccupied with only one hazard when multiple hazards are present in their surroundings. [Solution] A driving assistance device according to one embodiment of the present disclosure is capable of estimating the presence or absence of multiple potential hazards caused by the actions of multiple traffic participants around the vehicle, based on at least road data from among road data and traffic participant data around the vehicle. When the presence of multiple potential hazards is estimated, this driving assistance device is capable of performing data processing in which an agent generated using agent data performs actions corresponding to the danger level of the multiple potential hazards obtained by estimation, while the agent's gaze or posture is directed towards the driver.

Description

[Technical Field]

[0001] This disclosure relates to driver assistance systems and vehicles. [Background technology]

[0002] Driving assistance devices are known that assist the vehicle's movement in accordance with the surrounding conditions (see, for example, Patent Documents 1 to 5). [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2002-213986 [Patent Document 2] Japanese Patent Publication No. 2013-078969 [Patent Document 3] Japanese Patent Publication No. 2017-049943 [Patent Document 4] Japanese Patent Publication No. 2023-019245 [Patent Document 5] Japanese Patent Publication No. 2023-151589 [Overview of the project]

[0004] A driver assistance device according to one embodiment of the present disclosure comprises a data acquisition unit, a data processing unit, and an output unit. The data acquisition unit is capable of acquiring road data and traffic participant data around the vehicle, as well as agent data. The data processing unit is capable of estimating the presence or absence of multiple potential hazards caused by the actions of multiple traffic participants around the vehicle, based on at least the road data from the road data and traffic participant data. When the data processing unit estimates the presence of multiple potential hazards, it is capable of performing data processing to cause an agent generated using agent data to perform actions corresponding to the danger level of the multiple potential hazards obtained by estimation, while the agent's gaze or posture is directed towards the driver. The output unit is capable of generating a video signal including the agent based on the data obtained by data processing and outputting it to a display unit.

[0005] A vehicle according to one embodiment of this disclosure includes a driver assistance device and a display unit. The driver assistance device includes a data acquisition unit, a data processing unit, and an output unit. The data acquisition unit is capable of acquiring road data and traffic participant data around the vehicle, as well as agent data. The data processing unit is capable of estimating the presence or absence of multiple potential hazards caused by the actions of multiple traffic participants around the vehicle, based on at least the road data from the road data and traffic participant data. When the data processing unit estimates the presence of multiple potential hazards, it is capable of performing data processing to cause an agent generated using agent data to perform actions corresponding to the danger level of the multiple potential hazards obtained by estimation, while the agent's gaze or posture is directed towards the driver. The output unit is capable of generating a video signal including the agent based on the data obtained by data processing and outputting it to the display unit. [Brief explanation of the drawing]

[0006] The accompanying drawings are provided to further understand the present disclosure, are incorporated into this specification, and constitute a part of this specification. The drawings show an embodiment and, together with the specification, serve to explain the principles of the present disclosure.

[0007] [Figure 1] Figure 1 is a diagram showing an example of the appearance around the driver's seat inside a vehicle according to an embodiment of the present disclosure. [Figure 2] Figure 2 is a diagram showing an example of the view of the front of the vehicle when looking ahead from the driver's seat of the vehicle in Figure 1. [Figure 3] Figure 3 is a diagram showing a schematic configuration example of the vehicle in Figure 1. [Figure 4] Figure 4(A) is a diagram showing an example of the appearance of the agent in Figure 1. Figure 4(B) is a diagram showing an example of the appearance of the agent in Figure 1. Figure 4(C) is a diagram showing an example of the appearance of the agent in Figure 1. [Figure 5] Figure 5 is a diagram showing an example of the appearance of the agent in Figure 1. [Figure 6] Figure 6 is a diagram showing an example of the traffic situation around the vehicle in Figure 1. [Figure 7] Figure 7 is a diagram showing an example of a driving support procedure in the vehicle in Figure 5. [Figure 8] Figure 8 is a diagram showing a modified example of the driving support procedure in the vehicle in Figure 5. [Figure 9] Figure 9 is a diagram showing a modified example of the appearance around the driver's seat inside the vehicle in Figure 1.

Mode for Carrying Out the Invention

[0008] Hereinafter, several exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. The following description is intended to illustrate specific examples of the present disclosure and should not be construed as limiting the disclosure. For example, elements such as numerical values, shapes, materials, parts, the location of each part, and the method of connecting each part are merely examples and should not be construed as limiting the disclosure. Furthermore, in the following exemplary embodiments, components not described in separate sections based on the highest-level concepts of the present disclosure are optional and may be provided as needed. The drawings are schematic and are not intended to be to scale. Throughout this specification and the drawings, components having substantially the same function and substantially the same configuration are denoted by the same reference numerals, and redundant descriptions are omitted. Furthermore, components not directly related to an embodiment of the present disclosure are not shown in the drawings.

[0009] <1. Background> There are known driver assistance devices that assist the vehicle's driving in accordance with the situation around the vehicle. For example, Patent Documents 1 to 5 describe how characters or agents that drivers can easily feel a sense of familiarity with can speak to the driver, or direct their gaze and facial expressions towards the driver, thereby making the driver interested in what the character or agent is saying, looking at them, and making facial expressions, and encouraging them to take action.

[0010] For example, the invention described in Patent Document 1 can display a character with the appearance of a lover in "lover mode" and a character with the appearance of a mother in "mother mode." Furthermore, the invention described in Patent Document 1 changes the character's facial expression (joy, anger, sadness, etc.) according to the current situation (surrounding scenery, traffic congestion, traffic manners). This allows the driver to enjoy driving the vehicle more.

[0011] For example, in the invention described in Patent Document 2, when the vehicle approaches a dangerous location where traffic accidents are likely to occur, the character behaves as if it were a real person and notifies the driver of the dangerous location, as well as warns against dangerous driving, reckless driving, driving without concentration, and drowsy driving. This is intended to make the driver feel a strong emotional connection to the character and more likely to heed the notifications and warnings.

[0012] For example, in the invention described in Patent Document 3, after a driver is in danger, such as when another vehicle suddenly cuts in aggressively, the system outputs a calming message to the driver in the voice of a lover or family member. This is said to influence the driver's psychological state and help them achieve a calm state of mind.

[0013] For example, in the invention described in Patent Document 4, a personified character notifies the driver of the presence of a fallen object in front of the vehicle. This makes it possible to draw the driver's attention to the apparent danger.

[0014] For example, in the invention described in Patent Document 5, a real person, such as the driver's family or a friend, is set as a virtual passenger, and when the driver approaches the location of an incident, the driver is notified of the incident through the voice of the real person, such as the family or a friend. This allows the driver to virtually experience the feeling of being notified of danger by a passenger, and is said to improve the driver's awareness of safe driving.

[0015] However, in the inventions described in Patent Documents 1, 2, and 4, the driver's attention is directed to the hazard indicated by the character. Therefore, if other hazards not indicated by the character exist, the driver may become preoccupied with the hazard indicated by the character and be slow to notice the other hazards.

[0016] Furthermore, in the invention described in Patent Document 3, the system outputs a message to the driver after the danger has passed. As a result, the driver may become distracted by the system's message and be slow to notice potential dangers that may occur in the future.

[0017] Furthermore, in the invention described in Patent Document 5, the driver's attention is directed to the hazard indicated by the system. Therefore, if other hazards exist that are not indicated by the system, the driver may become preoccupied with the hazard indicated by the system and be slow to notice the other hazards.

[0018] Thus, with conventional technology, drivers may become preoccupied with hazards indicated by characters or agents, and may be slow to notice hazards not indicated by characters or agents. Therefore, after careful consideration, the inventors of this application have conceived of a technology that, when multiple hazards are present in the surroundings, provides notifications in a way that prevents the driver from becoming fixated on only one hazard, thereby enabling the driver to take actions that contribute to safety against multiple hazards. The following describes in detail the driver assistance device and vehicle that realize this technology.

[0019] <2. Embodiments> [Example Configuration] First, the configuration of vehicle 100 according to one embodiment of the present disclosure will be described. Figure 1 shows an example of the exterior view around the driver's seat inside vehicle 100. Figure 2 shows an example of the view in front of the vehicle when looking forward from the driver's seat of vehicle 100 in Figure 1. Figure 3 shows a schematic example of the configuration of vehicle 100 in Figure 1. Vehicle 100 corresponds to one specific example of the "vehicle" according to one embodiment of the present disclosure.

[0020] Vehicle 100 is capable of moving by the drive of a prime mover 70 (engine or motor). Vehicle 100 includes, for example, a sensor unit 10, a communication unit 20, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 40, a memory unit 50, a control unit 60, a prime mover 70, a brake 80, and an EPS (Electric Power Steering) motor 90, as shown in Figure 3.

[0021] The sensor unit 10 is comprised of various sensors mounted on the vehicle 100. For example, the sensor unit 10 comprises an accelerator opening sensor, a vehicle speed sensor, an acceleration sensor, an angular velocity sensor, a steering angle sensor, and a steering torque sensor. The sensor unit 10 may also include sensors other than those listed above.

[0022] The accelerator pedal position sensor can detect the accelerator pedal position from the amount the accelerator pedal is pressed. The accelerator pedal position sensor can output time-series data (accelerator pedal position data) about the detected accelerator pedal position to the control unit 60.

[0023] The vehicle speed sensor is capable of detecting the speed of vehicle 100. The vehicle speed sensor can output time-series data (vehicle speed data) about the detected vehicle speed to the control unit 60. The acceleration sensor is capable of detecting the acceleration applied to vehicle 100. The acceleration sensor can output time-series data (acceleration data) about the acceleration in the three detected directions to the control unit 60. The angular velocity sensor is capable of detecting the angular velocity of vehicle 100. The angular velocity sensor can output time-series data (angular velocity data) about the three detected angular velocities (yaw angular velocity, roll angular velocity, and pitch angular velocity) to the control unit 60.

[0024] The steering angle sensor is capable of detecting the steering angle of the steering wheel of the vehicle 100. The steering angle sensor can output time-series data (steering angle data) of the detected steering angle to the control unit 60. The steering torque sensor is capable of detecting the steering torque generated by the driver's steering wheel operation. The steering torque sensor can output time-series data (steering torque data) of the detected steering torque to the control unit 60.

[0025] The sensor unit 10 further comprises a front camera 11 mounted on the vehicle 100 and a driving environment detection unit. The front camera 11 is an autonomous sensor that senses the real space around the vehicle 100. The front camera 11 is a stereo camera that is positioned symmetrically on either side of the central part of the vehicle 100 in the width direction, and is capable of stereo imaging the area in front of the vehicle 100 from different viewpoints. The stereo camera is capable of outputting image data Ia (a pair of stereo image data) obtained by imaging to the control unit 60. Image data Ia may include, for example, the road on which the vehicle 100 travels (roadway La) and an intersection IS provided on roadway La, as shown in Figure 2. A wall WL is provided along roadway La, and a part of the intersection Lb that intersects with roadway La at intersection IS is a blind spot from the perspective of the vehicle 100 driver. Therefore, the intersection IS in front of the vehicle 100 is an intersection with poor visibility from the perspective of the vehicle 100 driver.

[0026] The stereo camera is capable of generating distance image data based on image data Ia (a pair of stereo image data) obtained through imaging, calculated from the amount of displacement of the corresponding object's position. The driving environment detection unit can, for example, determine the lane markings that demarcate the road around the vehicle 100 based on the distance image data. The driving environment detection unit can further determine the road curvature of the markings that demarcate the left and right sides of the road (driving lane) on which the vehicle 100 travels, and the width between the left and right markings (vehicle width). The driving environment detection unit can further perform predetermined pattern matching on the distance image data to detect lanes and three-dimensional objects such as structures present around the vehicle 100. The driving environment detection unit is composed of, for example, an MPU (Micro Processing Unit).

[0027] In the driving environment detection unit, the detection of three-dimensional objects includes, for example, the type of object, the distance to the object, the speed of the object, and the relative speed between the object and the vehicle (the vehicle itself). Examples of objects to be detected include traffic lights, intersections, road signs, stop lines, other vehicles, pedestrians, bicycles, and buildings. Examples of buildings include detached houses, apartment buildings, commercial facilities, factories, and signs. The driving environment detection unit can output driving environment information around the vehicle 100, including the information on three-dimensional objects acquired in this way, to the control unit 60.

[0028] The communication unit 20 can acquire data to supplement data that cannot be obtained from image data Ia and distance image data, for example, through vehicle-to-vehicle communication, vehicle-to-infrastructure communication, and satellite communication. The communication unit 20 can output the acquired data to the control unit 60.

[0029] The communication unit 20 can acquire data obtained from other vehicles (e.g., vehicle position, vehicle speed) through vehicle-to-vehicle communication, for example. The communication unit 20 can also receive positioning signals transmitted from multiple positioning satellites through satellite communication, for example.

[0030] The communication unit 20 is capable of acquiring road map data around the vehicle 100, for example, through vehicle-to-infrastructure communication. The road map data consists of, for example, high-precision road map information (dynamic map) and mainly comprises static and quasi-static information that constitutes road information, and quasi-dynamic and dynamic information that mainly constitutes traffic information. The communication unit 20 is also capable of acquiring weather information around the vehicle 100, for example, through vehicle-to-infrastructure communication.

[0031] The static information that constitutes road information consists of information that requires updates at a frequency of no more than one month, such as roads and structures on roads, structures surrounding roads, lane information, road surface information, and permanent regulatory information. "Roads" include, for example, the location and shape of roads, intersections, and road attributes (e.g., national roads, prefectural roads, municipal roads, private roads, priority roads, non-priority roads, general roads, expressways). "Structures on roads" include, for example, traffic signs, traffic lights, convex mirrors, pedestrian overpasses, bus stops, and garbage collection points. "Structures surrounding roads" include, for example, various buildings and parks.

[0032] The quasi-static information that makes up road information consists of information that needs to be updated within an hour, such as traffic restriction information due to road construction or events, wide-area weather information, and congestion forecasts.

[0033] The semi-dynamic information that makes up traffic information consists of information that needs to be updated within one minute, such as actual congestion conditions and driving restrictions at the time of observation, temporary driving obstructions such as fallen objects and obstacles, actual accident conditions, and local weather information.

[0034] The dynamic information that constitutes traffic information consists of information that requires updates every second, such as information transmitted and exchanged between moving objects, information on currently displayed traffic signals, information on pedestrians and cyclists at intersections, and information on vehicles traveling on the roads. This road map information is maintained and updated in cycles until the next information is received from each vehicle, and the updated road map information is transmitted to each vehicle as appropriate through the communication unit 20.

[0035] The DMS30 consists of an in-vehicle camera 31 mounted inside the vehicle 100, a microphone, and an in-vehicle environment detection unit. The in-vehicle camera 31 is an autonomous sensor that senses the real space inside the vehicle 100. The in-vehicle camera 31 is a monocular camera that is positioned, for example, near the front windshield FW of the vehicle 100 and capable of imaging the interior of the vehicle 100. The monocular camera is capable of imaging the driver sitting in the driver's seat of the vehicle 100, or one or more passengers sitting in the passenger seat or rear seat of the vehicle 100. The monocular camera is capable of outputting image data Ib (photographic data) and video data Ic obtained by imaging to the in-vehicle environment detection unit. The microphone is capable of collecting sound from the vehicle 100. The microphone is capable of outputting sound data Va obtained by collection to the in-vehicle environment detection unit.

[0036] The in-vehicle environment detection unit is configured to include, for example, an MPU. The in-vehicle environment detection unit is capable of detecting the position and orientation of the driver's face in vehicle 100 based on, for example, image data Ib or video data Ic. The in-vehicle environment detection unit is also capable of detecting the presence or absence of one or more passengers in vehicle 100 based on, for example, image data Ib, video data Ic or audio data Va.

[0037] The in-vehicle environment detection unit can, for example, extract image data Id (photo data) or video data Ie containing the detected one or more passengers from image data Ib or video data Ic when it detects one or more passengers in vehicle 100. The in-vehicle environment detection unit may also be able to extract image data Id (photo data) and video data Ie containing the detected one or more passengers from image data Ib and video data Ic when it detects one or more passengers in vehicle 100. The in-vehicle environment detection unit can, for example, extract audio data Vb of one or more passengers in vehicle 100 based on audio data Va.

[0038] The DMS30 is capable of storing data Dd about the position and orientation of the driver's face in the vehicle 100, along with time data, in the history data 52 of the storage unit 50. The DMS30 is also capable of storing image data Id, video data Ie, and audio data Vb of one or more passengers in the vehicle 100, along with time data, in the history data 52 of the storage unit 50.

[0039] The HMI 40 is comprised of, for example, a steering wheel 41, switches 42, an instrument panel display 43, a center panel display 44, a speaker 45, and steering switches 46. Switch 42 is, for example, a switch for controlling the flashing of each turn signal. The instrument panel display 43 is comprised of, for example, a liquid crystal display panel or an organic EL display panel, and is capable of displaying information such as speed and engine RPM. The center panel display 44 is comprised of, for example, a touch-input liquid crystal display panel or an organic EL display panel, and is capable of making various settings of the vehicle 100. The center panel display 44 is capable of displaying animations, images (photographs), or videos, including an agent AG, as shown in Figure 1. The center panel display 44 corresponds to one specific example of the "display unit" according to one embodiment of the present disclosure. The agent AG corresponds to one specific example of the "agent" and "close agent" according to one embodiment of the present disclosure. The speaker 45 is capable of outputting, for example, voice messages that support various settings on the center panel display 44 and voice messages from the agent AG. The steering wheel switch 46 is, for example, a switch for setting the on / off status of driver assistance using agent AG.

[0040] The steering switch 46 is capable of receiving control flag input from the driver. The steering switch 46 is, for example, a switch attached to the steering wheel 41. The steering switch 46 can store "1" as a control flag in the memory unit 50 when, for example, the driver presses and holds down the steering switch 46 for a long time. The steering switch 46 can store "0" as a control flag in the memory unit 50 when, for example, the driver presses and holds down the steering switch 46 for a long time again after previously storing "1" as a control flag in the memory unit 50. The steering switch 46 can store "1" as a control flag in the memory unit 50 when, for example, the driver presses and holds down the steering switch 46 for a long time after previously storing "0" as a control flag in the memory unit 50.

[0041] When the control flag is "1", it means that the system is in a driving assistance mode that utilizes the agent AG. When the control flag is "0", it means that the system is in a normal mode that does not utilize the agent AG for driving assistance. Note that the values ​​that can be taken as the control flag are not limited to those listed above.

[0042] The storage unit 50 is composed of, for example, non-volatile memory, such as EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, or resistive random-access memory. The storage unit 50 stores, for example, a road map DB 51, historical data 52, and agent data 53, as shown in Figure 3.

[0043] The road map DB51 contains high-precision road map information (dynamic map). This high-precision road map information, similar to road map information acquired externally via vehicle-to-infrastructure communication, mainly consists of static and quasi-static information that constitutes road information, and quasi-dynamic and dynamic information that mainly constitute traffic information.

[0044] The history data 52 includes data obtained by the DMS 30. The history data 52 includes data Dd about the position and orientation of the driver's face in the vehicle 100, image data Id or video data Ie of one or more passengers in the vehicle 100, and audio data Vb of one or more passengers in the vehicle 100. The image data Id or video data Ie of one or more passengers in the vehicle 100 includes data corresponding to one specific example of "agent data" according to one embodiment of the present disclosure. The history data 52 may also include data Dd about the position and orientation of the driver's face in the vehicle 100, image data Id and video data Ie of one or more passengers in the vehicle 100, and audio data Vb of one or more passengers in the vehicle 100. The image data Id and video data Ie of one or more passengers in the vehicle 100 includes data corresponding to one specific example of "agent data" according to one embodiment of the present disclosure. In the historical data 52, the time data obtained by DMS30 is associated with each of the data Dd, image data Id, video data Ie, and audio data Vb.

[0045] The agent data 53 includes data that can be used in the driving assistance mode described above. The agent data 53 consists of multiple materials consisting of images (photographs) or videos of a person or pet that is close to the driver of the vehicle 100. Multiple materials consisting of images (photographs) or videos of a pet correspond to one specific example of the "agent data" according to one embodiment of the present disclosure. The agent data 53 may also consist of multiple materials consisting of images (photographs) of a pet and multiple materials consisting of videos of a pet. Multiple materials consisting of images (photographs) of a pet and multiple materials consisting of videos of a pet correspond to one specific example of the "agent data" according to one embodiment of the present disclosure. Examples of a person close to the driver of the vehicle 100 include a lover or family member (spouse or child).

[0046] Agent data 53 stores images (photographs) or videos of intimate creatures and danger levels corresponding to the emotional state of the intimate creatures in the images (photographs) or videos, in relation to each other. Agent data 53 may also store images (photographs) of intimate creatures and danger levels corresponding to the emotional state of the intimate creatures in the images (photographs), in relation to each other, as well as videos of intimate creatures and danger levels corresponding to the emotional state of the intimate creatures in the videos, in relation to each other.

[0047] The intimate creature is an entity with which the driver of vehicle 100 has communicated frequently, or has communicated frequently in the past, making it easy to understand the intention behind the intimate creature's actions. Furthermore, the intimate creature is an entity with which the driver of vehicle 100 feels affection and empathy. Therefore, when the intimate creature exhibits anxious behavior, the driver of vehicle 100 can easily understand the intention behind that behavior (i.e., that the intimate creature is feeling anxious). The driver of vehicle 100 interprets the intimate creature's actions favorably, as being done out of concern for the driver's safety. Therefore, when the intimate creature exhibits anxious behavior, the driver of vehicle 100 is more likely to take preventative action.

[0048] The agent data 53 further includes multiple agent data 53A, agent data 53B, and multiple agent data 53C, each with different actions. Each agent data 53A, agent data 53B, and each agent data 53C is stored in agent data 53 in association with a risk level.

[0049] Each agent data set 53A includes data that includes an agent AG performing a predetermined action while directing its gaze or posture toward the driver of the vehicle 100. Agent AG is an agent that mimics an intimate creature (intimate agent), and is data generated using one or more images (photographs) or one or more videos of an intimate creature, or one or more images (photographs) and one or more videos of an intimate creature, which are included in agent data 53. Multiple agent data sets 53A include multiple data sets with different danger levels corresponding to the emotional state of agent AG. Multiple agent data sets 53A include, for example, data with a low danger level, data with a medium danger level, and data with a high danger level.

[0050] Agent data 53B includes data in which agent AG performs a predetermined action while facing the rear of vehicle 100 with their gaze or posture directed toward the rear. Agent data 53B may include data in which the danger level corresponding to agent AG's emotional state is at a no-alert level.

[0051] Multiple agent data 53C includes data in which an agent AG performs a predetermined action while facing a specific direction different from the direction of the driver of the vehicle 100, either by gaze or posture. Multiple agent data 53C belong to one of several groups in which the gaze or posture differs from each other. Within each group, multiple agent data 53C includes several data in which the danger levels corresponding to the emotional state of the agent AG differ from each other. Multiple agent data 53C may include, for example, data in which the danger level is low alert, data in which the danger level is medium alert, and data in which the danger level is high alert.

[0052] Agent data 53A, 52B, and 52C are, for example, 2D animation data, 3D animation data, 2D image data (photo data), or 2D video data.

[0053] 2D animation data is, for example, data generated from at least one image (photograph) from among multiple images (photographs) included in agent data 53, at least one video from among multiple videos included in agent data 53, or at least one data from among multiple images (photographs) and multiple videos included in agent data 53. 3D animation data is, for example, data generated from at least one image (photograph) from among multiple images (photographs) included in agent data 53, at least one video from among multiple videos included in agent data 53, or at least one data from among multiple images (photographs) and multiple videos included in agent data 53.

[0054] 2D image data (photo data) is data generated from at least one image (photo) among multiple images (photos) included in agent data 53, for example. 2D video data is data generated from at least one video among multiple videos included in agent data 53, for example.

[0055] Let's assume the intimate creature is a pet (dog) owned by the driver of vehicle 100. In this case, the multiple agent data 53A includes, for example, three types of agent AGs with different danger levels, as shown in Figures 4(A), (B), and (C). Figure 4(A) shows an example of an agent AG with a low danger level. The agent AG shown in Figure 4(A) is represented as a pet (dog) with its ears perked up. Figure 4(B) shows an example of an agent AG with a medium danger level. The agent AG shown in Figure 4(B) is represented as a pet (dog) in a forward-leaning posture. Figure 4(C) shows an example of an agent AG with a high danger level. The agent AG shown in Figure 4(C) is represented as a pet (dog) with its ears flattened and its tail tucked between its hind legs. When the data of the pet (dog) shown in Figures 4(A), (B), and (C) is displayed as an agent AG on the center panel display 44, the agent AG's gaze or posture is directed towards the face of the driver of vehicle 100.

[0056] Agent data 53B includes, for example, agent AGs with a no-warning level, as shown in Figure 5. Figure 5 shows an example of an agent AG with a no-warning level. The agent AG shown in Figure 5 is represented as a pet (dog) sitting with its gaze directed towards the rear of the vehicle.

[0057] When multiple potential hazards are present, the direction of the agent AG's gaze or posture is important in preventing the driver from becoming preoccupied with a specific potential hazard. For example, if the agent AG exhibits unsettling behavior while facing a direction different from the driver, the driver may perceive a hazard in the direction the agent AG is facing. In this case, there is a concern that the driver's attention being directed towards the agent AG may cause other potential hazards in other directions to become apparent, potentially leading to a collision between vehicle 100 and something. For example, if the agent AG exhibits unsettling behavior while facing the driver, the driver may perceive multiple potential hazards and attribute the agent AG's behavior to the driver's actions (e.g., excessive speed).

[0058] Therefore, in this embodiment, when multiple potential hazards exist, each agent data 53A is used to prevent the driver from paying attention to a specific potential hazard. When only one potential hazard exists, each agent data 53C is used to direct the driver to that specific potential hazard. When no potential hazards exist, agent data 53B is used to inform the driver that no potential hazards exist.

[0059] For drivers to understand the intent behind information presentation and find meaning in acting accordingly, it is crucial that the information is presented by someone with whom they communicate frequently, whose intentions are easy to understand, and with whom they have affection and empathy. Family members (especially children) and pets are prime examples of such beings. Drivers can easily understand the intent behind anxious behaviors exhibited by such beings. Furthermore, drivers are expected to proactively take actions to alleviate the anxiety of those they are attached to (such as preventative slowing). Alternatively, drivers are expected to interpret such actions favorably as being done with their own safety in mind, making them more likely to take preventative measures such as slowing.

[0060] Therefore, in this embodiment, the agent AG used to present information is a character that reflects a being with whom the user communicates frequently, whose intentions are easy to understand, and with whom the user feels affection and empathy. Specifically, a character that reflects the aforementioned intimate creature is used as the agent AG that presents information.

[0061] The control unit 60 is capable of controlling the entire vehicle 100. The control unit 60 is, for example, a so-called ECU (Electronic Control Unit) and is composed of, for example, one or more processors and one or more memories. The control unit 60 may also be composed of, for example, a CPU (Central Processing Unit). In this case, the control unit 60 is capable of controlling the entire vehicle 100 by, for example, executing a program stored in a memory unit.

[0062] The control unit 60 includes, for example, a locator unit. The locator unit is capable of acquiring the position coordinates of the vehicle 100 based on the positioning signal received through the communication unit 20. The locator unit is capable of estimating the vehicle's position on the road map by map matching the acquired position coordinates onto route map information. Based on the acquired position coordinates of the vehicle 100, the locator unit acquires map information for a predetermined range including the vehicle 100 from the map information stored in the road map DB (database) 41, which will be described later.

[0063] The locator unit can switch to autonomous navigation, which estimates the vehicle's position on a road map based on vehicle speed, angular velocity, and longitudinal acceleration detected by the sensor unit 10, in environments where it is not possible to receive effective positioning signals from positioning satellites due to reduced sensitivity, such as when driving in a tunnel.

[0064] As described above, the locator unit estimates the position of the vehicle 100 on the road map (vehicle position) based on the positioning signal received through the communication unit 20 or the information detected by the sensor unit 10. Based on the estimated vehicle position on the road map, it is possible to determine the type of road the vehicle 100 is traveling on.

[0065] The locator unit can update the road map information stored in the road map DB 51 to the latest state using road map information acquired through external communication (vehicle-to-infrastructure communication and vehicle-to-vehicle communication) via the communication unit 20. This information update is performed not only on static information but also on quasi-static, quasi-dynamic, and dynamic information. As a result, the road map information is composed of road information and traffic information acquired through communication with the outside of the vehicle, and information on moving objects such as vehicles traveling on the road is updated in near real time.

[0066] The locator unit verifies the road map information based on the driving environment information recognized as described above, and updates the road map information stored in the road map DB51 to the latest state. This information update is performed not only on static information, but also on quasi-static, quasi-dynamic, and dynamic information. As a result, information on moving objects such as vehicles traveling on the road, as recognized as described above, is updated in real time.

[0067] The control unit 60 further includes, for example, a driving support unit 61, as shown in Figure 3. The driving support unit 61 is capable of performing data processing for driving support using an agent AG. The driving support unit 61 includes, for example, a data acquisition unit 62, a potential hazard estimation unit 63, an agent generation unit 64, and a notification control unit 65. The data acquisition unit 62 corresponds to a specific example of the "data acquisition unit" according to one embodiment of the present disclosure. The potential hazard estimation unit 63 and the agent generation unit 64 correspond to a specific example of the "data processing unit" according to one embodiment of the present disclosure. The notification control unit 65 corresponds to a specific example of the "output unit" according to one embodiment of the present disclosure.

[0068] The data acquisition unit 62 is capable of acquiring various data obtained from the sensor unit 10, various data obtained from external sources via the communication unit 20, various data obtained from the DMS 30, and various control signals obtained from the HMI 40. Based on the acquired data and various control signals, the data acquisition unit 62 is capable of acquiring road data Da and traffic participant data Db. From the various data obtained from the DMS 30, the data acquisition unit 62 is capable of acquiring image data Id or video data Ie of one or more passengers of the vehicle 100. Based on the acquired data and various control signals, the data acquisition unit 62 is capable of acquiring position data and speed data of the vehicle 100. Road data Da corresponds to one specific example of "road data" according to one embodiment of this disclosure. Traffic participant data Db corresponds to one specific example of "traffic participant data" according to one embodiment of this disclosure.

[0069] The data acquisition unit 62 is capable of outputting acquired road data Da and traffic participant data Db to the potential hazard estimation unit 63. The data acquisition unit 62 is capable of outputting acquired image data Id or video data Ie to the potential hazard estimation unit 63. The data acquisition unit 62 is capable of outputting acquired vehicle 100 position data and speed data to the potential hazard estimation unit 63.

[0070] Road data Da includes, for example, information about the road and structures on the road surrounding vehicle 100, and structures around the road. Traffic participant data Db includes, for example, information about the position and speed of each traffic participant around vehicle 100. Traffic participants are a concept that includes, for example, vehicles, bicycles, and pedestrians. Traffic participant data Db includes, for example, information about the position and speed of vehicles traveling on the road around vehicle 100, and information about the position and speed of pedestrians and bicycles in intersections.

[0071] The potential hazard estimation unit 63 is capable of estimating the presence or absence of multiple potential hazards caused by the actions of multiple traffic participants around the vehicle 100, based on at least the road data Da, out of the road data Da and traffic participant data Db.

[0072] Here, the multiple traffic participants around vehicle 100 may include multiple traffic participants included in image data Ib. The multiple traffic participants around vehicle 100 may include multiple traffic participants not included in image data Ib, but for example, included in data obtained by vehicle-to-infrastructure communication. The multiple traffic participants around vehicle 100 may include one or more traffic participants included in image data Ib and one or more traffic participants not included in image data Ib, but for example, included in data obtained by vehicle-to-infrastructure communication.

[0073] Potential hazard refers to a traffic situation (hereinafter referred to as "the first dangerous traffic situation") in which, currently, there is no possibility of a traffic accident (e.g., a collision) occurring between vehicle 100 and other traffic participants around vehicle 100, but depending on the subsequent actions of the other traffic participants around vehicle 100, there is a possibility of a traffic accident (e.g., a collision) occurring between vehicle 100 and other traffic participants around vehicle 100. In the first dangerous traffic situation, the other traffic participants around vehicle 100 may be in a position visible to the driver of vehicle 100, or they may be in a position that is difficult for the driver of vehicle 100 to see. Potential hazard further refers to a traffic situation (hereinafter referred to as "the second dangerous traffic situation") in which there is a possibility of a traffic accident (e.g., a collision) occurring between vehicle 100 and other traffic participants that are currently difficult for the driver of vehicle 100 to see.

[0074] The potential hazard estimation unit 63 can estimate the possibility of a traffic accident (e.g., a collision) occurring between the vehicle 100 and other traffic participants around the vehicle 100, that is, the presence or absence of a potential hazard (a first dangerous traffic situation or a second dangerous traffic situation), based on the time required (TTC) for the vehicle 100 and other traffic participants around the vehicle 100 to come into contact with each other. For example, if the TTC is shorter than a predetermined threshold, the potential hazard estimation unit 63 can estimate that there is a possibility of a traffic accident (e.g., a collision) occurring between the vehicle 100 and other traffic participants around the vehicle 100, that is, that a potential hazard exists. For example, if the TTC is longer than or equal to a predetermined threshold, the potential hazard estimation unit 63 can estimate that there is no possibility of a traffic accident (e.g., a collision) occurring between the vehicle 100 and other traffic participants around the vehicle 100, that is, that a potential hazard does not exist.

[0075] A potential hazard may also refer to a traffic situation in which, although the driver of vehicle 100 cannot see the traffic participant, if we assume that the traffic participant is located in a place that is difficult for the driver of vehicle 100 to see, there is a possibility of a traffic accident (for example, a collision) occurring between vehicle 100 and the hypothetical traffic participant (hereinafter referred to as the "third type of dangerous traffic situation").

[0076] The potential hazard estimation unit 63 may be capable of estimating the possibility of a traffic accident (e.g., a collision) occurring between vehicle 100 and hypothetical traffic participants, that is, the presence or absence of a potential hazard (a third dangerous traffic situation), based on the hazard level α. The potential hazard estimation unit 63 may be capable of deriving the acceleration or deceleration necessary for vehicle 100 to avoid contact with traffic participants around vehicle 100 as the hazard level α. For example, if the hazard level α is greater than a predetermined threshold, the potential hazard estimation unit 63 can estimate that there is a possibility of a traffic accident (e.g., a collision) occurring between vehicle 100 and traffic participants around vehicle 100, that is, that a potential hazard exists. For example, if the hazard level α is less than or equal to a predetermined threshold, the potential hazard estimation unit 63 can estimate that there is no possibility of a traffic accident (e.g., a collision) occurring between vehicle 100 and traffic participants around vehicle 100, that is, that a potential hazard does not exist.

[0077] The potential hazard estimation unit 63, when the data acquisition unit 62 acquires road data Da, traffic participant data Db, and position and speed data of vehicles 100, 100a, and 100c in the traffic conditions shown in Figure 6, for example, the data acquisition unit 62 acquires road data Da, traffic participant data Db, and position and speed data of vehicles 100, 100a, and 100c, and based on the various data acquired by the data acquisition unit 62, is able to calculate the hazard levels of multiple potential hazards in the traffic conditions shown in Figure 6, according to the time required (time to traffic congestion) for vehicle 100, vehicle 100a, and vehicle 100c to come into contact with each other, or the magnitude of the deceleration (hazard level α) required for vehicle 100 to avoid colliding with vehicles 100a and 100c.

[0078] The potential hazard estimation unit 63 can output a flag to the agent generation unit 64 indicating that it has estimated the existence of multiple potential hazards when it has estimated the existence of multiple potential hazards. The potential hazard estimation unit 63 can output a flag to the agent generation unit 64 indicating that it has estimated the existence of no potential hazards when it has estimated that no potential hazards exist. The potential hazard estimation unit 63 can output a flag to the agent generation unit 64 indicating that it has estimated the existence of one potential hazard when it has estimated the existence of one potential hazard.

[0079] As described above, the multiple potential hazards can arise from the actions of multiple traffic participants around vehicle 100, and not from the actions of the driver of vehicle 100 (e.g., distracted driving, running a red light, exceeding the speed limit, entering a no-entry road, etc.). In other words, the multiple potential hazards can arise from the actions of multiple traffic participants around vehicle 100, even when the driver of vehicle 100 is complying with traffic laws and driving safely.

[0080] For example, at an unsignaled intersection IS with poor visibility as shown in Figure 2, several potential hazards include the risk of another vehicle suddenly appearing from the intersecting road Lb, and the risk of a motorcycle passing vehicle 100 on its left side while vehicle 100 is attempting to turn left. Also, for example, at a three-lane road as shown in Figure 6, several potential hazards include the risk of vehicle 100a, traveling in lane La1 adjacent to lane La2 in which vehicle 100 is traveling, suddenly changing lanes in front of vehicle 100 at a constant speed to avoid a stationary vehicle 100b, and the risk of vehicle 100c, traveling in lane La3 adjacent to lane La2, suddenly changing lanes in front of vehicle 100 at a predetermined acceleration from a line of cars traveling at a low speed due to congestion.

[0081] The potential hazard estimation unit 63 is capable of calculating the hazard level of the potential hazard obtained through estimation based on the time limit (TTC) or the hazard degree α. For example, the potential hazard estimation unit 63 uses the following formula (1) to calculate the hazard level D of the potential hazard. TTCIt is possible to calculate. In the following formula (1), the time to collision (TTC) is, for example, TTC_ MIN or more, and it is assumed that the value is smaller than TTC_ MAX . D_ TTC =(TTC_ MAX -TTC) / (TTC_ MAX -TTC_ MIN )…(1) TTC_ MAX : The maximum value that can be taken as the time to collision (TTC) TTC_ MIN : The minimum value that can be taken as the time to collision (TTC)

[0082] The potential danger estimation unit 63 can calculate the danger level D_α of the potential danger obtained by estimation, for example, using the following formula (2). In the following formula (2), the danger degree α is, for example, α_ MIN or more, and it is assumed that the value is smaller than α_ MAX . D_α=(α - α_ MIN ) / (α_ MAX -α_ MIN )…(2) α_ MAX : The maximum value that can be taken as the danger degree α α_ MIN : The minimum value that can be taken as the danger degree α

[0083] When the potential danger estimation unit 63 estimates the existence of a plurality of potential dangers, for each potential danger, it is possible to calculate the danger level D_ TTC or the danger level D_α. When the potential danger estimation unit 63 estimates a plurality of potential dangers, it is possible to set the highest value among the calculated plurality of danger levels as the maximum danger level D_ MAX . When the potential danger estimation unit 63 estimates a plurality of potential dangers, it is possible to output the maximum danger level D_ MAX to the agent generation unit 64. When the potential danger estimation unit 63 estimates one potential danger, the calculated danger degree (danger level D_ TTCAlternatively, it is possible to output a danger level D_α) to the agent generation unit 64.

[0084] The agent generation unit 64 is capable of acquiring one or more materials (images (photos) or videos) containing intimate creatures from the history data 52 of the memory unit 50. The agent generation unit 64 is capable of identifying one or more passengers (passenger creatures) of the vehicle 100 included in the history data 52, for example, based on the history data 52. The agent generation unit 64 is capable of deriving the passenger frequency and the elapsed time since the start of passenger riding for each passenger (passenger creature) based on the data (image data Id, video data Ie, and time data) of the identified one or more passengers (passenger creatures). The elapsed time refers, for example, to the period from the day the passenger (passenger creature) first got into the vehicle 100 to the present. The passenger frequency refers, for example, to the value obtained by dividing the number of times the passenger (passenger creature) rode in the vehicle during the elapsed time by the number of times the driver of the vehicle 100 rode in the vehicle during the elapsed time.

[0085] The agent generation unit 64 can, for example, set a close companion from among one or more passengers (passenger creatures) of the vehicle 100 based on the frequency and elapsed time of each passenger. The agent generation unit 64 can, for example, set the passenger (passenger creature) with the highest frequency of riding among one or more passengers (passenger creatures) whose riding period exceeds a predetermined period as a close companion. The agent generation unit 64 can, for example, acquire one or more materials (images (photos) or videos) showing the passenger (passenger creature) set as a close companion from the history data 52 of the storage unit 50.

[0086] The agent generation unit 64 is capable of estimating the emotional state of intimate creatures depicted in each material (image (photo) or video) acquired from the historical data 52. The agent generation unit 64 may, for example, estimate the emotional state based on FACS (Facial Action Coding System) theory, and may be capable of determining the degree of a certain emotion (e.g., anxiety). The agent generation unit 64 is capable of setting a danger level corresponding to the estimated emotional state for each material. For example, if the intimate creature included in the material is a pet (dog), and the pet (dog) is expressing strong anxiety, the agent generation unit 64 can set a high alert level for this material. For example, if the intimate creature included in the material is a pet (dog), and the pet (dog) is expressing moderate anxiety, the agent generation unit 64 can set a medium alert level for this material. For example, if the intimate creature included in the material is a pet (dog), and the pet (dog) is expressing low anxiety, the agent generation unit 64 can set a low alert level for this material. The agent generation unit 64 is capable of associating the set danger level with each material and storing it in the agent data 53 of the memory unit 50.

[0087] The agent generation unit 64 can read multiple materials of a common danger level from the agent data 53 in the memory unit 50, and use the read materials of a common danger level to generate a friendly agent (agent AG) that mimics a friendly creature. The agent generation unit 64 can generate an agent AG for each danger level. For example, the agent generation unit 64 can generate an agent AG that performs low-alert level actions, an agent AG that performs medium-alert level actions, and an agent AG that performs high-alert level actions. The agent generation unit 64 may also read one material from the agent data 53 in the memory unit 50, and use the read material to generate a friendly agent (agent AG) that mimics a friendly creature. In this way, the agent generation unit 64 can generate a friendly agent (agent AG) that mimics a friendly creature using one or more materials (images (photos) or videos) that show a friendly creature. The agent generation unit 64 can perform data processing to make the agent AG perform actions that mimic the actions of a friendly creature. The agent generation unit 64 is capable of performing data processing to generate agent data 53A, agent data 53B, and agent data 53C, as described later.

[0088] The agent generation unit 64 is capable of generating agent data 53A that causes agent AG to perform actions according to the danger level while directing its gaze or posture toward the driver of vehicle 100. The agent generation unit 64 can generate agent data 53A using a learning model such as GAN (Generative Adversarial Networks). Here, the learning model used to generate agent data 53A is a model that has been trained using multiple materials of a common danger level and data about gaze or posture as teaching data. This learning model may also be a model that has been trained using one material and data about gaze or posture as teaching data. The agent generation unit 64 is capable of storing the generated agent data 53A in the agent data 53 of the storage unit 50 in association with the danger level.

[0089] The agent generation unit 64 is capable of generating agent data 53B that causes agent AG to perform actions at a non-vigilant level while facing the rear of vehicle 100 with its gaze or posture directed toward the rear. The agent generation unit 64 can generate agent data 53B using, for example, a learning model such as GAN. Here, the learning model used to generate agent data 53B is a model that has learned multiple materials with a danger level of non-vigilant level and data about gaze or posture as teaching data. This learning model may also be a model that has learned one material with a danger level of non-vigilant level and data about gaze or posture as teaching data. The agent generation unit 64 is capable of storing the generated agent data 53B in the agent data 53 of the memory unit 50 in association with the danger level (non-vigilant level).

[0090] The agent generation unit 64 is capable of generating agent data 53C that causes agent AG to perform actions according to the danger level while facing a specific direction different from the direction of the vehicle 100's driver, either by gaze or posture. The agent generation unit 64 can generate agent data 53C using, for example, a learning model such as a GAN. Here, the learning model used to generate agent data 53C is a model that has learned multiple materials of a common danger level and data about gaze or posture as teaching data. This learning model may also be a model that has learned one material and data about gaze or posture as teaching data. The agent generation unit 64 is capable of storing the generated agent data 53C in the agent data 53 of the storage unit 50 in association with the danger level.

[0091] The agent generation unit 64 receives a flag from the potential risk estimation unit 63 indicating that multiple potential risks have been estimated, and the risk levels (maximum risk level D_) of the multiple potential risks obtained through estimation. MAX When the input is received, the agent data 53A associated with the input danger level can be read from the agent data 53 in the storage unit 50. The agent data 53A is obtained by estimation with the gaze or posture directed towards the driver of the vehicle 100, and contains multiple danger levels of potential hazards (maximum danger level D_ MAX This is data that causes Agent AG to perform actions corresponding to the given situation.

[0092] When the agent generation unit 64 receives a flag from the potential danger estimation unit 63 indicating that it has estimated that no potential danger exists, it can read agent data 53B from the agent data 53 in the memory unit 50. Agent data 53B is data that causes agent AG to perform actions at a level of no vigilance while facing the rear of the vehicle 100 with its gaze or posture directed toward the rear.

[0093] The agent generation unit 64 receives a flag from the potential risk estimation unit 63 indicating that one potential risk has been estimated, and the risk level (risk level D_) of the specific potential risk obtained through the estimation. TTC When a danger level (D_α) is input, the agent data 53C associated with the input danger level can be read from the agent data 53 in the memory unit 50. The agent data 53C is data that causes agent AG to perform actions according to the danger level while facing the direction of the estimated potential danger, either by gaze or posture.

[0094] Thus, when the agent generation unit 64 estimates multiple potential hazards in the potential hazard estimation unit 63, it directs its gaze or posture toward the driver of the vehicle 100 and determines the hazard level of the multiple potential hazards obtained by the estimation (maximum hazard level D_ MAX As data processing to cause agent AG to perform actions corresponding to the risk level (no-warning level) obtained by estimation, the agent generation unit 64 can read agent data 53A from agent data 53 in the storage unit 50. Furthermore, when the potential risk estimation unit 63 estimates that there are no potential risks, the agent generation unit 64 can read agent data 53B from agent data 53 in the storage unit 50 as data processing to cause agent AG to perform actions corresponding to the risk level (no-warning level) obtained by estimation, with their gaze or posture directed toward the rear of the vehicle 100. Furthermore, when the potential risk estimation unit 63 estimates one potential risk, the agent generation unit 64 can read agent data 53B from agent data 53 in the storage unit 50 as data processing to cause agent AG to perform actions corresponding to the risk level (risk level D_) of the specific potential risk obtained by estimation, with their gaze or posture directed toward a specific direction different from the direction the driver of the vehicle 100 is facing. TTC Alternatively, as data processing to cause agent AG to perform actions corresponding to danger level D_α), it is possible to read agent data 53C from agent data 53 in the storage unit 50.

[0095] The notification control unit 65 is capable of generating a video signal including agent AG based on data obtained from data processing performed by the agent generation unit 64 to cause agent AG to perform predetermined actions. The notification control unit 65 is capable of generating a video signal including agent AG based on agent data 53A, agent data 53B, or agent data 53C obtained from the agent generation unit 64. The notification control unit 35 is capable of outputting the generated video signal to, for example, the center panel display 44. As a result, the center panel display 44 is capable of displaying agent AG performing predetermined actions, for example, as shown in Figure 1.

[0096] The control unit 60 further includes a driving control unit 66, as shown in Figure 3, for example. The driving control unit 66 is capable of controlling the driving of the vehicle 100 (for example, the torque of the prime mover 70, the amount of pressure applied to the brake 80, and the steering angle of the steering wheel 90). The driving control unit 66 includes an accelerator control unit 67, a brake control unit 68, and a steering control unit 69, as shown in Figure 3, for example.

[0097] The accelerator control unit 67 is capable of controlling the torque of the prime mover 70 based on the requested torque corresponding to the amount the driver of the vehicle 100 depresses the accelerator pedal. The prime mover 70 is configured to drive the steering wheels of the vehicle 100 and is capable of driving the steering wheels of the vehicle 100 according to the requested torque or target torque input from the accelerator control unit 67.

[0098] The brake control unit 68 is capable of controlling the torque of the brake 80 based on the requested torque corresponding to the amount the driver of the vehicle 100 presses the brake pedal. The brake 80 is configured to brake the steering wheels of the vehicle 100 and is capable of braking the steering wheels of the vehicle 100 according to the requested torque or target torque input from the brake control unit 68.

[0099] The steering control unit 69 can derive a steering assist torque to assist the steering torque generated by the driver's steering wheel operation, and can set an EPS torque corresponding to the derived steering assist torque. The steering control unit 69 can output a control signal to the EPS motor 90 so that the output torque of the EPS motor 90 becomes the set EPS torque. The EPS motor 90 generates an output torque based on the input control signal and can control the steering angle of the steering wheel.

[0100] Next, we will explain the driving assistance procedures for vehicle 100.

[0101] Figure 7 shows an example of a driving assistance procedure in vehicle 100. The control unit 60 determines whether the control flag for the driving assistance mode is ON or OFF (step S101). If the control flag is ON (step S101; Y), the control unit 60 determines whether agent data 53 is stored in the memory unit 50 (step S102). If agent data 53 is not stored in the memory unit 50 (step S102; N), the control unit 60 generates agent data 53 (step S103).

[0102] If agent data 53 is stored in the storage unit 50 (step S102; Y), or if agent data 53 is generated, the control unit 60 acquires road data Da and traffic participant data Db, etc. (step S104). Based on at least the road data Da from the road data Da and traffic participant data Db, the control unit 60 estimates whether there are multiple potential hazards caused by the actions of multiple traffic participants around the vehicle 100 (step S105).

[0103] When the control unit 60 estimates the existence of multiple potential hazards (step S106; Y), it determines the maximum hazard level D_ among the multiple potential hazards obtained by the estimation. MAXThe control unit 60, with its gaze or posture directed towards the driver of vehicle 100, determines the maximum danger level D_ MAX Agent data 53A, which causes agent AG to perform the corresponding action, is obtained from agent data 53 (step S107).

[0104] When the control unit 60 estimates that no potential hazard exists (step S106;N, step S108;N), it obtains agent data 53B from agent data 53, which causes agent AG to perform a no-vigilance level action with its gaze or posture directed towards the rear of the vehicle 100 (step S109). When the control unit 60 estimates one potential hazard (step S106;N, step S108;Y), it directs its gaze or posture toward the direction of the specific potential hazard, and determines the hazard level (hazard level D_ TTC Alternatively, agent data 53C is obtained from agent data 53 to cause agent AG to perform an action corresponding to the danger level D_α) (step S110). Based on the acquired agent data, the control unit 60 displays a video including agent AG performing an action that mimics the actions of an intimate creature (step S111). In this way, driving assistance for vehicle 100 is provided.

[0105] [effect] Next, I will explain the effects of vehicle 100.

[0106] In this embodiment, based on at least the road data Da of the road data Da and traffic participant data Db surrounding the vehicle 100, the presence or absence of multiple potential hazards caused by the actions of multiple traffic participants around the vehicle 100 is estimated. When the presence of multiple potential hazards is estimated, data processing is performed in which an agent AG, generated using agent data, performs actions corresponding to the danger levels of the multiple potential hazards obtained by estimation, while keeping its gaze or posture directed towards the driver. Specifically, as part of this data processing, agent data 52A is acquired from agent data 53. Then, based on the data obtained by data processing, a video signal including agent AG is generated and output to, for example, the center panel display 44. At this time, when the presence of multiple potential hazards is estimated, the agent AG output to the center panel display 44 performs actions corresponding to the danger levels of the multiple potential hazards obtained by estimation, while keeping its gaze or posture directed towards the driver. This allows for notification to the driver of vehicle 100 that multiple potential hazards exist, while also preventing the driver from becoming preoccupied with only a specific hazard. As a result, the driver of vehicle 100 can take preventative actions (such as slowing down) that contribute to safety against multiple hazards.

[0107] In this embodiment, an agent AG that mimics a person or pet that is close to the driver of vehicle 100 is generated using an image or video of the pet. Furthermore, data processing is performed to make the agent AG perform actions that mimic the actions of the pet. Specifically, this data processing involves generating agent data 52A. This allows the video using the generated agent data 52A to be displayed on the center panel display 44. As a result, when the agent AG, which the driver of vehicle 100 is likely to feel a sense of familiarity with, directs its gaze and facial expressions towards the driver of vehicle 100, the driver of vehicle 100 can become interested in the agent AG's gaze and facial expressions, prompting them to take action based on the existence of multiple potential hazards. This allows the driver of vehicle 100 to be informed of the existence of multiple potential hazards, while preventing the driver from becoming preoccupied with only one hazard. At the same time, it is possible to provide notification that makes the driver understand the intention of the information presentation and feel that taking action in accordance with that intention is meaningful. As a result, the driver of vehicle 100 can take preventative actions (e.g., slowing down) that contribute to safety against multiple hazards.

[0108] In this embodiment, the emotional state of the intimate creature depicted in each material (image (photo) or video) consisting of images or videos is estimated, and the danger level corresponding to the estimated emotional state is associated with each material and stored in the agent data 53 of the memory unit 50. This allows the agent AG to perform actions according to the danger level. As a result, the agent AG can more effectively provide notifications to the driver of the vehicle 100, making them understand the intent behind the information presentation and feel that taking action in accordance with that intent is meaningful. Consequently, the driver of the vehicle 100 can take preventative actions (e.g., slowing down) that contribute to safety against multiple dangers.

[0109] In this embodiment, the risk level of the potential hazard obtained through estimation is calculated, one or more materials associated with the calculated risk level are read from the agent data 53 of the storage unit 50, and an agent AG is generated using the one or more materials read out. This makes it possible to make the agent AG perform actions according to the risk level. As a result, the agent AG can more effectively provide notifications to the driver of the vehicle 100 so that they understand the intention of presenting the information and feel that taking action in accordance with that intention is meaningful. As a result, the driver of the vehicle 100 can take preventive actions (e.g., deceleration) that contribute to safety against multiple hazards.

[0110] In this embodiment, for example, when road data Da, traffic participant data Db, and position and speed data of vehicles 100, 100a, and 100c are acquired in the traffic situation shown in Figure 6, the danger levels of multiple potential hazards in the traffic situation shown in Figure 6 are derived based on the acquired data, according to the time required for vehicle 100, vehicle 100a, and vehicle 100c to come into contact with each other (time limit TTC), or the magnitude of the deceleration required for vehicle 100 to avoid colliding with vehicles 100a and 100c (danger level α). This allows the agent AG to perform actions corresponding to the danger level. As a result, the agent AG can more effectively provide information to the driver of vehicle 100, making them understand the intent behind the information presentation and feel that taking action in accordance with that intent is meaningful. As a result, the driver of vehicle 100 can take preventive actions (e.g., deceleration) that contribute to safety against multiple hazards.

[0111] In this embodiment, based on the history data 52, one or more passengers (passenger creatures) of the vehicle 100 included in the history data 52 are identified, and based on the data of the identified one or more passengers (passenger creatures) (image data Id, video data Ie, and time data), the frequency of riding and the elapsed time since the start of riding for one or more passengers (passenger creatures) of the vehicle 100 are derived. Based on the frequency of riding and the elapsed time for one or more passengers (passenger creatures) of the vehicle 100, a close creature is set from among the one or more passengers (passenger creatures) of the vehicle 100. Among the passengers whose riding period exceeds a predetermined period, the passenger with the highest frequency of riding is set as the close creature. This allows Agent AG, which is more likely to evoke a sense of familiarity in the driver of Vehicle 100, to direct its gaze and facial expressions towards the driver of Vehicle 100. This makes it possible to attract the driver's attention to Agent AG's gaze and facial expressions, more effectively enabling them to understand the intent behind the information presentation and to provide notification that makes them feel that taking action in accordance with that intent is meaningful. As a result, the driver of Vehicle 100 becomes able to take preventative actions (such as slowing down) that contribute to safety against multiple hazards.

[0112] <3. Variant> Next, a modified example of the vehicle 100 according to the above embodiment will be described.

[0113] [Differentiation A] In the above embodiment, the control unit 60 pre-generates a plurality of agent data 53A with different actions (risk levels) and stores them in the storage unit 50. The control unit 60 then extracts agent data 53A from the storage unit 50 corresponding to the risk levels of multiple potential hazards obtained by estimation, and uses the extracted agent data 53A to cause agent AG to perform predetermined actions.

[0114] However, in the above embodiment, when the control unit 60 (agent generation unit 64) obtains risk levels for multiple potential hazards through estimation, it may read one or more materials (images (photos) or videos of intimate creatures) associated with the obtained risk levels from the history data 52 of the storage unit 50, and use the read one or more materials (images (photos) or videos of intimate creatures) to generate an intimate agent (agent AG) that mimics an intimate creature. At this time, the control unit 60 (agent generation unit 64) is capable of performing data processing to make the agent AG perform actions that mimic the actions of an intimate creature. As such data processing, the control unit 60 (agent generation unit 64) is capable of performing processing to generate agent data 53A.

[0115] Next, the driving assistance procedure for vehicle 100 related to this modified example will be described.

[0116] Figure 8 shows an example of the driving assistance procedure in the vehicle 100 according to this modified example. The control unit 60 determines whether the control flag for the driving assistance mode is ON or OFF (step S201). If the control flag is ON (step S201; Y), the control unit 60 acquires road data Da and traffic participant data Db, etc. (step S202). Based on at least the road data Da from the road data Da and traffic participant data Db, the control unit 60 estimates the presence or absence of multiple potential hazards caused by the actions of multiple traffic participants around the vehicle 100 (step S203).

[0117] When the control unit 60 estimates the existence of multiple potential hazards (step S204; Y), it sets the hazard level for the multiple potential hazards obtained by the estimation (maximum hazard level D_ MAX The control unit 60 calculates the maximum danger level D_ with its gaze or posture directed towards the driver of the vehicle 100. MAXAgent data 53A is generated to cause agent AG to perform actions corresponding to the (step S205). At this time, the control unit 60 calculates the danger level (maximum danger level D_ MAX The control unit 60 reads one or more materials (images (photos) or videos) associated with the intimate creature from the history data 52 of the storage unit 50. The control unit 60 then uses the read one or more materials (images (photos) or videos) to generate an intimate agent (agent AG) that mimics the intimate creature. The control unit 60 performs data processing to make the agent AG perform actions that mimic the actions of the intimate creature. As part of such data processing, the control unit 60 performs processing to generate agent data 53A.

[0118] When the control unit 60 estimates that no potential hazard exists (step S204;N, step S206;N), it generates agent data 53B instructing agent AG to perform actions at a level of no vigilance while facing the rear of the vehicle 100 (step S207). When the control unit 60 estimates the presence of one potential hazard (step S204;N, step S206;Y), it directs agent AG's gaze or posture toward the direction of the specific potential hazard and generates agent data 53B instructing agent AG to perform actions at a level of no vigilance. TTC Alternatively, agent data 53C is generated to cause agent AG to perform actions corresponding to the danger level D_α) (step S208). Based on the acquired agent data, the control unit 60 displays a video including agent AG performing actions that mimic the actions of an intimate creature (step S209). In this way, driving assistance for the vehicle 100 according to this modified example is provided.

[0119] In this modified example, based on at least the road data Da of the road data Da and traffic participant data Db surrounding the vehicle 100, the presence or absence of multiple potential hazards caused by the actions of multiple traffic participants around the vehicle 100 is estimated. When multiple potential hazards are estimated, data processing is performed in which an agent AG, generated using agent data, performs actions corresponding to the hazard levels of the multiple potential hazards obtained by the estimation, while the driver's gaze or posture is directed towards the driver. Specifically, this data processing involves generating agent data 52A. Then, based on the data obtained by the data processing, a video signal including the agent AG is generated and output to, for example, the center panel display 44. As a result, the agent AG output to the center panel display 44 can suggest to the driver of the vehicle 100 that multiple potential hazards exist and provide notification to prevent the driver from becoming preoccupied with only one hazard. As a result, the driver of the vehicle 100 can take preventive actions (e.g., deceleration) that contribute to safety against multiple hazards.

[0120] [Variation B] In the above embodiment and modified example A, the HMI 40 may further include a HUD (Head-Up Display) 47, as shown in Figure 9, for example. The HUD 47 corresponds to one specific example of the "display unit" according to one embodiment of the present disclosure.

[0121] At this time, the notification control unit 35 can output the generated video signal to, for example, the HUD 47. The HUD 47 is a display device that can project an image onto the display surface 47A of the front windshield FW, thereby superimposing the projected image onto the scenery in front of the vehicle 100. The HUD 47 can display animations, images (photographs), or videos including an agent AG on the display surface 47A, as shown in Figure 9, for example.

[0122] In this modified version, the HUD 47 is used to display an animation, image (photo), or video including agent AG on the display surface 47A. In this case, agent AG output to the display surface 47A can inform the driver of vehicle 100 that multiple potential hazards exist, and can also provide notification to prevent the driver from becoming preoccupied with only one hazard. As a result, the driver of vehicle 100 can take preventative actions (e.g., deceleration) that contribute to safety against multiple hazards.

[0123] [Differentiation C] In the above embodiments and modified versions A and B, the agent generation unit 64 may be capable of setting a person or pet specified by the driver of the vehicle 100 as a close companion. In this case, the agent generation unit 64 is capable of acquiring an image (photo) or video of the person or pet specified by the driver of the vehicle 100 from the history data 52 of the storage unit 50. In this case, the series of information processing required for setting a close companion in the agent generation unit 64 can be omitted.

[0124] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0125] Furthermore, for example, this disclosure can take the following configuration. (1) A data acquisition unit capable of acquiring road data and traffic participant data around the vehicle, as well as agent data, A data processing unit capable of estimating the presence or absence of multiple potential hazards caused by the actions of multiple traffic participants around the vehicle, based at least on the road data, and when the presence or absence of multiple potential hazards is estimated, using the agent data, causing the agent to perform actions corresponding to the risk level of the multiple potential hazards obtained by estimation, while facing the driver with its gaze or posture; Based on the data obtained by the above data processing, an output unit capable of generating a video signal including the agent and outputting it to a display unit. Equipped with Driving assistance system. (2) The data acquisition unit is capable of acquiring images or videos of intimate animals, such as people or pets, that are intimate with the driver of the vehicle, as agent data. The data processing unit is capable of generating a closeness agent that mimics the closeness organism using the image or video, and further performing data processing to cause the closeness agent to perform actions that mimic the actions of the closeness organism. (1) The driving assistance device described above. (3) The agent data comprises multiple materials consisting of images or videos. The data processing unit is capable of estimating the emotional state of the intimate creatures depicted in each of the materials, and storing the estimated danger level corresponding to the emotional state in the memory unit in association with each of the materials. (2) The driving assistance device described above. (4) The data processing unit can calculate the risk level of the potential risk obtained by estimation, read one or more of the materials associated with the calculated risk level from the storage unit, and generate the agent using the one or more of the materials read out. (3) The driving assistance device described above. (5) The data acquisition unit is capable of acquiring the vehicle's position data and speed data, as well as the traffic participant data, which includes the position data and speed data of a first traffic participant and a second traffic participant surrounding the vehicle. The data processing unit calculates the risk level of the multiple potential hazards based on the road data, the traffic participant data, the vehicle's position data and speed data, and the respective position data and speed data of the first and second traffic participants around the vehicle, according to the margin of time before the vehicle collides with the first and second traffic participants, or the amount of deceleration required to avoid a collision between the vehicle and the first and second traffic participants. A driving assistance device described in any one of (1) to (4). (6) The data processing unit can identify one or more passengers riding with the driver, and based on the frequency of each passenger and the time elapsed since the start of the ride, it can set the "closest" passenger from among the one or more passengers and acquire one or more images or videos of the closest passenger as agent data. A driving assistance device described in any one of (1) to (5). (7) A vehicle equipped with a driver assistance system and a display unit, The aforementioned driving support device, A data acquisition unit that acquires road data and traffic participant data around the vehicle, and agent data, A data processing unit capable of estimating the presence or absence of multiple potential hazards caused by the actions of multiple traffic participants around the vehicle, based at least on the road data, and when the presence or absence of multiple potential hazards is estimated, using the agent data, causing the agent to perform actions corresponding to the risk level of the multiple potential hazards obtained by estimation, while facing the driver with its gaze or posture; An output unit capable of generating a video signal including the agent based on the data obtained by the above data processing and outputting it to the display unit. It has, The display unit is capable of displaying video including the agent based on the video signal input from the output unit. vehicle.

[0126] In one embodiment of the driver assistance system described herein, the presence or absence of multiple potential hazards caused by the actions of multiple traffic participants around the vehicle is estimated based on at least the road data, among the road data and traffic participant data. Thus, this driver assistance system estimates the presence or absence of multiple potential hazards that are not attributable to the driver and are not easily predictable by the driver, rather than multiple potential hazards that are attributable to the driver. When the presence or absence of multiple potential hazards is estimated in this driver assistance system, data processing is performed using agent data to cause the agent to perform actions corresponding to the danger level of the multiple potential hazards obtained by estimation, while facing the driver with its gaze or posture. Then, based on the data obtained by the data processing, a video signal including the agent is generated and output to the display unit.

[0127] This allows the vehicle driver to observe the actions performed by the agent while its gaze or posture is directed towards the driver, understand the intent behind the agent's actions, and take action accordingly. However, if the agent looks in the direction of a potential hazard, the vehicle driver may try to look in the direction the agent is looking. In this case, for example, a potential hazard in a direction the vehicle driver is not looking may become apparent, raising concerns that the vehicle may collide with another vehicle approaching from a direction the driver is not looking. However, with this driver assistance system, since the agent's gaze or posture is directed towards the driver, the apparent manifestation of a potential hazard in a direction the vehicle driver is not looking at is suppressed. Furthermore, if the agent performs an unsettling action while its gaze or posture is directed towards the driver, the vehicle driver can understand that the potential hazard exists not just in one specific location, but in multiple specific locations. As a result, the vehicle driver can take action to prepare for multiple potential hazards present around the vehicle. Therefore, this driver assistance system can inform the vehicle driver of the presence of multiple hazards while preventing the driver from becoming preoccupied with only one specific hazard.

[0128] In a vehicle according to one embodiment of this disclosure, the presence or absence of multiple potential hazards caused by the actions of multiple traffic participants around the vehicle is estimated based on at least the road data, among the road data and traffic participant data. Thus, this driver assistance system estimates the presence or absence of multiple potential hazards that are not attributable to the vehicle driver and are not easily predictable by the vehicle driver, rather than multiple potential hazards that are attributable to the vehicle driver. When the presence or absence of multiple potential hazards is estimated in this driver assistance system, data processing is performed using agent data to cause the agent to perform actions corresponding to the danger level of the multiple potential hazards obtained by estimation, while facing the driver with its gaze or posture. Then, based on the data obtained by the data processing, a video signal including the agent is generated and output to the display unit.

[0129] This allows the vehicle driver to observe the actions performed by the agent while its gaze or posture is directed towards the driver, understand the intent behind the agent's actions, and take action accordingly. However, if the agent looks in the direction of a potential hazard, the vehicle driver may try to look in the direction the agent is looking. In this case, for example, a potential hazard in a direction the vehicle driver is not looking may become apparent, raising concerns that the vehicle may collide with another vehicle approaching from a direction the driver is not looking. However, with this driver assistance system, since the agent's gaze or posture is directed towards the driver, the apparent manifestation of a potential hazard in a direction the vehicle driver is not looking at is suppressed. Furthermore, if the agent performs an unsettling action while its gaze or posture is directed towards the driver, the vehicle driver can understand that the potential hazard exists not just in one specific location, but in multiple specific locations. As a result, the vehicle driver can take action to prepare for multiple potential hazards present around the vehicle. Therefore, this driver assistance system can inform the vehicle driver of the presence of multiple hazards while preventing the driver from becoming preoccupied with only one specific hazard.

[0130] The control unit 60 shown in Figure 3 can be implemented by a circuit including at least one semiconductor integrated circuit, such as at least one processor (e.g., a central processing unit (CPU)), at least one application-specific integrated circuit (ASIC) and / or at least one field-programmable gate array (FPGA). At least one processor can be configured to perform all or some of the functions of the control unit 60 shown in Figure 3 by reading instructions from at least one non-temporary, tangible computer-readable medium. Such a medium can take various forms, including, but is not limited to, various magnetic media such as hard disks, various optical media such as CDs or DVDs, and various semiconductor memories (i.e., semiconductor circuits) such as volatile or non-volatile memory. Volatile memory may include DRAM and SRAM. Non-volatile memory may include ROM and NVRAM. An ASIC is an integrated circuit (IC) specialized to perform all or some of the functions of the control unit 60 shown in Figure 3. An FPGA is an integrated circuit designed to be configurable after manufacturing to perform all or some of the functions of the control unit 60 shown in Figure 3. [Explanation of Symbols]

[0131] 10...Sensor unit, 11...Front camera, 20...Communication unit, 30...DMS, 31...In-car camera, 40...HMI, 41...Steering wheel, 42...Switch, 43...Meter panel display, 44...Center panel display, 45...Speaker, 46...Steering switch, 47...HUD, 47A...Display area, 50...Storage unit, 51...Road map DB, 52...Agent data, 60...Control unit, 61...Driving control unit, 62...Data acquisition unit, 63...Latent Hazard estimation unit, 64... Agent generation unit, 65... Notification control unit, 66... ​​Driving control unit, 67... Accelerator control unit, 68... Brake control unit, 69... Steering control unit, 70... Prime mover, 80... Brake, 90... EPS motor, 100, 100a, 100b, 100c... Vehicle, AG... Agent, Da... Road data, Db... Traffic participant data, FW... Front windshield, Ia... Image data, IS... Intersection, La... Driving path, La1, La2, La3... Lanes, Lb... Crossroads.

Claims

1. A data acquisition unit capable of acquiring road data and traffic participant data around the vehicle, as well as agent data, A data processing unit capable of estimating the presence or absence of multiple potential hazards caused by the actions of multiple traffic participants around the vehicle, based at least on the road data, and when the presence or absence of multiple potential hazards is estimated, having an agent generated using the agent data perform actions corresponding to the risk level of the multiple potential hazards obtained by estimation, while facing the driver with its gaze or posture; Based on the data obtained by the above data processing, an output unit capable of generating a video signal including the agent and outputting it to a display unit. Equipped with Driving assistance system.

2. The data acquisition unit is capable of acquiring images or videos of intimate animals, such as people or pets, that are intimate with the driver of the vehicle, as agent data. The data processing unit is capable of generating a closeness agent that mimics the closeness organism using the image or video, and further performing data processing to cause the closeness agent to perform actions that mimic the actions of the closeness organism. The driving support device according to claim 1.

3. The agent data comprises multiple materials consisting of images or videos. The data processing unit is capable of estimating the emotional state of the intimate creatures depicted in each of the materials, and storing the estimated danger level corresponding to the emotional state in the memory unit in association with each of the materials. The driving support device according to claim 2.

4. The data processing unit can calculate the risk level of the potential risk obtained by estimation, read one or more of the materials associated with the calculated risk level from the storage unit, and generate the agent using the read one or more of the materials. The driving support device according to claim 3.

5. The data acquisition unit is capable of acquiring the vehicle's position data and speed data, as well as the traffic participant data, which includes the position data and speed data of a first traffic participant and a second traffic participant surrounding the vehicle. The data processing unit calculates the risk level of the multiple potential hazards based on the road data, the traffic participant data, the vehicle's position data and speed data, and the respective position data and speed data of the first and second traffic participants around the vehicle, according to the margin of time before the vehicle collides with the first and second traffic participants, or the amount of deceleration required to avoid a collision between the vehicle and the first and second traffic participants. The driving support device according to claim 1.

6. The data processing unit can identify one or more passengers riding with the driver, and based on the frequency of each passenger and the time elapsed since the start of the ride, it can set a close companion from among the one or more passengers and acquire one or more images or videos of the close companion as agent data. The driving support device according to claim 1.

7. A vehicle equipped with a driver assistance system and a display unit, The aforementioned driving support device, A data acquisition unit that acquires road data and traffic participant data around the vehicle, and agent data, A data processing unit capable of estimating the presence or absence of multiple potential hazards caused by the actions of multiple traffic participants around the vehicle, based at least on the road data, and when the presence or absence of multiple potential hazards is estimated, having an agent generated using the agent data perform actions corresponding to the risk level of the multiple potential hazards obtained by estimation, while facing the driver with its gaze or posture; An output unit capable of generating a video signal including the agent based on the data obtained by the above data processing and outputting it to the display unit. It has, The display unit is capable of displaying video including the agent based on the video signal input from the output unit. vehicle.

Images