Driver assistance device, driver assistance method, and driver assistance program

Abstract

We provide technology that enables appropriate driving assistance, taking into account the impact on the driver from the assistance system. [Solution] The driving support device detects a second vehicle that is present around the first vehicle being driven by the first driver, acquires emotional information indicating the emotions of the second driver who is driving the second vehicle, and outputs support information to assist the first driver's driving if the emotional information satisfies predetermined conditions.

Description

Technical Field

[0007] , , ,

[0001] The present invention relates to a driving support device, a driving support method, and a driving support program.

Background Art

[0002] There is known a technique for providing driving support to a driver of a vehicle by providing advice according to the driving situation of the driver.

Prior Art Documents

[0008] [Figure 1] Figure 1 is a diagram showing the configuration of the support system according to the embodiment. [Figure 2] Figure 2 shows the configuration of the driver assistance system. [Figure 3] Figure 3 shows the arrangement of cameras mounted on the vehicle. [Figure 4] Figure 4 shows an example of data recorded by a driver assistance system. [Figure 5] Figure 5 shows an example of a condition data table in which the conditions that determine the timing for when the driver assistance system outputs support information are registered. [Figure 6] Figure 6 shows an example of a support data table in which the emotional information of the first driver and the emotional information of the second driver, along with corresponding support information, are registered. [Figure 7] Figure 7 shows a specific example of support information registered in the support data table. [Figure 8] Figure 8 shows the flow of the driver assistance method that the driver assistance system controller executes based on the driver assistance program. [Modes for carrying out the invention]

[0009] The driver assistance devices, driver assistance methods, and driver assistance programs according to the embodiments of this disclosure described below are examples, and their configurations can be added, omitted, replaced, combined, and otherwise modified as appropriate without departing from the spirit of this disclosure.

[0010] Figure 1 shows the configuration of the support system 100 according to this embodiment. The support system 100 includes a driver assistance device 10 that provides support information to the driver of a vehicle. In Figure 1, for example, if vehicle 21 is the first vehicle, then vehicles 22 to 25 surrounding vehicle 21 are second vehicles. In this case, the driver of vehicle 21, which is the first vehicle, is the first driver. Also, the drivers of vehicles 22 to 25, which are the second vehicles, are second drivers.

[0011] The driver assistance device 10 is installed in vehicle 21, for example, and provides support information to the first driver based on the emotions of the second driver operating vehicles 22-24. The driver assistance device 10 may be installed in vehicles 22-25 in addition to vehicle 21. When the driver assistance device 10 installed in vehicle 22 provides support information to the driver of vehicle 22, vehicle 22 is considered the first vehicle, the driver of vehicle 22 is considered the first driver, the surrounding vehicles 21, 23-25 ​​are considered second vehicles, and the drivers of vehicles 21, 23-25 ​​are considered second drivers. In other words, the driver to whom the support information is provided is the first driver, the vehicle driven by the first driver is the first vehicle, and the vehicles surrounding it are second vehicles.

[0012] The driver assistance device 10 in this embodiment is a drive recorder mounted near the top of the vehicle's windshield, which photographs the area around the first vehicle and records the captured images. The driver assistance device 10 then identifies the vehicle captured in the images as the second vehicle, creates support information based on the emotions of the second driver operating the second vehicle, and provides this information to the first driver in various forms. Note that the driver assistance device 10 is not limited to a drive recorder, but may also be other electronic devices such as a control device that controls features such as cruise control, lane keeping assist, and collision mitigation braking, a navigation device, an audio device, or a head unit equipped with multiple of these functions.

[0013] FIG. 2 is a diagram showing the configuration of the driving support device 10. As illustrated in FIG. 2, the driving support device 10 is an information processing device (computer) having a controller (control unit) 101, a storage unit 102, an input / output interface (IF) 103, and a communication IF 104, which are interconnected by a connection bus 110.

[0014] The controller 101 controls the entire driving support device, and is configured by, for example, a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), and a main storage device (main memory). The controller 101 is not limited to a configuration including a single processor, and may have a multi-processor configuration. Further, the controller 101 may have a multi-core configuration with a single processor connected by a single socket. The main storage device is used as a work area of the controller 101, a storage area for programs and data, and a buffer area for communication data. The main storage device includes, for example, a Random Access Memory (RAM), or a combination of a RAM and a Read Only Memory (ROM). The main storage device is a storage medium in which programs and data are cached by the processor and used as a work area.

[0015] The storage unit 102 is an auxiliary storage device that stores programs executed by the controller 101, operation setting information, and the like. The storage unit 102 is not limited to an internal storage device built in the driving support device 10, and may be an external storage device such as an external storage device or a NAS (Network Attached Storage). The storage unit 102 is, for example, an HDD (Hard-disk Drive) or an S SD (Solid State Drive), EPROM (Erasable Programmable ROM), flash me mory, a USB memory, a memory card, or the like.

[0016] The input / output IF 103 is an interface for inputting and outputting data with peripheral devices. The input / output IF 103 performs data input and output with devices such as a disk drive that reads data from storage media such as CDs and DVDs, an operation unit that receives user input, a display device that displays information to the user, and a position detection unit 38. The input / output IF 103 may also perform data input and output with devices such as a tuner that receives radio and television broadcast waves, a reader / writer that reads and writes data to storage media such as memory cards, a microphone, a sensor, and a camera. For example, the input / output IF 103 is connected to cameras 31 and 32 and transmits control signals to cameras 31 and 32 for ON / OFF control, and also receives images captured by cameras 31 and 32. The operation unit is an input means that receives user input and inputs operation information indicating this input to the controller 101. The operation unit may be, for example, a button, a switch, a dial (rotary knob), a lever, etc. Alternatively, the operation unit may be a touch panel superimposed on the display surface of the display device. The position detection unit 38 uses a satellite positioning system such as GPS (Global Positioning System) to determine the vehicle's position. This is a device for acquiring information. The position detection unit 38 may also be a device that determines position information by autonomous navigation based on the vehicle's movement information obtained from an acceleration sensor or the like. The input / output IF 103 may be connected to a navigation device, and position information may be acquired from the navigation device.

[0017] The communication IF104 is an interface for communicating with other devices via a communication line. The communication IF104 communicates with, for example, an external data server or a driving support device mounted on another vehicle using a wireless communication method. Also, the communication IF104 may transmit and receive data, control signals, etc. to and from cameras and sensors via a network such as CAN. Furthermore, the communication IF104 has a function of making a call to an external telephone via a wireless telephone line. In the example of FIG. 2, the communication IF104 is connected to cameras 33 to 35 via a network, transmits a control signal for performing control such as ON / OFF to the cameras 33 to 35, and receives a captured image captured by the cameras 33 to 35. Also, the communication IF104 communicates with the vehicle's ECU39 via a network and receives information indicating the situation of the vehicle (hereinafter also referred to as vehicle information) detected by sensors connected to the ECU39. Here, the vehicle information is information indicating, for example, the speed (vehicle speed) of the vehicle, acceleration, jerk (jolt), change in steering angle, operation of a siren, operation of flashing a headlight (passing operation), etc.

[0018] In the driving support device 10, the controller 101 functions as a processing unit that executes predetermined information processing by executing an application program (such as a driving support program). However, at least a part of the processing of each of the above processing units may be provided by a DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), etc. . Also, at least a part of each of the above processing units may be a dedicated LSI (large scale integration) such as an FPGA (Field-Programmable Gate Array ), or other digital circuits. . Also, the driving support device 10 may be configured to include an analog circuit in at least a part of each of the above processing units.

[0019] The controller 101 detects a second vehicle that is present around the first vehicle being driven by the first driver. In this embodiment, the controller 101 processes the images captured by cameras 32-35 and detects the vehicle shown in the captured image as the second vehicle. For example, the controller 101 uses pattern matching to recognize an object image from the captured image that matches a predetermined pattern, such as a bicycle, motorcycle, passenger car, bus, or truck, and detects it as the second vehicle.

[0020] Figure 3 shows the arrangement of cameras 31-35 mounted on the vehicle. Camera 31 is Camera 31 is built into the rear of the steering assistance device 10, and its shooting lens (shooting direction) is positioned facing inward. In other words, camera 31 is positioned to photograph the driver inside the vehicle. Camera 32 is built into the front of the steering assistance device 10, and its shooting lens is positioned facing forward. In other words, camera 31 is positioned to photograph the second vehicle located in front of the first vehicle.

[0021] Furthermore, camera 33 is positioned at the rear of the vehicle, with its shooting lens facing the rear of the vehicle. That is, camera 33 is positioned to be able to photograph the second vehicle located behind the first vehicle. Note that camera 33 may be a dedicated camera provided for the driver assistance device (drive recorder) 10, or it may be a rear camera (backup camera) that photographs the area behind the vehicle when reversing and provides the first driver with the image of the area behind the vehicle. Camera 34 is positioned on the left side of the vehicle, with its shooting lens facing to the left of the vehicle. That is, camera 34 is positioned to be able to photograph the second vehicle located to the left of the first vehicle. Camera 35 is positioned on the right side of the vehicle, with its shooting lens facing to the right of the vehicle. That is, camera 35 is positioned to be able to photograph the second vehicle located to the right of the first vehicle. Note that cameras 34 and 35 may be dedicated cameras provided for the driver assistance device (drive recorder) 10, or they may be cameras that provide the first driver with an image of the area around the first vehicle, or cameras for electronic side mirrors. Note that the arrangement shown in Figure 3 is just one example, and the number and placement of cameras are not limited to this.

[0022] As the arrangement of cameras 31-35 is known, the controller 101 can determine which camera 32-35 captured the second vehicle and the direction in which the second vehicle is located relative to the first vehicle based on the vehicle's position in the images captured by each camera 32-35. The controller 101 may also recognize the lanes based on the captured images and determine whether the second vehicle is traveling in the same lane as the first vehicle or in an adjacent lane. For example, if the second vehicle is detected from an image captured by camera 33, the controller 101 can recognize it as a vehicle located behind the first vehicle. The controller 101 can also determine whether the vehicle located behind the first vehicle is moving in the same direction as the first vehicle without any other vehicles in between, and if it determines that this is true, the controller 101 can recognize the second vehicle as a vehicle following the first vehicle. In this case, the controller 101 may recognize a second vehicle traveling in the same lane as the first vehicle as a vehicle following the first vehicle, but may not recognize a second vehicle traveling in a different lane as a vehicle following the first vehicle.

[0023] Furthermore, the controller 101 detects a second driver from the area of ​​the captured image where the second vehicle is detected. For example, using image recognition technology, the controller 101 detects a person holding the steering wheel of the second vehicle, or a person sitting in the seat (driver's seat) where the steering wheel of the second vehicle is located, as the second driver from the area of ​​the captured image where the second vehicle is detected. The controller 101 also uses image recognition technology to detect a person holding the steering wheel of the first vehicle, or a person sitting in the seat (driver's seat) where the steering wheel of the first vehicle is located, as the first driver from the image captured by the camera 31.

[0024] Next, the controller 101 acquires emotional information indicating the emotions of the first and second drivers. Since a driver's emotions are expressed in their facial expressions and actions, images of these expressions and actions, along with the driver's emotions at that time, are used as training data. An AI model (emotion estimation model) is created by machine learning a pre-trained AI model using this training data. The emotional data in the training data can be acquired by attaching biosensors for emotion estimation (EEG sensors, heart rate sensors) to subjects for training data collection in a vehicle driving environment, and estimating emotions based on the biosignals detected by these sensors. In addition, image data of the driver (subject) in the training data can be acquired by taking images of the subject with a camera during the emotion estimation process. Then, by inputting the captured images of the first and second drivers into the AI ​​model thus trained, the first The emotions of the first and second drivers can be inferred.

[0025] Furthermore, since the driver's emotions are reflected in the vehicle's driving state, it is also possible to create an AI model (emotion estimation model) by using the vehicle's driving state and the driver's emotions at that time as training data, and then using this training data to machine-learn a pre-trained AI model. In this case, the vehicle's driving state can be estimated using the vehicle's control signals or the vehicle's driving trajectory information (vehicle images, etc.). Then, during emotion estimation, the driver's emotions are estimated by inputting the vehicle's control signals or the vehicle's driving trajectory information into the trained AI model. Examples of a driver's emotions include happy, joyful, comfortable, angry, irritated, excited, sad, depressed, anxious, frightened, uncomfortable, relaxed, and calm. The emotion estimation model in this embodiment may, for example, determine whether the state is negative, such as anger, sadness, depression, anxiety, fear, or discomfort. However, it is not limited to this, and the emotion estimation model may determine whether the state is neutral, such as relaxed or calm, or negative, or something else. Furthermore, the emotion estimation model may also determine whether the state is positive (such as happy, joyful, or comfortable), neutral, or negative. Regarding negative, positive, and neutral emotions, the designers and developers can create an emotion type data table indicating which emotion each emotion type corresponds to. The acquired or detected emotion can then be compared with this emotion type data table to determine whether it is negative, positive, or neutral. The controller 101 obtains the first driver's emotion information by inputting the first driver's image into this emotion estimation model. The controller 101 also obtains the second driver's emotion information by inputting the second driver's image into this emotion estimation model.

[0026] Furthermore, a driver's emotions are reflected in the movement of the vehicle they are driving. For example, when a driver is experiencing negative emotions, there is a tendency for the system to detect acceleration and jerk exceeding predetermined values ​​when sudden braking or acceleration occurs, changes in acceleration and steering angle when the vehicle swerves, and the activation of the horn or flashing headlights in situations where they are not actually necessary. For this reason, an emotion estimation model may output the driver's emotions using vehicle information as a parameter. For example, one method can be applied to create an AI model (emotion estimation model) by using the vehicle's driving state and the driver's emotions at that time as training data, and then machine learning a pre-trained AI model using this training data. In this case, the vehicle's driving state can be estimated using the vehicle's control signals or by using vehicle trajectory information (vehicle images, etc.). Then, during emotion estimation, the driver's emotions are estimated by inputting the vehicle's control signals or vehicle trajectory information into the trained AI model.

[0027] Furthermore, if the second vehicle is equipped with a driver assistance device 10, the controller 101 of the first vehicle may communicate with the driver assistance device 10 of the second vehicle and obtain emotional information of the second driver from the driver assistance device 10 of the second vehicle. That is, the controller 101 of the first vehicle may request emotional information from the controller 101 of the second vehicle, and the controller 101 of the second vehicle, upon receiving this request, may obtain emotional information from an image of its own vehicle (the second vehicle) driver and transmit it to the controller 101 of the first vehicle. By transmitting the emotional information obtained by the driver assistance device 10 of the second vehicle to the driver assistance device 10 of the first vehicle in this way, the emotional feelings of the second driver can be determined with greater accuracy than by determining the emotional feelings of the second driver based on an image captured by the camera of the first vehicle.

[0028] Figure 4 shows an example of data recorded by the driver assistance device 10, and this data is stored in the storage unit 102 as a driving information data table 61. As shown in Figure 4, the driver assistance device 10 records images 51 taken by cameras 31-35, location information 52, driving information 53 indicating driving conditions, emotional information of the first driver 54, emotional information of the second driver 55, and weather conditions during driving. The weather information 56 is stored in the memory unit 102 along with the time information 57. The location information indicates the position of the first vehicle. The driving status information includes, for example, the driving scene, the dangerous driving flag, and the operation of the headlights.

[0029] Driving scenes are information that indicates scenes during driving, such as turning left, turning right, overtaking, stopping, driving straight, starting, merging, and waiting at traffic lights. Dangerous driving flags are flags that indicate the first driver engaged in dangerous driving (abnormal driving conditions that deviate from normal driving conditions), such as sudden acceleration, sudden deceleration, sudden lane changes, speeding, swerving, and lane departure. Headlight operation is information that indicates, for example, high beam on / off, low beam on / off, and flashing of headlights.

[0030] In the driving information data table 61 in Figure 4, the symbol T1 indicates the timing when driving begins, and the symbol T2 indicates the timing when driving ends. Driving start T1 is, for example, the timing when the accessory power of the first vehicle is changed from off to on, and driving end T2 is the timing when the accessory power of the vehicle is changed from on to off. In addition to the vehicle's accessory power being changed from on to off, driving end T2 may also be the timing when the doors are locked or when the motion sensor detects that there are no people inside the vehicle (all people have disembarked). Furthermore, if a driving route is defined, driving start T1 may be the timing when the first vehicle reaches the departure position, and driving end T2 may be the timing when it reaches the arrival position (destination position).

[0031] Figure 5 shows an example of a condition data table 62 in which conditions are registered to determine the timing at which the driver assistance device 10 outputs support information. This condition data is created by the designers and developers of the driver assistance device 10 and stored in the memory unit 102. When the data shown in Figure 4 satisfies the predetermined conditions in the condition data table 62, the controller 101 of the driver assistance device 10 provides support information to the primary driver at a timing corresponding to the urgency of the information.

[0032] In the condition data table 62, condition 1 is the condition that the first vehicle has finished driving, that is, that the driving has ended T2, and the urgency of providing information under condition 1 is "not urgent". When the first vehicle has finished driving and condition 1 is met, the controller 101 generates support information based on the emotional information of the second driver detected from the start of driving T1 to the end of driving T2. In this case, the controller 101 uses the position of the first vehicle, the driving status, the dangerous driving flag, the position of the second vehicle relative to the first vehicle (relative position), the weather, the time, and the time of day as attribute values, or a combination thereof, and aggregates the number of times the second driver's emotions were negative for each of these attribute values ​​(hereinafter also referred to as the negative count). Then, it provides (outputs) support information for the driving scene corresponding to the attribute value where the negative count exceeds a predetermined number (support information threshold) to the first driver, etc. For example, if the negative count is aggregated and the negative count for stopping while turning left exceeds a predetermined value, support information such as "When stopping while turning left, the impression of those around you is negative. Please check the driving situation in the image." is output. In other words, support information is output for driving scenes (attribute values) that meet the output criteria.

[0033] The controller 101 may also provide support information when the frequency of negative occurrences (the number of occurrences in a predetermined period of length) exceeds a support information threshold. Furthermore, the controller 101 may output support information based on statistics of negative occurrences (the relationship between past performance and the number of negative occurrences during the operating period under consideration, for example, the peculiarity of negative occurrences during the operating period under consideration using variance, standard deviation, median, mean, truncated mean, etc.) (for example, when the frequency of negative occurrences is unusually high).

[0034] For example, if the number of negative occurrences is aggregated for each position (attribute value) of the second vehicle relative to the first vehicle, the controller 101 provides support information that suggests positional relationships where negative occurrences are frequent. The controller may output support information (indicating that the driver of the second vehicle in the positional relationship is likely to experience negative emotions). For example, if the driver of the following vehicle frequently experiences negative emotions when stopped, the controller 101 may output support information including the position of the second vehicle, such as, "It appears that the driver of the following vehicle often experiences negative emotions when stopping." The output of the support information is transmitted to a user terminal, such as the driver's smartphone or tablet. Alternatively, the output of the support information may be registered on a server via a network from the driver assistance device 10, and the user may access the server to view it.

[0035] In the condition data table 62, condition 2 is defined as the condition that dangerous driving has occurred and the emotions of either the first driver or the second driver are negative, and the urgency of providing information under condition 2 is "urgent". For example, in Figure 4, at timing Ta, the dangerous driving flag is recorded and the emotions of the second driver are negative, thus satisfying condition 2. In situations where condition 2 is satisfied, the controller 101 outputs support information at the time condition 2 is satisfied, without waiting for the end of driving T2. In other words, when dangerous driving occurs, support information is usually provided to the first driver immediately, considering the urgency. However, since the situation satisfying condition 2 does not adversely affect the first driver or the second driver of the surrounding vehicles, support information is not provided to the first driver immediately to avoid causing excessive tension or anxiety in the first driver's subsequent driving, but rather to prepare for the next drive after the end of driving. In this case, support information may be provided again at the end of driving.

[0036] In the condition data table 62, condition 3 is defined as the condition that no dangerous driving occurs and the emotions of both the first and second drivers are negative, and the urgency of providing information under condition 3 is "urgent". For example, in Figure 4, at timing Tb, the dangerous driving flag is not recorded and the emotions of both the first and second drivers are negative, thus satisfying condition 3. In situations where condition 3 is satisfied, the controller 101 outputs support information at the moment condition 3 is satisfied, without waiting for the end of driving T2. In other words, if no dangerous driving occurs, support information is usually provided to the first driver after the end of driving to prevent the first driver from feeling excessive tension or anxiety while driving afterward. However, since the state in which condition 3 is satisfied has a negative impact on the first driver and the second driver of the surrounding vehicle (the system does not judge it as dangerous driving, but the second driver may judge it as dangerous driving (for example, novice drivers often lack confidence in their driving and are therefore more likely to feel danger)), support information is provided to the first driver immediately, considering that the urgency is somewhat high. In this case, support information may be provided again at the end of the operation.

[0037] In the condition data table 62, condition 4 is that the first driver's emotions are not negative, and the frequency of negative emotional states of the second driver in the surrounding vehicles is equal to or greater than a predetermined frequency (a predetermined judgment threshold) (for example, in a process that detects the emotions of the drivers of following vehicles at predetermined time intervals (e.g., 1 minute), the number of times they were negative (negative frequency) in a predetermined period (e.g., the most recent 5 minutes) is equal to or greater than the judgment threshold (e.g., 3 times)). In this case, the urgency of providing information under condition 4 is "urgent." For example, in Figure 4, at timing Tc, the first driver's emotions are negative, and the second driver's negative frequency is above the judgment threshold. Therefore, the controller 101 outputs support information at timing Tc (end of Tc) when condition 4 is met, without waiting for the end of driving T2. In other words, if dangerous driving is not occurring, support information is normally provided to the first driver after the end of driving to prevent the first driver from feeling excessive tension or anxiety while driving afterward. However, since the state in which condition 4 is met means that the second driver of the surrounding vehicles is being continuously negatively affected, and considering that it is highly likely that the second driver will continue to be negatively affected in the future, support information is provided to the first driver immediately. In this case, support information is provided again at the end of driving. I may provide it.

[0038] Thus, conditions 2 to 4 are conditions for outputting support information without waiting for the end of operation T2 (hereinafter referred to as emergency conditions). Note that the conditions in the condition data table 62 in Figure 5 are just examples and are not limited to the example in Figure 5. The design and development team may determine appropriate conditions based on experiments, etc., and store them in the memory unit 102. When these emergency conditions are met, the support information may be displayed on the display device of the driving support device 10 or output as an audio message from the speaker.

[0039] Figure 6 shows an example of a support data table 63 in which emotional information of the first driver and the second driver, along with corresponding support information, are registered. This support data table 63 is set up for each attribute value of the vehicle state, and support information for the vehicle state corresponding to the attribute value is stored (set). If A1 to A3 are detected as the emotions of the first driver and B1 to B3 are detected as the emotions of the second driver, in the example in Figure 6, support information C1 to C9 indicating the support content is registered for the records defined in the columns A1 to A3 of the first driver's emotions and the rows B1 to B3 of the second driver's emotions. For example, when the emotion of the first driver is A1 and the emotion of the second driver is B1, the controller 101 refers to the support data table 63 for the attribute value corresponding to the vehicle state and obtains C1 as the support information corresponding to these emotions A1 and B1. Furthermore, when the first driver's emotion is A3 and the second driver's emotion is B2, the controller 101 refers to the support data table 63 and obtains C6 as support information corresponding to these emotions A3 and B2.

[0040] Figure 7 shows a specific example of support information registered in the support data table 63. Note that, for the sake of explanation, some of the correspondence between the emotions of the first driver and the second driver is omitted in Figure 7 compared to Figure 6. For example, when the vehicle status attribute value is "turning right" and the emotions of the first and second drivers are negative emotions A1 and B1, the controller 101 refers to the support data table 63 with the attribute value "turning right", reads the support information stored in the record corresponding to these emotions A1 and B1, such as "Take a deep breath and calm down before turning right", which helps to improve the negative emotions, and provides it to the first driver. Furthermore, for example, if the vehicle status attribute value is "short distance to following vehicle," the emotion of the following vehicle's driver is anger (e.g., B2), and the emotion of the first driver is fear (e.g., A2), the controller 101 refers to the support data table 63 with the attribute value "short distance to following vehicle" and provides support information that prompts external notification (request for assistance), such as "It seems I'm being tailgated, should I report it?", stored in the records corresponding to these emotions A2 and B2. If the first driver responds with something like "I'll report it," the controller 101 may use the communication IF 104 to make a phone call to the police or other relevant authorities via a wireless telephone line. Control data (programs) for implementing appropriate control corresponding to each record in the support data table 63 in Figure 6 are also stored, and the controller 101 can realize these functions by reading and executing this control data. Note that the support data table 63 in Figure 6 is just an example and is not limited to the configuration shown in Figure 6.

[0041] Figure 8 shows the flow of the driver assistance method executed by the controller 101 of the driver assistance device 10 based on the driver assistance program. When the accessory power of the first vehicle is turned on and power supply to the driver assistance device 10 is started or when an operation to start assistance is performed, the controller 101 starts the process shown in Figure 8.

[0042] In step S10, the controller 101 acquires images from each camera 31-35 and various vehicle information indicating driving conditions such as vehicle speed, stores them in the memory unit 102, and proceeds to step S20. Note that if vehicle information is not used for emotion estimation, the acquisition and storage of vehicle information is performed as follows: This can be omitted.

[0043] In step S20, the controller 101 inputs the captured images and vehicle information acquired in step S10 into the emotion estimation model to obtain the emotions of the first driver and the second driver, and then proceeds to step S30. If the emotions of the first driver are not used to obtain support information, the acquisition of the emotions of the first driver may be omitted. The captured images are pre-processed as necessary for input into the emotion estimation model, for example, by extracting images (face images) of the first and second drivers from the captured images. The controller 101 also stores the acquired data in the driving information data table 61.

[0044] In step S30, the controller 101 determines which conditions in the condition data table 62 are met based on the emotions obtained in step S20 and attribute values ​​such as vehicle information acquired in step S10, and determines whether the emergency condition is met based on the urgency information for the conditions that are met. In step S30, if the determination is positive, the controller 101 proceeds to step S40, and if the determination is negative, it proceeds to step S60.

[0045] In step S40, the controller 101 refers to the support data table 63 corresponding to the vehicle status attribute values ​​to obtain support information corresponding to the emotions of the first driver and the second driver, and then proceeds to step S50.

[0046] In step S50, the controller 101 displays the support information acquired in step S40 on the display device of the driver assistance device 10, and also outputs it as an audio message from the speaker, before proceeding to step S60.

[0047] In step S60, the controller 101 determines whether the operation has ended, that is, whether condition 1 has been met. If the determination in step S60 is negative, the controller 101 returns to step S10. If the determination in step S60 is positive, the controller 101 proceeds to step S70.

[0048] In step S70, the controller 101 creates support information based on the vehicle status, the emotions of the first driver, and the emotions of the second driver, etc., detected from the start of driving T1 to the end of driving T2, which are stored in the driving information data table 61 shown in Figure 4, and then proceeds to step S80. For example, the controller 101 generates support information by statistically processing the emotions detected from the start of driving T1 to the end of driving T2 for each attribute value.

[0049] In step S80, the controller 101 displays the support information created in step S70 on the display device, outputs it as an audio message from the speaker, and also outputs it to the user terminal or server. Based on this support information, the user terminal provides the support information to the user (first driver, etc.) (display, audio output), and the server uses this support information in various services.

[0050] <Effects of the Embodiment> (1) As described above, the driving support device 10 of this embodiment detects a second vehicle, acquires emotional information indicating the emotions of the second driver operating the second vehicle, and outputs support information to assist the first driver when the emotional information satisfies predetermined conditions. As a result, the driving support device 10 provides support information to the first driver at an appropriate time when the conditions are met, rather than providing it sequentially while the first driver is driving, thus preventing inconveniences such as making the first driver anxious and enabling appropriate support.

[0051] (2) The driving support device 10 of this embodiment has a controller 101 that controls the emotions of the first driver The content of the support information is determined based on the emotions of the second driver. The content of the support may differ depending on whether the first driver is stressed or calm. Therefore, the driving support device 10 of this embodiment can provide appropriate support information by determining the support information based on the emotions of both the first and second drivers.

[0052] (3) In this embodiment, the driver assistance device 10 has a controller 101 that refers to a data table in which the emotions of the first driver and the second driver are registered in association with the content of the assistance, and determines the content of the assistance that corresponds to the emotions of the first driver and the second driver, and then determines the assistance information. As a result, the driver assistance device 10 can provide appropriate assistance information according to the emotions of the first driver and the second driver.

[0053] (4) In this embodiment, the driver assistance device 10 outputs support information based on the emotions of the second driver detected from the start to the end of the first vehicle's operation, after the first vehicle's operation has finished. As a result, the driver assistance device 10 can provide information from the start to the end of the operation in one place, and can provide appropriate support information that is easy for the first driver to understand.

[0054] (5) When the controller 101 determines that the first driver has engaged in dangerous driving and detects that the second driver's emotions are negative, it outputs support information to the first driver based on the timing of the determination of dangerous driving. This allows the driving support device 10 to immediately provide support information in urgent situations.

[0055] (6) When the controller 101 detects that both the first driver's and the second driver's emotions are negative, it outputs support information to the first driver based on the timing of the acquisition of those emotions. As a result, the driving support device 10 can immediately provide support information when the first driver's emotions are in a negative state such as anxiety or fear, and the second driver's emotions are in a negative state such as anger or frustration, and the situation is urgent and prone to trouble.

[0056] (7) When the controller 101 detects the emotions of the second driver, if the frequency of the second driver's emotions being negative for each attribute value is above a threshold, it outputs support information. As a result, the driving support device 10 can organize the information by attribute value and provide appropriate support information that is easy for the first driver to understand. [Explanation of symbols]

[0057] 10: Driving assistance devices 21-24: Vehicles 31-35: Camera 38: Position detection unit 39: ECU 51: Photographed image 52: Location information 53: Driving Information 54: Emotion information 55: Emotion information 56: Driving Information 57: Time information 100: Support System 101: Controller 102: Storage section 110: Connecting bus 103: Input / Output Interface 104: Communication Interface

Claims

1. To detect a second vehicle that is present around the first vehicle being driven by the first driver, To obtain emotional information indicating the emotions of the second driver operating the second vehicle, When the aforementioned emotional information satisfies predetermined conditions, support information to assist the first driver's driving is output. A driver assistance system equipped with a controller that performs the following actions.

2. The aforementioned controller, The driving support device according to claim 1, which determines the content of the support information based on the emotions of the first driver and the emotions of the second driver.

3. The aforementioned controller, The driving support device according to claim 2, which refers to a data table in which the emotions of the first driver and the emotions of the second driver are registered in association with the support content, and determines the support information by finding the support content corresponding to the emotions of the first driver and the emotions of the second driver.

4. The aforementioned controller, The driving support device according to claim 1, which outputs the support information based on the emotions of the second driver detected from the start to the end of the operation after the operation of the first vehicle has finished.

5. The aforementioned controller, The driving support device according to claim 1 or 4, which determines whether the first driver has engaged in dangerous driving, and if it is determined that the first driver has engaged in dangerous driving, and the second driver's emotions are detected to be negative, outputs the support information to the first driver based on the timing of the determination of dangerous driving.

6. The aforementioned controller, The driving support device according to claim 1 or 4, which, when it is detected that both the emotions of the first driver and the emotions of the second driver are negative, outputs the support information to the first driver based on the timing of the acquisition of those emotions.

7. The aforementioned controller, The driving support device according to claim 1 or 4, which outputs the support information if the frequency of the second driver's emotions being negative is above a threshold.

8. To detect a second vehicle that is present around the first vehicle being driven by the first driver, To obtain emotional information indicating the emotions of the second driver operating the second vehicle, When the aforementioned emotional information satisfies predetermined conditions, support information to assist the first driver's driving is output. A driver assistance method performed by the controller.

9. To detect a second vehicle that is present around the first vehicle being driven by the first driver, To obtain emotional information indicating the emotions of the second driver operating the second vehicle, When the aforementioned emotional information satisfies predetermined conditions, support information to assist the first driver's driving is output. A driver assistance program that instructs the controller to execute a command.

Images