Warning device and warning method

Abstract

This reduces the possibility of misinterpretation of information notified to the driver and increases the reliability of notifications to the driver. [Solution] The system includes a risk indicator value acquisition unit (155) that acquires risk indicator values, a geographical characteristics acquisition unit (151) that acquires the geographical characteristics of the driver D of the vehicle (1), a driver state recognition unit (152) that recognizes the state of the driver D of the vehicle (1), a driver emotion estimation unit (153) that estimates the emotions of the driver D based on the recognition result of the state of the driver D, and a notification determination unit (156) that determines the level of notification to notify the driver D of the existence of risk and the type of HMI device (30) to be used for notification, based on the emotions of the driver estimated by the driver emotion estimation unit (153), the geographical characteristics of the driver D acquired by the geographical characteristics acquisition unit (151), and the risk indicator values ​​acquired by the risk indicator value acquisition unit (155).

Description

Technical Field

[0001] The present invention relates to an alert device and an alert method.

Background Art

[0002] In recent years, a device that outputs information using a language corresponding to a passenger in a vehicle has been known.

[0003] For example, Patent Document 1 discloses a system having a function of accepting a setting of a country or language by a user and a function of outputting information using the set country or language.

[0004] Also, the in-vehicle information display device disclosed in Patent Document 2 includes a face image capturing unit that captures a face image of a user, a nationality narrowing-down unit that narrows down the nationality of the user based on the captured face image, a calling language selection unit that selects a calling language for the user based on the narrowed-down nationality and makes a call, a used language estimation unit that estimates the language used by the user based on the response result of the user to the call, and a display control unit that displays a display screen that can be understood by the user using the estimated used language.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] When notifying drivers of the presence of a risk, there is a challenge in minimizing driver misperception by using multiple means of notification, such as sound and color, in addition to text display. However, differences in the emotions evoked by drivers due to their cultural characteristics and social background can lead to misperception. This application aims to solve the above problem by improving visibility. Ultimately, this will further improve traffic safety and contribute to the development of a sustainable transportation system. [Means for solving the problem]

[0007] One aspect of the present invention is a warning device that notifies a vehicle driver of the presence of a risk using a plurality of HMI devices, comprising: an index value acquisition unit that acquires a risk index value indicating the degree of possibility of contact between the vehicle and an object present around the vehicle based on the output of a plurality of sensors mounted on the vehicle; a geographical characteristics acquisition unit that acquires the geographical characteristics of the vehicle driver; a state recognition unit that recognizes the state of the vehicle driver; an emotion estimation unit that estimates the driver's emotions based on the state recognition result of the state recognition unit; and a determination unit that determines the level of notification to notify the driver of the presence of a risk and the type of HMI device to be used for notification, based on the emotions of the driver estimated by the emotion estimation unit, the geographical characteristics of the driver acquired by the geographical characteristics acquisition unit, and the risk index value acquired by the index value acquisition unit. Another aspect of the present invention includes a visual HMI device that transmits information to the driver visually, an auditory HMI device that transmits information to the driver audibly, and a tactile HMI device that provides tactile stimulation to the driver, wherein the determination unit determines, based on the driver's emotions, the driver's geographical characteristics, and the risk index value, that at least one of the visual HMI device, the auditory HMI device, and the tactile HMI device be the HMI device used for notification. In another aspect of the present invention, the determination unit determines the value of a visual parameter that modifies the image displayed by the visual HMI device from the level of notification determined based on the risk index value, wherein the visual parameter includes a parameter that modifies at least one of the enlargement and reduction of the image, the display area of ​​the image, the display brightness of the image, the display color of the image, the shape of the image, and the flashing period of the image. In another aspect of the present invention, the determination unit determines the value of an auditory parameter that modifies the sound output by the auditory HMI device from the level of notification determined based on the risk index value, wherein the auditory parameter includes a parameter that modifies at least one of the playback speed or playback cycle, volume, and pitch of the audio data that forms the basis of the sound. In another aspect of the present invention, the determination unit determines a value of a tactile parameter that modifies the vibration level of the vibration output by the tactile HMI device from the level of notification determined based on the risk index value, wherein the tactile parameter includes a parameter that modifies at least one of the period of the vibration, the amplitude of the vibration, and the waveform of the vibration. One aspect of the present invention is a warning method that causes a processor mounted on a warning device that notifies a vehicle driver of the presence of a risk using multiple types of HMI devices to execute an index value acquisition step that acquires a risk index value indicating the degree of possibility of contact between the vehicle and an object present around the vehicle based on the output of multiple sensors mounted on the vehicle; a geographical characteristics acquisition step that acquires the geographical characteristics of the vehicle driver; a state recognition step that recognizes the state of the vehicle driver; an emotion estimation step that estimates the driver's emotions based on the results of the recognition of the driver's state by the state recognition step; and a decision step that determines the level of notification to notify the driver of the presence of a risk and the type of HMI device to be used for notification, based on the emotions of the driver estimated by the emotion estimation step, the geographical characteristics of the driver acquired by the geographical characteristics acquisition step, and the risk index value acquired by the index value acquisition step. [Effects of the Invention]

[0008] According to one aspect of the present invention, when a driver is notified of the presence of a risk using multiple types of HMI devices, differences in the emotions evoked by the driver may occur depending on the driver's cultural characteristics and social background, potentially leading to misinterpretation of the risk. Therefore, by determining the level of notification to the driver regarding the presence of a risk and the type of HMI device used for notification based on the driver's geographical characteristics and emotions, the possibility of misinterpretation of the information notified to the driver can be reduced, and the reliability of notification to the driver can be increased. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows the configuration of the vehicle's control system. [Figure 2] Figure 2 shows the layout of the vehicle's interior. [Figure 3] Figure 3 shows an example of an emotion map. [Figure 4] Figure 4 shows an example of the first notification settings table. [Figure 5] Figure 5 shows an example of the second notification settings table. [Figure 6] Figure 6 shows an example of the third notification settings table. [Figure 7] Figure 7 is a flowchart showing the operation of the warning device. [Figure 8] Figure 8 is a flowchart showing the emotion estimation process. [Figure 9] Figure 9 is a flowchart showing the object detection process. [Modes for carrying out the invention]

[0010] [1. Vehicle Configuration] FIG. 1 is a diagram showing the configuration of the control system of the alert device 100, and FIG. 2 is a diagram showing the configuration in front of the passenger compartment of the vehicle 1 equipped with the alert device 100. The three axes of the X-axis, Y-axis, and Z-axis shown in FIG. 2 are perpendicular to each other. The Z-axis indicates the vertical direction. The X-axis and Y-axis are parallel to the horizontal direction in the driving state of the vehicle 1. The X-axis indicates the left-right direction, which is the vehicle width direction of the vehicle 1. The Y-axis indicates the front-rear direction of the vehicle 1. The positive direction of the X-axis indicates the right direction. The positive direction of the Y-axis indicates the front direction. The positive direction of the Z-axis indicates the upward direction.

[0011] The vehicle 1 is equipped with a vehicle sensor 10, a HMI (Human Machine Interface) device 30, a microphone 40, a driver monitor camera 50, a navigation device 60, and an alert device 100. In addition, the driver D of the vehicle 1 is wearing a wearable device 70, for example, on the driver D's arm or the like.

[0012] The vehicle sensor 10 includes at least one of one or more cameras, radars, lidars, and sonars distributed on the vehicle body of the vehicle 1, exists around the vehicle 1, and detects an object to be notified. The object is an object that may collide with or contact the vehicle 1, and is also an object that needs to notify the driver of the vehicle 1 of its presence. The vehicle sensor 10 outputs the data measured by the camera, radar, lidar, or sonar as sensor data to the alert device 100. FIG. 2 shows, as an example of the vehicle sensor 10, a front camera 10A mounted on the upper part of the front glass 3 of the vehicle 1 and imaging the front of the vehicle 1.

[0013] The HMI device 30 is a device that gives a stimulus to the driver D of the vehicle 1. The HMI device 30 of the present embodiment includes a touch panel 310 that transmits visual information to the vision of the driver D, a speaker 330 that transmits auditory information to the hearing of the driver D, and a seat vibration unit 350 that gives a tactile stimulus to the driver D. The touch panel 310 corresponds to a visual HMI device, the speaker 330 corresponds to an auditory HMI device, and the seat vibration unit 350 corresponds to a tactile HMI device.

[0014] In the configuration example of the front of the passenger compartment of the vehicle 1 shown in FIG. 2, the touch panel 310 is mounted on the dashboard 4 of the vehicle 1, and the speaker 330 includes a right speaker 330A provided on the driver's seat side of the vehicle 1 and a left speaker 330B provided on the passenger seat side. The touch panel 310 functions as a reception unit that displays visual information and receives input information input by the driver D.

[0015] The seat vibration unit 350 is provided on the seat on which the driver D is seated. The seat vibration unit 350 includes, for example, a vibration motor that generates vibrations inside the seat on which the driver is seated, and drives the vibration motor based on a vibration control signal input from the attention - calling device 100. Thereby, a tactile stimulus is given to the driver D. The HMI device 30 that gives a tactile stimulus to the driver D may be an electric seat belt that changes the tension of the seat belt to give a tactile stimulus to the driver.

[0016] Also, a microphone 40 and a driver monitor camera 50 are arranged on the dashboard 4 of the vehicle 1. The microphone 40 collects the voice of the driver D's speech. The microphone 40 converts the collected speech voice into digital voice data and outputs the converted voice data to the attention - calling device 100. The driver monitor camera 50 images the driver D seated on the driver's seat of the vehicle 1. The driver monitor camera 50 generates an imaging image including the face of the driver D and outputs the generated imaging image to the attention - calling device 100.

[0017] The wearable device 70 worn on the arm of the driver D measures biological information such as the heart rate, body temperature, blood pressure, etc., which are the biological information of the wearer, the driver D. The wearable device 70 is wirelessly connected to the attention - calling device 100 by short - range wireless communication and transmits the measured biological information to the attention - calling device 100.

[0018] The navigation device 60 searches for a guidance route, which is a route to a destination set by the passengers including the driver D of the vehicle 1, and executes route guidance to the destination by displaying the searched guidance route on the touch panel 310.

[0019] [2. Configuration of the warning device] Next, the configuration of the warning device 100 will be described. The warning device 100 is composed of an ECU or computer equipped with a communication unit 110, a storage unit 130, a processor 150, and input / output interfaces (not shown).

[0020] The communication unit 110 performs short-range wireless communication with the wearable device 70 using specifications such as Bluetooth® and UWB (Ultra Wide Band).

[0021] The storage unit 130 may include, for example, non-volatile RAM (Read Only Memory), or ROM and RAM (Random Access Memory). Alternatively, the storage unit 130 may also include an auxiliary storage device such as an SSD (Solid State Drive).

[0022] The memory unit 130 stores the control program 131 executed by the processor 150, the emotion map 133, the first notification setting table 135, the second notification setting table 137, and the third notification setting table 139. Details of the emotion map 133, the first notification setting table 135, the second notification setting table 137, and the third notification setting table 139 will be described later.

[0023] The processor 150 is an arithmetic processing unit equipped with a processor such as a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit). The processor 150 may be composed of a single processor or multiple processors. Furthermore, the processor 150 may be composed of a SoC integrated with part or all of the memory unit 130 or other circuits. The processor 150 may also be composed of a combination of a CPU that executes programs and a DSP (Digital Signal Processor) that performs predetermined arithmetic processing. Additionally, the processor 150 may be configured with all its functions implemented in hardware, or it may be configured using programmable devices.

[0024] The warning device 100 comprises, as functional units, a geographical characteristics acquisition unit 151, a driver state recognition unit 152, a driver emotion estimation unit 153, an object detection unit 154, a risk indicator value acquisition unit 155, a notification decision unit 156, and an execution control unit 157. These functional units are realized by the processor 150 executing control according to the control program 131.

[0025] The geographical characteristics acquisition unit 151 acquires the geographical characteristics of the driver D of vehicle 1. Geographical characteristics include the language spoken by driver D and the country or region where driver D resides. Alternatively, the system may acquire geographical characteristics such as the culture, traffic environment, and social background of each country or region. For example, if the communication unit 110 is connected to the internet, the system may acquire the culture, traffic environment, and social background of the country or region where driver D resides via the internet.

[0026] In this embodiment, the geographical characteristics acquisition unit 151 acquires information about the country where driver D resides or the language spoken by driver D as the geographical characteristics of driver D. For example, the geographical characteristics acquisition unit 151 acquires language setting information from the navigation device 60. Alternatively, the geographical characteristics acquisition unit 151 may analyze the voice of driver D input from the microphone 40 to identify the language and acquire language information. The geographical characteristics acquisition unit 151 outputs the information about the language used by driver D, which has been acquired as geographical characteristics information, to the notification determination unit 156.

[0027] The driver state recognition unit 152 corresponds to the state recognition unit. The driver state recognition unit 152 recognizes the state of driver D, who is riding in vehicle 1, based on at least one of the following: information input by touch operation on the touch panel 310, images of driver D captured by the driver monitor camera 50, voice of driver D input to the microphone 40, and biometric information of driver D detected by the wearable device 70.

[0028] The driver state recognition unit 152 recognizes, for example, the following elements as the state of driver D. Element 1: The response entered by touching the touch panel 310 in response to a question such as "How are you feeling today?" that is displayed on the touch panel 310 when driver D gets into vehicle 1. Second element: Driver D's facial expressions and behavior as recognized from the image of Driver D. Third element: The content, intonation, pitch, volume, and intonation of driver D's voice. Fourth element: Driver D's biometric information (heart rate, blood pressure, body temperature, etc.).

[0029] Figure 3 shows an example of an emotion map 133. The driver emotion estimation unit 153 corresponds to the emotion estimation unit. The driver emotion estimation unit 153 estimates the emotions of driver D based on the state of driver D recognized by the driver state recognition unit 152. The driver emotion estimation unit 153 estimates the emotions of driver D using the emotion map 133 shown in Figure 3, with emotional valence and arousal level as evaluation elements. Emotional valence is an evaluation element that indicates the level of driver D's discomfort to pleasure, and arousal level is an evaluation element that indicates the level of driver D's excitement to calmness. For example, as shown in Figure 3, the driver emotion estimation unit 153 estimates the emotions of driver D in seven stages: apathetic, sad 1, sad 2, sad 3, happy 1, happy 2, and happy 3. The driver emotion estimation unit 153 outputs the estimated emotion information of driver D to the notification decision unit 156.

[0030] The driver emotion estimation unit 153 estimates the emotions of driver D by, for example, setting the emotional valence and arousal levels based on the first to fourth elements as follows.

[0031] Regarding the first element described above, the driver emotion estimation unit 153 sets the driver D's emotional valence to a relatively high value (e.g., 3) if the response to the question is "as usual," and sets the arousal level to a relatively low value (e.g., -3) if the response to the question is "different from usual."

[0032] Regarding the second element described above, the driver emotion estimation unit 153 determines, based on the driver D's facial expression, that if the driver D has a smiling expression, it sets the emotional valence to a relatively high value (e.g., 5), and if the driver D has a sullen expression, it sets the emotional valence to a relatively low value (e.g., -5). Furthermore, based on the driver D's behavior, the driver emotion estimation unit 153 determines, based on the driver D's behavior, that the arousal level is set to a relatively high value (e.g., 3) if the driver D is moving restlessly, and if the driver D is moving very little, it sets the arousal level to a relatively low value (e.g., -3).

[0033] Regarding the third element described above, the driver emotion estimation unit 153, based on the content of driver D's utterance, sets the emotional valence to a relatively high value (e.g., 5) if it determines that the utterance is positive, such as praising or expressing expectations, and sets the emotional valence to a relatively low value (e.g., -5) if it determines that the utterance is negative, such as criticizing something. In addition, if the driver emotion estimation unit 153 includes specific keywords in driver D's utterance (e.g., "like," "super like"), it sets the emotional valence and arousal level values ​​associated with those keywords.

[0034] Furthermore, the driver emotion estimation unit 153, based on the pitch (frequency) of driver D's voice, sets the level of alertness to a relatively high value (e.g., 3) if driver D's voice is above a predetermined pitch, and sets the level of alertness to a relatively low value (e.g., -3) if driver D's voice is below the predetermined pitch.

[0035] Regarding the fourth element described above, the driver emotion estimation unit 153 applies the detected values ​​of driver D's biometric information (heart rate, blood pressure, body temperature, etc.) to a pre-prepared correspondence table of detected values, emotional valence, and arousal level to set the emotional valence and arousal level. The correspondence table may be created based on driver D's profile entered by driver D or on driver D's biometric information detected in the past.

[0036] Furthermore, the driver emotion estimation unit 153 may recognize values ​​for emotional valence and arousal based on changes in the vehicle's speed detected by the speed sensor, the vehicle's behavior detected by the gyro sensor, and fluctuations in the vehicle's position detected by the GNSS (Global Navigation Satellite Systems) sensor. The speed sensor, gyro sensor, and GNSS sensor are not shown in the diagram. The driver emotion estimation unit 153 may also estimate the driver D's emotions based on either emotional valence or arousal, or on other evaluation factors.

[0037] The object detection unit 154 detects objects present around the vehicle 1 based on sensor data input from the vehicle sensor 10. In this embodiment, the object detection unit 154 detects objects that are located in front of the vehicle 1 in the direction of travel and that have the potential to come into contact with or collide with the vehicle 1. That is, when the vehicle 1 is traveling forward, it detects objects located in front of the vehicle 1, and when the vehicle 1 is traveling backward, it detects objects located behind the vehicle 1. The object detection unit 154 detects the position, speed, and direction of movement of the detected objects. The object detection unit 154 outputs information indicating the position, speed, and direction of movement of the detected objects to the risk index value acquisition unit 155. The object detection unit 154 may detect objects and their relative positions using sensors such as the front camera provided by the vehicle sensor 10, as well as vehicle-to-vehicle communication devices, GNSS units, V2X (Vehicle-to-Everything) communication, etc. Furthermore, the object detection unit 154 may detect objects in the captured image and their relative positions using methods such as judgment in a virtual environment via a server. Communication partners for V2X communication include other vehicles, pedestrians, networks, infrastructure, etc.

[0038] The risk index value acquisition unit 155 acquires the risk index value of the object detected by the object detection unit 154. The risk index value is a value indicating the possibility of contact or collision between the vehicle 1 and the object. The risk index value acquisition unit 155 corresponds to the index value acquisition unit.

[0039] The risk index acquisition unit 155 receives information such as the position, speed, and direction of movement of the object detected by the object detection unit 154. The risk index acquisition unit 155 also receives sensor data such as images captured by the camera. Based on the position, direction, and speed of movement of the object, the position, direction, and speed of movement of vehicle 1, and the lighting status of the traffic lights detected from the camera's captured images, the risk index acquisition unit 155 predicts the movement path between the object and vehicle 1. The risk index acquisition unit 155 then calculates a risk index value based on the position where vehicle 1 and the object are closest, and the distance between vehicle 1 and the object at that position. The risk index value is calculated, for example, by assigning points to the position, direction, speed of movement, and distance between vehicle 1 and the object, and then summing the values ​​obtained by multiplying these points by a weighting coefficient.

[0040] When predicting the movement of an object, the risk indicator value acquisition unit 155 may consider the illumination status of the vehicle's turn signals and brake lights if the object is a vehicle, and may consider the direction of the pedestrian's face if the object is a pedestrian. Similarly, when predicting the movement of vehicle 1, the risk indicator value acquisition unit 155 may consider the illumination status of vehicle 1's turn signals and brake lights obtained from the vehicle sensor 10, and may consider the route being guided if the navigation device 60 is providing route guidance.

[0041] Furthermore, the risk indicator value acquisition unit 155 may detect the risk of contact between the vehicle 1 and objects actually detected in the surrounding environment of the vehicle 1, as well as estimate and detect the risk of contact with objects that have not yet been detected but may appear (such as virtual vehicles or pedestrians) based on factors such as the visibility at intersections and the number of accidents. The risk indicator value acquisition unit 155 outputs the acquired risk indicator values ​​to the notification decision unit 156.

[0042] The notification decision unit 156 corresponds to the decision unit. The notification decision unit 156 receives information on the language spoken by driver D, acquired by the geographical characteristics acquisition unit 151, information on driver D's emotions estimated by the driver emotion estimation unit 153, and risk index values ​​for each object acquired by the risk index value acquisition unit 155. The notification determination unit 156 determines whether or not there is an object to be notified based on the input information. If there is an object to be notified, the notification determination unit 156 also determines the type of notification to notify of the existence of the object and the level of notification.

[0043] Figure 4 shows an example of the first notification setting table 135. The first notification setting table 135 is a table that registers the correspondence between risk indicator values ​​and notification levels. Hereafter, the risk indicator value will be referred to as the risk indicator value R. First, the notification decision unit 156 determines whether or not there are any objects with a risk index value R of 2 or greater, which is input from the risk index value acquisition unit 155. The notification decision unit 156 determines that if there are any objects with a risk index value R of 2 or greater, these objects are objects that should be notified. A risk index value R of 2 or greater corresponds to a threshold value.

[0044] Furthermore, if there are objects with a risk index value R of 2 or higher, the notification determination unit 156 determines the notification level based on the risk index value of these objects and the first notification setting table 135. In the example of the first notification setting table 135 shown in Figure 4, the notification level is set to "low" when the risk index value R is 2 ≤ R < 4 and when 4 ≤ R < 6. The notification level is set to "medium" when the risk index value R is 6 ≤ R < 8. The notification level is set to "high" when the risk index value R is 8 ≤ R < 10 and when R = 10.

[0045] Figure 5 shows an example of the second notification setting table 137. The second notification setting table 137, shown in Figure 5, registers information relating to geographical characteristics such as language, the emotions of driver D, and the type of notification. Driver D's emotions include "neutral," "happy 1," "happy 2," "happy 3," "sad 1," "sad 2," and "sad 3." The first notification setting table 135 registers the type of notification and the level of notification for each of these emotions.

[0046] Next, the notification determination unit 156 determines the type of HMI device 30 to be used to notify the driver D of the risks present around the vehicle 1, based on the input language information of the driver D and the emotional information of the driver D. The notification determination unit 156, based on the language information of driver D input from the geographical characteristics acquisition unit 151 and the emotional information of driver D input from the driver emotional estimation unit 153, refers to the second notification setting table 137 and obtains the type of notification indicating the type of HMI device 30 to be used for risk notification. In this embodiment, if the type of notification is visual, image display on the touch panel 310 is selected. If the type of notification is auditory, in this embodiment, a voice notification output from speaker 330 is selected. If the type of notification is tactile, in this embodiment, vibration of the sheet by the sheet vibration unit 350 is selected.

[0047] Figure 6 shows an example of the third notification setting table 139. The third notification setting table 139 shown in Figure 6 includes parameters and settings for each notification level for each type of notification: visual, auditory, and tactile. The parameters set for each type of notification are as follows: Visual parameters are variables that change the display manner of images and figures displayed on the touch panel 310. Auditory parameters are variables that change the output manner of sound output from the speaker 330. Tactile parameters are variables that change the output manner of vibrations output from the seat vibration unit 350.

[0048] Visual parameters include display size, resizing range, display color, graphic shape, and blinking cycle. These parameters correspond to visual parameters. The third notification settings table 139 registers the settings for each of these parameters for each notification level. The notification levels include high notification levels, medium notification levels, and low notification levels. Hereafter, high notification levels will be referred to as high level, medium notification levels as medium level, and low notification levels as low level.

[0049] If the parameter is display size, the high-level setting will register a display size of "large," the medium-level setting will register a display size of "medium," and the low-level setting will register a display size of "small." When the parameter is the width of the change size, "Large" is registered for high-level settings, "Medium" for medium-level settings, and "Small" for low-level settings. The width of the change size is the range by which the display size of the shapes displayed on the touch panel 310 is enlarged and reduced. For example, if the width of the change size is "Large," the size of the shapes displayed on the touch panel 310 will be significantly changed from "Large" to "Small," and from "Small" to "Large." By significantly changing the display size of the shapes displayed on the touch panel 310 in this way, the driver's attention is attracted. If the parameter is luminance, the high-level setting will register "bright," the medium-level setting will register "medium," and the low-level setting will register "dark." If the parameter is a display color, then "red" will be registered as the display color for high-level settings, "yellow" for medium-level settings, and "blue" for low-level settings. When the parameter is a geometric shape, "sharp" is registered as the geometric shape for high-level settings, "angular" is registered as the geometric shape for medium-level settings, and "rounded" is registered as the geometric shape for low-level settings. The sharper the geometric shape displayed by the visual HMI device 310, the higher the perceived risk, and the rounder the geometric shape, the lower the perceived risk. If the parameter is the blinking period, "fast" will be registered as the blinking period for high-level settings, "medium" for medium-level settings, and "slow" for low-level settings.

[0050] Hearing parameters include electronic sounds or spoken voices, playback speed, volume, and pitch. These parameters correspond to auditory parameters. The third notification settings table 139 registers the settings for each of these parameters for each notification level.

[0051] If the parameter is either an electronic sound or a spoken voice, "Spoken voice" will be registered for high-level settings, "Spoken voice" for medium-level settings, and "Electronic sound" for low-level settings. If the parameter is playback speed, "fast" will be registered as the playback speed for high-level settings, "medium" for medium-level settings, and "slow" for low-level settings. If the parameter is volume, "High" will be registered as the volume for high-level settings, "Medium" for medium-level settings, and "Low" for low-level settings. If the parameter is voice pitch (tone), a high level setting will register "high" as the voice pitch, and a medium level setting will register "medium" as the voice pitch. In the case of a low level setting, an electronic sound is output, so no voice pitch is registered.

[0052] The parameters of tactile sensation include frequency and period, amplitude, and waveform. These parameters correspond to tactile parameters. The second notification settings table 137 registers the settings for each of these parameters for each notification level.

[0053] When the parameters are frequency and period, "fast" is registered for high-level settings, "medium" for medium-level settings, and "slow" for low-level settings. If the parameter is amplitude, "Large" is registered for high-level settings, "Medium" for medium-level settings, and "Small" for low-level settings. If the parameter is a waveform, "Rectangle" is registered for high-level settings, "Sawtooth" is registered for medium-level settings, and "Steady" is registered for low-level settings.

[0054] The notification determination unit 156 obtains the notification type determined in the second notification setting table 137 and the parameter settings corresponding to the notification type level determined in the first notification setting table 135 from the third notification setting table 139 shown in Figure 6. The notification determination unit 156 outputs the obtained notification type, notification level, and parameter settings to the execution control unit 157.

[0055] The execution control unit 157 controls the operation of the HMI device 30 according to the type of notification, the level of notification, and the parameter settings input from the notification decision unit 156. If the type of notification input from the notification decision unit 156 is visual, the execution control unit 157 displays images, shapes, or text on the touch panel 310 according to the parameter settings. If the type of notification input from the notification decision unit 156 is auditory, the execution control unit 157 outputs spoken voice or electronic sound from the speaker 330 according to the parameter settings. If the type of notification input from the notification decision unit 156 is tactile, the execution control unit 157 controls the seat vibration unit 350 according to the parameter settings to vibrate the seat on which the driver D is seated.

[0056] The settings in the second notification settings table 137 shown in Figure 5 and the third notification settings table 139 shown in Figure 6 are settings that reflect geographical characteristics. If driver D's geographical characteristics, such as cultural characteristics and social background, differ, there may be differences in the emotions driver D recalls, potentially leading to misrecognition in notifications using the HMI device 30. Therefore, the second notification setting table 137 and the third notification setting table 139 register settings that reflect the cultural characteristics, traffic environment, and social background of each country or region. For example, if the driver emotion estimation unit 153 estimates that driver D's emotion is "sadness," the auditory parameters are set to produce a "calm" voice output that soothes driver D's "sadness." Also, if the driver emotion estimation unit 153 estimates that driver D's emotion is "drowsiness, fatigue, or boredom," the auditory parameters are set to excite driver D's emotions and create feelings of happiness. The following describes examples of settings for the second notification setting table 137 and the third notification setting table 139.

[0057] [3. Example of a notification] (Example of notification 1) For example, the display color can be changed depending on the country of driver D. In China, the color white is used to represent both "relief" and "sadness." In Nigeria, the color red is used to represent "anger," "love," and "fear." In Egypt, the color yellow is used to represent positive emotions such as "joy." In Greece, the color purple is used to represent sadness.

[0058] (Example of notification 2) For example, when using a touch panel 310, which is a visual communication device, to notify users of risk information using a human face or a face icon, the Good Mark is permitted for use in Japan, but it is set to "do not use" in Brazil. Similarly, a nodding action is permitted in Japan, but set to "do not use" in India and Brazil.

[0059] (Notification Example 3) For example, in Asian and Nordic countries where physical contact is less common, tactile notifications are set to "disabled" or the notification level is set to "weak." Furthermore, in Latin American countries where physical contact is common, such as Italy, France, Spain, Portugal, and South American countries, tactile notifications are set to "use" or the level of tactile notifications is set to "high".

[0060] (Notification example 4) For example, in Asian countries where horns are frequently used, auditory notification is set to "use," or the level of auditory notification is set to "high."

[0061] [4. Operation of the warning device] Figures 7 to 9 are flowcharts showing the operation of the warning device 100. The operation of the warning device 100 will be explained with reference to the flowcharts shown in Figures 7 to 9. First, when the vehicle's ignition switch is turned on (step S1), the warning device 100 acquires geographical characteristics (step S2). Geographical characteristics may be acquired, for example, from the language settings of the navigation system, or they may be identified by analyzing the driver's speech or the sound of the radio or music.

[0062] Next, the warning device 100 performs emotion estimation processing for driver D (step S3). After estimating driver D's emotions through emotion estimation processing, the warning device 100 detects an object, which is an obstacle, around the vehicle 1 through object detection processing (step S4).

[0063] Next, the alerting device 100 determines whether or not it has detected an object to be notified through object detection processing (step S5). If the alerting device 100 has not detected an object to be notified (step S5 / NO), it returns to the process in step S3.

[0064] Furthermore, when the alert device 100 detects an object to be notified (step S5 / YES), it refers to the first notification setting table 135, the second notification setting table 137, and the third notification setting table 139 based on the geographical characteristics obtained in step S2, the emotions of driver D identified in step S3, and the risk indicator values ​​obtained in step S5, to determine the type of notification to notify driver D of the presence of risk and the level of notification (step S6).

[0065] Next, the alert device 100 operates the HMI device 30 corresponding to the type of notification determined in step S6 with an output corresponding to the notification level, and performs the notification operation (step S7).

[0066] Next, the alert device 100 determines whether the IG switch has been turned off or not (step S8). If the IG switch is not turned off (step S8 / NO), the alert device 100 returns to the emotion estimation process in step S3. Also, if the IG switch is turned off (step S8 / YES), the alert device 100 terminates this processing flow.

[0067] Figure 8 is a flowchart detailing the emotion estimation process in step S3. The details of the emotion estimation process will be explained with reference to the flowchart shown in Figure 8. First, the warning device 100 acquires the voice of driver D input from microphone 40 (step S31). Next, the warning device 100 outputs a voice message from speaker 330 asking driver D about their condition, and acquires the response to this message via microphone 40 or touch operation (step S32). Furthermore, the warning device 100 acquires the image captured by driver monitor camera 50 (step S33) and acquires driver D's biometric information from wearable device 70 (step S34).

[0068] Next, the alert device 100 recognizes the state of driver D based on the information acquired in steps S31 to S34 (step S35). Then, the alert device 100 refers to the emotion map 133 based on the first to fourth elements described above and estimates the emotion of driver D (step S36).

[0069] Figure 9 is a flowchart showing the details of the object detection process. The details of the object detection process will be explained with reference to the flowchart shown in Figure 9. First, the warning device 100 acquires sensor data from the vehicle sensor 10 (step S41).

[0070] Next, the warning device 100 detects objects present around the vehicle 1 based on sensor data (step S42), and detects the position, speed, and direction of movement of the detected objects.

[0071] Next, the warning device 100 acquires the risk index value of the detected object (step S3). For example, the warning device 100 scores the object's position, direction of travel, speed, and distance between vehicle 1 and the object, multiplies each of these scored values ​​by a weighting coefficient, and calculates the sum of these values ​​as the risk index value.

[0072] Next, the alert device 100 determines whether the acquired risk indicator value is equal to or greater than a preset threshold (step S44). If the alert device 100 determines that the risk indicator value is not equal to or greater than the threshold (step S44 / NO), it returns to the process in step S41.

[0073] Next, the alerting device 100 determines that it has detected an object to be notified if the risk indicator value is above the threshold (step S44 / YES) (step S45).

[0074] [5. Other Embodiments] The embodiments described above are merely examples and can be modified and applied as needed.

[0075] In the embodiment described above, the case in which the visual HMI device 310 is a touch panel 310 was explained, but the visual HMI device 310 may also be a display means such as an LED (Light Emitting Diode), or a head-up display provided on the instrument panel 4 of the vehicle 1. The head-up display notifies the presence of an object that may come into contact with the vehicle 1 by displaying an image on the windshield 3, which serves as a projection surface.

[0076] Furthermore, the configuration of each part of Vehicle 1 shown in Figure 2 is merely an example, and the specific implementation is not particularly limited. In other words, it is not necessarily required that hardware corresponding to each part be implemented individually; it is certainly possible to configure a system where a single processor executes a program to realize the functions of each part. Also, in the embodiments described above, some of the functions realized by software may be implemented as hardware, or conversely, some of the functions realized by hardware may be implemented as software.

[0077] Furthermore, the operation steps shown in Figures 7 to 9 are divided according to the main processing content, and the present invention is not limited by the way the processing units are divided or their names. Depending on the processing content, it may be further divided into more steps. Alternatively, it may be divided so that one step unit includes even more processing. Also, the order of the steps may be changed as appropriate, as long as it does not impede the spirit of the present invention.

[0078] Furthermore, when the warning method by the warning device 100 described above is implemented using the processor 150, the program to be executed by the processor 150 can be configured as a recording medium or a transmission medium for transmitting this program. In other words, the control program 131 can also be implemented by recording it on a portable information recording medium. Examples of information recording media include magnetic recording media such as hard disks, optical recording media such as CDs, and semiconductor storage devices such as USB (Universal Serial Bus) memory and SSDs (Solid State Drives), but other recording media can also be used.

[0079] [6. Configurations supported by the above embodiments] The above embodiment supports the following configuration:

[0080] (Composition 1) An alert device that notifies a vehicle driver of the presence of a risk using multiple types of HMI devices, comprising: an index value acquisition unit that acquires a risk index value indicating the degree of possibility of contact between the vehicle and an object present around the vehicle based on the output of multiple sensors mounted on the vehicle; a geographical characteristics acquisition unit that acquires the geographical characteristics of the vehicle driver; a state recognition unit that recognizes the state of the vehicle driver; an emotion estimation unit that estimates the driver's emotions based on the state recognition result of the state recognition unit; and a determination unit that determines the level of notification to notify the driver of the presence of a risk and the type of HMI device to be used for notification, based on the emotions of the driver estimated by the emotion estimation unit, the geographical characteristics of the driver acquired by the geographical characteristics acquisition unit, and the risk index value acquired by the index value acquisition unit.

[0081] According to this configuration, the level of notification to inform the driver of the presence of a risk, and the type of HMI device used for notification, are determined based on the driver's geographical characteristics, the driver's emotions, and a risk index value indicating the degree of likelihood of contact with a vehicle occurring. When drivers are notified of the presence of risks using multiple types of HMI devices, differences in the emotions evoked by drivers due to their cultural characteristics and social backgrounds may lead to misinterpretations of the risk. Therefore, by determining the level of notification and the type of HMI device used for notification based on the driver's geographical characteristics and emotions, it is possible to reduce the possibility of misinterpretations and increase the reliability of notifications to drivers.

[0082] (Configuration 2) The alert device according to Configuration 1, wherein the plurality of HMI devices include a visual HMI device that transmits information to the driver visually, an auditory HMI device that transmits information to the driver audibly, and a tactile HMI device that provides tactile stimulation to the driver, and the determination unit determines, based on the driver's emotions, the driver's geographical characteristics, and the risk index value, that at least one of the visual HMI device, the auditory HMI device, and the tactile HMI device be the HMI device used for notification.

[0083] In this configuration, multiple types of HMI devices include visual HMI devices, auditory HMI devices, and tactile HMI devices. The decision unit determines that at least one of the visual HMI device, auditory HMI device, and tactile HMI device will be used to notify the driver of the presence of risk. Therefore, the presence of risk can be notified to the driver through visual information transmission, auditory information transmission, and tactile stimulation.

[0084] (Composition 3) The warning device according to configuration 2, wherein the determination unit determines the value of a visual parameter that changes the image displayed by the visual HMI device from the level of notification determined based on the risk index value, and the visual parameter includes a parameter that changes at least one of the following: the enlargement and reduction of the image, the display area of ​​the image, the display brightness of the image, the display color of the image, the shape of the image, and the flashing period of the image.

[0085] In this configuration, the values ​​of the visual parameters are determined by the notification level, which is determined based on the risk indicator value. Therefore, at least one of the following can be changed to correspond to the notification level: the enlargement and reduction of the image displayed by the visual HMI device, the display area of ​​the image, the display brightness of the image, the color of the image, the shape of the image, and the flashing cycle of the image.

[0086] (Composition 4) The warning device according to configuration 2, wherein the determination unit determines the value of an auditory parameter that modifies the sound output by the auditory HMI device from the level of notification determined based on the risk index value, and the auditory parameter includes a parameter that modifies at least one of the playback speed or playback cycle, volume, and pitch of the audio data that is the source of the sound.

[0087] In this configuration, the auditory parameter values ​​are determined by the notification level, which is determined based on the risk indicator value. Therefore, at least one of the following—the playback speed or playback cycle, volume, and pitch—of the audio data that forms the basis of the sound output by the auditory HMI device can be changed to correspond to the notification level.

[0088] (Composition 5) The warning device according to configuration 2, wherein the determination unit determines the value of a tactile parameter that changes the vibration level of the vibration output by the tactile HMI device from the level of notification determined based on the risk index value, and the tactile parameter includes a parameter that changes at least one of the period of the vibration, the amplitude of the vibration, and the waveform of the vibration.

[0089] In this configuration, the value of the tactile parameter is determined by the notification level, which is determined based on the risk indicator value. Therefore, at least one of the vibration period, amplitude, and waveform of the vibration output by the tactile HMI device can be changed to correspond to the notification level.

[0090] (Composition 6) An alerting method that causes a processor mounted on an alerting device that notifies a vehicle driver of the presence of a risk using multiple types of HMI devices to execute an index value acquisition step that acquires a risk index value indicating the degree of possibility of contact between the vehicle and an object present around the vehicle based on the output of multiple sensors mounted on the vehicle; a geographical characteristics acquisition step that acquires the geographical characteristics of the vehicle driver; a state recognition step that recognizes the state of the vehicle driver; an emotion estimation step that estimates the driver's emotions based on the results of the recognition of the driver's state by the state recognition step; and a decision step that determines the level of notification to notify the driver of the presence of a risk and the type of HMI device to be used for notification, based on the driver's emotions estimated by the emotion estimation step and the geographical characteristics of the driver acquired by the geographical characteristics acquisition step.

[0091] According to this configuration, the level of notification to inform the driver of the presence of a risk, and the type of HMI device used for notification, are determined based on the driver's geographical characteristics, the driver's emotions, and a risk index value indicating the degree of likelihood of contact with a vehicle occurring. When drivers are notified of the presence of risks using multiple types of HMI devices, differences in the emotions evoked by drivers due to their cultural characteristics and social backgrounds may lead to misinterpretations of the risk. Therefore, by determining the level of notification and the type of HMI device used for notification based on the driver's geographical characteristics and emotions, it is possible to reduce the possibility of misinterpretations and increase the reliability of notifications to drivers. [Explanation of Symbols]

[0092] 1...Vehicle, 3...Windshield, 4...Dashboard, 10...Vehicle sensor, 10A...Front camera, 30...HMI device, 40...Microphone, 50...Driver monitor camera, 60...Navigation device, 70...Wearable device, 100...Warning device, 110...Communication unit, 130...Memory unit, 131...Control program, 133...Emotion map, 135...First notification setting table, 137...Second notification setting table, 139...Third notification setting table, 150...Processor, 151...Geographic characteristics acquisition unit, 152...Driver state recognition unit, 153...Driver emotion estimation unit, 154...Object detection unit, 155...Risk index value acquisition unit, 156...Notification decision unit, 157...Execution control unit, 310...Touch panel, 330...Speaker, 330A...Right speaker, 330B...Left speaker, 350...Seat vibration unit.

Claims

1. A warning device that notifies the vehicle driver of the presence of a risk using multiple types of HMI devices, An index value acquisition unit that acquires a risk index value indicating the degree of possibility of contact occurring between the vehicle and an object present around the vehicle, based on the output of multiple sensors mounted on the vehicle. A geographic characteristics acquisition unit that acquires the geographic characteristics of the driver of the vehicle, A state recognition unit that recognizes the state of the driver of the vehicle, Based on the state recognition result of the state recognition unit, an emotion estimation unit estimates the driver's emotions, A determination unit determines the level of notification to inform the driver of the presence of risk and the type of HMI device to be used for notification, based on the driver's emotions estimated by the emotion estimation unit, the driver's geographical characteristics acquired by the geographical characteristics acquisition unit, and the risk indicator values ​​acquired by the indicator value acquisition unit. A warning device equipped with [a specific feature / feature].

2. The aforementioned multiple types of HMI devices are, The system includes a visual HMI device that transmits information to the driver visually, an auditory HMI device that transmits information to the driver audibly, and a tactile HMI device that provides tactile stimulation to the driver. The attention-raising device according to claim 1, wherein the determination unit determines, based on the driver's emotions, the driver's geographical characteristics, and the risk index value, that at least one of the visual HMI device, the auditory HMI device, and the tactile HMI device be the HMI device to be used for notification.

3. The determination unit determines the value of the visual parameter that changes the image displayed by the visual HMI device from the level of notification determined based on the risk indicator value. The warning device according to claim 2, wherein the visual parameters include parameters for changing at least one of the following: the enlargement and reduction of the image, the display area of ​​the image, the display brightness of the image, the display color of the image, the shape of the image, and the flashing period of the image.

4. The determination unit determines the value of the auditory parameter that changes the sound output by the auditory HMI device from the level of notification determined based on the risk indicator value. The attention-raising device according to claim 2, wherein the auditory parameters include a parameter that changes at least one of the playback speed or playback cycle, volume, and pitch of the audio data that is the source of the sound.

5. The determination unit determines the value of the tactile parameter that changes the vibration level of the vibration output by the tactile HMI device from the level of notification determined based on the risk index value. The attention-raising device according to claim 2, wherein the tactile parameters include a parameter that modifies at least one of the period of the vibration, the amplitude of the vibration, and the waveform of the vibration.

6. A processor installed in a warning device that uses multiple types of HMI devices to notify the vehicle driver of the presence of risks, An index value acquisition step, which acquires a risk index value indicating the degree of possibility of contact occurring between the vehicle and an object present around the vehicle, based on the output of multiple sensors mounted on the vehicle, A geographic characteristics acquisition step to acquire the geographic characteristics of the driver of the vehicle, A state recognition step to recognize the state of the driver of the vehicle, Based on the results of the state recognition step for recognizing the driver's state, an emotion estimation step is performed to estimate the driver's emotions. A decision step to determine the level of notification to inform the driver of the presence of risk and the type of HMI device to be used for notification, based on the driver's emotions estimated by the emotion estimation step, the driver's geographical characteristics acquired by the geographical characteristics acquisition step, and the risk indicator values ​​acquired by the indicator value acquisition step. A method of raising awareness and prompting action.

Images