Attention reminding device and attention reminding method

Abstract

The invention provides an attention reminding device and an attention reminding method. The present invention is provided with: a risk index value acquisition unit (155) that acquires a risk index value; a geographic characteristic acquisition unit (151) that acquires geographic characteristics of a driver (D) of the vehicle (1); a driver state recognition unit (152) that recognizes the state of a driver (D) of the vehicle (1); a driver mood estimation unit (153) that estimates the mood of the driver (D) on the basis of the recognition result of the state of the driver (D); and a notification determination unit (156) that determines, on the basis of the driver emotion estimated by the driver emotion estimation unit (153), the geographic characteristics of the driver (D) acquired by the geographic characteristic acquisition unit (151), and the risk index value acquired by the risk index value acquisition unit (155), the level of notification for notifying the driver (D) of the presence of risk, and the type of HMI device (30) for notification.

Description

Technical Field

[0001] This invention relates to a attention reminder device and a attention reminder method. Background Technology

[0002] In recent years, devices have been known to output information using a language corresponding to the occupants of a vehicle.

[0003] For example, Patent Document 1 discloses a system that has the function of accepting user settings for a country or language and the function of outputting information using the set country or language.

[0004] In addition, the vehicle information display device disclosed in Patent Document 2 includes: a facial image capturing unit that captures a user's facial image; a nationality filtering unit that filters the user's nationality based on the captured facial image; a language estimation unit that selects a calling language for the user based on the filtered nationality and makes a call, and estimates the language used by the user based on the user's response to the call; and a display control unit that uses the estimated language to display a screen that the user can understand.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2022-056875

[0008] Patent Document 2: Japanese Patent Application Publication No. 2019-113520 Summary of the Invention

[0009] The problem that the invention aims to solve

[0010] When informing drivers of the existence of risks, the following challenges exist: notification should not only be provided through text, but also through multiple methods such as sound and color to minimize driver misidentification. However, due to differences in drivers' cultural characteristics and social backgrounds leading to varying emotional responses, misidentification may occur. This application aims to address these challenges by improving visual recognition. Furthermore, it contributes to further improving traffic safety and promoting the development of sustainable transportation systems.

[0011] Methods for solving problems

[0012] One aspect of the present invention is a attention alert device that notifies a driver of a vehicle of the presence of a risk through multiple human-machine interface devices. The attention alert device comprises: an index value acquisition unit that acquires a risk index value based on the outputs of multiple sensors mounted on the vehicle, the risk index value representing the degree of probability that an object in the vicinity of the vehicle will come into contact with the vehicle; a geographic characteristic acquisition unit that acquires the geographic characteristics of the driver; a state recognition unit that identifies the state of the driver; an emotion estimation unit that estimates the driver's emotion based on the state recognition unit's identification result of the driver's state; and a decision unit that determines the level of notification of the presence of risk to the driver and the type of human-machine interface device used for notification, based on the driver's emotion estimated by the emotion estimation unit, the driver's geographic characteristics acquired by the geographic characteristic acquisition unit, and the risk index value acquired by the index value acquisition unit.

[0013] In other embodiments of the invention, the plurality of human-machine interface devices includes: a visual human-machine interface device that conveys information to the driver visually; an auditory human-machine interface device that conveys information to the driver verbally; and a tactile human-machine interface device that applies tactile stimulation to the driver, wherein the decision unit determines at least one of the visual human-machine interface device, the auditory human-machine interface device, and the tactile human-machine interface device as the human-machine interface device for notification based on the driver's mood, the driver's geographical characteristics, and the risk index value.

[0014] In other embodiments of the invention, the decision unit determines the value of a visual parameter that alters the image displayed by the visual human-machine interface device based on the level of the notification determined by the risk index value. The visual parameter includes a parameter that alters at least one of the following: magnification and reduction of the image, display area of ​​the image, display brightness of the image, display color of the image, shape of the image, and flicker period of the image.

[0015] In another aspect of the invention, the decision unit determines the value of an auditory parameter that alters the voice output by the auditory human-machine interface device based on the level of the notification determined by the risk index value. The auditory parameter includes parameters that alter at least one of the playback speed or playback period, volume, and pitch of the voice data that is the source of the voice.

[0016] In another aspect of the invention, the decision unit determines the value of a tactile parameter that changes the vibration level of the vibration output by the tactile human-machine interface device based on the level of the notification determined based on the risk index value. The tactile parameter includes a parameter that changes at least one of the period of the vibration, the amplitude of the vibration, and the waveform of the vibration.

[0017] One aspect of the present invention is a attention alert method that causes a processor in an attention alert device equipped with a vehicle that notifies the driver of the presence of a risk via a variety of human-machine interface devices to perform the following steps: an index value acquisition step, which acquires a risk index value based on the output of multiple sensors mounted on the vehicle, the risk index value representing the degree of probability that an object existing in the vicinity of the vehicle will come into contact with the vehicle; a geographic characteristic acquisition step, which acquires the geographic characteristics of the driver of the vehicle; a state recognition step, which identifies the state of the driver of the vehicle; an emotion estimation step, which estimates the driver's emotion based on the identification result of the driver's state by the state recognition step; and a decision step, which determines the level of notification of the presence of a risk to the driver and the type of human-machine interface device used for notification based on the driver's emotion estimated by the emotion estimation step, the driver's geographic characteristics acquired by the geographic characteristic acquisition step, and the risk index value acquired by the index value acquisition step.

[0018] Invention Effects

[0019] According to one aspect of the present invention, when a driver is notified of the presence of a risk through multiple HMI devices, the driver's emotional state may vary depending on their cultural characteristics, social background, and the notification of the risk's existence, potentially leading to misinterpretation. Therefore, by determining the level of the risk notification and the type of HMI device used for the notification based on the driver's geographical location and emotional state, the likelihood of the driver misinterpreting the notification information can be reduced, thus improving the reliability of the notification to the driver. Attached Figure Description

[0020] Figure 1 This is a diagram showing the structure of a vehicle's control system.

[0021] Figure 2 It is a diagram showing the structure inside the vehicle's interior.

[0022] Figure 3 This is a diagram illustrating an example of an emotion mapping.

[0023] Figure 4 This is a diagram showing an example of the first notification settings table.

[0024] Figure 5 This is a diagram showing an example of a second notification settings table.

[0025] Figure 6 This is a diagram showing an example of a third notification settings table.

[0026] Figure 7 This is a flowchart illustrating the operation of the attention reminder device.

[0027] Figure 8 This is a flowchart illustrating the emotion estimation process.

[0028] Figure 9 This is a flowchart illustrating the object detection process.

[0029] Explanation of reference numerals in the attached figures

[0030] 1…Vehicle; 3…Windshield; 4…Instrument panel; 10…Vehicle sensors; 10A…Front camera; 30…HMI device; 40…Microphone; 50…Driver monitoring camera; 60…Navigation device; 70…Wearable device; 100…Attention reminder device; 110…Communication unit; 130…Storage unit; 131…Control program; 133…Emotion map; 135…First notification setting table; 137…Second notification setting table; 139…Third notification setting table; 150…Processor; 151…Geographical characteristics acquisition unit; 152…Driver status recognition unit; 153…Driver emotion estimation unit; 154…Object detection unit; 155…Risk indicator value acquisition unit; 156…Notification decision unit; 157…Execution control unit; 310…Touch panel; 330…Speaker; 330A…Right speaker; 330B…Left speaker; 350…Seat vibration unit. Detailed Implementation

[0031] [1. Vehicle Structure]

[0032] Figure 1 This is a diagram showing the structure of the control system of the attention reminder device 100. Figure 2 This is a diagram showing the structure at the front of the passenger compartment of a vehicle 1 equipped with a attention reminder device 100. Figure 2 The X, Y, and Z axes shown are orthogonal to each other. The Z axis represents the vertical direction. The X and Y axes are parallel to the horizontal direction when vehicle 1 is in motion. The X axis represents the width of vehicle 1, i.e., the left-right direction. The Y axis represents the front-back direction of vehicle 1. The positive direction of the X axis represents the right. The positive direction of the Y axis represents the front. The positive direction of the Z axis represents the top.

[0033] The vehicle 1 is equipped with vehicle sensors 10, an HMI (Human Machine Interface) device 30, a microphone 40, a driver monitoring camera 50, a navigation device 60, and a attention reminder device 100. In addition, the driver D of the vehicle 1 wears a wearable device 70, for example, on the driver D's arm.

[0034] The vehicle sensor 10 includes at least one of one or more cameras, radars, lidar, and sonars distributed around the vehicle body of the vehicle 1, and detects objects present around the vehicle 1 that are the subject of notification. An object refers to an object that has the potential to collide or come into contact with the vehicle 1, and also an object whose presence needs to be notified to the driver of the vehicle 1. The vehicle sensor 10 outputs the data measured by the camera, radar, lidar, or sonar as sensor data to the attention alert device 100.

[0035] exist Figure 2 As an example of vehicle sensor 10, a front camera 10A is shown mounted on the upper part of the windshield 3 of vehicle 1 and captures images of the front of vehicle 1.

[0036] The HMI device 30 is a device that applies stimulation to the driver D of the vehicle 1. The HMI device 30 of this embodiment includes a touch panel 310 that transmits visual information to the driver D's vision, a speaker 330 that transmits auditory information to the driver D's hearing, and a seat vibration unit 350 that applies tactile stimulation to the driver D. The touch panel 310 corresponds to a visual HMI device, the speaker 330 corresponds to an auditory HMI device, and the seat vibration unit 350 corresponds to a tactile HMI device.

[0037] exist Figure 2 In the example of the structure at the front of the vehicle compartment shown in the vehicle 1, a touch panel 310 is mounted on the instrument panel 4 of the vehicle 1, and the speaker 330 includes a right speaker 330A located on the driver's side and a left speaker 330B located on the passenger side. The touch panel 310 displays visual information and functions as a receiving unit for input information input by the driver D.

[0038] A seat vibration unit 350 is installed in the seat where the driver D sits. The seat vibration unit 350 includes, for example, a vibration motor that generates vibrations within the driver's seat, driven by a vibration control signal input from the attention alert device 100. This provides tactile stimulation to the driver D. The HMI device 30 that provides tactile stimulation to the driver D can also be an electrically operated seatbelt that provides tactile stimulation by changing the tension of the seatbelt.

[0039] Additionally, a microphone 40 and a driver monitoring camera 50 are installed in the dashboard 4 of vehicle 1. The microphone 40 collects the speech of driver D. The microphone 40 converts the collected speech into digital voice data and outputs the converted voice data to the attention alert device 100. The driver monitoring camera 50 captures an image of driver D sitting in the driver's seat of vehicle 1. The driver monitoring camera 50 generates an image containing driver D's face and outputs the generated image to the attention alert device 100.

[0040] Wearable device 70, worn on the arm of driver D, measures the biometric information of driver D, including heart rate, body temperature, and blood pressure. Wearable device 70 is wirelessly connected to attention reminder device 100 via near-field communication and sends the measured biometric information to attention reminder device 100.

[0041] The navigation device 60 searches for a route, i.e. a guidance route, to the destination set by the passengers, including the driver D of vehicle 1, and performs route guidance to the destination by displaying the searched guidance route on the touch panel 310.

[0042] [2. Note the structure of the reminder device]

[0043] Next, the structure of the attention reminder device 100 will be described. The attention reminder device 100 consists of an ECU and a computer, which include a communication unit 110, a storage unit 130, a processor 150, and input / output interfaces (not shown).

[0044] The communication unit 110 and the wearable device 70 conduct short-range wireless communication based on standards such as Bluetooth (registered trademark) and UWB (Ultra-Wide Band).

[0045] The storage unit 130 may include, for example, non-volatile RAM (Read Only Memory), or ROM and RAM (Random Access Memory). Alternatively, the storage unit 130 may also have an auxiliary storage device such as an SSD (Solid State Drive).

[0046] The storage unit 130 stores the control program 131, the emotion mapping diagram 133, the first notification setting table 135, the second notification setting table 137, and the third notification setting table 139 executed by the processor 150. Details about the emotion mapping diagram 133, the first notification setting table 135, the second notification setting table 137, and the third notification setting table 139 will be described later.

[0047] Processor 150 is an arithmetic processing device equipped with processors such as CPU (Central Processing Unit) and MPU (Micro-Processing Unit). Processor 150 can be composed of a single processor or multiple processors. Alternatively, processor 150 can be composed of a System-on-a-Chip (SoC) integrating part or all of the storage unit 130 with other circuitry. Furthermore, processor 150 can be a combination of a CPU that executes programs and a DSP (Digital Signal Processor) that performs specified arithmetic operations. Moreover, processor 150 can be configured to have all its functions installed in hardware, or it can be configured to use a programmable device.

[0048] The attention reminder device 100 includes a geographic location acquisition unit 151, a driver status recognition unit 152, a driver emotion estimation unit 153, an object detection unit 154, a risk indicator value acquisition unit 155, a notification decision unit 156, and an execution control unit 157 as functional units. These functional units are implemented by the processor 150 executing control according to the control program 131.

[0049] The geographic characteristics acquisition unit 151 acquires the geographic characteristics of the driver D of vehicle 1. Geographical characteristics include the language spoken by driver D, the country and region where driver D resides, etc. Alternatively, it can be configured to acquire the culture, transportation environment, social background, etc., of each country and region as geographic characteristics. For example, when the communication unit 110 is connected to the Internet, the culture, transportation environment, social background, etc., of the country and region where driver D resides can also be acquired via the Internet.

[0050] In this embodiment, the geographic characteristic acquisition unit 151 acquires information about the country where driver D resides or the language spoken by driver D as the geographic characteristic of driver D. For example, the geographic characteristic acquisition unit 151 acquires the language setting information of navigation device 60 from navigation device 60. The geographic characteristic acquisition unit 151 may also analyze the voice of driver D input from microphone 40 to determine the language and acquire language information. The geographic characteristic acquisition unit 151 outputs the language information used by driver D, acquired as geographic characteristic information, to the notification decision unit 156.

[0051] The driver state recognition unit 152 is equivalent to a state recognition unit. The driver state recognition unit 152 identifies the state of the driver D based on at least one of the following: information input through touch operation of touch panel 310, image of driver D captured by driver monitoring camera 50, voice input to microphone 40 of driver D, and biometric information of driver D detected by wearable device 70, and identifies the state of driver D riding in vehicle 1.

[0052] The driver state recognition unit 152 identifies the following elements as the state of driver D, for example.

[0053] The first element is the answer input via touch operation of the touch panel 310 in response to inquiries such as "How are you feeling today?" displayed on the touch panel 310 when the driver D is in the vehicle 1.

[0054] The second element is… the facial expressions and actions of driver D as identified from the image of driver D.

[0055] The third element is... the content, intonation, pitch, and volume of the driver D's voice.

[0056] The fourth element is the driver D's biological information (heart rate, blood pressure, body temperature, etc.).

[0057] Figure 3 This is a diagram illustrating an example of the emotion mapping diagram 133.

[0058] The driver emotion estimation unit 153 is equivalent to an emotion estimation unit. The driver emotion estimation unit 153 estimates the driver D's emotion based on the state of the driver D identified by the driver state recognition unit 152. The driver emotion estimation unit 153, based on... Figure 3 The emotion mapping diagram 133 shown uses emotional valence and alertness as evaluation factors to estimate driver D's emotion. Emotional valence represents driver D's level of unpleasantness to pleasantness, while alertness is an evaluation factor representing driver D's level of excitement to calmness. For example, as... Figure 3 As shown, the driver emotion estimation unit 153 estimates the driver D's emotion using seven levels: no emotion, sadness 1, sadness 2, sadness 3, happiness 1, happiness 2, and happiness 3. The driver emotion estimation unit 153 outputs the estimated driver D's emotion information to the notification decision unit 156.

[0059] The driver emotion estimation unit 153 estimates the driver D's emotion, for example, by setting the levels of emotional valence and alertness based on the first to fourth elements as follows.

[0060] Regarding the first element mentioned above, the driver emotion estimation unit 153 sets the driver D's emotional valence to a relatively high value (e.g., 3) when the driver's answer to the question is "as usual" and sets the alertness to a relatively low value (e.g., -3) when the driver's answer to the question is "different from usual".

[0061] Regarding the second element mentioned above, the driver emotion estimation unit 153 sets a relatively high emotional valence (e.g., 5) when it judges that driver D is making a smiling expression based on driver D's facial expression, and sets a relatively low emotional valence (e.g., -5) when it judges that driver D is making an angry expression. Furthermore, the driver emotion estimation unit 153 sets a relatively high alertness (e.g., 3) when it judges that driver D is acting irrationally based on driver D's actions, and sets a relatively low alertness (e.g., -3) when it judges that driver D has hardly acted at all.

[0062] Regarding the third element mentioned above, the driver emotion estimation unit 153 sets a relatively high emotional valence (e.g., 5) when it determines that the driver D's speech contains positive content such as praise or expectation, and a relatively low emotional valence (e.g., -5) when it determines that the speech contains negative content such as criticism. Furthermore, when the driver D's speech contains specific keywords ("good," "excellent," etc.), the driver emotion estimation unit 153 sets emotional valence and alertness values ​​associated with those keywords.

[0063] In addition, the driver emotion estimation unit 153 sets the alertness to a relatively high value (e.g., 3) based on the pitch (frequency) of the driver D's voice. For example, if the driver D's voice is above a predetermined pitch, the alertness is set to a relatively low value (e.g., -3).

[0064] Regarding the fourth element mentioned above, the driver emotion estimation unit 153 applies the detected values ​​of driver D's biological information (heart rate, blood pressure, body temperature, etc.) to a pre-prepared correspondence table between the detected values ​​and emotional valence and alertness to set the emotional valence and alertness. The correspondence table can be created based on the driver D's profile input by driver D and the biological information of driver D detected in the past.

[0065] Furthermore, the driver emotion estimation unit 153 can also identify the values ​​of emotional valence and alertness based on changes in the vehicle 1's speed detected by the speed sensor, the vehicle 1's behavior detected by the gyroscope sensor, and changes in the vehicle 1's position detected by the GNSS (Global Navigation Satellite Systems) sensor. Illustrations of the speed sensor, gyroscope sensor, and GNSS sensor are omitted. Additionally, the driver emotion estimation unit 153 can estimate the driver D's emotion using either emotional valence or alertness, or using other evaluation factors.

[0066] The object detection unit 154 detects objects present around the vehicle 1 based on sensor data input from the vehicle sensor 10. In this embodiment, the object detection unit 154 detects objects present in front of the vehicle 1 in its direction of travel that may come into contact with or collide with the vehicle 1. That is, when the vehicle 1 is moving forward, it detects objects in front of the vehicle 1; when the vehicle 1 is moving backward, it detects objects behind the vehicle 1. The object detection unit 154 detects the position, speed, and direction of movement of the detected objects. The object detection unit 154 outputs information indicating the position, speed, and direction of movement of the detected objects to the risk indicator value acquisition unit 155.

[0067] In addition to sensors such as the front camera included in the vehicle sensor 10, the object detection unit 154 can also use vehicle-to-everything (V2X) communication devices, GNSS units, and V2X (Vehicle-to-Everything) communication devices to detect objects and their relative positions. Furthermore, the object detection unit 154 can also use judgments performed by a server in a virtual environment to detect objects reflected in captured images and their relative positions. V2X communication targets include other vehicles, pedestrians, networks, and infrastructure.

[0068] The risk index value acquisition unit 155 acquires the risk index value of the object detected by the object detection unit 154. The risk index value is a value that indicates the probability of vehicle 1 contacting or colliding with the object. The risk index value acquisition unit 155 is equivalent to the index value acquisition unit.

[0069] The risk index value acquisition unit 155 is input with information such as the position, speed, and direction of movement of the object detected by the object detection unit 154. Additionally, images captured by a camera, serving as sensor data, are also input to the risk index value acquisition unit 155. Based on the position, direction, and speed of the object, the position, direction, and speed of vehicle 1, and the illumination status of traffic lights detected from the camera images, the risk index value acquisition unit 155 predicts the movement path of the object and vehicle 1. Then, the risk index value acquisition unit 155 calculates a risk index value based on the closest position between vehicle 1 and the object and the distance between vehicle 1 and the object at that position. For example, the risk index value is obtained by scoring the position, direction, speed, and distance between vehicle 1 and the object, multiplying the scores by weighting coefficients, and summing the results.

[0070] When predicting the movement route of an object, the risk indicator value acquisition unit 155 can refer to the illumination status of the vehicle's turn signals and brake lights if the object is a vehicle, and to the direction the pedestrian's face is facing if the object is a pedestrian. Similarly, when predicting the movement route of vehicle 1, the risk indicator value acquisition unit 155 can also refer to the illumination status of vehicle 1's turn signals and brake lights obtained from vehicle sensor 10, and if navigation device 60 is providing route guidance, it can also refer to the route being guided.

[0071] In addition to detecting the contact risk between the vehicle 1 and objects actually detected in the surrounding environment, the risk indicator value acquisition unit 155 can also estimate the contact risk with objects that have not yet been detected but may appear (hypothetical vehicles, pedestrians, etc.) based on factors such as visibility at intersections and the number of accidents. The risk indicator value acquisition unit 155 outputs the acquired risk indicator values ​​to the notification decision unit 156.

[0072] The notification decision unit 156 is equivalent to the decision unit. The notification decision unit 156 is input with the language information spoken by the driver D obtained by the geographic characteristics acquisition unit 151, the emotional information of the driver D estimated by the driver emotion estimation unit 153, and the risk index value of each object obtained by the risk index value acquisition unit 155.

[0073] Based on the input information, the notification decision unit 156 determines whether there is an object to be notified. Furthermore, if there is an object to be notified, the notification decision unit 156 determines the type and level of notification to be sent regarding the existence of the object.

[0074] Figure 4 This is a diagram showing an example of the first notification settings table 135.

[0075] Table 135, the first notification setting table, is a table for registering risk indicator values ​​and corresponding notification levels. Hereinafter, the risk indicator values ​​will be referred to as risk indicator values ​​R.

[0076] First, the notification decision unit 156 determines whether there is an object whose risk indicator value R, input from the risk indicator value acquisition unit 155, is 2 or higher. If there is an object with a risk indicator value R of 2 or higher, the notification decision unit 156 determines that object to be notified. A risk indicator value R of 2 or higher is equivalent to a threshold.

[0077] Furthermore, when there is an object with a risk index value R of 2 or higher, the notification decision unit 156 determines the notification level based on the risk index value of that object and the first notification setting table 135. Figure 4 In the example of the first notification setting table 135 shown, the notification level is set to "low" when the risk indicator value R is 2 ≤ R < 4 and 4 ≤ R < 6. The notification level is set to "medium" when the risk indicator value R is 6 ≤ R < 8. The notification level is set to "high" when the risk indicator value R is 8 ≤ R < 10 and R = 10.

[0078] Figure 5 This is a diagram showing an example of the second notification settings table 137.

[0079] exist Figure 5 In the second notification setting table 137 shown, information about language as a geographical characteristic, the driver D's mood, and the type of notification are registered together. The driver D's mood includes "no mood," "happy 1," "happy 2," "happy 3," "sad 1," "sad 2," and "sad 3." In the second notification setting table 137, for each of these moods, the type of notification and the level of notification are registered together.

[0080] Next, the notification decision unit 156 determines the type of HMI device 30 used to notify of risks present around vehicle 1 based on the input language information and emotional information of driver D.

[0081] The notification decision unit 156 obtains the notification type, which indicates the type of HMI device 30 used in risk notifications, based on the language information of driver D input from the geographic characteristics acquisition unit 151 and the emotion information of driver D input from the driver emotion estimation unit 153, and with reference to the second notification setting table 137.

[0082] In this embodiment, when the notification type is visual, the image display on the touch panel 310 is selected.

[0083] In this embodiment, when the notification type is auditory, a notification based on sound output from speaker 330 is selected.

[0084] In the case where the notification type is tactile, in this embodiment, the vibration of the seat based on the seat vibration unit 350 is selected.

[0085] Figure 6 This is a diagram showing an example of the third notification settings table 139.

[0086] Figure 6 The third notification setting table 139 shown includes parameters for each type of visual, auditory, and tactile notification, and settings for these parameters according to each notification level. Among the parameters set for each notification type, the visual parameters are variables used to change the display mode of the image or graphic displayed on the touch panel 310. The auditory parameters are variables that change the output mode of the sound output from the speaker 330. The tactile parameters are variables that change the output mode of the vibration output from the seat vibration unit 350.

[0087] For visual purposes, these parameters include display size, the magnitude of size changes, display color, graphic shape, and flicker cycle. These parameters are equivalent to visual parameters.

[0088] Register these parameters in the third notification settings table 139 according to the settings for each notification level. Notification levels include high notification level, medium notification level, and low notification level. Hereinafter, high notification level will be displayed as high level, medium notification level as medium level, and low notification level as low level.

[0089] When the parameter is set to display size, the display size is registered as "large" in the high-level setting, "medium" in the medium-level setting, and "small" in the low-level setting.

[0090] When the parameter is set to the magnitude of the size change, "Large" is registered in the high-level setting, "Medium" in the medium-level setting, and "Small" in the low-level setting. The magnitude of the size change refers to the amount by which the display size of the graphics displayed on the touch panel 310 is enlarged or reduced. For example, when the magnitude of the size change is set to "Large," the size of the graphics displayed on the touch panel 310 changes drastically from a "Large" size to a "Small" size, and also drastically changes from a "Small" size to a "Large" size. By drastically changing the display size of the graphics displayed on the touch panel 310 in this way, the driver's attention will be drawn.

[0091] With the parameter set to brightness, the brightness was registered as "bright" in the high-level setting, "medium" in the medium-level setting, and "dark" in the low-level setting.

[0092] When the parameter is set to display color, "red" is registered as the display color in the high-level setting, "yellow" is registered as the display color in the medium-level setting, and "blue" is registered as the display color in the low-level setting.

[0093] When the parameter is a graphic shape, "sharp" is registered as the graphic shape in the high-level setting, "angular" is registered as the graphic shape in the medium-level setting, and "circular" is registered as the graphic shape in the low-level setting. The sharper the shape of the graphic displayed on the touch panel 310, the higher the perceived danger level; the more rounded the shape of the graphic, the lower the perceived danger level.

[0094] With the parameter set to blink period, "fast" is registered as the blink period in the high-level setting, "medium" is registered as the blink period in the medium-level setting, and "slow" is registered as the blink period in the low-level setting.

[0095] For hearing, the parameters include electronic sounds or spoken language, playback speed, volume, and pitch. These parameters are equivalent to auditory parameters.

[0096] Register these parameters in the third notification settings table 139 according to the settings for each notification level.

[0097] When the parameter is set to electronic voice or spoken voice, "spoken voice" is registered in the high-level setting, "spoken voice" is registered in the medium-level setting, and "electronic voice" is registered in the low-level setting.

[0098] With playback speed as the parameter, "Fast" was registered as the playback speed in the high-level setting, "Medium" was registered as the playback speed in the medium-level setting, and "Slow" was registered as the playback speed in the low-level setting.

[0099] When the parameter is volume, "Large" is registered as volume in the high-level setting, "Medium" is registered as volume in the medium-level setting, and "Small" is registered as volume in the low-level setting.

[0100] When the parameter is the pitch (tone), in the high-level setting, "high" is registered as the pitch, and in the medium-level setting, "medium" is registered as the pitch. In the low-level setting, the output is an electronic sound, so the pitch is not registered.

[0101] For tactile sensation, parameters include frequency, period, amplitude, and waveform. These parameters are equivalent to tactile parameters.

[0102] Register these parameters in the third notification settings table 139 according to the settings for each notification level.

[0103] With the parameters being frequency and period, "fast" was registered in the high-level setting, "medium" was registered in the medium-level setting, and "slow" was registered in the low-level setting.

[0104] With the parameter being amplitude, "large" was registered in the high-level setting, "medium" was registered in the medium-level setting, and "small" was registered in the low-level setting.

[0105] When the parameter is a waveform, "rectangle" is registered in the high-level setting, "sawtooth" is registered in the medium-level setting, and "standing wave" is registered in the low-level setting.

[0106] Notification Decision Department 156 Figure 6 The third notification setting table 139 shown obtains parameter settings corresponding to the notification type determined in the second notification setting table 137 and the notification level determined in the first notification setting table 135. The notification decision unit 156 outputs the obtained notification type, notification level, and parameter settings to the execution control unit 157.

[0107] The execution control unit 157 controls the operation of the HMI device 30 based on the notification type, notification level, and parameter settings input from the notification decision unit 156.

[0108] When the type of notification input from the notification decision unit 156 is visual, the execution control unit 157 causes the touch panel 310 to display images, graphics, or text according to the parameter settings.

[0109] When the notification type input from the notification decision unit 156 is auditory, the execution control unit 157 causes the speaker 330 to output speech or electronic voice based on the parameter settings.

[0110] When the type of notification input from the notification decision unit 156 is tactile, the execution control unit 157 controls the seat vibration unit 350 according to the parameter settings, so that the seat where the driver D is seated vibrates.

[0111] Figure 5 The second notification setting table 137 shown and Figure 6 The settings in the third notification settings table 139 shown are settings that reflect geographical characteristics.

[0112] When driver D's geographical characteristics, i.e., cultural characteristics and social background, differ, the emotions that driver D experiences may vary, and notifications made using HMI device 30 may be misrecognized. Therefore, settings reflecting the cultural characteristics, traffic environment, social background, etc. of each country and region are registered in the second notification setting table 137 and the third notification setting table 139.

[0113] For example, if the driver emotion estimation unit 153 estimates that the driver D's emotion is "sadness," the auditory parameters are set to produce a "smooth" voice output that soothes the driver D's "sadness." Alternatively, if the driver emotion estimation unit 153 estimates that the driver D's emotion is "drowsy, tired, or bored," the auditory parameters are set to produce a happy emotion that uplifts the driver D's spirits.

[0114] The following describes the setting examples for the second notification setting table 137 and the third notification setting table 139.

[0115] [3.Notification example]

[0116] (Notification Example 1)

[0117] For example, the display color can be changed based on the driver's country (D).

[0118] In China, the color "white" is used to represent both "peace of mind" and "sadness".

[0119] In Nigeria, the color "red" is used to represent "anger," "love," and "fear."

[0120] In Egypt, the color "yellow" is used to represent positive emotions such as "happiness".

[0121] In Greece, the color "purple" was used to represent sadness.

[0122] (Notification Example 2)

[0123] For example, when using a human face or facial icon to notify risk information on a touch panel 310 for visual communication, using a thumbs-up gesture is allowed in Japan, but is set to "not use" in Brazil. Similarly, a nodding gesture is allowed in Japan, but is set to "not use" in India and Brazil.

[0124] (Notification Example 3)

[0125] For example, in Asian and Nordic countries where physical contact is less frequent, tactile-based notifications are set to "not used" or the notification level is set to "weak".

[0126] In addition, in Latin American countries where there is more physical contact, such as Italy, France, Spain, Portugal, and South American countries, tactile stimulation-based notifications are set to "use" or the level of tactile stimulation-based notifications is set to "high".

[0127] (Notification Example 4)

[0128] For example, in Asian countries where loudspeakers are widely used, notifications based on auditory communication are set to "use" or the level of notifications based on auditory communication is set to "high".

[0129] [4. Pay attention to the operation of the reminder device]

[0130] Figures 7-9 This is a flowchart illustrating the operation of the attention reminder device 100.

[0131] Reference Figures 7-9 The flowchart shown illustrates the operation of the attention reminder device 100.

[0132] First, when the vehicle's IG (ignition) switch is turned on (step S1), the attention reminder device 100 acquires the geographic characteristics (step S2). The geographic characteristics can be obtained, for example, from the language settings of the navigation device, or by analyzing the driver's speech, radio, and music voice to determine the geographic characteristics.

[0133] Next, the attention reminder device 100 performs emotion estimation processing for driver D (step S3). When estimating driver D's emotion through emotion estimation processing, the attention reminder device 100 detects obstacles, i.e., objects, existing around vehicle 1 through object detection processing (step S4).

[0134] Next, the attention reminder device 100 determines whether an object to be notified has been detected through object detection processing (step S5). If no object to be notified is detected (step S5 / No), the attention reminder device 100 returns to the processing in step S3.

[0135] Additionally, when the reminder device 100 detects an object that is the subject of a notification (step S5 / Yes), based on the geographical characteristics obtained in step S2, the driver D's mood determined in step S3, and the risk index value obtained in step S5, and referring to the first notification setting table 135, the second notification setting table 137, and the third notification setting table 139, it determines the type and level of the notification to driver D that the risk exists (step S6).

[0136] Next, the attention reminder device 100 causes the HMI device 30 corresponding to the notification type determined in step S6 to operate with an output corresponding to the notification level, and executes the notification action (step S7).

[0137] Next, the attention reminder device 100 determines whether the IG switch has been turned off (step S8). If the IG switch has not been turned off (step S8 / No), the attention reminder device 100 returns to the emotion estimation processing in step S3. Alternatively, if the IG switch is turned off (step S8 / YES), the attention reminder device 100 terminates the processing flow.

[0138] Figure 8 This is a flowchart showing the details of the emotion estimation process in step S3.

[0139] Reference Figure 8 The flowchart shown illustrates the details of the emotion estimation process.

[0140] First, the attention reminder device 100 acquires the driver D's voice input from the microphone 40 (step S31). Next, the attention reminder device 100 outputs a voice message from the speaker 330 inquiring about the driver D's status, and obtains a response to the inquiry via the microphone 40 and touch operation (step S32). Furthermore, the attention reminder device 100 acquires the image captured by the driver monitoring camera 50 (step S33) and obtains the driver D's biometric information from the wearable device 70 (step S34).

[0141] Next, the attention reminder device 100 identifies the state of driver D based on the information obtained in steps S31 to S34 (step S35). Then, based on the first to fourth elements described above and referring to the emotion mapping diagram 133, the attention reminder device 100 estimates the emotion of driver D (step S36).

[0142] Figure 9 This is a flowchart showing the detailed process of object detection.

[0143] Reference Figure 9 The flowchart shown illustrates the detailed process of object detection and processing.

[0144] First, the attention reminder device 100 obtains sensor data from the vehicle sensor 10 (step S41).

[0145] Next, the attention reminder device 100 detects objects around the vehicle 1 based on sensor data (step S42), and detects the position, speed and direction of movement of the detected objects.

[0146] Next, the attention reminder device 100 obtains the risk index value of the detected object (step S43). The attention reminder device 100 scores, for example, the position of the object, the direction of travel, the speed of movement, and the distance between the vehicle 1 and the object, and multiplies the scores by weighting coefficients, and calculates the sum of these values ​​after multiplying by the weighting coefficients as the risk index value.

[0147] Next, the attention reminder device 100 determines whether the obtained risk indicator value is above a preset threshold (step S44). If the risk indicator value is not above the threshold (step S44 / No), the attention reminder device 100 returns to the processing in step S41.

[0148] Next, if the risk indicator value is above the threshold (step S44 / Yes), the reminder device 100 determines that an object as the notification target has been detected (step S45).

[0149] [5. Other Implementation Methods]

[0150] The above-described implementation is merely one approach, which can be arbitrarily modified and applied.

[0151] In the above embodiment, the case where the visual HMI device is a touch panel 310 has been described. However, the visual HMI device may also be a display unit such as an LED (Light Emitting Diode), or a head-up display installed on the dashboard 4 of the vehicle 1. The head-up display notifies the presence of objects that may come into contact with the vehicle 1 by displaying an image on the windshield 3, which serves as the projection surface.

[0152] in addition, Figure 2 The structure of each part of the vehicle 1 shown is an example, and the specific installation method is not particularly limited. That is, it is not necessarily necessary to install hardware corresponding to each part separately; of course, it is also possible to configure the vehicle so that the functions of each part are implemented by a processor executing a program. In addition, some of the functions implemented by software in the above embodiments can be implemented as hardware, or some of the functions implemented by hardware can be implemented by software.

[0153] in addition, Figures 7-9 The steps shown are divided according to the main processing content, and this invention is not limited to the method or name of the processing unit division. It is also possible to divide the process into more step units based on the processing content. Alternatively, it is possible to divide the process into units containing more processes within a single step unit. Furthermore, the order of these steps can be appropriately changed without affecting the spirit of this invention.

[0154] Furthermore, when implementing the attention reminder method of the attention reminder device 100 described above using the processor 150, the program executed by the processor 150 can also be configured as a recording medium or a transmission medium for transmitting the program. That is, the control program 131 can also be implemented by recording the control program 131 in a removable information recording medium. Examples of information recording media include magnetic recording media such as hard disks, optical recording media such as CDs, USB (Universal Serial Bus) memory, SSD (Solid State Drive) and other semiconductor storage devices, or other recording media can be used.

[0155] [6. Structures supported by the above embodiments]

[0156] The above implementation supports the following structures.

[0157] (Structure 1)

[0158] A warning device notifies a driver of a vehicle of the presence of a risk via multiple human-machine interface devices. The warning device comprises: an index value acquisition unit that acquires a risk index value based on the outputs of multiple sensors mounted on the vehicle, the risk index value representing the degree of probability that an object in the vicinity of the vehicle will come into contact with the vehicle; a geographic characteristic acquisition unit that acquires the geographic characteristics of the driver; a state recognition unit that identifies the state of the driver; an emotion estimation unit that estimates the driver's emotion based on the state recognition unit's identification result of the driver's state; and a decision unit that determines the level of notification of the presence of risk to the driver and the type of human-machine interface device used for notification based on the driver's emotion estimated by the emotion estimation unit, the driver's geographic characteristics acquired by the geographic characteristic acquisition unit, and the risk index value acquired by the index value acquisition unit.

[0159] According to this structure, the level of notification to the driver regarding the existence of risk and the type of HMI device used for notification are determined based on the driver's geographical characteristics, the driver's mood, and the risk index value indicating the degree of likelihood of contact with the vehicle.

[0160] When drivers are notified of the presence of risks through various HMI devices, their emotional state can vary depending on their cultural characteristics, social background, and the resulting differences in perception, potentially leading to misinterpretation of the risk notification. Therefore, by determining the level of risk notification and the type of HMI device used for notification based on the driver's geographical location and emotional state, the likelihood of misinterpretation can be reduced, and the reliability of driver notifications can be improved.

[0161] (Structure 2)

[0162] According to the attention reminder device of structure 1, the multiple human-machine interface devices include: a visual human-machine interface device that conveys information to the driver visually; an auditory human-machine interface device that conveys information to the driver verbally; and a tactile human-machine interface device that applies tactile stimulation to the driver. The decision unit determines at least one of the visual human-machine interface device, the auditory human-machine interface device, and the tactile human-machine interface device as the human-machine interface device for notification based on the driver's emotions, the driver's geographical characteristics, and the risk index value.

[0163] According to this structure, various HMI devices include visual HMI devices, auditory HMI devices, and tactile HMI devices. The decision unit determines at least one of the visual HMI device, auditory HMI device, and tactile HMI device as the HMI device that notifies the driver of the presence of a risk. Therefore, the presence of a risk can be notified to the driver through visual information transmission, voice-based information transmission, and tactile stimulation.

[0164] (Structure 3)

[0165] According to the attention reminder device of structure 2, the decision unit determines the value of a visual parameter that changes the image displayed by the visual human-machine interface device based on the level of the notification determined based on the risk index value. The visual parameter includes a parameter that changes at least one of the following: magnification and reduction of the image, display area of ​​the image, display brightness of the image, display color of the image, shape of the image, and blinking period of the image.

[0166] According to this structure, the values ​​of the visual parameters are determined based on the notification level, which is determined by the risk indicator value. Therefore, at least one of the following parameters of the image displayed by the visual HMI device—magnification and reduction, display area, brightness, color, shape, and flicker cycle—can be changed to correspond to the notification level.

[0167] (Structure 4)

[0168] According to the attention reminder device of structure 2, the decision unit determines the value of the auditory parameter that changes the voice output by the auditory human-machine interface device based on the level of the notification determined based on the risk index value. The auditory parameter includes a parameter that changes at least one of the playback speed or playback period, volume, and pitch of the voice data that is the source of the voice.

[0169] According to this structure, the values ​​of the auditory parameters are determined based on the notification level, which is determined by the risk index value. Therefore, at least one of the following changes—the playback speed or playback period, volume, and pitch—of the voice data that can become the source of the voice output by the auditory HMI device, is modified to correspond to the notification level.

[0170] (Structure 5)

[0171] According to the attention reminder device of structure 2, the decision unit determines the value of a tactile parameter that changes the vibration level of the vibration output by the tactile human-machine interface device based on the level of the notification determined based on the risk index value. The tactile parameter includes a parameter that changes at least one of the vibration period, the vibration amplitude, and the vibration waveform.

[0172] According to this structure, the values ​​of the tactile parameters are determined based on the notification level, which is determined by the risk index value. Therefore, it is possible to change at least one of the period, amplitude, and waveform of the vibration output by the tactile HMI device to correspond to the notification level.

[0173] (Structure 6)

[0174] A method for alerting drivers involves a processor in an alerting device that alerts a driver of a vehicle to the presence of a risk via multiple human-machine interface devices performing the following steps: an index value acquisition step, which acquires a risk index value based on the outputs of multiple sensors mounted on the vehicle, the risk index value representing the degree of probability that an object in the vicinity of the vehicle will come into contact with the vehicle; a geographic characteristic acquisition step, which acquires the geographic characteristics of the driver of the vehicle; a state recognition step, which identifies the state of the driver of the vehicle; an emotion estimation step, which estimates the driver's emotion based on the state recognition result of the driver's state; and a decision step, which determines the level of notification of the presence of a risk to the driver and the type of human-machine interface device used for notification based on the driver's emotion estimated by the emotion estimation step, the driver's geographic characteristics acquired by the geographic characteristic acquisition step, and the risk index value acquired by the index value acquisition step.

[0175] According to this structure, the level of notification to the driver regarding the existence of risk and the type of HMI device used for notification are determined based on the driver's geographical characteristics, the driver's mood, and the risk index value indicating the degree of likelihood of contact with the vehicle.

[0176] When drivers are notified of the presence of risks through various HMI devices, their emotional state can vary depending on their cultural characteristics, social background, and the resulting differences in perception, potentially leading to misinterpretation of the risk notification. Therefore, by determining the level of risk notification and the type of HMI device used for notification based on the driver's geographical location and emotional state, the likelihood of misinterpretation can be reduced, and the reliability of driver notifications can be improved.

Claims

1. A attention reminder device that notifies the driver of a vehicle of the presence of a risk through various human-machine interface devices, wherein, The attention reminder device includes: The indicator value acquisition unit acquires a risk indicator value based on the output of multiple sensors mounted on the vehicle. This risk indicator value represents the degree of likelihood that an object existing in the vicinity of the vehicle will come into contact with the vehicle. The geographic characteristics acquisition unit acquires the geographic characteristics of the driver of the vehicle; A status recognition unit that identifies the status of the driver of the vehicle; An emotion estimation unit estimates the driver's emotion based on the state recognition unit's recognition result of the driver's state. as well as The decision unit, based on the driver's emotion estimated by the emotion estimation unit, the driver's geographical characteristics obtained by the geographical characteristics acquisition unit, and the risk index value obtained by the index value acquisition unit, determines the level of notification to the driver regarding the existence of risk and the type of human-machine interface device used for notification.

2. The attention reminder device according to claim 1, wherein, The various human-machine interface devices include: A visual human-machine interface device that conveys information to the driver visually; an auditory human-machine interface device that conveys information to the driver via voice; and a tactile human-machine interface device that applies tactile stimulation to the driver. The decision-making unit determines at least one of the visual human-machine interface device, the auditory human-machine interface device, and the tactile human-machine interface device as the human-machine interface device for notification, based on the driver's mood, the driver's geographical characteristics, and the risk index value.

3. The attention reminder device according to claim 2, wherein, The decision unit determines the value of the visual parameter that causes the image displayed by the visual human-machine interface device to change, based on the level of the notification determined by the risk indicator value. The visual parameters include parameters that modify at least one of the following: magnification and reduction of the image, display area of ​​the image, display brightness of the image, display color of the image, shape of the image, and flicker period of the image.

4. The attention reminder device according to claim 2, wherein, The decision-making unit determines the value of the auditory parameter that alters the voice output by the auditory human-machine interface device based on the notification level determined by the risk index value. The auditory parameters include parameters that modify at least one of the playback speed or playback period, volume, and pitch of the speech data that is the source of the speech.

5. The attention reminder device according to claim 2, wherein, The decision-making unit determines the value of the tactile parameter that changes the vibration level of the vibration output by the tactile human-machine interface device based on the notification level determined by the risk index value. The tactile parameters include parameters that modify at least one of the period of the vibration, the amplitude of the vibration, and the waveform of the vibration.

6. A method for alerting a driver, wherein a processor mounted on an alerting device that notifies the driver of a vehicle of the presence of a risk via various human-machine interface devices performs the following steps: The indicator value acquisition step involves obtaining a risk indicator value based on the output of multiple sensors mounted on the vehicle. This risk indicator value represents the degree of likelihood that an object existing around the vehicle will come into contact with the vehicle. The geographical characteristics acquisition step obtains the geographical characteristics of the driver of the vehicle. The state recognition step identifies the state of the driver of the vehicle; The emotion estimation step estimates the driver's emotion based on the identification result of the driver's state through the state recognition step; as well as The decision step, based on the driver's emotion estimated through the emotion estimation step, the driver's geographical characteristics obtained through the geographical characteristics acquisition step, and the risk index value obtained through the index value acquisition step, determines the level of notification to the driver regarding the existence of risk and the type of human-machine interface device used for notification.

Images