Information notification control device, information notification control method, on-vehicle information notification system, data table generation method, and program

Abstract

To provide a technique that can notify information in a notification mode suitable for a user.SOLUTION: An exemplary information notification control device comprises a memory for storing weighting data in association with a notification mode of information, notifies information in a notification mode determined based on the weighting data, estimates the emotion of a user who has notified of the information, and updates the weighting data based on a result of the estimation of the emotion.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present invention relates to a technique for controlling a mode of notification of information. [Background technology]

[0002] Conventionally, information is notified to a driver while driving using a display device or a voice output device. A configuration that enables switching of the notification mode so that information can be notified in a mode preferred by the driver is known. Examples of switching of the notification mode include switching between a setting to use both the display device and the voice output device, a setting to use either one of them, and a setting to use neither. Other examples include switching of visual settings such as display tone, and switching of voice settings such as voice quality.

[0003] Furthermore, Patent Document 1 discloses an emotion expression device that estimates an emotion from input information, synthesizes a facial expression and a voice corresponding to the estimated emotion, and outputs the synthesized expression and voice via a display device and a voice output device. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 3018865 Summary of the Invention [Problem to be solved by the invention]

[0005] When there are various options for the notification mode of information, the driver needs to find a notification mode that suits his / her preference from among the many options, which may be troublesome. In addition, after deciding on a preferred notification mode, the driver may get bored and be forced to perform a cumbersome setting process for the notification mode again. In order to solve these problems, it is conceivable to randomly select a notification mode from a plurality of options at regular intervals, but there is a possibility that the notification of information will be continued endlessly in a notification mode that does not suit the driver's preference. If the notification of information is continued endlessly in a notification mode that does not suit the driver's preference, it may have an undesirable effect on the driver's driving.

[0006] Also, it is possible to set the notification mode of information to correspond to the driver's emotion by referring to the technology disclosed in Patent Document 1. However, it is considered that the notification mode suitable for the driver's emotion is not universally determined, and even if the notification mode is simply matched to the driver's emotion, it is not necessarily a comfortable notification for each driver.

[0007] Although the above describes the issues using an information notification system applied to a vehicle as an example, it is believed that similar issues exist in information notification systems for various purposes, such as home, factory, or office use.

[0008] In view of the above, an object of the present invention is to provide a technique capable of notifying a user of information in a notification format suitable for the user. [Means for solving the problem]

[0009] An exemplary information notification control device of the present invention has a memory that stores weighting data associated with the notification mode of information, notifies information in a notification mode determined based on the weighting data, estimates the emotions of a user who has been notified of the information, and updates the weighting data based on the estimated emotion result. Effect of the Invention

[0010] According to the exemplary embodiment of the present invention, it is possible to notify a user of information in a notification mode suitable for the user. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 shows a schematic configuration of an information notification system. [Diagram 2] A diagram showing a conceptual configuration of an emotion estimation device. [Diagram 3] A flowchart showing an emotion estimation process executed by the emotion estimation device. [Figure 4] A block diagram showing the functional configuration of a controller provided in an information notification control device. [Diagram 5] FIG. 1 is a diagram showing an example of a weighting data table. [Figure 6] FIG. 13 is a diagram showing another example of a weighting data table. [Figure 7A] FIG. 7 is a diagram for explaining a plurality of selectable modes in the item “notification timing” of FIG. 6. [Figure 7B] FIG. 7 is a diagram for explaining a plurality of selectable modes in the item “notification time length” in FIG. 6; [Figure 8] FIG. 13 is a diagram showing an example of a weighting data table when grouping notification information is assumed; [Figure 9] FIG. 1 is a diagram for explaining an example of a method for obtaining weighting data from an estimated emotion. [Figure 10] A functional block diagram showing the function of the notification control unit in more detail. [Figure 11] A flowchart illustrating a process for generating a weighting data table. [Figure 12] A flowchart illustrating the flow of an information notification control process using a weighting data table. [Figure 13] A diagram showing the structure of the table to be used [Figure 14] FIG. 13 is a schematic diagram illustrating an example of an expression form table; [Figure 15] FIG. 7 is a schematic diagram showing the updated weighting data table of FIG. 6; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.

[0013] <1. Information notification system> Fig. 1 is a diagram showing a schematic configuration of an information notification system 100 according to an embodiment of the present invention. As shown in Fig. 1, the information notification system 100 includes an information source device 1, an information notification control device 2, and an information notification device 3. More specifically, the information notification system 100 is an in-vehicle information notification system mounted on a vehicle. However, the information notification system of the present invention is not limited to in-vehicle use, and can be applied to various uses such as home use, factory use, or office use.

[0014] The information source device 1 transmits information to be notified to a user. In this embodiment, the information source device 1 is mounted on a vehicle. The user to whom the information is notified is the driver who drives the vehicle. The information source device 1 may be configured to read and transmit information stored in a memory (not shown), for example. The information source device 1 may also be configured to generate and transmit information by itself. The information source device 1 may also be configured to receive information from another device, such as a server device, and transmit the information.

[0015] The information source device 1 is, for example, a navigation device that provides route guidance to a destination, or a part thereof. The information source device 1 is, for example, an in-vehicle safety device that monitors the behavior of the vehicle itself and the surrounding conditions of the vehicle and issues warnings, etc., or a part thereof. The information source device 1 is provided so as to be able to communicate with the information notification control device 2 wirelessly or via wire, and outputs information to be notified to a user (notification information) to the information notification control device 2.

[0016] The information notification control device 2 receives information (notification information) from the information source device 1. The information notification control device 2 decides in what notification mode the input information is to be notified, and performs processing so that the information notification device 3 performs notification in the decided notification mode. The information notification control device 2 may also have the function of the information source device 1. In this case, the information source device 1 is not necessary. The information notification control device 2 may be, for example, the above-mentioned navigation device, or an in-vehicle safety device, or a part of them. Furthermore, the information notification control device 2 may be a device separate from the navigation device or the in-vehicle safety device.

[0017] The information notification mode may include at least one of a notification mode related to audio and a notification mode related to display (visual). By notifying the user of the information by audio or display, the user can easily understand the information. By configuring the notification mode by audio or display to be selectable from a plurality of types, the user can be notified in a form that suits his / her preference. In this embodiment, the information can be notified by audio and display. For each of the notification by audio and the notification by display, the notification mode can be selected from a plurality of types.

[0018] Among the notification modes related to voice, elements that can notify the same information in multiple types of modes include, for example, voice volume, voice quality, intonation, voice speed, and the way of expressing the notification information. Among the notification modes related to display, elements that can notify the same information in multiple types of modes include, for example, the display color of the notification information, the background color against the display color, the brightness of the screen, the notification position on the screen, the display size of the notification information, and the way of expressing the notification information. Note that, examples of the ways of expressing the notification information include, for example, a command tone, a gentle tone, and the like.

[0019] In this embodiment, the information notification control device 2 is mounted on a vehicle. However, the information notification control device 2 may be a device (e.g., a server device) disposed outside the vehicle. Also, some of the components of the information notification control device 2 may be disposed outside the vehicle. As shown in FIG. 1, the information notification control device 2 includes a controller 21, a memory 22, and a feeling estimation device 23.

[0020] The controller 21 includes an arithmetic circuit that performs arithmetic processing. In this embodiment, the arithmetic circuit includes a processor. The processor includes, for example, a CPU (Central Processing Unit). The controller 21 may include one processor or multiple processors. When the controller 21 includes multiple processors, the processors only need to be capable of communicating with each other.

[0021] The memory 22 includes a volatile memory and a non-volatile memory. The volatile memory is specifically a RAM (Random Access Memory). The non-volatile memory is specifically a ROM (Read Only Memory). The non-volatile memory may also be a flash memory, a hard disk drive, or the like. The non-volatile memory stores programs and data that can be read by a computer. The functions of the controller 21 are realized by the processor executing arithmetic processing in accordance with the programs stored in the memory 22. The functions of the controller 21 will be described in detail later.

[0022] The feeling estimation device 23 estimates the feeling of a user (a driver in this embodiment). The feeling estimation device 23 is connected to the controller 21 via a communication bus or the like and can exchange information with the controller 21. The feeling estimation device 23 outputs a feeling estimation result to the controller 21. At least a part of the functions of the feeling estimation device 23 may be included in the functions of the controller 21. At least a part of the feeling estimation device 23 may not be mounted on the vehicle and may be provided to be able to communicate with the controller 21 via a communication network such as the Internet. That is, at least a part of the feeling estimation device 23 may be realized as a server device. The feeling estimation device 23 may be a device separate from the information notification control device 2. Details regarding the feeling estimation device 23 will be described later.

[0023] The information notification device 3 is controlled by the information notification control device 2 to notify information (notification information). As described above, in this embodiment, information is notified using display and sound. For this purpose, the information notification device 3 includes a display device 31 and an audio output device 32. The display device 31 is, for example, a liquid crystal display device having a liquid crystal display, or an organic EL (Electroluminescence) display device having an organic EL display. The display device 31 displays the notification information on a screen. The audio output device 32 is, for example, a speaker. The audio output device 32 outputs the notification information by sound.

[0024] The information notification device 3 may be, for example, a part of the above-mentioned navigation device or an in-vehicle safety device (using the display and speaker of the navigation device or the in-vehicle safety device).

[0025] <2. Information notification control device> Next, the information notification control device 2 will be described in detail.

[0026] [2-1. Emotion estimation device] First, the feeling estimation device 23 included in the information notification control device 2 will be described. Note that the configuration of the feeling estimation device 23 described below is an example, and a feeling estimation device with another configuration may be used. In addition, the feeling estimation device 23 described below is suitable as a device that estimates the feeling of a driver of a vehicle and is used for vehicle driving control, etc. The feeling estimation device 23 included in the information notification control device 2 may be configured to double as a feeling estimation device applied to such vehicle driving control, etc.

[0027] 2 is a diagram showing a conceptual configuration of the feeling estimation device 23. The feeling estimation device 23 is a computer device equipped with a controller including a processor such as a CPU or a GPU (Graphics Processing Unit), and a memory including a volatile memory and a non-volatile memory. The function of the feeling estimation device 23 is realized by the processor executing arithmetic processing according to a program stored in the memory.

[0028] The emotion estimation device 23 estimates emotions based on the emotion estimation model 233. In the emotion estimation model 233, emotions are estimated based on the index values ​​of two emotion indices that indicate mental and physical states related to emotions. In this embodiment, one of the two emotion indices is a central nervous system arousal level (hereinafter referred to as arousal level), and its index value can be calculated using "β waves / α waves of electroencephalograms." The other of the two emotion indices is an autonomic nervous system activity level (hereinafter referred to as activity level), and its index value can be calculated using the "standard deviation of heartbeat LF (Low Frequency) components (low frequency components of heartbeat waveform signals)."

[0029] The emotion estimation model 233 is configured by a model (calculation formula, conversion data table, etc.) for emotion estimation based on the arousal level and activity level. The emotion estimation model 233 is configured by a multidimensional model that estimates emotions using the arousal level and activity level as parameters. In this embodiment, the multidimensional model is a two-dimensional model with the arousal level and activity level as two axes. The two-dimensional model is created based on medical evidence (papers, etc.) showing the relationship between each of a plurality of indexes and emotions (the relationship between the arousal level and activity level and the emotion). Alternatively, the two-dimensional model is created based on the results of a questionnaire by many subjects. The questionnaire results are, for example, configured by the subject's emotion declaration and data consisting of the arousal level and activity level at that time. The arousal level is obtained based on the electroencephalogram measurement value, and the activity level is obtained based on the heart rate measurement value. The emotion estimation model 233 may be not only a two-dimensional model but also a multidimensional model of three or more dimensions.

[0030] The psychological plane table 234 is an example of a multi-dimensional model for emotion estimation. According to various medical evidences related to psychology, the psychology can be estimated based on two types of indicators showing the physical state. In the psychological plane table 234, the vertical axis is "arousal level" and the horizontal axis is "activity level." The upper side of the vertical axis is the arousal side, and the lower side is the non-arousal side. The right side of the horizontal axis is the sympathetic nerve activity (strong emotion) side, and the left side is the parasympathetic nerve activity (weak emotion) side.

[0031] In this psychological plane table 234, a corresponding psychological state is assigned to each of four quadrants separated by a vertical axis and a horizontal axis. The distance from each axis indicates the intensity of the corresponding psychological state. The first quadrant is assigned to the psychological states of "fun, joy, anger, and sadness." The second quadrant is assigned to the psychological state of "melancholy." The third quadrant is assigned to the psychological state of "relaxation and calm." The fourth quadrant is assigned to the psychological states of "anxiety, fear, and discomfort."

[0032] The shaded area 234a in the psychological plane table 234 is a neutral area where the indicator of the mental and physical state is vague and the emotion cannot be specified.

[0033] For example, the psychological state can be estimated from the coordinates obtained by plotting two types of index values ​​of mental and physical states (arousal level and activity level) obtained based on biosignals on a psychological plane. Specifically, the psychological state and its intensity can be estimated based on which quadrant of the psychological plane the plotted coordinates are in, where they are located within the quadrant, and how far they are from the origin. Note that it is also possible to use a method in which only the psychological state is estimated (without estimating the intensity of the psychological state) based on which quadrant of the psychological plane the plotted coordinates are in.

[0034] The emotion estimation model 233 utilizes the arousal and activity of emotion indices calculated based on electroencephalogram data and heartbeat data. For this purpose, the electroencephalogram data and heartbeat data may be measured by a measuring device (sensor), and the measurement results may be input to the emotion estimation model 233. However, in this embodiment, the configuration does not use the electroencephalogram data and heartbeat data detected by the electroencephalogram sensor and heartbeat sensor.

[0035] In this embodiment, the arousal level and activity level of the emotion index estimated by the AI ​​model are input from the gaze, face direction (face orientation), and voice. With this configuration, it is possible to save the trouble of attaching an electroencephalogram sensor and a heart rate sensor, which are contact sensors, to the subject of emotion estimation. In this configuration, gaze, face direction, and voice data are collected using a camera and a microphone, which are non-contact sensors, and the electroencephalogram data and heart rate data are estimated by the AI ​​model. For example, when estimating the emotion of a driver of a vehicle and using it for vehicle driving control, it is practically difficult to attach a contact sensor to the driver, and the technology using the AI ​​model as in this configuration is particularly useful. Note that in this embodiment, the AI ​​model does not estimate the electroencephalogram data and heart rate data themselves, but estimates the arousal level and activity level, which are emotion indexes calculated from the electroencephalogram data and heart rate data.

[0036] In the example shown in FIG. 2, a user U is a driver of a vehicle V. The first AI model 232a and the second AI model 232b of the emotion estimation device 23 are used to estimate the emotion of the user U, who is the driver. The first AI model 232a and the second AI model 232b are trained models that have been subjected to machine learning. The first AI model 232a and the second AI model 232b are AI models that have been designed and trained according to the type of input data.

[0037] The vehicle V is equipped with a camera C and a microphone M as on-board sensors. Then, gaze data and facial direction data of the user U based on an image captured by the camera C, and voice data of the user U based on a voice collected by the microphone M are input to the first AI model 232a and the second AI model 232b.

[0038] 2, the image captured by the camera C and the audio collected by the microphone M are processed by the behavior determination unit 231, which is exercised as a function of a controller provided in the emotion estimation device 23. Then, data obtained by the processing is input to the first AI model 232a and the second AI model 232b. However, this configuration is merely an example, and the image captured by the camera C and the audio collected by the microphone M may be directly input to the AI ​​models.

[0039] The behavior determination unit 231 performs an analysis process on the image information and voice information of the user U to determine predetermined types of behavior such as the gaze, facial direction, facial expression, and voice of the user U. The predetermined types of behavior are behaviors corresponding to the types of data input by the first AI model 232a and the second AI model 232b, and in this embodiment, are the gaze, facial direction, and voice (contents).

[0040] In the behavior determination unit 231, for example, a process of cutting out images (still images or videos) and audio data during a processing period is performed. Then, a process of cutting out the face and eyeballs of the user U from the image captured by the camera C and a process of detecting the orientation of the cut-out face and eyeballs are performed. In addition, a frequency analysis process is performed on the output signal of the microphone M. Note that the above-mentioned processing period is, for example, 10 seconds immediately before the emotion estimation timing. In addition, the programs and data for the behavior determination unit 231 to realize these operations are stored in a memory provided in the emotion estimation device 23.

[0041] The first AI model 232a is an AI model that inputs gaze data, face direction data, and voice data of the user U, and outputs the arousal level, which is an emotional index, and is designed to have a structure suitable for the input and output. The first AI model 232a is trained with a large number of learning data sets in which the gaze data, face direction data, and voice data are input data, and the arousal level corresponding to the input data is used as the correct answer data.

[0042] The second AI model 232b is an AI model that inputs gaze data, face direction data, and voice data of the user U, and outputs activity, which is an emotional index, and is designed to have a structure suitable for the input and output. The second AI model 232b is trained with a large number of training data sets in which gaze data, face direction data, and voice data are input data, and activity corresponding to the input data is used as correct answer data.

[0043] The learning data used for learning the first AI model 232a and the second AI model 232b is created based on observation data of a subject who boarded an experimental vehicle for creating the learning data. Specifically, a camera and a microphone are set in the experimental vehicle similar to the camera C and microphone M mounted on the vehicle V in Fig. 2, and an image and a voice of the subject are acquired. Then, a behavior determination unit similar to the behavior determination unit 231 in Fig. 2 processes the acquired image and voice, and gaze data, face direction data, and voice data are generated, which become input data for the learning data.

[0044] On the other hand, the subject is fitted with an electroencephalogram sensor and a heart rate sensor to measure the subject's electroencephalogram and heart rate. Based on the measured electroencephalogram and heart rate, the emotional indexes of arousal and activity are calculated and used as the correct answer data in the learning data.

[0045] Then, the first AI model 232a and the second AI model 232b are generated from input data (gaze data, face direction data, and voice data) generated and calculated from paired, i.e., simultaneous image, voice, electroencephalogram, and heart rate data, and correct answer data (awakening level and activity level). A learning data set is generated from many pieces of learning data generated in this way.

[0046] The awakening level output by the first AI model 232a and the activity level output by the second AI model 232b are input to the emotion estimation model 233, and emotion estimation is performed based on the emotion estimation model 233. The emotion estimation result estimated by the emotion estimation model 233 is output to the controller 21 included in the information notification control device 2.

[0047] 3 is a flowchart showing the emotion estimation process executed by the emotion estimation device 23. This flowchart shows the technical contents of a computer program that causes a computer device to realize the emotion estimation process. In addition, the computer program is stored in various computer-readable non-volatile recording media and provided (sold, distributed, etc.). The computer program may be composed of only one program, or may be composed of multiple programs that work together.

[0048] The process shown in Fig. 3 can be started from the time when the vehicle V starts and various controls based on the estimated emotion can be started. The process shown in Fig. 3 may be executed from the time when it is possible to start. Furthermore, the process shown in Fig. 3 may be executed at an appropriate timing after the time when it is possible to start, that is, at a timing when emotion estimation is required, or repeatedly over a time period when emotion estimation is required.

[0049] In step S1, the feeling estimation device 23 (specifically, a controller not shown) acquires data indicating the behavior of the user U, specifically, an image captured by a camera and a voice collected by a microphone of the user U. The acquired various pieces of information are stored in a memory provided in the feeling estimation device 23 as necessary. When the data indicating the behavior is acquired, the process proceeds to the next step S2.

[0050] In step S2, the feeling estimation device 23 judges the behavior of the user U, specifically, the gaze, face direction, and voice (content), based on each piece of data acquired in step S1 (images captured by the camera and audio collected by the microphone). When the behavior judgment is completed, the process proceeds to the next step S3.

[0051] In step S3, the feeling estimation device 23 inputs each data based on the behavior determination result determined in step S2 to the first AI model 232a and the second AI model 232b, and estimates the emotion indexes, awakeness and activity. When the emotion index values ​​(awakeness and activity) are estimated, the process proceeds to the next step S4.

[0052] In step S4, the feeling estimation device 23 applies (inputs) the two types of estimated emotion index values ​​(arousal level and activity level) to the emotion estimation model 233 (psychological plane table 234) to estimate the emotion of the user U. Assuming that the emotion is estimated, the process proceeds to the next step S5.

[0053] In step S5, the feeling estimation device 23 provides (outputs) the estimated emotion information of the user U to the controller 21 of the information notification control device 2. This ends the process shown in FIG.

[0054] In the above description, the emotion estimation model 233 (psychological plane table 234) is used to estimate the emotion. However, this is merely an example, and the emotion estimation model 233 may add emotion estimation results obtained by other methods to the emotion estimation model 233 to arrive at a final emotion estimation result.

[0055] In this embodiment, as described in detail later, the user's reaction (emotion) when the user is notified of information is observed to estimate (determine) whether the current notification mode of information is good or bad. Assuming such a configuration, if the user has a good impression (good emotion) with respect to the notification mode, the user's emotion can be said to be positive. Also, if the user has a bad impression (bad emotion) with respect to the notification mode, the user's emotion can be said to be negative. In this embodiment, it is important to determine (classify) whether the user's emotion is positive or negative with respect to the notification mode of information. That is, in this embodiment, it is necessary to be able to estimate whether the user's emotion is positive or negative from the user's emotion estimation result obtained from the emotion estimation device 23.

[0056] In the example using the emotion estimation model 233 described above, if the estimated emotion index (arousal level and activation level) falls in the first quadrant of the psychological plane table 234, the psychological states of "fun, joy, anger, sadness" are assigned. "Fun" and "joy" give a positive impression, while "anger" and "sadness" give a negative impression. That is, in the above example, if the estimated emotion index falls in the first quadrant of the psychological plane table 234, it may be difficult to determine whether it is positive or negative. Therefore, for example, a configuration may be adopted in which facial expression analysis is performed by image processing of the user's face image to estimate the emotion obtained by the emotion estimation model 233 in more detail. For example, the estimated psychological state of "fun, joy, anger, sadness" may be classified into one of the three, "fun, joy", "anger", and "sadness", depending on the result of the facial expression analysis.

[0057] Note that since it is sufficient to determine whether the user’s emotion is positive or negative with respect to the manner of notification of information, in some cases, the emotion estimation device 23 may be configured to perform emotion estimation based only on the result of facial expression analysis.

[0058] [2-2. Functions of the controller for the information notification control device] Next, the functions of the controller 21 included in the information notification control device 2 will be described in detail.

[0059] Fig. 4 is a block diagram showing a functional configuration of the controller 21 included in the information notification control device 2. As shown in Fig. 4, the controller 21 includes, as its functions, an emotion acquisition unit 211, an information acquisition unit 212, a notification mode setting unit 213, a weight processing unit 214, and a notification control unit 215. The functions of the controller 21 are realized by a processor executing arithmetic processing in accordance with a program stored in the memory 22. The number of programs that realize the functions of the controller 21 may be one or more.

[0060] The program stored in the memory 22 is a computer program that causes a computer to realize the functions of the controller 21 (in other words, the information notification control device 2). Such a computer program may be provided, for example, by a computer-readable nonvolatile recording medium. The nonvolatile recording medium may be, for example, an optical recording medium (for example, an optical disk), a magneto-optical recording medium (for example, a magneto-optical disk), a USB memory, or an SD card, in addition to the above-mentioned nonvolatile memory. As another example, the computer program may be provided from a program providing server via a communication line such as the Internet, or provided by so-called downloading.

[0061] Further, each of the functional units 211 to 215 may be realized by one program, but may be configured to be realized by separate programs for each functional unit. Further, each of the functional units 211 to 215 may be realized by having a processor execute a program, that is, by software, as described above, but may be realized by other methods. Each of the functional units 211 to 215 may be realized by using, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). That is, each of the functional units 211 to 215 may be realized by hardware using a dedicated IC or the like. Further, each of the functional units 211 to 215 may be realized by using both software and hardware. Further, each of the functional units 211 to 215 is a conceptual component. A function executed by one component may be distributed to multiple components. Further, functions possessed by multiple components may be integrated into one component.

[0062] In this embodiment, as will be described later in detail, the controller 21 has a function of determining whether to maintain or change the notification mode of the current information by using the emotion estimation result. The controller 21 also has a function of generating a data table (weighting data table 221, the details of which will be described later) required for determining whether to maintain or change the above. The multiple function units 211 to 215 include those used for both of these two functions.

[0063] The emotion acquisition unit 211 acquires an emotion estimation result from the emotion estimation device 23. The emotion acquisition unit 211 may be configured to periodically acquire an emotion estimation result, or may be configured to acquire the emotion estimation result at a necessary timing. In this embodiment, it is important to know the user's reaction (emotion) to the notification mode after the information is notified. For this reason, the necessary timing is immediately after the information is notified by the information notification device 3. In other words, the emotion estimation device 23 (information notification control device 2) estimates the emotion of the user who has been notified of the information.

[0064] The information acquisition unit 212 acquires information (notification information) to be notified to the user from the information source device 1. In this embodiment, the information acquisition unit 212 not only acquires the notification information when the information source device 1 transmits (outputs) the notification information at its own timing, but also acquires the notification information by requesting it from the information source device 1. The request for notification information to the information source device 1 is made, for example, when generating a weighting data table 221, the details of which will be described later.

[0065] The notification mode setting unit 213 selects a notification mode for notification from the information notification device 3 from among a plurality of selectable notification modes (information notification modes). Then, the notification mode setting unit 213 performs a notification mode setting process in the selected notification mode. The method of selecting the notification mode differs between normal control in which a generated weighting data table 221 is used and data table generation in which a new weighting data table 221 is generated.

[0066] Here, the weighting data table 221 will be described. The weighting data table 221 is stored in the memory 22 as shown in Fig. 1. The weighting data table 221 includes weighting data as described later. That is, the information notification control device 2 includes the memory 22 that stores the weighting data.

[0067] 5 is a diagram showing an example of weighting data table 221. In weighting data table 221, the items "visual", "audio", and "notification content" indicate the types of elements for which the notification mode settings can be changed. In weighting data table 221, multiple specific notification modes (in detail, three in this example) are prepared for each element for which the notification mode settings can be changed. Note that the number of selectable notification modes prepared for each item may be the same or different.

[0068] In detail, the item "visual" indicates that a plurality of display modes of the display device 31 can be selected. More specifically, it indicates that three display modes ("dark tone", "normal tone", and "bright tone") can be selected for the brightness of the display. Note that "normal tone" means a tone intermediate between "dark tone" and "bright tone". Display brightness may be other than the three modes. In some cases, a mode of "not displaying" may be included. As described above, there are a plurality of elements other than the brightness of the display that can be displayed in a plurality of modes. For this reason, a configuration may be adopted in which a plurality of modes can be selected for elements other than the brightness of the display. In addition, there may be a plurality of elements in the item "visual" that can have a plurality of modes selected. For example, the item "visual" may include "brightness of display" and "display position" as elements that can have a plurality of modes selected.

[0069] The item "audio" indicates that a plurality of types of audio modes of the audio output device 32 can be selected. More specifically, it indicates that three types of voice quality modes ("dark tone", "normal tone", and "bright tone") can be selected. Note that voice quality may be other than the three types. In some cases, a mode of "no audio output" may be included. As described above, there are a plurality of elements other than voice quality that can be output in a plurality of types of audio modes. For this reason, a configuration may be made in which a plurality of types of modes can be selected for elements other than voice quality. The item "audio" may also include a plurality of types of elements for which a plurality of types of modes can be selected. For example, the item "audio" may include "voice quality" and "volume" as elements for which a plurality of types of modes can be selected.

[0070] The item "notification content" indicates that multiple types of notification content can be selected for the notification content of the information. The notification content is a type that can be selected in common for both the display mode of the display device 31 and the audio mode of the audio output device 32, but different notification content items may be used for the display mode and the audio mode. The example shown in FIG. 5 indicates that three types of notification content can be selected ("Speeding and exceeding the speed limit. Please slow down," "It's dangerous, so slow down," and "Drive with ample space to maneuver."). The configuration of the item "notification content" shown in FIG. 5 is an example, just like the items "visual" and "audio," and the configuration may be changed as appropriate.

[0071] In the example shown in Fig. 5, the types of elements for which the notification mode setting can be changed are "visual", "audio", and "notification content", but this is also merely an example. The types and number of elements (items) for which the notification mode setting can be changed may be changed as appropriate. Fig. 6 is a diagram showing another example of the weighting data table 221.

[0072] In FIG. 6, the types of elements for which the notification mode setting can be changed include "visual", "audio", and "notification content", as well as "notification time". The "notification time" includes items "notification timing" and "notification time length" as elements for which multiple types of modes can be selected. The items "notification timing" and "notification time length" are each configured to allow three types of modes to be selected. Note that the multiple selectable modes included in the item "notification time" ("notification timing" and "notification time length") are selectable modes that are common to both the display mode of the display device 31 and the audio mode of the audio output device 32, but may be different notification content items for the display mode and the audio mode.

[0073] Fig. 7A is a diagram for explaining a plurality of selectable modes in the item "notification timing" in Fig. 6. As shown in Figs. 6 and 7A, the selectable modes in the item "notification timing" include "early", "normal", and "late". In "early", notification is started earlier than the notification start timing of "normal" by a time T1 (e.g., 5 seconds). On the other hand, in "late", notification is started later than the notification start timing of "normal" by a time T2 (e.g., 5 seconds). Note that the time T1 in the case of "early" and the time T2 in the case of "late" may be the same or different.

[0074] FIG. 7B is a diagram for explaining a plurality of selectable modes in the item "notification time length" in FIG. 6. As shown in FIG. 6 and FIG. 7B, selectable modes in "notification time length" include "longer", "normal", and "shorter". "Longer" is extended by time T3 (e.g., 5 seconds) based on the notification time length of "normal" (e.g., 10 seconds). In display, the display time is extended. For audio, for example, a message text is added or the audio speed is slowed. On the other hand, "shorter" is shortened by time T4 (e.g., 5 seconds) based on the notification time length of "normal". In display, the display time is shortened. For audio, for example, a part of the message text is cut or the audio speed is increased. Note that the time T3 in the case of "longer" and the time T4 in the case of "shorter" may be the same or different.

[0075] In the example shown in FIG. 6, the notification mode of information includes a notification mode related to the notification time. By configuring the notification timing and time length to be selectable from a plurality of types, it is possible to perform notification in a form that suits the user's preference. In this embodiment, as shown in FIG. 6, the notification mode related to the notification time can also be selected from a plurality of types. Note that, in the example shown in FIG. 6, the preferred mode can be selected for both the notification timing and the notification time length, but it is also possible to select only one of them.

[0076] Incidentally, the weighting data table 221 shown in FIG. 5 and FIG. 6 has an item called "weight" for each of a plurality of selectable modes present for each item (element for which the notification mode setting can be changed). "Weight" means weighting data, and is expressed, for example, by a numerical value. The weighting data is data that associates the notification mode of information with changes in the user's emotions. More specifically, the weighting data is data that indicates a tendency (direction) of change in emotions that the user who has received the notification of information varies according to the notification mode. How the weighting data is set will be described later. Also, how to create the weighting data table 221 in which such weighting data is tabulated, and how to use it will be described later.

[0077] Note that the weighting data table 221 stored in the memory 22 may not be singular, but may be plural. The examples shown in Figs. 5 and 6 show table structures used when notifying a warning (warning information) against speeding. In addition to this, a plurality of pieces of notification information are assumed, such as warnings and cautions when turning right or left. For this reason, a configuration may be adopted in which a separate weighting data table 221 is created for each piece of information. That is, a plurality of weighting data tables 221 in which weighting data is tabulated may be stored in the memory 22. This makes it possible to use weighting data tables 221 appropriately associated with each of a plurality of pieces of notification information, and to appropriately set the notification mode.

[0078] However, even when multiple pieces of notification information are assumed, it is not necessary to prepare one weighting data table 221 for each piece of notification information. For example, the notification information may be grouped into warning / caution notification information that notifies warnings and cautions and general notification information that notifies general information (simple directions, etc.), and the weighting data table 221 may be created for each group. FIG. 8 is a diagram showing an example of the weighting data table 221 when grouping of notification information is assumed. In the example shown in FIG. 8, the multiple selectable modes of the item "notification contents" are not distinguished by specific sentences as shown in FIG. 5, but are distinguished by the expression method of the notification contents. In the example shown in FIG. 8, one of three types, "command tone", "normal tone", and "mild tone", can be selected as the expression method (tone of expression) of the notification contents. The "normal tone" here is an intermediate expression mode between the "command tone" and the "mild tone".

[0079] Note that specific expression forms (sentences) corresponding to the "command tone", "normal tone" and "mild tone" need to be tabulated separately and stored in the memory 22. This allows the specific expression form (sentence) corresponding to the information to be notified to be selected and played back when notifying information. FIG. 14 is a schematic diagram illustrating an example of an expression form table. In the example shown in FIG. 14, notification information is classified into a plurality of groups in the form of warning / caution type notification information and general notification information, and specific expression forms (sentences) for each tone (tone of expression) are set for each notification content in each group. Also, in the example shown in FIG. 14, an expression form table is set for each user.

[0080] In addition, since the weighting data is set in association with changes in the user's emotions, it is assumed that the weighting data will differ for each user. For this reason, it is preferable that the weighting data table 221 is generated for each user. In this sense, multiple weighting data tables 221 may be stored in the memory 22. For example, when a family shares one vehicle, a weighting data table 221 may be created for each member of the family.

[0081] Returning to Fig. 4, the weighting processing unit 214 performs processing related to weighting data indicating changes (change trends or change trends) in the user's emotions that are changed according to the notification mode. For example, when generating (creating) the weighting data table 221, the weighting processing unit 214 determines the weight of the notification mode used in the notification according to the estimated emotion of the user after the notification of information (preferably immediately after the notification), and assigns weighting data to each notification mode. Furthermore, during normal control, the weighting processing unit 214 updates the weighting data assigned to each notification mode as necessary.

[0082] Here, we will explain a method for deriving weighting data from the estimated emotion of the user obtained from the emotion estimation device 23. Fig. 9 is a diagram for explaining an example of the method for deriving weighting data from the estimated emotion.

[0083] 9, when the estimated emotion is assigned to the first quadrant I and the third quadrant III of the psychological plane table 234, the emotion is estimated to be positive, and the weighting data is set to a positive value. When the estimated emotion is assigned to the second quadrant II and the fourth quadrant IV of the psychological plane table 234, the emotion is estimated to be negative, and the weighting data is set to a negative value. When the estimated emotion is assigned to the neutral region 234a of the psychological plane table 234, the emotion is estimated to be indefinite and therefore neither positive nor negative, and the weighting data is set to zero.

[0084] The weighting data may be classified into three categories: positive, negative, and zero. In the example shown in FIG. 9, the weighting data is further classified into two categories: positive (plus) and negative (minus). That is, the weighting data for a slightly positive case is "+1", and the weighting data for a very positive case is "+2". The weighting data for a slightly negative case is "-1", and the weighting data for a very negative case is "-2". In each of the positive and negative cases, whether the degree is small or large may be determined by the distance from each axis of the psychological plane table 234. A preset threshold may be used to determine whether the degree is small or large.

[0085] Some concrete examples will be given. In the psychological plane table 234, when an emotion is at a coordinate P1 relatively close to each axis of the first quadrant I, the weighting data is set to "+1". When an emotion is at a coordinate P2 relatively close to each axis of the second quadrant II, the weighting data is set to "-1". When an emotion is at a coordinate P3 relatively far from each axis of the second quadrant II, the weighting data is set to "-2". When an emotion is at a coordinate P4 relatively close to each axis of the third quadrant III, the weighting data is set to "+1". When an emotion is at a coordinate P5 relatively far from each axis of the third quadrant III, the weighting data is set to "+2". When an emotion is at a coordinate P6 relatively far from each axis of the fourth quadrant IV, the weighting data is set to "-2". When an emotion is at a coordinate P7 in the neutral area 234a, the weighting data is set to "0".

[0086] 9, the estimated emotion is converted into weighted data of five categories, but this is merely an example, and the estimated emotion may be converted into weighted data of multiple categories other than the five categories. Also, as described above, it may be difficult to classify the estimated emotion into positive and negative only based on the result of psychological plane table 234. In this case, the estimated emotion from psychological plane table 234 may be estimated in more detail by taking into account other methods (such as facial expression analysis), and conversion to weighted data may be performed according to the estimated emotion thus estimated.

[0087] Returning to Fig. 4, notification control unit 215 controls display device 31 and audio output device 32 so that information is notified in the notification mode set by notification mode setting unit 213. Fig. 10 is a functional block diagram showing the function of notification control unit 215 in further detail. As shown in Fig. 10, notification control unit 215 includes visual control unit 2151, notification content control unit 2152, audio control unit 2153, and notification time control unit 2154.

[0088] The visual control unit 2151 controls the display device 31 to perform display in the mode set by the notification mode setting unit 213. In response to this, the display device 31 displays the notification information in one of a "dark tone", a "normal tone", and a "bright tone".

[0089] The notification content control unit 2152 controls the display device 31 to perform display in the mode set by the notification mode setting unit 213. The notification content control unit 2152 also controls the audio output device 32 to output audio in the mode set by the notification mode setting unit 213. In response to these controls, the display device 31 displays notification information in any one of a "commanding tone", a "normal tone", and a "mild tone". The audio output device 32 also outputs notification information in any one of a "commanding tone", a "normal tone", and a "mild tone".

[0090] The audio control unit 2153 controls the audio output device 32 to output audio in the mode set by the notification mode setting unit 213. In response to this, the audio output device 32 outputs the notification information in one of a "dark tone", a "normal tone", and a "bright tone".

[0091] The notification time control unit 2154 controls the units 2151 to 2153 so that the display device 31 performs display and the audio output device 32 performs audio output in the mode set by the notification mode setting unit 213. In response to this, the display device 31 and the audio output device 32 output notification information at the set notification timing and notification time length.

[0092] <3. Control processing by the information notification control device> Next, the control process (control method) executed by the information notification control device 2 will be described in detail.

[0093] [3-1. Data table generation process] First, the process of generating the weighting data table 221 executed by the information notification control device 2 will be described.

[0094] 11 is a flowchart illustrating the flow of a process for generating the weighting data table 221. This flowchart shows the technical contents of a computer program that causes a computer device to realize the data table generation process. The computer program is stored in various computer-readable non-volatile recording media and is provided (sold, distributed, etc.). The computer program may be composed of only one program, or may be composed of multiple programs that work together.

[0095] The process shown in Fig. 11 can be started when the vehicle starts and various controls for notifying information become executable. The process shown in Fig. 11 is started, for example, when the user (driver) performs an operation to instruct the start of generation of the weighting data table 221. The user issues an instruction to generate the data table by, for example, a touch panel operation, a button operation, or a voice operation.

[0096] In step S11, the controller 21 sets the variable X to "1." As will be understood from the following explanation, weighting data is determined for each item (such as the items "visual" and "audio" in FIG. 6, etc.). The variable X is a variable provided in response to this, and is a variable that changes each time the item changes. When the variable X is set to "1," the process proceeds to the next step S12.

[0097] In step S12, the controller 21 (notification mode setting unit 213) starts a process of determining weighting data for the Xth item. In this embodiment, the items "visual", "audio", "notification contents", "notification timing", and "notification time length" are the targets for which weighting data is determined. A processing order is determined for these in advance. For example, the first item is "visual", the second item is "audio", the third item is "notification contents", the fourth item is "notification timing", and the fifth item is "notification time length". For example, when the variable X=1, a process of determining weighting data for the first item "visual" is started. When the variable X=2, a process of determining weighting data for the second item "audio" is started. With the start of the process of determining weighting data, the process proceeds to the next step S12.

[0098] In step S13, the controller 21 (notification mode setting unit 213) sets the variable Y to "1". As described above, there are multiple selectable notification modes for each item. In this embodiment, for example, the item "visual" has three types: "dark tone", "normal tone", and "bright tone". The variable Y is a variable provided in response to this, and is a variable that changes each time the notification mode setting changes. When the variable Y is set to "1", the process proceeds to the next step S14.

[0099] In step S14, the controller 21 (notification mode setting unit 213) decides to perform notification in the Yth notification mode. Here, as an example, it is assumed that the variable X=1. That is, it is assumed that the target of the process for determining the weighting data is the first item "visual". In this case, the controller 21 decides to perform notification in the Yth notification mode for the first item "visual". In this embodiment, for the first item "visual", any one of "dark tone", "normal tone", and "bright tone" can be set as the notification mode. A processing order (setting order) is determined in advance for these. For example, it is determined that the first notification mode is "dark tone", the second notification mode is "normal tone", and the third item is "bright tone". For example, when the variable Y=1, the first notification mode, "dark tone", is set for "visual". When the variable Y=2, the second notification mode, "normal tone", is set for "visual".

[0100] In order to perform notification using the display device 31 and the audio output device 32, it is necessary to set the notification mode not only for the item "visual" but also for other items such as "audio". For these other items, the same notification mode is set regardless of the value of the variable Y, since the purpose is to determine the weighting data for the item "visual" and to eliminate the influence of fluctuations in other items. For example, for the other items, the second notification mode is set regardless of the value of the variable Y. Specifically, for the item "visual", when the notification mode is set to "dark tone" with the variable Y=1, when the notification mode is set to "normal tone" with the variable Y=2, and when the notification mode is set to "bright tone" with the variable Y=3, the other items are as follows. The item "audio" is set to the second notification mode of "normal tone", the item "notification content" is set to the second notification mode of "normal tone", the item "notification timing" is set to the second notification mode of "normal", and the item "notification duration" is set to the second notification mode of "normal". When the notification mode is determined, the process proceeds to the next step S15.

[0101] In step S15, the controller 21 (notification control unit 215) controls the information notification device 3 (the display device 31 and the audio output device 32) to perform notification in the notification mode determined in step S14. The information to be notified is acquired by the information acquisition unit 212 requesting the information source device 1. When the process of having the information notification device 3 perform the information notification is completed, the process proceeds to the next step S16. The information notification device 3 notifies the user of the information in the notification mode determined in step S14 under the control of the notification control unit 215. The notification performed here is a notification for setting weighting data. For this reason, the notification performed here is not a notification in the usual sense, and has no substantial meaning as a notification. It is preferable to notify the user that the notification is to be output for setting weighting data, for example, at the timing of step S1 or in conjunction with the notification of step S15.

[0102] In step S16, the feeling estimation device 23 estimates the user's feeling immediately after the information notification device 3 notifies the user of the information in response to the process of step S15. The feeling estimation device 23 outputs a result of the feeling estimation to the controller 21. After the feeling estimation and the estimation result are output, the process proceeds to the next step S17.

[0103] In step S17, the controller 21 (emotion acquisition unit 211) acquires the emotion estimation result by the emotion estimation device 23. Once the emotion estimation result is acquired, the process proceeds to the next step S18.

[0104] In step S18, the controller 21 (weighting processing unit 214) determines the value of weighting data for the notification mode of the information in step S15 (specifically, the Yth notification mode of the Xth item). For example, when the variable X=1 and the variable Y=1, the value of the weighting data for "dark tone" of the item "visual" is determined. For example, when the variable X=1 and the variable Y=2, the value of the weighting data for "normal tone" of the item "visual" is determined. For example, when the variable X=2 and the variable Y=2, the value of the weighting data for "normal tone" of the item "audio" is determined. The determined weighting data is stored in the memory 22.

[0105] The method of determining the value of the weighting data is as described above with reference to FIG. 9. For example, when the variable X=1 and the variable Y=1, and the user's emotion is relatively close to each axis of the second quadrant of the psychological plane table 234 and is estimated to be "slightly negative", the weighting data of "dark tone" of the item "visual" is set to "-1". Also, when the variable X=1 and the variable Y=2, and the emotion is in the neutral region 234a of the psychological plane table 234 and it cannot be estimated whether it is positive or negative, the weighting data of "normal tone" of the item "visual" is set to "0". Also, when the variable X=1 and the variable Y=3, and the user's emotion is relatively close to each axis of the third quadrant of the psychological plane table 234 and is estimated to be "slightly positive", the weighting data of "bright tone" of the item "visual" is set to "+1". When the value of the weighting data is determined, the process proceeds to the next step S19.

[0106] In step S19, the controller 21 (notification mode setting unit 213) performs a process of adding "1" to the variable Y. When this process is completed, the process proceeds to the next step S20.

[0107] In step S20, the controller 21 (notification mode setting unit 213) judges whether or not the variable Y exceeds the total number of selectable notification modes prepared for each item (three in this embodiment). If the variable Y exceeds the total number of selectable notification modes prepared for each item (Yes in step S20), the process proceeds to the next step S21. On the other hand, if the variable Y is equal to or less than the total number of selectable notification modes prepared for each item (No in step S20), the process returns to step S14, and the processes from step S14 onwards are repeated.

[0108] In step S21, the controller 21 (notification mode setting unit 213) performs a process of adding "1" to the variable X. When this process is completed, the process proceeds to the next step S22.

[0109] In step S22, the controller 21 (notification mode setting unit 213) judges whether or not the variable X exceeds the number of prepared items (five in this embodiment). If the variable X exceeds the number of prepared items (Yes in step S22), the process flow shown in Fig. 11 ends. On the other hand, if the variable X is equal to or less than the number of prepared items (No in step S22), the process returns to step S12, and the processes from step S12 onwards are repeated.

[0110] 11, a weighting data table 221 such as that shown in FIG. 6 is obtained. In this embodiment, the weighting data table 221 is generated for each user and for each type of information (each piece of information or each information group). That is, the process shown in FIG. 11 is executed for each type of information for each user, and multiple types of weighting data tables 221 are generated.

[0111] As can be seen from the above, in the method for generating the weighting data table 221 of this embodiment, information notification is sequentially executed while changing the notification mode of information. In the generation method, the user's emotion for the notification mode is estimated, and weighting data indicating the change in emotion is assigned for each notification mode according to the estimation result of the emotion to generate a weighting data table.

[0112] This makes it possible to create a data table showing the tendency of changes in the user's emotions for each notification mode when notifying the user of information. By using such a data table, it is possible to notify the user of information in a mode that matches their preferences, and to quickly update the table taking into account the tendency of changes in the user's emotions.

[0113] [3-2. Information notification control processing] Next, an information notification control process using the weighting data table 221 executed by the information notification control device 2 will be described.

[0114] 12 is a flowchart illustrating the flow of information notification control processing using the weighting data table 221. This flowchart shows the technical contents of a computer program that causes a computer device to realize the information notification control processing. The computer program is stored in various computer-readable non-volatile recording media and provided (sold, distributed, etc.). The computer program may be composed of only one program, or may be composed of multiple programs that work together.

[0115] The process shown in Fig. 12 can be started when the vehicle starts and various controls for notifying information can be executed. The process shown in Fig. 12 may be executed from the start-ready time point. The process shown in Fig. 12 may be executed at an appropriate timing after the start-ready time point, that is, at a timing when information notification control is required, or during a time period when information notification control is required.

[0116] Furthermore, when starting the process shown in Fig. 12, it is necessary to use the weighting data table 221 created for the user to be notified of information, among the weighting data tables 221 stored for each user in the memory 22 (set the weighting data table 221 created for the user as the table to be used). That is, when starting the process shown in Fig. 12, it is necessary to identify the user. The user may be identified, for example, by face authentication using an image captured by a camera C (see Fig. 2) that captures an image of the user. Alternatively, the user may be identified by biometric authentication using a fingerprint or the like, or by the user inputting data that identifies the user.

[0117] In step S31, the controller 21 (information acquisition unit 212) monitors whether or not information to be notified to the user has been acquired. If the information acquisition unit 212 has acquired information (Yes in step S31), the process proceeds to the next step S32. If the information acquisition unit 212 has not acquired information (No in step S31), the process of step S31 is repeated.

[0118] In step S32, the controller 21 (notification mode setting unit 213) sets the notification mode of the information. In setting the notification mode of the information, first, the table usage specification table 222 shown in Fig. 13 is used. The table usage specification table 222 is stored in the memory 22 as shown in Fig. 1.

[0119] Fig. 13 is a diagram showing the configuration of the table-usage specification table 222. As shown in Fig. 13, the table-usage specification table 222 is a two-dimensional matrix table with the user ID as a vertical axis parameter and the information ID as a horizontal axis parameter. The user ID is identification code information assigned to each user. The information ID is identification code information assigned to each type of information (each piece of information or each information group) to be notified to the user.

[0120] As described above, in this embodiment, the weighting data table 221 is generated for each user for each type of information. For this purpose, the notification mode setting unit 213 specifies the weighting data table 221 to be used in the table-used specification table 222 when setting the notification mode for notifying information. For example, when the user ID is "US01" and the information ID is "IN02", the controller 21 specifies the weighting data table 221 whose weighting data table ID is "WT01-02". The weighting data table ID is identification code information assigned to each weighting data table 221.

[0121] When the notification mode setting unit 213 specifies the identification information of the weighting data table 221 to be used, it reads out the weighting data table 221 having the identification information from the memory 22. Then, the notification mode setting unit 213 sets the notification mode of the information using the read weighting data table 221. In detail, the notification mode setting unit 213 selects the mode with the largest weighting data value for each item (such as "visual" or "audio") and sets it as the mode to be notified. For example, when the weighting data table 221 shown in FIG. 6 is read out, the items "visual" and "audio" are set to "bright tone". Also, the item "notification content" is set to "run with plenty of time". Also, the item "notification timing" is set to "late". Also, the item "notification time length" is set to "shorter". Returning to FIG. 12, when the notification mode is set, the process proceeds to the next step S33.

[0122] In step S33, the controller 21 (notification control unit 215) controls the information notification device 3 (the display device 31 and the audio output device 32) to perform notification in the notification mode determined in step S32. When the process of causing the information notification device 3 to notify the information is completed, the process proceeds to the next step S34. Note that under the control of the notification control unit 215, the information notification device 3 notifies the user of the information in the notification mode determined in step S32.

[0123] In step S34, the feeling estimation device 23 estimates the user's feeling immediately after the information notification device 3 notifies the user of the information in response to the process of step S33. The feeling estimation device 23 outputs a result of the feeling estimation to the controller 21. After the feeling estimation and the estimation result are output, the process proceeds to the next step S35.

[0124] In step S35, the controller 21 (emotion acquisition unit 211) acquires the emotion estimation result by the emotion estimation device 23. Once the emotion estimation result is acquired, the process proceeds to the next step S36.

[0125] In step S36, the controller 21 (notification mode setting unit 213) judges whether the user's estimated emotion is bad or not. When it is estimated from the estimated emotion that the user is dissatisfied with the notification mode, the user's emotion is judged to be bad. For example, when the estimated emotion is "melancholy" or "anxiety, fear, unpleasantness", the user's estimated emotion is judged to be bad. When it is judged that the user's estimated emotion is bad (Yes in step S36), the process proceeds to step S37. On the other hand, when it is judged that the user's estimated emotion is not bad (No in step S36), the process proceeds to step S38.

[0126] In addition, cases where the estimated emotion of the user is determined to be not bad include cases where the estimated emotion of the user is estimated to be good and cases where the user emotion is indefinite. As can be seen from the above explanation, the weighting data table 221 is a table generated so that a mode in which the user emotion is good can be selected. Then, in step S32, a notification mode is set so that the user emotion is good based on the table. For this reason, in the process of step S36, the probability that the estimated emotion of the user is determined to be bad is low.

[0127] In step S37, the controller 21 (notification mode setting unit 213) decides to change the notification mode of the information. When the change in the notification mode of the information is decided, the process returns to step S31, and the processes from step S31 onwards are carried out. However, since the change in the notification mode of the information has been decided, the process of step S32 is carried out in a partially different manner from that described above. In order to determine whether to carry out this partially different process, for example, a process may be carried out in the process of step S37 to set a flag indicating that the process will be carried out for the notification of the type of information (the flag is released in step S39).

[0128] If it has been decided that the information notification mode should be changed and the type of information is the same, the information notification mode is changed. However, if the type of information to be notified has changed, the weighting data table 221 will be different, and it is determined that the notification mode should not be changed. In this case, the process in step S32 is the same as that described above. Note that, when notifying information of a type for which it has been decided that the information notification mode should be changed thereafter, the change in the information notification mode is continuously performed for the notification of that type of information.

[0129] When the information notification mode is changed, rather than selecting the mode with the largest weighting data value for each item, for example, the following selection is made: for one of the items, the mode with the second largest weighting data value is selected, and for the remaining items, the mode with the largest weighting data value is selected. The item for which the second largest mode is selected may be determined, for example, according to a predetermined order.

[0130] For example, the predetermined order is assumed to be "visual", "audio", "notification contents", "notification timing", and "notification time length". In this case, if the estimated emotion of the user is bad in the pattern of selecting the mode with the largest weighting data value for all modes, the mode with the second largest weighting data value for "visual", which is the first in the predetermined order, is selected. Then, the mode with the largest weighting data value for the remaining items is selected. If the emotion of the user is bad in this notification mode in step S36 and the notification mode is changed, the mode with the second largest weighting data value for "audio", which is the second in the predetermined order, is selected. Then, the mode with the largest weighting data value for the remaining items is selected. If the emotion of the user is bad in this notification mode in step S36 and the notification mode is changed, the mode with the second largest weighting data value for "notification contents", which is the third in the predetermined order, is selected. Then, the mode with the largest weighting data value for the remaining items is selected.

[0131] Even if the last item is selected in the above-mentioned order system, the user's emotion may still remain bad. In such a case, for example, the mode with the third largest weighting data value may be selected for one of the items, and the mode with the largest weighting data value may be selected for the remaining items, in the above-mentioned order system or the like. However, the weighting data table 221 is created while observing the change tendency of the user's emotion with respect to the notification mode, and if the weighting data value is selected, the user's emotion should basically be good. For this reason, if the user's emotion cannot be made good even after the above-mentioned notification mode change, such as selecting the mode with the second largest weighting data value for one of the items, is performed, the user's preference may have changed significantly. If it is estimated that the user's preference has changed significantly in this way, the user may be notified to re-create the weighting data table 221.

[0132] In the above, the item for selecting the second largest aspect is selected in a preset order, but this is merely an example. For example, the item may be selected in order of the likelihood that a change in the selection aspect will cause a large change in the user's emotion based on the tendency of changes in the weighting data value. For example, in the example shown in FIG. 6, the weighting data for "notification timing" and "notification duration" change significantly with the change in aspect. For this reason, the item for selecting the second largest aspect may be selected with priority.

[0133] In step S38, the controller 21 (weighting processing unit 214) determines whether or not to update the weighting data table 221. When the user's emotion changes from bad to good by changing the notification mode, the weighting processing unit 214 determines to update the weighting data table 221. When the notification mode has not been changed and when the user's emotion is indefinite, it is determined not to update the weighting data table 221. When it is determined to update the weighting data table 221 (Yes in step S38), the process proceeds to step S39. On the other hand, when it is determined not to update the weighting data table 221 (No in step S38), the process proceeds to step S40.

[0134] In step S39, the controller 21 (weighting processing unit 214) updates the weighting data table 221. The weighting processing unit 214 changes the weighting data for an item in which the user's emotion has changed due to a change in the notification mode. For example, for a notification mode in which the user's emotion has become good for an item to be changed, the weighting data value is determined according to the estimated emotion of the user. Then, for a notification mode in which the user's emotion was previously bad for an item to be changed, the weighting data value is determined according to the previous estimated emotion of the user.

[0135] Here, a specific example will be described with reference to FIG. 6 and FIG. 15. FIG. 15 is a schematic diagram showing a state in which the weighting data table 221 in FIG. 6 has been updated. For example, suppose that the user's emotion changes from bad (small negative) to good (large positive) by changing the notification mode for the item "visual" from "bright tone" to "normal tone". In this case, the weighting data for the mode "normal tone" is changed from "0" to "+2" (weighting is added by a predetermined value). The weighting data for the mode "bright tone" is changed from "+1" to "-1" (weighting is subtracted by a predetermined value). The weighting data may be changed in a different manner, and the value of the weighting data may be increased or decreased by a predetermined rank. In addition, in the updated table shown in FIG. 15, the display order of the display modes determined by the magnitude of the weighting data value is also changed in accordance with the change in the value of the weighting data (the display order of "normal tone" and "bright tone" is reversed). This is an example, and the value of the weighting data may simply be changed. Once the weighting data table 221 has been updated, the process proceeds to step S40.

[0136] In step S40, it is confirmed whether the information notification process is to be ended. The information notification process is ended, for example, by a command from the user, the occurrence of an error, a command to turn off the vehicle power, etc. If the information notification process is to be ended (Yes in step S40), the process shown in Fig. 12 is ended. If the information notification process is not to be ended (No in step S40), the process returns to step S31, and the processes from step S31 onwards are performed.

[0137] As can be seen from the above, the information notification control device 2 of this embodiment estimates a notification mode that will make the emotion good based on the emotion estimation result and weighting data, and notifies information in the estimated notification mode.

[0138] In such a configuration, a notification mode that makes the user's emotions good is estimated based on the emotions of the user who has been notified of the information (i.e., who has been exposed to a certain notification mode) and the user's emotion change tendency with respect to the notification mode that has been acquired in advance. This makes it possible to set the notification mode that the user prefers while automatically changing the notification mode. Since the notification mode is changed according to the emotion change tendency of the user with respect to the notification mode that is acquired in advance, the probability of unnecessary setting changes is reduced, and the notification mode that the user prefers can be set efficiently.

[0139] As can be seen from the above, the information notification control device 2 of the present embodiment updates the weighting data according to the estimation result of the emotion after the information notification in the estimated notification mode. In detail, the weighting data is updated when the emotion becomes better as a result of the change in the notification mode compared to before the change in the notification mode.

[0140] In addition, the method of updating the weighting data is not limited to the specific method described above, and for example, a method of increasing or decreasing the weighting data for the notification mode used in the notification depending on the estimated emotion after the information is notified (good emotion: adding a predetermined value, bad emotion: subtracting a predetermined value) can also be applied.

[0141] According to these configurations, even if the user's preferences change, the notification mode can be automatically set to match the user's preferences. In other words, the effort required of the user to set the notification mode of information can be reduced as much as possible.

[0142] As can be seen from the above, the information notification control device 2 of this embodiment maintains the current notification mode when the emotion is estimated to be good based on the emotion estimation result, and repeats changing the notification mode until the emotion is estimated to be good based on the emotion estimation decision. With this configuration, it is basically possible to notify information in a notification mode that matches the user's preferences, and when the user's preferences change, it is possible to automatically find a notification mode that matches the user's preferences. Although the information notification control device 2 of this embodiment has been described as an example of visual (display) and audio notification, it can also be applied to notification by vibration, etc. In the case of notification by vibration, the notification mode can be vibration intensity, vibration frequency, and vibration pattern (for example, vibration on / off / pause).

[0143] <5. Things to keep in mind> Various technical features disclosed in the description of the invention may be modified in various ways without departing from the spirit of the technical creation. Furthermore, the embodiments and modifications disclosed in the description of the invention may be combined to the extent possible. [Explanation of symbols]

[0144] 1... Information source device 2. Information notification control device 3...Information notification device 22. Memory 23... Emotion estimation device 31...Display device (information notification device) 32 Audio output device 100 Information notification system 221...Weighting data table

Claims

1. A memory is provided for storing weighting data in association with a notification mode of information, notifying information in a notification manner determined based on the weighting data; Estimating the emotion of a user who has been notified of the information; The information notification control device updates the weighting data based on the emotion estimation result.

2. The information notification control device according to claim 1 , wherein the weighting data is updated when the emotion becomes better as a result of the change in the notification mode compared to before the change in the notification mode.

3. maintain the current notification mode when the emotion is estimated to be good based on the emotion estimation result; The information notification control device according to claim 1 , wherein, when the emotion is estimated to be not good by the emotion estimation determination, the change of the notification mode is repeated until the emotion is estimated to be good.

4. 4. The information notification control device according to claim 1, wherein a plurality of weighting data tables, each of which is a table of the weighting data, are stored in the memory according to a type of information, and a selected weighting data table is selected according to a type of information to be notified.

5. 4. The information notification control device according to claim 1, wherein a weighting data table in which the weighting data is compiled is stored in the memory for each user and is selected according to a user to be notified.

6. The information notification control device according to claim 1 , wherein the notification modes include a notification mode relating to a sound and a notification mode relating to a display, and the weighting data is set for each notification mode.

7. The information notification control device according to claim 1 , wherein the notification mode includes a notification mode relating to a notification time.

8. Notifying the information in a notification manner determined based on weighting data associated with the notification manner of the information; Estimating the emotion of a user who has been notified of the information; and updating the weighting data based on the emotion estimation result.

9. Notifying the information in a notification manner determined based on weighting data associated with the notification manner of the information; Estimating an emotion of a user who has been notified of the information; and updating the weighting data based on the emotion estimation result; An information notification control program that causes a computer to execute the above.

10. An information source device that transmits information to be notified to a driver; an information notification control device to which the information is input from the information source device; an information notification device that is controlled by the information notification control device and notifies the information; An in-vehicle information notification system comprising: The information notification control device includes: a memory that stores weighting data that associates the notification mode of the information with a change in the driver's emotion; a feeling estimation device that estimates the feeling; Equipped with notifying information in a notification manner determined based on the weighting data; an in-vehicle information notification system that updates the weighting data based on a feeling estimation result of a user who has been notified of the information, the feeling estimation device.

11. A method for generating a data table used when determining a notification mode of information based on weighting data corresponding to the notification mode, comprising: Sequentially notifying the information while changing the notification mode of the information; Estimating a user's emotion for each of the notification modes; a weighting data table generating method for generating a weighting data table by assigning weighting data indicating a change in emotion for each notification mode according to the emotion estimation result.

12. A program for generating a data table used when determining a notification mode of information based on weighting data corresponding to the notification mode, Sequentially notifying the information while changing the notification mode of the information; Estimating a user's emotion for each of the notification modes; and generating a weighting data table by assigning weighting data indicating a change in emotion for each notification mode according to the emotion estimation result; A data table generating program that causes a computer to execute the above.

Images