Emotion estimation device, on-vehicle emotion estimation system, emotion estimation method and emotion estimation program

Abstract

To improve the estimation accuracy of an emotion.SOLUTION: An emotion estimation device for deriving an emotion estimation result (Pest) by estimating an emotion of an object person estimates emotion candidates to be candidates for the emotion estimation result on the basis of a plurality of emotion indices in the object person, and determines a scene for the object person on the basis of scene determination information. Then, it determines the emotion estimation result (Pest) on the basis of an estimation result of the motion candidates and a determination result (Sdet) of the scene.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to an emotion estimation device, an in-vehicle emotion estimation system, an emotion estimation method, and an emotion estimation program. [Background technology]

[0002] A method for estimating a person's emotion based on biometric information has been proposed (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-63324 A Summary of the Invention [Problem to be solved by the invention]

[0004] It is possible to control various devices using emotion estimation results. However, it is difficult to sufficiently improve the estimation accuracy using emotion estimation methods based on biometric information. There is a demand for improving emotion estimation accuracy.

[0005] An object of the present invention is to provide a technique that contributes to improving the accuracy of emotion estimation. [Means for solving the problem]

[0006] An emotion estimation device according to the present invention is an emotion estimation device that derives an emotion estimation result by estimating the emotion of a subject, estimates emotion candidates that are candidates for the emotion estimation result based on a plurality of emotion indicators of the subject, determines a scene for the subject based on scene determination information, and decides the emotion estimation result based on the estimation result of the emotion candidates and the determination result of the scene. Effect of the Invention

[0007] According to the present invention, it is possible to provide a technique that contributes to improving the emotion estimation accuracy. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram illustrating a relationship between a subject of emotion estimation and a vehicle in the first embodiment of the present invention. [Diagram 2] 1 is a schematic block diagram of an in-vehicle system according to a first embodiment of the present invention. [Diagram 3] 1 is an internal block diagram of a biosensor according to a first embodiment of the present invention. [Figure 4] FIG. 2 is a diagram showing an example of a two-dimensional model related to emotion estimation in the first embodiment of the present invention. [Diagram 5] FIG. 11 is a diagram showing another example of the two-dimensional model related to emotion estimation in the first embodiment of the present invention. [Figure 6] FIG. 1 is an internal block diagram of a feeling estimation device according to a first embodiment of the present invention. [Figure 7] FIG. 2 is a structural diagram of an emotion determination table according to the first embodiment of the present invention. [Figure 8] 1 is an internal block diagram of a vehicle control device according to a first embodiment of the present invention. [Figure 9] FIG. 2 is an internal block diagram of a vehicle sensor unit according to the first embodiment of the present invention. [Figure 10] 4 is a flowchart of emotion estimation processing according to the first embodiment of the present invention. [Figure 11] FIG. 11 is a structural diagram of a table for emotion estimation according to example EX_1a belonging to the first embodiment of the present invention. [Figure 12] FIG. 11 is a structural diagram of a table for emotion estimation according to example EX_1b belonging to the first embodiment of the present invention. [Figure 13] 11 is a flowchart of an algorithm improvement operation according to an example EX_2a belonging to the first embodiment of the present invention. [Figure 14] FIG. 11 is a diagram illustrating an input / output relationship of a biological information estimation model according to a second embodiment of the present invention. [Figure 15]FIG. 11 is a schematic configuration diagram of a learning device according to a second embodiment of the present invention. [Figure 16] FIG. 11 is a diagram illustrating an input / output relationship of an emotion index estimation model in a third embodiment of the present invention. [Figure 17] FIG. 11 is a schematic configuration diagram of a learning device according to a third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, examples of embodiments of the present invention will be described in detail with reference to the drawings. In each of the drawings, the same parts are given the same reference numerals, and duplicated descriptions of the same parts are omitted as a rule. In this specification, for the sake of simplicity, by writing a symbol or code referring to information, signal, physical quantity, functional part, circuit, element, or part, the name of the information, signal, physical quantity, functional part, circuit, element, or part corresponding to the symbol or code may be omitted or abbreviated.

[0010] <<First embodiment>> A first embodiment of the present invention will be described. In the first embodiment, a technology for estimating a person's emotion will be described. FIG. 1 shows the relationship between a vehicle V1 and a person U1 assumed in the first embodiment. The person U1 is a target of emotion estimation, and is hereinafter referred to as a target person U1. The vehicle V1 is any type of vehicle. Here, the vehicle V1 is assumed to be an automobile or the like that runs on a road surface. The target person U1 is an occupant of the vehicle V1. In this embodiment, it is assumed that the target person U1 is the driver of the vehicle V1. Hereinafter, when simply referring to a driver, this refers to the driver of the vehicle V1 (hence the target person U1). However, the target person U1 may be an occupant other than the driver (i.e. a passenger of the vehicle V1). In accordance with the person U1 being referred to as a target person U1, the vehicle V1 may be referred to as a target vehicle V1. An in-vehicle system SYS is mounted on the vehicle V1.

[0011] 2 shows a schematic block diagram of the in-vehicle system SYS. The in-vehicle system SYS includes an emotion estimation device 10, a vehicle control device 20, an actuator unit 30, a vehicle sensor unit 40, a biosensor 60, an exterior camera 71, an interior camera 72, a display unit 73, a speaker 74, and a microphone 75. The components of the in-vehicle system SYS can transmit and receive any signal and information to and from each other through an in-vehicle network formed in the vehicle V1. The in-vehicle network includes, for example, a CAN (Controller Area Network) and an AVCLAN (Audio Visual Communication Local Area Network).

[0012] The emotion estimation device 10 has an emotion estimation model 121, and estimates the emotion of the subject U1 using the emotion estimation model 121. The vehicle control device 20 controls the running of the vehicle V1 using an actuator unit 30. The actuator unit 30 has various driving parts constituted by a motor and the like that realize the running of the vehicle V1. The actuator unit 30 includes an engine and a motor that generate a driving force for the vehicle V1, a steering actuator that drives the steering of the vehicle V1, a brake actuator that drives the brakes of the vehicle V1, and the like.

[0013] The vehicle sensor unit 40 has a sensor that detects the content of the driving operation of the vehicle V1 by the driver of the vehicle V1 and a sensor that detects various states of the vehicle V1. The vehicle sensor unit 40 outputs vehicle sensor information having the detection results. The vehicle control device 20 realizes driving control of the vehicle V1 by driving and controlling the actuator unit 30 according to the vehicle sensor information. At this time, the vehicle control device 20 can also perform driving control according to the emotion estimation result. The biosensor 60 detects bioinformation of the subject U1. The emotion of the subject U1 is estimated based on the detection result of the bioinformation by the biosensor 60. The emotion estimation device 10, the vehicle control device 20, the vehicle sensor unit 40, and the biosensor 60 will be described in detail later.

[0014] The exterior camera 71 is composed of one or more cameras that capture images of the outside of the vehicle V1. The exterior camera 71 has a capture area set outside the vehicle V1, and generates an exterior camera image by capturing images of the scene within the capture area. The exterior camera image is an image captured in the capture area of ​​the exterior camera 71. Image information showing the exterior camera image is referred to as exterior image information. The exterior camera 71 captures images at a predetermined frame rate.

[0015] The vehicle V1 is assumed to travel in a forward direction. A vehicle located in front of the vehicle V1 is referred to as a forward vehicle, and a vehicle located behind the vehicle V1 is referred to as a rearward vehicle. Although it depends on the distance between the vehicle V1 and the forward vehicle, it is assumed here that the forward vehicle is located within the shooting area of ​​the exterior camera 71, and therefore the image of the forward vehicle is included in the exterior camera image. Although it depends on the distance between the vehicle V1 and the rearward vehicle, it is assumed here that the rearward vehicle is located within the shooting area of ​​the exterior camera 71, and therefore the image of the rearward vehicle is included in the exterior camera image. The exterior camera image may be composed of a photographed image of the front area of ​​the vehicle V1, a photographed image of the rear area of ​​the vehicle V1, a photographed image of the right side area of ​​the vehicle V1, and a photographed image of the left side area of ​​the vehicle V1. Alternatively, a wide-angle image including information on these four photographed images may be the exterior camera image. In addition, the configuration of the exterior camera image is arbitrary.

[0016] The in-vehicle camera 72 is composed of one or more cameras that capture images of the interior of the vehicle V1. The in-vehicle camera 72 has a shooting area set inside the vehicle V1 (i.e., the interior of the vehicle V1), and generates an in-vehicle camera image by capturing images of the shooting area. The in-vehicle camera image is an image captured in the shooting area by the in-vehicle camera 72. Image information showing the in-vehicle camera image is referred to as in-vehicle image information. The in-vehicle camera 72 captures images at a predetermined frame rate.

[0017] The subject U1 is located within the shooting area of ​​the in-vehicle camera 72, and therefore the image of the subject U1 is included in the in-vehicle camera image. In particular, it is assumed here that the face of the subject U1 is included within the shooting area of ​​the in-vehicle camera 72, and therefore the in-vehicle camera image includes a facial image of the subject U1 (a captured image of the face of the subject U1).

[0018] The display unit 73 is a display device having a liquid crystal display panel or the like, and displays any image under the control of the emotion estimation device 10, the vehicle control device 20, or a display control device not shown. The display unit 73 is installed at an appropriate location in the cabin of the vehicle V1 so that each occupant of the vehicle V1 can visually recognize the display content of the display unit 73. A plurality of display units 73 may be installed in the cabin of the vehicle V1. The display unit 73 may be a component of a car navigation system installed in the vehicle V1. The car navigation system may be included in the in-vehicle system SYS.

[0019] The speaker 74 outputs any sound (message, music, etc.) under the control of the emotion estimation device 10, the vehicle control device 20, or an audio device (not shown). The speaker 74 is installed at an appropriate position in the cabin of the vehicle V1 so that each occupant of the vehicle V1 can hear the output sound of the speaker 74. A plurality of speakers 74 may be installed in the cabin of the vehicle V1.

[0020] The microphone 75 converts ambient sounds around the microphone 75 into an electric audio signal. The audio signal obtained by the conversion of the microphone 75 is called a microphone signal. In the in-vehicle system SYS, the microphone signal is transmitted to the emotion estimation device 10 and the vehicle control device 20. The microphone 75 is installed at an appropriate location in the cabin of the vehicle V1 so that the signal components of the speech sounds of each occupant of the vehicle V1 are included in the microphone signal (i.e., so that the speech sounds are included in the contents of the microphone 75). A plurality of microphones 75 may be installed in the cabin of the vehicle V1. Hereinafter, the speech sounds picked up by the microphone 75 are assumed to be the speech sounds of the target person U1.

[0021] The feeling estimation device 10 estimates the feeling of the subject U1 based on the subject U1's biometric information. A biometric sensor 60 is attached to the subject U1 to acquire the biometric information of the subject U1. FIG. 3 shows the configuration of the biometric sensor 60. The biometric sensor 60 includes at least an electroencephalogram sensor 61 and a heartbeat sensor 62, and the electroencephalogram sensor 61 and the heartbeat sensor 62 are attached to the subject U1. The electroencephalogram sensor 61 detects the electroencephalogram of the subject U1 and outputs electroencephalogram data indicating the detection result of the electroencephalogram. That is, the electroencephalogram data includes information on the detected electroencephalogram. The heartbeat sensor 62 detects the heartbeat of the subject U1 and outputs heartbeat data indicating the detection result of the heartbeat. That is, the heartbeat data includes information on the detected heartbeat. For example, a headgear-type electroencephalogram sensor is used as the electroencephalogram sensor 61. For example, a chest belt-type electrocardiogram-type heartbeat sensor is used as the heartbeat sensor 62. An optical heartbeat sensor can also be used as the heartbeat sensor 62.

[0022] The biosensor 60 further includes a sweat sensor 63 and a body temperature sensor 64, which are attached to the subject U1. The sweat sensor 63 detects the amount of sweat (amount of sweat per predetermined time) of the subject U1 and outputs sweat data indicating the detection result of the amount of sweat. The body temperature sensor 64 detects the body temperature of the subject U1 and outputs body temperature data indicating the detection result of the body temperature.

[0023] Data obtained from the biosensor 60 attached to the subject U1 is called biodata. The biodata includes brainwave data, heartbeat data, sweat data, and body temperature data, and represents the bioinformation of the subject U1. Depending on the type of bioinformation to be acquired or the wearability, other sensors may be included in the biosensor 60 and attached to the subject U1. Examples of other sensors include a blood pressure monitor or a NIRS (Near Infrared Spectroscopy) device. Note that it is not essential that the biosensor 60 is provided with the sweat sensor 63, and therefore the biodata may not include the sweat data. Similarly, it is not essential that the biosensor 60 is provided with the body temperature sensor 64, and therefore the biodata may not include the body temperature data.

[0024] The biometric data obtained by the biometric sensor 60 is provided to the emotion estimation device 10. The emotion estimation device 10 refers to the biometric data and estimates emotions based on two emotion indices that are indices indicating the mental and physical state of the subject U1. The mental and physical state of an arbitrary person includes a psychological state, and therefore each emotion indices can also be said to be an index indicating a psychological state. The value of the emotion indices is called an emotion index value. One of the emotion indices used in this embodiment is the arousal level of the central nervous system (hereinafter referred to as the arousal level), and the other emotion indices used in this embodiment is the activity level of the autonomic nervous system (hereinafter referred to as the activity level). That is, one emotion index value is expressed by the arousal level, and the other emotion index value is expressed by the activity level. The arousal level can be derived based on, for example, electroencephalogram data. The activity level can be derived based on, for example, heart rate data.

[0025] The emotion estimation model 121 provided in the emotion estimation device 10 is a model for emotion estimation based on arousal and activity, and may be configured with a calculation formula or a conversion data table that identifies an emotion from arousal and activity. However, as described later, the emotion estimation model 121 estimates one or more emotion candidates as candidates for the emotion of the subject U1, and the emotion estimation model 121 alone may not narrow down the emotion of the subject U1 to one.

[0026] The emotion estimation model 121 is a multi-dimensional model with arousal and activity as parameters, and here is a two-dimensional model with arousal and activity as two axes. The two-dimensional model is created based on medical evidence (papers, etc.) showing the relationship between arousal and activity and emotions. Alternatively, the two-dimensional model is created based on the results of a questionnaire given to many subjects. The results of the questionnaire given to the subjects include the arousal and activity of the subjects identified from the measured values ​​of the subjects' electroencephalograms and heartbeats, and the emotions declared by the subjects when the electroencephalograms and heartbeats were measured. Note that a multi-dimensional model with three or more dimensions may be used as the emotion estimation model 121.

[0027] Figure 4 shows an example of a two-dimensional model related to emotion estimation. Figure 4 shows a two-dimensional psychological plane PP defined in the two-dimensional model. According to various medical evidence related to psychology, psychology can be estimated based on two types of indices that indicate mental and physical states. In the psychological plane PP shown in Figure 4, a vertical axis corresponding to arousal level and a horizontal axis corresponding to activity level are defined.

[0028] Points indicating the awakening and activity of the subject U1 can be plotted (placed) on the psychological plane PP. The higher the awakening of the subject U1, the more the coordinates of the plotted (placed) points move in the positive direction of the vertical axis, and the lower the awakening of the subject U1, the more the coordinates of the plotted (placed) points move in the negative direction of the vertical axis. The higher the activity of the subject U1, the more the coordinates of the plotted (placed) points move in the positive direction of the horizontal axis, and the lower the activity of the subject U1, the more the coordinates of the plotted (placed) points move in the negative direction of the horizontal axis. From the origin of the psychological plane PP, the positive side of the vertical axis corresponds to an awakened state, and the negative side of the vertical axis corresponds to an unawakened state. From the origin of the psychological plane PP, the positive side of the horizontal axis corresponds to a state in which the sympathetic nerves are activated (a state in which a relatively strong emotion is present), and the negative side of the horizontal axis corresponds to a state in which the parasympathetic nerves are activated (a state in which a relatively weak emotion is present).

[0029] The point where the vertical axis and the horizontal axis intersect on the psychological plane PP is the origin. The psychological plane PP has first to fourth quadrants separated by the vertical axis and the horizontal axis. The first quadrant is an area where the vertical axis component has a positive value when viewed from the origin of the psychological plane PP, and the horizontal axis component also has a positive value. The second quadrant is an area where the vertical axis component has a positive value when viewed from the origin of the psychological plane PP, and the horizontal axis component also has a negative value. The third quadrant is an area where the vertical axis component has a negative value when viewed from the origin of the psychological plane PP, and the horizontal axis component also has a negative value. The fourth quadrant is an area where the vertical axis component has a negative value when viewed from the origin of the psychological plane PP, and the horizontal axis component has a positive value.

[0030] In the psychological plane PP, a corresponding psychological state (in other words, an emotion) is assigned to each quadrant. The first quadrant is assigned the psychological states of "fun, joy, anger, sadness". The first quadrant may further be assigned the psychological state of "moderate tension". The second quadrant is assigned the psychological state of "melancholy, crying". It can be said that "crying" in the second quadrant is one form of "sadness". The third quadrant is assigned the psychological states of "boredom, relaxation, calmness". The fourth quadrant is assigned the psychological states of "anxiety, fear, unpleasantness". The distance from the origin indicates the intensity of the corresponding psychological state. For example, when a point indicating the awakening level and activity level of the subject U1 is plotted in the second quadrant, the greater the distance between the plotted point and the origin, the stronger the psychological state of "melancholy, crying" of the subject U1.

[0031] In this way, the emotion of any person depends on the arousal level and activity level of the person. The position of each axis on the psychological plane PP may be appropriately set based on an experiment or the like. Each axis may be set based on the position of a neutral area described later.

[0032] The emotion estimation model 121 can estimate the psychological state of the subject U1 from the coordinates obtained by plotting (arranging) points indicating the arousal and activity of the subject U1 on the psychological plane PP. The psychological state and its intensity of the subject U1 can be estimated based on which quadrant of the psychological plane PP the coordinates of the plotted point are in, where in the quadrant they are located, and the distance between the plotted point and the origin. The first to fourth quadrants on the psychological plane PP can be considered as first to fourth classes, respectively, and in this case, it can be said that the emotion estimation model 121 classifies the psychological state of the subject U1 into one of the first to fourth classes. Note that when the emotion estimation model 121 is a multidimensional model of three or more dimensions, a multidimensional psychological space of three or more dimensions is used depending on the number of types of emotion index values ​​to be used.

[0033] Incidentally, when emotion intensity is strong, that is, when the emotion index value swings significantly toward the maximum or minimum value, emotion estimation accuracy tends to increase. On the other hand, when emotion intensity is weak, that is, when the emotion index value is near the median, emotion estimation accuracy tends to decrease. For this reason, as shown in Figure 5, a method can be adopted in which the area near the median of the emotion index value is set as a neutral area (corresponding to the diagonal line area in Figure 5), and emotion index values ​​that fall into the neutral area are judged to be unable to be estimated or to have no estimated emotion.

[0034] The neutral regions include an awakening neutral region NR1 and an activity neutral region NR2. In the psychological plane PP, the region whose distance from the horizontal axis is equal to or less than a predetermined first boundary distance is the awakening neutral region NR1, and the region whose distance from the vertical axis is equal to or less than a predetermined second boundary distance is the activity neutral region NR2. The awakening neutral region NR1 and the activity neutral region NR2 overlap each other in a region including the origin of the psychological plane PP. The combined region of the awakening neutral region NR1 and the activity neutral region NR2 is hereinafter referred to as a neutral region NR. It may be understood that each position in the neutral region NR does not belong to any of the first to fourth quadrants of the psychological plane PP.

[0035] [Emotion estimation device 10] FIG. 6 shows a configuration block diagram of the emotion estimation device 10. The emotion estimation device 10 includes a communication unit 11, a storage unit 12, and a controller 13. In the in-vehicle system SYS, the emotion estimation result by the emotion estimation device 10 can be used to control the vehicle. When the emotion estimation result is used to perform vehicle control that requires quick response, it is desirable to mount the emotion estimation device 10 on the vehicle V1 as in this example. However, the emotion estimation device 10 may be configured using a server device provided outside the vehicle V1. In this case, an in-vehicle block having a part of the emotion estimation device 10 is installed on the vehicle V1, while a server block having the remaining part of the emotion estimation device 10 is installed on the server device, and the emotion estimation device 10 is configured by the cooperation of the in-vehicle block and the server block. The in-vehicle block and the server block share any information by communication via a predetermined communication network.

[0036] For example, among the functional blocks 131 to 136 provided in the emotion estimation device 10, the functional blocks 132 and 136 are realized in the server block, and the other functional blocks 131 and 133 to 135 are realized in the in-vehicle block. Alternatively, among the functional blocks 131 to 136 provided in the emotion estimation device 10, the functional blocks 132 to 134 and 136 are realized in the server block, and the other functional blocks 131 and 135 are realized in the in-vehicle block. In addition, the allocation of the functional blocks to the server block and the in-vehicle block is arbitrary. The server device is composed of one or more computer devices connected to a communication network. In addition, when the emotion estimation device 10 is configured by the cooperation of the in-vehicle block and the server block, a part of the memory contents of the memory unit 12 necessary for realizing the operation of the server block is held in a memory unit (not shown) in the server device. That is, for example, when the function block 132 is provided in a server block, the emotion estimation model 121 and the psychological plane table 122 shown in FIG. 6 are held in a storage unit (not shown) in the server device.

[0037] The communication unit 11 transmits and receives any signal between the emotion estimation device 10 and a counterpart device different from the emotion estimation device 10. The counterpart device for the communication unit 11 is a component other than the emotion estimation device 10 among the components of the in-vehicle system SYS shown in FIG. 1. The communication unit 11 can communicate with the counterpart device via an in-vehicle network formed in the vehicle V1. The counterpart device for the communication unit 11 may further include an external device (such as a server device connected to the above-mentioned communication network) provided outside the vehicle V1. Note that the controller 13 can transmit and receive any information to and from the counterpart device (the counterpart device for the communication unit 11) using the communication unit 11, but hereinafter, a description of the communication unit 11 may be omitted.

[0038] The storage unit 12 is configured to have a non-volatile memory such as a ROM (Read only memory) or a flash memory, and a volatile memory such as a RAM (Random access memory). An emotion estimation model 121, a psychological plane table 122, and an emotion determination table 123 are stored in the storage unit 12. The psychological plane table 122 is data forming the psychological plane PP shown in FIG. 4 or FIG. 5, and has a data group in which two types of emotion index values ​​(arousal level and activity level) are associated with the corresponding emotion information. The emotion determination table 123 will be described later. The storage unit 12 further stores various programs to be executed by the controller 13. The programs stored in the storage unit 12 include a program PR1.

[0039] The controller 13 comprehensively controls the operation of each part in the feeling estimation device 10. The controller 13 includes a calculation processing unit including a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) as hardware resources. The controller 13 includes functional blocks 131 to 136.

[0040] The controller 13 is a program execution device (computer) capable of executing any program. The controller 13 executes the program PR1 to realize each function of the controller 13 (including each function of the functional blocks 131 to 136). All of the operations of the controller 13 described in this embodiment may be operations that are realized by the controller 13 executing the program PR1. The program PR1 may be composed of multiple programs.

[0041] A description will be given of each functional block of the controller 13. Functional blocks 131, 132, 133, 134, 135, and 136 are an acquisition unit, an emotion candidate estimation unit, a scene determination unit, an estimation result determination unit, an estimation result output unit, and an adjustment unit, respectively.

[0042] The acquisition unit 131 acquires information, signals, and data used to estimate the emotion of the subject U1. The contents acquired by the acquisition unit 131 include vehicle sensor information output from the vehicle sensor unit 40 and information generated by the biosensor 60 (i.e., biodata of the subject U1). The contents acquired by the acquisition unit 131 further include outside-vehicle image information, inside-vehicle image information, and microphone signals. The contents acquired by the acquisition unit 131 are shared by each functional block in the controller 13.

[0043] The controller 13 estimates the emotion of the subject U1 to derive an emotion estimation result Pest. The emotion estimation result Pest represents the estimated content of the emotion of the subject U1 (the estimated emotion of the subject U1). The emotion candidate estimation unit 132 derives the arousal level and activity level of the subject U1 based on the biological data of the subject U1. Thereafter, the emotion candidate estimation unit 132 uses the psychological plane table 122 to estimate (identify) emotion candidates as candidates for the emotion estimation result Pest based on the arousal level and activity level of the subject U1.

[0044] Coordinates obtained by plotting (arranging) points indicating the arousal and activity of the subject U1 on the psychological plane PP are called plot coordinates. The plot coordinates represent the positions of the points indicating the arousal and activity of the subject U1 (positions on the psychological plane PP). The emotion candidate estimation unit 132 estimates and identifies emotion candidates of the subject U1 by identifying in which quadrant of the psychological plane PP the plot coordinates are located. It may be considered that the estimation of emotion candidates is performed by the emotion estimation model 121.

[0045] When the plot coordinates are located within the i-th quadrant, the emotion candidate estimation unit 132 (emotion estimation model 121) estimates that the emotion of the subject U1 corresponds to the psychological state assigned to the i-th quadrant. i represents an arbitrary integer. In this case, if there are multiple psychological states assigned to the i-th quadrant, the multiple psychological states assigned to the i-th quadrant are estimated and identified as multiple emotion candidates.

[0046] As described above, the first to fourth quadrants in the psychological plane PP can be considered as the first to fourth classes, respectively. In this case, it can be said that the emotion candidate estimation unit 132 (emotion estimation model 121) classifies (classifies) the emotion of the subject U1 into one of the first to fourth classes based on the arousal level and activity level of the subject U1. One or more psychological states are assigned to the first to fourth classes. In the psychological plane PP assumed in this embodiment, multiple psychological states are assigned to each of the first to fourth classes. When multiple psychological states are assigned to the classified classes, the emotion candidate estimation unit 132 (emotion estimation model 121) sets the multiple psychological states as multiple emotion candidates.

[0047] That is, the fact that the plot coordinates are located in the first quadrant corresponds to the emotion of the subject U1 being classified into the first class. When the emotion of the subject U1 is classified into the first class, the emotion candidate estimation unit 132 sets the four psychological states "fun", "joy", "anger", and "sadness" assigned to the first class (first quadrant) as four emotion candidates. Similarly, the fact that the plot coordinates are located in the second quadrant corresponds to the emotion of the subject U1 being classified into the second class. When the emotion of the subject U1 is classified into the second class, the emotion candidate estimation unit 132 sets the two psychological states "melancholy" and "crying" assigned to the second class (second quadrant) as two emotion candidates. Similarly, the fact that the plot coordinates are located in the third quadrant corresponds to the emotion of the subject U1 being classified into the third class. When the emotion of the subject U1 is classified into the third class, the emotion candidate estimation unit 132 sets the three psychological states "boredom," "relaxed," and "calm" assigned to the third class (third quadrant) as three emotion candidates. Similarly, the fact that the plot coordinates are located in the fourth quadrant corresponds to the emotion of the subject U1 being classified into the fourth class. When the emotion of the subject U1 is classified into the fourth class, the emotion candidate estimation unit 132 sets the three psychological states "anxiety," "fear," and "unpleasantness" assigned to the fourth class (fourth quadrant) as three emotion candidates.

[0048] However, if the plot coordinates are located in the i-th quadrant and there is only one psychological state assigned to the i-th quadrant, the one psychological state assigned to the i-th quadrant may be set as the emotion estimation result Pest as it is. That is, for example, a transformed psychological plane in which only one psychological state "melancholy" is assigned to the second quadrant may be used. When the transformed psychological plane is used and the plot coordinates are located in the second quadrant, the psychological state "melancholy" assigned to the second quadrant may be set as the emotion estimation result Pest. In the following, unless otherwise specified, it is assumed that multiple psychological states are assigned to each of the first to fourth classes as shown in FIG. 4 or FIG. 5.

[0049] The scene determination unit 133 performs a scene determination process based on the scene determination information. In the scene determination process, a scene for the subject U1 is determined. The scene for the subject U1 is one of all situations (situations) that the subject U1 may face. In this embodiment, since the subject U1 is an occupant of the vehicle V1, the scene determined in the scene determination process (i.e., the scene for the subject U1) is a driving scene of the vehicle V1.

[0050] A driving scene is one of all the scenes (situations) that the vehicle V1 may encounter during the driving of the vehicle V1. For example, various driving scenes are assumed, such as a scene in which sudden braking or steering is performed, a scene in which the vehicle is driving in a parking lot, on a city street or on a highway, or a scene in which the vehicle is driving at an intersection, on a curve, on a slope, on an unpaved road, or on an icy road.

[0051] The scene determination unit 133 defines a total of n types of driving scenes, 1st to nth scenes. n represents an integer of 2 or more, and in many cases has an integer value of 10 or more. In the scene determination process, the scene determination unit 133 determines which of the 1st to nth scenes the driving scene of the vehicle V1 corresponds to.

[0052] The scene determination information includes information necessary to determine which of the first to nth scenes the driving scene belongs to. Specifically, the scene determination information includes at least vehicle sensor information. The scene determination information may further include all or any part of the outside camera image, the inside camera image, and the microphone signal.

[0053] The estimation result determination unit 134 determines the emotion estimation result Pest based on the estimation result of the emotion candidate by the emotion candidate estimation unit 132 and the judgment result of the traveling scene by the scene judgment unit 133 (the scene judgment result at the timing corresponding to the estimation timing of the emotion candidate). That is, by narrowing down the multiple emotion candidates identified by the estimation by the emotion candidate estimation unit 132 based on the judgment result of the traveling scene, one of the multiple emotion candidates is selected and determined as the emotion estimation result Pest. Hereinafter, the result of the scene judgment process is referred to as the scene judgment result Sdet. The scene judgment result Sdet indicates which of the above-mentioned first to nth scenes the determined traveling scene of the vehicle V1 is.

[0054] The inference result determination unit 134 determines the emotion estimation result Pest using the emotion determination table 123. The emotion determination table 123 has a data group that specifies the relationship between the emotion candidate inference result by the emotion candidate estimation unit 132, the determination result of the driving scene by the scene determination unit 133, and the emotion estimation result Pest. The emotion determination table 123 may be configured as a lookup table that uses the emotion candidate inference result and the determination result of the driving scene as input parameters and outputs the emotion estimation result Pest.

[0055] FIG. 7 shows an example of the emotion determination table 123. When the emotion determination table 123 in FIG. 7 is used, the emotion estimation result Pest is determined as follows. That is, when the emotion of the subject U1 is classified into the first class and the scene determination result Sdet indicates the first or seventh scene, the psychological state "fun" is selected and determined as the emotion estimation result Pest. When the emotion of the subject U1 is classified into the first class and the scene determination result Sdet indicates the second or eighth scene, the psychological state "joy" is selected and determined as the emotion estimation result Pest. When the emotion of the subject U1 is classified into the first class and the scene determination result Sdet indicates the third or fifth scene, the psychological state "anger" is selected and determined as the emotion estimation result Pest. When the emotion of the subject U1 is classified into the first class and the scene determination result Sdet indicates the fourth or sixth scene, the psychological state "sadness" is selected and determined as the emotion estimation result Pest.

[0056] Similarly, when the emotion of the subject U1 is classified into the second, third, or fourth class, the emotion estimation result Pest is determined based on the scene determination result Sdet. However, in Fig. 7, the specific relationship between the scene determination result Sdet and the emotion estimation result Pest when the emotion of the subject U1 is classified into the second, third, or fourth class is omitted.

[0057] Ideally, any one of the multiple emotion candidates is always selected and determined as the emotion estimation result Pest. However, there may be cases where it is not possible to narrow down one emotion candidate from the multiple emotion candidates, and in this case, for example, the emotion estimation result Pest indicates that it is one of two emotion candidates. That is, for example, when the emotion of the subject U1 is classified into the first class and the scene determination result Sdet indicates the 10th scene, the emotion estimation result Pest indicates that the emotion of the subject U1 is either "fun" or "joy". Also, for example, when the emotion of the subject U1 is classified into the first class and the scene determination result Sdet indicates the 11th scene, the emotion estimation result Pest indicates that the emotion of the subject U1 is either "anger" or "sadness".

[0058] In addition, when the classification result of the emotion of the subject U1 and the judgment result of the scene judgment unit 133 do not match (when there is no corresponding data in the emotion determination table 123), the estimation result determination unit 134 may execute the exception process J1 to generate an emotion estimation result Pest indicating that emotion estimation is impossible. That is, for example, when the emotion of the subject U1 is classified into the third class, when the scene judgment unit 133 judges that the driving scene is an abrupt steering scene, the estimation result determination unit 134 may execute the above-mentioned exception process J1. This is because it is difficult to think that the emotion of the subject U1 corresponds to "boredom", "relaxation", or "calmness" in an abrupt steering scene. The abrupt steering scene is one of the above-mentioned first to n-th classes, and corresponds to a scene in which the steering angle is changed abruptly. Similarly, when the emotion of the subject U1 is classified into the first, third, or fourth class, the exception process J1 may be executed.

[0059] The estimation result output unit 135 outputs the emotion estimation result Pest to the vehicle control device 20. This enables the vehicle control device 20 to control the vehicle V1 based on the emotion of the subject U1.

[0060] The adjustment unit 136 will be described later. It is also possible to form the emotion estimation device 10 without the adjustment unit 136. When the psychological plane PP including the neutral region NR is used, the plot coordinates may be located within the neutral region NR. In this case, the controller 13 may derive an emotion estimation result Pest indicating that the subject U1 does not have any particular emotion or that emotion estimation of the subject U1 is impossible.

[0061] [Vehicle control device 20] 8 shows a configuration block diagram of the vehicle control device 20. The vehicle control device 20 is configured by, for example, an ECU (Electronic Control Unit) for vehicle control. The vehicle control device 20 includes a communication unit 21, a storage unit 22, and a controller 23.

[0062] The communication unit 21 transmits and receives any signal to and from a counterpart device different from the vehicle control device 20. The counterpart device for the communication unit 21 is a component other than the vehicle control device 20 among the components of the in-vehicle system SYS shown in FIG. 1. The communication unit 21 can communicate with the counterpart device via an in-vehicle network formed in the vehicle V1. The counterpart device for the communication unit 21 may further include an external device (such as a server device connected to the above-mentioned communication network) provided outside the vehicle V1. The controller 23 can transmit and receive any information to and from the counterpart device (the counterpart device for the communication unit 21) using the communication unit 21, but hereinafter, a description of the communication unit 21 may be omitted.

[0063] The storage unit 22 is configured to include a non-volatile memory such as a ROM or a flash memory, and a volatile memory such as a RAM. The storage unit 22 stores programs to be executed by the controller 23, as well as any data referenced by the controller 23.

[0064] The controller 23 performs driving control of the vehicle V1 based on the driving operation of the vehicle V1 by the driver and vehicle sensor information from the vehicle sensor unit 40. The controller 23 also receives the emotion estimation result Pest from the emotion estimation device 10 via the communication unit 21. The controller 23 can perform vehicle control (driving control) using the emotion estimation result Pest. In the vehicle control using the emotion estimation result Pest, for example, the speed, accelerator sensitivity, and brake sensitivity of the vehicle V1 are controlled. More specifically, for example, when the driver (U1) is excited, the controller 23 can perform driving control that is closer to the safe side than when the driver is not excited. Examples of driving control that is closer to the safe side include control that lowers the accelerator sensitivity (to make it harder to accelerate), increases the brake sensitivity (to make it easier to stop), and lowers the upper limit of the maximum speed control.

[0065] The controller 23 also performs notification control for notifying the occupants of the vehicle V1 of various types of notification information. The notification information is transmitted to the notification unit via the communication unit 21. Here, the notification unit includes at least one of a display unit 73 and a speaker 74, and notifies the occupants of the vehicle V1 of the notification information from the vehicle control device 20 visually or audibly. The notification unit can notify the occupants of the vehicle control device 20 by displaying an image on the display unit 73. The notification unit can notify the occupants of the vehicle control device 20 by outputting an audio signal from the speaker 74.

[0066] The controller 23 can cause the notification unit to perform a notification based on the emotion estimation result Pest. At this time, the controller 23 can use the notification unit to notify the emotion estimation result Pest itself, or can perform a notification according to the emotion estimation result Pest. For example, when it is estimated that the subject U1 is excited or scared, the controller 23 uses the notification unit to display a phrase such as "calm down" or output a voice. At this time, the controller 23 controls the quality of the voice in the voice output to be a calm speaking style. To achieve this, the storage unit 22 is provided with a data table for various processes in which the emotion estimation result Pest is associated with the vehicle control and notification information of the vehicle V1. Note that the control to cause the notification unit to perform a notification based on the emotion estimation result Pest may be performed by the controller 13 of the emotion estimation device 10.

[0067] In this way, the in-vehicle system SYS estimates the emotion of the subject U1 and controls the driving of the vehicle V1 taking the estimated emotion into consideration. As a result, if the subject U1 is the driver of the vehicle V1, it becomes possible to control the driving of the vehicle V1 according to the driver's emotion, which is expected to contribute to safe driving of the vehicle V1.

[0068] [Vehicle sensor unit 40] FIG. 9 shows an internal block diagram of the vehicle sensor unit 40. The vehicle sensor unit 40 includes sensors 41 to 51. The sensors 41, 42, and 43 are an accelerator pedal sensor, a brake pedal sensor, and a steering wheel sensor, respectively. The vehicle V1 is provided with operation parts that receive driving operations from the driver, and the operation parts include an accelerator pedal, a brake pedal, and a steering wheel. The sensors 44, 45, 46, and 47 are a vehicle speed sensor, a steering angle sensor, a G sensor, and a distance measurement sensor, respectively. The sensors 48, 49, 50, and 51 are a GPS sensor, a temperature sensor, a rainfall sensor, and an illuminance sensor, respectively.

[0069] The accelerator pedal sensor 41 detects the operation of the accelerator pedal of the vehicle V1 by the driver of the vehicle V1, and generates and outputs accelerator pedal operation information indicating the operation of the accelerator pedal. The brake pedal sensor 42 detects the operation of the brake pedal of the vehicle V1 by the driver of the vehicle V1, and generates and outputs brake pedal operation information indicating the operation of the brake pedal. The steering wheel sensor 43 detects the operation of the steering wheel of the vehicle V1 by the driver of the vehicle V1, and generates and outputs steering wheel operation information indicating the operation of the steering wheel.

[0070] The vehicle speed sensor 44 detects the speed of the vehicle V1, and generates and outputs vehicle speed information (vehicle speed pulse) representing the detected speed. The steering angle sensor 45 detects the steering angle (steering angle) of the vehicle V1, and generates and outputs steering angle information representing the detected steering angle. The G sensor 46 detects acceleration in a predetermined axial direction applied to the vehicle V1, and generates and outputs the detection result of the acceleration as acceleration information. The G sensor 46 may detect acceleration in two mutually orthogonal axial directions, or may detect acceleration in three mutually orthogonal axial directions.

[0071] The distance measurement sensor 47 generates and outputs distance measurement information by performing distance measurement. In the distance measurement by the distance measurement sensor 47, the distance between the vehicle V1 and a solid object (three-dimensional object) located within the distance measurement area around the vehicle V1 is detected, and the orientation of the solid object as viewed from the vehicle V1 is also detected. These detection results are included in the distance measurement information. The distance measurement sensor 47 may be configured with a LIDAR (Light Detection and Ranging) that performs distance measurement using light, or may be configured with a radar that performs distance measurement using radio waves. The distance measurement sensor 47 may be configured with a combination of LIDAR and radar.

[0072] The GPS sensor 48 receives signals from a plurality of GPS satellites that form a GPS (Global Positioning System), and generates and outputs vehicle position information based on the reception results. The vehicle position information generated by the GPS sensor 48 represents the current location (current position) of the vehicle V1 by longitude and latitude, or represents the current location of the host vehicle V1 by longitude, latitude, and altitude.

[0073] The temperature sensor 49 detects the temperature inside the vehicle V1, and generates and outputs temperature information representing the detected temperature. The rainfall sensor 50 detects whether it is raining outside the vehicle V1, and generates and outputs rainfall information representing the detection result. The rainfall sensor 50 detects whether it is raining by detecting whether water droplets are attached to the windshield of the vehicle V1. The illuminance sensor 51 detects the illuminance outside the vehicle V1, and generates and outputs illuminance information representing the detected illuminance.

[0074] Other sensors (e.g., a shift lever sensor, a door lock sensor) are also provided in the vehicle sensor unit 40. The vehicle sensor information generated by the vehicle sensor unit 40 has output information of each sensor in the vehicle sensor unit 40. Therefore, the vehicle sensor information has accelerator pedal operation information, brake pedal operation information, steering wheel operation information, vehicle speed information, steering angle information, acceleration information, distance measurement information, vehicle position information, temperature information, precipitation information, and illuminance information. However, any of these pieces of information may not be included in the vehicle sensor information.

[0075] [Flowchart of emotion estimation process] Fig. 10 is a flowchart of the emotion estimation process executed by the controller 13. The emotion estimation process may be a process realized by the controller 13 executing the above-mentioned program PR1. The emotion estimation process shown in Fig. 10 is activated at the timing when various controls based on the estimated emotion are started after the start of the vehicle V1, and is thereafter executed once or repeatedly executed multiple times at any timing when emotion estimation is required. The emotion estimation process has the processes of steps S11 to S20.

[0076] In the emotion estimation process, first, a data acquisition process is performed in step S11. In the data acquisition process, the acquisition unit 131 acquires biometric data of the subject U1 during the estimation evaluation period from the biometric sensor 60, and acquires scene determination information during the estimation evaluation period from the vehicle sensor unit 40, etc. As described above, the scene determination information includes at least vehicle sensor information, and may further include outside-vehicle image information, inside-vehicle image information, and microphone signals. The estimation evaluation period is an arbitrary period having a predetermined time length (for example, several seconds to several tens of seconds), and in the emotion estimation process, the emotion of the subject U1 during the estimation evaluation period is estimated. After step S11, the process proceeds to step S12.

[0077] In step S12, the emotion candidate estimation unit 132 derives and outputs the arousal level and activity level of the subject U1 as emotion index values ​​of the subject U1 based on the biological data of the subject U1 during the estimation evaluation period.

[0078] The emotion candidate estimation unit 132 derives the awakening level of the subject U1 based on the electroencephalogram data of the subject U1 during the estimation evaluation period. The awakening level is derived based on the beta / alpha waves of the electroencephalogram. That is, a value according to the beta / alpha waves of the electroencephalogram (for example, a value proportional to the beta / alpha waves of the electroencephalogram) is derived as the awakening level. The beta / alpha waves of the electroencephalogram are the ratio of the beta waves of the electroencephalogram to the alpha waves of the electroencephalogram. More specifically, for example, the beta / alpha waves of the electroencephalogram are "INT_β / INT_α" or "PW_β / PW_α". INT_β and INT_α represent the intensities of the beta and alpha waves of the electroencephalogram, respectively. PW_β and PW_α represent the powers of the beta and alpha waves of the electroencephalogram, respectively.

[0079] The emotion candidate estimation unit 132 derives the activity of the subject U1 based on the heartbeat data of the subject U1 during the estimation evaluation period. More specifically, the activity is calculated based on the LF component of the time interval between heartbeats (hereinafter referred to as the heartbeat interval). The heartbeat sensor 62 may detect the action potential of the subject U1's heart, which is an electric potential generated on the skin surface of the subject U1, and generate a cardiac potential signal indicating the detected action potential. The heartbeat sensor 62 detects the heartbeat interval, which is the time interval between the heartbeats of the subject U1, based on the cardiac potential signal. The heartbeat interval corresponds to the RR interval of the cardiac potential signal (however, it may be the interval between two adjacent Q waves in the cardiac potential signal, etc.). The heartbeat data from the heartbeat sensor 62 includes a signal indicating the heartbeat interval (may further include a cardiac potential signal). It is not essential that the heartbeat sensor 62 generates a cardiac potential signal. The emotion candidate estimation unit 132 derives the LF component of the heartbeat interval based on the heartbeat data. The LF components of the heartbeat interval are frequency components in a predetermined low frequency band (for example, a frequency band from 0.04 to 0.15 Hz) among the frequency components included in the heartbeat interval. For the subject U1, the emotion candidate estimation unit 132 derives the activity level of the subject U1 by, for example, calculating the standard deviation in the amplitude of the LF components of the heartbeat interval. At this time, a value according to the standard deviation (for example, a value proportional to the standard deviation) is derived as the activity level. In the amplitude spectrum of the heartbeat interval, the frequency components in the low frequency band of the heartbeat interval are made up of 1st to kth frequency components (k is an integer equal to or greater than 2), and the standard deviation corresponds to the standard deviation in the amplitude of the 1st to kth frequency components.

[0080] After step S12, the process proceeds to step S13. In step S13, the emotion candidate estimation unit 132 estimates (specifies) emotion candidates of the subject U1 based on the emotion index value (the awakening level and activity level of the subject U1) derived in step S12. The emotion candidate estimation unit 132 estimates (specifies) emotion candidates as candidates of the emotion estimation result Pest by inputting the awakening level and activity level derived in step S12 to the emotion estimation model 121. More specifically, the emotion candidate estimation unit 132 specifies in which of the first to fourth quadrants of the psychological plane PP the plot coordinates obtained by plotting (arranging) the points indicating the awakening level and activity level of the subject U1 on the psychological plane PP are located. When the plot coordinates are located in the i-th quadrant, the emotion of the subject U1 is determined to belong to the i-th class, and each psychological state assigned to the i-th class (each psychological state assigned to the i-th quadrant) is estimated and specified as an emotion candidate. After step S13, the process proceeds to step S14.

[0081] In step S14, the controller 13 checks whether there are multiple estimated emotion candidates. If there are multiple estimated emotion candidates (Y in step S14), the process proceeds to step S15, and if there is one or zero estimated emotion candidates (N in step S14), the process proceeds to step S17. In step S17, the controller 13 checks whether there is one estimated emotion candidate. If there is one estimated emotion candidate (Y in step S17), the process proceeds to step S18, and if there is not one estimated emotion candidate, i.e., if there is zero estimated emotion candidate (N in step S17), the process proceeds to step S20.

[0082] When the emotion candidate is estimated using the psychological plane PP in FIG. 4, two or more emotion candidates are always estimated in step S13. However, the flowchart of the emotion estimation process in FIG. 10 is a flowchart that takes into consideration a case where only one emotion candidate can be estimated in step S13 and a case where no emotion candidate is estimated in step S13. The former case may occur, for example, when a modified psychological plane in which only one psychological state "melancholy" is assigned to the second quadrant is used. The latter case occurs, for example, when the plot coordinates according to the awakening level and activity level of the subject U1 belong to the neutral region NR (see FIG. 5). If the emotion candidate estimation unit 132 is configured so that two or more emotion candidates are always estimated in step S13, the process always proceeds directly to step S15 after step S13.

[0083] In step S15, the scene determination unit 133 performs a scene determination process to determine the traveling scene of the vehicle V1 during the estimation evaluation period based on the scene determination information acquired in step S11. The scene determination unit 133 generates a scene determination result Sdet as a result of the scene determination process. After step S15, the process proceeds to step S16.

[0084] In step S16, the inference result determination unit 134 determines the emotion estimation result Pest by selecting one of a plurality of emotion candidates based on the scene determination result Sdet. At this time, the inference result determination unit 134 determines the emotion estimation result Pest using the emotion determination table 123.

[0085] When the emotion of the subject U1 is classified into the first class (first quadrant), in step S13, four emotion candidates are estimated, which are "fun", "joy", "anger" and "sadness". In this case, if the emotion determination table 123 in FIG. 7 is used and the scene determination result Sdet indicates the first or seventh scene, the emotion estimation result Pest representing "fun" is generated. Similarly, when the emotion of the subject U1 is classified into the first class, if the emotion determination table 123 in FIG. 7 is used and the scene determination result Sdet indicates the second or eighth scene, the emotion estimation result Pest representing "joy" is generated. When the scene determination result Sdet indicates another scene and when the emotion of the subject U1 is classified into the second, third or fourth class, the emotion estimation result Pest is determined in the same manner.

[0086] When proceeding to step S18, one emotion candidate was estimated in step S13. Therefore, in step S18, the estimation result determination unit 134 sets the emotion candidate estimated in step S13 as the emotion estimation result Pest. The emotion estimation result Pest set here indicates the emotion candidate estimated in step S13 itself. After step S18, proceed to step S19.

[0087] In step S19, the inference result output unit 135 outputs the emotion estimation result Pest determined or set in step S16 or S18 to the vehicle control device 20. In step S20, the inference result determination unit 134 determines that the emotion of the subject U1 cannot be estimated with high accuracy, and generates an emotion estimation result Pest indicating that emotion estimation is impossible. When the emotion estimation result Pest indicating that emotion estimation is impossible is generated, the emotion of the subject U1 is not classified into any of the first to fourth classes. The emotion estimation result Pest indicating that emotion estimation is impossible may or may not be output to the vehicle control device 20. With the processing of step S19 or S20, one cycle of emotion estimation processing ends.

[0088] Incidentally, although the operational flowchart of Fig. 10 does not assume a case in which the above-mentioned exception processing J1 is executed, the exception processing J1 may be executed in the operational flowchart of Fig. 10. Also, the emotion estimated in one execution of the emotion estimation process may be affected by noise, etc. For this reason, the estimation result output unit 135 may perform statistical processing on multiple emotion estimation results Pest obtained by executing the emotion estimation process multiple times, and output information obtained thereby to the vehicle control device 20.

[0089] As described above, the controller 13 estimates emotion candidates based on multiple emotion indices while performing scene determination based on scene determination information, and determines the emotion estimation result Pest based on the emotion candidate estimation result and the scene determination result (see FIG. 10). As a result, the emotion is estimated according to the scene of the subject U1 as well as multiple emotion indices, and therefore the accuracy of emotion estimation is expected to improve.

[0090] When there are multiple emotion candidates, the controller 13 determines the emotion estimation result Pest by selecting one of the multiple emotion candidates based on the scene determination result Sdet (S15 and S16 in FIG. 1). This allows the emotion to be estimated with high accuracy. Generating the emotion estimation result Pest by narrowing down multiple emotion candidates corresponds to estimating the emotion in detail, and can be considered to belong to improving the emotion estimation accuracy.

[0091] The plurality of emotion indices may include the awakening level and activity level of the subject U1. Then, in the process of estimating emotion candidates (step S13), the controller 13 classifies the emotion of the subject U1 into one of the first to fourth classes (the first to fourth quadrants of the psychological plane PP) based on the awakening level and activity level of the subject U1. When a plurality of psychological states are assigned to the classified classes, the controller 13 sets the plurality of psychological states as a plurality of emotion candidates. For example, when the emotion of the subject U1 is classified into the first class based on the awakening level and activity level of the subject U1, the four psychological states "fun", "joy", "anger" and "sadness" are assigned to the classified first class. Therefore, the four psychological states "fun", "joy", "anger" and "sadness" are set as a plurality of emotion candidates. Then, the controller 13 determines the emotion estimation result Pest by selecting one of the plurality of emotion candidates set based on the result (Sdet) of the scene determination process (step S16). The emotion can be estimated in detail by narrowing down the emotion estimation result Pest from multiple emotion candidates based on the scene determination result Sdet. In addition, since there is a correlation between arousal and activity levels and emotions, estimating emotion candidates from arousal and activity levels contributes to accurate emotion estimation.

[0092] As described above, the awakening level and activity level of the subject U1 are estimated based on the information of the brain waves and heart rate of the subject U1 (step S12). There is a correlation between the awakening level and the brain waves, and there is also a correlation between the activity level and the heart rate. Therefore, based on the information of the brain waves and the heart rate, the awakening level and activity level can be estimated well, and thus appropriate emotion candidates can be estimated and the emotion estimation result Pest can be determined.

[0093] In this embodiment, information on the brain waves and heart rate of the subject U1 is acquired from the biosensor 60 attached to the subject U1. This allows accurate acquisition of information on the brain waves and heart rate. As a result, the arousal level and activity level can be estimated well, thereby allowing estimation of appropriate emotion candidates and determination of the emotion estimation result Pest.

[0094] In this embodiment, the subject U1 is a passenger of the vehicle V1, and the scene for the subject U1 is a driving scene of the vehicle V1. The scene determination information is formed including vehicle sensor information from the vehicle sensor unit 40 installed in the vehicle V1 (step S11). The driving scene of the vehicle V1 can be properly determined by the scene determination information including the vehicle sensor information, and the proper scene determination result is used to optimize the emotion estimation result Pest.

[0095] The in-vehicle system SYS includes an in-vehicle emotion estimation system. The in-vehicle emotion estimation system includes an emotion estimation device 10 and a vehicle sensor unit 40. Any of the components of the in-vehicle system SYS other than the emotion estimation device 10 and the vehicle sensor unit 40 may be further included in the components of the in-vehicle emotion estimation system. The vehicle sensor unit 40 is installed in the vehicle V1 and generates vehicle sensor information used for driving control of the vehicle V1 (see FIG. 2 and FIG. 9). The driving scene of the vehicle V1 is determined based on scene determination information including the vehicle sensor information (step S15). The controller 13 estimates emotion candidates based on multiple emotion indicators, while determining the driving scene of the vehicle V1 based on the scene determination information, and determines the emotion estimation result Pest based on the estimation result of the emotion candidates and the determination result of the driving scene (see FIG. 10). As a result, emotions are estimated according to not only multiple emotion indicators but also the driving scene of the vehicle V1, so that the estimation accuracy of emotions is expected to be improved. In addition, since the accuracy of emotion estimation is improved using output sensor information that is originally used in the driving control of the vehicle V1, there is no need to add dedicated parts for improving the estimation accuracy to the vehicle V1.

[0096] There is also a method of extracting so-called near-miss scenes based only on the output of the G sensor 46. However, this method often extracts scenes that are not appropriate for extraction, such as intentionally climbing over a step, as near-miss scenes. By taking into consideration the output of the G sensor 46, in combination with the emotion of the subject U1 at that time, the vehicle distance based on the vehicle V1, and the driving operation of the vehicle V1, it is possible to extract scenes that should truly be extracted as near-miss scenes.

[0097] The first embodiment includes the following multiple examples. The matters described above in the first embodiment are applied to the following examples unless otherwise specified and unless there is a contradiction. However, in each example, the description in each example may take precedence for matters that contradict the matters described above. Furthermore, unless there is a contradiction, matters described in any of the multiple examples shown below can also be applied to any other example (i.e., any two or more of the multiple examples can be combined).

[0098] [Example EX_1a] Example EX_1a will be described. The structure of a table 123a relating to the example EX_1a is shown in Fig. 11. The table 123a in Fig. 11 is a part of the emotion determination table 123 relating to the example EX_1a.

[0099] The above-mentioned first to nth scenes include three of them: a traffic jam scene, a stable driving scene, and a scene where another vehicle cuts in. A traffic jam scene for the vehicle V1 refers to a scene where the vehicle V1 is driving in an area where there is a traffic jam. A stable driving scene for the vehicle V1 refers to a scene where the vehicle V1 is driving stably while maintaining an average speed equal to or higher than a certain value. A scene where another vehicle cuts in ahead of the vehicle V1 refers to a scene where another vehicle cuts in ahead of the vehicle V1.

[0100] 11 also shows criteria for determining whether the driving scene of the vehicle V1 corresponds to a traffic jam scene, a stable driving scene, or a scene where another vehicle cuts in. However, the items of the criteria may or may not be included in the table 123a.

[0101] The scene determination unit 133 can determine whether the driving scene of the vehicle V1 corresponds to a traffic jam scene based on the distance between the vehicles ahead and the speed of the vehicle V1. The scene determination unit 133 sets the evaluation period for estimation to the scene determination period, or sets the evaluation period for estimation to the scene determination period including the evaluation period for estimation and the period before that. For example, the scene determination unit 133 can determine that the driving scene of the vehicle V1 corresponds to a traffic jam scene when the average value of the distance between the vehicles ahead during the scene determination period is equal to or less than a predetermined distance and the average speed of the vehicle V1 during the scene determination period is equal to or less than a predetermined speed.

[0102] The speed of vehicle V1 is represented by vehicle speed information (see FIG. 9). The forward inter-vehicle distance refers to the distance between vehicle V1 and another vehicle (hereinafter referred to as the immediately ahead vehicle) traveling immediately ahead of vehicle V1. The immediately ahead vehicle is located within the distance measurement area of ​​distance measurement sensor 47, and controller 13 can determine the forward inter-vehicle distance based on the distance measurement information of distance measurement sensor 47. In addition, the immediately ahead vehicle is located within the shooting area of ​​exterior camera 71, and therefore the image of the immediately ahead vehicle is included in the exterior camera image. Controller 13 may determine the forward inter-vehicle distance based on exterior image information.

[0103] When determining whether the driving scene of the vehicle V1 corresponds to a traffic jam scene, the scene determination unit 133 may further refer to in-vehicle image information (in-vehicle camera image) and microphone signals while considering the distance ahead and the speed of the vehicle V1 (see Figs. 2 and 9). For example, the scene determination unit 133 detects the facial expression and behavior of the subject U1 based on the in-vehicle image information. Then, based on whether the subject U1's face shows a negative expression such as eyes lifted, eyes glaring, or eyebrows frowning, the scene determination unit 133 may determine whether the driving scene of the vehicle V1 corresponds to a traffic jam scene. Alternatively, based on whether the driver as the subject U1 performs a behavior such as hitting the steering wheel, the scene determination unit 133 may determine whether the driving scene of the vehicle V1 corresponds to a traffic jam scene. Moreover, the scene determination unit 133 may determine whether the target person U1 is complaining based on the microphone signal, and, taking the determination result into consideration, determine whether the driving scene of the vehicle V1 corresponds to a traffic jam scene.

[0104] 11, when the emotion of the subject U1 is classified into the first class (first quadrant), if the scene determination result Sdet indicates a traffic jam scene, the psychological state "anger" is determined as the emotion estimation result Pest. The estimation result determination unit 134 determines the emotion estimation result Pest according to the regulation.

[0105] The scene determination unit 133 can determine whether the driving scene of the vehicle V1 corresponds to a stable driving scene based on the distance between the vehicles ahead and the speed of the vehicle V1. For example, when the average value of the distance between the vehicles ahead during the scene determination period is equal to or greater than a predetermined appropriate distance and the average speed of the vehicle V1 during the scene determination period is equal to or greater than a predetermined appropriate speed, the scene determination unit 133 can determine that the driving scene of the vehicle V1 corresponds to a stable driving scene. Note that when no vehicle ahead is detected, the distance between the vehicles ahead is deemed to be sufficiently greater than the appropriate distance. The appropriate speed may be determined according to the speed limit of the road on which the vehicle V1 is located.

[0106] When determining whether the driving scene of the vehicle V1 corresponds to a stable driving scene, the scene determination unit 133 may further refer to the in-vehicle image information (in-vehicle camera image) and the microphone signal while considering the distance ahead and the speed of the vehicle V1 (see FIG. 2 and FIG. 9). For example, the scene determination unit 133 detects the facial expression of the target person U1 and the behavior of the target person U1 based on the in-vehicle image information. Then, based on whether the target person U1 has a positive expression such as a fun expression, the scene determination unit 133 may determine whether the driving scene of the vehicle V1 corresponds to a stable driving scene. Alternatively, based on whether the target person U1 behaves in a manner such as being in tune with music, the scene determination unit 133 may determine whether the driving scene of the vehicle V1 corresponds to a stable driving scene. Also, based on the microphone signal, the scene determination unit 133 may determine whether the target person U1 is humming a song, and considering the determination result, the scene determination unit 133 may determine whether the driving scene of the vehicle V1 corresponds to a stable driving scene.

[0107] In the table 123a in Fig. 11, it is stipulated that when the emotion of the subject U1 is classified into the first class and the scene determination result Sdet indicates a stable driving scene, the psychological state "fun" is determined as the emotion estimation result Pest. The estimation result determination unit 134 determines the emotion estimation result Pest according to the stipulation. Alternatively, when the emotion of the subject U1 is classified into the first class and the scene determination result Sdet indicates a stable driving scene, the emotion estimation result Pest may be determined to be either "fun" or "joy."

[0108] The scene determination unit 133 can determine whether the driving scene of the vehicle V1 corresponds to an other vehicle cutting in scene by determining whether another vehicle has cut in front of the vehicle V1 based on at least one of the distance measurement information and the outside-vehicle image information.

[0109] When determining whether the driving scene of the vehicle V1 corresponds to the other vehicle cutting-in scene, the scene determination unit 133 may further refer to the in-vehicle image information (in-vehicle camera image) and the microphone signal while considering at least one of the distance measurement information and the vehicle exterior image information (see FIG. 2 and FIG. 9). For example, the scene determination unit 133 detects the facial expression and behavior of the subject U1 based on the in-vehicle image information. Then, based on whether the subject U1's face shows a negative expression such as eyes lifted, eyes glaring, or eyebrows frowning, the scene determination unit 133 may determine whether the driving scene of the vehicle V1 corresponds to the other vehicle cutting-in scene. Alternatively, based on whether the driver as the subject U1 performs a behavior such as hitting the steering wheel, the scene determination unit 133 may determine whether the driving scene of the vehicle V1 corresponds to the other vehicle cutting-in scene. In addition, the scene determination unit 133 may determine whether the subject U1 is complaining based on the microphone signal, and taking the determination result into consideration, determine whether the driving scene of the vehicle V1 corresponds to a scene in which another vehicle cuts in.

[0110] 11, when the emotion of the subject U1 is classified into the first class, and the scene determination result Sdet indicates a scene in which another vehicle cuts in, the psychological state "anger" is determined as the emotion estimation result Pest. The estimation result determination unit 134 determines the emotion estimation result Pest according to the regulation.

[0111] Table 123a in Fig. 11 includes an item indicating post-processing. Post-processing indicates processing that is recommended to be performed by the controller 13 or 23 after the emotion estimation result Pest is determined, or processing that can be performed by the controller 13 or 23. The controller 13 or 23 may perform post-processing after determining the emotion estimation result Pest. Post-processing is determined for each combination of the scene determination result Sdet and the emotion estimation result Pest.

[0112] When the scene determination result Sdet represents a traffic jam scene and the emotion estimation result Pest represents "anger", the corresponding post-processing is a detour route suggestion and a calming music suggestion. In the detour route suggestion, the controller 13 or 23 cooperates with a navigation system mounted on the vehicle V1 to suggest a detour route suitable for getting out of the traffic jam to the occupants of the vehicle V1 (including the subject U1). The navigation system may be incorporated in the in-vehicle system SYS. The detour route is suggested using the display unit 73 and the speaker 74. In the calming music suggestion, the controller 13 or 23 suggests to the occupants of the vehicle V1 (including the subject U1) that music suitable for calming negative emotions be output from the speaker 74.

[0113] When the scene determination result Sdet represents a stable driving scene and the emotion estimation result Pest represents "fun", the corresponding post-processing is a stopover spot suggestion and a light music suggestion. In the stopover spot suggestion, the controller 13 or 23 makes a suggestion to the occupants of the vehicle V1 (including the subject U1) to stop by tourist facilities or the like in the vicinity of the current location of the vehicle V1. In the light music suggestion, the controller 13 or 23 suggests to the occupants of the vehicle V1 (including the subject U1) to output light music suitable for the current emotion (positive emotion) from the speaker 74.

[0114] When the scene determination result Sdet indicates a scene where another vehicle cuts in and the emotion estimation result Pest indicates "anger," the corresponding post-processing is a calming music suggestion and a sympathetic comment notification. The calming music suggestion is as described above. In the sympathetic comment notification, the controller 13 or 23 outputs a comment from the speaker 74 that is in sympathy with the emotion felt by the subject U1 due to the cut-in of the other vehicle, thereby calming the subject U1's mood.

[0115] In addition, the optional post-processing may include an estimation result notification (the same applies to the optional post-processing in each embodiment described later). In the estimation result notification, the controller 13 or 23 notifies the subject U1 of the contents of the emotion estimation result Pest. This enables the subject U1 to objectively recognize his / her own emotion and to control his / her emotion in a manner suitable for safe driving based on the recognition result.

[0116] [Example EX_1b] Example EX_1b will be described. The structure of a table 123b relating to example EX_1b is shown in Fig. 12. The table 123b in Fig. 12 is a part of the emotion determination table 123 relating to example EX_1b.

[0117] The above-mentioned first to nth scenes include two of the sudden braking scene and the traffic jam scene. The sudden braking scene of the vehicle V1 refers to a scene in which the speed of the vehicle V1 is suddenly decreased by the sudden application of the brakes of the vehicle V1. The traffic jam scene is as described in the embodiment EX_1a.

[0118] 12 also shows criteria for determining whether the driving scene of the vehicle V1 corresponds to a sudden braking scene or a traffic jam scene. However, the items of the criteria may or may not be included in the table 123b.

[0119] The scene determination unit 133 can determine whether the driving scene of the vehicle V1 corresponds to a sudden braking scene based on the brake pedal operation information and the speed information (i.e., the speed of the vehicle V1). The scene determination unit 133 sets the estimation evaluation period to the scene determination period, or sets it to a scene determination period including the estimation evaluation period and the period before it. For example, when the brake pedal is pressed hard during the scene determination period, resulting in a sudden decrease in the speed of the vehicle V1, the scene determination unit 133 can determine that the driving scene of the vehicle V1 corresponds to a sudden braking scene.

[0120] When determining whether the driving scene of the vehicle V1 corresponds to a sudden braking scene, the scene determination unit 133 may further refer to the distance between the vehicles ahead while considering the brake pedal operation information and the speed information. When the brake pedal is depressed hard during the scene determination period, and as a result the distance between the vehicles ahead is suddenly reduced, the scene determination unit 133 can determine that the driving scene of the vehicle V1 corresponds to a sudden braking scene.

[0121] 12, when the emotion of the subject U1 is classified into the fourth class (fourth quadrant), if the scene determination result Sdet indicates a sudden braking scene, the psychological state "fear" is determined as the emotion estimation result Pest. The estimation result determination unit 134 determines the emotion estimation result Pest according to the regulation.

[0122] The scene determination unit 133 can determine whether the driving scene of the vehicle V1 corresponds to a traffic jam scene, and the method of determination is as shown in the embodiment EX_1a.

[0123] 12, it is stipulated that when the emotion of the subject U1 is classified into the fourth class and the scene determination result Sdet indicates a traffic jam scene, the psychological state of “unpleasant” is determined as the emotion estimation result Pest. The estimation result determination unit 134 determines the emotion estimation result Pest according to the stipulation.

[0124] 12 includes an item indicating post-processing. As described above, the controller 13 or 23 may perform post-processing after determining the emotion estimation result Pest. The post-processing is determined for each combination of the scene determination result Sdet and the emotion estimation result Pest.

[0125] When the scene determination result Sdet represents a sudden braking scene and the emotion estimation result Pest represents "fear", the corresponding post-processing is the transmission of near-miss information and the notification of the occurrence of a near-miss. In the transmission of the near-miss information, the controller 13 or 23 creates the near-miss information and transmits the near-miss information to a predetermined server device (not shown). The server device here is any computer device connected to a predetermined communication network including the Internet, and collects and stores the near-miss information. The near-miss information includes outside image information, inside image information, and vehicle sensor information during a near-miss target period including the time when the sudden braking occurred. In the notification of the occurrence of a near-miss, the controller 13 or 23 outputs from the speaker 74 a comment explaining the transmission of the near-miss information and the situation that caused the transmission (for example, the comment "sudden braking was detected").

[0126] When the scene determination result Sdet represents a traffic jam scene and the emotion estimation result Pest represents "unpleasant", the corresponding post-processing is a detour route suggestion and a calming music suggestion, as shown in the embodiment EX_1a.

[0127] [Example EX_1c] An example EX_1c will be described. In the emotion estimation process, the controller 13 may determine the emotion estimation result Pest based on specific judgment material, without relying on the scene determination result Sdet.

[0128] For example, when the emotion of the subject U1 is classified into the first class (first quadrant) based on the awakening level and activity level of the subject U1, the controller 13 refers to the in-car image information as a specific judgment material, and detects the facial expression of the subject U1 based on the in-car image information. Then, when an expression corresponding to "sadness" is shown on the face of the subject U1, the estimation result determination unit 134 may generate an emotion estimation result Pest representing "sadness". The controller 13 can judge whether the facial expression of the subject U1 is an expression corresponding to "sadness" based on whether the degree of eye opening of the subject U1 is sufficiently low, whether the mouth of the subject U1 is in a so-called "downward" position, whether the eyes of the subject U1 are bloodshot or watery, etc. The specific judgment material may include a microphone signal.

[0129] The controller 13 or 23 may perform post-processing after determining the emotion estimation result Pest. When the emotion of the subject U1 is classified into the first class and the emotion estimation result Pest representing "sadness" is generated regardless of the scene determination result Sdet, the post-processing may include a rest suggestion or an uplifting music suggestion. In the rest suggestion, the controller 13 or 23 uses the display unit 73 and the speaker 74 to suggest to the occupants of the vehicle V1 (including the subject U1) that they take a rest. In the uplifting music suggestion, the controller 13 or 23 suggests to the occupants of the vehicle V1 (including the subject U1) that they output music suitable for uplifting the mood of the subject U1 from the speaker 74.

[0130] [Example EX_2a] Example EX_2a will be described. In Example EX_2a, a technique for adjusting the emotion estimation algorithm will be described. The emotion estimation algorithm is an algorithm for determining the emotion estimation result Pest in the controller 13, and includes all processing algorithms from the acquired information of the acquisition unit 131 to determining the emotion estimation result Pest.

[0131] Although the post-processing is performed according to the estimation result of the emotion of the subject U1, the estimation result of the emotion of the subject U1 may not be correct. Furthermore, due to the influence of individual differences, the emotion estimation result Pest that is correct for a certain person as the subject U1 may not be correct for another person. In the technology according to the embodiment EX_2a, the appropriateness of the emotion estimation result Pest that was the basis for the execution of the post-processing is evaluated based on the reaction of the subject U1 after the post-processing. Then, the emotion estimation algorithm is adjusted to be suitable for the subject U1 according to the evaluation result. In other words, the emotion estimation algorithm is personalized.

[0132] FIG. 13 shows a flowchart of an algorithm improvement operation for adjusting the emotion estimation algorithm. In the algorithm improvement operation shown in FIG. 13, first, emotion estimation processing is executed in step S61. The emotion estimation processing in step S61 is the same as the emotion estimation processing described above (see FIG. 10). Therefore, the algorithm improvement operation shown in FIG. 13 may be started at the timing when various controls based on the estimated emotion are started after the start of the vehicle V1. Thereafter, the algorithm improvement operation in FIG. 13 may be executed every time the emotion estimation processing is executed. The algorithm improvement operation in FIG. 13 includes the processing of steps S61 to S65. The algorithm improvement operation in FIG. 13 is executed by the controller 13. However, the post processing in step S62 may be executed by the controller 23. The process proceeds to step S62 via the emotion estimation processing in step S61.

[0133] In step S62, the controller 13 or 23 executes post-processing according to the emotion estimation result Pest obtained in the emotion estimation process in step S61. The post-processing in step S62 includes some kind of suggestion or notification to the occupants of the vehicle V1 (including the target person U1). The suggestion or notification is realized using at least one of the display unit 73 and the speaker 74. Examples of suggestions in the post-processing include the above-mentioned detour route suggestion, calming music suggestion, stop-off spot suggestion, or light music suggestion (see Figs. 11 and 12). In addition to these, various other suggestions (route change suggestion, rest suggestion, etc.) according to the emotion estimation result Pest are performed in the post-processing. Examples of notifications in the post-processing include the above-mentioned sympathy comment notification, near-miss occurrence notification, or estimation result notification (see Figs. 11 and 12). In addition to these, various other notifications (detour route notification, route change notification, service area presence or absence notification, etc.) according to the emotion estimation result Pest are performed in the post-processing. After step S62, the process proceeds to step S63.

[0134] In step S63, the controller 13 performs a reaction evaluation process. For example, the reaction evaluation process can be performed by the adjustment unit 136 (see FIG. 6). In the reaction evaluation process, the controller 13 evaluates whether the subject U1 had a positive emotion or a negative emotion in response to the post-processing proposal or notification. The result of the evaluation is derived by the controller 13 as an evaluation value. The evaluation value is a digital value expressed in three or more stages. Here, the evaluation value has one of "1", "-1" and "0". When it is evaluated that the subject U1 had a positive emotion, an evaluation value having a value of "1" is derived, and when it is evaluated that the subject U1 had a negative emotion, an evaluation value having a value of "-1" is derived. When it is evaluated that the subject U1 had neither a positive emotion nor a negative emotion, or when it is not possible to identify whether the emotion of the subject U1 was a positive emotion or a negative emotion, an evaluation value having a value of "0" is derived.

[0135] When the evaluation value is "-1", it is presumed that the post-processing was inappropriate. When the evaluation value is "1", it is presumed that the post-processing was appropriate or at least not inappropriate. When the evaluation value is "0", it is presumed that the post-processing was at least not inappropriate.

[0136] The reaction evaluation process is performed based on the reaction evaluation information. The reaction evaluation information may include in-vehicle image information during the reaction evaluation period, and may further include a microphone signal. The reaction evaluation period is a period from when the post-processing of step S62 is performed until a certain time (for example, 10 seconds) has elapsed. In the reaction evaluation process, the controller 13 detects the facial expression of the subject U1 based on the in-vehicle image information during the reaction evaluation period. Then, the detected expression is classified into a positive expression, a negative expression, or a neutral expression. A positive expression is a happy expression, typically a smiling expression. A negative expression is an unpleasant expression, for example, an expression with furrowed brows and a closed mouth. When the detected expression is not classified into either a positive expression or a negative expression, the detected expression is classified into a neutral expression.

[0137] In the reaction evaluation process, the controller 13 detects the behavior of the subject U1 based on the in-vehicle image information and the microphone signal during the reaction evaluation period. The behavior of the subject U1 includes a first positive behavior in which the subject U1 is into music. The behavior of the subject U1 includes a second positive behavior in which the subject U1 is humming music. The music here is music that is played and output from the speaker 74 based on a suggestion in the post-processing. The behavior of the subject U1 includes a first negative behavior in which the subject U1 tilts his / her head. The behavior of the subject U1 includes a second negative behavior in which the subject U1 makes a negative statement (a statement that denies the post-processing performed immediately before). The controller 13 can detect whether the subject U1 has performed the first or second positive behavior, or the first or second negative behavior, based on the in-vehicle image information and the microphone signal during the reaction evaluation period.

[0138] In the reaction evaluation process, the controller 13 comprehensively judges the facial expression of the subject U1 and the detection results of the behavior of the subject U1, and sets the evaluation value to "1", "0", or "-1". For example, the evaluation value can be set as follows. In a case where the face of the subject U1 has a positive expression and the subject U1 has a first or second positive behavior during the reaction evaluation period, the controller 13 sets the evaluation value to "1". Conversely, in a case where the face of the subject U1 has a negative expression and the subject U1 has a first or second negative behavior during the reaction evaluation period, the controller 13 sets the evaluation value to "-1". In other cases, the controller 13 sets the evaluation value to "0".

[0139] After completing the reaction evaluation process in step S63, the process proceeds to step S64. In step S64, the controller 13 checks whether the evaluation value is "-1". If the evaluation value is "-1" (Y in step S64), the controller 13 infers that the content of the post-processing was inappropriate, and proceeds to step S65. If the evaluation value is "1" or "0", the controller 13 infers that the content of the post-processing was at least not inappropriate, and ends the algorithm improvement operation in FIG. 13 without performing the process of step S65.

[0140] In step S65, adjustment unit 136 executes adjustment processing to adjust the emotion estimation algorithm under the assumption that the emotion estimation result Pest in the emotion estimation processing in step S61 is inappropriate.

[0141] Specific examples of the adjustment process in step S65 are given below. Note that in each case showing specific examples of the adjustment process in step S65, it is assumed that notification of the estimation result is included in the post-processing.

[0142] Consider case CS1. In case CS1, the emotion estimation result Pest is determined according to table 123a in FIG. 11. In case CS1, the emotion of the subject U1 is classified into the first class in the emotion estimation process in step S61, and the scene determination result Sdet indicates a traffic jam scene. As a result, in case CS1, it is estimated in step S61 that the subject U1 has an emotion of "anger," and the post-processing in step S62 includes a suggestion of calming music and a notification of the estimation result. Then, in case CS1, the evaluation value is set to "-1," and the adjustment process in step S65 is performed.

[0143] In case CS1, although the emotion estimation result Pest indicates "anger", it is estimated that the actual emotion of the target U1 is not "anger". Therefore, in case CS1, for example, in step S65, a process is performed to shift the origin of the psychological plane PP to the positive side of the horizontal axis by a predetermined amount, or a process is performed to shift the origin of the psychological plane PP to the positive side of the vertical axis by a predetermined amount (both processes may be performed). Alternatively, the neutral area NR may be expanded so that the first quadrant of the psychological plane PP is reduced. As a result, the possibility that the emotion estimation result Pest will be "anger" in cases similar to case CS1 will be reduced.

[0144] In the following cases CS1a and CS1b in which the emotion estimation result Pest is determined according to the table 123a in FIG. 11, the same adjustment process as that of case CS1 may be performed. Case CS1a is a case in which the emotion of the subject U1 is classified into the first class in the emotion estimation process of step S61 and the scene determination result Sdet indicates a stable driving scene, so that the emotion estimation result Pest is “fun”. However, in case CS1a, as in case CS1, the evaluation value is set to “−1”, so that the adjustment process of step S65 is performed. Case CS1b is a case in which the emotion of the subject U1 is classified into the first class in the emotion estimation process of step S61 and the scene determination result Sdet indicates a scene in which another vehicle cuts in, so that the emotion estimation result Pest is “anger”. However, in case CS1b, as in case CS1, the evaluation value is set to “−1”, so that the adjustment process of step S65 is performed.

[0145] Consider case CS2. In case CS2, the emotion estimation result Pest is determined according to table 123b in FIG. 12. In case CS2, the emotion of the subject U1 is classified into the fourth class in the emotion estimation process in step S61, and the scene determination result Sdet indicates a sudden braking scene. As a result, in case CS2, it is estimated in step S61 that the subject U1 is experiencing the emotion of "fear," and a near-miss occurrence notification and an estimation result notification are included in the post-processing in step S62. Then, in case CS2, the evaluation value is set to "-1," and the adjustment process in step S65 is performed.

[0146] In case CS2, although the emotion estimation result Pest indicates "fear", it is estimated that the actual emotion of the subject U1 is not "fear". Therefore, in case CS2, for example, in step S65, a process is performed to shift the origin of the psychological plane PP to the positive side of the horizontal axis by a predetermined amount, or a process is performed to shift the origin of the psychological plane PP to the negative side of the vertical axis by a predetermined amount (both processes may be performed). Alternatively, the neutral area NR may be expanded so that the fourth quadrant of the psychological plane PP is reduced. As a result, the possibility that the emotion estimation result Pest will be "fear" in cases similar to case CS2 will decrease.

[0147] In the following case CS2a in which the emotion estimation result Pest is determined according to table 123b in Fig. 12, the same adjustment process as in case CS2 may be performed. Case CS2a is a case in which the emotion of the subject U1 is classified into the fourth class in the emotion estimation process in step S61 and the scene determination result Sdet indicates a traffic jam scene, so that the emotion estimation result Pest is "unpleasant". However, in case CS2a, as in case CS2, the evaluation value is set to "-1" and the adjustment process in step S65 is performed.

[0148] The emotion estimation algorithm includes the algorithm of the scene determination process. Therefore, the algorithm of the scene determination process may be adjusted in the adjustment process of step S65. For example, in the above-mentioned case CS2, although the emotion estimation result Pest indicates "fear" as described above, it is presumed that the actual emotion of the subject U1 is not "fear". This suggests that the braking operation corresponding to case CS2 does not correspond to sudden braking for the subject U1. Therefore, in the adjustment process of step S65 related to case CS2, the algorithm of the scene determination process may be adjusted so that the driving scene of the vehicle V1 is less likely to be determined to be a sudden braking scene thereafter.

[0149] In this way, after the emotion estimation result Pest is determined, post-processing according to the emotion estimation result Pest is executed (steps S61 and S62). Then, the reaction of the subject U1 when the post-processing is executed is evaluated (step S63), and the emotion estimation algorithm (algorithm for determining the emotion estimation result Pest) is adjusted according to the evaluation result (steps S64 and S65). This adjusts the emotion estimation algorithm to one suitable for the subject U1 (i.e., the emotion estimation algorithm can be personalized). It is considered that the adjustment contributes to improving the accuracy of emotion estimation.

[0150] [Example EX_2b] An example EX_2b will be described. The reaction evaluation process in step S63 in Fig. 13 may include an emotion estimation process, and an evaluation value may be set based on the result of the emotion estimation process performed in step S63.

[0151] For example, in step S63 of FIG. 13, the arousal and activity of the subject U1 may be derived by performing a part of the emotion estimation process, and it may be determined whether the plot coordinates based on the derived arousal and activity are located in the fourth quadrant of the psychological plane PP. If the plot coordinates are located in the fourth quadrant, it may be determined that the subject U1 has a negative emotion in response to the post-processing in step S62, and the evaluation value may be set to "-1". This is because emotions assigned to the fourth quadrant are basically negative emotions (see FIG. 4 or FIG. 5). In step S63, if the plot coordinates are located in the first, second, or third quadrant, the evaluation value may be set to "0". If the plot coordinates are within the neutral region NR, the evaluation value may also be set to "0".

[0152] [Example EX_3a] An embodiment EX_3a will be described. The controller 13 may determine the emotion estimation result Pest by considering bio-information (biological information of the subject U1) other than the electroencephalogram data and the heartbeat data in addition to the electroencephalogram data and the heartbeat data. The controller 13 may derive the awakening level and activity level of the subject U1 by considering bio-information (biological information of the subject U1) other than the electroencephalogram data and the heartbeat data in addition to the electroencephalogram data and the heartbeat data. The bio-information other than the electroencephalogram data and the heartbeat data may include at least one of sweat data and body temperature data. The accuracy of emotion estimation may be improved by considering bio-information other than the electroencephalogram data and the heartbeat data.

[0153] For example, the controller 13 may determine the emotion estimation result Pest by taking into consideration a judgment that the subject U1 is in a tense state when the subject U1 is sweating relatively large, compared to when the subject U1 is sweating relatively small. Also, for example, the controller 13 may determine the emotion estimation result Pest by taking into consideration a judgment that the subject U1 is in a tense state when the subject U1's body temperature is relatively high, compared to when the subject U1's body temperature is relatively low. Also, for example, the controller 13 may determine the emotion estimation result Pest by taking into consideration a judgment that the subject U1 is sleepy when the subject U1's body temperature is relatively low, compared to when the subject U1's body temperature is relatively high.

[0154] However, subject U1 may sweat simply because the ambient temperature is high. For this reason, when referring to sweat data when deriving the alertness and activity of subject U1 or determining the emotion estimation result Pest, it is advisable to also take into consideration temperature information from the temperature sensor 49 (see FIG. 9). That is, for example, when the temperature inside the vehicle cabin indicated by the temperature information is equal to or higher than a predetermined reference temperature, it is advisable to derive the alertness and activity of subject U1 and determine the emotion estimation result Pest without relying on sweat data. This makes it possible to prevent sweating due to high temperature inside the vehicle cabin from affecting the emotion estimation result Pest.

[0155] [Example EX_3b] Example EX_3b will be described. Although arousal and activity have been given as examples of emotion index values, the emotion index values ​​handled by the emotion estimation device 10 are not limited to arousal and activity. When the biometric data of the subject U1 includes electroencephalogram data, one emotion index value derived by the emotion candidate estimation unit 132 may be any value based on the electroencephalogram data of the subject U1. When the biometric data of the subject U1 includes heartbeat data, another emotion index value derived by the emotion candidate estimation unit 132 may be any value based on the heartbeat data of the subject U1.

[0156] <<Second embodiment>> A second embodiment of the present invention will be described. The second embodiment and the third embodiment described below are embodiments based on the first embodiment. For matters not specifically mentioned in the second embodiment, the description of the first embodiment also applies to the second embodiment unless there is a contradiction (similar to the third embodiment). However, when interpreting the description of the second embodiment, the description of the second embodiment may take precedence for matters that contradict each other between the first and second embodiments (similar to the third embodiment).

[0157] In the first embodiment, a biosensor 60 is attached to a subject U1, and based on electroencephalogram data and heart rate data acquired from the biosensor 60, an emotion index value (typically, arousal level and activity level) of the subject U1 is derived.

[0158] However, a technology (hereinafter referred to as a first inverse estimation technology) in which a biological information estimation model that estimates the brain waves and heart rate of the subject U1 from image information of the subject U1 is installed in the feeling estimation device 10 may be adopted. In the second embodiment, a feeling estimation device 10 in which the first inverse estimation technology is adopted will be described below.

[0159] The model 126 in FIG. 14 is a biometric information estimation model. When the first inverse estimation technique is adopted, the biometric information estimation model 126 may be stored in the storage unit 12 of the feeling estimation device 10. The biometric information estimation model 126 is a trained AI model that estimates (infers) the brain waves and heart rate of any person from image information of the person. Therefore, by inputting image information of the subject U1 to the biometric information estimation model 126, the brain waves and heart rate of the subject U1 are estimated (inferred) in the biometric information estimation model 126. By adopting the first inverse estimation technique, it is not necessary to attach the biometric sensor 60 to the subject U1. Incidentally, AI is an abbreviation for artificial intelligence.

[0160] The image information of the subject U1 input to the bioinformation estimation model 126 includes image information of the subject U1's face. The image information of the subject U1 is included in the in-vehicle image information, and the controller 13 extracts the image information of the subject U1 from the in-vehicle image information and inputs it to the bioinformation estimation model 126. As a result, the obtained estimation results of the brain waves and heart rate of the subject U1 are supplied to the emotion candidate estimation unit 132 as the brain wave data and heart rate data of the subject U1. The emotion candidate estimation unit 132 derives an emotion index value (typically, arousal level and activity level) of the subject U1 based on the brain wave data and heart rate data obtained by estimation by the bioinformation estimation model 126. The operation after the emotion index value is derived is as shown in the first embodiment.

[0161] The biological information estimation model 126 is created in a learning device 200 shown in Fig. 15. The learning device 200 may be configured with any one or more server devices. A method for creating the biological information estimation model 126 in the learning device 200 will be described.

[0162] First, in the data collection step, a plurality of sets of learning data including image information of a subject and electroencephalogram data and heart rate data of the subject are collected. A large amount of learning data is collected (for example, tens of thousands of sets are collected).

[0163] The image information of the subject includes image information of the subject's face, and is obtained by photographing the subject with any camera. An electroencephalogram sensor equivalent to the electroencephalogram sensor 61 is attached to the subject. The electroencephalogram sensor attached to the subject measures the subject's electroencephalogram to obtain the subject's electroencephalogram data (hereinafter referred to as measured electroencephalogram data). A heartbeat sensor equivalent to the heartbeat sensor 62 is attached to the subject. The heartbeat of the subject is measured by the heartbeat sensor attached to the subject to obtain the subject's heartbeat data (hereinafter referred to as measured heartbeat data).

[0164] A unit period of a certain length of time is set, and image information of the subject obtained by photographing the subject during the unit period is associated with the measured brain wave data and measured heart rate data during the unit period. A set of learning data is composed of the image information of the subject and the measured brain wave data and measured heart rate data of the subject that have been associated. In other words, one set of learning data is composed of the image information of the subject and the measured brain wave data and measured heart rate data obtained by photographing and measuring during the same period. The subject may be the subject U1 itself, or may be someone other than the subject U1. There may be multiple subjects.

[0165] After the data collection step, a learning step is performed in the learning device 200. In the learning step, learning data is input to the learning device 200. The learning device 200 is provided with an AI model 210 consisting of a neural network. The learning device 200 provides the AI ​​model 210 with image information of the subject as input data, and provides the AI ​​model 210 with measured brain wave data and measured heart rate data of the subject as correct answer data. The AI ​​model 210 estimates (infers) the brain wave data and heart rate data of the subject from the image information of the subject. In the learning step, the learning device 200 evaluates the error between the brain wave data and heart rate data estimated by the AI ​​model 210 and the measured brain wave data and measured heart rate data provided as correct answer data. Then, the learning device 200 executes learning of the AI ​​model 210 so that the error is reduced using a learning algorithm such as an error backpropagation method. The learning here is supervised machine learning, and the parameters (weights, etc.) of the AI ​​model 210 are adjusted in the learning. The learning ends when a predetermined learning end condition is satisfied, such as when the error converges to a sufficiently small value. After the learning is completed, the AI ​​model 210 is incorporated into the feeling estimation device 10 as the biological information estimation model 126.

[0166] <<Third embodiment>> A third embodiment of the present invention will be described. In the first embodiment, a technology (hereinafter referred to as a second inverse estimation technology) in which an emotion index estimation model that estimates the arousal level and activity level of the subject U1 from image information of the subject U1 is installed in the emotion estimation device 10 may be adopted. In the third embodiment, a emotion estimation device 10 in which the second inverse estimation technology is adopted will be described below.

[0167] The model 127 in FIG. 16 is an emotion index estimation model. When the second inverse estimation technique is adopted, the emotion index estimation model 127 may be stored in the storage unit 12 of the emotion estimation device 10. The emotion index estimation model 127 is a trained AI model that estimates (infers) the arousal and activity of any person from image information of the person. Therefore, by inputting image information of the subject U1 to the emotion index estimation model 127, the arousal and activity of the subject U1 are estimated (inferred) by the emotion index estimation model 127. By adopting the second inverse estimation technique, it is not essential to attach the biosensor 60 to the subject U1.

[0168] The image information of the subject U1 input to the emotion index estimation model 127 includes image information of the subject U1's face. The image information of the subject U1 is included in the in-vehicle image information, and the controller 13 extracts the image information of the subject U1 from the in-vehicle image information and inputs it to the emotion index estimation model 127. The emotion index estimation model 127 estimates and derives the arousal and activity of the subject U1 as the emotion index value of the subject U1. The operation after the derivation of the emotion index value is as shown in the first embodiment. For example, in the flowchart of Fig. 10, the emotion index value (arousal and activity) estimated by the emotion index estimation model 127 may be treated as the emotion index value to be derived in step S12.

[0169] The emotion index estimation model 127 is created in a learning device 300 shown in Fig. 17. The learning device 300 may be configured with any one or more server devices. A method for creating the emotion index estimation model 127 in the learning device 300 will be described.

[0170] First, in the data collection step, a plurality of sets of learning data including image information of a subject and the subject's alertness and activity are collected. A large amount of learning data is collected (for example, tens of thousands of sets).

[0171] The image information of the subject includes image information of the subject's face, and is obtained by photographing the subject with any camera. An electroencephalogram sensor equivalent to the electroencephalogram sensor 61 is attached to the subject. The electroencephalogram sensor attached to the subject measures the subject's electroencephalogram to obtain the subject's electroencephalogram data (referred to as measured electroencephalogram data as described above). A heartbeat sensor equivalent to the heartbeat sensor 62 is attached to the subject. The heartbeat sensor attached to the subject measures the subject's heartbeat to obtain the subject's heartbeat data (referred to as measured heartbeat data as described above).

[0172] In the data collection step, a first model is prepared for estimating the alertness of any person based on the measured electroencephalogram data of the person, and the first model is used to estimate the alertness of the subject based on the measured electroencephalogram data of the subject. In the data collection step, a second model is prepared for estimating the activity of any person based on the measured heart rate data of the person, and the second model is used to estimate the activity of the subject based on the measured heart rate data of the subject. The alertness and activity of the subject estimated by the first and second models are included in the learning data.

[0173] More specifically, a unit period having a certain time is set, and image information of the subject obtained by photographing the subject during the unit period is associated with the subject's alertness and activity during the unit period. The alertness and activity of the subject during a certain unit period are estimated by the first and second models based on the measured brain wave data and measured heart rate data of the subject during the unit period. A set of learning data is composed of the associated image information of the subject and the alertness and activity of the subject. The subject may be the subject U1 itself, or may be someone other than the subject U1. There may be a plurality of subjects. The first and second models may be created in advance by known supervised machine learning. The first and second models may be a common model.

[0174] After the data collection step, a learning step is performed in the learning device 300. In the learning step, learning data is input to the learning device 300. The learning device 300 is provided with an AI model 310 consisting of a neural network. The learning device 300 provides the AI ​​model 310 with image information of the subject in the learning data as input data, and provides the subject's awakening and activity in the learning data as correct answer data. The AI ​​model 310 estimates (infers) the subject's awakening and activity from the subject's image information. In the learning step, the learning device 300 evaluates the error between the awakening and activity estimated by the AI ​​model 310 and the awakening and activity provided as correct answer data. Then, the learning device 300 executes learning of the AI ​​model 310 so that the error is reduced using a learning algorithm such as an error backpropagation method. The learning here is supervised machine learning, and the parameters (weights, etc.) of the AI ​​model 310 are adjusted in the learning. The learning ends when a predetermined learning end condition is satisfied, such as when the error converges to a sufficiently small value. After the learning is completed, the AI ​​model 310 is incorporated into the emotion estimation device 10 as the emotion index estimation model 127.

[0175] <<Additional Information>> In the above-described embodiments, an occupant of a vehicle V1 is assumed as an example of the subject U1. However, the subject U1 may be any person whose emotion is to be estimated. For example, the subject U1 may be a patient at a medical institution. In this case, the estimated emotion can be used for examination, treatment, and the like. Alternatively, the subject U1 may be a student at an educational institution. In this case, the estimated emotion can be used for improving the content of lessons, and the like. Alternatively, the subject U1 may be an athlete in e-sports. In this case, the estimated emotion can be used for mental training of the athlete, and the estimated emotion can also be made public to viewers of the e-sports.

[0176] A program for causing a computer device to execute any of the methods described in the embodiments of the present invention, and a non-volatile recording medium having the program recorded thereon, are included in the scope of the embodiments of the present invention. Any of the processes in the embodiments of the present invention may be realized by hardware such as a semiconductor integrated circuit, software equivalent to the program, or a combination of hardware and software.

[0177] The embodiments of the present invention can be modified in various ways as appropriate within the scope of the technical ideas set forth in the claims. The above embodiments are merely examples of the present invention, and the meanings of the terms of the present invention and each component are not limited to those described in the above embodiments. The specific numerical values ​​shown in the above description are merely examples, and can be changed to various numerical values ​​as a matter of course. [Explanation of symbols]

[0178] U1 Target V1 vehicle (target vehicle) SYS In-vehicle system 10 Emotion estimation device 12 Storage section 121 Emotion estimation model 123 Psychological Plane Table 123 Emotion Determination Table PR1 Program 13 Controller 131 Acquisition Department 132 Emotion candidate estimation unit 133 Scene Judgment Section 134 Estimation result determination section 135 Estimation result output section 136 Adjustment section 20 Vehicle control device 30 Actuator section 40 Vehicle sensor section 60 Biometric Sensor 61 Brainwave Sensor 62 Heart Rate Sensor 63 Sweat Sensor 64 Body Temperature Sensor 71 Exterior Camera 72 In-car camera 73 Display section 74 Speaker 75 Microphone PP psychological plane NR1 Neutral awakening area NR2 Neutral activity region

Claims

1. An emotion estimation device that derives an emotion estimation result by estimating an emotion of a subject, Estimating emotion candidates that are candidates for the emotion estimation result based on a plurality of emotion indices of the subject; determining a scene for the subject based on scene determination information; The emotion estimation result is determined based on the emotion candidate estimation result and the scene determination result. , emotion estimation device.

2. When there are a plurality of emotion candidates, the emotion estimation result is determined by selecting one of the plurality of emotion candidates based on the scene determination result. The emotion estimation device according to claim 1 .

3. The plurality of emotion indexes include a degree of arousal of the central nervous system and a degree of activity of the autonomic nervous system in the subject, In the process of estimating the emotion candidate, the emotion of the subject is classified into one of first to fourth classes based on the arousal level and the activity level of the subject; One or more psychological states are assigned to each of the first to fourth classes; When a plurality of mental states are assigned to the classified classes, the plurality of mental states are set as a plurality of emotion candidates, and the emotion estimation result is determined by selecting one of the plurality of emotion candidates based on the scene determination result. The emotion estimation device according to claim 1 .

4. The alertness and activity are derived based on information on the subject's brainwaves and heart rate. The emotion estimation device according to claim 3 .

5. Acquiring information on the subject's brainwaves and heartbeat from a biosensor attached to the subject, or acquiring information on the subject's brainwaves and heartbeat by inputting image information of the subject into a model that estimates the person's brainwaves and heartbeat from image information of the person. The emotion estimation device according to claim 4 .

6. The image information of the subject is input to a model that estimates the arousal level and activity level of the subject from the image information of the subject, thereby deriving the arousal level and activity level of the subject. The emotion estimation device according to claim 3 .

7. Adjusting an algorithm for determining the emotion estimation result according to a reaction of the subject when post-processing according to the emotion estimation result is performed after the emotion estimation result is determined. The emotion estimation device according to any one of claims 1 to 6.

8. the subject is an occupant of a subject vehicle; the scene is a driving scene of the target vehicle, The scene determination information includes vehicle sensor information from a vehicle sensor unit installed in the target vehicle. The emotion estimation device according to any one of claims 1 to 6.

9. an emotion estimation device that estimates an emotion of a subject who is an occupant of a target vehicle to derive an emotion estimation result; a vehicle sensor unit that is installed in the target vehicle and generates vehicle sensor information that is used for driving control of the target vehicle; The emotion estimation device estimates emotion candidates that are candidates for the emotion estimation result based on a plurality of emotion indexes of the subject, determines a driving scene of the target vehicle based on scene determination information including the vehicle sensor information, and determines the emotion estimation result based on the estimation result of the emotion candidates and the determination result of the driving scene. ,In-vehicle emotion estimation system.

10. An emotion estimation method for deriving an emotion estimation result by estimating an emotion of a subject, comprising: Estimating emotion candidates that are candidates for the emotion estimation result based on a plurality of emotion indices of the subject; determining a scene for the subject based on scene determination information; The emotion estimation result is determined based on the emotion candidate estimation result and the scene determination result. ,Emotion estimation method.

11. An emotion estimation method for deriving an emotion estimation result by estimating an emotion of a subject, comprising: Estimating emotion candidates that are candidates for the emotion estimation result based on a plurality of emotion indices of the subject; determining a scene for the subject based on scene determination information; A method for determining the emotion estimation result based on the emotion candidate estimation result and the scene determination result is performed by a computer. , an emotion estimation program.

Images