Emotion estimation device, calibration method, emotion estimation program and emotion estimation system

Abstract

To provide an emotion estimation device capable of starting emotion estimation processing early, a calibration method, an emotion estimation program, and an emotion estimation system.SOLUTION: An emotion estimation device for estimating emotions according to an emotion index value calculated on the basis of biological data has a controller. The controller detects a singular point where the emotion index value in a time series has changed singularly, receives a real emotion when the singular point occurs, and performs calibration of data to be used when estimating an emotion on the basis of the real emotion and fluctuation data to be the emotion index value during a period before and after the singular point.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an emotion estimation device, a calibration method, an emotion estimation program, and an emotion estimation system. [Background technology]

[0002] Conventionally, a technique has been disclosed in which an emotional index value is calculated based on biometric data measured from a subject, and the subject's emotion is estimated from the calculated index value (see, for example, Patent Document 1). Furthermore, with this type of technique, the way the biometric data is expressed varies depending on individual differences and differences in the surrounding environment, which causes the index value to vary accordingly, and therefore calibration is required to adjust the threshold used for emotion estimation in accordance with the variation. For example, Patent Document 1 discloses a technique in which the subject is asked to perform a task that can identify changes in emotion, and calibration is performed based on the biometric data during the task execution. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-071507 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional techniques require a subject to repeatedly perform multiple different tasks to collect a certain amount of biometric data, which takes time to collect a certain amount of biometric data and for the subject to perform the tasks multiple times. As a result, it takes a long time to complete calibration, which can make it difficult to start emotion estimation processing early.

[0005] The present application has been made in view of the above, and aims to provide an emotion estimation device, a calibration method, an emotion estimation program, and an emotion estimation system that are capable of starting emotion estimation processing early. [Means for solving the problem]

[0006] The emotion estimation device according to the present application is an emotion estimation device that performs emotion estimation from emotion index values ​​calculated based on biological data, and includes a controller. The controller detects singular points at which the emotion index values ​​in a time series change uniquely, accepts an actual emotion at the time the singular point occurred, and calibrates data used in emotion estimation based on the actual emotion and fluctuation data that is the emotion index values ​​for periods before and after the singular point. [Effects of the Invention]

[0007] In the present disclosure, when a change (singularity) occurs in an emotion index value obtained without calibration, the actual emotion is obtained from the subject, and a threshold for the emotion index value is set based on the singularity and the actual emotion. Therefore, according to the present disclosure, a threshold can be set without performing calibration, allowing the emotion estimation process to start early. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing the relationship between a vehicle and a person assumed in the embodiment. [Figure 2] FIG. 2 is a diagram showing a schematic block diagram of the in-vehicle system. [Figure 3] FIG. 3 is a diagram showing an example of a two-dimensional model related to emotion estimation. [Figure 4] FIG. 4 is a diagram for explaining the neutral region. [Figure 5] FIG. 5 is a diagram illustrating an example of the configuration of the feeling estimation device according to the embodiment. [Figure 6] FIG. 6 is a diagram showing an example of the psychological plane table. [Figure 7] FIG. 7 is a diagram showing an example of the psychological plane table. [Figure 8] FIG. 8 is a diagram illustrating an example of the neutral region table. [Figure 9] FIG. 9 is a diagram illustrating an outline of the emotion estimation process performed by the controller. [Figure 10] FIG. 10 is a diagram illustrating an example of time-series data of the wakefulness level. [Figure 11] FIG. 11 is a diagram schematically showing changes in wakefulness over time. [Figure 12] FIG. 12 is a diagram showing an example of suggested content for estimating actual emotions. [Figure 13] FIG. 13 is a diagram showing an example of suggested content for estimating actual emotions. [Figure 14] FIG. 14 is a diagram showing an example of suggested content for estimating actual emotions. [Figure 15] FIG. 15 is a flowchart showing the processing steps of the data integration process executed by the feeling estimation device. [Figure 16] FIG. 16 is a flowchart showing the procedure of the emotion estimation process executed by the emotion estimation device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of an emotion estimation device, a calibration method, an emotion estimation program, and an emotion estimation system will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the embodiments described below.

[0010] <<Embodiment>> An embodiment of the present invention will be described. A technology for estimating a person's emotion will be described in the embodiment. FIG. 1 is a diagram showing the relationship between a vehicle V1 and a person U1 assumed in the embodiment. The person U1 is a target of emotion estimation, and will be referred to as the target person U1 hereinafter. The vehicle V1 may be any type of vehicle. Herein, the vehicle V1 is assumed to be an automobile or the like traveling on a road. The target person U1 is an occupant of the vehicle V1. In the present embodiment, it is assumed that the target person U1 is the driver of the vehicle V1. Hereinafter, when simply referring to the driver, this refers to the driver of the vehicle V1 (hence the target person U1). However, the target person U1 may also be an occupant other than the driver (i.e., a passenger in the vehicle V1). In line with the person U1 being referred to as the target person U1, the vehicle V1 may also be referred to as the target vehicle V1. An in-vehicle system SYS is installed in the vehicle V1. The in-vehicle system SYS is an example of an emotion estimation system.

[0011] In this disclosure, an example is given in which the target U1 of emotion estimation is a driver, but the target U1 is not limited to a driver. For example, the target U1 may be an e-sports player, a patient at a medical institution, a student at an educational institution, or a viewer of content such as video or music.

[0012] 2 is a diagram showing a schematic block diagram of the in-vehicle system SYS. The in-vehicle system SYS includes an emotion estimation device 10, a vehicle control device 20, an actuator unit 30, a vehicle sensor unit 40, a biosensor 60, an exterior camera 71, an interior camera 72, a display unit 73, a speaker 74, and a microphone 75. The components of the in-vehicle system SYS can transmit and receive any signals and information to and from each other through an in-vehicle network formed in the vehicle V1. The in-vehicle network includes, for example, a CAN (Controller Area Network) and an AVCLAN (Audio Visual Communication Local Area Network).

[0013] The emotion estimation device 10 has an emotion estimation model 131 and estimates the emotion of the subject U1 using the emotion estimation model 131. The vehicle control device 20 controls the driving of the vehicle V1 using an actuator unit 30. The actuator unit 30 has various driving components such as a motor that realizes the driving of the vehicle V1. The actuator unit 30 includes an engine and a motor that generate driving force for the vehicle V1, a steering actuator that drives the steering of the vehicle V1, and a brake actuator that drives the brakes of the vehicle V1.

[0014] The vehicle sensor unit 40 has sensors that detect the details of the driving operation of the vehicle V1 by the driver of the vehicle V1 and sensors that detect various states of the vehicle V1. The vehicle sensor unit 40 outputs vehicle sensor information containing these detection results. The vehicle control device 20 realizes driving control of the vehicle V1 by driving and controlling the actuator unit 30 in accordance with the vehicle sensor information. At this time, the vehicle control device 20 can also perform driving control in accordance with the emotion estimation result. The biometric sensor 60 detects biometric information of the subject U1. The emotion estimation device 10 estimates the emotion of the subject U1 based on the detection result of the biometric information by the biometric sensor 60. The emotion estimation device 10, the vehicle control device 20, the vehicle sensor unit 40, and the biometric sensor 60 will be described in detail below.

[0015] The exterior camera 71 consists of one or more cameras that capture images of the scene outside the vehicle V1. The exterior camera 71 has a capture area set outside the vehicle V1, and generates an exterior camera image by capturing images of the scene within the capture area. The exterior camera image is an image captured by the capture area of ​​the exterior camera 71. Image information showing the exterior camera image is referred to as exterior image information. The exterior camera 71 captures images at a predetermined frame rate.

[0016] The exterior camera 71 assumes that the vehicle V1 is traveling forward. A vehicle located in front of the vehicle V1 is referred to as a leading vehicle, and a vehicle located behind the vehicle V1 is referred to as a trailing vehicle. Although it depends on the distance between the vehicle V1 and the leading vehicle, it is assumed here that the leading vehicle is located within the imaging range of the exterior camera 71, and therefore the exterior camera image includes an image of the leading vehicle. Although it also depends on the distance between the vehicle V1 and the trailing vehicle, it is assumed here that the trailing vehicle is located within the imaging range of the exterior camera 71, and therefore the exterior camera image includes an image of the trailing vehicle. The exterior camera image may be composed of a captured image of the area in front of the vehicle V1, a captured image of the area behind the vehicle V1, a captured image of the area to the right of the vehicle V1, and a captured image of the area to the left of the vehicle V1. Alternatively, the exterior camera image may be a wide-angle image containing information from these four captured images. The exterior camera image may have any configuration.

[0017] The in-vehicle camera 72 consists of one or more cameras that capture images of the interior of the vehicle V1. The in-vehicle camera 72 has a capture area set inside the vehicle V1 (i.e., the interior of the vehicle V1), and generates an in-vehicle camera image by capturing images of the capture area. The in-vehicle camera image is an image captured in the capture area by the in-vehicle camera 72. Image information showing the in-vehicle camera image is referred to as in-vehicle image information. The in-vehicle camera 72 captures images at a predetermined frame rate.

[0018] The subject U1 is located within the imaging area of ​​the in-vehicle camera 72, and therefore the image of the subject U1 is included in the in-vehicle camera image. That is, if the subject U1 is the driver, for example, the in-vehicle camera 72 is installed so as to capture an image of the area around the driver's seat from the front of the vehicle interior. In particular, it is assumed here that the face of the subject U1 is included within the imaging area of ​​the in-vehicle camera 72, and therefore the in-vehicle camera image includes a facial image of the subject U1 (a captured image of the face of the subject U1).

[0019] The display unit 73 is a display device having a liquid crystal display panel or the like, and displays any image under the control of the emotion estimation device 10, the vehicle control device 20, or a display control device (not shown). The display unit 73 is installed at an appropriate location in the cabin of the vehicle V1 so that each occupant of the vehicle V1 can see the display content of the display unit 73. A plurality of display units 73 may be installed in the cabin of the vehicle V1. The display unit 73 may be a component of a car navigation system installed in the vehicle V1. The car navigation system may be included in the in-vehicle system SYS.

[0020] The speaker 74 outputs any sound (message, music, etc.) under the control of the emotion estimation device 10, the vehicle control device 20, or an audio device (not shown). The speaker 74 is installed at an appropriate location in the cabin of the vehicle V1 so that each occupant of the vehicle V1 can hear the output sound from the speaker 74. A plurality of speakers 74 may be installed in the cabin of the vehicle V1.

[0021] The microphone 75 converts ambient sounds around the microphone 75 into an electrical audio signal. The audio signal obtained by the conversion of the microphone 75 is referred to as a microphone signal. In the in-vehicle system SYS, the microphone signal is transmitted to the emotion estimation device 10 and the vehicle control device 20. The microphone 75 is installed at an appropriate location in the cabin of the vehicle V1 so that the signal components of the speech sounds of each occupant of the vehicle V1 are included in the microphone signal (i.e., so that the speech sounds are included in the sound pickup content of the microphone 75). Multiple microphones 75 may be installed in the cabin of the vehicle V1. Hereinafter, the speech sounds picked up by the microphone 75 are assumed to be the speech sounds of the target person U1.

[0022] The emotion estimation device 10 estimates the emotion of the subject U1 based on the subject U1's biometric information. A biometric sensor 60 is attached to the subject U1 to acquire the subject U1's biometric information. The biometric sensor 60 includes at least an electroencephalogram (EEG) sensor and a heartbeat sensor, and the EEG sensor and heartbeat sensor are attached to the subject U1. The EEG sensor detects the subject U1's brain waves and outputs EEG data indicating the results of the brainwave detection. That is, the EEG data includes information on the detected EEG. The heartbeat sensor detects the subject U1's heartbeat and outputs heartbeat data indicating the results of the heartbeat detection. That is, the heartbeat data includes information on the detected heartbeat. For example, a headgear-type EEG sensor is used as the EEG sensor. For example, a chest belt-type electrocardiogram heartbeat sensor is used as the heartbeat sensor. An optical heartbeat sensor can also be used as the heartbeat sensor.

[0023] The biosensor 60 further includes a sweat sensor and a body temperature sensor, which are attached to the subject U1. The sweat sensor detects the amount of sweat (amount of sweat per predetermined time) of the subject U1 and outputs sweat data indicating the detected amount of sweat. The body temperature sensor detects the body temperature of the subject U1 and outputs body temperature data indicating the detected body temperature.

[0024] Data obtained from the biosensor 60 attached to the subject U1 is referred to as biodata. The biodata includes brain wave data, heart rate data, sweat data, and body temperature data, and represents bioinformation of the subject U1. Depending on the type of bioinformation to be acquired or the wearing comfort, other sensors may be included in the biosensor 60 and attached to the subject U1. Examples of other sensors include a blood pressure monitor or a NIRS (Near Infrared Spectroscopy) device. Note that it is not essential that the biosensor 60 be equipped with a sweat sensor, and therefore the biodata may not include sweat data. Similarly, it is not essential that the biosensor 60 be equipped with a body temperature sensor, and therefore the biodata may not include body temperature data.

[0025] The biometric data obtained by the biometric sensor 60 is provided to the emotion estimation device 10. The emotion estimation device 10 references the biometric data and estimates an emotion based on two emotion indices, which are indices indicating the mental and physical state of the subject U1. The mental and physical state of any given person includes a psychological state, and therefore each emotion indices can also be said to be an index indicating a psychological state. The value of an emotion index is referred to as an emotion index value. One of the emotion indices used in this embodiment is the arousal level of the central nervous system (hereinafter referred to as arousal level), and the other emotion index used in this embodiment is the activity level of the autonomic nervous system (hereinafter referred to as activity level). That is, one emotion index value is expressed by arousal level, and the other emotion index value is expressed by activity level. Arousal level can be derived, for example, based on electroencephalogram data. Activity level can be derived, for example, based on heart rate data. Specifically, arousal level is calculated based on beta waves and alpha waves of the electroencephalogram. The activity level is calculated from the standard deviation of the heartbeat LF (Low Frequency) component (the low frequency component of the heartbeat waveform signal).

[0026] The emotion estimation model 131 provided in the emotion estimation device 10 is a model for emotion estimation based on arousal and activity, and may be configured with a calculation formula or a conversion data table that identifies an emotion from arousal and activity. However, as will be described later, the emotion estimation model 131 estimates one or more emotion candidates as candidates for the emotion of the subject U1, and a single emotion estimation model 131 may not narrow down the emotion of the subject U1 to one.

[0027] The emotion estimation model 131 is a multidimensional model with arousal and activity as parameters, and in this case, it is a two-dimensional model with arousal and activity as two axes. The two-dimensional model is created based on medical evidence (such as papers) showing the relationship between arousal and activity and emotions. Alternatively, the two-dimensional model is created based on the results of a questionnaire administered to many subjects. The subject questionnaire results include the subject's arousal and activity identified from the measured values ​​of the subject's electroencephalogram and heart rate, and the subject's declared emotions at the time of measuring the electroencephalogram and heart rate. Note that the emotion estimation model 131 may be a multidimensional model with three or more dimensions, that is, a model that further includes parameters (emotion indices) other than arousal and activity for estimating emotions.

[0028] FIG. 3 is a diagram showing an example of a two-dimensional model related to emotion estimation. FIG. 3 shows a two-dimensional psychological plane PP defined in the two-dimensional model. According to various medical evidence related to psychology, psychology can be estimated based on two types of indices that indicate mental and physical states. In the psychological plane PP shown in FIG. 3, a vertical axis corresponding to arousal level and a horizontal axis corresponding to activity level are defined.

[0029] Points indicating the arousal and activity of subject U1 can be plotted (placed) on the psychological plane PP. The coordinates of the plotted (placed) points move in the positive direction on the vertical axis as subject U1's arousal level increases, and move in the negative direction on the vertical axis as subject U1's arousal level decreases. The coordinates of the plotted (placed) points move in the positive direction on the horizontal axis as subject U1's activity level increases, and move in the negative direction on the horizontal axis as subject U1's activity level decreases. From the origin of the psychological plane PP, the positive side of the vertical axis corresponds to an aroused state, and the negative side of the vertical axis corresponds to an unconscious state. From the origin of the psychological plane PP, the positive side of the horizontal axis corresponds to a state in which the sympathetic nervous system is activated (a state in which relatively strong emotions are present), and the negative side of the horizontal axis corresponds to a state in which the parasympathetic nervous system is activated (a state in which relatively weak emotions are present).

[0030] The point where the vertical and horizontal axes intersect on the psychological plane PP is the origin. The psychological plane PP has first to fourth quadrants separated by the vertical and horizontal axes. The first quadrant is the area where, as viewed from the origin of the psychological plane PP, the vertical axis component has a positive value and the horizontal axis component also has a positive value. The second quadrant is the area where, as viewed from the origin of the psychological plane PP, the vertical axis component has a positive value and the horizontal axis component has a negative value. The third quadrant is the area where, as viewed from the origin of the psychological plane PP, the vertical axis component has a negative value and the horizontal axis component also has a negative value. The fourth quadrant is the area where, as viewed from the origin of the psychological plane PP, the vertical axis component has a negative value and the horizontal axis component has a positive value.

[0031] In the psychological plane PP, a corresponding psychological state (in other words, emotion) is assigned to each quadrant. The psychological states of "fun, joy, anger, sadness" are assigned to the first quadrant. The psychological state of "moderate tension" may also be assigned to the first quadrant. The psychological state of "melancholy, crying" is assigned to the second quadrant. "Crying" in the second quadrant can also be considered a form of "sadness." The psychological states of "boredom, relaxation, calmness" are assigned to the third quadrant. The psychological states of "anxiety, fear, unpleasantness" are assigned to the fourth quadrant. The distance from the origin indicates the intensity of the corresponding psychological state. For example, when points indicating the arousal and activity levels of subject U1 are plotted in the second quadrant, the greater the distance between the plotted point and the origin, the stronger the psychological state of "melancholy, crying" for subject U1.

[0032] In this way, the emotions of any person depend on the arousal level and activity level of that person. The position of each axis on the psychological plane PP may be set appropriately based on experiments, etc. Each axis may also be set based on the position of the neutral area described below.

[0033] The emotion estimation model 131 can estimate the psychological state of the subject U1 from coordinates obtained by plotting (arranging) points indicating the arousal and activity of the subject U1 on the psychological plane PP. The psychological state and its intensity of the subject U1 can be estimated based on which quadrant of the psychological plane PP the coordinates of the plotted point are in, the position within the quadrant, and the distance between the plotted point and the origin. The first to fourth quadrants on the psychological plane PP can also be considered as first to fourth classes, respectively. In this case, it can be said that the emotion estimation model 131 classifies the psychological state of the subject U1 into one of the first to fourth classes. Note that when the emotion estimation model 131 is a multidimensional model with three or more dimensions, a multidimensional psychological space with three or more dimensions is used depending on the number of types of emotion index values ​​used.

[0034] Incidentally, when emotion intensity is strong, that is, when the emotion index value swings significantly toward the maximum or minimum value, emotion estimation accuracy tends to increase. On the other hand, when emotion intensity is weak, that is, when the emotion index value is near the median, emotion estimation accuracy tends to decrease. For this reason, the region near the median of the emotion index value is set as the neutral region. Figure 4 is a diagram for explaining the neutral region. As shown in Figure 4, a method can be adopted in which the region near the median of the emotion index value is set as the neutral region (corresponding to the diagonal line region in Figure 4), and an emotion estimation is determined to be impossible or no emotion is estimated for emotion index values ​​that fall into the neutral region.

[0035] The neutral regions include the arousal neutral region NR1 and the activity neutral region NR2. In the psychological plane PP, the region whose distance from the horizontal axis is equal to or less than a predetermined first boundary distance is the arousal neutral region NR1, and the region whose distance from the vertical axis is equal to or less than a predetermined second boundary distance is the activity neutral region NR2. The arousal neutral region NR1 and the activity neutral region NR2 overlap each other in the region including the origin of the psychological plane PP. The combined region of the arousal neutral region NR1 and the activity neutral region NR2 is hereinafter referred to as the neutral region NR. It may be understood that each position within the neutral region NR does not belong to any of the first to fourth quadrants of the psychological plane PP.

[0036] This disclosure provides a method for setting a neutral region without performing the calibration required for setting the neutral region. Specifically, the emotion estimation device 10 detects the occurrence of a singularity, which is a value that has changed specifically from a time-series emotion index value. When a singularity occurs, the emotion estimation device 10 receives the user's actual emotion at that time from the subject U1 (user). Then, the emotion estimation device 10 sets a neutral region (threshold value of emotion index values) based on the emotion index values ​​in the vicinity of the singularity and the received actual emotion.

[0037] In other words, in the present disclosure, a neutral region is set by combining singular points where the subject U1's emotion is likely to have changed with the actual emotion (correct emotion value) received from the subject U1. This makes it possible to set a neutral region without performing calibration that involves having the subject U1 perform a specific task that may change their emotion. Therefore, according to the present disclosure, the processing time for calibration can be reduced (eliminated), allowing the emotion estimation process to begin earlier. Note that the method for setting a neutral region in the present disclosure will be described in detail later.

[0038] Next, a configuration example of the feeling estimation device 10 will be described. FIG. 5 is a diagram showing a configuration example of the feeling estimation device 10 according to the embodiment. Note that FIG. 5 shows components necessary for explaining the features of this embodiment, and general components are omitted. As shown in FIG. 5, the feeling estimation device 10 includes a communication unit 11 and a storage unit 12. The feeling estimation device 10 also includes a controller 13. The feeling estimation device 10 may be a so-called computer device. Note that the feeling estimation device 10 may be configured to include an input device such as a keyboard and an output device such as a display.

[0039] The communication unit 11 is an interface for communicating data with other devices via the network N. The communication unit 11 is, for example, a network interface card (NIC).

[0040] The storage unit 12 is configured to include a volatile memory and a non-volatile memory. The volatile memory may include, for example, a random access memory (RAM). The non-volatile memory may include, for example, a read-only memory (ROM), a flash memory, or a hard disk drive. The non-volatile memory stores computer-readable programs and data. Note that at least some of the programs and data stored in the non-volatile memory may be obtained from another computer device connected via a wired or wireless connection, or from a portable recording medium.

[0041] As shown in Fig. 5, in this embodiment, the storage unit 12 includes a table (data table) 121. In detail, the table 121 includes a plurality of tables for various processes. For example, the table 121 includes a psychological plane table 121a and a neutral region table 121b. Figs. 6 and 7 are diagrams showing an example of the psychological plane table 121a. Fig. 8 is a diagram showing an example of the neutral region table 121b.

[0042] First, psychological plane table 121a will be described with reference to Fig. 6 and Fig. 7. Fig. 6 shows a search-type psychological plane table 121a1, and Fig. 7 shows a formula-type psychological plane table 121a2.

[0043] As shown in FIG. 6, the psychological plane table 121a1 is a two-dimensional matrix table with index types as parameters on the vertical and horizontal axes. In the search-type psychological plane table 121a1, emotion data identified from two types of index type data is stored in memory cells (memory frames) determined by two types of index types. Then, using each calculated value of the two types of index types, a corresponding memory cell in the psychological plane table 121a1 is searched, and the data (emotion type) stored in the searched memory cell becomes the emotion estimation result (including "neutral"). In FIG. 6, when the arousal level is A1 and the activity level is B1, the emotion is "melancholy / sadness." In addition, in the psychological plane table 121a1, "neutral" refers to the neutral region. For example, when the arousal level is A3 and the activity level is B1, an emotion is estimated to be indeterminate. Note that the emotion types and "neutral" stored in each memory cell in the psychological plane table 121a1 are determined during the initial calibration.

[0044] Next, as shown in FIG. 7, the formula-type psychological plane table 121a2 is a table that stores the upper and lower limits of the neutral region for each index type (arousal level, activity level) for each emotion type. In the psychological plane table 121a2, the boundary (upper or lower limit) between the neutral region of arousal level and the neutral region of activity level for each emotion type is determined during the initial calibration. For example, if the emotion type is "depressed / sad," the upper limit of the neutral region of arousal level is the boundary (neutral lower limit) between the "depressed / sad region" and the "neutral region of arousal level," and the lower limit of the neutral region of activity level is the boundary (neutral upper limit) between the "depressed / sad region" and the "neutral region of arousal level." For example, if the measured arousal level is higher than the neutral lower limit (boundary) and the activity level is lower than the neutral upper limit (boundary), the emotion is estimated to be "depressed / sad." Then, it is determined based on psychological plane table 121a2 which emotion type the calculated values ​​of the two index types fall within, and the corresponding emotion type (including "neutral") is the emotion estimation result.

[0045] As shown in FIG. 8, the items in the neutral region table 121b include "neutral data ID," "user ID," "sensor type," "corresponding index type," "upper limit value," "lower limit value," and "acquisition date and time."

[0046] The item "Neutral Data ID" of the neutral area table 121b stores neutral data ID data, which is identification information for identifying each neutral data in the neutral area table 121b. Note that each data record forming the neutral area table 121b is created for each neutral data ID data.

[0047] The item "user ID" of the neutral area table 121b stores user ID data, which is identification information for identifying a user.

[0048] The "Upper limit value" and "Lower limit value" items of the neutral region table 121b store information on the upper limit value and lower limit value determined by the controller 13. The "Acquisition date and time" item of the neutral region table 121b stores information on the date and time when the information on the upper limit value and lower limit value was stored. The upper limit value and lower limit value determined for each neutral data ID are updated each time the latest information is acquired. Furthermore, when the information on the upper limit value and lower limit value is updated, the acquisition date and time is also updated.

[0049] Returning to FIG. 5, the controller 13 includes a processor that performs arithmetic processing and the like. The processor may be configured to include, for example, a CPU (Central Processing Unit). The controller 13 may be configured with one processor or multiple processors. When configured with multiple processors, the processors may be connected to each other so that they can communicate with each other. Note that when the emotion estimation device 10 is configured as a cloud server, the CPU that configures the processor may be a virtual CPU.

[0050] In this embodiment, the functions of the controller 13 are realized by the processor executing arithmetic processing in accordance with a program PG1 stored in the storage unit 12.

[0051] The scope of this embodiment may include a computer program that causes a processor (computer) to realize at least a portion of the functions of the emotion estimation device 10. The scope of this embodiment may also include a computer-readable nonvolatile recording medium that records such a computer program. The nonvolatile recording medium may be, for example, the nonvolatile memory described above, an optical recording medium (e.g., an optical disk), a magneto-optical recording medium (e.g., a magneto-optical disk), a USB memory, an SD card, or the like.

[0052] Furthermore, each function of the controller 13 may be realized by a single program, or, for example, a configuration in which each function is realized by a separate program. Each function may also be realized as a separate server. As described above, each function may be realized by having a processor execute a program, i.e., by software, but may also be realized by other methods. At least a portion of each function may be realized using, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). That is, each function may be realized by hardware using a dedicated IC or the like. Each function may also be realized by a combination of software and hardware. Each function is a conceptual component. A function performed by one component may be distributed among multiple components. Furthermore, functions possessed by multiple components may be integrated into one component.

[0053] Next, the emotion estimation process performed by the controller 13 will be described.

[0054] Fig. 9 is a diagram showing an overview of the emotion estimation process performed by the controller 13. As shown in Fig. 9, the controller 13 starts measuring biometric data after the subject U1 wears the biometric sensor 60 (step S1). Then, the controller 13 calculates an emotion index value based on the measured biometric data.

[0055] Next, the controller 13 arranges the emotion index values ​​in chronological order and acquires fluctuation data, which are singular points of the emotion index values ​​(step S2). Here, the fluctuation data will be explained using FIG.

[0056] FIG. 10 is a diagram illustrating an example of time-series data on arousal levels. In FIG. 10, a graph is shown with arousal levels on the vertical axis and time on the horizontal axis. Fluctuation data refers to data including emotion index values ​​for a period before and after a specific point (a specific time point) where a change of a predetermined value or more occurs within a specific period (e.g., several seconds). In other words, fluctuation data refers to data obtained when emotion index values ​​change suddenly within a short period of time, i.e., data obtained near a time point when the emotion of subject U1 is likely to have changed. For example, as shown by the data circled in FIG. 10, fluctuation data is data obtained when arousal levels suddenly increase or decrease. Because fluctuation data is data obtained when the emotion of subject U1 is likely to have changed, the accuracy of the neutral region determined using the fluctuation data in a later stage can be improved. The threshold values ​​for determining whether data is fluctuation data and the amount of change in values ​​are set in advance through experiments or other means. Specifically, the emotion index value of the subject during normal times (such as when driving a vehicle) is measured, and the period and amount of change that serve as the threshold for determining fluctuation data are determined based on the measured emotion index value and emotion change information confirmed from the subject's request, etc. The controller 13 may correct the set threshold (period and amount of change). For example, the controller 13 corrects the threshold based on the user's actual emotion and the fluctuation data in step S3, which will be described later. For example, if the fluctuation data indicates anger but the actual emotion is not anger, the controller 13 corrects the threshold used to determine the fluctuation data so that it is not determined to be anger.

[0057] Furthermore, when fluctuation data occurs in the wakefulness level, the controller 13 acquires data on the activity level at the same time.

[0058] Returning to FIG. 9, next, when fluctuation data occurs, the controller 13 receives emotion data indicating the user's actual emotion at the time the fluctuation data occurred (step S3). For example, the controller 13 receives the actual emotion through voice input by the subject U1 or input via a screen (such as operating a touch panel). The controller 13 may also output a question regarding the emotional change, such as "You felt uncomfortable just now, didn't you?" by voice or text, and identify the actual emotion from the answer received from the subject U1. Note that detailed examples of question output by the controller 13 will be described later with reference to FIGS. 12 to 14.

[0059] Next, the controller 13 performs a data integration process and an emotion estimation process based on the fluctuation data acquired in step S2 and the emotion data accepted in step S3 (step S4). The data integration process is a process of setting emotion index values ​​into three regions consisting of a neutral region, a positive region, and a negative region. The emotion estimation process is a process of estimating the emotion of the subject U1 from each measured emotion index value based on the three regions for each emotion index value (arousal level and activity level) set by the data integration process. That is, in step S4, the emotion estimation process is performed after the data integration process is completed.

[0060] Emotion estimation in data integration processing will be explained using Fig. 11. Fig. 11 is a diagram for explaining, using a specific example, a method for calculating quantized values ​​of emotion indices used in emotion estimation, and is a diagram schematically showing changes in arousal level over time. In Fig. 11, the horizontal axis represents time, and the vertical axis represents arousal level.

[0061] As shown in Figure 11, controller 13 sets a high arousal level as a positive region, a low arousal level as a negative region, and an ambiguous state as a neutral region. Specifically, controller 13 determines the boundary of the neutral region based on the fluctuation data and emotion data that is the correct value. More specifically, controller 13 determines the central axis of the neutral region (the intermediate value between the upper and lower limits) and the upper and lower limits of the neutral region.

[0062] For example, the controller 13 determines the average value of the maximum arousal level in the fluctuation data in which the arousal level fluctuates upward (toward the positive region) and the minimum arousal level in the fluctuation data in which the arousal level fluctuates downward (toward the negative region) as the central axis value of the neutral region. The controller 13 also determines the upper and lower limit values ​​of the neutral region based on the standard deviations (e.g., +2σ and -2σ) of the maximum and minimum arousal levels in each fluctuation data around the average value.

[0063] The controller 13 also determines, as an upper limit value, a value obtained by multiplying the difference between the maximum value of the arousal level in fluctuation data in which the arousal level fluctuates upward and the above-mentioned central axis value (average of the maximum and minimum values) by a predetermined appropriate coefficient (less than 1) and adding the result to the central axis value. The controller 13 also determines, as a lower limit value, a value obtained by multiplying the difference between the minimum value of the arousal level in fluctuation data in which the arousal level fluctuates downward and the above-mentioned central axis value by a predetermined appropriate coefficient (less than 1) and adding the result to the central axis value.

[0064] The controller 13 may also instruct the subject U1 to perform a deep breathing task and determine the neutral region based on fluctuation data generated while the subject U1 is deep breathing. This deep breathing task is likely to result in emotional changes toward relaxation on the psychological plane PP, making it easier to obtain fluctuation data in which arousal and activity levels fluctuate downward. That is, the controller 13 determines the lower limit by subtracting the difference between the fluctuation data generated while the subject U1 is deep breathing and the average value (which can be estimated as the central axis value) of the emotion index value data following the fluctuation data (data that is likely to represent an emotionally unstable state) from the average value. Furthermore, since the neutral region is estimated to expand approximately symmetrically with respect to the central axis value, the controller 13 determines the upper limit by adding the difference to the average value.

[0065] The controller 13 may also estimate the central axis, upper limit, and lower limit of the neutral region by performing cluster analysis, which divides the fluctuation data and the emotion data representing the ground truth value into the three regions, as a data set. A known method may be used for the cluster analysis. For example, the k-means method, Otsu's multi-value method, a Gaussian mixture model, or the like may be used as the cluster analysis method. In this way, the controller 13 can improve the accuracy of the neutral region by setting the neutral region using fluctuation data linked to the actual emotion received from the subject U1.

[0066] The controller 13 then determines whether the state of arousal is high, neutral, or low depending on which of three regions, centered around the neutral region, the arousal level calculated based on the measured biological signals (electroencephalograms) is located in, and uses this as an index value for emotion estimation. Similar processing is also performed on activity levels, which are other index values.

[0067] Next, after the data integration process is completed, i.e., after the neutral region is set, the controller 13 estimates the emotion of the subject U1 based on the measured emotion index values. Specifically, the controller 13 identifies in which quadrant or neutral region on the psychological plane PP shown in Figure 4 the combination of arousal and activity measured based on the biological data is located, and estimates the psychological state (in other words, emotion) based on the identified position.

[0068] Furthermore, as shown in FIG. 9, after setting the neutral region by the data integration process, the controller 13 compares the emotion estimation result with the actual emotion at predetermined intervals, and adjusts the neutral region based on the result (step S5).

[0069] For example, each time fluctuation data occurs in the emotion index value, the controller 13 receives input of an actual emotion from the subject U1 and determines whether the emotion estimated by the emotion estimation model matches the received actual emotion. If the emotions match, the controller 13 does not change the neutral region. If the emotions do not match, the controller 13 changes the neutral region. For example, if the estimated emotion is in the "melancholy, crying (sad)" quadrant on the psychological plane PP and the actual emotion received from the subject U1 is "melancholy, crying (sad)," the controller 13 does not change the neutral region. On the other hand, if the actual emotion received from the subject U1 is "boredom, relaxation, calm" rather than "melancholy, crying (sad)" (in the case of an emotionally indeterminate state), the controller 13 adjusts the neutral region so that the "boredom, relaxation, calm" side is wider and the "melancholy, crying (sad)" side is narrower (moving the neutral region for the arousal level stratification toward the arousal side). That is, the controller 13 performs correction to raise the upper limit value and lower the lower limit value of the neutral region of the awakening level.

[0070] Specifically, in the above-described situation, for example, if the arousal level A2 and activity level B2 are in FIG. 6, the controller 13 changes "melancholy / sadness" of the arousal level A2 and activity level B2 in FIG. 6 to "neutral" (sets the arousal level A2 to the value of arousal level A3, and sets the arousal level A4 to the value of arousal level A5). Furthermore, the controller 13 increases the neutral lower limit and neutral upper limit corresponding to the arousal level neutral region of "melancholy / sadness" in FIG. 7. In this way, if fluctuation data occurs in the emotion estimation process after the neutral region is set, the controller 13 can correct the neutral region by accepting the actual emotion of the subject U1, thereby improving the accuracy of the neutral region as the emotion estimation process is performed.

[0071] The controller 13 may estimate the actual emotion of the target person U1 from a response to a proposal to the target person U1 based on the estimated emotion. This point will be described with reference to FIGS.

[0072] 12 to 14 are diagrams showing examples of suggested content for estimating an actual emotion. For convenience of explanation, Fig. 12 to 14 show an example in which the feeling estimation device 10 directly suggests the subject U1, but in practice, there are other methods, such as displaying the suggested content on a display unit of a navigation device mounted on a vehicle, or outputting the suggested content by voice via an in-vehicle speaker.

[0073] For example, in the example shown in FIG. 12, the controller 13 directly asks the subject U1 about the estimated emotion. Specifically, the controller 13 estimates the emotion of the subject U1 using an emotion estimation model and presents the estimated emotion to the subject U1, such as, "You were feeling this way just now, right?" to confirm with the subject U1 whether the estimated emotion is correct. In practice, the "this emotion" part is replaced with the content of the estimated emotion. Assume that the controller 13 receives a response from the subject U1 to the above question, such as, "No, this is what I was feeling just now." That is, when the controller 13 receives a response indicating that the actual emotion of the subject U1 is different from the estimated emotion, it corrects the neutral region based on the received response. Specifically, the controller 13 corrects the neutral region set in the psychological plane table 121a shown in FIGS. 6 and 7.

[0074] Next, the example shown in FIG. 13 illustrates a method in which the controller 13 presents a comment indicating an estimated emotion to the subject U1. Specifically, the controller 13 estimates the emotion of the subject U1 using an emotion estimation model and presents the subject U1 with a comment indicating the estimated emotion, such as, "This is how I felt today." That is, the controller 13 presents the emotion in a format similar to that of the subject U1 actually conversing with a person. Note that, in reality, the "This is how I felt" part is replaced with the content of the estimated emotion. Then, it is assumed that the controller 13 receives a response from the subject U1 in response to the comment, such as, "I see. Thank you." That is, when the controller 13 receives a response indicating that the emotion indicated by the presented comment matches the actual emotion of the subject U1, the controller 13 determines that the setting of the neutral region is correct and does not correct the neutral region.

[0075] Next, the example shown in FIG. 14 illustrates a method in which the controller 13 presents content tailored to the estimated emotion to the subject U1. Specifically, the controller 13 estimates the emotion of the subject U1 using an emotion estimation model and presents content tailored to the estimated emotion to the subject U1, such as "Shall I play some relaxing music?" In the example shown in FIG. 14, when the estimated emotion is in the upper left quadrant (melancholy) or the lower right quadrant (anxiety) of the psychological plane PP shown in FIG. 4, the controller 13 presents content (relaxing music) that changes the emotion to the lower left quadrant (relaxation). Note that, for example, when there are multiple passengers in the vehicle, the controller 13 may present content (e.g., upbeat, energetic music) that changes the emotion to the upper right quadrant (fun). Then, it is assumed that the controller 13 receives a response from the subject U1 in response to the presented content, such as "No, I prefer upbeat music." That is, when the content based on the estimated emotion does not match the actual emotion of the subject U1, that is, when the controller 13 receives a response indicating that the estimated emotion and the actual emotion are different, the controller 13 corrects the neutral region based on the received response. Specifically, the controller 13 corrects the neutral region set in the psychological plane table 121a shown in FIGS. 6 and 7.

[0076] Next, the flow of processing executed by the feeling estimation device 10 will be described with reference to FIGS. 15 and 16. FIG. 15 is a flowchart showing the processing steps of data integration processing executed by the feeling estimation device 10. FIG. 16 is a flowchart showing the processing steps of emotion estimation processing executed by the feeling estimation device 10. Note that the data integration processing shown in FIG. 15 is executed each time the IG of the vehicle is turned on. Also, the data integration processing shown in FIG. 15 is just an example. For example, when the IG of the vehicle is turned off, fluctuation data and correct values ​​may be stored, and the next time the IG is turned on, the stored fluctuation data and correct values ​​may be read out and the data integration processing may be performed.

[0077] 15, the controller 13 first acquires biological data measured by the biological sensor 60 worn by the subject U1 (step S101). The biological data includes, for example, brain wave data, heart rate data, sweat data, and body temperature data.

[0078] Next, the controller 13 calculates the emotion index value of the subject U1 based on the acquired biological data (step S102). For example, the controller 13 calculates the level of arousal based on the electroencephalogram data, and calculates the activity level based on the heartbeat data. More specifically, the controller 13 calculates the level of arousal based on the beta / alpha waves of the electroencephalogram. The controller 13 also calculates the activity level based on the standard deviation of the low-frequency components of the heartbeat.

[0079] Next, the controller 13 stores and accumulates the calculated emotion index value data in the storage unit 12 in chronological order (step S103).

[0080] Next, the controller 13 determines whether or not fluctuation data that becomes a singular point has occurred among the time-series emotion index values ​​accumulated in the storage unit 12 (step S104). Specifically, the controller 13 determines whether or not an emotion index value has occurred that has changed by more than a predetermined value within a predetermined period (for example, several seconds).

[0081] If a singular point has occurred (step S104: Yes), the controller 13 receives the actual emotion at that time from the subject U1 (step S105). If a singular point has not occurred (step S104: No), the controller 13 returns to step S101.

[0082] Next, the controller 13 accumulates the fluctuation data and the actual emotion as a data set, and determines whether or not a certain amount of the data set has been accumulated (step S106). If a certain amount of the data set has been accumulated (step S106: Yes), the controller 13 performs a data integration process (step S107) and ends the process. Specifically, for example, as the data integration process, the controller 13 performs a cluster analysis on the accumulated data set to set a neutral region. Furthermore, the controller 13 estimates the emotion using the set neutral region. Note that if a certain amount of the data set has not been accumulated (step S106: No), the controller 13 returns to step S101.

[0083] 16, the controller 13 first acquires biometric data measured by the biometric sensor 60 worn by the subject U1 (step S201). The biometric data includes, for example, brain wave data, heart rate data, sweat data, and body temperature data.

[0084] Next, the controller 13 calculates the emotion index value of the subject U1 based on the acquired biological data (step S202). For example, the controller 13 calculates the level of arousal based on the electroencephalogram data, and calculates the activity level based on the heartbeat data. More specifically, the controller 13 calculates the level of arousal based on the beta / alpha waves of the electroencephalogram. The controller 13 also calculates the activity level based on the standard deviation of the low-frequency components of the heartbeat.

[0085] Next, the controller 13 applies the calculated arousal level and activity level to the emotion estimation model to estimate the emotion of the subject U1 (step S203). Specifically, the controller 13 estimates the emotion by identifying in which quadrant (or neutral region) the pair of arousal level and activity level is located in the psychological plane PP in which the neutral region is set.

[0086] Next, the controller 13 acquires the actual emotion from the target person U1 (step S204). For example, the controller 13 presents the target person U1 with the content as shown in Fig. 12 to Fig. 14, and acquires the actual emotion from the target person U1 based on the response to the presented content.

[0087] Next, the controller 13 determines whether the estimated emotion and the actual emotion match (step S205). If the emotions match (step S205: Yes), the controller 13 ends the process. That is, the controller 13 does not correct the neutral region.

[0088] On the other hand, if the emotions do not match (step S205; No), the controller 13 corrects the neutral region based on the actual emotion (step S206), and ends the process.

[0089] As described above, the feeling estimation device 10 according to the embodiment is a feeling estimation device that performs feeling estimation from a feeling index value calculated based on biological data, and includes the controller 13. The controller 13 receives from the subject U1 the actual feeling when a singularity occurs in the feeling index value, and sets a threshold value for the feeling index value when estimating the feeling, based on the received actual feeling and fluctuation data, which is the feeling index value including the period before and after the singularity.

[0090] As a result, when a change (singularity) occurs in the emotion index value obtained without calibration, the actual emotion is obtained from the subject, and a threshold for the emotion index value is set based on the singularity and the actual emotion. Therefore, according to the present disclosure, a threshold can be set without performing calibration, allowing the emotion estimation process to start early.

[0091] <Modification> Note that the biometric data is not limited to being acquired from the biometric sensor 60. For example, a technology (hereinafter referred to as a first inverse estimation technology) in which a biometric information estimation model that estimates the brain waves and heart rate of the subject U1 from image information of the subject U1 is installed in the emotion estimation device 10 may be adopted.

[0092] When the first inverse estimation technique is employed, a biometric information estimation model may be stored in the storage unit 12 of the emotion estimation device 10. The biometric information estimation model is a trained AI model that estimates (infers) the brain waves and heart rate of any person from image information of the person. Therefore, by inputting image information of the subject U1 into the biometric information estimation model, the brain waves and heart rate of the subject U1 are estimated (inferred) by the biometric information estimation model. The employment of the inverse estimation technique makes it unnecessary for the subject U1 to wear a biometric sensor 60. Note that AI is an abbreviation for artificial intelligence.

[0093] The image information of the subject U1 input to the biological information estimation model includes image information of the subject U1's face. The image information of the subject U1 is included in the in-vehicle image information, and the controller 13 extracts the image information of the subject U1 from the in-vehicle image information and inputs it to the biological information estimation model. As a result, the obtained estimation results of the subject U1's brain waves and heart rate are supplied to the controller 13 as brain wave data and heart rate data of the subject U1. The controller 13 derives an emotion index value (typically, arousal level and activity level) of the subject U1 based on the brain wave data and heart rate data obtained by estimation using the biological information estimation model.

[0094] The biological information estimation model is created, for example, in a learning device (not shown). The learning device may be configured with any one or more server devices. A method for creating the biological information estimation model in the learning device will be described.

[0095] First, in the data collection step, a plurality of sets of training data including image information of a subject and electroencephalogram data and heart rate data of the subject are collected. A large amount of training data (for example, tens of thousands of sets) is collected.

[0096] Image information of the subject includes image information of the subject's face and is obtained by photographing the subject with any camera. An EEG sensor equivalent to the EEG sensor is attached to the subject. The EEG sensor attached to the subject measures the subject's brain waves to obtain the subject's brain wave data (hereinafter referred to as measured EEG data). A heart rate sensor equivalent to the heart rate sensor is attached to the subject. The heart rate sensor attached to the subject measures the subject's heart rate to obtain the subject's heart rate data (hereinafter referred to as measured heart rate data).

[0097] A unit period of a certain length of time is set, and image information of the subject obtained by photographing the subject during that unit period is associated with measured brain wave data and measured heart rate data during that unit period. The associated image information of the subject and the measured brain wave data and measured heart rate data of the subject constitute one set of learning data. In other words, one set of learning data consists of image information of the subject and measured brain wave data and measured heart rate data obtained by photographing and measuring during the same period. Note that the subject may be subject U1 itself, or may be someone other than subject U1. There may be multiple subjects.

[0098] After the data collection step, a learning step is performed in the learning device. In the learning step, learning data is input to the learning device. An AI model consisting of a neural network is provided in the learning device. The learning device provides the AI ​​model with image information of the subject as input data and with the subject's measured electroencephalogram data and measured heart rate data as correct answer data. The AI ​​model estimates (infers) the subject's electroencephalogram data and heart rate data from the subject's image information. In the learning step, the learning device evaluates the error between the electroencephalogram data and heart rate data estimated by the AI ​​model and the measured electroencephalogram data and measured heart rate data provided as correct answer data. The learning device then performs learning of the AI ​​model to reduce the error using a learning algorithm such as backpropagation. The learning here is supervised machine learning, and the parameters (weights, etc.) of the AI ​​model are adjusted during the learning. The learning ends when a predetermined learning termination condition is met, such as when the error converges to a sufficiently small value. The AI ​​model after learning is incorporated into the emotion estimation device 10 as a biological information estimation model.

[0099] Furthermore, in the above modification, an example of estimating biometric data from image information has been described, but an emotion index value may be estimated from image information. In this case, the emotion estimation device 10 employs a technology (hereinafter referred to as a second inverse estimation technology) in which an emotion index estimation model that estimates the arousal and activity levels of the subject U1 from the image information of the subject U1 is installed in the emotion estimation device 10.

[0100] When the second inverse estimation technique is employed, an emotion index estimation model may be stored in the storage unit 12 of the emotion estimation device 10. The emotion index estimation model is a trained AI model that estimates (infers) the arousal and activity level of any person from image information of the person. Therefore, by inputting image information of the subject U1 into the emotion index estimation model, the arousal and activity level of the subject U1 are estimated (inferred) by the emotion index estimation model. By employing the second inverse estimation technique, it is no longer necessary for the subject U1 to wear a biosensor 60.

[0101] The image information of subject U1 input to the emotion index estimation model includes image information of the subject U1's face. The image information of subject U1 is included in the in-vehicle image information, and the controller 13 extracts the image information of subject U1 from the in-vehicle image information and inputs it into the emotion index estimation model. The emotion index estimation model estimates and derives the subject U1's arousal level and activity level as the emotion index value of subject U1.

[0102] The emotion index estimation model is created in a learning device (not shown). The learning device may be configured with any one or more server devices. A method for creating an emotion index estimation model in the learning device will be described below.

[0103] First, in the data collection step, a plurality of sets of learning data including image information of the subject and the subject's alertness and activity level are collected. A large amount of learning data (for example, tens of thousands of sets) is collected.

[0104] Image information of the subject includes image information of the subject's face, and is obtained by photographing the subject with any camera. An EEG sensor equivalent to the EEG sensor is attached to the subject. The EEG sensor attached to the subject measures the subject's brain waves, thereby obtaining the subject's brain wave data (referred to as measured EEG data as described above). A heart rate sensor equivalent to the heart rate sensor is attached to the subject. The heart rate sensor attached to the subject measures the subject's heart rate, thereby obtaining the subject's heart rate data (referred to as measured heart rate data as described above).

[0105] In the data collection step, a first model is prepared for estimating the alertness of any person based on the measured electroencephalogram data of the person, and the first model is used to estimate the alertness of the subject based on the measured electroencephalogram data of the subject. In the data collection step, a second model is prepared for estimating the activity level of any person based on the measured heart rate data of the person, and the second model is used to estimate the activity level of the subject based on the measured heart rate data of the subject. The alertness and activity levels of the subject estimated by the first and second models are included in the learning data.

[0106] More specifically, a unit period having a certain time is set, and image information of the subject obtained by photographing the subject during that unit period is associated with the subject's alertness and activity level during that unit period. The subject's alertness and activity level during a certain unit period are estimated using first and second models based on the subject's measured electroencephalogram data and measured heart rate data during that unit period. A set of learning data is composed of the associated image information of the subject and the subject's alertness and activity level. Note that the subject may be subject U1 itself or may be someone other than subject U1. There may be multiple subjects. The first and second models may be created in advance using known supervised machine learning. The first and second models may be a common model.

[0107] After the data collection step, a learning step is performed in the learning device. In the learning step, learning data is input to the learning device. An AI model consisting of a neural network is provided in the learning device. The learning device provides the AI ​​model with image information of the subject in the learning data as input data and with the subject's arousal and activity levels in the learning data as ground truth data. The AI ​​model estimates (infers) the subject's arousal and activity levels from the subject's image information. In the learning step, the learning device evaluates the error between the arousal and activity levels estimated by the AI ​​model and the arousal and activity levels provided as ground truth data. The learning device then performs learning of the AI ​​model using a learning algorithm such as backpropagation to reduce the error. The learning here is supervised machine learning, and the parameters (weights, etc.) of the AI ​​model are adjusted during the learning. Learning ends when a predetermined learning termination condition is met, such as when the error converges to a sufficiently small value. The AI ​​model after learning is incorporated into the emotion estimation device 10 as an emotion index estimation model.

[0108] Furthermore, the data integration process shown in Figure 15 is just one example. For example, when the vehicle's IG is turned off, the fluctuation data and correct value can be stored, and the next time the IG is turned on, the stored fluctuation data and correct value can be read out and the data integration process can be performed.

[0109] Specifically, when the vehicle's IG is turned off, the emotion estimation device 10 stores a dataset of fluctuation data and ground truth values ​​acquired up to that point for each subject U1 in the storage unit 12. Then, when the vehicle's IG is turned on, the emotion estimation device 10 reads out the dataset stored in the storage unit 12 and performs cluster analysis based on the dataset to set a neutral region. Thereafter, the emotion estimation device 10 performs cluster analysis based on a newly obtained dataset to correct the neutral region. This allows the emotion estimation process to start immediately after the vehicle's IG is turned on.

[0110] Further advantages and modifications will readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents. [Explanation of symbols]

[0111] 10 Emotion estimation device 11 Communications Department 12 Storage section 13 Controller 20 Vehicle control device 30 Actuator section 40 Vehicle sensor unit 60 Biometric Sensor 71 Exterior camera 72 In-car camera 73 Display section 74 Speaker 75 microphone 121 Table 121a Psychological Plane Table 121b Neutral Area Table 131 Emotion estimation model PP psychological plane SYS In-vehicle system

Claims

1. An emotion estimation device that estimates emotions from emotion index values ​​calculated based on biometric data, Has a controller The controller Detecting singular points where the emotion index values ​​in the time series have changed specifically; Accepting the actual emotions at the time the singularity occurred, Calibrating data used when estimating emotions based on the actual emotions and fluctuation data, which is the emotion index value in the period before and after the singular point. Emotion estimation device.

2. The controller The emotion estimation is performed by stratifying the emotion index values ​​using a threshold value; Calibrating the threshold value The emotion estimation device according to claim 1 .

3. The controller The emotion is estimated based on a combination state of stratification results using thresholds for a plurality of types of emotion index values, The threshold value for each emotion index value is used to perform the calibration using the fluctuation data for the corresponding emotion index value. The emotion estimation device according to claim 2 .

4. The plurality of types of emotion index values ​​are The degree of central nervous system arousal calculated based on brain waves, The activity of the autonomic nervous system calculated based on the heart rate The emotion estimation device according to claim 3 .

5. an arousal level estimation model trained using training data including images of a person and arousal levels of the central nervous system at the same time; an activity estimation model trained using training data including images of a person and activity levels of the autonomic nervous system at the same time; The controller applying the captured image of the subject to the arousal level estimation model to calculate the arousal level of the subject; The captured image of the subject is applied to the activity estimation model to calculate the activity of the subject. The emotion estimation device according to claim 4 .

6. 1. A calibration method executed by an emotion estimation device that performs emotion estimation from an emotion index value calculated based on biological data, comprising: Detecting singular points where the emotion index values ​​in the time series have changed specifically; Accepting the actual emotions at the time the singularity occurred, Calibrating data used when estimating emotions based on the actual emotions and fluctuation data, which is the emotion index value in the period before and after the singular point. Calibration method.

7. An emotion estimation program executed by an emotion estimation device that performs emotion estimation from an emotion index value calculated based on biometric data, Detecting singular points where the emotion index values ​​in the time series have changed specifically; Accepting the actual emotions at the time the singularity occurred, Calibrating data used when estimating emotions based on the actual emotions and fluctuation data, which is the emotion index value in the period before and after the singular point. Emotion estimation program.

8. a biosensor that measures biometric data of a subject; a feeling estimation device that calculates an emotion index value based on the biological data measured by the biological sensor, detects a singular point at which the emotion index value in a time series changes uniquely, accepts an actual emotion at the time when the singular point occurred, and calibrates data used for estimating the emotion based on the actual emotion and fluctuation data that is the emotion index value in a period before and after the singular point; An emotion estimation system including:

Images