Estimation device, estimation method, on-vehicle system, and estimation program

Abstract

To provide a technique capable of more appropriately narrowing an emotion estimated on the basis of an emotion index value.SOLUTION: An exemplary estimation device is an estimation device that estimates an emotion, estimates an emotion candidate being a candidate of an emotion estimation result on the basis of a plurality of types of emotion index value of an emotion estimation subject, and determines the emotion estimation result on the basis of an emotion transition state of the emotion estimation subject from the emotion candidate.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a technique for estimating emotions. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there is known a device that estimates a person's emotion based on a biosignal measured by a biosensor and displays the estimated emotion (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-63324 A Summary of the Invention [Problem to be solved by the invention]

[0004] Depending on the emotion estimation model used in the above device, the emotion estimation result may be, for example, a psychological state of "joy, happiness, anger, or sadness" (an estimation result indicating that the user is in one of these psychological states).

[0005] The psychological state estimation result of "fun, joy, anger, sadness" may indicate that the subject is feeling "fun," "joy," "anger," or "sadness," and there is a problem that the estimation result is difficult to use for various controls due to restrictions. In particular, in the case of an estimation result that includes the possibility of both positive emotions such as "fun" and "joy" and negative emotions such as "anger" and "sadness," the estimation result is difficult to use. In other words, even though positive emotions and negative emotions are opposite emotions, if it is not possible to distinguish whether the emotion is positive or negative, it may be difficult to use the emotion estimation result depending on the mode of using the emotion estimation result.

[0006] In view of the above, an object of the present invention is to provide a technique that can more appropriately narrow down emotions estimated based on emotion index values. [Means for solving the problem]

[0007] An exemplary estimation device of the present invention is an estimation device that estimates emotions, estimates emotion candidates that are candidates for an emotion estimation result based on multiple types of emotion index values ​​of a subject of emotion estimation, and determines the emotion estimation result from the emotion candidates based on an emotional transition state of the subject of emotion estimation. Effect of the Invention

[0008] According to the exemplary embodiment of the present invention, although there are often multiple emotion candidates estimated based on emotion index values, the accuracy of the estimation can be expected to be high. Since the estimated emotion, which is the emotion estimation result, can be determined from among the emotion candidates based on the emotion transition state of the emotion estimation target, the estimated emotion can be expected to be highly accurate. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an estimation system according to an embodiment; [Diagram 2] A diagram showing an example of an emotion estimation model (psychological plane) [Diagram 3] FIG. 1 shows an example of the configuration of an estimation device. [Figure 4A] FIG. 13 is a diagram showing an example of a sensor table. [Figure 4B] A diagram showing an example of a psychological plane table. [Diagram 5] 1 is a flowchart showing an example of an estimation process executed by an estimation device; [Figure 6] FIG. 13 is a diagram showing an example of table data for decision processing; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.

[0011] <1. First embodiment> [1-1. Estimation system] 1 is a diagram showing an example of the configuration of an estimation system 1 according to an exemplary embodiment of the present invention. In this embodiment, a user U1 is an occupant of a vehicle 20. The estimation system 1 is configured as a system that estimates the emotion of the user U1.

[0012] 1 shows only one user U1, there may be multiple users U1. For example, user U1 may be an e-sports player, a patient at a medical institution, a student at an educational institution, a viewer of content such as video or music, etc. In the case of these users, a part of the configuration of the estimation system 1 (such as a biosensor) will be installed at the location where user U1 is present, for example, the location where the e-sports operation terminal is installed, a hospital, a school, a content viewing room, etc.

[0013] 1, the estimation system 1 includes a server 10 (computer) and a biosensor 21. In this embodiment, the server 10 is disposed outside the vehicle 20, for example, in an emotion estimation service center building or the like, and the biosensor 21 is disposed inside the vehicle 20.

[0014] The server 10 may be a physical server or a virtual server. In this embodiment, the server 10 constitutes an estimation device that estimates emotions. That is, the estimation system 1 includes the estimation device 10. Hereinafter, the server 10 will be referred to as the estimation device 10. Details of the estimation device 10 will be described later. Note that the estimation device 10 may be constituted by one server or multiple servers.

[0015] The biosensor 21 detects a plurality of types of biosignals of the user U1 as sensor signals. That is, the estimation system 1 includes the biosensor 21 that measures a plurality of types of biosignals.

[0016] In this embodiment, the biosensor 21 includes a first sensor and a second sensor. The first sensor is a brain wave sensor. The second sensor is a heart rate (pulse) sensor. However, the first sensor and the second sensor may be changed to other biosensors depending on the bioinformation to be acquired. The other biosensor may be, for example, a blood pressure monitor, a NIRS (Near Infrared Spectroscopy) device, etc.

[0017] The biosensor 21 may be composed of only a sensor worn by the user U1, for example. The biosensor 21 may also be composed of a sensor worn by the user U1 and a sensor equipped in the vehicle 20, for example. The biosensor 21 may also be composed of only a sensor equipped in the vehicle 20, for example. The sensor worn by the user U1 may be, for example, a headgear-type brainwave sensor, a wristband-type optical heart rate (pulse) sensor, or the like. The sensor equipped in the vehicle 20 may be, for example, an optical heart rate (pulse) sensor installed on the handlebar, or the like.

[0018] The biosensor 21 may be configured as a single sensor, not a plurality of sensors, so long as it is capable of measuring a plurality of types of biosignals of the user U1.

[0019] Here, the flow of processing in the estimation system 1 will be described with reference to FIG.

[0020] The biosensor 21 measures multiple types of biosignals of a user U1 who is a target of emotion estimation. A sensor signal, which is a measurement result of the biosensor 21, is transmitted to the estimation device (server) 10 via a network N1 (step S1). For example, a communication device equipped in the vehicle 20 may be involved in transmitting the sensor signal from the biosensor 21 to the estimation device 10. The estimation device 10 estimates an emotion of the user U1, which is a candidate for an estimation result, based on the multiple types of biosignals (step S2).

[0021] Here, an overview of emotion estimation in step S2 will be described. The estimation device 10 generates an index value, which is a value of an index (physiological response) indicating a mental and physical state, based on the input sensor signal. The index value, which is a value of an index (physiological response) indicating a mental and physical state, is also an emotion index value related to emotion. In this embodiment, the estimation device 10 generates index values ​​of indexes indicating two mental and physical states related to electroencephalograms and heartbeat. For example, the index indicating the mental and physical state related to electroencephalograms is a central nervous system arousal level (hereinafter simply referred to as arousal level), and the index value can be given by "β waves / α waves of electroencephalograms". Also, for example, the index indicating the mental and physical state related to heartbeats is an activity level of the autonomic nervous system, and the index value can be given by "standard deviation of heartbeat LF (low frequency range) component". The index value is calculated using a calculation model (calculation formula or conversion data table) stored in advance in a memory or the like.

[0022] The biosensor 21 may be configured to have a calculation function (a configuration having a built-in computer or the like) and to calculate the index value.

[0023] The estimation device 10 estimates emotion candidates of the user U1, which are candidates for the estimation result, based on multiple types of emotion index values ​​of the user U1. More specifically, the estimation device 10 estimates emotion candidates of the user U1, which are candidates for the estimation result, using the calculated index values ​​and an emotion estimation model stored in advance in a memory or the like. The emotion estimation model is created based on medical evidence (papers, etc.) or a data group obtained by sampling experiments (for example, measuring emotion index values ​​of subjects and acquiring emotion information at that time (questionnaire responses by the subjects)).

[0024] FIG. 2 is a diagram showing an example of an emotion estimation model (mental plane). According to various medical evidence related to psychology, it is said that mentality can be estimated based on two types of indices that indicate physical conditions. The psychological plane with the two types of indices of mental and physical conditions shown in FIG. 2 as axes is an example of an emotion estimation model according to the technical idea. In FIG. 2, the vertical axis is "arousal level (arousal-unaroused)" and the horizontal axis is "activity level of the autonomic nervous system (sympathetic nervous activity (strong emotion)-parasympathetic nervous activity (weak emotion)" and is an example of an emotion estimation model devised by the applicant of the present application that is expected to have high estimation accuracy.

[0025] In the psychological plane shown in Figure 2, psychological states (emotions) are assigned to each of the four quadrants separated by the vertical and horizontal axes. The distance from each axis indicates the intensity of the corresponding psychological state. In the example in Figure 2, the psychological states of "fun, joy, anger, sadness" are assigned to the first quadrant. The psychological state of "melancholy" is assigned to the second quadrant. The psychological states of "relaxation, calm" are assigned to the third quadrant. The psychological states of "anxiety, fear, unpleasantness" are assigned to the fourth quadrant.

[0026] The psychological state can be estimated from the coordinates obtained by plotting two types of mental and physical state index values ​​(emotion index values) obtained based on the measurement results of biosignals on a psychological plane. Specifically, the psychological state and its intensity can be estimated based on which quadrant of the psychological plane the plotted coordinates are in and how far they are from the origin.

[0027] In this model, in which the vertical axis represents "alertness (awake-unaroused): calculated, for example, based on beta / alpha waves of the brainwaves" and the horizontal axis represents "activity of the autonomic nervous system (sympathetic nervous activity (strong emotion)-parasympathetic nervous activity (weak emotion): calculated, for example, based on the standard deviation of the heart rate LF component), the subject's emotion is estimated to be "joy, joy, anger, sadness," "melancholy," "relaxed, calm" or "anxiety, fear, unpleasantness" depending on which quadrant the subject's state is in. Note that this model has been verified as significant through experiments by the applicant. In this model, the first to fourth quadrants of the psychological plane shown in FIG. 2 represent the respective multiple emotion index values ​​of user U1. The first quadrant of the psychological plane shown in FIG. 2 corresponds to a region corresponding to a combination of a positive value of the arousal level of the user U1 and a positive value of the autonomic nervous system activity level. The second quadrant of the psychological plane shown in FIG. 2 corresponds to a region corresponding to a combination of a positive value of the arousal level of the user U1 and a negative value of the autonomic nervous system activity level. The third quadrant of the psychological plane shown in FIG. 2 corresponds to a region corresponding to a combination of a negative value of the arousal level of the user U1 and a negative value of the autonomic nervous system activity level. The fourth quadrant of the psychological plane shown in FIG. 2 corresponds to a region corresponding to a combination of a negative value of the arousal level of the user U1 and a positive value of the autonomic nervous system activity level.

[0028] In the example shown in FIG. 2, the emotion estimation model is a two-dimensional plane, but it may be a three or more dimensional space (a space with three or more types of emotion index values ​​as spatial axes).

[0029] Furthermore, since error in emotion estimation is larger than in physical measurements, etc., it is possible to determine that emotion estimation is impossible (no emotion) by setting a neutral region NR as shown in Fig. 2, taking into consideration the error in the emotion estimation result and each parameter / index related to emotion estimation. In this case, each of the multiple emotion index values ​​of user U1 is converted to a ternary value (positive, neutral, negative) using two appropriate thresholds calculated through experiments, etc.

[0030] It is also possible to apply a device (detection method) that estimates a biosignal (or an emotion index value based on a biosignal) from a facial image (such as a change in facial expression or complexion) of the user U1 captured by a camera. For example, an AI (artificial intelligence) model trained with a data set of learning data in which the facial image of the user U1 is used as input data and the biosignal (or the emotion index value based on the biosignal) at that time is the correct answer value can be used instead of a biosensor. That is, a captured image of the emotion estimation target for creating learning data is used as input data, and a trained biosignal learning model (or an emotion index value learning model) machine-learned with learning data in which the biosignal (or the emotion index value based on the biosignal) of the emotion estimation target at the time of capturing the captured image is used as correct answer data is created, and the captured image of the emotion estimation target is applied to this created biosignal learning model (or emotion index value learning model) to estimate the biosignal (or the emotion index value based on the biosignal) of the emotion estimation target. In other words, an estimated biosignal is output from the biosignal learning model (or an estimated emotion index value is output from the emotion index value learning model).

[0031] The camera that captures the facial image of user U1 may be, for example, a vehicle-mounted camera provided in vehicle 20, a camera mounted on a smartphone brought into the passenger compartment of vehicle 20, or the like.

[0032] Returning to FIG. 1, the process flow of the estimation system 1 will be described below. When a predetermined condition is satisfied, the estimation device 10 determines an emotion estimation result based on the emotion transition state of the user U1 from the emotion candidates obtained in step S2 (step S3). The predetermined condition will be described later. The estimation device 10 provides (transmits) the determined estimation result to the vehicle 20 (step S4).

[0033] Then, the control device (on-board device) mounted on the vehicle 20 controls the vehicle 20 using, for example, the estimation result provided by the estimation device 10. For example, an angry driver tends to operate the accelerator, brake, and steering angle roughly. Therefore, when the user U1 is the driver and the estimation result is "anger," the control device mounted on the vehicle 20 performs control to reduce sensitivity to the accelerator, brake, and steering angle operations (to convert them into operation signals that change gently).

[0034] In another application example, when the user U1 is a sports player, the estimation result can be used for the mental training of the user U1. For example, when the user U1 feels unfavorable emotions (anxiety, anger, etc.) during playing a video game, it is determined that intensive training corresponding to the emotional state is necessary. Therefore, the estimation result of the estimation device 10 is used in the training by providing the player, etc. with information on the estimated emotions in each scene of the sport, or by providing appropriate advice information on the estimated emotions.

[0035] In addition, various types of e-sports include a type in which multiple users U1 play together and a type in which multiple users U1 play against each other. In such types of e-sports, if the emotional state of each player is displayed, it becomes possible to perform advanced game play, such as changing game tactics according to the emotional state. In addition, it is also possible to configure the e-sports spectators to understand the emotional state of each player while watching the game, which can increase the fun factor of watching the game.

[0036] Furthermore, if the user U1 is a patient at a medical institution, the estimation result can be used for examinations, treatment, etc. For example, the staff at the medical institution can understand that the patient is feeling anxious based on the estimated emotion and provide countermeasures such as counseling.

[0037] In addition, if the user U1 is a student in an educational institution, the estimation result can be used to improve the content of lessons. For example, a teacher can understand that a student finds a lesson boring based on the estimated emotion and improve the content of the lesson to make it more interesting for the student.

[0038] Furthermore, if the user U1 is a viewer of content such as video or music, the estimation result can be used to create further content. For example, a distributor of video content can create a highlight video by collecting scenes that viewers found enjoyable based on the estimated emotions.

[0039] In the above, an example has been given in which the server 10 disposed outside the vehicle 20 is the estimation device, but the estimation device may be mounted on the vehicle 20. In a configuration in which the estimation device is mounted on the vehicle 20, the estimation system becomes an in-vehicle system that controls the in-vehicle device. Such an in-vehicle system includes an emotion estimation device (estimation device 10) that estimates the emotion of the driver of the vehicle, and an in-vehicle control device that controls the in-vehicle device. In addition, for example, the estimation device may be composed of multiple devices. For example, a device disposed outside the vehicle 20 and a device mounted on the vehicle 20 may perform processing for estimating emotions in a distributed manner.

[0040] [1-2. Estimation device] FIG. 3 is a diagram showing an example of the configuration of an estimation device 10 according to an exemplary embodiment of the present invention. Note that FIG. 3 shows components necessary for explaining the features of this embodiment, and a description of general components is omitted. As shown in FIG. 3, the estimation device 10 includes a communication unit 11 and a storage unit 12. The estimation device 10 also includes a controller 13. The estimation device 10 may be a so-called computer device. Note that the estimation device 10 may be configured to include an input device such as a keyboard and an output device such as a display.

[0041] The communication unit 11 is an interface for communicating data with other devices via the network N1. The communication unit 11 is, for example, a network interface card (NIC).

[0042] The storage unit 12 includes a volatile memory and a non-volatile memory. The volatile memory may include, for example, a random access memory (RAM). The non-volatile memory may include, for example, a read only memory (ROM), a flash memory, or a hard disk drive. The non-volatile memory stores computer-readable programs and data. At least a part of the programs and data stored in the non-volatile memory may be obtained from another computer device connected by wire or wirelessly, or from a portable recording medium.

[0043] As shown in Fig. 3, in this embodiment, the storage unit 12 includes a table (data table) 121. In detail, the table 121 includes a plurality of tables for various processes. For example, the table 121 includes a sensor table 121a and a psychological plane table 121b. Fig. 4A is a diagram showing an example of the sensor table 121a. Fig. 4B is a diagram showing an example of the psychological plane table 121b.

[0044] 4A, the items of the sensor table 121a include “sensor ID”, “sensor type”, “biological signal type”, “corresponding index ID”, “corresponding index type”, and “index conversion information”. The items of the table correspond to data storage cells (storage slots).

[0045] The item "sensor ID" of the sensor table 121a stores sensor ID data, which is identification information for identifying a data record in the sensor table 121a. The sensor ID data is also a primary key of the data record in the sensor table 121a. That is, in the sensor table 121a, a data record is configured for each sensor ID data, and data of each item associated with the sensor ID data is stored in the data record.

[0046] The item "sensor type" of the sensor table 121a stores information for identifying the type of biosensor. In this example, the sensor name (or data such as model number) is stored.

[0047] The item "biological signal type" of the sensor table 121a stores the type of measurement value based on the biological signal detected by the biological sensor. This biological signal type data is data correlated with the corresponding index type data. Academically, it is recognized that the corresponding index type data can be estimated (calculated) by acquiring the biological signal type data corresponding thereto.

[0048] The item "corresponding index ID" of the sensor table 121a stores identification information for identifying a mental and physical state index generated (calculated) based on a signal from a biosensor that detects a biosignal. The item "corresponding index type" of the sensor table 121a stores the type (such as name) of the index.

[0049] The item "index conversion information" of the sensor table 121a stores conversion information (such as an arithmetic formula or a conversion data table) for calculating an index value based on a signal obtained from a biosensor that detects a biosignal. In other words, the index value of the mental and physical state index identified by the corresponding index ID is estimated (calculated) by converting the biosignal detected by the biosensor corresponding to the sensor ID data according to the index conversion information.

[0050] For example, the data record of the sensor ID "SN01" in the sensor table 121a shown in Fig. 4A has the following information. "Electroencephalogram beta / alpha waves" are measured by the output signal of "electroencephalogram sensor BA". Then, this "electroencephalogram beta / alpha waves" is converted using the index conversion information of "FX01" to obtain an index value of "wakefulness".

[0051] As shown in Fig. 4B, the psychological plane table 121b is a two-dimensional matrix table with index types (specifically, index ID data is used) as parameters of the vertical and horizontal axes. In the psychological plane table 121b, data of psychological plane types that can be used with the two index type data are stored in memory cells (memory frames) determined by two index types. For example, if the index types used as indexes are index type VSm and index type VSn, the psychological plane used for emotion estimation is psychological plane mn. Information for performing processing using psychological plane mn is read out and used for emotion estimation processing.

[0052] In addition, the sensor table 121a and the psychological plane table 121b use a common index ID. That is, based on the sensor table 121a and the psychological plane table 121b, a psychological plane mn corresponding to two types of biosensors attached to the user U1 can be determined. For example, the types of sensors attached to the user U1 are "brain wave sensor BA (index ID: VS01)" and "heartbeat sensor HA (index ID: VS02)". In this case, index ID data corresponding to "brain wave sensor BA (index ID: VS01)" and "heartbeat sensor HA (index ID: VS02)" is determined based on the sensor table 121a. Then, based on the psychological plane table 121b, the psychological plane 01-02 with "arousal level" (VS01) and "autonomic nervous system activity level" (VS02) as indexes is determined as the psychological plane to be used for emotion estimation.

[0053] Returning to FIG. 3, an example of the configuration of the estimation device 10 will be described. The controller 13 includes a processor that performs arithmetic processing and the like. The processor may include, for example, a CPU (Central Processing Unit). The controller 13 may be configured with one processor or multiple processors. When configured with multiple processors, the processors may be connected to each other so that they can communicate with each other. When the estimation device 10 is configured as a cloud server, the CPU that configures the processor may be a virtual CPU.

[0054] 3, the controller 13 includes, as its functions, an acquisition unit 131, a feeling estimation unit 132, a determination unit 133, and a providing unit 134. In this embodiment, the functions of the controller 13 are realized by a processor executing arithmetic processing in accordance with a program stored in the storage unit 12.

[0055] The scope of the present embodiment may include a computer program that causes a processor (computer) to realize at least a part of the functions of the estimation device 10. The scope of the present embodiment may also include a computer-readable non-volatile recording medium that records such a computer program. The non-volatile recording medium may be, for example, an optical recording medium (e.g., an optical disk), a magneto-optical recording medium (e.g., a magneto-optical disk), a USB memory, or an SD card, in addition to the above-mentioned non-volatile memory.

[0056] Further, each of the functional units 131 to 134 may be realized by one program, or may be realized by a separate program for each functional unit. Further, each of the functional units 131 to 134 may be realized as a separate server. Further, each of the functional units 131 to 134 may be realized by having a processor execute a program, that is, by software, as described above, or may be realized by other methods. At least a part of each of the functional units 131 to 134 may be realized by using, for example, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like. That is, each of the functional units 131 to 134 may be realized by hardware using a dedicated IC, or the like. Further, each of the functional units 131 to 134 may be realized by using both software and hardware. Further, each of the functional units 131 to 134 is a conceptual component. A function executed by one component may be distributed to multiple components. Further, functions possessed by multiple components may be integrated into one component.

[0057] The acquiring unit 131 acquires a sensor signal output from the biosensor 21 via the communication unit 11. The acquiring unit 131 stores the received data in the storage unit 12 for subsequent processing as necessary.

[0058] The emotion estimation unit 132 selects a model to be used for estimating emotion candidates based on the sensor signal acquired by the acquisition unit 131. Specifically, the emotion estimation unit 132 compares the acquired sensor signal with the sensor table 121a to extract data of the “corresponding index type” corresponding to the sensor type and the biological signal type included in the acquired sensor signal (index type data). Then, the emotion estimation unit 132 compares the “index ID” ​​of the extracted index type data with the psychological plane table 121b to select a model (psychological plane) to be used for estimating emotion candidates. For example, when the sensor IDs are “SN01” (sensor type: electroencephalogram sensor BA) and “SN02” (sensor type: heart rate sensor HA), the corresponding index IDs are “VS01” (index type: arousal level) and “VS02” (index type: autonomic nervous system activity level) (see FIG. 4A). Based on these, the emotion estimation unit 132 selects the psychological plane 01-02 as the emotion estimation model to be used.

[0059] When the type of sensor to be used by the estimation device 10 is predetermined (when the sensor type is fixedly connected (installed) to the estimation device 10, for example), the emotion estimation model to be used is also a predetermined emotion estimation model (a selection process for the emotion estimation model is not required).

[0060] The emotion estimation unit 132 executes emotion candidate estimation processing using the selected psychological plane. The emotion estimation unit 132 converts information based on the biological signal acquired by the acquisition unit 131 into an index value using the index conversion information in the sensor table 121a. In this embodiment, an index value of the arousal level and an index value of the autonomic nervous system activity level are obtained using data acquired from the first sensor (sensor type: electroencephalogram sensor BA) and the second sensor (sensor type: heart rate sensor HA).

[0061] Then, the emotion estimation unit 132 estimates emotion candidates according to the coordinate positions obtained by plotting each index value on the selected psychological plane. In this embodiment, emotion candidates are estimated according to the coordinate positions obtained by plotting the index value of the arousal level and the index value of the autonomic nervous system activity level on a psychological plane (see FIG. 2) with the arousal level on the vertical axis and the autonomic nervous system activity level on the horizontal axis. Note that, as a specific method for determining the area in which the coordinates of each index value are located (area in which emotion candidates are set), a method of calculating the positional relationship between each area boundary and each index value based on each function formula corresponding to each area boundary, or a method of extracting an area corresponding to each combination of index values ​​using a data table simulating a psychological plane (a data table in which area values ​​(corresponding to emotion candidates) for each combination of index values ​​are stored), or the like, can be applied. Then, as a method for determining emotion candidates from the area determined in this way, a method of determining emotion candidates corresponding to an area using a data table in which areas and emotion candidates are associated with each other can be applied.

[0062] Moreover, the feeling estimation unit 132 obtains a plurality of types of index values. In this embodiment, the plurality of types is two. However, a configuration may be adopted in which three or more types of index values ​​are obtained. In this case, the feeling estimation model may be a three or more dimensional space, rather than a two-dimensional plane.

[0063] The determination unit 133 determines an estimation result based on the emotion estimation result based on the emotion transition state of the user U1 from the emotion candidates estimated by the emotion estimation unit 132, that is, emotion candidates that are candidates for the estimation result.

[0064] The providing unit 135 provides the estimation result determined by the determining unit 133 to an in-vehicle device in the vehicle 20 via the network N1.

[0065] [1-3. Estimation method] Next, a method for estimating an emotion executed by the estimation device 10 will be described.

[0066] The estimation method of the present embodiment can be realized by a computer program that causes a computer device to operate in accordance with the estimation method. Such a computer program is provided to an administrator of the estimation device 10 by a computer-readable non-volatile recording medium or by using a communication network (so-called download), and is installed in the estimation device 10. Such a computer program is included in the scope of the present embodiment. Furthermore, the computer program that causes a computer device to realize the estimation method of the present embodiment may be composed of only one program, or may be composed of multiple programs.

[0067] Fig. 5 is a flowchart showing an example of an estimation process executed by the estimation device 10. The estimation process shown in Fig. 5 is started when emotion estimation is started, for example, when the acquisition unit 131 of the estimation device 10 acquires a signal notifying door unlocking from the vehicle 20 in order to estimate the emotion of the user from the time the user gets into the vehicle and use the emotion for various vehicle controls.

[0068] In step S11, the acquisition unit 131 (controller 13) acquires (inputs) the sensor signals (brain wave sensor signal and heart rate sensor signal) output from the biosensor 21, and stores them in the storage unit 12. The sensor signals stored in the storage unit 12 are read out as necessary when used by the controller 13. Once the sensor signals are stored in the storage unit 12, the process proceeds to the next step S12.

[0069] In step S12, the decision unit 133 (controller 13) determines whether or not to decide the estimation result. The decision unit 133 can decide whether or not to decide the estimation result from various viewpoints. The various judgment conditions are stored in advance (at the time of system production, etc.) in the storage unit 12, and are read out from the storage unit 12 at the time of judgment (step S12) and used for judgment. As a basic idea, when it is the timing to use the emotion estimation result for vehicle control, etc. (when emotion estimation result information is necessary for various controls), it is determined that it is necessary to decide the estimation result. Furthermore, when starting vehicle control using the estimated emotion, it is determined that it is necessary to decide the estimation result in order to make the initial control on the safe side (to increase the accuracy of the data used in the control).

[0070] If it is determined that the estimation result is not to be determined (No in step S12), the process returns to step S11. If it is determined that the estimation result is to be determined (Yes in step S12), the process proceeds to step S13.

[0071] In step S13, the feeling estimation unit 132 (controller 13) reads out from the storage unit 12 the biological signals acquired by the acquisition unit 131 during a period for feeling estimation (first predetermined period) going back from the timing at which it is determined in step S12 that the estimation result is to be determined. The feeling estimation unit 132 (controller 13) calculates an arousal level based on the biological signal output from the electroencephalogram sensor for each electroencephalogram sensor signal (read-out data of the electroencephalogram sensor signal) read out from the storage unit 12. The feeling estimation unit 132 (controller 13) calculates an autonomic nervous system activity level based on the biological signal output from the heartbeat sensor for each heartbeat sensor signal (read-out data of the heartbeat sensor signal) read out from the storage unit 12. When the arousal level and the autonomic nervous system activity level are calculated, the process proceeds to the next step S14.

[0072] In step S14, the feeling estimation unit 132 (controller 13) estimates a feeling based on the arousal level and the autonomic nervous system activity level calculated in step S13. The feeling estimated in step S14 becomes a candidate for the final estimation result.

[0073] Depending on the setting of the length of the period for emotion estimation (first predetermined period), one arousal level and one autonomic nervous system activity level may be calculated in step S13, or multiple arousal levels and multiple autonomic nervous system activity levels may be calculated in step S13. When multiple arousal levels and multiple autonomic nervous system activity levels are calculated in step S13, the emotion estimation unit 132 (controller 13) estimates the emotion based on the arousal levels and the autonomic nervous system activity levels obtained by statistical processing of the arousal levels and the autonomic nervous system activity levels. For example, the emotion estimation unit 132 estimates the emotion based on the average value of the multiple arousal levels calculated in step S13 and the average value of the multiple autonomic nervous system activity levels calculated in step S13.

[0074] When emotion is estimated in step S14 based on the arousal level and autonomic nervous system activity level calculated in step S13, the process proceeds to the next step S15.

[0075] In step S15, the feeling estimation unit 132 (controller 13) reads out from the storage unit 12 the biological signal acquired by the acquisition unit 131 in the period for emotion estimation pattern determination (second predetermined period). The period for emotion estimation pattern determination (second predetermined period) is a period in the past (may include the first predetermined period) before the period for emotion estimation (first predetermined period) and is a period immediately before the period for emotion estimation (first predetermined period). The period for emotion estimation pattern determination (second predetermined period) is longer than the period for emotion estimation (first predetermined period). The feeling estimation unit 132 (controller 13) calculates the degree of awakening based on the biological signal output from the electroencephalogram sensor for each electroencephalogram sensor signal (read data of the electroencephalogram sensor signal) read out from the storage unit 12. The feeling estimation unit 132 (controller 13) calculates the degree of autonomic nervous system activity based on the biological signal output from the heartbeat sensor for each heartbeat sensor signal (read data of the heartbeat sensor signal) read out from the storage unit 12. Then, the emotion estimation unit 132 (controller 13) calculates an emotion transition pattern (time-series information of a coordinate position (or a quadrant in which the emotion transition pattern is located) on the psychological plane (see FIG. 2) based on the arousal level and autonomic nervous system activity level at the same timing) from the calculated arousal level and autonomic nervous system activity level.

[0076] In this embodiment, the emotion transition pattern (time series information relating to arousal level and autonomic nervous system activity level) includes inter-quadrant transition information indicating that the coordinates plotted on the psychological plane shown in Figure 2 have moved from one quadrant to another quadrant.

[0077] In this embodiment, the emotion transition pattern (time series information relating to arousal level and autonomic nervous system activity level) includes information on the time required for inter-quadrant transition, which indicates the time required for the coordinates plotted on the psychological plane shown in Figure 2 to move from near a fixed point (area) in one quadrant to near a fixed point (area) in another quadrant.

[0078] In this embodiment, the emotion transition pattern (time-series information relating to arousal level and autonomic nervous system activity level) includes information on the time during which the coordinates plotted on the psychological plane shown in FIG. 2 stayed in each quadrant.

[0079] Once the emotion transition pattern (time-series information relating to the arousal level and the activity level of the autonomic nervous system) has been calculated, the process proceeds to the next step S16.

[0080] In step S16, the determination unit 133 (controller 13) determines whether to determine the estimation result. The determination unit 133 determines not to determine (update) the estimation result when it is determined that there is no change in emotion (change from the previous estimation result), and determines to determine (update) the estimation result in other cases.

[0081] For example, the determination unit 133 may determine whether to determine the estimation result based on the coordinates (emotion index value) plotted on the psychological plane by the emotion estimation unit 132. The determination unit 133 may determine to determine the estimation result when at least one of the multiple emotion index values ​​has changed. Here, the change is a change that satisfies a predetermined condition and does not include a change that does not satisfy the predetermined condition (e.g., a small change). The same applies to changes other than the emotion index value described below.

[0082] Specifically, for example, when the coordinates plotted on the psychological plane shown in Fig. 2 move from a quadrant other than the first quadrant to the first quadrant, the determination unit 133 may determine that the estimation result is determined. This is because the first quadrant of the psychological plane shown in Fig. 2 includes both positive emotions such as "fun" and "joy" and negative emotions such as "anger" and "sadness," and it is desirable to be able to distinguish between these emotions.

[0083] 2 moves from a region in the first quadrant where the distance from the origin is equal to or less than a certain value to a region in the first quadrant where the distance from the origin exceeds the certain value, the determination unit 133 may determine to determine the estimation result. This is because, when the coordinates move to a region in the first quadrant where the distance from the origin exceeds the certain value, it is highly likely that a certain emotion of the user U1 (one of the emotions corresponding to the first quadrant) has become stronger, and it is desirable for the estimation device 10 to accurately estimate the emotion after the change.

[0084] For example, the determination unit 133 may determine to determine the estimation result when the surrounding environment of the vehicle 20 changes. This is because when the surrounding environment of the vehicle 20 changes, the emotion of the user U1 is likely to change, and it is desirable for the estimation device 10 to accurately estimate the emotion after the change.

[0085] Specifically, for example, the acquisition unit 131 acquires a surrounding image of the vehicle 20 captured by an on-board camera mounted on the vehicle 20 via the communication unit 11. Then, the decision unit 133 may determine to decide on the estimation result when detecting a change in the surrounding environment, such as a change in weather or scenery (driving in a city, driving in the suburbs, driving on a mountain road, etc.), from the surrounding image.

[0086] More specifically, for example, the acquisition unit 131 acquires traffic information provided from a car navigation device mounted on the vehicle 20 via the communication unit 11. The decision unit 133 may determine to decide on the estimation result when detecting the occurrence of a peculiar surrounding traffic condition such as a traffic jam or a change in the surrounding traffic condition from the traffic information, vehicle speed information, or the like.

[0087] For example, the determination unit 133 may determine to determine the estimation result when the behavior of the vehicle 20 changes (there is a large change in the steering angle, the vehicle speed, or the acceleration). This is because when the behavior of the vehicle 20 changes, it is highly likely that the emotion of the user U1 has changed, and it is desirable for the estimation device 10 to accurately estimate the emotion after the change.

[0088] Specifically, for example, the acquisition unit 131 acquires information on the behavior of the vehicle 20 from a car navigation device, an ECU (Electronic Control Unit), or the like mounted on the vehicle 20 via the communication unit 11. Then, the decision unit 133 determines to decide on the estimation result when detecting the occurrence of sudden braking, speeding, or the like from the information.

[0089] For example, the determination unit 133 may determine to determine the estimation result when the vehicle 20 starts control using the estimation result of the estimation device 10.

[0090] Specifically, for example, when the vehicle 20 starts control using the estimation result of the estimation device 10, it outputs an estimation result request signal that requests the estimation result. The acquisition unit 131 acquires the estimation result request signal via the communication unit 11. Then, the decision unit 133 determines to decide the estimation result in response to the estimation result request signal.

[0091] If it is determined that the estimation result is not to be determined (updated) (No in step S16), the process proceeds to step S18, and the previously determined estimation result is read and used from the storage unit 12. If it is determined that the estimation result is to be determined (updated) (Yes in step S16), the process proceeds to step S17.

[0092] In step S17, the determination unit 133 (controller 13) determines an estimation result based on the calculated emotion transition pattern (time series information related to arousal and autonomic nervous system activity) from the emotion candidates estimated by the emotion estimation unit 132 in step S14, and stores the determined estimated emotion in the storage unit 12 (updates it as the latest estimated emotion). That is, the estimation device 10 further appropriately narrows down the emotion candidates estimated in step S14 using the emotion transition pattern of the user U1 (time series information related to arousal and autonomic nervous system activity). It is considered that the emotion of the user U1 is influenced by the emotion transition state of the user U1 up to the present time, and it is considered that there is a correlation between the emotion of the user U1 and the emotion transition pattern of the user U1 up to the present time (time series information related to arousal and autonomic nervous system activity). Therefore, high accuracy of the estimated emotion can be expected by narrowing down the emotion candidates estimated in step S14 based on the emotion transition pattern of the user U1 (time series information related to arousal and autonomic nervous system activity).

[0093] In this embodiment, the determination unit 133 estimates the emotion transition state (emotion transition pattern) of the user U1 based on time-series information related to multiple emotion index values ​​(arousal level, autonomic nervous system activity level) of the user U1. When the emotion transition state of the user U1 is estimated in this manner, the emotion transition state of the user U1 can also be estimated using the emotion index values ​​(arousal level, autonomic nervous system activity level) of the user U1 used to estimate emotion candidates in the past, so that the data acquisition burden on the estimation device 10 can be reduced.

[0094] Specifically, the determination unit 133 determines an estimation result from emotion candidates and emotion transition patterns (time-series information relating to arousal level and autonomic nervous system activity level) using the determination process table data shown in Fig. 6. The determination process table data shown in Fig. 6 is included in the table 121. That is, the determination process table data shown in Fig. 6 is stored in the storage unit 12.

[0095] As shown in FIG. 6, the items of the determination process table data include "emotion candidates" and "emotion transition patterns." The items of the table correspond to data storage cells (storage slots). Emotion transition pattern A shown in FIG. 6, for example, stays in the third quadrant for ○ seconds, then transitions to the first quadrant over △ seconds, and then stays in the first quadrant for x seconds. Emotion transition patterns B to E are similar to emotion transition pattern A, and only the specific transition states are different.

[0096] In this embodiment, the estimation result is determined using the determination process table data shown in FIG. 6, but the estimation result may be determined using a first emotion determination learning model.

[0097] For example, the determination unit 133 (controller 13) may have, instead of table data for determination processing, a first emotion determination learning model machine-learned with learning data in which input data are multiple types of emotion index values ​​(time-series information related to a period for emotion estimation pattern determination (second predetermined period)) generated based on biological signals detected from the data creation subject for creating learning data, and an estimated emotion estimated from the multiple types of emotion index values ​​(period for emotion estimation (first predetermined period)), and the emotion of the data creation subject at the corresponding timing is used as correct answer data. The correct answer data, which is the emotion of the data creation subject, is generated based on a declaration of emotion by the data creation subject (so-called: questionnaire survey).

[0098] The length of the period for acquiring a biosignal required to obtain time-series information (a period corresponding to the second predetermined period in the above-described embodiment) and the length of the period for acquiring a biosignal required to obtain an estimated emotion (a period corresponding to the first predetermined period in the above-described embodiment) may be set to an appropriate value (a value that increases the accuracy rate) through experiments, etc.

[0099] The determination unit 133 may apply time-series information relating to the multiple types of emotion index values ​​generated based on the biological signal detected from the user U1 and the multiple types of emotion index values ​​of the user U1 to the above-mentioned first emotion determination learning model, thereby determining an output emotion output from the first emotion determination learning model as an emotion estimation result.

[0100] The estimated emotion estimated from the multiple emotion index values, which are part of the input data for machine learning, may be estimated using the psychological plane shown in FIG. 2, as in the above-mentioned embodiment. The first to fourth quadrants of the psychological plane shown in FIG. 2 correspond to areas corresponding to the case classification of combinations of the binarized values ​​of the multiple emotion index values ​​of the user U1. The first quadrant of the psychological plane shown in FIG. 2 corresponds to an area corresponding to a combination of the positive value of the awakening degree of the user U1 and the positive value of the autonomic nervous system activity. The second quadrant of the psychological plane shown in FIG. 2 corresponds to an area corresponding to a combination of the positive value of the awakening degree of the user U1 and the negative value of the autonomic nervous system activity. The third quadrant of the psychological plane shown in FIG. 2 corresponds to an area corresponding to a combination of the negative value of the awakening degree of the user U1 and the negative value of the autonomic nervous system activity. The fourth quadrant of the psychological plane shown in FIG. 2 corresponds to an area corresponding to a combination of the negative value of the awakening degree of the user U1 and the positive value of the autonomic nervous system activity.

[0101] It should be noted that a second emotion determination learning model may be used in place of the above first emotion determination learning model.

[0102] The second emotion determination learning model is a learning model that uses time-series information related to multiple emotion index values ​​generated based on biological signals detected from a data creation subject for creating learning data as input data, and machine-learned learning data in which a selected emotion candidate selected from emotion candidates estimated based on multiple emotion index values ​​is used as correct answer data. The correct answer data, which is the emotion of the data creation subject, is generated based on the data creation subject's declaration of emotion (so-called questionnaire survey) in a format in which the emotion candidate is selected from emotion candidates estimated based on multiple emotion index values ​​of the data creation subject. This allows the second emotion determination learning model to also have the function of estimating emotion candidates through machine learning.

[0103] The determination unit 133 may determine, as an emotion estimation result, an output emotion output from the second emotion determination learning model by applying time-series information relating to a plurality of types of emotion index values ​​generated based on a biological signal detected from the user U1 to the above-mentioned second emotion determination learning model.

[0104] In the above-described embodiment and modified example, the use of a biosensor that detects the biosignal of the user U1 is assumed. However, in order to eliminate the trouble of wearing a biosensor, it is desirable to eliminate the need for a biosensor that detects the biosignal of the user U1. For this purpose, the feeling estimation unit 132 (controller 13) may have an emotion index value learning model that is machine-learned with learning data in which a photographed image of the data creation subject for creating learning data is used as input data, and an emotion index value calculated based on the biosignal of the data creation subject at the time of photographing the photographed image is used as correct answer data. Then, the multiple types of emotion index values ​​used in the emotion estimation unit 132 (controller 13) may be emotion index values ​​output from the emotion index value learning model by applying the photographed image of the user U1 to the emotion index value learning model.

[0105] When the estimation result is determined by the determination unit 133, the process proceeds to the next step S18.

[0106] In step S18, the providing unit 134 reads out the latest estimated emotion from the storage unit 12, and provides it to the on-board control device in the vehicle 20 via the network N1. The on-board control device then performs various controls on the on-board device in accordance with the provided estimated emotion.

[0107] When the estimation result is provided to the on-board control device in the vehicle 20 by the providing unit 134, the process returns to step S12, and the above-mentioned processes are repeated.

[0108] In addition, when the operation of the device that uses the emotion estimation result is terminated, for example, when the acquisition unit 131 of the estimation device 10 acquires a signal from the vehicle 20 notifying that the doors are locked (end of various controls while the vehicle is running), an interrupt process for terminating the estimation process shown in FIG. 5 is started, and the estimation process shown in FIG. 5 is terminated.

[0109] Through the above-described processing, estimated emotion candidates are selected based on the bio-signals of the user U1 (emotion estimation target person), and the estimated emotion is narrowed down and determined from the estimated emotion candidates based on the emotion transition state of the user U1.

[0110] <2. Points to note> Various technical features disclosed in the description of the invention may be modified in various ways without departing from the spirit of the technical creation. Furthermore, the embodiments and modifications disclosed in the description of the invention may be combined to the extent possible.

[0111] For example, the estimation process of the estimation device 10 may be started when the vehicle 20 starts control using the estimation result of the estimation device 10. In this way, the processing load on the estimation device 10 can be reduced. [Explanation of symbols]

[0112] 1. Estimation system 10... Estimation device 11. Communications Department 12...Storage section 13. Controller 20. Vehicle 21. Biometric sensor U1...User

Claims

1. An emotion estimation device, comprising: Estimating emotion candidates that are candidates for an emotion estimation result based on a plurality of types of emotion index values ​​of a subject of emotion estimation; determining the emotion estimation result from the emotion candidates based on an emotion transition state of the emotion estimation target;

2. The estimation device according to claim 1 , wherein an emotion transition state of the emotion estimation subject is estimated based on time-series information related to a plurality of types of emotion index values ​​of the emotion estimation subject.

3. 3. The estimation device according to claim 2, wherein the time-series information related to the plurality of emotion index values ​​of the emotion estimation subject includes at least one of transition information between regions corresponding to case classification of combinations of binarized values ​​of the plurality of emotion index values ​​of the emotion estimation subject, information about a time required for the transition between the regions, and information about a time spent in each of the regions.

4. an emotion determination learning model that is machine-learned using learning data that uses time-series information relating to a plurality of emotion index values ​​generated based on biological signals detected from a data creation subject for creating learning data and an estimated emotion estimated from the plurality of emotion index values ​​as input data and the emotion of the data creation subject as correct answer data; 4. The estimation device according to claim 2 or 3, wherein an output emotion output from the emotion determination learning model is determined as the emotion estimation result by applying to the emotion determination learning model time-series information relating to a plurality of types of emotion index values ​​generated based on a biological signal detected from the emotion estimation subject, and the plurality of types of emotion index values ​​of the emotion estimation subject.

5. the plurality of emotion index values ​​are an arousal state index value representing an arousal state and an emotion intensity index value representing an emotion intensity, 4. The estimation device according to claim 1, wherein the emotion candidate is estimated by applying the arousal state index value and the emotion intensity index value to an emotion model in which an emotion estimated value is determined by case-by-case classification of a combination of a binarized value of the arousal state index value and a binarized value of the emotion intensity index value.

6. an emotion index value learning model that is machine-learned using learning data in which a captured image of a data creation subject for creating learning data is used as input data, and an emotion index value calculated based on a biological signal of the data creation subject at the time the captured image is taken is used as correct answer data; The estimation device according to any one of claims 1 to 3, wherein the plurality of types of emotion index values ​​are emotion index values ​​output from the emotion index value learning model by applying a photographed image of the emotion estimation subject to the emotion index value learning model.

7. an emotion determination learning model is generated by machine learning using learning data in which time-series information related to a plurality of emotion index values ​​generated based on biological signals detected from a data creation subject for creating learning data is used as input data, and a selected emotion candidate selected from emotion candidates estimated based on the plurality of emotion index values ​​is used as correct answer data; a time-series information related to a plurality of emotion index values ​​generated based on a biological signal detected from a subject of emotion estimation to the emotion determination model, thereby determining an emotion output from the emotion determination learning model as the emotion estimation result.

8. An in-vehicle system including an emotion estimation device that estimates an emotion of a driver of a vehicle and an in-vehicle control device that controls an in-vehicle device, The emotion estimation device includes: Estimating emotion candidates that are candidates for an emotion estimation result based on a plurality of types of emotion index values ​​of a subject of emotion estimation; and determining the emotion estimation result from the emotion candidates based on an emotion transition state of the emotion estimation subject.

9. Further equipped with a camera for photographing the driver, the feeling estimation device has an emotion index value learning model that is machine-learned using learning data in which a captured image of a data creation subject for creating learning data is used as input data, and an emotion index value calculated based on a biological signal of the data creation subject at the time the captured image is taken is used as correct answer data; 9. The in-vehicle system according to claim 8, wherein the plurality of types of emotion index values ​​are emotion index values ​​output from the emotion index value learning model by applying an image of the driver captured by the camera to the emotion index value learning model.

10. 10. The in-vehicle system according to claim 8, wherein the emotion estimation result is determined in at least one of the following cases: when at least one of a plurality of emotion index values ​​of the emotion estimation subject has changed; when a surrounding environment of the vehicle has changed; when a behavior of the vehicle has changed; and when the vehicle starts control using the emotion estimation result.

11. Estimating emotion candidates that are candidates for an emotion estimation result based on a plurality of emotion index values ​​of an emotion estimation target person; determining the emotion estimation result based on an emotion transition state of the emotion estimation target from the emotion candidates; The estimation program causes a computer to execute the above.

Images