Estimation device, estimation system, method of estimation and estimation program
The estimation device improves emotion estimation accuracy by selecting candidates from biosignal data and using an emotion estimation model to distinguish between positive and negative emotions, addressing the limitations of existing systems.
Patent Information
- Application Number
- JP2023086245
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-25
- Publication Date
- 2024-12-05
AI Technical Summary
Existing emotion estimation systems struggle to accurately distinguish between positive and negative emotions, such as 'fun' and 'joy' versus 'anger' and 'sadness', leading to difficulties in utilizing the estimation results effectively.
An estimation device that acquires related information from biosensors, selects emotion candidates based on multiple types of biological signals, and determines the emotion estimation result using an emotion estimation model to enhance accuracy.
The system achieves high accuracy in emotion estimation by narrowing down emotion candidates based on appropriate related information, enabling precise emotion determination.
Smart Images

Figure 2024169061000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a technique for estimating emotions. [Background technology]
[0002] 2. Description of the Related Art Conventionally, there is known a device that estimates a person's emotion based on a biosignal measured by a biosensor and displays the estimated emotion (see, for example, Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2019-63324 A Summary of the Invention [Problem to be solved by the invention]
[0004] Depending on the emotion estimation model used in the above device, the emotion estimation result may be, for example, a psychological state of "joy, happiness, anger, or sadness" (an estimation result indicating that the user is in one of these psychological states).
[0005] The psychological state estimation result of "fun, joy, anger, sadness" may indicate that the subject is feeling "fun," "joy," "anger," or "sadness," and there is a problem that the estimation result is difficult to use for various controls due to restrictions. In particular, in the case of an estimation result that includes the possibility of both positive emotions such as "fun" and "joy" and negative emotions such as "anger" and "sadness," the estimation result is difficult to use. In other words, even though positive emotions and negative emotions are opposite emotions, if it is not possible to distinguish whether the emotion is positive or negative, it may be difficult to use the emotion estimation result depending on the mode of using the emotion estimation result.
[0006] In view of the above, an object of the present invention is to provide a technique that can more appropriately narrow down emotions estimated based on a biosignal output from a biosensor. [Means for solving the problem]
[0007] An exemplary estimation device of the present invention is an estimation device that estimates emotions, obtains relevant information related to the emotions of a subject of emotion estimation, estimates emotion candidates that are candidates for the emotion estimation result based on a plurality of types of biological signals, and determines the emotion estimation result from the emotion candidates based on the relevant information. Effect of the Invention
[0008] According to the exemplary present invention, there are often multiple emotion candidates estimated based on biosignals measured by a biosensor, but the accuracy of the estimation can be expected to be high. Since an estimated emotion, which is an emotion estimation result, can be determined from among the emotion candidates based on appropriate related information, a high accuracy of the estimated emotion can be expected. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an estimation system according to first and second embodiments; [Diagram 2] A diagram showing an example of an emotion estimation model (psychological plane) [Diagram 3] FIG. 1 shows an example of the configuration of an estimation device. [Figure 4A] FIG. 13 is a diagram showing an example of a sensor table. [Figure 4B] A diagram showing an example of a psychological plane table. [Diagram 5] 1 is a flowchart showing an example of an estimation process executed by an estimation device; [Figure 6] FIG. 13 is a diagram showing an example of an evaluation table. [Figure 7] FIG. 1 is a diagram showing an example of a weighting table. [Figure 8] FIG. 13 is a diagram showing an example of the configuration of an estimation system according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.
[0011] <1. First embodiment> [1-1. Estimation system] 1 is a diagram showing an example configuration of an estimation system 1 according to a first exemplary embodiment of the present invention. In this embodiment, a user U1 is an occupant of a vehicle 20. The estimation system 1 is configured as a system that estimates the emotion of the user U1.
[0012] 1 shows only one user U1, there may be multiple users U1. For example, user U1 may be an e-sports player, a patient at a medical institution, a student at an educational institution, a viewer of content such as video or music, etc. In the case of these users, some components of the estimation system 1 (biometric sensor, camera, microphone, etc.) will be installed at the location where user U1 is present, for example, the location where the e-sports operation terminal is installed, a hospital, a school, a content viewing room, etc.
[0013] 1, the estimation system 1 includes a server 10 (computer), a biosensor 21, a camera 22, and a microphone 23. In this embodiment, the server 10 is disposed outside the vehicle 20, for example, in an emotion estimation service center building or the like, and the biosensor 21, the camera 22, and the microphone 23 are disposed inside the vehicle 20.
[0014] The server 10 may be a physical server or a virtual server. In this embodiment, the server 10 constitutes an estimation device that estimates emotions. That is, the estimation system 1 includes the estimation device 10. Hereinafter, the server 10 will be referred to as the estimation device 10. Details of the estimation device 10 will be described later. Note that the estimation device 10 may be constituted by one server or multiple servers.
[0015] The biosensor 21 detects a plurality of types of biosignals of the user U1 as sensor signals. That is, the estimation system 1 includes the biosensor 21 that measures a plurality of types of biosignals.
[0016] In this embodiment, the biosensor 21 includes a first sensor and a second sensor. The first sensor is a brain wave sensor. The second sensor is a heart rate (pulse) sensor. However, the first sensor and the second sensor may be changed to other biosensors depending on the bioinformation to be acquired. The other biosensor may be, for example, a blood pressure monitor, a NIRS (Near Infrared Spectroscopy) device, etc.
[0017] The biosensor 21 may be composed of only a sensor worn by the user U1, for example. The biosensor 21 may also be composed of a sensor worn by the user U1 and a sensor equipped in the vehicle 20, for example. The biosensor 21 may also be composed of only a sensor equipped in the vehicle 20, for example. The sensor worn by the user U1 may be, for example, a headgear-type brainwave sensor, a wristband-type optical heart rate (pulse) sensor, or the like. The sensor equipped in the vehicle 20 may be, for example, an optical heart rate (pulse) sensor installed on the handlebar, or the like.
[0018] The biosensor 21 may be configured as a single sensor, not a plurality of sensors, so long as it is capable of measuring a plurality of types of biosignals of the user U1.
[0019] It is also possible to apply a device (detection method) that estimates a biosignal (or an emotion index value based on a biosignal) from a facial image (such as a change in facial expression or blood color) of the user U1 captured by the camera 22. For example, it is also possible to apply an AI (artificial intelligence) model trained with a data set of learning data in which the facial image of the user U1 is used as input data and the biosignal (or the emotion index value based on the biosignal) at that time is used as a correct answer value as a biosensor. In other words, a captured image of the emotion estimation subject for creating learning data is used as input data, and a trained biosignal learning model that has been machine-learned with learning data in which the biosignal of the emotion estimation subject at the time of capturing the captured image is used as correct answer data is created, and the captured image of the emotion estimation subject is applied to the created biosignal learning model to estimate the biosignal of the emotion estimation subject (an estimated biosignal is output from the biosignal learning model).
[0020] The camera 22 captures images of the interior of the vehicle 20. The camera 22 may be, for example, a camera equipped in the vehicle 20, a camera mounted on a smartphone brought into the interior of the vehicle 20, or the like.
[0021] The microphone 23 generates and outputs audio information corresponding to sounds generated in the cabin of the vehicle 20, including the voice emitted by the user U1. The microphone 23 may be, for example, a microphone equipped in the vehicle 20 (which may be used in conjunction with other devices), a microphone mounted on a smartphone brought into the cabin of the vehicle 20, or the like.
[0022] The camera 22 and the microphone 23 are each an example of an output device that outputs relevant information related to the emotion of the user U1. Also, the camera 22 and the microphone 23 are each an example of a sensor that is disposed around the user U1 and does not come into contact with the user U1.
[0023] Here, the flow of processing in the estimation system 1 will be described with reference to FIG.
[0024] The biosensor 21 measures multiple types of biosignals of a user U1 who is a target of emotion estimation. A sensor signal, which is a measurement result of the biosensor 21, is transmitted to the estimation device (server) 10 via a network N1 (step S1). For example, a communication device equipped in the vehicle 20 may be involved in transmitting the sensor signal from the biosensor 21 to the estimation device 10. The estimation device 10 estimates an emotion of the user U1, which is a candidate for an estimation result, based on the multiple types of biosignals (step S2).
[0025] Here, an overview of the emotion estimation in step S2 will be described. The estimation device 10 generates an index value, which is a value of an index (physiological response) indicating a mental and physical state, based on an input sensor signal. In this embodiment, the estimation device 10 generates index values of indexes indicating two mental and physical states related to electroencephalograms and heartbeat. For example, the index indicating the mental and physical state related to electroencephalograms is a central nervous system arousal level (hereinafter, simply referred to as arousal level), and the index value can be given by "β waves / α waves of electroencephalograms". Also, for example, the index indicating the mental and physical state related to heartbeats is an activity level of the autonomic nervous system, and the index value can be given by "standard deviation of heartbeat LF (low frequency range) component". The index value is calculated using a calculation model (calculation formula or conversion data table) stored in advance in a memory or the like.
[0026] The biosensor 21 may be configured to have a calculation function (a configuration having a built-in computer or the like) and to calculate the index value.
[0027] The estimation device 10 estimates emotion candidates of the user U1, which are candidates for the estimation result, by using the calculated index value and an emotion estimation model stored in advance in a memory or the like. The emotion estimation model is created based on medical evidence (papers, etc.).
[0028] FIG. 2 is a diagram showing an example of an emotion estimation model (mental plane). According to various medical evidence related to psychology, it is said that mentality can be estimated based on two types of indices that indicate physical conditions. The psychological plane with the two types of indices of mental and physical conditions shown in FIG. 2 as axes is an example of an emotion estimation model according to the technical idea. In FIG. 2, the vertical axis is "arousal level (arousal-unaroused)" and the horizontal axis is "activity level of the autonomic nervous system (sympathetic nervous activity (strong emotion)-parasympathetic nervous activity (weak emotion)" and is an example of an emotion estimation model devised by the applicant of the present application that is expected to have high estimation accuracy.
[0029] In the psychological plane shown in Figure 2, each of the four quadrants separated by the vertical and horizontal axes is assigned to a corresponding psychological state. The distance from each axis indicates the intensity of the corresponding psychological state. In the example of Figure 2, the psychological states of "fun, joy, anger, and sadness" are assigned to the first quadrant. The psychological state of "melancholy" is assigned to the second quadrant. The psychological states of "relaxation and calm" are assigned to the third quadrant. And the psychological states of "anxiety, fear, and discomfort" are assigned to the fourth quadrant.
[0030] The psychological state can be estimated from the coordinates obtained by plotting the index values of two types of mental and physical states obtained based on the measurement results of biosignals on a psychological plane. Specifically, the psychological state and its intensity can be estimated based on which quadrant of the psychological plane the plotted coordinates are in and how far they are from the origin.
[0031] In this example, in a model in which the vertical axis represents "alertness (awake-unaroused): calculated, for example, based on the beta / alpha waves of the brainwaves" and the horizontal axis represents "activity of the autonomic nervous system (sympathetic activity (strong emotion)-parasympathetic activity (weak emotion): calculated, for example, based on the standard deviation of the heart rate LF component," the subject's emotions are estimated to be "fun, joy, anger, sadness," "melancholy," "relaxed, calm" or "anxiety, fear, unpleasantness" depending on which quadrant the subject's state is in. This model has been verified as significant through experiments etc. by the applicant.
[0032] In the example shown in FIG. 2, the emotion estimation model is a two-dimensional plane, but it may be a three- or higher-dimensional space.
[0033] In addition, since the error in emotion estimation is larger than that in physical measurements, etc., it is possible to take into account the error in the emotion estimation result and each parameter / index related to emotion estimation, set a neutral region NR as shown in Figure 2, and make a judgment that emotion estimation is impossible (no emotion).
[0034] Returning to FIG. 1, the process flow of the estimation system 1 will be described below. When a predetermined condition is satisfied, the estimation device 10 requests related information related to the emotion of the user U1 from the camera 22 and the microphone 23 via the network N1 (step S3). The predetermined condition will be described later. For example, a communication device equipped in the vehicle 20 is involved, and the camera 22 and the microphone 23 receive the request from the estimation device 10.
[0035] In response to a request from the estimation device 10, the camera 22 transmits a captured image corresponding to the scenery inside the vehicle 20 as related information to the estimation device 10 via the network N1, and the microphone 23 transmits audio information corresponding to sounds generated inside the vehicle 20 as related information to the estimation device 10 via the network N1 (step S4). For example, a communication device equipped in the vehicle 20 may be involved and audio information may be transmitted from the microphone 23 to the estimation device 10. The scenery inside the vehicle 20 becomes a captured image in its original form or in a form that has been subjected to appropriate image processing (facial expression recognition, etc.). The sounds generated inside the vehicle 20 become audio information in its original form (audio waveform) or in a form that has been subjected to appropriate audio processing (audio recognition, frequency analysis, audio intensity analysis, etc.).
[0036] The estimation device 10 determines an estimation result based on the emotion candidates obtained in step S2 and the related information received in step S4 (step S5). The estimation device 10 provides (transmits) the determined estimation result to the vehicle 20 (step S6).
[0037] Then, the control device mounted on the vehicle 20 controls the vehicle 20 using, for example, the estimation result provided by the estimation device 10. For example, an angry driver tends to operate the accelerator, brake, and steering angle roughly. Therefore, when the user U1 is the driver and the estimation result is "anger," the control device mounted on the vehicle 20 performs control to reduce sensitivity to the accelerator, brake, and steering angle operations (to convert them into operation signals that change gently).
[0038] In another application example, when the user U1 is a sports player, the estimation result can be used for the mental training of the user U1. For example, when the user U1 feels unfavorable emotions (anxiety, anger, etc.) during playing a video game, it is determined that intensive training corresponding to the emotional state is necessary. Therefore, the estimation result of the estimation device 10 is used in the training by providing the player, etc. with information on the estimated emotions in each scene of the sport, or by providing appropriate advice information on the estimated emotions.
[0039] In addition, various types of e-sports include a type in which multiple users U1 play together and a type in which multiple users U1 play against each other. In such types of e-sports, if the emotional state of each player is displayed, it becomes possible to perform advanced game play, such as changing game tactics according to the emotional state. In addition, it is also possible to configure the e-sports spectators to understand the emotional state of each player while watching the game, which can increase the fun factor of watching the game.
[0040] Furthermore, if the user U1 is a patient at a medical institution, the estimation result can be used for examinations, treatment, etc. For example, the staff at the medical institution can understand that the patient is feeling anxious based on the estimated emotion and provide countermeasures such as counseling.
[0041] In addition, if the user U1 is a student in an educational institution, the estimation result can be used to improve the content of lessons. For example, a teacher can understand that a student finds a lesson boring based on the estimated emotion and improve the content of the lesson to make it more interesting for the student.
[0042] Furthermore, if the user U1 is a viewer of content such as video or music, the estimation result can be used to create further content. For example, a distributor of video content can create a highlight video by collecting scenes that viewers found enjoyable based on the estimated emotions.
[0043] In the above, an example has been given in which the server 10 disposed outside the vehicle 20 is the estimation device, but the estimation device may be mounted on the vehicle 20. In addition, for example, the estimation device may be composed of a plurality of devices. For example, a device disposed outside the vehicle 20 and a device mounted on the vehicle 20 may perform a process for estimating emotions in a distributed manner.
[0044] [1-2. Estimation device] FIG. 3 is a diagram showing an example of the configuration of an estimation device 10 according to a first exemplary embodiment of the present invention. Note that FIG. 3 shows components necessary for explaining the features of this embodiment, and a description of general components is omitted. As shown in FIG. 3, the estimation device 10 includes a communication unit 11 and a storage unit 12. The estimation device 10 also includes a controller 13. The estimation device 10 may be a so-called computer device. Note that the estimation device 10 may be configured to include an input device such as a keyboard and an output device such as a display.
[0045] The communication unit 11 is an interface for communicating data with other devices via the network N1. The communication unit 11 is, for example, a network interface card (NIC).
[0046] The storage unit 12 includes a volatile memory and a non-volatile memory. The volatile memory may include, for example, a random access memory (RAM). The non-volatile memory may include, for example, a read only memory (ROM), a flash memory, or a hard disk drive. The non-volatile memory stores computer-readable programs and data. At least a part of the programs and data stored in the non-volatile memory may be obtained from another computer device connected by wire or wirelessly, or from a portable recording medium.
[0047] As shown in Fig. 3, in this embodiment, the storage unit 12 includes a table (data table) 121. In detail, the table 121 includes a plurality of tables for various processes. For example, the table 121 includes a sensor table 121a and a psychological plane table 121b. Fig. 4A is a diagram showing an example of the sensor table 121a. Fig. 4B is a diagram showing an example of the psychological plane table 121b.
[0048] 4A, the items of the sensor table 121a include “sensor ID”, “sensor type”, “biological signal type”, “corresponding index ID”, “corresponding index type”, and “index conversion information”. The items of the table correspond to data storage cells (storage slots).
[0049] The item "sensor ID" of the sensor table 121a stores sensor ID data, which is identification information for identifying a data record in the sensor table 121a. The sensor ID data is also a primary key of the data record in the sensor table 121a. That is, in the sensor table 121a, a data record is configured for each sensor ID data, and data of each item associated with the sensor ID data is stored in the data record.
[0050] The item "sensor type" of the sensor table 121a stores information for identifying the type of biosensor. In this example, the sensor name (or data such as model number) is stored.
[0051] The item "biological signal type" of the sensor table 121a stores the type of measurement value based on the biological signal detected by the biological sensor. This biological signal type data is data correlated with the corresponding index type data. Academically, it is recognized that the corresponding index type data can be estimated (calculated) by acquiring the biological signal type data corresponding thereto.
[0052] The item "corresponding index ID" of the sensor table 121a stores identification information for identifying a mental and physical state index generated (calculated) based on a signal from a biosensor that detects a biosignal. The item "corresponding index type" of the sensor table 121a stores the type (such as name) of the index.
[0053] The item "index conversion information" of the sensor table 121a stores conversion information (such as an arithmetic formula or a conversion data table) for calculating an index value based on a signal obtained from a biosensor that detects a biosignal. In other words, the index value of the mental and physical state index identified by the corresponding index ID is estimated (calculated) by converting the biosignal detected by the biosensor corresponding to the sensor ID data according to the index conversion information.
[0054] For example, the data record of the sensor ID "SN01" in the sensor table 121a shown in Fig. 4A has the following information. "Electroencephalogram beta / alpha waves" are measured by the output signal of "electroencephalogram sensor BA". Then, this "electroencephalogram beta / alpha waves" is converted using the index conversion information of "FX01" to obtain an index value of "wakefulness".
[0055] As shown in Fig. 4B, the psychological plane table 121b is a two-dimensional matrix table with index types (specifically, index ID data is used) as parameters of the vertical and horizontal axes. In the psychological plane table 121b, data of psychological plane types that can be used with the two index type data are stored in memory cells (memory frames) determined by two index types. For example, if the index types used as indexes are index type VSm and index type VSn, the psychological plane used for emotion estimation is psychological plane mn. Information for performing processing using psychological plane mn is read out and used for emotion estimation processing.
[0056] In addition, the sensor table 121a and the psychological plane table 121b use a common index ID. That is, based on the sensor table 121a and the psychological plane table 121b, a psychological plane mn corresponding to two types of biosensors attached to the user U1 can be determined. For example, the types of sensors attached to the user U1 are "brain wave sensor BA (index ID: VS01)" and "heartbeat sensor HA (index ID: VS02)". In this case, index ID data corresponding to "brain wave sensor BA (index ID: VS01)" and "heartbeat sensor HA (index ID: VS02)" is determined based on the sensor table 121a. Then, based on the psychological plane table 121b, the psychological plane 01-02 with "arousal level" (VS01) and "autonomic nervous system activity level" (VS02) as indexes is determined as the psychological plane to be used for emotion estimation.
[0057] Returning to FIG. 3, an example of the configuration of the estimation device 10 will be described. The controller 13 includes a processor that performs arithmetic processing and the like. The processor may include, for example, a CPU (Central Processing Unit). The controller 13 may be configured with one processor or multiple processors. When configured with multiple processors, the processors may be connected to each other so that they can communicate with each other. When the estimation device 10 is configured as a cloud server, the CPU that configures the processor may be a virtual CPU.
[0058] 3, the controller 13 includes, as its functions, an acquisition unit 131, a feeling estimation unit 132, a request unit 133, a determination unit 134, and a providing unit 135. In this embodiment, the functions of the controller 13 are realized by a processor executing arithmetic processing in accordance with a program stored in the storage unit 12.
[0059] The scope of the present embodiment may include a computer program that causes a processor (computer) to realize at least a part of the functions of the estimation device 10. The scope of the present embodiment may also include a computer-readable non-volatile recording medium that records such a computer program. The non-volatile recording medium may be, for example, an optical recording medium (e.g., an optical disk), a magneto-optical recording medium (e.g., a magneto-optical disk), a USB memory, or an SD card, in addition to the above-mentioned non-volatile memory.
[0060] Further, each of the functional units 131 to 135 may be realized by one program, or may be realized by a separate program for each functional unit. Further, each of the functional units 131 to 135 may be realized as a separate server. Further, each of the functional units 131 to 135 may be realized by having a processor execute a program, that is, by software, as described above, or may be realized by other methods. At least a part of each of the functional units 131 to 135 may be realized by using, for example, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like. That is, each of the functional units 131 to 135 may be realized by hardware using a dedicated IC, or the like. Further, each of the functional units 131 to 135 may be realized by using both software and hardware. Further, each of the functional units 131 to 135 is a conceptual component. A function executed by one component may be distributed to multiple components. Further, functions possessed by multiple components may be integrated into one component.
[0061] The acquisition unit 131 acquires the sensor signal output from the biosensor 21, the captured image output from the camera 22, and the audio information output from the microphone 23 via the communication unit 11. The acquisition unit 131 stores the received data in the storage unit 12 as necessary for subsequent processing.
[0062] The emotion estimation unit 132 selects a model to be used for estimating emotion candidates based on the sensor signal acquired by the acquisition unit 131. Specifically, the emotion estimation unit 132 compares the acquired sensor signal with the sensor table 121a to extract data of the “corresponding index type” corresponding to the sensor type and the biological signal type included in the acquired sensor signal (index type data). Then, the emotion estimation unit 132 compares the “index ID” of the extracted index type data with the psychological plane table 121b to select a model (psychological plane) to be used for estimating emotion candidates. For example, when the sensor IDs are “SN01” (sensor type: electroencephalogram sensor BA) and “SN02” (sensor type: heart rate sensor HA), the corresponding index IDs are “VS01” (index type: arousal level) and “VS02” (index type: autonomic nervous system activity level) (see FIG. 4A). Based on these, the emotion estimation unit 132 selects the psychological plane 01-02 as the emotion estimation model to be used.
[0063] When the type of sensor to be used by the estimation device 10 is predetermined (when the sensor type is fixedly connected (installed) to the estimation device 10, for example), the emotion estimation model to be used is also a predetermined emotion estimation model (a selection process for the emotion estimation model is not required).
[0064] The emotion estimation unit 132 executes emotion candidate estimation processing using the selected psychological plane. The emotion estimation unit 132 converts information based on the biological signal acquired by the acquisition unit 131 into an index value using the index conversion information in the sensor table 121a. In this embodiment, an index value of the arousal level and an index value of the autonomic nervous system activity level are obtained using data acquired from the first sensor (sensor type: electroencephalogram sensor BA) and the second sensor (sensor type: heart rate sensor HA).
[0065] Then, the emotion estimation unit 132 estimates emotion candidates according to the coordinate positions obtained by plotting each index value on the selected psychological plane. In this embodiment, emotion candidates are estimated according to the coordinate positions obtained by plotting the index value of the arousal level and the index value of the autonomic nervous system activity level on a psychological plane (see FIG. 2) with the arousal level on the vertical axis and the autonomic nervous system activity level on the horizontal axis. Note that, as a specific method for determining the area in which the coordinates of each index value are located (area in which emotion candidates are set), a method of calculating the positional relationship between each area boundary and each index value based on each function formula corresponding to each area boundary, or a method of extracting an area corresponding to each combination of index values using a data table simulating a psychological plane (a data table in which area values (corresponding to emotion candidates) for each combination of index values are stored), or the like, can be applied. Then, as a method for determining emotion candidates from the area determined in this way, a method of determining emotion candidates corresponding to an area using a data table in which areas and emotion candidates are associated with each other can be applied.
[0066] Moreover, the feeling estimation unit 132 obtains a plurality of types of index values. In this embodiment, the plurality of types is two. However, a configuration may be adopted in which three or more types of index values are obtained. In this case, the feeling estimation model may be a three or more dimensional space, rather than a two-dimensional plane.
[0067] In order to acquire related information for use in the process of narrowing down emotion candidates, the request unit 133 requests captured images from the camera 22 and requests audio information from the microphone 23 via the communication unit 11.
[0068] The determination unit 134 determines an estimation result from the emotion candidates estimated by the emotion estimation unit 132, i.e., emotion candidates that are candidates for the estimation result, based on the related information (captured image and audio information) acquired by the acquisition unit 131.
[0069] The providing unit 135 provides the estimation result determined by the determining unit 134 to an in-vehicle device in the vehicle 20 via the network N1.
[0070] [1-3. Estimation method] Next, a method for estimating an emotion executed by the estimation device 10 will be described.
[0071] The estimation method of the present embodiment can be realized by a computer program that causes a computer device to operate in accordance with the estimation method. Such a computer program is provided to an administrator of the estimation device 10 by a computer-readable non-volatile recording medium or by using a communication network (so-called download), and is installed in the estimation device 10. This is included in the scope of the present embodiment. The computer program that causes a computer device to realize the estimation method of the present embodiment may be composed of only one program, or may be composed of multiple programs.
[0072] Fig. 5 is a flowchart showing an example of an estimation process executed by the estimation device 10. The estimation process shown in Fig. 5 is started when emotion estimation is started, for example, when the acquisition unit 131 of the estimation device 10 acquires a signal notifying door unlocking from the vehicle 20 in order to estimate the emotion of the user from the time the user gets into the vehicle and use the emotion for various vehicle controls.
[0073] In step S11, the feeling estimation unit 132 (controller 13) estimates a feeling based on a sensor signal (biological signal) output from the biosensor 21. The feeling estimated in step S11 becomes a candidate for a final estimation result. When the feeling estimation unit 132 estimates a feeling, the process proceeds to the next step S12.
[0074] In step S12, the request unit 133 (controller 13) judges whether or not to request related information. The request unit 133 can judge whether or not to request related information from various viewpoints. The various judgment conditions are stored in advance (at the time of system production, etc.) in the storage unit 12, and are read out from the storage unit 12 at the time of judgment (step S12) and used for judgment. As a basic idea, when there is no need to narrow down the estimated emotion that will be the final estimation result from the estimated emotion candidates, or when it is judged that there is no change in emotion (change from the previous estimation result) such as a small change in the biosignal, it is judged that there is no need to request related information. Also, when starting vehicle control using the estimated emotion, it is judged that there is a need to request related information in order to make the initial control on the safe side (to increase the accuracy of the data used in the control).
[0075] For example, the request unit 133 may determine whether to request related information based on the coordinates (biological signal values) plotted on the psychological plane by the emotion estimation unit 132. The request unit 133 may determine to request related information when at least one of the multiple types of biological signals has changed. Here, the change is a change that satisfies a predetermined condition and does not include a change that does not satisfy the predetermined condition (e.g., a small change). The same applies to changes other than biological signals described below.
[0076] Specifically, for example, when the coordinates plotted on the psychological plane shown in Fig. 2 move from a quadrant other than the first quadrant to the first quadrant, the request unit 133 may determine that related information is requested. This is because the first quadrant of the psychological plane shown in Fig. 2 includes both positive emotions such as "fun" and "joy" and negative emotions such as "anger" and "sadness," and it is desirable to be able to distinguish between these.
[0077] 2 moves from a region in the first quadrant where the distance from the origin is equal to or less than a certain value to a region in the first quadrant where the distance from the origin exceeds the certain value, the request unit 133 may determine that related information is requested. This is because, when the coordinates move to a region in the first quadrant where the distance from the origin exceeds the certain value, it is highly likely that a certain emotion of the user U1 (one of the emotions corresponding to the first quadrant) has become stronger, and it is desirable for the estimation device 10 to accurately estimate the emotion after the change.
[0078] For example, the request unit 133 may determine to request related information when the surrounding environment of the vehicle 20 changes. This is because when the surrounding environment of the vehicle 20 changes, the emotion of the user U1 is likely to change, and it is desirable for the estimation device 10 to accurately estimate the emotion after the change.
[0079] Specifically, for example, the acquisition unit 131 acquires a surrounding image of the vehicle 20 captured by an on-board camera mounted on the vehicle 20 via the communication unit 11. Then, the request unit 133 may determine to request related information when detecting a change in the surrounding environment, such as a change in weather or scenery (driving in a city, driving in the suburbs, driving on a mountain road, etc.), from the surrounding image.
[0080] More specifically, for example, the acquisition unit 131 acquires traffic information provided from a car navigation device mounted on the vehicle 20 via the communication unit 11. The request unit 133 may determine to request related information when detecting the occurrence of a peculiar surrounding traffic condition such as a traffic jam or a change in the surrounding traffic condition from the traffic information, vehicle speed information, or the like.
[0081] For example, the request unit 133 may determine to request related information when the behavior of the vehicle 20 changes (there is a large change in the steering angle, the vehicle speed, or the acceleration). This is because when the behavior of the vehicle 20 changes, it is highly likely that the emotion of the user U1 has changed, and it is desirable for the estimation device 10 to accurately estimate the emotion after the change.
[0082] Specifically, for example, the acquisition unit 131 acquires information about the behavior of the vehicle 20 from a car navigation device, an ECU (Electronic Control Unit), or the like mounted on the vehicle 20 via the communication unit 11. Then, the request unit 133 determines to request related information when detecting the occurrence of sudden braking, speeding, or the like from the information.
[0083] For example, the request unit 133 may determine to request the related information when the vehicle 20 starts control using the estimation result of the estimation device 10.
[0084] Specifically, for example, when the vehicle 20 starts control using the estimation result of the estimation device 10, it outputs an estimation result request signal for requesting the estimation result. The acquisition unit 131 acquires the estimation result request signal via the communication unit 11. Then, the request unit 133 determines to request related information in response to the estimation result request signal.
[0085] If the related information is not requested (No in step S12), the process returns to step S11. If the related information is requested (Yes in step S12), the process proceeds to step S13.
[0086] In step S13, the request unit 133 requests the related information from the camera 22 and the microphone 23 via the communication unit 11. After the request unit 133 requests the related information, the process proceeds to the next step S14.
[0087] In step S14, the acquisition unit 131 acquires the related information output from the camera 22 and the microphone 23 via the communication unit 11. After the acquisition unit 131 acquires the related information, the process proceeds to the next step S15.
[0088] In step S15, the determination unit 134 determines an estimation result from the emotion candidates estimated by the emotion estimation unit 132 in step S11 based on the related information acquired by the acquisition unit 131 in step S14. That is, the estimation device 10 can further appropriately narrow down the emotion candidates estimated in step S11 by using the related information. Specifically, the determination unit 134 determines an estimation result from the emotion candidates estimated by the emotion estimation unit 132 in step S11 based on the facial expression of the user U1 analyzed from the captured image acquired by the acquisition unit 131 in step S14, the dialogue content of the user U1 analyzed from the voice information acquired by the acquisition unit 131 in step S14, and the volume of the sound emitted by the user U1 analyzed from the voice information acquired by the acquisition unit 131 in step S14. Note that the dialogue of the user U1 may be a dialogue between the user U1 and another passenger, or may be a dialogue between the user U1 and a dialogue system such as a smart speaker.
[0089] For example, the determination unit 134 determines the estimation result by referring to the evaluation table 121c and the weighting table 121d. The evaluation table 121c and the weighting table 121d are included in the table 121.
[0090] Fig. 6 is a diagram showing an example of the evaluation table 121c. As shown in Fig. 6, the items of the evaluation table 121c include "emotion", "facial expression", "content of dialogue", and "volume of sound".
[0091] The item "emotion" of the evaluation table 121c stores emotions included in psychological states assigned to each quadrant of each psychological plane.
[0092] The evaluation item "facial expression" of the evaluation table 121c stores facial expressions classified into a plurality of patterns.
[0093] The evaluation item "conversation content" of the evaluation table 121c stores, for example, the content of words included in the conversation.
[0094] The evaluation item "loudness" of the evaluation table 121c stores the range of loudness, which is one element of sound. Note that elements of sound other than "loudness", such as "pitch", "tone", and "intonation (fluctuation of volume)", may be added to the items of the evaluation table 121c.
[0095] In addition, these data in the evaluation table 121c are generated based on data indicating the relationship between emotions, facial expressions, dialogue contents, volume of sound (voice), etc., collected by, for example, a designer / developer of the emotion estimation device 10 system in an experiment, etc. In addition, the evaluation table 121c may be a general-purpose evaluation table unrelated to a user, or may be an individual evaluation table created for each user according to the user characteristics.
[0096] 7 is a diagram showing an example of the weighting table 121d. As shown in FIG 7, in the weighting table 121d, a weighting coefficient is determined for each evaluation item in the evaluation table 121c shown in FIG 6.
[0097] The weighting coefficient is determined based on the frequency of past matches (or similarities) with the stored contents of the evaluation items in the evaluation table 121c shown in FIG. 6. For example, if user U1 is the type of person whose emotions are easily expressed in facial expressions, the value of the weighting coefficient corresponding to the evaluation item "facial expressions" will be large. Also, for example, if the content of the conversation within the vehicle 20 has a large impact on the emotional changes of user U1, the value of the weighting coefficient corresponding to the evaluation item "conversation content" will be large. Also, for example, if user U1 is the type of person whose voice volume easily changes depending on his / her emotions, the value of the weighting coefficient corresponding to the evaluation item "sound volume" will be large.
[0098] These data are generated based on data indicating the degree of influence of facial expressions, dialogue contents, sound (voice) volume, etc. on emotions, collected by, for example, a designer / developer of the emotion estimation device 10 system through experiments, etc. The weighting table 121d may be a general-purpose weighting table unrelated to a user, or may be an individual weighting table created for each user according to the user characteristics.
[0099] The determination unit 134 then calculates an evaluation score for each estimated emotion candidate based on the degree of agreement for each evaluation item in the evaluation table 121c. The determination unit 134 then multiplies the evaluation score for each evaluation item by the weighting coefficient for each evaluation item in the weighting table 121d. The determination unit 134 then determines the sum of these multiplication results for each emotion candidate as an evaluation value for each emotion candidate, and determines the emotion candidate with the highest evaluation value as the estimated emotion.
[0100] For example, if the "candidates for the estimation result" are "joy, happiness, anger, and sadness," the determination unit 134 assigns a score to each of the evaluation items "joy," "happiness," "anger," and "sadness" based on whether or not the scores match (or the degree of similarity) with the stored contents of the evaluation items "facial expression," "conversation content," and "volume." The determination unit 134 then multiplies the scores for each of the evaluation items "joy," "happiness," "anger," and "sadness" using the weighting coefficients in the weighting table 121d, and performs a weighted averaging process to sum up the results of the multiplication. The determination unit 134 then determines, as the estimation result, the "joy," "happiness," "anger," or "sadness" that has the largest weighted average value obtained by the weighted averaging process.
[0101] The determination unit 134 may determine the emotion estimation result using AI (artificial intelligence) instead of using a table to determine the estimation result. Specifically, for example, emotion candidates (emotion types set in the emotion estimation model (psychological plane)) estimated by a method similar to that of the emotion estimation unit 132 using biosignals and environmental information for creating learning data, and related information (such as a face image of the subject set to be acquired as related information) acquired by a method similar to that of the acquisition unit 131 are used as input information. A learning data set is created in which emotions determined (estimated) by an experiment (declaration of emotions by the subject under that state (so-called questionnaire survey)) in that state (biological signals and environmental information state for that input information) are used as correct answer data (relationship between related information and emotion). Then, a trained AI model trained by machine learning is created using the created training data set. Then, the acquired emotion candidates and related information of the user are applied (input) to the trained AI model created in this way, and the estimation result is determined as the estimated emotion.
[0102] When the estimation result is determined by the determination unit 134, the process proceeds to the next step S16.
[0103] In step S16, the providing unit 135 provides the estimation result determined by the determining unit 134 to an in-vehicle device in the vehicle 20 via the network N1. Then, the in-vehicle device performs various controls according to the provided estimated emotion.
[0104] When the estimation result is provided to the in-vehicle device in the vehicle 20 by the providing unit 135, the process returns to step S11, and the above-mentioned process is repeated.
[0105] In addition, when the operation of the device that uses the emotion estimation result is terminated, for example, when the acquisition unit 131 of the estimation device 10 acquires a signal from the vehicle 20 notifying that the doors are locked (end of various controls while the vehicle is running), an interrupt process for terminating the estimation process shown in FIG. 5 is started, and the estimation process shown in FIG. 5 is terminated.
[0106] Through the above-described processing, estimated emotion candidates are selected based on the biological signal of the user U1 (emotion estimation target person), and the estimated emotion is narrowed down and determined from the estimated emotion candidates according to the related information.
[0107] <2. Second embodiment> An example configuration of an estimation system 1 according to the second exemplary embodiment of the present invention is the configuration shown in FIG. 1, similar to the above-described first embodiment.
[0108] An example configuration of an estimation device 10 according to a second exemplary embodiment of the present invention is the configuration shown in Fig. 3, similar to the above-described first embodiment. However, in this embodiment, the storage unit 12 of the estimation device 10 includes a sensor table 121a and a psychological plane table 121b as tables (data tables) 121, and also includes a learned emotion prediction model.
[0109] An example of the estimation process executed by the estimation device 10 in this embodiment is the flowchart shown in Fig. 5, similar to the first embodiment described above. However, in this embodiment, the specific process contents of step S13 and step S15 are different from those in the first embodiment.
[0110] The learned emotion prediction model included in the memory unit 12 of the estimation device 10 is a model constructed by machine learning of correct answer data (relationship between events and emotions) collected by experiments (such as a questionnaire survey on the relationship between an event by a subject and the emotion at the time when the event occurred (emotions within the emotion types set in the emotion estimation model (psychological plane))). Specifically, for example, it is an AI model trained with a set of learning data in which an event to a subject is used as input data and the emotion of the subject at the time when the event occurred is used as correct answer data.
[0111] It is desirable that the training data includes not only the relationship between one event and the emotion when that event occurred, but also the relationship between multiple events (input data) and the emotion when those events occurred (correct answer data). This can improve the accuracy of emotion prediction by the trained emotion prediction model when multiple events occur.
[0112] In addition, when the relationship between multiple events and the emotions at the time when the events occurred is included in the learning data, it is desirable that the order in which the multiple events occurred or the time when each of the multiple events occurred is also included in the learning data (input data). This allows the order or time when the multiple events occurred to be taken into account in emotion prediction by the learned emotion prediction model when multiple events occurred, so that the accuracy of emotion prediction by the learned emotion prediction model can be further improved. In this case, when estimating emotions, it is necessary to input (apply) the order in which the events occurred or the time when each of the multiple events occurred as input data to the learned emotion prediction model.
[0113] Next, in the flowchart shown in FIG. 5, differences from the first embodiment will be described.
[0114] In step S13, the request unit 133 requests the camera 22 and the microphone 23 for related information within a target event period (for example, within one hour from the current time) that is set in advance by a design developer or the like, via the communication unit 11. In this embodiment, the camera 22 and the microphone 23 have built-in or externally connected storage devices that store and update the related information within the target event period.
[0115] In step S15, the determination unit 134 determines an estimation result from the emotion candidates estimated by the emotion estimation unit 132 in step S11, based on the related information acquired by the acquisition unit 131 in step S14.
[0116] Specifically, first, the determining unit 134 infers an event that will occur to the user U1 within the event target period from the related information acquired by the acquiring unit 131 in step S14.
[0117] Specifically, the determination unit 134 estimates events that occurred during the event target period prior to the emotion determination timing as emotion-related information, for example, by image analysis processing using an AI image analysis device or the like, or audio analysis processing using an AI audio analysis device or the like.
[0118] Through such processing, for example, if a crying child is captured in an image taken at a first time and the audio information at the first time includes the child's crying voice, the determination unit 134 infers that an incident occurred at the first time in which the child was having a tantrum.
[0119] Also, for example, if the image captured at the second time shows multiple smiling occupants and the audio information at the second time includes the laughter of multiple occupants, the determination unit 134 infers that an event occurred at the second time in which the occupants were chatting with each other.
[0120] Also, for example, if the image captured at the third time shows a window glass with water droplets on it and the audio information at the third time includes the sound of rain, the determination unit 134 infers that an event in which it is raining is occurring at the third time.
[0121] Next, the determination unit 134 inputs the estimated event into the learned emotion prediction model to obtain the emotion output from the learned emotion prediction model. If an estimated event exists and the learned emotion prediction model is a model that considers the order in which multiple events occurred or the time at which each of the multiple events occurred, the determination unit 134 also inputs the order in which the multiple estimated events occurred or the time at which each of the multiple estimated events occurred into the learned emotion prediction model.
[0122] The trained emotion prediction model outputs the probability of the emotion corresponding to the estimated event. For example, when an event that a child is throwing a tantrum is input, the trained emotion prediction model outputs, for example, a 60% probability that the emotion of user U1 is anger, a 30% probability that the emotion of user U1 is depression, and a 10% probability that the emotion of user U1 is unpleasant. Note that the operation of calculating the probability of each output candidate and determining the output is known as the operation of a general AI model.
[0123] Then, the determination unit 134 determines an estimation result based on the output (probability) of the trained emotion prediction model from the emotion candidates estimated by the emotion estimation unit 132 in step S11. That is, in the above case, the emotion candidate with the highest probability of correct answer estimated by the trained emotion prediction model is determined as the estimated emotion among the emotion candidates.
[0124] Through the above-mentioned process, estimated emotion candidates are selected based on the biosignals of user U1 (emotion estimation target person), and the estimated emotion is narrowed down and determined from the estimated emotion candidates according to events that occurred to user U1 within a preset event estimation target period estimated from related information. In this embodiment, even if it is difficult to predict the emotion of user U1 from the appearance of user U1 (for example, the facial expression, dialogue content, voice, etc. of user U1), it is possible to predict the emotion of user U1 by utilizing the fact that the events that occur to user U1 affect the emotion of user U1, and it becomes possible to narrow down the estimated emotion.
[0125] Furthermore, in this embodiment, since a trained emotion prediction model is used, there is no need to perform tasks such as closely examining the relationship between events and emotions and creating a data table.
[0126] In addition, the determination unit 134 may determine the emotion estimation result using an AI (artificial intelligence) having both the estimation function by the emotion estimation unit 132 and the output function of the learned emotion prediction model, instead of determining the estimation result based on the output (probability) of the learned emotion prediction model from the emotion candidates estimated by the emotion estimation unit 132 in step S11. Specifically, for example, emotion candidates (emotion types set in the emotion estimation model (psychological plane)) estimated by a method similar to that of the emotion estimation unit 132 based on biosignals and environmental information for creating learning data, and related information (events estimated by a method similar to that of the determination unit 134) acquired by a method similar to that of the acquisition unit 131 are input. In addition, an AI model machine-learned with a learning dataset is created as correct answer data (relationship between related information and emotion) collected by an experiment (declaration of emotion by a subject under that state (so-called: questionnaire survey)) in that state (biological signal and environmental information state for that input information). Then, the acquired emotion candidates and related information of the user are applied (input) to the learned AI model created in this way, and the estimation result is determined as the estimated emotion.
[0127] In addition, in this embodiment, instead of the trained emotion prediction model, a data table showing the relationship between the events and emotions estimated from related information may be used. In this case, a weighting coefficient indicating the degree to which each event affects emotions when multiple events occur may be set. The weighting coefficient may be a value according to the type of event, or a value according to the order in which the multiple events occurred or the time at which each of the multiple events occurred.
[0128] <3. Third embodiment> 8 is a diagram showing an example of the configuration of an estimation system 1 according to an exemplary third embodiment of the present invention. In this embodiment, detailed descriptions of parts similar to those in the second embodiment will be omitted.
[0129] 8, the estimation system 1 includes a server 10, a biosensor 21, and a smartphone 24. The smartphone 24 is placed (brought in) in the cabin of the vehicle 20. The smartphone 24 can be communicatively connected to a communication device provided in the vehicle 20 via a communication method such as Wi-Fi (registered trademark).
[0130] The smartphone 24 stores schedule information of the user U1 using a schedule management application, a calendar application, a task management application, etc. The smartphone 24 is an example of an output device that outputs relevant information related to the emotion of the user U1 (schedule information of the user U1). In addition to the smartphone 24, a portable terminal device (including a smart phone) such as a tablet terminal or a mobile phone capable of executing the above application is also applicable.
[0131] An example configuration of an estimation device 10 according to the third exemplary embodiment of the present invention is the configuration shown in FIG. 3, similar to the above-described second embodiment.
[0132] An example of the estimation process executed by the estimation device 10 in this embodiment is the flowchart shown in Fig. 5, similar to the second embodiment described above. However, in this embodiment, the specific processing content of step S13 is different from that in the first embodiment.
[0133] In step S13, the request unit 133 requests related information (schedule information of a schedule management app of the smartphone 24, etc.) within a schedule target period previously set by a design developer, etc., from the smartphone 24 via the communication unit 11. The schedule target period described above may include both a past period and a future period (for example, within 30 minutes before or after the present time), may include only a past period (for example, within one hour before the present time), or may include only a future period (for example, within one hour after the present time). In other words, since past and future schedules that are relatively close to the present time are considered to have an effect on emotions, the schedules are treated as related information.
[0134] In this embodiment, the determination unit 134 infers in step S15 events that will occur to the user U1 within the schedule target period from the related information acquired by the acquisition unit 131 in step S14, as in the second embodiment. However, in this embodiment, the related information is schedule information within a predetermined period, such as the schedule target period of the user U1, so the nature of the events is different from that in the second embodiment. Events inferred in this embodiment are mainly events related to the actions of the user U1 himself, such as going to lunch with a friend or visiting a business partner, and are acquired as so-called text information.
[0135] In this embodiment, as in the second embodiment described above, even if it is difficult to predict the emotions of user U1 from the appearance of user U1 (for example, the facial expression, content of the dialogue, voice, etc. of user U1), it is possible to predict the emotions of user U1 by utilizing the fact that events that occur to user U1 affect the emotions of user U1, and it becomes possible to narrow down the estimated emotions.
[0136] In addition, in this embodiment, as in the second embodiment described above, a trained emotion prediction model is used, so there is no need to perform tasks such as examining the relationship between events and emotions and creating a data table.
[0137] In addition, in this embodiment, unlike the second embodiment described above, it is also possible to predict the emotion of user U1 by utilizing the fact that future events occurring to user U1 will affect the emotion of user U1. This increases the number of events that can be used to predict the emotion of user U1, and is expected to improve the accuracy of emotion prediction.
[0138] <4. Things to keep in mind> Various technical features disclosed in the description of the invention may be modified in various ways without departing from the spirit of the technical creation. Furthermore, the embodiments and modifications disclosed in the description of the invention may be combined to the extent possible.
[0139] For example, the estimation process of the estimation device 10 may be started when the vehicle 20 starts control using the estimation result of the estimation device 10. In this way, the processing load on the estimation device 10 can be reduced. [Explanation of symbols]
[0140] 1. Estimation system 10... Estimation device 13. Controller 21. Biometric sensor 22. Camera 23 Microphone 24. Smartphone
Claims
1. An emotion estimation device, comprising: Acquire relevant information related to the emotion of the emotion estimation target person; Estimating emotion candidates based on a plurality of types of biosignals; and determining the emotion estimation result from the emotion candidates based on the related information.
2. the related information is sensor information output from a sensor arranged in the vicinity of the emotion estimation target and not in contact with the emotion estimation target, The estimation device includes: Calculating an evaluation score for each of the emotion candidates based on a difference between the sensor information and an evaluation criterion set for each of the emotion candidates; The estimation device according to claim 1 , wherein the emotion estimation result is determined based on a comparison result of the evaluation points of the emotion candidates.
3. The estimation device according to claim 1 , wherein the related information is event information for the emotion estimation subject during a set period that is set as a target period of the related information.
4. The estimation device according to claim 3 , wherein the event information is schedule information acquired from a mobile terminal of the emotion estimation subject.
5. a trained emotion prediction model that is machine-learned using training data in which the related information and the emotion candidates are used as input data and emotions in situations indicated by the related information and the emotion candidates are used as correct answer data; 4. The estimation device according to claim 1, wherein the emotion estimation result is determined by applying the acquired emotion candidates and the estimated emotion candidates to the learned emotion prediction model.
6. A biosignal learning model is machine-trained using learning data in which a captured image of a data creation subject for creating learning data is used as input data, and the biosignal of the data creation subject at the time of capturing the captured image is used as correct answer data, 4. The estimation device according to claim 1, wherein the plurality of types of biosignals are biosignals output from the biosignal learning model by applying a captured image of a subject of emotion estimation to the biosignal learning model.
7. 2. The estimation device according to claim 1, wherein the process of determining the emotion estimation result is at least one of when at least one of the plurality of types of biological signals has changed, when an environment surrounding the vehicle has changed, when a behavior of the vehicle has changed, and when the vehicle starts control using the emotion estimation result.
8. The plurality of types of biosignals are brain waves and heart rates. calculating an alertness index value representing an alertness state based on the beta wave / alpha wave of the electroencephalogram; calculating an emotion intensity index value representing the intensity of the emotion based on a standard deviation of the LF component of the heart rate; 2. The estimation device according to claim 1, wherein an emotion estimation value is determined by case-by-case classification of a combination of a binarized value of the arousal state index value and a binarized value of the emotion intensity index value, and an emotion that is a candidate for an estimation result is estimated by applying the arousal state index value and the emotion intensity index value to an emotion model.
9. An estimation system including an estimation device that estimates an emotion, a biosensor that measures a plurality of types of biosignals, and an output device that outputs related information related to an emotion of a subject of emotion estimation, The estimation device includes: Estimating emotion candidates that are candidates for an emotion estimation result based on the plurality of types of biological signals; and determining the emotion estimation result from the emotion candidates based on the related information.
10. An estimation method, comprising: an apparatus executing a process of estimating emotion candidates that are candidates for an emotion estimation result based on a plurality of types of biological signals of a subject of emotion estimation, and determining the emotion estimation result from the emotion candidates based on associated information related to the emotion of the subject of emotion estimation.
11. Estimating emotion candidates that are candidates for an emotion estimation result based on a plurality of types of biosignals of an emotion estimation subject; determining the emotion estimation result based on related information related to the emotion of the emotion estimation target from the emotion candidates; The estimation program causes a computer to execute the above.
Citation Information
Patent Citations
Emotion determination apparatus, emotion determination method, and program
JP2019063324A
