Estimation device, estimation system, and estimation method

Abstract

To provide a technology to execute emotion estimation in a state suitable for estimating emotions.SOLUTION: An estimation device includes a controller. The controller gives a user a plurality of types of tasks. The controller acquires an index value based on a biological signal when each of the plurality of types of tasks is executed. The controller estimates suitability for emotion estimation on the user on the basis of relationships among a plurality of index values.SELECTED DRAWING: Figure 16

Description

Technical Field

[0001] The present invention relates to a technique for estimating emotions.

Background Art

[0002] Conventionally, an apparatus that estimates a person's emotion based on physical quantities acquired by sensors such as cameras and heart rate monitors and presents the estimated emotion is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, biological information is easily affected by the environment, physical condition, etc. This tendency is likely to be significant, for example, in electroencephalograms. For this reason, it is considered that the person who is the target of emotion estimation may be in a state unsuitable for estimating emotions due to the influence of the environment or the like. If emotion estimation is performed in a state unsuitable for emotion estimation, the possibility of making an incorrect emotion estimation increases.

[0005] In view of the above points, an object of the present invention is to provide a technique that enables emotion estimation to be performed in a state suitable for emotion estimation.

Means for Solving the Problems

[0006] An exemplary estimation apparatus of the present invention includes a controller. The controller gives a plurality of types of tasks to a user. The controller acquires an index value based on a biological signal at the time of execution of each of the plurality of types of tasks. The controller estimates the suitability of the user for emotion estimation based on the relationship between the plurality of index values.

Effects of the Invention

[0007] According to an exemplary version of the present invention, emotion estimation can be performed in a state suitable for emotion estimation. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram showing an example of the estimation system configuration. [Figure 2] A diagram showing an example of an emotion estimation model (psychological plane). [Figure 3] This figure shows an example configuration of the estimation device according to the first embodiment. [Figure 4A] A diagram showing an example of a sensor table. [Figure 4B] A diagram showing an example of a psychological planar table. [Figure 5] A diagram showing an example of a special task table. [Figure 6] This figure schematically shows the time-dependent changes in indicator values ​​of physiological responses during the execution of the first and second tasks. [Figure 7] A diagram that simulates the time evolution of physiological responses. [Figure 8] A schematic diagram showing the time evolution of the estimated upper limit of the neutral region. [Figure 9] A diagram showing an example of a neutral region table. [Figure 10] A diagram showing an example of a psychological plane that includes a neutral zone. [Figure 11] A flowchart showing an example of the preliminary stage of the estimation process performed by the estimation device of the first embodiment. [Figure 12] A flowchart showing an example of the subsequent stage of the estimation process performed by the estimation device of the first embodiment. [Figure 13A] A diagram illustrating a screen displaying the results of an emotion estimation. [Figure 13B] A diagram illustrating a screen displaying the results of an emotion estimation. [Figure 14] A flowchart illustrating another example of the preliminary stage of the estimation process performed by the estimation device of the first embodiment. [Figure 15] A diagram showing the schematic configuration of a modified example of the estimation system of the first embodiment. [Figure 16] Figure showing a configuration example of the estimation device according to the second embodiment [Figure 17] Figure showing an example of a task table for suitability estimation [Figure 18] Flowchart showing an example of the suitability estimation process executed by the estimation device according to the second embodiment [Figure 19] Flowchart showing a modified example of the suitability estimation process executed by the estimation device according to the second embodiment [Figure 20] Figure showing an example of a suitability level table [Figure 21] Example of a screen for notifying that the user is unsuitable for emotion estimation [Figure 22] Figure showing an example of temporarily displaying the emotion estimation result on the screen [Figure 23] Flowchart showing another modified example of the suitability estimation process executed by the estimation device according to the second embodiment

Mode for Carrying Out the Invention

[0009] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.

[0010] <1. First Embodiment> [1-1. Estimation System] FIG. 1 is a diagram showing a configuration example of an estimation system 1 according to the first embodiment of the present invention. In the present embodiment, the user U1 is a player of e-sports. The user U1 is a person for whom emotion estimation is performed. That is, the estimation system 1 is configured as a system for estimating the emotion of the e-sports player U1.

[0011] Note that only one user U1 is shown in FIG. 1, but there may be a plurality of users U1. Also, the user U1 may be other than an e-sports player. For example, the user U1 may be a patient in a medical institution, a student in an educational institution, a driver of a vehicle, a viewer of content such as video or music, or the like.

[0012] As shown in Figure 1, the estimation system 1 comprises a server 10, a terminal device 20, a biosensor 30, and a game device 40. The server 10 and the terminal device 20 are connected via a network N. Network N is, for example, the internet or an intranet. The terminal device 20, the biosensor 30, and the game device 40 are provided to communicate with each other via wired or wireless connections. The terminal device 20, the biosensor 30, and the game device 40 are connected according to a communication standard such as Wi-Fi® or Bluetooth®.

[0013] Server 10 may be a physical server or a virtual server. In this embodiment, Server 10 constitutes an estimation device for estimating emotions. That is, the estimation system 1 includes the estimation device 10. Hereafter, Server 10 will be referred to as the estimation device 10. Details of the estimation device 10 will be described later. Note that the estimation device 10 may be composed of one server or multiple servers.

[0014] The terminal device 20 is, for example, a personal computer, a smartphone, or a tablet computer. In this embodiment, the terminal device 20 is used by operator O1. However, the terminal device 20 may be configured to be used by user U1. In this case, the terminal device 20 may also function as the game device 40. Furthermore, there may be multiple terminal devices 20, for example, if there are multiple users U1.

[0015] The biosensor 30 is attached to user U1 and detects user U1's biological signals as sensor signals. That is, the estimation system 1 includes a biosensor 30 that measures biological signals. In this embodiment, the biosensor 30 includes a first sensor 31 and a second sensor 32. The first sensor 31 is a headgear-type electroencephalogram (EEG) sensor. The second sensor 32 is a wristband-type optical heart rate (pulse) sensor. However, the first sensor 31 and the second sensor 32 may be changed to other biosensors depending on the biological information to be acquired. Other biosensors may be, for example, an electrocardiogram (ECG) heart rate sensor, a blood pressure monitor, or a NIRS (Near Infrared Spectroscopy) device.

[0016] The game device 40 includes an operation unit 41 in which user U1 performs operations related to the game, and a display unit 42 having a display screen. The game device 40 appropriately transmits information such as that of user U1 operating the device to the terminal device 20. The game device 40 also appropriately receives information from the terminal device 20.

[0017] Now, with reference to Figure 1, the processing flow of estimation system 1 will be explained.

[0018] The first sensor 31 and the second sensor 32 measure the biosignals of user U1 and output the measurement results as sensor signals to the terminal device 20 (step S1). The terminal device 20 transmits the input sensor signals to the estimation device (server) 10. The estimation device 10 estimates the emotional state of user U1 based on the input sensor signals (step S3).

[0019] Here, we will explain the overview of emotion estimation in step S3. The estimation device 10 generates index values, which are values ​​of indicators (physiological responses) that indicate the mental and physical state, based on the input sensor signals. In this embodiment, the estimation device 10 generates index values ​​for two indicators of mental and physical state related to electroencephalography (EEG) and heart rate. For example, the indicator of mental and physical state related to EEG is the level of central nervous system arousal (hereinafter simply referred to as arousal), and its index value can be given as "beta waves / alpha waves of EEG". Also, for example, the indicator of mental and physical state related to heart rate is the activity level of the autonomic nervous system, and its index value can be given as "standard deviation of the heart rate LF component". The index values ​​are calculated using a calculation model (calculation formula or conversion data table) that is stored in memory or the like in advance.

[0020] Furthermore, the first sensor 31 and the second sensor 32 may be configured to have calculation functions (a configuration in which a computer or the like is built in), and the sensors 31 and 32 may be configured to calculate the index value. Alternatively, the terminal device 20 may be configured to calculate the index value based on the sensor signals input from the first sensor 31 and the second sensor 32.

[0021] The estimation device 10 estimates the emotion of user U1 using the calculated index value and an emotion estimation model that is pre-stored in memory or the like. The emotion estimation model is created based on medical evidence (papers, etc.).

[0022] Figure 2 shows an example of an emotion estimation model (psychological plane). According to various medical evidence related to psychology, psychology can be estimated based on two types of indicators that show physical state. The psychological plane shown in Figure 2, which has two types of indicators of mind and body state as axes, is an example of an emotion estimation model in accordance with the aforementioned technological concept. In Figure 2, as an example, the vertical axis is "arousal level (aroused-unaroused)" and the horizontal axis is "autonomic nervous system activity level (sympathetic nervous system activity (strong emotion)-parasympathetic nervous system activity (weak emotion))".

[0023] In the psychological plane shown in Figure 2, each of the four quadrants separated by the vertical and horizontal axes is assigned a corresponding psychological state. The distance from each axis indicates the intensity of the corresponding psychological state. In the example in Figure 2, the psychological states of "happy, joyful, angry, and sad" are assigned to the first quadrant. The psychological state of "depressed" is assigned to the second quadrant. The psychological states of "relaxed and calm" are assigned to the third quadrant. The psychological states of "anxious, fearful, and unhappy" are assigned to the fourth quadrant.

[0024] By plotting two types of mental and physical state indicators obtained based on the measurement results of biosignals onto a psychological plane, it is possible to estimate the psychological state from the resulting coordinates. Specifically, the psychological state and its intensity can be estimated based on which quadrant of the psychological plane the plotted coordinates lie in and how far they are from the origin.

[0025] In this embodiment, the emotion estimation model shown in Figure 2 is calibrated, and the emotion is estimated using the calibrated emotion estimation model. This will be explained later. In the example shown in Figure 2, the emotion estimation model is a two-dimensional plane, but it may be a three-dimensional or higher space.

[0026] Returning to Figure 1, the estimation device 10 performs emotion estimation and then provides (transmits) the obtained estimation result to the terminal device 20 (step S4). For example, the terminal device 20 displays the received emotion information of user U1 on its screen based on the operation of operator O1. Alternatively, the terminal device 20 may display the received emotion information of user U1 on the display unit 42 of the game device 40. The terminal device 20 may also provide the received emotion information of user U1 to an external game system.

[0027] The results of the emotion estimation can be used, for example, for the mental training of user U1 in esports. For instance, if user U1 experiences an emotion that puts them at a disadvantage in a game (such as anxiety or anger) while playing a video game, it is determined that intensive training corresponding to that emotional state is necessary. In this training, the emotion information of user U1 estimated by estimation system 1 is used.

[0028] Furthermore, various types of esports include those where multiple users (U1) cooperate and those where multiple users (U1) compete against each other. In these types of esports, if the emotional state of each player is displayed, it becomes possible to perform advanced gameplay, such as changing game tactics according to the emotional state. It is also possible to create a system where esports spectators can understand the emotional state of each player while watching the game, which can increase the enjoyment of watching games.

[0029] Furthermore, if the subject whose emotions are estimated by Estimation System 1 (User U1) is a patient in a medical institution, the estimated emotions can be used for examinations and treatments. For example, medical staff can understand that the patient is feeling anxious and take appropriate measures such as counseling. Note that the medical staff may be just one example of Operator O1 (see Figure 1).

[0030] Furthermore, if the target (user U1) whose emotions are estimated by estimation system 1 is a student in an educational institution, the estimated emotions can be used to improve the content of lessons. For example, a teacher can understand that students are finding lessons boring and improve the content of the lessons to make them more interesting to students. Note that the teacher may be an example of operator O1 (see Figure 1).

[0031] Furthermore, if the target (user U1) whose emotions are estimated by the estimation system 1 is the vehicle driver, the estimated emotions can be used to promote safe driving. For example, the in-vehicle device can detect that the driver is not feeling a moderate level of tension while driving and output a message encouraging them to concentrate on driving. Note that the in-vehicle device may be an example of a terminal device 20.

[0032] Furthermore, if the target of emotion estimation by estimation system 1 (user U1) is a viewer of content such as video or music, the estimated emotion can be used to create further content. For example, a video content distributor can create a highlight video by collecting scenes that viewers found enjoyable. Note that the video content distributor may be just one example of operator O1 (see Figure 1).

[0033] In the above example, the server 10 is shown as the estimation device, but for example, the terminal device 20 or the game device 40 used by the e-sports player may also be the estimation device. Furthermore, for example, the estimation device may consist of multiple devices. For example, the server 10, terminal device 20, and game device 40, which are connected to each other, may distribute the processing for estimating emotions.

[0034] [1-2. Estimation device] Figure 3 shows an example configuration of an estimation device 10 according to the first embodiment of the present invention. In Figure 3, only the components necessary to explain the features of this embodiment are shown, and descriptions of general components are omitted. As shown in Figure 3, the estimation device 10 includes a communication unit 11 and a storage unit 12. The estimation device 10 also includes a controller 13. The estimation device 10 may be a so-called computer device. The estimation device 10 may also be configured to include an input device such as a keyboard and an output device such as a display.

[0035] The communication unit 11 is an interface for communicating data with other devices via the network N. The communication unit 11 is, for example, a NIC (Network Interface Card).

[0036] The storage unit 12 is configured to include volatile memory and non-volatile memory. The volatile memory may include, for example, RAM (Random Access Memory). The non-volatile memory may include, for example, ROM (Read Only Memory), flash memory, or a hard disk drive. Programs and data that can be read by a computer are stored in the non-volatile memory. At least a portion of the programs and data stored in the non-volatile memory may be obtained from other computer devices connected by wired or wireless connections, or from a portable recording medium.

[0037] As shown in Figure 3, in this embodiment, the storage unit 12 includes a table (data table) 121. More specifically, table 121 includes multiple tables for various processing. For example, table 121 includes a sensor table 121a and a psychological plane table 121b. Figure 4A shows an example of the sensor table 121a. Figure 4B shows an example of the psychological plane table 121b.

[0038] As shown in Figure 4A, the items in sensor table 121a include "Sensor ID," "Sensor Type," "Biometric Signal Type," "Corresponding Indicator ID," "Corresponding Indicator Type," and "Indicator Conversion Information." Note that the items in the table correspond to data storage cells (storage frames).

[0039] The "Sensor ID" field in sensor table 121a stores sensor ID data, which is identification information used to identify data records in sensor table 121a. Sensor ID data is also the primary key for data records in sensor table 121a. In other words, in sensor table 121a, a data record is created for each sensor ID data, and the data for each item associated with the sensor ID data is stored in that data record.

[0040] The "Sensor Type" item in sensor table 121a stores information to identify the sensor type. In this example, the sensor name (or model number) is stored.

[0041] The "Biological Signal Type" item in sensor table 121a stores the type of measurement value based on the biological signal detected by the sensor. This biological signal type data is correlated with the corresponding index type data. Academically, it is recognized that the corresponding index type data can be estimated (calculated) by obtaining the corresponding biological signal type data.

[0042] The "Corresponding Indicator ID" item in sensor table 121a stores identification information for identifying a mental / physical state index generated (calculated) based on the signal from a sensor that detects biological signals. The "Corresponding Indicator Type" item in sensor table 121a stores the type of index (name, etc.).

[0043] The "Index Conversion Information" item in sensor table 121a stores conversion information (such as calculation formulas and conversion data tables) for calculating index values ​​based on signals obtained from sensors that detect biological signals. In other words, by converting the biological signals detected by the sensor corresponding to the sensor ID data according to the index conversion information, the index value of the mental and physical state index identified by the corresponding index ID is estimated (calculated).

[0044] For example, the data record for sensor ID "SN01" in the sensor table 121a shown in Figure 4A contains the following information: The output signal of the "EEG sensor BA" measures "beta waves / alpha waves of the brainwave." Then, by converting these "beta waves / alpha waves of the brainwave" using the index conversion information of "FX01," an index value for "alertness level" is obtained.

[0045] As shown in Figure 4B, the psychological plane table 121b is a two-dimensional matrix table in which the indicator type (specifically, indicator ID data) is used as the parameters on the vertical and horizontal axes. In the psychological plane table 121b, data for psychological plane types that can be used with the data of the two indicator types is stored in memory cells (memory frames) determined by the two indicator types. For example, if the indicator types used as indicators are indicator type VSm and indicator type VSn, the psychological plane used for emotion estimation will be psychological plane mn. Information for processing using psychological plane mn is read out and used for emotion estimation processing.

[0046] Furthermore, a common index ID is used in both the sensor table 121a and the psychological plane table 121b. That is, based on the sensor table 121a and the psychological plane table 121b, the psychological plane mn corresponding to the two types of sensors attached to user U1 can be determined. For example, suppose the types of sensors attached to user U1 are "EEG sensor BA (index ID: VS01)" and "heart rate sensor HA (index ID: VS02)". In this case, the index ID data corresponding to "EEG sensor BA (index ID: VS01)" and "heart rate sensor HA (index ID: VS02)" is determined based on the sensor table 121a. Then, based on the psychological plane table 121b, psychological planes 01-02, using "alertness level" (VS01) and "autonomic nervous system activity level" (VS02) as indices, are determined as the psychological planes used to estimate emotions.

[0047] Returning to Figure 3, the controller 13 includes a processor that performs calculations and other processing. The processor may include, for example, a CPU (Central Processing Unit). The controller 13 may consist of one processor or multiple processors. If it consists of multiple processors, they just need to be connected to each other so that they can communicate with one another. Note that if the estimation device 10 is configured as a cloud server, the CPU that makes up the processor may be a virtual CPU.

[0048] As shown in Figure 3, the controller 13 includes, as its functions, an acquisition unit 131, a region determination unit 132, an emotion estimation unit 133, and a provision unit 134. In this embodiment, the functions of the controller 13 are realized by the processor executing calculation processing according to a program stored in the storage unit 12.

[0049] Furthermore, the scope of this embodiment may include a computer program that enables a processor (computer) to implement at least some of the functions of the estimation device 10. The scope of this embodiment may also include a computer-readable non-volatile recording medium for recording such a computer program. The non-volatile recording medium may be, for example, the non-volatile memory described above, an optical recording medium (e.g., an optical disc), a magneto-optical recording medium (e.g., a magneto-optical disc), a USB memory, or an SD card.

[0050] Furthermore, each functional unit 111 to 114 may be implemented by a single program, but for example, each functional unit may be implemented by a separate program. Also, each functional unit 111 to 114 may be implemented as a separate server. In addition, each functional unit 111 to 114 may be implemented by having a processor execute a program, i.e., by software, as described above, but may also be implemented by other methods. At least a part of each functional unit 111 to 114 may be implemented using, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). In other words, each functional unit 111 to 114 may be implemented by hardware using a dedicated IC, etc. Also, each functional unit 111 to 114 may be implemented using a combination of software and hardware. Furthermore, each functional unit 111 to 114 is a conceptual component. The function performed by one component may be distributed among multiple components. Also, the functions of multiple components may be integrated into one component.

[0051] The acquisition unit 131 receives data (sensor signals) measured by the first sensor 31 and the second sensor 32 from the terminal device 20 via the communication unit 11. The acquisition unit 131 stores the received data in the storage unit 12 as needed for subsequent processing.

[0052] The domain determination unit 132 performs preparatory processing to calibrate the emotion estimation model (psychological plane in this example) to a model suitable for estimating the emotions of each user U1. Before explaining the specific contents of this preparatory processing, we will explain why calibration is necessary.

[0053] In the psychological plane shown in Figure 2 above, each index (vertical and horizontal axes) based on biosignals is divided into two regions. When using such a psychological plane, for example, it would be as follows: The index value representing the degree of arousal is classified into either "aroused" or "non-aroused," and the index representing the degree of autonomic nervous system activity is classified into either "sympathetic nervous system activity (strong emotion)" or "parasympathetic nervous system activity (weak emotion)." However, taking the degree of arousal as an example, the mental and physical state of a person is not limited to aroused and non-aroused states, but also includes a neutral state that cannot be classified as either. And the range of this neutral state generally varies from person to person. The same can be said for autonomic nervous system activity. For this reason, if emotions are estimated by simply using the psychological plane shown in Figure 2, there is a possibility that the estimation of emotions will be inaccurate. Taking these points into consideration, in this embodiment, the psychological plane, which is an emotion estimation model, is configured to be used after calibration.

[0054] As a preparatory process for calibration, the region determination unit 132 divides each index based on the biological signal into three regions: a neutral region, a positive region, and a negative region. In other words, the region determination unit 132 is configured to perform a region determination process to determine the neutral region in order to divide each physiological response that constitutes the vertical and horizontal axes of the psychological plane into the three regions described above.

[0055] The neutral region is a region where physiological responses (physical and mental states) are ambiguous and cannot be specifically identified. The positive region is the region on the other side of the neutral region, where the index value is larger than that of the neutral region. The negative region is the region on the other side of the neutral region, where the index value is smaller than that of the neutral region. Physiological responses are clear in the positive and negative regions. For example, if the type of physiological response is arousal, the positive region indicates an aroused state, and the negative region indicates a non-aroused state. For example, if the type of physiological response is autonomic nervous system activity, the positive region indicates a state where the sympathetic nervous system is active, and the negative region indicates a state where the parasympathetic nervous system is active.

[0056] Below, we will explain the region determination process by which the region determination unit 132 determines the neutral region for each index (physiological response) using two examples.

[0057] In the first example of the region determination process, user U1 is instructed to perform a special task, and the neutral region is determined according to the measurement results of biological signals during the execution of the task. The special tasks are determined based on medical evidence and are prepared in advance for each type of physiological response. The prepared special tasks are stored in the storage unit 12 as a special task table 121c. That is, in this embodiment, table 121 (see Figure 3) includes the special task table 121c.

[0058] Figure 5 shows an example of a special task table 121c. As shown in Figure 5, the items in the special task table 121c include "Task ID," "Sensor Type," "Corresponding Indicator Type," "Task Type," and "Task Content." The items "Sensor Type" and "Corresponding Indicator Type" are the same as those in Figure 4A above, so their explanation is omitted. Note that "Corresponding Indicator Type" may also be expressed as "Corresponding Physiological Response Type."

[0059] The "Task ID" field in the special task table 121c stores ID data, which is identification information used to identify task information in the special task table 121c.

[0060] The "Task Type" item in Special Task Table 121c stores whether a task is a first-class task (which elicits a large physiological response) or a second-class task (which elicits a small physiological response). A large physiological response results in a large index value, while a small physiological response results in a small index value. Whether a task elicits a large or small physiological response is determined based on medical evidence. For example, if the type of physiological response is arousal level, tasks that increase arousal level are first-class tasks, and tasks that decrease arousal level are second-class tasks. Similarly, if the type of physiological response is autonomic nervous system activity level, tasks that activate the sympathetic nervous system are first-class tasks, and tasks that activate the parasympathetic nervous system (i.e., deactivate the sympathetic nervous system) are second-class tasks.

[0061] The "Task Content" item in Special Task Table 121c stores the specific details of the task to be performed by user U1. The task content is determined based on medical evidence along with the task type mentioned above. For example, if the type of physiological response is arousal, the task content of the first task is "mentally add the displayed numbers." The number of mental additions may be, for example, 10. Also, for example, if the type of physiological response is arousal, the task content of the second task is "remain still and clench your fist." The time spent clenching your fist and remaining still may be, for example, 3 minutes. Also, for example, if the type of physiological response is autonomic nervous system activity, the task content of the first task is "standing test." In the standing test, for example, the user is required to remain still in a supine position with eyes open for a predetermined time, and then raise their upper body and remain still in a long sitting position with eyes open for a predetermined time. The predetermined time may be, for example, 3 minutes. Also, for example, if the type of physiological response is autonomic nervous system activity, the task content of the second task is "Aschnell test." The Aschnell test, for example, requires the subject to gently press one eyelid with the pads of their index and middle fingers for a predetermined time while their eyes are closed and at rest. Another example of the Aschnell test may involve wearing an eye mask for a predetermined time, such as 3 minutes.

[0062] As can be seen from the contents of the special task table 121c, determining the neutral region for each physiological reaction requires the user U1 to execute a first task and a second task for each type of physiological reaction. The region determination unit 132 (i.e., the controller 13) performs the following processing when determining the neutral region for each physiological reaction. First, the region determination unit 132 has user U1 execute the first task, which is expected to increase the index value obtained when the tasks are executed, and the second task, which is expected to decrease it when the tasks are executed. Then, the region determination unit 132 determines the neutral region based on the index values ​​obtained from the execution of the first and second tasks. Once the neutral region is determined, the positive and negative regions are automatically determined. In other words, the physiological reaction (index) can be divided into three regions. In addition, there is a time difference between the timing of the first and second tasks, which are performed for each type of physiological reaction, and the physiological reaction is measured for each task.

[0063] In this configuration, before emotion estimation, user U1 is required to perform a special task, and the positive, neutral, and negative regions are determined based on the results. This allows for the division of physiological responses (indicators) into three regions, appropriately reflecting user U1's current environment. In other words, it is expected that emotion estimation will be performed accurately.

[0064] A specific example of the neutral region determination method described above will be explained with reference to Figure 6. Figure 6 is a schematic diagram showing the time change of the index value of the physiological response during the execution of the first and second tasks. In Figure 6, the horizontal axis is time, and the vertical axis is the physiological response (index). In Figure 6, signal α corresponds to the time change of the index value obtained during the execution of the first task. Signal β corresponds to the time change of the index value obtained during the execution of the second task. In Figure 6, the special tasks are executed in the order of the first task followed by the second task, but this is an example, and the special tasks may be executed in the order of the second task followed by the first task. Also, in Figure 6, the region indicated by arrow NR (a part of the vertical axis) corresponds to the neutral region.

[0065] The region determination unit 132 statistically processes the time-varying data of the index value obtained during the execution of the first task to determine the upper limit of the neutral region NR. For example, the region determination unit 132 smooths the time-varying data of the index value obtained during the execution of the first task, and sets the minimum value of the data after smoothing as the upper limit of the neutral region NR. The region determination unit 132 also statistically processes the time-varying data of the index value obtained during the execution of the second task to determine the lower limit of the neutral region NR. For example, the region determination unit 132 smooths the time-varying data of the index value obtained during the execution of the second task, and sets the maximum value of the data after smoothing as the lower limit of the neutral region NR. The smoothing process may be performed, for example, to remove noise components such as minute peak signals.

[0066] In other words, the region determination unit 132 (i.e., the controller 13) determines the boundary between the neutral region and the positive region based on the index value obtained during the execution of the first task. The region determination unit 132 also determines the boundary between the neutral region and the negative region based on the index value obtained during the execution of the second task. This configuration is suitable when it is possible to prepare two tasks (the first task and the second task) with large differences in the magnitude of physiological responses.

[0067] The region determination unit 132 determines the upper and lower limits of the neutral region for each type of physiological response. For this purpose, in this embodiment, the user U1 is requested to perform special tasks consisting of a first task and a second task for each of the two physiological responses (level of arousal and level of autonomic nervous system activity). Then, for each of the two physiological responses, the neutral region determination process is performed according to the measurement results of the biological signals during the execution of the special task.

[0068] Furthermore, the above assumes that the first fluctuation range, which is the range of fluctuation of the index value obtained when the first task is executed, and the second fluctuation range, which is the range of fluctuation of the index value obtained when the second task is executed, are far apart, and the area between the two fluctuation ranges is defined as the neutral region NR (see Figure 6). Note that the fluctuation range of the index value refers to the range of value fluctuations that occur with the change in the index value over time. However, for example, if the difference in the magnitude of the physiological response between the first task and the second task is small, the first fluctuation range and the second fluctuation range may overlap. In such cases, for example, the minimum value of the data obtained by smoothing the time change data of the index value obtained when the first task is executed may be used as the lower limit of the neutral region NR. Alternatively, for example, the maximum value of the data obtained by smoothing the time change data of the index value obtained when the second task is executed may be used as the upper limit of the neutral region NR.

[0069] In other words, the domain determination unit 132 (i.e., the controller 13) may determine the boundary between the neutral domain and the positive domain based on the index value obtained during the execution of the second task. The domain determination unit 132 may also determine the boundary between the neutral domain and the negative domain based on the index value obtained during the execution of the first task. To put it another way, the region where the first variation range and the second variation range overlap may be determined as the neutral domain NR. The neutral domain NR may be determined based on the region where the first variation range and the second variation range overlap. By adopting such a configuration, the constraints on determining the first and second tasks can be relaxed, making it easier to set up special tasks.

[0070] Next, we will describe a second example of the region determination process for determining the neutral region. In this second example, we do not request user U1 to perform any special tasks. In this second example, we determine the neutral region by focusing on the fact that as time-series data of physiological responses accumulate, the upper and lower limits of the neutral region estimated by statistical processing converge to a certain value. Once the neutral region is determined, the positive and negative regions are automatically determined. In other words, once the neutral region is determined, the physiological response can be divided into three regions.

[0071] In other words, in the second example, the domain determination unit 132 (i.e., the controller 13) accumulates time-series data of the indicator values ​​and determines the neutral region based on the statistical processing results of the time-series data. With this configuration, there is no need to ask user U1 to perform a special task before sentiment estimation, thus reducing the burden on user U1.

[0072] A method for determining the neutral region without special tasks will be explained with reference to Figures 7 and 8. Figure 7 is a schematic diagram showing the temporal change of a physiological response. In Figure 7, the horizontal axis represents time, and the vertical axis represents the physiological response (indicator). Figure 8 is a schematic diagram showing the temporal change of the estimated upper limit of the neutral region (see Figure 7). In Figure 8, the horizontal axis represents time, and the vertical axis represents the estimated upper limit of the neutral region. Note that the temporal change of the estimated lower limit of the neutral region shows the same temporal change as the upper limit shown in Figure 8. That is, the vertical axis of the graph shown in Figure 8 can be interpreted as the lower limit.

[0073] As shown in Figure 7, a person's physiological response (which can also be called their mental and physical state) changes in various states over time (based on changes in the environment, etc., that occur over time), even without being given a special task. More specifically, with time, the physiological response fluctuates between a large state, a small state, or an ambiguous state (neutral state). Since the physiological response changes up and down due to external stimuli, the distribution of physiological response values ​​is statistically distinctive. For example, since there are often no external stimuli, the physiological response values ​​usually fall within the neutral region. When there is a stimulus, the physiological response values ​​tend to change suddenly. By accumulating time-series data of physiological responses and performing cluster analysis to divide the accumulated data into the three states mentioned above, the upper and lower limits of the neutral region can be estimated. Any known method can be used for cluster analysis. For example, k-means, Otsu's multi-level model, and Gaussian mixture models can be used as cluster analysis methods.

[0074] In Figure 8, it is assumed that the amount of accumulated time-series data of physiological responses increases over time. That is, in Figure 8, it is assumed that the amount of data used to estimate the upper limit of the neutral region using statistical methods (cluster analysis is a detailed example) increases over time. As the amount of data increases over time, the estimated upper limit converges to a constant value over time, as shown in Figure 8. This trend is also true for the lower limit, as described above. The neutral region identified by the converged upper and lower limits is considered to be highly reliable. For this reason, in this example, the neutral region is determined when it is determined that the estimated upper and lower limits have converged.

[0075] Whether or not the upper limit has converged can be determined by comparing the current estimated upper limit with the previous estimated upper limit. For example, if the ratio of the two is within a predetermined range relative to 1, it can be determined that convergence has occurred. In this example as well, the region determination unit 132 determines the upper and lower limits of the neutral region for each type of physiological response.

[0076] The region determination unit 132 stores the upper and lower limits determined by performing a special task, or without performing a special task, in the storage unit 12. More specifically, the region determination unit 132 stores the determined upper and lower limits in a data table in the storage unit 12. That is, table 121 (see Figure 3) includes a neutral region table 121d containing neutral region information for each user U1.

[0077] Figure 9 shows an example of the neutral area table 121d. As shown in Figure 9, the items in the neutral area table 121d include "User ID", "Sensor Type", "Corresponding Indicator Type", "Upper Limit", "Lower Limit", and "Acquisition Date and Time". The items "Sensor Type" and "Corresponding Indicator Type" are the same as in Figures 4A and 5 above, so their explanation is omitted.

[0078] The "User ID" item in the neutral area table 121d stores user ID data, which is identification information used to identify user information in the neutral area table 121d. User ID data is transmitted, for example, from the game device 40 to the estimation device 10 via the terminal device 20.

[0079] The "Upper Limit" and "Lower Limit" fields in the neutral area table 121d store the upper and lower limit information determined by the area determination unit 132. The "Acquisition Date and Time" field in the neutral area table 121d stores the date and time information when the upper and lower limit information was stored. The upper and lower limits determined for each item, such as the user ID, are updated each time the latest information is acquired. In addition, when the upper and lower limit information is updated, the acquisition date and time are also updated.

[0080] Returning to Figure 3, the emotion estimation unit 133 selects a model to use for emotion estimation based on the data acquired by the acquisition unit 131. Specifically, the emotion estimation unit 133 compares the acquired data with the sensor table 121a and extracts "corresponding index type" data (index type data) that corresponds to the sensor type and biosignal type included in the acquired data. Then, the emotion estimation unit 133 compares the "index ID" of the extracted index type data with the psychological plane table 121b and selects a model (psychological plane) to use for emotion estimation. For example, if the sensor IDs are "SN01" (sensor type: electroencephalogram sensor BA) and "SN02" (sensor type: heart rate sensor HA), the corresponding index IDs are "VS01" (index type: arousal level) and "VS02" (index type: autonomic nervous system activity level) (see Figure 4A). Based on these, the emotion estimation unit 133 selects psychological planes 01-02 as the emotion estimation model to use.

[0081] When the emotion estimation unit 133 selects a psychological plane to be used, it uses the user ID of user U1, the target of emotion estimation, to extract the upper and lower limits of the neutral region corresponding to the index type of the selected psychological plane from the neutral region table 121d. Then, the emotion estimation unit 133 processes the selected psychological plane into a psychological plane that includes the neutral region using the extracted upper and lower limits. In other words, the emotion estimation unit 133 performs calibration on a pre-prepared emotion estimation model.

[0082] Figure 10 shows an example of a psychological plane including the neutral region. The psychological plane including the neutral region shown in Figure 10 is a modified version of the psychological plane shown in Figure 2. In Figure 10, the shaded region NR is the neutral region. The upper limit of the index type "Arousal Level" determines the boundary between the neutral region and the positive region in the arousal level (vertical axis). The lower limit of the index type "Arousal Level" determines the boundary between the neutral region and the negative region in the arousal level (vertical axis). The upper limit of the index type "Autonomic Nervous System Activity Level" determines the boundary between the neutral region and the positive region in the autonomic nervous system activity level (horizontal axis). The lower limit of the index type "Autonomic Nervous System Activity Level" determines the boundary between the neutral region and the negative region in the autonomic nervous system activity level (horizontal axis).

[0083] In the processed psychological plane, the first neutral region NR1, determined by the upper and lower limits of the indicator type "arousal level," is a band-shaped region extending parallel to the horizontal axis. Similarly, in the processed psychological plane, the second neutral region NR2, determined by the upper and lower limits of the indicator type "autonomic nervous system activity level," is a band-shaped region extending parallel to the vertical axis. The neutral region NR established on the psychological plane consists of the first neutral region NR1 and the second neutral region NR2, and its shape is cross-shaped. The widths of the first neutral region NR1 and the second neutral region NR2, which are each composed of bands, may be the same or different. In the example shown in Figure 10, the first neutral region NR1 and the second neutral region NR2 overlap with the vertical axis or the horizontal axis, respectively. The vertical axis and horizontal axis referred to here are axes that were initially predicted (set) from the biosignal values. However, it is possible that at least one of the first neutral region NR1 and the second neutral region NR2 may be located off-center from the axis (the initially planned axis). In other words, determining the neutral region NRs will, as a result, determine the vertical and horizontal axes (passing through the origin). The midpoint between each neutral region NR1 and NR2 may be considered the vertical and horizontal axes.

[0084] The emotion estimation unit 133 generates a calibrated psychological plane and then uses the psychological plane to perform emotion estimation processing. The emotion estimation unit 133 converts information based on biosignals acquired by the acquisition unit 131 into index values ​​using index conversion information in the sensor table 121a. In this embodiment, index values ​​for arousal level and index values ​​for autonomic nervous system activity are obtained using data from the first sensor 31 (sensor type: electroencephalogram sensor BA) and the second sensor 32 (sensor type: heart rate sensor HA).

[0085] The emotion estimation unit 133 then estimates emotions according to the position of the coordinates obtained by plotting each index value on a psychological plane that includes the neutral region NR obtained by the previous processing. In this embodiment, the emotional estimation is performed according to the position of the coordinates obtained by plotting the index value of arousal and the index value of autonomic nervous system activity on a psychological plane (see Figure 10) that includes the neutral region, with arousal level on the vertical axis and autonomic nervous system activity level on the horizontal axis.

[0086] The emotion estimation unit 133 determines that if the coordinate position obtained from plotting the index values ​​is in the neutral region NR, it cannot determine the type of emotion of user U1. In other words, the emotion estimation unit 133 determines that the emotion is in an undetermined state. On the other hand, if the coordinate position obtained from plotting the index values ​​is outside the neutral region NR, it estimates (specifies) that the type of emotion of user U1 corresponds to the emotion at that coordinate position. For example, if the coordinate position is above the first neutral region NR1 in Figure 10 and to the right of the second neutral region NR2 (i.e., in the first quadrant), it estimates (specifies) that the user has emotions such as "happy, joyful, angry, or sad."

[0087] As can be seen from the above, the controller 13 identifies which of the three regions the index value based on the biological signal belongs to: the neutral region, the positive region on one side of the neutral region, and the negative region on the other side of the neutral region. The controller 13 then determines that the person is in an emotionally indecisive state if the index value belongs to the neutral region. This determination of an emotionally indecisive state may be a direct determination, such as setting an "emotionally indecisive state flag." In addition to the direct determination described above, this determination of an emotionally indecisive state may also include an indirect determination, such as changing the subsequent processing content if the index value belongs to the neutral region.

[0088] Furthermore, the controller 13 calculates multiple types of index values. The controller 13 identifies which of the three regions described above each of the multiple types of index values ​​belongs to. If all of the multiple types of index values ​​are outside the neutral region, the controller 13 uses the multiple types of index values ​​and a pre-prepared emotion estimation model to identify the type of emotion. In this embodiment, there are two types of multiple values. However, the configuration may involve three or more types of index values. In this case, the emotion estimation model may be a three-dimensional or higher space rather than a two-dimensional plane.

[0089] As in this embodiment, by configuring the system to estimate emotions by providing a neutral region, the possibility of making incorrect estimations can be reduced in emotion estimation processing, which tends to have many error factors and is prone to misjudgments. Furthermore, a neutral region is created for each user U1. The position and size (width) of the neutral region usually differ for each user U1. By providing a neutral region, emotion estimation can be performed in a way that is tailored to the characteristics of each user, thereby improving the accuracy of emotion estimation.

[0090] Returning to Figure 3, the provisioning unit 134 provides the emotion information estimated by the emotion estimation unit 133 to the terminal device 20 via the network N. Operator O1 then uses the emotion information provided by the provisioning unit 134, such as by viewing it. Operator O1 may also provide the emotion information to user U1, who is an e-sports player.

[0091] [1-3. Estimation method] Next, we will describe the estimation method performed by the estimation device 10.

[0092] Furthermore, a computer program that implements the estimation method of this embodiment in a computer device is included within the scope of this embodiment. Also, a computer-readable non-volatile recording medium that records such a computer program is included within the scope of this embodiment. Moreover, the computer program that implements the estimation method of this embodiment in a computer device may consist of only one program, or it may consist of multiple programs.

[0093] (1-3-1. When performing special tasks) First, we will describe an example of the process when emotion estimation is performed after determining the neutral region by performing a special task. Figure 11 is a flowchart showing an example of the preliminary part of the estimation process performed by the estimation device 10 of the first embodiment. Figure 12 is a flowchart showing an example of the final part of the estimation process performed by the estimation device 10 of the first embodiment. Here, the preliminary part of the estimation process corresponds to the region determination process that determines the neutral region as described above. In the example shown in Figure 11, the preliminary part of the estimation process corresponds to the region determination process when performing a special task. The final part of the estimation process corresponds to the emotion estimation process using an emotion estimation model.

[0094] The process shown in Figure 11 is initiated, for example, when the acquisition unit 131 of the estimation device 10 starts acquiring signals from the biosensor 30, which includes the first sensor 31 and the second sensor 32. This acquisition may be initiated, for example, by an instruction operation from user U1. The process shown in Figure 12 is performed following the process shown in Figure 11. It is preferable that the preceding process shown in Figure 11 is performed before the e-sports event starts.

[0095] In step S11, the region determination unit 132 identifies the target indicator type based on the information from the biosensor 30 acquired by the acquisition unit 131 and the sensor table 121a. In this embodiment, "alertness level" and "autonomic nervous system activity level" are identified as indicator types. Once the indicator type is identified, the process proceeds to the next step S12.

[0096] In step S12, the domain determination unit 132 determines whether the upper and lower limits of the neutral domain for each indicator type have been determined for user U1, who is the target of emotion estimation. Whether the upper and lower limits of the neutral domain for user U1, the target of emotion estimation, have been determined is determined by the information contained in the neutral domain table 121d (see Figure 9). In this embodiment, it is confirmed whether the upper and lower limits of the neutral domain for "arousal level" and "autonomic nervous system activity level" for user U1, the target of emotion estimation, have been determined. If they have been determined (Yes in step S12), the process proceeds to step S20 (see Figure 12). If they have not been determined (No in step S12), the process proceeds to step S13.

[0097] Note that the process in step S12 is not required. That is, regardless of whether the upper and lower limits of the neutral region for each indicator type have been determined in the past, the process of determining the upper and lower limits of the neutral region for each indicator type may be performed. The neutral region may fluctuate depending on, for example, the environment in which user U1 is placed or the physical condition of user U1. For this reason, it is preferable to determine the upper and lower limits of the neutral region before each start of an esports event.

[0098] Furthermore, in this embodiment, if there are any indicator types identified in step S11 for which the upper and lower limits of the neutral region have not yet been determined, the process of determining the upper and lower limits is performed for all previously identified indicator types. However, this is merely an example. In other words, the configuration may be such that the process of determining the upper and lower limits of the neutral region is performed only for indicator types for which the upper and lower limits of the neutral region have not yet been determined.

[0099] In step S13, the region determination unit 132 sets variable x to 1. In this embodiment, if variable x is 1, it is determined to perform processing for the first indicator type out of the two indicator types identified in step S11. If variable x is 2, it is determined to perform processing for the second indicator type out of the two indicator types identified in step S11. Once variable x is set to 1, the process proceeds to the next step S14.

[0100] In step S14, the domain determination unit 132 requests user U1 to perform a first task of the first indicator type because the variable is 1. The task request is made, for example, by displaying the request on the display unit 42 of the game device 40 used by user U1. The task request may also be made by voice guidance instead of, or in addition to, a screen display. Preferably, the request includes the specific content of the first task. Information regarding the content of the first task is contained in the special task table 121c (see Figure 5). In this embodiment, for example, the first indicator type is "awareness level". And the content of the first task is "mentally add up the numbers displayed on the display unit 42". When the request to perform a first task of the first indicator type is made, the process proceeds to the next step S15.

[0101] In step S15, the region determination unit 132 determines the upper limit of the neutral region based on the signal obtained from the first sensor 31 when the first task of the first index type is executed. The start and end of the user U1's task may be determined, for example, by requiring the user U1 to operate an operating member at the start and end of the task, and by the operation information of said operating member. The operating member may be, for example, included in the operating unit 41 of the game device 40. Alternatively, the start and end of the user U1's task may be determined by information obtained from a camera that photographs the user U1, which is placed in the game device 40 or the like. In this embodiment, the region determination unit 132 determines the upper limit of the neutral region in the arousal level of the user U1 who is the target of emotion estimation. The determined upper limit is stored in the neutral region table 121d (see Figure 9). Once the upper limit of the neutral region is determined, the process proceeds to the next step S16.

[0102] In step S16, the domain determination unit 132 requests user U1 to execute the second task of the first indicator type. The processing related to the task request may be the same as in the case of the first task, and a detailed explanation is omitted here. In this embodiment, since the first indicator type is "level of arousal", the content of the second task is "rest and clench your fist". Once the request to execute the second task of the first indicator type is made, the process proceeds to the next step S17.

[0103] In step S17, the region determination unit 132 determines the lower limit of the neutral region based on the signal obtained from the first sensor 31 when the second task of the first index type is executed. The determination of the start and end of the user U1's task may be the same as in the case of the first task. In this embodiment, the region determination unit 132 determines the lower limit of the neutral region in terms of arousal level for user U1, who is the target of emotion estimation. The determined lower limit is stored in the neutral region table 121d. Once the lower limit of the neutral region is determined, the process proceeds to the next step S18.

[0104] In this example, the special tasks are executed in the order of Task 1 followed by Task 2, but this order can be reversed.

[0105] In step S18, the region determination unit 132 performs the operation of adding 1 to the variable x. Once the addition operation on variable x is completed, the process proceeds to the next step S19.

[0106] In step S19, the region determination unit 132 determines whether the variable x is 3 or greater. If the variable x is less than 3 (No in step S19), the process returns to step S14, and the processing from step S14 onward is repeated. However, at this stage, the variable x is 2. For this reason, processing related to the second indicator type is performed. In this embodiment, the second indicator type is "autonomic nervous system activity level". For this reason, the content of the first task required in step S14 is the "standing test". In step S15, the upper limit of the neutral region in the autonomic nervous system activity level of user U1, who is the target of emotion estimation, is determined. Also, the content of the second task required in step S16 is the "Aschnell test". In step 17, the lower limit of the neutral region in the autonomic nervous system activity level of user U1, who is the target of emotion estimation, is determined. If the variable x is 3 or greater (Yes in step S19), the process proceeds to step S20 (see Figure 12).

[0107] In this embodiment, the upper and lower limits of the neutral region are determined in the order of the first indicator type followed by the second indicator type. However, this is merely an example, and the upper and lower limits of the neutral region may be determined in the order of the second indicator type followed by the first indicator type.

[0108] In step S20, the emotion estimation unit 133 processes the pre-stored psychological plane using the neutral region information (upper and lower limits) for each index type obtained in the previous region determination process. Then, the emotion estimation unit 133 performs emotion estimation processing using the psychological plane including the neutral region. In this embodiment, for example, the processed psychological plane shown in Figure 10 is used to estimate emotions. As described above, there are cases where the emotion is determined to be in an undetermined state and the type of emotion is not specified, and cases where the type of emotion is specifically specified. Once the emotion estimation processing using the psychological plane is completed, the process proceeds to the next step S21.

[0109] In step S21, the providing unit 134 provides the emotion estimation result to an external party. The providing unit 134 provides the emotion estimation result to, for example, the terminal device 20. The emotion estimation result may be displayed on the display screen of the terminal device 20 as appropriate. The operator O1 of the terminal device 20 may also perform processing using the emotion estimation result as appropriate. The result of the emotion estimation processing provided to the terminal device 20 may be output to the game device 40 automatically from the terminal device 20, or by operation of the operator O1 of the terminal device 20. The emotion estimation result output to the game device 40 may then be displayed on, for example, the display unit 42.

[0110] Figures 13A and 13B illustrate screens displaying the estimated emotion results. In the examples shown in Figures 13A and 13B, an emotion map is displayed on the screen (more specifically on the right side of the screen) to allow for easy recognition of the estimated emotion results. The emotion map is graphic information that includes, for example, a graph plotting the measurement results (coordinate positions) on a psychological plane. In addition, in the examples shown in Figures 13A and 13B, an area is provided on the screen to display the estimated emotion results as text.

[0111] Figure 13A shows the results when the coordinate positions identified by the two index values ​​are outside the neutral region (NR). In this case, the type of emotion can be identified, and the result of identifying the type of emotion is shown. Figure 13B shows the results when the coordinate positions identified by the two index values ​​are on the neutral region (NR). In this case, the type of emotion cannot be identified because the emotion is in an undetermined state, and this is indicated by text. Note that in the example shown in Figure 13B, if we focus only on autonomic nervous system activity (horizontal axis), the plotted position is not in the neutral region (NR). For this reason, for example, a message such as "The autonomic nervous system activity is estimated to be in a state of 'weak emotion'" may be displayed.

[0112] (1-3-2. If no special tasks are performed) Next, we will describe an example of processing when emotion estimation is performed after determining the neutral region without performing a special task. Figure 14 is a flowchart showing another example of the preliminary part of the estimation process performed by the estimation device 10 of the first embodiment. In the example shown in Figure 14, the preliminary part of the estimation process corresponds to the region determination process when no special task is performed. In this example as well, once the preliminary part of the estimation process is completed, the subsequent part of the estimation process shown in Figure 12 is performed.

[0113] The process shown in Figure 14 is initiated, for example, when the acquisition unit 131 of the estimation device 10 starts acquiring signals from the biosensor 30, which includes the first sensor 31 and the second sensor 32. Preferably, the process shown in Figure 14 is started immediately before the start of the e-sports event (for example, 20 minutes before). This allows the determination of the neutral region to be performed in a state as close as possible to the actual environment in which the e-sports event is held. For example, the user may be made to warm up before the start of the e-sports event (by running the game's warm-up mode), and the process shown in Figure 14 may be performed using the data accumulated during that time. However, the process shown in Figure 14 may be performed several hours or more before the start of the e-sports event. Also, the process shown in Figure 14 may be performed while the user U1 is going about their daily life. Furthermore, the process shown in Figure 14 may, in some cases, be performed after the start of the e-sports event.

[0114] In step S31, the region determination unit 132 identifies the target index type based on the information from the biosensor 30 acquired by the acquisition unit 131. This process is the same as step S11 shown in Figure 11, so a detailed explanation is omitted. Once the index type is identified, the process proceeds to the next step S32.

[0115] In step S32, the domain determination unit 132 determines whether the upper and lower limits of the neutral domain for each index type have been determined for user U1, who is the target of emotion estimation. If they have been determined (Yes in step S32), the process proceeds to step S20 (see Figure 12). If they have not been determined (No in step S32), the process proceeds to step S33. The process in step S32 is the same as that in step S12 shown in Figure 11, so a detailed explanation is omitted. Note that, as in the example shown in Figure 11, the process in step S32 may be omitted.

[0116] In step S33, the region determination unit 132 estimates the upper and lower limits of the neutral region for each indicator type using statistical processing methods such as cluster analysis. The data subject to statistical processing such as cluster analysis is the time-series data of indicator values ​​accumulated for each indicator type. In this embodiment, the upper and lower limits of the neutral region are estimated for each indicator type using the time-series data of the indicator values ​​for "alertness" and "autonomic nervous system activity." Once the upper and lower limits for each indicator type have been estimated, the process proceeds to the next step S34.

[0117] In step S34, the region determination unit 132 determines whether the upper and lower limits of the neutral region have converged for each indicator type. In this embodiment, it is determined whether the upper and lower limits of the neutral region for "alertness" have converged. It is also determined whether the upper and lower limits of the neutral region for "autonomic nervous system activity" have converged. If it is determined that the upper and lower limits of the neutral region for all indicator types have converged (Yes in step S34), the process proceeds to the next step S35. On the other hand, if there are indicator types for which the upper and lower limits of the neutral region have not converged (No in step S34), the process returns to step S33, and the processing from step S33 onward is carried out.

[0118] In step S35, the domain determination unit 132 determines the upper and lower limits of the neutral domain for each indicator type, using the upper and lower limits that were previously determined to have converged. The determined upper and lower limits are stored in the neutral domain table 121d. Once the upper and lower limits of the neutral domain for each indicator type are determined, the process proceeds to step S20 shown in Figure 12, where the emotion estimation process is performed. The emotion estimation process has been described above, so its explanation is omitted here.

[0119] If the criteria (thresholds) for determining when the upper and lower limits have converged are set too strictly, the time it takes for the upper and lower limits to converge will be longer. Therefore, when the amount of accumulated time-series data is small, the criteria can be set loosely to determine the neutral region as quickly as possible. In this way, the accuracy of setting the neutral region will decrease, but it will be possible to perform emotion estimation processing in a short time. Then, while securing a state where emotion estimation can be performed as early as possible, the processing from step S33 onwards shown in Figure 14 may be continued while changing the criteria according to the amount of accumulated data. In this way, the accuracy of setting the neutral region can be gradually increased, and the accuracy of emotion estimation can also be gradually improved.

[0120] The configuration described above, which gradually increases the accuracy of the neutral zone settings, can be used in esports, for example, in the following way: At the beginning of the game, when the accuracy of the neutral zone settings is low, the user U1 is notified of the low accuracy of emotion estimation, and the emotion estimation results are displayed on the screen. When the accuracy of the neutral zone settings increases and the emotion estimation accuracy rises above a certain level, the notification that the emotion estimation accuracy is low is removed, and the emotion estimation results are officially displayed on the screen.

[0121] Furthermore, the configuration described above, which gradually increases the accuracy of the neutral zone setting, can be used in, for example, an autonomous vehicle in the following way: When the accuracy of the neutral zone setting is low, only the emotion estimation result is displayed to the driver. When the accuracy of the neutral zone setting increases and the emotion estimation accuracy rises above a predetermined level, the system automatically switches between autonomous and manual driving according to the emotion estimation result. For example, if the emotion estimation result determines that the driver is irritated, the system can determine that it is dangerous and switch from manual to autonomous driving.

[0122] [1-4. Variations] Next, a modified example of the first embodiment will be described.

[0123] (1-4-1. First variation) In the above configuration, the neutral region determined by the region determination unit 132 is used directly to calibrate the emotion estimation model (psychological plane). Alternatively, the controller 13 may adjust the neutral region depending on the intended use of the emotion estimation results. Specifically, adjusting the neutral region means adjusting the width of the neutral region determined by the region determination unit 132. With such a configuration, the possibility of identifying the type of emotion through the emotion estimation process can be varied depending on the intended use.

[0124] For example, if the goal is to inform user U1 of the estimated emotion type as much as possible, even at the expense of some accuracy in emotion estimation, the width of the neutral region, which forms a band on the psychological plane, can be narrowed. This adjustment reduces the likelihood that the index value obtained from the biosignal will fall within the neutral region. As a result, the likelihood of identifying the emotion type can be increased. One example of a situation where the goal is to inform user U1 of the estimated emotion type as much as possible, even at the expense of some accuracy in emotion estimation, is simply to display the emotion type on the screen without performing any control processing based on the emotion type.

[0125] Furthermore, if you want to identify the type of emotion as accurately as possible, you can adjust the settings so that the width of the neutral region, which forms a band on the psychological plane, is wider. With this adjustment, if the index value obtained from the biosignal is not sufficiently large, it will fall into the neutral region. As a result, the possibility of misidentifying the type of emotion can be reduced. One example of a situation where you want to identify the type of emotion as accurately as possible is when you want to use the emotion estimation results for mental training. Another example of a situation where you want to identify the type of emotion as accurately as possible is when you want to use the type of emotion to control the operation of a vehicle.

[0126] (1-4-2. Second variation) The above configuration uses only an emotion estimation model to estimate emotions, but this is merely an example. When estimating emotions using the emotion estimation model, the controller 13 may take into account the emotion estimation results derived from information other than biological signals. By configuring it in this way, the type of emotion can be estimated more appropriately.

[0127] Figure 15 shows a schematic configuration of a modified version of the estimation system 1 of the first embodiment. As shown in Figure 15, the estimation device (server) 10A of the modified version is also connected to the terminal device 20A via a network N, similar to the first embodiment described above. However, in this modified version, an image signal is input to the terminal device 20A in addition to the biological signal, and this image signal is transmitted to the estimation device 10A, which is different from the embodiment described above. The biological signal is, more precisely, a sensor signal obtained by measuring the biological signal with a biological sensor 30. The image signal is a signal output from a camera (not shown) that photographs the user U1. This camera may be built into, for example, the game device 40. Alternatively, it may be installed separately from the game device 40.

[0128] The estimation device 10A, similar to the first embodiment described above, estimates emotions using the emotion estimation model shown in Figure 10, etc. Here, let's assume that the emotion estimation model estimates that the user U1 has one of the emotions "enjoyment, happiness, anger, or sadness" (corresponding to the first quadrant of the psychological plane). In such a case, the estimation device 10A performs image processing on the user U1's face image, analyzes the facial expression, and classifies the type of emotion. For example, it uses the image information to classify the emotion into one of the three categories: "enjoyment, happiness," "anger," or "sadness." In other words, by taking image information into account, the type of emotion estimated by the emotion estimation model can be classified in more detail.

[0129] In the above example, we have used the emotion estimation model (psychological plane) to estimate emotions in the first quadrant, but classification using image information may also be performed when emotions in other quadrants are estimated. For example, if the types of emotions estimated using the emotion estimation model include both positive and negative emotions, classification using image information may be performed. Furthermore, although image information was used in the above example, other types of information, such as audio information, may also be used.

[0130] Let's explain using the example of estimating the emotions of a vehicle driver. In this case, for example, an in-vehicle camera for a dashcam can be used as the camera that obtains the image information mentioned above. If information other than image information is used, for example, audio information obtained from a microphone in an in-vehicle device such as a dashcam may be used. When using audio information, the driver may be asked predetermined questions, and the driver's responses may be analyzed to classify their emotions. As an example of using information other than image information, CAN (Controller Area Network) information or driving behavior information may also be used.

[0131] Examples of CAN information used to estimate emotions include accelerator information, brake information, and steering angle information. For example, accelerator, brake, and steering angle operations tend to be rough when a driver is angry or irritated, and emotions can be classified by analyzing this information. In addition, driving behavior information can be obtained from at least one of the following: CAN information, radar information, LiDAR (Light Detection and Ranging) information, and image information that captures the surrounding environment of the vehicle. For example, if a tendency to yield to pedestrians or other vehicles is observed, it can be determined that the driver is not angry or sad. Also, for example, if the distance between vehicles is short, it can be determined that the driver is angry or irritated.

[0132] <2. Second Embodiment> Next, a second embodiment will be described. In describing the second embodiment, components similar to those in the first embodiment will be denoted by the same reference numerals, and explanations will be omitted unless they require special explanation.

[0133] [2-1. Estimation System] The estimation system of the second embodiment has the same configuration as the estimation system 1 of the first embodiment. For this reason, a detailed description of the estimation system of the second embodiment will be omitted. The estimation system of the second embodiment includes an estimation device 10B (see Figure 16, described later) and a biosensor 30 for measuring biological signals (see Figure 1). The biosensor 30 includes a first sensor 31 configured as an electroencephalogram sensor and a second sensor 32 configured as a heart rate sensor.

[0134] [2-2. Estimation device] Figure 16 shows an example configuration of the estimation device 10B according to the second embodiment of the present invention. The configuration of the estimation device 10B is generally the same as that of the estimation device 10 of the first embodiment (see Figure 3). Similar to the first embodiment, the estimation device 10B includes a communication unit 11 and a storage unit 12. The estimation device 10B also includes a controller 13B.

[0135] The configuration of controller 13B is the same as that of controller 13 in the first embodiment. Furthermore, the functions of controller 13B are generally the same as those of controller 13 in the first embodiment. However, controller 13B differs from controller 13 in that it includes a suitability estimation unit 135. The suitability estimation unit 135, like the other functional units 131 to 134, is implemented, for example, by software. The suitability estimation unit 135 may also be implemented by hardware using a dedicated IC, or by a combination of software and hardware.

[0136] Furthermore, the estimation device 10B of this embodiment may be configured to include the acquisition unit 131 and the aptitude estimation unit 135 among the functional units 131 to 135, but may not include the domain determination unit 132, the emotion estimation unit 133, and the provision unit 134. In this case, the estimation device 10B may be a device that prepares for emotion estimation. In addition, an emotion estimation device comprising at least the emotion estimation unit 133 among the domain determination unit 132, the emotion estimation unit 133, and the provision unit 134 may be provided separately from the estimation device 10B.

[0137] Before explaining the suitability estimation unit 135, let's explain why it is provided. Information obtained based on biosignals is easily affected by the environment, physical condition, etc. For this reason, at the time of emotion estimation based on biosignals, the person whose emotion is to be estimated may be in a state unsuitable for emotion estimation due to the influence of the environment, etc. If emotion estimation is performed in such a state, there is a high possibility of errors in emotion estimation, and the accuracy of emotion estimation may decrease. In this embodiment, the suitability estimation unit 135 is provided to suppress such a decrease in emotion estimation accuracy.

[0138] The aptitude estimation unit 135 estimates the user U1's aptitude for emotion estimation based on biosignals. Specifically, the aptitude estimation unit 135 (i.e., the controller 13B) gives user U1 multiple types of tasks. The aptitude estimation unit 135 also obtains index values ​​based on biosignals during the execution of each of the multiple types of tasks. Furthermore, the aptitude estimation unit 135 estimates user U1's aptitude for emotion estimation based on the relationships between the multiple index values.

[0139] With this configuration, by having user U1 perform multiple tasks before sentiment estimation, it is possible to estimate user U1's suitability for sentiment estimation. As a result, the likelihood of performing sentiment estimation in a suitable state increases, and the accuracy of sentiment estimation can be improved.

[0140] The aptitude estimation unit 135, which has the functions described above, will now be explained in more detail.

[0141] Each of the multiple tasks that the aptitude estimation unit 135 has user U1 perform is a task that can evoke a predetermined state of physiological response (e.g., level of arousal) based on medical evidence. Furthermore, the magnitude (index value) of the physiological response that each of the multiple tasks can evoke differs from one another. In other words, there is a hierarchy in the magnitude (index value) of the physiological response obtained when user U1 performs each of the multiple tasks.

[0142] In this embodiment, table 121 (see Figure 16) includes a task table 121e for aptitude estimation. The task table 121e for aptitude estimation includes information on the multiple types of tasks described above. Figure 17 shows an example of the task table 121e for aptitude estimation. As shown in Figure 17, the items in the task table 121e for aptitude estimation include "Task ID," "Sensor Type," "Corresponding Indicator Type," "Ranking," and "Task Content." The items "Sensor Type" and "Corresponding Indicator Type" are the same as those in Figure 4A, etc., described above, so their explanation is omitted. Note that "Corresponding Indicator Type" may also be expressed as "Corresponding Physiological Response Type."

[0143] The "Task ID" item in the aptitude estimation task table 121e stores ID data, which is identification information used to identify task information in the aptitude estimation task table 121e.

[0144] The "Ranking" item in the aptitude estimation task table 121e stores ranking information of the magnitude of physiological responses (index values) when user U1 performs the various tasks described above. This ranking information is determined based on medical evidence. For example, if the type of physiological response (index type) is arousal level, the ranking is determined by the magnitude of the index value obtained when performing the task. In the aptitude estimation table 121e, tasks that are judged to have larger index values ​​based on medical evidence are ranked higher (the numbers indicating the ranking become smaller).

[0145] The "Task Content" item in the aptitude estimation task table 121e stores the specific details of the task to be performed by user U1. The task content is determined based on medical evidence, along with the hierarchy mentioned above. For example, if the type of physiological response is arousal level, the highest-ranking task content is "apply tenderness." Specifically, "applying tenderness" means applying pressure that everyone would find painful, such as a foot massage, for a predetermined period of time. Also, for example, if the type of physiological response is arousal level, the second-ranking task content is "mental arithmetic addition of displayed numbers." Also, for example, if the type of physiological response is arousal level, the third-ranking task content is "rest and clench your fist." Also, for example, if the type of physiological response is arousal level, the fourth-ranking task content is "close your eyes and rest (think nothing)."

[0146] The aptitude estimation unit 135, for example, compares the sensor type information of the biosensor 30 worn by user U1 with the task table 121e for aptitude estimation to determine several types of tasks for user U1 to perform. The sensor type information is acquired by the acquisition unit 131 via a terminal device 20 that is connected to the biosensor 30. If user U1 is wearing multiple types of biosensors 30, aptitude estimation may be performed using each type of biosensor 30. Alternatively, if user U1 is wearing multiple types of biosensors 30, aptitude estimation may be performed using only one type of biosensor 30.

[0147] For example, in this embodiment, the biosensors 30 worn by user U1 consist of two sensors: a first sensor 31 and a second sensor 32. If the aptitude estimation task table 121e contains aptitude estimation tasks corresponding to these two sensors 31 and 32, aptitude estimation may be performed using both sensors 31 and 32, or aptitude estimation may be performed using only one of the sensors. When aptitude estimation is performed using only one of the sensors, the type of sensor used for aptitude estimation may be determined, for example, according to pre-prepared priority information. Furthermore, if the aptitude estimation task table 121e contains only aptitude estimation tasks corresponding to one of the sensors, aptitude estimation may be performed using that sensor.

[0148] For example, let's consider the case where the sensor type is an electroencephalogram (EEG) sensor BA. The aptitude estimation unit 135 has user U1 perform four tasks. The order in which the four tasks are performed is not particularly limited, but for example, they are performed in order from the lowest-ranking task (i.e., the one with the lowest level of arousal). The aptitude estimation unit 135 acquires the index value (magnitude of arousal) for each of the four tasks performed by user U1. The index value is obtained using the signal from the EEG sensor BA that measures user U1's brainwaves and the index conversion information contained in the sensor table 121a (see Figure 4A). The aptitude estimation unit 135 determines the order of the four acquired index values ​​according to their magnitudes.

[0149] The aptitude estimation unit 135 then determines whether the sequence of index values ​​obtained when user U1 performs each task matches the sequence in the task table 121e for aptitude estimation. If they match, the aptitude estimation unit 135 estimates that user U1 is in a state suitable for emotion estimation. If they do not match, the aptitude estimation unit 135 estimates that user U1 is in a state unsuitable for emotion estimation.

[0150] In other words, the controller 13B estimates that user U1 is in a state suitable for emotion estimation if the order of magnitudes of the indicator values ​​obtained during the execution of each of the multiple types of tasks matches a pre-set order. Conversely, the controller 13B estimates that user U1 is in a state unsuitable for emotion estimation if the order of magnitudes of the indicator values ​​obtained during the execution of each of the multiple types of tasks differs from the pre-set order. With this configuration, it is possible to easily determine whether user U1 is in a state suitable for emotion estimation based on the order of magnitudes of physiological responses determined based on medical evidence.

[0151] In the example shown above, if the type of physiological response is arousal level, there are four types of tasks that user U1 is required to perform for aptitude estimation (see Figure 17). However, this is merely an example. User U1 may be required to perform multiple types of tasks for aptitude estimation, for example, just two. If the system is configured to have user U1 perform only two tasks with significantly different magnitudes of physiological responses, the order obtained when user U1 performs the tasks may not change from the set order even if user U1's physical condition or environment changes. In other words, the significance of aptitude estimation may be diminished. Taking these points into consideration, it is preferable to determine the types of tasks that user U1 is required to perform in such a way that the aptitude estimation is appropriate (so that the order obtained when performing tasks changes appropriately in response to environmental changes, etc.).

[0152] In this embodiment, when user U1 is estimated to be in a state suitable for emotion estimation, the region determination unit 132 and emotion estimation unit 133 perform emotion estimation in the same manner as in the first embodiment described above. That is, when user U1 is estimated to be in a state suitable for emotion estimation, the controller 13B performs emotion estimation using index values ​​based on biosignals. The controller 13B identifies which of the three regions, consisting of the neutral region, the positive region, and the negative region, the index value based on biosignals belongs to. If the index value belongs to the neutral region, the controller 13B determines that the user is in an emotionless state. The controller 13B obtains multiple types of index values, and if all of the multiple types of index values ​​are outside the neutral region, it estimates the type of emotion using the multiple types of index values ​​and a pre-prepared emotion estimation model.

[0153] The emotion estimation method in the second embodiment may be completely different from the emotion estimation method in the first embodiment. The emotion estimation method may also be a known method.

[0154] Furthermore, in this embodiment, the emotion estimation result is provided to the terminal device 20 via the network N by the providing unit 134.

[0155] [2-3. Estimation method] Next, we will describe the estimation method performed by the estimation device 10B.

[0156] Furthermore, a computer program that implements the estimation method of this embodiment in a computer device is included within the scope of this embodiment. Also, a computer-readable non-volatile recording medium that records such a computer program is included within the scope of this embodiment. Moreover, the computer program that implements the estimation method of this embodiment in a computer device may consist of only one program, or it may consist of multiple programs.

[0157] Figure 18 is a flowchart showing an example of the aptitude estimation process performed by the estimation device 10B of the second embodiment. The process shown in Figure 18 is started, for example, when the acquisition unit 131 of the estimation device 10B starts acquiring signals from the biosensor 30, which includes the first sensor 31 and the second sensor 32. It is preferable that the aptitude estimation process shown in Figure 18 is performed before the start of the e-sports event.

[0158] In step S41, the aptitude estimation unit 135 sets the variable n to zero. Note that the variable n represents the number of times. Once the variable n is set to zero, the process proceeds to the next step S42.

[0159] In step S42, the aptitude estimation unit 135 identifies target index values ​​based on the information from the biosensor 30 acquired by the acquisition unit 131 and the aptitude estimation table 121e. In this embodiment, for example, an electroencephalogram (EEG) sensor is identified as the sensor type, and the level of arousal is identified as the index. The aptitude estimation unit 135 then sequentially assigns tasks corresponding to the identified index to the user U1, and acquires index values ​​based on biosignals while the user is performing the assigned tasks.

[0160] In this embodiment, for example, the user U1 is sequentially given tasks such as "close eyes and rest (not thinking about anything)," "rest and clench your fist," "mentally add the displayed numbers," and "apply pressure to the user." Each time user U1 performs a task, an index value (an index value of alertness) based on electroencephalography is acquired. The tasks may be given, for example, using the screen display of the game device 40 used by user U1. Alternatively, the tasks may be given by voice guidance instead of screen display, or in addition to screen display. Once the aptitude estimation unit 135 acquires the index value for each given task, it proceeds to the next step S43.

[0161] In step S43, the aptitude estimation unit 135 arranges the multiple (four in this embodiment) index values ​​obtained in response to the user U1's task execution in order of ranking from largest to smallest. The aptitude estimation unit 135 then determines whether the ranking obtained from the user U1's task execution is the same as the pre-set ranking (the ranking shown in the item "Ranking") stored in the aptitude estimation table 121e. If the two rankings are the same (Yes in step S43), the aptitude estimation unit 135 proceeds to the next step S44. On the other hand, if the two rankings are different (No in step S43), the aptitude estimation unit 135 returns to step S42.

[0162] In step S44, the aptitude estimation unit 135 adds 1 to the variable n. Once the process of adding 1 to the variable n is completed, the process proceeds to the next step S45.

[0163] In step S45, the suitability estimation unit 135 determines whether the variable n is greater than or equal to a pre-prepared threshold number nth. The threshold number nth is determined, for example, by conducting an experiment. If the variable n is greater than or equal to the threshold number (Yes in step S45), it is estimated that user U1 is in a suitable state for emotion estimation, and the process proceeds to step S11 shown in Figure 11 or step S31 shown in Figure 14. After the completion of the process shown in Figure 11 or Figure 14, the process proceeds to step S20 shown in Figure 12 to perform emotion estimation. Note that the calibration preparation process shown in Figure 11 or Figure 14 may be omitted. On the other hand, if the variable n is less than the threshold number (No in step S45), the process returns to step S42.

[0164] As can be seen from the above, in this embodiment, when the number of times the sequence obtained by the execution of user U1's task matches the pre-prepared setting sequence reaches a predetermined number of times, user U1 is presumed to be in a state suitable for emotion estimation. However, this is an example, and user U1 may be presumed to be in a state suitable for emotion estimation if the sequence obtained by the execution of user U1's task matches the pre-prepared setting sequence just once.

[0165] Furthermore, in this embodiment, if the number of times the sequence obtained by user U1's task execution matches the pre-prepared setting sequence does not reach a predetermined number of times, it is presumed that user U1 is in an unsuitable state for emotion estimation, and the suitability estimation task is provided indefinitely. However, this is an example. For example, when the number of times the sequence obtained by user U1's task execution matches the pre-prepared setting sequence reaches a predetermined number, it may be presumed that user U1 is in an unsuitable state for emotion estimation, and the suitability estimation process may be terminated.

[0166] [2-4. Variations] Next, a modified example of the second embodiment will be described.

[0167] (2-4-1. First variation) Figure 19 is a flowchart showing a modified example of the suitability estimation process performed by the estimation device 10B of the second embodiment.

[0168] In step S51, the aptitude estimation unit 135 identifies the target indicators based on the information from the biosensor 30 acquired by the acquisition unit 131 and the aptitude estimation table 121e. The aptitude estimation unit 135 then sequentially assigns tasks corresponding to the identified indicators to the user U1, and acquires indicator values ​​based on biosignals while the user is performing the assigned tasks. These processes are the same as those in step S42 of Figure 18 described above. Once the aptitude estimation unit 135 has acquired the indicator values ​​for each assigned task, it proceeds to the next step S52.

[0169] In step S52, the aptitude estimation unit 135 arranges the multiple index values ​​obtained in response to user U1's task execution in order of ranking from largest to smallest. The aptitude estimation unit 135 then compares the ranking obtained from user U1's task execution with the pre-set ranking (the ranking shown in the item "Ranking") stored in the aptitude estimation table 121e. For example, this comparison may result in the two rankings being a perfect match. Alternatively, for example, this comparison may result in one change in the order of the rankings between the two rankings. For example, let's consider the case where the physiological response (index) is arousal level. One example of a case where there is one change in the order of the rankings is when the index value obtained when "rest and clench your fist" is performed is ranked 2nd, and when "mentally add the displayed numbers" is performed is ranked 3rd. Once the comparison results are obtained, the process proceeds to the next step S53.

[0170] In step S53, the aptitude estimation unit 135 estimates the aptitude level according to the comparison results obtained earlier. The aptitude level table 121f (see Figure 20) is used to estimate the aptitude level. Figure 20 is a diagram showing an example of the aptitude level table 121f. As shown in Figure 20, the items of the aptitude level table 121f include "Aptitude Estimation Task ID", "Sensor Type", "Corresponding Indicator Type", "Comparison Status", and "Aptitude Level". The items "Sensor Type" and "Corresponding Indicator Type" are the same as those in Figure 17 above, so their explanation is omitted.

[0171] The "Aptitude Task ID" item in the aptitude level table 121f stores information corresponding to the "Task ID" item in the aptitude estimation task table 121e (see Figure 17).

[0172] The "Comparison Status" item in the aptitude level table 121f stores information to be compared with the comparison result in step S52. Note that the number of tasks may differ depending on the type of physiological response. For this reason, the correspondence between "Comparison Status" and "Aptitude Level" is not necessarily unique. Taking this into consideration, in this example, a "Comparison Status" is provided for each type of physiological response. For example, if the type of physiological response (indicator type) is arousal level, the information in the comparison status includes "Exact match," "One change in the order of events," and "Other." "Other" is the case when the comparison result is neither "Exact match" nor "One change in the order of events."

[0173] Next, we will explain using the example of a physiological response type being arousal level. "Other" includes cases where there is one instance of a change in rank between two positions. For example, when "resting and clenching a fist" is performed, the index value obtained is ranked 1st, and when user U1 performs "tendering pressure," the index value obtained is ranked 3rd. "Other" also includes cases where there are two instances of rank changes. For example, when both the following first and second cases are met. The first case is when, for example, "resting and clenching a fist" is performed, the index value obtained is ranked 4th, and when "closed eyes and resting" is performed, the index value obtained is ranked 3rd. The second case is when, for example, "tendering pressure" is performed, the index value obtained is ranked 2nd, and when "mentally adding the displayed numbers" is performed, the index value obtained is ranked 1st.

[0174] The "Aptitude Level" column in Aptitude Level Table 121f contains information about the aptitude level determined by the content of the comparison results. In the example shown in Figure 20, the aptitude level is "Aptitude" when the comparison result is "Exact Match". The aptitude level is "Somewhat Unstable" when the comparison result is "Other". The aptitude level is "Unsuitable" when the comparison result is "Other". Note that in this example there are three aptitude levels, but this is just an example, and there may be four or more levels.

[0175] Once the aptitude level is estimated, the process proceeds to the next step, S54.

[0176] In step S54, the aptitude estimation unit 135 determines whether or not to perform emotion estimation for user U1. For example, in a configuration where the aptitude level is estimated in three stages as shown in Figure 20, it may be determined to perform emotion estimation if the user is at the top two levels. The top two levels are "suitable" and "somewhat unstable". In the case of the lowest level, it may be determined not to perform emotion estimation. The lowest level is "unsuitable".

[0177] If it is determined that emotion estimation should be performed (Yes in step S54), the process proceeds to step S11 shown in Figure 11 or step S31 shown in Figure 14. After the completion of the process shown in Figure 11 or Figure 14, the process proceeds to step S20 shown in Figure 12, where emotion estimation is performed. If it is determined that emotion estimation should not be performed (No in step S54), the process proceeds to step S55.

[0178] In step S55, the aptitude estimation unit 135 performs a process to notify user U1 that he is unsuitable for emotion estimation. For example, the aptitude estimation unit 135 causes the screen of the game device 40 (see Figure 1) used by user U1 to display a notification that user U1 is unsuitable for emotion estimation. Figure 21 is an example of a screen that notifies user U1 that he is unsuitable for emotion estimation. As shown in Figure 21, if user U1 is unsuitable for emotion estimation, he may be prompted to perform an aptitude check again, for example, after 20 minutes.

[0179] As can be seen from the above, in this modified example, the controller 13B estimates the user's level of suitability for emotion estimation according to the degree of discrepancy between the order of magnitude of the index values ​​obtained when each of the multiple types of tasks is executed and the pre-set order which is the order set for the multiple types of tasks. With this configuration for estimating the level of suitability, the controller 13B can change the way in which the results of emotion estimation are used according to the level.

[0180] For example, if the aptitude level is estimated to be "appropriate," the emotion estimation result may be displayed normally as shown in Figures 13A and 13B above. On the other hand, if the aptitude level is estimated to be "somewhat unstable," the emotion estimation result may be displayed provisionally, as shown in Figure 22. Figure 22 shows an example of provisionally displaying the emotion estimation result on the screen. In Figure 22, a cautionary note is included in the screen display to inform the user U1 that it is a provisional display. Furthermore, taking the case of autonomous driving control as an example, if the aptitude level is "appropriate," the system may be configured to perform autonomous driving-related control according to the emotion estimation result. On the other hand, if the aptitude level is "somewhat unstable," the emotion estimation result may be simply displayed on the in-vehicle device screen for reference only and not used for autonomous driving-related control.

[0181] Furthermore, the controller 13B may adjust the neutral region NR (see Figure 10) according to the aptitude level. By configuring it in this way, the possibility of errors in emotion estimation can be suppressed. For example, when the aptitude level is "somewhat unstable," the width of the neutral region may be adjusted to be wider than when the aptitude level is "appropriate."

[0182] (2-4-2. Second variation) Figure 23 is a flowchart showing another variation of the aptitude estimation process performed by the estimation device 10B of the second embodiment. The flowchart shown in Figure 23 is generally similar to the flowchart shown in Figure 19. For this reason, we will focus on explaining the differences. In the example shown in Figure 23, if it is determined that emotion estimation should not be performed (No in step S54), the process does not immediately proceed to step S55, but rather the process in step S56 is performed beforehand. This is different from the flowchart shown in Figure 19. In this variation, instead of estimating the aptitude level in step S53, an estimation of whether or not the state is suitable for emotion estimation may be performed.

[0183] In step S56, the suitability estimation unit 135 determines whether there are other means for estimating emotions. Other means may be, for example, other biosensors 30 worn by user U1. For example, even if the first sensor 31, which is an electroencephalogram (EEG) sensor, determines that the user is in an unsuitable state for emotion estimation, the second sensor 32, which is a heart rate sensor, may determine that the user is in a suitable state for emotion estimation. In such cases, emotion estimation may be performed using the heart rate sensor instead of the EEG sensor. Alternatively, other means may be, for example, a camera (an example of a sensor) that captures an image of user U1's face. That is, emotion estimation may be performed using an image of user U1's face captured by the camera.

[0184] As can be seen from the above, if the controller 13B estimates that user U1 is in an unsuitable state for emotion estimation, it may perform emotion estimation using information from a sensor different from the one that provided the information leading to that estimation. By configuring it in this way, the possibility of a situation where no emotion estimation result can be obtained can be suppressed.

[0185] <3. Things to keep in mind> The various technical features disclosed in the embodiments for carrying out the invention as described herein can be modified in various ways without departing from the spirit of the technical creation. Furthermore, the multiple embodiments and modifications disclosed in the embodiments for carrying out the invention as described herein may be combined to the extent possible. [Explanation of symbols]

[0186] 1. Estimation System 10, 10A, 10B... Estimation device 13, 13B... Controller 30.. Biosensors

Claims

1. An estimation device equipped with a controller, The aforementioned controller, Give the user multiple types of tasks, Obtain index values ​​based on biosignals during the execution of each of the aforementioned multiple tasks, An estimation device that estimates the user's suitability for emotion estimation based on the relationships between a plurality of the aforementioned index values.

2. The aforementioned controller, When the order of magnitude of the index values ​​obtained during the execution of each of the aforementioned tasks matches the pre-set order for the aforementioned tasks, it is estimated that the user is in a state suitable for emotion estimation. The estimation device according to claim 1, wherein if the order of magnitudes of the index values ​​obtained during the execution of each of the aforementioned multiple tasks differs from the set order, the device estimates that the user is in a state unsuitable for emotion estimation.

3. The estimation device according to claim 1, wherein the controller estimates the level of suitability for the user's emotion estimation according to the degree of discrepancy between the order of magnitudes of the index values ​​obtained when each of the multiple types of tasks is executed and the set order which is a pre-set order for the multiple types of tasks.

4. The estimation device according to claim 1, wherein the controller performs emotion estimation using index values ​​based on biosignals when it is estimated that the user is in a state suitable for emotion estimation.

5. The aforementioned controller, If the user is estimated to be in a state suitable for emotion estimation, emotion estimation is performed using index values ​​based on biosignals. The estimation device according to claim 3, wherein the form of using the emotion estimation result is changed according to the level.

6. The aforementioned controller, If the user is estimated to be in a state suitable for emotion estimation, emotion estimation is performed using index values ​​based on biosignals. The system identifies which of the three regions—the neutral region, the positive region on one side of the neutral region, and the negative region on the other side of the neutral region—the index value based on the biological signal belongs to. The estimation device according to claim 3, which determines that the state of emotion is indecisive when the index value falls within the neutral range.

7. The estimation device according to claim 6, wherein the controller adjusts the neutral region according to the level.

8. The estimation device according to claim 4, wherein the controller, when it is estimated that the user is in a state unsuitable for emotion estimation, performs emotion estimation using information from a sensor different from the sensor that led to the estimation of unsuitability.

9. An estimation system comprising an estimation device for estimating emotions and a biosensor for measuring biological signals, The estimation device is, Give the user multiple types of tasks, Obtain index values ​​based on biosignals during the execution of each of the aforementioned multiple tasks, Based on the relationships between multiple aforementioned index values, the suitability of the user for emotion estimation is estimated. An estimation system that performs emotion estimation using index values ​​based on biosignals when the user is estimated to be in a state suitable for emotion estimation.

10. Give the user multiple types of tasks, Obtain index values ​​based on biosignals during the execution of each of the aforementioned multiple tasks, Based on the relationships between multiple aforementioned index values, the suitability of the user for emotion estimation is estimated. An estimation method in which a device performs the processing.

Images