Emotion estimation device

Abstract

Provided is an emotion estimation device capable of carrying out highly accurate emotion estimation, while improving the convenience with respect to acquiring biological information when an emotion is estimated. An emotion estimation device (1) has: an emotion estimation model (30) for estimating an emotion on the basis of biological information (Bi); and a control unit (20) for estimating the emotion of a user (U) on the basis of the biological information (Bi) of the user (U). The emotion estimation model (30) includes an nth degree biological information distillation model made by using n-1th degree knowledge information. The control device (20) uses the nth degree biological information distillation model to estimate the emotion of the user (U) on the basis of the biological information (Bi) of the user (U) which is included in the types of biological information (Bi) required for estimating emotion in a first degree model (100), and for which the number of types of the biological information (Bi) of the user (U) is less than the number of types required to estimate emotion in a first-degree model (100). Information pertaining to the estimated emotion is output as emotion information (Ei).

Description

Emotion estimation device

[0001] The present invention relates to an emotion estimation device.

[0002] A person's emotions can affect their behavior, thoughts, and physical state. Furthermore, a person's behavior, thoughts, and physical reactions can affect their emotions. For example, a person's behavior, way of thinking, and physical state change when they feel joy, sadness, anger, disgust, fear, surprise, etc. Furthermore, a person's emotions change when they move their body, feel fatigue, sleepiness, or pain, receive external stimuli, or recall memories, etc. In this way, emotions and physical states are related.

[0003] Emotions such as joy, sadness, anger, disgust, fear, and surprise are difficult to directly recognize because they are mental activities. On the other hand, emotions are related to physical states. Therefore, techniques for estimating a person's emotions from changes in their mental and physical states are known. Patent Literature 1 discloses an emotion analysis method and emotion analysis device for analyzing a subject's emotions based on their heart rate signal, which is one type of biometric information representing the subject's mental and physical state. The emotion analysis device analyzes the acquired heart rate signal of the subject and calculates feature quantities such as the RR interval, the power spectrum of the low-frequency band, the power spectrum of the high-frequency band, and the power spectrum ratio. Furthermore, the emotion analysis device estimates emotions associated with the heart rate signal based on the feature quantities using an emotion analysis model created by learning.

[0004] Human biometric information, such as electrocardiogram signals and pulse waves, can be easily measured using a wearable sensor, such as an easily wearable wristband. By wearing a wearable sensor, the heart rate signal can be measured in real time under various circumstances. Therefore, the emotion analysis device of Patent Document 1, which utilizes easily measurable heart rate signals, improves the convenience of emotion estimation.

[0005] EP 3263024

[0006] The emotion analysis model that estimates emotions based on heart rate signals can improve emotion estimation accuracy by training using training data that combines various patterns of heart rate signals from various subjects with the subjects' emotions. However, when analyzing emotions using training data consisting of a combination of one type of biometric information and at least one emotion, differences in biometric information due to different emotions are difficult to clarify. Therefore, when training is performed using training data containing only one type of biometric information, the emotion analysis model must be trained using a large amount of training data. Moreover, measuring heart rate signals to create a large amount of training data requires many subjects, time, and effort. For this reason, it has been difficult to improve emotion estimation accuracy with emotion analysis models that estimate a subject's emotion based on one type of biometric information.

[0007] On the other hand, when analyzing emotions using training data composed of a combination of multiple types of biometric information, including electrocardiogram signals, pulse waves, electroencephalograms, sweating, exhalation, etc., and at least one emotion, differences in the biometric information due to different emotions tend to become clear. Therefore, when learning is performed using training data including multiple types of biometric information, the emotion analysis model can improve the accuracy of emotion estimation using a smaller amount of training data than the amount of training data used when learning using training data including only one type of biometric information.

[0008] However, when estimating a subject's emotion based on multiple types of biometric information, the emotion analysis device must simultaneously acquire multiple types of biometric information from the subject. Therefore, the convenience of the emotion analysis model that estimates the subject's emotion based on the multiple types of biometric information decreases as the number of types of biometric information acquired increases. Therefore, there is a need for an emotion estimation device that can estimate emotions with high accuracy while improving the convenience of acquiring biometric information when estimating emotions.

[0009] An object of the present invention is to provide a feeling estimation device that is capable of estimating feelings with high accuracy while improving the convenience of acquiring biometric information when estimating feelings.

[0010] The present inventors have conceived of a configuration of an emotion estimation device that is capable of estimating emotions with high accuracy while improving the convenience of acquiring biometric information when estimating emotions.

[0011] A feeling estimation device according to one embodiment of the present invention includes: a memory unit that stores biometric information; and a control unit that has a feeling estimation model used to estimate a feeling based on the biometric information, acquires a subject's biometric information and stores it in the memory unit, estimates the subject's feeling based on the subject's biometric information stored in the memory unit using the feeling estimation model, and outputs information related to the estimated feeling as feeling information.

[0012] Here, a model used to estimate emotions based on multiple types of biometric information is defined as a primary model, and a model created using n-1-order knowledge information obtained by an n-1-order biometric information distillation model, such as a secondary biometric information distillation model created using primary knowledge information obtained by the primary model, or a tertiary biometric information distillation model created using secondary knowledge information obtained by the secondary biometric information distillation model, is defined as an n-order biometric information distillation model (n is an integer of 2 or greater).

[0013] The emotion estimation model includes the n-th order biometric information distillation model used to estimate the emotion of the subject based on the biometric information, which is included in the types of biometric information necessary for emotion estimation in the primary model and is a number of types less than the number of types of biometric information necessary for emotion estimation in the primary model. The control unit utilizes the n-th order biometric information distillation model to estimate the emotion of the subject based on the biometric information of the subject, which is included in the types of biometric information necessary for emotion estimation in the primary model and is a number of types less than the number of types of biometric information necessary for emotion estimation in the primary model, and outputs information related to the estimated emotion as emotion information.

[0014] In the above configuration, the emotion estimation model includes an n-th order biometric information distillation model created using n-1-th order knowledge information obtained from an n-1-th order biometric information distillation model. The n-1-th order knowledge information inherits information for estimating emotions based on the subject's biometric information, obtained by a first order model used to estimate emotions based on multiple types of biometric information through learning based on multiple types of biometric information. In other words, by utilizing the n-1-th order knowledge information, the n-th order biometric information distillation model inherits privileged information, which is information not included in the biometric information used for learning. Therefore, even when the n-th order biometric information distillation model is used to estimate the subject's emotion based on a number of types of biometric information that are included in the types of biometric information required for estimating the subject's emotion in the emotion estimation model and are fewer than the number of types of biometric information required for estimating the subject's emotion in the first order model, the control unit may be able to achieve estimation accuracy equivalent to that of the first order model. As a result, by using biometric information that can be easily obtained from the subject for emotion estimation, it is possible to improve the convenience of acquiring biometric information when estimating the subject's emotion while also improving emotion estimation accuracy.

[0015] According to another aspect, the emotion estimation device of the present invention preferably includes the following configuration: the emotion estimation device is created using the first-order knowledge information obtained by the first-order model created based on multiple types of biometric information including first biometric information, which is at least one of the biometric information that is not easily affected by the subject's surrounding environment, the subject's work, or the subject's behavior, the biometric information that can be easily obtained from the subject, or the biometric information that is not easily affected by the subject's surrounding environment, the subject's work, or the subject's behavior and that can be easily obtained from the subject, or the n-1-order knowledge information obtained by the n-1-order biometric information distillation model created based on the biometric information including the first biometric information. The control unit estimates the emotion of the subject based on the first biometric information using the n-th order biometric information distillation model, and outputs information related to the estimated emotion as the emotion information.

[0016] As described above, the nth-order biometric information distillation model is used to estimate the subject's emotion based on multiple types of biometric information, including first biometric information. Therefore, the control unit can use the nth-order biometric information distillation model to estimate the subject's emotion based on biometric information, including biometric information that can be easily acquired in a relatively stable state, from among the multiple types of biometric information of the subject. Furthermore, the emotion estimation accuracy of the nth-order biometric information distillation model is further improved by using multiple types of biometric information from the first biometric information. Thus, by using biometric information that can be easily acquired from the subject to estimate the subject's emotion, the convenience of acquiring biometric information when estimating the subject's emotion can be improved while also improving the emotion estimation accuracy.

[0017] According to another aspect, the emotion estimation device of the present invention preferably includes the following configuration. The nth-order biometric information distillation model is created using the first-order knowledge information obtained by the first-order model created based on multiple types of biometric information including second biometric information, the second biometric information being at least one of the biometric information that is easily affected by the subject's surrounding environment, the subject's work, or the subject's behavior, the biometric information that is difficult to easily obtain from the subject, or the biometric information that is easily affected by the subject's work or the subject's behavior and is difficult to easily obtain from the subject, or the n-1th-order knowledge information obtained by the n-1th-order biometric information distillation model created based on the biometric information that includes the second biometric information. The control unit estimates the emotion of the subject based on the subject's biometric information that does not include the second biometric information, using the nth-order biometric information distillation model, and outputs information about the estimated emotion as the emotion information.

[0018] As described above, the nth-order biometric information distillation model is created using knowledge information obtained by a first-order model or an n-1th-order biometric information distillation model created based on multiple types of biometric information including the second biometric information. Meanwhile, the control unit uses the nth-order biometric information distillation model to estimate the subject's emotion based on biometric information that does not include the second biometric information among multiple types of biometric information of the subject, and that includes biometric information that can be easily acquired in a relatively stable state. In other words, by using the nth-order biometric information distillation model that inherits knowledge information obtained by the first-order model or the n-1th-order biometric information distillation model, the control unit can estimate the subject's emotion based on knowledge information created based on the second biometric information and biometric information that does not include the second biometric information. This improves the convenience of acquiring biometric information when estimating the subject's emotion while improving the accuracy of emotion estimation.

[0019] According to another aspect, the emotion estimation device of the present invention preferably includes the following configuration. The nth-order biometric information distillation model is created using the primary knowledge information obtained by the primary model created based on multiple types of biometric information including third biometric information that is at least two of electroencephalograms, sweating, respiration, pulse waves, or electrocardiogram signals, or the n-1th-order knowledge information obtained by the n-1th-order biometric information distillation model created based on the biometric information including the third biometric information. The control unit uses the nth-order biometric information distillation model to estimate the emotion of the subject based on the biometric information that includes a small number of types but includes some of the multiple types of biometric information included in the third biometric information, and outputs information related to the estimated emotion as emotion information.

[0020] As described above, the nth-order biometric information distillation model is created using knowledge information obtained by a primary model or an n-1th-order biometric information distillation model created based on multiple types of biometric information including the third biometric information. Meanwhile, the nth-order biometric information distillation model is used to estimate emotions based on a small number of types of biometric information that includes some of the multiple types of biometric information included in the third biometric information. In other words, by using the nth-order biometric information distillation model that inherits knowledge information obtained by the primary model or the n-1th-order biometric information distillation model, the control unit can estimate the subject's emotion based on knowledge information created based on the third biometric information and a smaller number of types of biometric information than the third biometric information. This improves the convenience of acquiring biometric information when estimating the subject's emotion while also improving the accuracy of emotion estimation.

[0021] According to another aspect, the emotion estimation device of the present invention preferably includes the following configuration: the control unit utilizes the emotion estimation model to estimate the emotion of the subject based on bio-information related to at least one of an electrocardiogram signal, respiration, or a pulse wave among bio-information related to an electroencephalogram, sweating, respiration, an electrocardiogram signal, or a pulse wave of the subject, and outputs information related to the estimated emotion as the emotion information.

[0022] As described above, the control unit can estimate the emotion of the subject based on biometric information related to at least one of the subject's electrocardiogram signal, respiration, or pulse wave, which is relatively easy to measure, among biometric information related to the subject's electrocardiogram signal, respiration, or pulse wave, by utilizing an emotion estimation model including an n-th order biometric information distillation model that inherits the n-1-th order knowledge information. Furthermore, the emotion estimation accuracy of the emotion estimation device may be further improved by utilizing multiple pieces of biometric information related to the subject's electrocardiogram signal, respiration, or pulse wave. This allows for the use of biometric information that can be easily acquired from the subject for emotion estimation, thereby improving the convenience of acquiring biometric information when estimating the subject's emotion and improving the emotion estimation accuracy.

[0023] According to another aspect, the feeling estimation device of the present invention preferably includes the following configuration. The n-1th-order biometric information distillation model includes a plurality of internal models used for emotion estimation based on a plurality of types of biometric information. The feeling estimation model includes the n-th-order biometric information distillation model created using a plurality of n-1th-order knowledge information obtained based on each of the plurality of internal models based on the plurality of types of biometric information using the plurality of internal models.

[0024] As described above, an n-1-order biometric information distillation model having multiple internal models reduces error and variability in the emotion estimated based on the subject's biometric information by, for example, ensemble learning that averages the outputs of the internal models, or ensemble learning that improves the parameters of other internal models by taking into account the parameters of one internal model. Thus, the n-th order biometric information distillation model trained using the n-1-order knowledge information obtained by the n-1-order biometric information distillation model can reduce variability in the estimation accuracy of the subject's emotion based on the subject's biometric information. As a result, by using biometric information that can be easily obtained from the subject to estimate the emotion, it is possible to improve the convenience of obtaining biometric information when estimating the subject's emotion while also improving the accuracy of the emotion estimation.

[0025] According to another aspect, the feeling estimation device of the present invention preferably includes the following configuration: The feeling estimation model includes the n-th order biological information distillation model created by learning, using the n-th order biological information distillation model, the n-th order knowledge information obtained based on the multiple types of biological information including biological information related to at least one of an electrocardiogram signal, respiration, or pulse wave among biological information related to an electroencephalogram, sweating, respiration, pulse wave, or an electrocardiogram signal, and the biological information including biological information related to at least one of an electrocardiogram signal, respiration, or pulse wave.

[0026] As described above, the nth-order biological information distillation model is created using the (n-1)th-order knowledge information and biological information related to at least one of an electrocardiogram signal, respiration, or pulse wave. Therefore, the control unit can estimate the subject's emotion based on at least one of biological information related to an electrocardiogram signal, respiration, or pulse wave, which can be easily measured using the nth-order biological information distillation model. This improves the convenience of acquiring biological information when estimating the subject's emotion while also improving the accuracy of emotion estimation.

[0027] According to another aspect, the emotion estimation device of the present invention preferably includes the following configuration: the emotion estimation model includes the n-th order biometric information distillation model, which is a neural network model created using the n-th order knowledge information obtained based on the n-th order biometric information distillation model, which is a neural network model.

[0028] As described above, the n-th order biometric information distillation model is a neural network model created using the n-1-th order biometric information distillation model, which is a neural network model having intermediate layers. Since the n-th order biometric information distillation model has an increased number of parameters due to the intermediate layers, the emotion estimation accuracy improves in accordance with the number of intermediate layers. This makes it possible to improve the convenience of obtaining biometric information when estimating the emotion of the subject while also improving the emotion estimation accuracy by using a type of biometric information that can be easily obtained from the subject for emotion estimation.

[0029] The terminology used in this specification is used for the purpose of defining particular embodiments only, and is not intended to limit the invention.

[0030] As used herein, "and / or" includes all combinations of one or more of the associated listed members.

[0031] In this specification, the use of "including," "comprising," or "having" and variations thereof identify the presence of stated features, steps, operations, elements, components, and / or equivalents thereof, but may include one or more of the steps, operations, elements, components, and / or groups thereof.

[0032] As used herein, the terms "attached," "connected," "coupled," and / or their equivalents are used broadly to encompass both "direct and indirect" attachments, connections, and couplings. Furthermore, "connected" and "coupled" are not limited to physical or mechanical connections or couplings, but can also include direct or indirect electrical connections or couplings.

[0033] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0034] Terms defined in commonly used dictionaries should be construed to have a meaning consistent with the relevant art and meaning in the context of this disclosure, and should not be construed in an idealized or overly formal sense unless expressly defined herein.

[0035] It is understood that in describing the present invention, several techniques and processes are disclosed, each of which has distinct advantages and can be used in conjunction with one or more, or in some cases all, of the other disclosed techniques.

[0036] Thus, for the sake of clarity, the description of the present invention refrains from unnecessarily repeating every possible combination of the individual steps, but the specification and claims should be read with the understanding that all such combinations are within the scope of the present invention.

[0037] In this specification, an embodiment of a feeling estimation device according to the present invention will be described.

[0038] In the following description, numerous specific examples are set forth to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without these specific examples.

[0039] Accordingly, the following disclosure is to be considered as illustrative of the present invention and is not intended to limit the invention to the specific embodiments illustrated by the following drawings or description.

[0040] [Subject] In this specification, a subject is a person whose emotion is to be estimated by an emotion estimation device. Biometric information of the subject is acquired by the emotion estimation device. Furthermore, the emotion of the subject is estimated by the emotion estimation device based on the acquired biometric information.

[0041] [Information about the subject] In this specification, information about the subject includes information about the subject, such as the subject's gender, age, height, weight, and other physical characteristics, personality, hobbies, and preferences. The information about the subject is input into the emotion estimation device in advance by the subject himself / herself. The information about the subject does not include personal information that identifies the subject, such as name and address.

[0042] [External Event] In this specification, an external event includes an event that provides information, stimuli, etc. to the subject from the outside, such as the surrounding environment in which the subject is placed, the work the subject is doing, etc. Examples of external events include a state in which the subject is moving in a lean vehicle, a state in which the subject is moving on foot, a state in which the subject is looking at pictures of animals, etc.

[0043] [Subject] In this specification, a subject is a person whose biometric information is measured for use as training data for training an emotion estimation model used to estimate emotions. The subject's biometric information in response to a specific external event is measured, and the subject provides emotion information, which is information related to emotions, by self-report. The subject provides emotion information by self-report for each piece of biometric information in response to an external event.

[0044] [Learning] In this specification, learning or machine learning refers to the operation of adjusting parameters of a model (program) in a machine (processor) so that the output for an input is a correct answer. In this embodiment, for the primary model, secondary biometric information distillation model, and tertiary biometric information distillation model to be learned, at least supervised learning is performed out of supervised learning, which learns based on training data that combines input data and a correct answer for the input data, and unsupervised learning, which determines whether or not the answer is correct using only the input data.

[0045] [Biological Information] In this specification, biological information refers to various physiological and anatomical information emitted by a subject or test subject. Biological information refers to all life activities occurring in the subject's or test subject's living body. Biological information includes, for example, information on the subject's heart rate, heart sounds, heart rate waveform, heart rate cycle, heart rate changes, blood pressure, pulse waves, three-axis acceleration, body surface temperature, brain waves, respiratory rate, pupil state, electromyography, blood components, exhaled breath, exhaled breath volume, exhaled breath components, etc. Biological information is measured by wearable sensors worn by the subject, such as a heart rate sensor, acceleration sensor, and temperature sensor.

[0046] [Learning Biological Information] In this specification, the training biological information refers to the biological information of a subject used for training the primary model, the secondary biological information distillation model, and the tertiary biological information distillation model. The training biological information includes, for example, information on the subject's heart rate, heart sounds, heart rate waveform, heart rate cycle, heart rate changes, blood pressure, pulse waves, three-axis acceleration, body surface temperature, brain waves, respiratory rate, pupil state, electromyography, blood components, exhalation, exhaled air volume, exhaled air components, etc. The training biological information is measured by wearable sensors such as a heart rate sensor, an acceleration sensor, and a temperature sensor worn by the subject.

[0047] [Biological information related to heartbeat] In this specification, biological information related to heartbeat means biological information including the subject's heart rate, heartbeat cycle, changes in heart rate, blood pressure, electrocardiogram signal, pulse wave, etc. The biological information related to heartbeat is measured by a heartbeat sensor provided in a wearable sensor or the like, which is a biological information acquisition unit worn by the subject.

[0048] [Emotional Information] In this specification, emotional information refers to information indicating a state of feeling joy, anger, sadness, happiness, etc. The emotional information related to emotions includes a state in which the subject is feeling happy, a state in which the subject is feeling relaxed, a state in which the subject is feeling angry, and a state in which the subject is feeling sad. The emotional information includes information regarding the probability of the subject's emotion being estimated based on the acquired biometric information. The emotional information may include information regarding the degree of the emotion being felt by the subject. One piece of emotional information may include multiple emotions. The emotional information is created, for example, by a control unit provided in a mobile device held by the subject or a server that acquired the biometric information.

[0049] [Degree] In this specification, the term "degree" refers to a graded expression of the strength of an emotion. For example, the degree of comfort can be obtained by dividing comfort into multiple stages, ranging from a standard emotion in which the emotion related to comfort is dominant and most strongly expressed, to a standard emotion in which the emotion related to comfort is dominant and least strongly expressed (not unpleasant but not feeling comfortable), and assigning a strength of the standard emotion related to comfort to each stage.

[0050] [Model] In this specification, a model refers to a specific formula, function, or calculation method that has been learned. A model is a mathematical formula that expresses the relationship between input and output. A model converts input data according to the mathematical formula. Creating a model means learning the model. There are various models, such as a regression model that predicts values ​​and a classification model that classifies data.

[0051] [Neural Network Model] In this specification, a neural network model refers to a mathematical model in which multiple processing units that linearly transform inputs are connected to one another. The neural network model has an input layer, at least one hidden layer, an output layer, a function connecting the input layer and the hidden layer, and a function connecting the hidden layer and the output layer. The input layer, the hidden layer, and the output layer are multiple variables to which data is input. The neural network model sequentially transforms data input to the input layer using the function and outputs the transformed data. When training data consisting of a combination of data and a correct answer for the data is input to the input layer, the parameters of the function in the neural network model are adjusted (trained) so that the data output approaches the correct answer of the training data. By repeatedly training the neural network model, the error between the correct answer and the data output based on the input data can be reduced.

[0052] [Knowledge Information] In this specification, knowledge information refers to information that includes at least a portion of knowledge, which is the output of an intermediate layer of a model, the output of an output layer, information of an intermediate layer, or information of an output layer.

[0053] [Knowledge Distillation] In this specification, knowledge distillation refers to a technique for training a student model, which is another learning model, using knowledge information from a teacher model, which is a learning model that has been trained based on teacher data. For example, knowledge distillation is a learning method for the student model based on teacher data whose number of types is fewer than the number of types of teacher data used in training the teacher model, and the knowledge information created using the teacher model.

[0054] [First Biometric Information] In this specification, the term "first biological information" refers to biological information relating to at least one of the following: biological information that is not easily affected by the subject's surrounding environment, the subject's work, or the subject's movements; biological information that can be easily obtained from the subject; or biological information that is not easily affected by the subject's surrounding environment, the subject's work, or the subject's movements and can be easily obtained from the subject. The first biological information includes, for example, biological information relating to at least one of the subject's electrocardiogram signal, breathing, pulse wave, electroencephalogram, or sweating. The first biological information includes, for example, biological information relating to at least one of the subject's electrocardiogram signal, breathing, or pulse wave. The first biological information includes, for example, a combination of the subject's electrocardiogram signal, breathing, and pulse wave; a combination of the subject's electrocardiogram signal and breathing; a combination of the subject's electrocardiogram signal and pulse wave; a combination of the subject's breathing and pulse wave; or biological information relating to the subject's electrocardiogram signal, breathing, pulse wave, electroencephalogram, or sweating.

[0055] [Second Biometric Information] In this specification, the term "second biometric information" refers to at least one of biometric information that is easily affected by the subject's surrounding environment, the subject's work, or the subject's movements, biometric information that is difficult to easily obtain from the subject, or biometric information that is easily affected by the subject's work or movements and difficult to easily obtain from the subject. The second biometric information includes, for example, biometric information related to at least one of the subject's electrocardiogram signal, breathing, pulse wave, electroencephalogram, or sweating. The second biometric information includes, for example, a combination of the subject's electroencephalogram and sweating, or biometric information related to the subject's electroencephalogram or sweating. For example, because devices for detecting electroencephalograms have a large number of sensors, it is difficult to easily obtain electroencephalograms. For example, sweating may be easily affected by the subject's surrounding environment or may vary greatly from person to person.

[0056] [Relationship between First Biometric Information and Second Biometric Information] In this specification, the first biometric information and the second biometric information may be determined based on the relative relationship between the types of biometric information. For example, the first biometric information may be the subject's pulse wave or respiration, and the second biometric information may be the subject's electrocardiogram or electroencephalogram. For example, the first biometric information may be the subject's electrocardiogram, and the second biometric information may be the subject's electroencephalogram. In this way, for example, the subject's electrocardiogram may be both the first biometric information and the second biometric information.

[0057] [Third Biological Information] In this specification, the third biological information refers to biological information that is thought to be related to the subject's emotions. The third biological information includes, for example, biological information related to a combination of at least two of the five biological information items, i.e., brain waves, sweating, breathing, pulse waves, and electrocardiogram signals, a combination of at least three of the five biological information items, i.e., brain waves, sweating, breathing, pulse waves, and electrocardiogram signals, a combination of at least four of the five biological information items, i.e., brain waves, sweating, breathing, pulse waves, and electrocardiogram signals, or a combination including the five biological information items, i.e., brain waves, sweating, breathing, pulse waves, and electrocardiogram signals.

[0058] [Primary Model] In this specification, the primary model may be a primary biometric information distillation model. In this case, the primary biometric information distillation model is a student model created using knowledge information of a teacher model used to estimate emotions based on multiple types of biometric information and used to estimate emotions based on multiple types of biometric information. For example, if a model used to estimate emotions based on multiple types of biometric information is defined as the teacher model and a model created using knowledge information of the teacher model and used to estimate emotions based on multiple types of biometric information is defined as the student model, the primary model in the present invention may be the teacher model or the student model described above.

[0059] According to one embodiment of the present invention, it is possible to improve the convenience of acquiring biometric information when estimating emotions, while also improving the accuracy of emotion estimation.

[0060] FIG. 1 is a schematic diagram of training a secondary biometric information distillation model or a tertiary biometric information distillation model included in an emotion estimation model in an emotion estimation device according to embodiment 2 of the present invention, and shows the overall configuration of the emotion estimation device having a trained emotion estimation model. FIG. 2 shows the overall configuration of the emotion estimation device according to embodiment 2 of the present invention. FIG. 3 shows the configuration of the emotion estimation model in the emotion estimation device according to embodiment 2 of the present invention. FIG. 4 shows a graph representing classification indices of estimated emotion information in the emotion estimation device according to the present invention. FIG. 5 shows the configuration when training is performed on a primary model used in training the emotion estimation system in the emotion estimation device according to embodiment 2 of the present invention. FIG. 6 shows the configuration after completion of training of the primary model used in training the emotion estimation system in the emotion estimation device according to embodiment 2 of the present invention. FIG. 7 shows the configuration when training is performed on a secondary biometric information distillation model included in the emotion estimation model in the emotion estimation device according to embodiment 2 of the present invention using the output of the primary model. FIG. 8 shows the emotion estimation device according to embodiment 2 of the present invention worn by a subject or test subject. FIG. 9 shows the configuration when training is performed on a secondary biometric information distillation model included in the emotion estimation model in the emotion estimation device according to embodiment 3 of the present invention using the output of the primary model. FIG. 10 shows a configuration when learning of a third-order biometric information distillation model included in a feeling estimation model in a feeling estimation device according to embodiment 2 of the present invention is performed using the output of a second-order biometric information distillation model.

[0061] Each embodiment will be described below with reference to the drawings. In each drawing, the same parts are designated by the same reference numerals, and the description of the same parts will not be repeated. Note that the dimensions of the components in each drawing do not faithfully represent the actual dimensions of the components and the dimensional ratios of the components.

[0062] When estimating emotion from one type of biometric information acquired from a subject, differences in the biometric information due to differences in the subject's emotions are unlikely to be clearly apparent. Therefore, a model that estimates emotion based on only one type of biometric information acquired from the subject not only requires a large amount of training biometric information to improve emotion estimation accuracy, but also makes it difficult to improve the estimation accuracy even if the training biometric information is increased. In other words, in the technical field of emotion estimation based on biometric information, it is generally easy to come up with devices or methods that increase the number of types of biometric information to improve emotion estimation accuracy. However, because reducing the number of types of biometric information directly leads to a decrease in the emotion estimation accuracy of the model, it is difficult to imagine devices or methods that reduce the number of types of biometric information to estimate emotions. Therefore, no devices or methods have been proposed that improve emotion estimation accuracy while reducing the number of types of biometric information.

[0063] On the other hand, since the multiple types of biometric information acquired from the subject are biological reactions occurring within the subject's body, each is considered to be closely related to the subject's emotions. For example, the subject's brain waves, sweating, breathing, electrocardiogram signals, pulse waves, and other biometric information each change in accordance with changes in the subject's emotions. Therefore, in estimating the subject's emotions, the model is considered to be able to utilize the multiple types of biometric information that change in a complementary manner.

[0064] The nth-order biometric information distillation model in this specification can improve estimation accuracy even when convenience is improved by reducing the number of types of biometric information of a subject used to estimate emotions. The nth-order biometric information distillation model is created using knowledge information obtained from a first-order model or an n-1th-order model created by complementary use of multiple types of biometric information. This is thought to be because, when creating the nth-order biometric information distillation model, which estimates emotions based on the subject's biometric information with a small number of types, the knowledge information functions to complement the subject's biometric information with a small number of types. In this way, by focusing on the subject's biometric information and utilizing characteristics of the biometric information that are closely related to the subject's emotions, the inventors have conceived an invention that complementarily uses multiple types of biometric information to reduce the number of types of complemented biometric information while improving emotion estimation accuracy.

[0065] [Embodiment 1] <Emotion Estimation Using Biometric Information> <Overall Configuration of Emotion Estimation Device> A feeling estimation device 1 according to Embodiment 1 of the present invention will be described using Fig. 1. Fig. 1 shows a schematic diagram of learning a secondary biometric information distillation model 31 or a tertiary biometric information distillation model 37 included in an emotion estimation model 30 in the feeling estimation device 1 according to Embodiment 1 of the present invention, and the overall configuration of the feeling estimation device 1 having the trained emotion estimation model 30.

[0066] 1 , a feeling estimation device 1 according to a first embodiment of the present invention estimates the feeling of a subject U based on biometric information Bi of the subject U (hereinafter simply referred to as “biometric information Bi”) or learning biometric information BiL (hereinafter simply referred to as “learning biometric information BiL”), which is biometric information of a subject T. In the present embodiment, the feeling estimation device 1 creates secondary estimated emotion information Ei2, which is information related to the emotion of the subject U, from at least one type of biometric information Bi of the subject U.

[0067] In the following embodiments, a model used to estimate emotions based on multiple different types of biometric information Bi is defined as a primary model 100. The primary model 100 outputs primary knowledge information Ki1 based on multiple types of biometric information Bi. Models created using primary knowledge information obtained based on the primary model 100 are defined as secondary biometric information distillation models 31 and 34. A model created using secondary knowledge information Ki2 obtained based on the secondary biometric information distillation models 31 and 34 is defined as a tertiary biometric information distillation model 37. In this way, a model created using n-1th-order knowledge information obtained based on an n-1th-order biometric information distillation model is defined as an nth-order biometric information distillation model (n is an integer greater than or equal to 2). n is a number representing the generation of the biometric information distillation model, counting from the primary model as the first generation.

[0068] The feeling estimation device 1 includes a control unit 20 and a storage unit 40. The control unit 20 controls the feeling estimation device 1. The control unit 20 also creates secondary estimated emotion information Ei2 or tertiary estimated emotion information Ei3 based on externally input biometric information Bi or learning biometric information BiL.

[0069] The control unit 20 has an emotion estimation model 30 that is used to estimate the emotion of the subject U based on the biometric information Bi. The control unit 20 is also configured to be able to input the biometric information Bi or learning biometric information BiL from the outside.

[0070] The control unit 20 stores various programs and data for controlling the emotion estimation device 1 including the emotion estimation model 30 and the storage unit 40. The control unit 20 is configured to be able to transmit biometric information Bi or learning biometric information BiL input from outside to the storage unit 40. The control unit 20 is also configured to be able to acquire various information including the biometric information Bi or learning biometric information BiL and information related to emotions stored in the storage unit 40.

[0071] The emotion estimation model 30 is used to estimate the emotion of a subject U based on biometric information Bi. The emotion estimation model 30 is a machine learning model trained based on training data that combines learning biometric information BiL and self-reported emotions of the subject T. The emotion estimation model 30 also includes a secondary biometric information distillation model 31 created using primary knowledge information Ki1 obtained based on the primary model 100, or a tertiary biometric information distillation model 37 created using secondary knowledge information Ki2 obtained based on the secondary biometric information distillation model 31.

[0072] The secondary biometric information distillation model 31 is used to create secondary estimated emotion information Ei2, which is information indicating the emotion of the subject U, based on the biometric information Bi. The secondary biometric information distillation model 31 is configured to be able to input the biometric information Bi. The secondary biometric information distillation model 31 is configured to be able to output the secondary estimated emotion information Ei2.

[0073] The tertiary biometric information distillation model 37 is used to create tertiary inferred emotion information Ei3, which is information indicating the emotion of the subject U, based on the biometric information Bi. The tertiary biometric information distillation model 37 is configured to be able to input the biometric information Bi. The tertiary biometric information distillation model 37 is configured to be able to output the tertiary inferred emotion information Ei3.

[0074] The control unit 20 is configured to be able to output secondary estimated emotion information Ei2 created using the secondary biometric information distillation model 31 of the emotion estimation model 30, or tertiary estimated emotion information Ei3 created using the tertiary biometric information distillation model 37 of the emotion estimation model 30. The control unit 20 is configured to be able to output the secondary estimated emotion information Ei2 or tertiary estimated emotion information Ei3 output by the emotion estimation model 30 to the storage unit 40 and to the outside.

[0075] The storage unit 40 stores the biometric information Bi, the learning biometric information BiL, the secondary estimated emotion information Ei2 and the tertiary estimated emotion information Ei3, etc. that the control unit 20 has created using the emotion estimation model 30. The storage unit 40 has a RAM for storing the biometric information Bi, the learning biometric information BiL, the secondary estimated emotion information Ei2 and the tertiary estimated emotion information Ei3, etc., a memory within the processor, a hard disk, etc. The storage unit 40 is configured to be able to output the stored biometric information Bi, the learning biometric information BiL, the secondary estimated emotion information Ei2 and the tertiary estimated emotion information Ei3 to the control unit 20.

[0076] The feeling estimation device 1 configured in this manner causes the control unit 20 to store externally input biometric information Bi in the storage unit 40. The control unit 20 also generates secondary estimated emotion information Ei2 or tertiary estimated emotion information Ei3 of the subject U based on the biometric information Bi using the emotion estimation model 30. The control unit 20 outputs the generated secondary estimated emotion information Ei2 to the storage unit 40 or to the outside every unit time.

[0077] <Creating Primary Model 100> Next, we will explain the primary model 100 required to create the emotion estimation model 30. The trained primary model 100 is a model that provides primary knowledge information Ki1 created based on multiple different types of biometric information Bi to the emotion estimation model 30. In this embodiment, the primary model is a teacher model for the secondary biometric information distillation model 31.

[0078] The primary model 100 is a machine learning model used to estimate emotions based on a plurality of different types of training biometric information BiL. The primary model 100 is a model trained based on training data consisting of a combination of training emotion information EiL, which is information about emotions self-reported by subject T, and training biometric information BiL, which includes a plurality of different types of biometric information. The training emotion information EiL is the correct emotion in the training data.

[0079] Primary model 100 is used to create primary estimated emotion information Ei1 based on learning emotion information EiL. Primary model 100 is configured to be able to output primary estimated emotion information Ei1 when learning emotion information EiL is input. The parameters of trained primary model 100 have been adjusted so that the error between primary estimated emotion information Ei1 and learning emotion information EiL is as small as possible, in response to the input learning emotion information EiL.

[0080] <Creation of Secondary Biometric Information Distillation Model 31> Next, the creation of the emotion estimation model 30 including the secondary biometric information distillation model 31 will be described. The emotion estimation model 30 is created by learning the training biometric information BiL and primary knowledge information Ki1 obtained based on the training biometric information BiL including a plurality of different types of biometric information using a trained primary model 100. In this embodiment, the emotion estimation model 30 uses, as the primary knowledge information Ki1, primary estimated emotion information Ei1 created based on the training biometric information BiL using the trained primary model 100. In this embodiment, the secondary biometric information distillation model 31 is a student model for the primary model 100.

[0081] The secondary biometric information distillation model 31 is configured to be able to output secondary estimated emotion information Ei2 when learning emotion information EiL is input. The secondary biometric information distillation model 31 is configured to be able to adjust the parameters of the secondary biometric information distillation model 31 so that the sum of the error of the secondary estimated emotion information Ei2 from the learning emotion information EiL contained in the teacher data and the error of the secondary estimated emotion information Ei2 from the primary estimated emotion information Ei1 which is the primary knowledge information Ki1 is minimized. In other words, the secondary biometric information distillation model 31 is trained using primary estimated emotion information Ei1 obtained based on the learning biometric information BiL using the trained primary model 100, secondary estimated emotion information Ei2 obtained based on the learning biometric information BiL using the secondary biometric information distillation model 31, and the learning emotion information EiL of the teacher data.

[0082] The secondary biometric information distillation model 31 is used to estimate the emotions of the subject U based on the number of types of learning biometric information BiL that are included in the types of learning biometric information BiL used when obtaining the primary knowledge information Ki1 (when estimating emotions) using the primary model 100, and that are fewer than the number of types of learning biometric information BiL required to estimate emotions in the primary model 100.

[0083] The primary knowledge information Ki1 is an output of the primary model 100 obtained based on multiple types of learning biometric information BiL using the primary model 100. In other words, the primary knowledge information Ki1 is information (knowledge of the primary model 100) for estimating the emotions of the subject U obtained by the primary model 100 through learning based on multiple types of learning biometric information BiL.

[0084] By utilizing the primary knowledge information Ki1, the secondary biometric information distillation model 31 is trained while inheriting the primary estimated emotion information Ei1, which includes information (privileged information) that is not included in the training biometric information BiL used for training the secondary biometric information distillation model 31 (knowledge distillation).

[0085] Therefore, the secondary biometric information distillation model 31 trained using the primary knowledge information Ki1 in addition to the learning biometric information BiL can achieve higher emotion estimation accuracy than when training is performed using only the learning biometric information BiL. Furthermore, even when training is performed based on fewer types of biometric information than the number of types of biometric information used by the primary model 100 to obtain the primary knowledge information Ki1, the secondary biometric information distillation model 31 can achieve emotion estimation accuracy close to that of the primary model 100.

[0086] Furthermore, when the secondary biometric information distillation model is trained using the primary knowledge information Ki1, the feeling estimation device 1 can reduce the size of the model and the amount of biometric information required for emotion estimation compared to when the secondary biometric information distillation model is trained without using the primary knowledge information. Thus, the feeling estimation device 1 can estimate the emotion of the subject U with estimation accuracy close to the emotion estimation accuracy of the primary model 100 by using the primary knowledge information Ki1, while suppressing an increase in the processing load on the hardware.

[0087] <Creation of Tertiary Biometric Information Distillation Model> Next, a description will be given of the creation of the emotion estimation model 30 including the tertiary biometric information distillation model 37. The emotion estimation model 30 is created using the training biometric information BiL and secondary knowledge information Ki2 obtained based on multiple types of training biometric information BiL using a trained secondary biometric information distillation model 31. In this embodiment, the emotion estimation model 30 uses, as secondary knowledge information Ki2, secondary estimated emotion information Ei2 created based on the training biometric information BiL using the trained secondary biometric information distillation model 31.

[0088] The tertiary biometric information distillation model 37 is configured to be able to output tertiary estimated emotion information Ei3 when learning emotion information EiL is input. The tertiary biometric information distillation model 37 is configured to be able to adjust the parameters of the tertiary biometric information distillation model 37 so that the sum of the error of the tertiary estimated emotion information Ei3 from the learning emotion information EiL contained in the teacher data and the error of the tertiary estimated emotion information Ei3 from the secondary estimated emotion information Ei2 which is secondary knowledge information Ki2 is minimized. In other words, the tertiary biometric information distillation model 37 is trained using secondary estimated emotion information Ei2 obtained based on the learning biometric information BiL using the trained secondary biometric information distillation model 31, third estimated emotion information Ei3 obtained based on the learning biometric information BiL using the tertiary biometric information distillation model 37, and the learning emotion information EiL of the teacher data.

[0089] The tertiary biometric information distillation model 37 is used to estimate the emotions of the subject U based on the number of types of learning biometric information BiL that are included in the types of learning biometric information BiL used when obtaining the secondary knowledge information Ki2 (when estimating emotions) using the secondary biometric information distillation model 31, and that are fewer than the types of learning biometric information BiL required to estimate emotions in the secondary biometric information distillation model 31.

[0090] The control unit 20 adjusts the parameters of the tertiary biometric information distillation model 37 so as to minimize the sum of the error of the tertiary estimated emotion information Ei3 relative to the learning emotion information EiL contained in the teacher data input to the convolution layer 38, and the error of the tertiary estimated emotion information Ei3 relative to the secondary estimated emotion information Ei2, which is the secondary knowledge information Ki2 of the secondary biometric information distillation model 34.

[0091] The tertiary biometric information distillation model 37 configured in this manner is trained using secondary inferred emotion information Ei2 obtained based on the training biometric information BiL, which is training data, using the trained secondary biometric information distillation model 31, tertiary inferred emotion information Ei3 obtained based on the training biometric information BiL using the tertiary biometric information distillation model 37, and the training emotion information EiL of the training data.

[0092] The secondary knowledge information Ki2 is an output of the secondary biometric information distillation model 31 obtained based on at least one type of learning biometric information BiL using the secondary biometric information distillation model 31. In other words, the secondary knowledge information Ki2 is information (knowledge of the tertiary biometric information distillation model 37) for estimating the emotion of the subject U obtained by the secondary biometric information distillation model 31 through learning based on multiple different types of learning biometric information BiL.

[0093] By utilizing the secondary knowledge information Ki2, the tertiary biometric information distillation model 37 is trained while inheriting the secondary estimated emotion information Ei2, which includes information (privileged information) that is not included in the training biometric information BiL used for training the tertiary biometric information distillation model 37 (knowledge distillation).

[0094] Therefore, the tertiary biometric information distillation model 37 trained using the secondary knowledge information Ki2 in addition to the learning biometric information BiL can achieve higher emotion estimation accuracy than when training is performed using only the learning biometric information BiL. Furthermore, even when training is performed based on biometric information whose number of types is less than or equal to the number of types of biometric information used by the secondary biometric information distillation model 31 to obtain the secondary knowledge information Ki2, the tertiary biometric information distillation model 37 can achieve estimation accuracy close to that of the secondary biometric information distillation model 31.

[0095] The feeling estimation device 1 configured in this manner can estimate the feeling of the subject U with estimation accuracy close to that of the primary model 100 by utilizing the primary knowledge information Ki1 while suppressing an increase in the processing load on the hardware. For example, the feeling estimation device 1 can easily acquire, using a wearable sensor or the like, biometric information Bi of a subject U performing a specific task, movement, or the like, such as a subject U driving a four-wheeled motor vehicle, a two-wheeled motor vehicle, a ship, an airplane, various types of work machinery, a drone, or the like, a subject U walking, a subject U exercising such as running, a subject U watching a movie, a painting, content, or the like, and estimate the feeling of the subject U performing a specific task, movement, or the like.

[0096] [Embodiment 2] <Emotion Estimation Using Electrocardiogram Signal and Pulse Wave> <Overall Configuration of the Emotion Estimation Device> A feeling estimation device 1 according to Embodiment 2 of the present invention will be described using FIGS. 2 to 4 and 8. FIG. 2 shows the overall configuration of the feeling estimation device 1 according to Embodiment 2 of the present invention. FIG. 3 shows the configuration of the emotion estimation model 30 in the feeling estimation device 1. FIG. 4 shows a graph representing classification indices of secondary estimated emotion information Ei2 in the feeling estimation device 1. FIG. 8 shows a state in which the subject U or subject T wears the feeling estimation device 1. The feeling estimation device 1 according to Embodiment 2 is a feeling estimation device that discloses the configuration of the feeling estimation device 1 according to Embodiment 1 in more detail. Note that in the following embodiments, specific descriptions of points similar to those of the embodiments already described will be omitted, and the description will focus on differences.

[0097] 2 , the feeling estimation device 1 according to the second embodiment of the present invention estimates the feeling of a subject U based on the biometric information Bi of the subject U. In the present embodiment, the feeling estimation device 1 creates estimated emotion information, which is information on the emotion of the subject U, from the biometric information Bi related to the electrocardiogram signal and pulse wave of the subject U.

[0098] The feeling estimation device 1 includes a biometric information acquisition unit 10, a control unit 20, a storage unit 40, and an output unit 50. The storage unit 40, the output unit 50, and the control unit 20 of the feeling estimation device 1 are built into a mobile terminal Pt (see FIG. 8 ) such as a smartphone carried by a subject U.

[0099] That is, the feeling estimation device 1 shares part of the configuration of the mobile device Pt. Specifically, the memory included in the mobile device Pt functions as the storage unit 40 of the feeling estimation device 1. The liquid crystal monitor included in the mobile device Pt functions as the output unit 50 of the feeling estimation device 1. The processor included in the mobile device Pt functions as the control unit 20 of the feeling estimation device 1.

[0100] The biometric information acquiring unit 10 acquires (measures) biometric information Bi or learning biometric information BiL. The biometric information acquiring unit 10 also outputs the acquired biometric information Bi or learning biometric information BiL to the control unit 20. The biometric information acquiring unit 10 has a sensor 11 such as a heart rate sensor (not shown) that acquires the biometric information Bi and learning biometric information BiL, and a sensor communicator 12 that transmits and receives a control signal, the acquired biometric information Bi, the learning biometric information BiL, etc.

[0101] The biometric information acquisition unit 10 uses the sensor 11 to acquire, for example, at least one of biometric information Bi or learning biometric information BiL, such as electrocardiogram signal, heart rate, heart sound, heart rate waveform, heart rate cycle, heart rate change, blood pressure, pulse wave, three-axis acceleration, body surface temperature, brain wave, respiratory rate, pupil state, electromyography, blood components, exhalation, exhaled air volume, exhaled air components, etc.

[0102] In this embodiment, the biometric information acquiring unit 10 acquires biometric information Bi1 (hereinafter simply referred to as "electrocardiogram signal Bi1") and biometric information Bi2 (hereinafter simply referred to as "pulse wave Bi2") related to an electrocardiogram signal, which are biometric information related to heartbeats, from the biometric information Bi or the learning biometric information BiL. The biometric information acquiring unit 10 acquires the electrocardiogram signal Bi1 and the pulse wave Bi2 of the subject U or the subject T every unit time using, for example, a wearable sensor, which is a sensor 11 that can be easily worn by the subject U or the subject T. In this way, the biometric information acquiring unit 10 also outputs the biometric information Bi, including the electrocardiogram signal Bi1 and the pulse wave Bi2 of the subject U or the subject T, acquired by the sensor 11 using the sensor communicator 12, to the control unit 20.

[0103] The control unit 20 controls the biometric information acquisition unit 10, the storage unit 40, and the output unit 50. The control unit 20 also creates estimated emotion information based on the biometric information Bi of the subject U input from the biometric information acquisition unit 10.

[0104] The control unit 20 has an emotion estimation model 30 that is used to estimate the emotion of the subject U based on the electrocardiogram signal Bi1 and pulse wave Bi2 of the subject U or subject T. The control unit 20 also has a control unit communication device 21 that transmits and receives control signals and the like to the biological information acquisition unit 10 and acquires the electrocardiogram signal Bi1 and pulse wave Bi2 of the subject U or subject T.

[0105] The control unit 20 stores various programs and data for controlling the biometric information acquisition unit 10, the storage unit 40, the output unit 50, and the emotion estimation model 30. The control unit 20 is configured to be able to transmit and receive control signals to and from the biometric information acquisition unit 10 via a control unit communication device 21. Similarly, the control unit 20 is configured to be able to transmit and receive control signals to and from the storage unit 40 and the output unit 50.

[0106] The control unit 20 is configured to be able to transmit the biometric information Bi or the learning biometric information BiL acquired by the biometric information acquisition unit 10 to the storage unit 40. The control unit 20 is also configured to be able to acquire various information including the biometric information Bi and emotion information of the subject U stored in the storage unit 40. The control unit 20 is also configured to be able to output various information including the biometric information Bi and emotion information of the subject U to the output unit 50.

[0107] The emotion estimation model 30 estimates the emotion of the subject U based on the electrocardiogram signal Bi1 and pulse wave Bi2 of the subject U. The emotion estimation model 30 is a neural network model trained based on training data that combines the electrocardiogram signal Bi1 and pulse wave Bi2 (see FIG. 3 ), which are learning biometric information BiL, with the subject T's self-reported emotion. The emotion estimation model 30 also includes a secondary biometric information distillation model 31 (see FIG. 3 ) that has trained on primary knowledge information Ki1 (see FIG. 7 ) created using the primary model 100.

[0108] As shown in FIG. 3 , the emotion estimation model 30 estimates the emotion of the subject U based on the electrocardiogram signal Bi1 and pulse wave Bi2 of the subject U using a secondary biological information distillation model 31, and outputs the estimated emotion as secondary estimated emotion information Ei2 indicated by an arousal value Ar (Arousal) representing the intensity of the emotion and a valence value Va (Valence) representing the degree of the emotion's aggressiveness.

[0109] The secondary biometric information distillation model 31 included in the emotion estimation model 30 is a convolutional neural network model (CNN) that extracts feature quantities from image data of the electrocardiogram signal Bi1 and the pulse wave Bi2 and classifies the image data based on the feature quantities. The secondary biometric information distillation model 31 includes a convolutional layer 32, a fully connected layer 33, etc.

[0110] The convolutional layer 32 is used to extract feature quantities from the image data of the electrocardiogram signal Bi1 and the pulse wave Bi2. The convolutional layer 32 is configured to be able to input the electrocardiogram signal Bi1 and the pulse wave Bi2 of the subject U acquired by the sensor 11.

[0111] The secondary biological information distillation model 31 is configured to be able to calculate, when the electrocardiogram signal Bi1 and pulse wave Bi2 of the subject U are input to the convolutional layer 32, a multidimensional vector V which is a group of multiple feature quantities, such as time domain feature quantities such as the R-R interval, the average value of the R-R interval, and the standard deviation of the R-R interval, which are feature quantities of the electrocardiogram signal Bi1 and pulse wave Bi2 of the subject U, frequency domain feature quantities such as the power spectrum of the low frequency band (LF), the power spectrum of the high frequency band (HF), and the LF / HF ratio, and nonlinear domain feature quantities such as the sample entropy and the Lyapunov exponent, using the convolutional layer 32. The convolutional layer 32 outputs the calculated multidimensional vector V of the electrocardiogram signal Bi1 and pulse wave Bi2 to the fully connected layer 33.

[0112] The fully connected layer 33 is used to estimate the emotion of the subject U based on the multidimensional vector V. The fully connected layer 33 is used to create secondary estimated emotion information Ei2, which is information related to emotion, based on the input multidimensional vector V.

[0113] The secondary estimated emotion information Ei2 includes the arousal value Ar and valence value Va of the estimated emotion. The control unit 20 is configured to be able to output the secondary estimated emotion information Ei2 created using the secondary biological information distillation model 31. The control unit 20 is configured to be able to output the secondary estimated emotion information Ei2 output by the emotion estimation model 30 to the storage unit 40 and the output unit 50 (see FIG. 2 ).

[0114] 2, storage unit 40 stores biometric information Bi acquired by sensor 11 included in biometric information acquisition unit 10, secondary estimated emotion information Ei2 created by control unit 20 using emotion estimation model 30, etc. Storage unit 40 has RAM for storing biometric information Bi and secondary estimated emotion information Ei2, memory within the processor, a hard disk, etc.

[0115] In this embodiment, the storage unit 40 is a memory of the mobile terminal Pt. The storage unit 40 is configured to be able to acquire control signals and the like from the control unit 20. The storage unit 40 is also configured to be able to store, in chronological order, the biometric information Bi input from the biometric information acquisition unit 10 via the control unit 20, the secondary estimated emotion information Ei2 input from the control unit 20, and information about the subject U. The storage unit 40 is configured to be able to output the stored biometric information Bi and secondary estimated emotion information Ei2 to the control unit 20.

[0116] The output unit 50 outputs at least one of the biometric information Bi or the secondary estimated emotion information Ei2 to the subject U. The output unit 50 includes a display device, a speaker, and the like. In this embodiment, the output unit 50 is a liquid crystal monitor of the mobile terminal Pt (see FIG. 8 ).

[0117] The output unit 50 is configured to be able to acquire control signals and the like from the control unit 20. The output unit 50 is configured to be able to acquire biometric information Bi and secondary estimated emotion information Ei2 from the control unit 20. The output unit 50 is configured to be able to output the acquired biometric information Bi and secondary estimated emotion information Ei2 to a liquid crystal monitor or the like of the mobile terminal Pt as at least one of graphic information, character information, and color information.

[0118] The feeling estimation device 1 configured as described above includes a biometric information acquisition unit 10 including a sensor 11, a control unit 20 configured by the mobile device Pt of the subject U, a storage unit 40, and an output unit 50. The control unit 20 of the feeling estimation device 1 acquires biometric information Bi every unit time using the biometric information acquisition unit 10. The control unit 20 stores the acquired biometric information Bi in the storage unit 40. The control unit 20 also creates secondary estimated emotion information Ei2 of the subject U every unit time based on the biometric information Bi acquired using the emotion estimation model 30. At this time, the control unit 20 may use information about the subject U acquired in advance to estimate the emotion. The control unit 20 outputs the created secondary estimated emotion information Ei2 to the storage unit 40 and the output unit 50 every unit time.

[0119] As shown in Fig. 4, the control unit 20 displays the emotion of the subject U on the output unit 50 using an emotion circle diagram based on the secondary estimated emotion information Ei2 (see Fig. 3). The emotion circle diagram classifies emotions according to the emotion arousal value Ar and the valence value Va. The control unit 20 converts the emotion of the subject U classified based on the emotion arousal value Ar and the valence value Va using the emotion circle diagram into at least one of graphic information, text information, and color information, and outputs it to the output unit 50.

[0120] The valence value Va represents the degree of positivity of an emotion between positive and negative emotions. Positive emotions refer to positive mental states such as excitement, relief, elation, comfort, calmness, and enjoyment. Negative emotions refer to negative mental states such as tension, agitation, depression, anxiety, discomfort, and despair.

[0121] When the valence value Va is on the positive side, the control unit 20 determines that the subject U is feeling a positive emotion corresponding to the degree of the valence value Va. When the valence value Va is on the negative side, the control unit 20 determines that the subject U is feeling a negative emotion corresponding to the degree of the valence value Va.

[0122] The arousal value Ar represents the degree of emotion between an activated emotional state and a deactivated emotional state. An activated emotional state refers to an active mental state such as excitement, elation, joy, agitation, tension, etc. A calmed emotional state refers to a passive mental state such as relief, tranquility, calmness, depression, boredom, etc.

[0123] When the arousal value Ar is on the positive side, the control unit 20 determines that the subject U is experiencing an activated emotion corresponding to the level of the arousal value Ar. When the arousal value Ar is on the negative side, the control unit 20 determines that the subject U is experiencing a deactivated emotion corresponding to the level of the arousal value Ar.

[0124] When the valence value Va is on the positive side and the arousal value Ar is on the positive side, the control unit 20 assigns the subject U's emotion to an emotion included in the happy domain, which includes emotions such as excitement, elation, and pleasure. For example, when the valence value Va is high and the arousal value Ar is low within the happy domain, the control unit 20 selects the emotion of pleasant from the multiple types of emotions in the happy domain.

[0125] When the valence value Va is on the positive side and the arousal value Ar is on the negative side, the control unit 20 assigns the subject U's emotion to an emotion included in the Relaxed area, which includes relief, serenity, calmness, etc. For example, when the valence value Va is high and the arousal value Ar is low within the Relaxed area, the control unit 20 selects the emotion of Serenity from the multiple types of emotions in the Relaxed area.

[0126] When the valence value Va is on the negative side and the arousal value Ar is on the positive side, the control unit 20 assigns the emotion of the subject U to an emotion included in the anger domain, which includes tension, agitation, displeasure, etc. For example, when the valence value Va and the arousal value Ar are low within the anger domain, the control unit 20 selects the emotion of displeasure from the multiple types of emotions in the anger domain.

[0127] When the valence value Va is on the negative side and the arousal value Ar is on the negative side, the control unit 20 assigns the emotion of the subject U to an emotion included in the sadness (Sad) domain, which includes melancholy, anxiety, despair, etc. For example, when the valence value Va and the arousal value Ar are low within the sadness domain, the control unit 20 selects the emotion of despair from the multiple types of emotions in the sadness domain.

[0128] <Creation of Primary Model> Next, the primary model 100 necessary for creating the emotion estimation model 30 will be described using Fig. 5 and Fig. 6. Fig. 5 shows the configuration when learning of the primary model 100 used to learn the emotion estimation model 30 (see Fig. 2) in the emotion estimation device 1 is performed. Fig. 6 shows the configuration after learning of the primary model 100 used to learn the emotion estimation model 30 in the emotion estimation device 1. The trained primary model 100 is a teacher model that provides the emotion estimation model 30 with primary knowledge information Ki1 created based on multiple different types of biometric information Bi.

[0129] As shown in Figure 5, primary model 100 is a neural network model used to estimate emotions based on multiple different types of training biometric information BiL. Primary model 100 is a model trained based on training data consisting of a combination of training emotion information EiL, which is information related to emotions self-reported by subject T, and multiple different types of training biometric information BiL of subject T, including electrocardiogram signals Bi1, pulse waves Bi2, electroencephalogram-related biometric information Bi3 (hereinafter simply referred to as "EEG Bi3"), sweating-related biometric information Bi4 (hereinafter simply referred to as "sweating Bi4"), and respiration-related biometric information Bi5 (hereinafter simply referred to as "respiration Bi5"). Training emotion information EiL is the correct emotion in the training data. Training emotion information EiL includes an emotional arousal value Ar and an emotional valence value Va.

[0130] In this embodiment, the first biometric information Bif is defined as biometric information Bi that is not easily affected by the environment around the subject U, the work of the subject U, or the subject's movements, etc., biometric information Bi that can be easily obtained from the subject U using a wearable sensor, etc., or at least one of an electrocardiogram signal Bi1, a pulse wave Bi2, or respiration Bi5, which is biometric information Bi that is not easily affected by the environment around the subject U, the work of the subject U, or the subject's movements, etc. and can be easily obtained from the subject U using a wearable sensor, etc.

[0131] In addition, in this embodiment, the second biometric information Bis is defined as at least one of biometric information Bi that is easily influenced by the environment around the subject U, the subject U's work, or the subject U's movements, biometric information Bi that is difficult to easily obtain from the subject U, or brain waves Bi3 or sweating Bi4 that is biometric information Bi that is easily influenced by the subject U's work or the subject U's movements and difficult to easily obtain from the subject U.

[0132] In this embodiment, the third biological information is defined as at least two of the biological information considered to be related to the subject's emotions, such as electroencephalogram Bi3, sweating Bi4, respiration Bi5, pulse wave Bi2, or electrocardiogram Bi1, or at least one of the biological information considered to be related to the subject's emotions, such as electrocardiogram Bi1, pulse wave Bi2, or respiration Bi5, which is a combination of three of the biological information considered to be related to the subject's emotions, such as electroencephalogram Bi3, sweating Bi4, respiration Bi5, pulse wave Bi2, or electrocardiogram Bi1. In this embodiment, the first biological information Bif and the third biological information Bit include the same type of biological information Bi.

[0133] The primary model 100 is trained based on the multidimensional vector V of feature quantities possessed by the electrocardiogram signal Bi1, pulse wave Bi2, and respiration Bi5, which are the first biological information Bif, and the electroencephalogram Bi3 and sweating Bi4, which are the second biological information Bis, and the training emotion information EiL. In other words, the primary model 100 is trained based on the multidimensional vector V of feature quantities possessed by the electrocardiogram signal Bi1, pulse wave Bi2, and respiration Bi5, and the electroencephalogram Bi3 and sweating Bi4, which are multiple types of training biological information BiL including the electrocardiogram signal Bi1, pulse wave Bi2, and respiration Bi5, which are the third biological information Bit.

[0134] The features of the electrocardiogram signal Bi1 and the pulse wave Bi2 include the mean value and standard deviation of the RR interval, the power spectrum of the low- and high-frequency bands, the power spectrum ratio, the sample entropy, and the Lyapunov exponent. The features of the electroencephalogram Bi3 include the root mean square of theta, alpha, beta, and gamma waves, the Hjorth parameter representing movement speed and complexity, the mean value and standard deviation of the power spectrum and phase synchrony, and the like. The features of sweating Bi4 include the amplitude and peak time of the skin conductance response, the mean value, standard deviation, root mean square, and power spectrum of the isotonic level and phasic response of the electrodermal response, the mean value and standard deviation of the Mel-frequency cepstrum coefficient, and the like. The features of respiration Bi5 include the mean value, standard deviation, maximum value, minimum value, the power spectrum of the low- and high-frequency bands, the power spectrum ratio, the sample entropy, and the Lyapunov exponent at the zero-crossing time.

[0135] The primary model 100 has an autoencoder 110, which is a neural network model, and a first head unit 121, a second head unit 122, and a third head unit 123, which are fully connected layers of the neural network model.

[0136] The autoencoder 110 is a model that extracts features of a multidimensional vector V by reducing the dimension (number of variables) of the multidimensional vector V including multiple variables and then restoring it to its original dimension. The autoencoder 110 has multiple encoder units and multiple decoder units. In this embodiment, the autoencoder 110 includes a combination of a first encoder unit 111 and a first decoder unit 114, a combination of a second encoder unit 112 and a second decoder unit 115, and a combination of a third encoder unit 113 and a third decoder unit 116 to perform ensemble learning. The primary model 100 is configured to allow a multidimensional vector V created based on training biometric information BiL to be input to each of the first encoder unit 111, the second encoder unit 112, and the third encoder unit 113 of the autoencoder 110.

[0137] The first encoder unit 111, the second encoder unit 112, and the third encoder unit 113 are dimension compressors that use a function to reduce the dimension of a multidimensional vector V that includes multiple variables. In other words, the first encoder unit 111, the second encoder unit 112, and the third encoder unit 113 use a function to compress the dimension of the multidimensional vector V to a lower dimension. In this way, the first encoder unit 111, the second encoder unit 112, and the third encoder unit 113 are configured to be usable for creating a first compressed multidimensional vector Vca, a second compressed multidimensional vector Vcb, and a third compressed multidimensional vector Vcc that extract important characteristic portions from the multidimensional vector V.

[0138] The first encoder unit 111 is configured to be able to output the first compressed multidimensional vector Vca to the first decoder unit 114 and the first head unit 121. The second encoder unit 112 is configured to be able to output the second compressed multidimensional vector Vcb to the second decoder unit 115 and the second head unit 122. The third encoder unit 113 is configured to be able to output the third compressed multidimensional vector Vcc to the third decoder unit 116 and the third head unit 123.

[0139] The first decoder unit 114, the second decoder unit 115, and the third decoder unit 116 are dimension restorers that restore the dimensions of the first compressed multidimensional vector Vca, the second compressed multidimensional vector Vcb, and the third compressed multidimensional vector Vcc, which include multiple variables whose dimensions have been reduced using a function, to the dimensions before being reduced.

[0140] The first decoder unit 114 is configured to be available for generating a first restored multidimensional vector Vra by restoring the dimensions of the first compressed multidimensional vector Vca compressed by the first encoder unit 111 to the dimensions before compression. The second decoder unit 115 is configured to be available for generating a second restored multidimensional vector Vrb by restoring the dimensions of the second compressed multidimensional vector Vcb compressed by the second encoder unit 112 to the dimensions before compression. The third decoder unit 116 is configured to be available for generating a third restored multidimensional vector Vrc by restoring the dimensions of the third compressed multidimensional vector Vcc compressed by the third encoder unit 113 to the dimensions before compression.

[0141] The first head unit 121, the second head unit 122, and the third head unit 123 are fully connected layers used to estimate the emotion of the subject T based on all of the variables input to each unit. The first head unit 121 is used to estimate the emotion of the subject T based on all of the first compressed multidimensional vector Vca input from the first encoder unit 111. In other words, the first head unit 121 is configured to be usable for creating first estimated emotion information Eia that includes an emotional arousal value Ar and a valence value Va, based on the first compressed multidimensional vector Vca.

[0142] Similarly, second head unit 122 is configured to be usable for creating second estimated emotion information Eib based on second compressed multidimensional vector Vcb. Third head unit 123 is configured to be usable for creating third estimated emotion information Eic based on third compressed multidimensional vector Vcc.

[0143] Furthermore, the primary model 100 is configured so that it can be used to create primary estimated emotion information Ei1 based on the first estimated emotion information Eia created by the first head unit 121, the second estimated emotion information Eib created by the second head unit 122, and the third estimated emotion information Eic created by the third head unit 123. The primary estimated emotion information Ei1 includes an emotional arousal value Ar and a valence value Va. In the present embodiment, the primary estimated emotion information Ei1 is the average of the first estimated emotion information Eia, the second estimated emotion information Eib, and the third estimated emotion information Eic.

[0144] In the primary model 100, parameters of functions in the first encoder unit 111 and the first decoder unit 114 are adjusted so as to reduce the error between the multidimensional vector V input to the first encoder unit 111 and the first restored multidimensional vector Vra created using the first decoder unit 114. In the primary model 100, parameters of functions in the second encoder unit 112 and the second decoder unit 115 are adjusted so as to reduce the error between the multidimensional vector V input to the second encoder unit 112 and the second restored multidimensional vector Vrb created using the second decoder unit 115. In the primary model 100, parameters of functions in the third encoder unit 113 and the third decoder unit 116 are adjusted so as to reduce the error between the multidimensional vector V input to the third encoder unit 113 and the third restored multidimensional vector Vrc created using the third decoder unit 116.

[0145] Furthermore, in primary model 100, the function parameters in first head unit 121, second head unit 122 and third head unit 123 are adjusted so as to reduce the error between learning emotion information EiL included in the teacher data and primary estimated emotion information Ei1.

[0146] The primary model 100 configured in this manner is trained using multiple combinations of the multidimensional vectors V and training emotion information EiL contained in the training data, which are the electrocardiogram signal Bi1, pulse wave Bi2, electroencephalogram Bi3, sweating Bi4, and respiration Bi5.

[0147] In the primary model 100, a multidimensional vector V of a plurality of different types of learning biometric information BiL of subject T is input to an autoencoder 110. In the primary model 100, a first restored multidimensional vector Vra, a second restored multidimensional vector Vrb, and a third restored multidimensional vector Vrc are created for the multidimensional vector V using the autoencoder 110. In addition, in the primary model 100, primary estimated emotion information Ei1 is created using a first encoder unit 111 and a first head unit 121, a second encoder unit 112 and a second head unit 122, and a third encoder unit 113 and a third head unit 123.

[0148] In primary model 100, the parameters of autoencoder 110 and the parameters of first head unit 121, second head unit 122 and third head unit 123 are adjusted (learning is performed) so that the sum of the error in first restored multidimensional vector Vra relative to multidimensional vector V, the error in second restored multidimensional vector Vrb relative to multidimensional vector V, and the error in third restored multidimensional vector Vrc relative to multidimensional vector V, and the error in primary estimated emotion information Ei1 relative to learning emotion information EiL, is made as small as possible.

[0149] 6 , when learning is completed, the first decoder unit 114, the second decoder unit 115, and the third decoder unit 116 are removed from the primary model 100. Therefore, the primary model 100 has a first internal model 100a that combines the first encoder unit 111 and the first head unit 121, a second internal model 100b that combines the second encoder unit 112 and the second head unit 122, and a third internal model 100c that combines the third encoder unit 113 and the third head unit 123.

[0150] As a result, the primary model 100 is configured so that a plurality of first internal models 100a, second internal models 100b, and third internal models 100c can each be used to estimate the emotion of the subject T, based on the learning biological information BiL: electrocardiogram signal Bi1, pulse wave Bi2, electroencephalogram Bi3, sweating Bi4, and respiration Bi5. The primary model 100 is also configured to be able to output primary estimated emotion information Ei1, which is the average of first estimated emotion information Eia created using the first internal model 100a, second estimated emotion information Eib created using the second internal model 100b, and third estimated emotion information Eic created using the third internal model 100c.

[0151] <Creation of Secondary Biometric Information Distillation Model> Next, creation of the emotion estimation model 30 will be described with reference to Fig. 7. Fig. 7 shows a configuration when learning of the secondary biometric information distillation model 31 included in the emotion estimation model 30 in the emotion estimation device 1 is performed using the output of the primary model 100.

[0152] 7 , the emotion estimation model 30 is created by learning the training biometric information BiL and primary knowledge information Ki1 obtained based on the multidimensional vector V of the training biometric information BiL, which includes a plurality of different types of biometric information, using a trained primary model 100. In this embodiment, the emotion estimation model 30 uses primary estimated emotion information Ei1 created based on the multidimensional vector V of the training biometric information BiL using the trained primary model 100 as the primary knowledge information Ki1.

[0153] The emotion estimation model 30 includes a secondary biometric information distillation model 31 created by learning primary knowledge information Ki1 obtained based on a multidimensional vector V of training biometric information BiL including a plurality of different types of biometric information using a trained primary model 100. The secondary biometric information distillation model 31 includes a convolutional layer 32 used to calculate the biometric information Bi and the multidimensional vector V of the training biometric information BiL, and a fully connected layer 33 used to calculate secondary estimated emotion information Ei2 including an arousal value Ar and a valence value Va of an emotion estimated based on the multidimensional vector V.

[0154] Secondary biological information distillation model 31 is configured so that the parameters of convolutional layer 32 and fully connected layer 33 can be adjusted so that the sum of error L1 of secondary estimated emotion information Ei2 relative to learning emotion information EiL contained in the teacher data, and error L2 of secondary estimated emotion information Ei2 relative to primary estimated emotion information Ei1, which is primary knowledge information Ki1 of primary model 100, at an arbitrary ratio α is made as small as possible (see equation (1)).

[0155]

[0156] The secondary biometric information distillation model 31 configured in this manner is trained using primary estimated emotion information Ei1, which is primary knowledge information Ki1 obtained based on the multidimensional vector V of the training biometric information BiL, which is teacher data, using the trained primary model 100, secondary estimated emotion information Ei2 obtained based on the training biometric information BiL using the secondary biometric information distillation model 31, and the training emotion information EiL of the teacher data.

[0157] In this embodiment, the primary model 100 is trained using as training data the electrocardiogram signals Bi1, pulse waves Bi2, and respiration Bi5, which are the first biological information Bif in the training biological information BiL of a plurality of subjects T, as well as the electrocardiogram signals Bi3 and sweating Bi4, which are the second biological information Bis. When training is performed on the secondary biological information distillation model 31, the electrocardiogram signals Bi1, pulse waves Bi2, electroencephalograms Bi3, sweating Bi4, and respiration Bi5 are used for the primary model 100. Furthermore, the secondary biological information distillation model 31 uses training data including the electrocardiogram signals Bi1 and pulse waves Bi2, which are included in the first biological information Bif in the training biological information BiL, and training emotion information EiL, which is information on the subject T's self-reported emotion.

[0158] When learning is performed on the secondary biological information distillation model 31, the primary model 100 receives as input a multidimensional vector V of an electrocardiogram signal Bi1, a pulse wave Bi2, an electroencephalogram Bi3, sweating Bi4, and respiration Bi5, which constitute the learning biological information BiL, to a first internal model 100a including a first encoder unit 111 and a first head unit 121, a second internal model 100b including a second encoder unit 112 and a second head unit 122, and a third internal model 100c including a third encoder unit 113 and a third head unit 123. In the primary model 100, first estimated emotion information Eia is generated using the first encoder unit 111 and the first head unit 121, second estimated emotion information Eib is generated using the second encoder unit 112 and the second head unit 122, and third estimated emotion information Eic is generated using the third encoder unit 113 and the third head unit 123.

[0159] The primary model 100 creates primary estimated emotion information Ei1, which is the average of the first estimated emotion information Eia, second estimated emotion information Eib, and third estimated emotion information Eic. The primary model 100 outputs the primary estimated emotion information Ei1 to the secondary biological information distillation model 31 as primary knowledge information Ki1.

[0160] Of the electrocardiogram signal Bi1, pulse wave Bi2, electroencephalogram Bi3, sweating Bi4, and respiration Bi5 that constitute the learning biometric information BiL input to the primary model 100, the electrocardiogram signal Bi1 and pulse wave Bi2 are input to the convolutional layer 32 of the secondary biometric information distillation model 31. The control unit 20 uses the convolutional layer 32 of the secondary biometric information distillation model 31 to create a multidimensional vector V related to the electrocardiogram signal Bi1 and the pulse wave Bi2 based on the electrocardiogram signal Bi1 and the pulse wave Bi2. The control unit 20 outputs the created multidimensional vector V to the fully connected layer 33. The control unit 20 uses the fully connected layer 33 to create secondary estimated emotion information Ei2 based on the multidimensional vector V.

[0161] Control unit 20 calculates error L1 of secondary estimated emotion information Ei2 relative to learning emotion information EiL contained in the teacher data. Furthermore, control unit 20 calculates error L2 of secondary estimated emotion information Ei2 relative to primary estimated emotion information Ei1, which is primary knowledge information Ki1. Control unit 20 adjusts the parameters of convolutional layer 32 and fully connected layer 33 of secondary biometric information distillation model 31 so that the overall error L, which is the sum of errors L1 and L2 at an arbitrary ratio α, is as small as possible (see equation 1). At this time, the parameters of each encoder unit and each head unit of trained primary model 100 are not adjusted.

[0162] The primary knowledge information Ki1 is primary estimated emotion information Ei1 as the output of the primary model 100, obtained based on a plurality of different types of learning biological information BiL using the primary model 100. In other words, the primary knowledge information Ki1 is information (knowledge of the primary model 100) for estimating the emotion of the subject U, obtained by the primary model 100 learning based on an electrocardiogram signal Bi1, a pulse wave Bi2, an electroencephalogram Bi3, sweating Bi4, and respiration Bi5, which are a plurality of types of biological information Bi including the third biological information Bit.

[0163] By utilizing the primary knowledge information Ki1, the secondary biological information distillation model 31 learns while inheriting the primary estimated emotion information Ei1 containing information on the second biological information Bis and respiration Bi5, which include electrocardiogram Bi3 and sweating Bi4, which are information (privileged information) not included in the electrocardiogram signal Bi1 and pulse wave Bi2, which are the first biological information Bif used for learning the secondary biological information distillation model 31 (knowledge distillation).

[0164] In this way, the secondary biological information distillation model 31 that has been trained using the primary knowledge information Ki1 in addition to the electrocardiogram signal Bi1 and the pulse wave Bi2 can achieve higher emotion estimation accuracy than when training is performed using the electrocardiogram signal Bi1 and the pulse wave Bi2 without using the primary knowledge information Ki1. Furthermore, even when training is performed based on the electrocardiogram signal Bi1 and the pulse wave Bi2, which are two types of biological information less than the five types of biological information (electrocardiogram signal Bi1, pulse wave Bi2, electroencephalogram Bi3, sweating Bi4, and respiration Bi5) that the primary model 100 used to obtain the primary knowledge information Ki1, the secondary biological information distillation model 31 can achieve estimation accuracy close to that of the primary model 100.

[0165] The primary model 100 also includes a first internal model 100a including a first encoder unit 111 and a first head unit 121, which are used to estimate emotions based on a plurality of different types of learning biometric information BiL, a second internal model 100b including a second encoder unit 112 and a second head unit 122, and a third internal model 100c including a third encoder unit 113 and a third head unit 123. In the primary model 100, primary knowledge information Ki1 in which error and variation in the primary estimated emotion information Ei1 are suppressed is created by ensemble learning that averages the outputs of the first internal model 100a, the second internal model 100b, and the third internal model 100c, to which a plurality of different types of learning biometric information BiL have been input, respectively.

[0166] The secondary biological information distillation model 31 is trained based on primary knowledge information Ki1 that includes information on the electrocardiogram signal Bi1, pulse wave Bi2, electroencephalogram Bi3, sweating Bi4, and respiration Bi5 in addition to the electrocardiogram signal Bi1 and pulse wave Bi2. Thus, the secondary biological information distillation model 31 is trained using the primary knowledge information Ki1 that is the output of the primary model 100 in which errors and variations in emotion estimation accuracy have been reduced. As a result, by estimating emotions based on the electrocardiogram signal Bi1 and pulse wave Bi2 that can be easily acquired using the sensor 11, it is possible to improve the convenience of acquiring biological information Bi while also improving the accuracy of emotion estimation.

[0167] As shown in FIG. 3 , when estimating the emotion of a subject U, the control unit 20 of the emotion estimation device 1 uses a secondary biological information distillation model 31 to estimate the emotion of the subject U based on biological information related to the subject U's electrocardiogram signal Bi1 or pulse wave Bi2. The secondary biological information distillation model 31 has already learned the primary knowledge information Ki1 (see FIG. 7 ) obtained by the primary model 100 through learning. Thus, the control unit 20 can use the secondary biological information distillation model 31 to estimate the emotion of the subject U based on the electrocardiogram signal Bi1 or pulse wave Bi2, which is first biological information Bif of the subject U's biological information Bi that can be easily measured using a wearable sensor such as a smart watch or a band-type heart rate sensor (see FIG. 8 ). In other words, the control unit 20 uses the secondary biological information distillation model 31 to estimate the emotion of the subject U based on the subject U's biological information Bi that does not include second biological information Bis, which is difficult to measure using a wearable sensor, and outputs secondary estimated emotion information Ei2.

[0168] Furthermore, the control unit 20 utilizes the secondary biological information distillation model 31 to estimate the emotion of the subject U based on the biological information Bi related to the electrocardiogram signal Bi1 or the pulse wave Bi2 of the subject U, which is biological information Bi that includes a portion of the electrocardiogram signal Bi1, pulse wave Bi2, and respiration Bi5 included in the third biological information Bit and has a small number of types. The control unit 20 utilizes the secondary biological information distillation model 31 to estimate the emotion of the subject U based on the electrocardiogram signal Bi1 or the pulse wave Bi2, which can be easily measured by a wearable sensor such as a smart watch or a band-type heart rate sensor (see FIG. 8 ), among the training biological information BiL used in training the secondary biological information distillation model 31. In other words, the control unit 20 uses the secondary biometric information distillation model 31 to estimate the emotion of the subject U based on the biometric information Bi that is a part of the third biometric information Bit and has a small number of types out of multiple types of learning biometric information BiL including the third biometric information Bit used when training the secondary biometric information distillation model 31, and outputs the secondary estimated emotion information Ei2.

[0169] Furthermore, the emotion estimation device 1 can estimate emotions without restricting at least one of the environment around the subject U, the behavior of the subject U, or the activity of the subject U, by using the electrocardiogram signal Bi1 and the pulse wave Bi2 of the subject U, which are first biometric information Bif including at least one of biometric information Bi that is not easily affected by the environment around the subject U, the activity of the subject U, or the like, biometric information Bi that can be easily acquired from the subject U by a wearable sensor or the like, or biometric information Bi that is not easily affected by the environment around the subject U, the activity of the subject U, the activity of the subject U, or the like and that can be easily acquired from the subject U by a wearable sensor or the like. The emotion estimation device 1 can easily acquire, using a wearable sensor or the like, an electrocardiogram signal Bi1 and a pulse wave Bi2 of a subject U performing a specific task, action, or the like, such as a subject U driving a four-wheeled motor vehicle, a two-wheeled motor vehicle, a ship, an airplane, various types of work machinery, a drone, or the like, a subject U walking, a subject U exercising such as running, a subject U appreciating a movie, a painting, content, or the like.

[0170] Furthermore, the emotion estimation device 1 can estimate emotions without restricting at least one of the surrounding environment of the subject U, the behavior of the subject U, or the work of the subject U, based on the primary knowledge information Ki1 including information on the brain waves Bi3 and sweating Bi4 of the subject U, which is second biometric information Bi that includes at least one of the biometric information Bi that is easily influenced by the surrounding environment of the subject U, the work of the subject U, or the subject's movements, etc., the biometric information Bi that is difficult to easily obtain from the subject U using a wearable sensor or the like, or the biometric information Bi that is easily influenced by the surrounding environment of the subject U, the work of the subject U, the subject's movements, etc. and is difficult to easily obtain from the subject U using a wearable sensor or the like, and the biometric information Bi of the subject U that does not include the second biometric information Bi.

[0171] The secondary biological information distillation model 31 is created using the primary knowledge information Ki1 including information on the electroencephalogram Bi3 and sweating Bi4, which are the second biological information Bis, and therefore has higher estimation accuracy than a model that estimates emotion based only on the electrocardiogram signal Bi1 or pulse wave Bi2 of the subject U. As a result, by using the biological information Bi that can be easily acquired from the subject U to estimate emotion, it is possible to improve the convenience of acquiring the biological information Bi when estimating emotion while also improving the accuracy of emotion estimation.

[0172] Furthermore, the feeling estimation device 1, which estimates the subject U's feeling based on the subject U's electrocardiogram signal Bi1 and pulse wave Bi2 and the primary knowledge information Ki1, acquires a smaller amount of biometric information than when estimating the subject U's feeling using multiple different types of learning biometric information BiL, such as the electrocardiogram signal Bi1, pulse wave Bi2, electroencephalogram Bi3, sweating Bi4, and respiration Bi5. Thus, the feeling estimation device 1 can estimate the subject U's feeling with estimation accuracy close to that of the primary model 100 by using the primary knowledge information Ki1, while suppressing an increase in the processing load on the hardware.

[0173] [Embodiment 3] <Emotion Estimation Using Electrocardiogram Information or Pulse Wave> The overall configuration of a feeling estimation device 1A according to Embodiment 3 of the present invention will be described with reference to Fig. 9. Fig. 9 shows a configuration when training of a secondary biological information distillation model 34 included in an emotion estimation model 30 in the feeling estimation device 1A according to Embodiment 3 of the present invention is performed using the output of a primary model 100.

[0174] As shown in FIG. 9 , the feeling estimation device 1A uses the feeling estimation model 30 included in the control unit 20 to create emotion information Ei related to the emotion of the subject U based on biological information Bi related to the subject U's electrocardiogram signal Bi1.

[0175] The biometric information acquisition unit 10 of the feeling estimation device 1A uses a sensor 11 to acquire an electrocardiogram signal Bi1 or a pulse wave Bi2, which is biometric information Bi related to the heartbeat of the subject U. In the present embodiment, the feeling estimation device 1A acquires the electrocardiogram signal Bi1. The biometric information acquisition unit 10 is configured to acquire the electrocardiogram signal Bi1 of the subject U every unit time using, for example, a wearable sensor that can be easily worn by the subject U.

[0176] The emotion estimation model 30 included in the control unit 20 is a neural network model trained based on training data that is a combination of learning biometric information BiL related to the electrocardiogram signal Bi1 of the subject T and learning emotion information EiL self-reported by the subject T. The emotion estimation model 30 is configured to be able to input the electrocardiogram signal Bi1 of the subject U acquired by the biometric information acquisition unit 10.

[0177] <Creation of Secondary Biometric Information Distillation Model 34 (Emotion Inference Model 30)> Next, the creation of the emotion inference model 30 will be described. The emotion inference model 30 includes a secondary biometric information distillation model 34 created by learning primary knowledge information Ki1 obtained based on a plurality of different types of learning biometric information BiL, namely, an electrocardiogram signal Bi1, a pulse wave Bi2, an electroencephalogram Bi3, sweating Bi4, and respiration Bi5, using the trained primary model 100. The secondary biometric information distillation model 34 includes a convolutional layer 35 used to calculate a multidimensional vector V of the electrocardiogram signal Bi1, and a fully connected layer 36 used to calculate secondary estimated emotion information Ei2 including an emotional arousal value Ar and a valence value Va estimated based on the multidimensional vector V. The secondary biometric information distillation model 34 outputs secondary estimated emotion information Ei2 that estimates the emotion of the subject U in response to the electrocardiogram signal Bi1 included in the first biometric information Bif.

[0178] In the secondary biometric information distillation model 34, the electrocardiogram signal Bi1 is input to the convolutional layer 35. The convolutional layer 35 is used to create a multidimensional vector V based on the electrocardiogram signal Bi1. The secondary biometric information distillation model 34 outputs the multidimensional vector V to the fully connected layer 36. The fully connected layer 36 is used to create secondary estimated emotion information Ei2 based on the multidimensional vector V.

[0179] Control unit 20 calculates error L1 of secondary estimated emotion information Ei2 relative to learning emotion information EiL contained in the teacher data. Furthermore, control unit 20 calculates error L2 of secondary estimated emotion information Ei2 relative to primary estimated emotion information Ei1, which is primary knowledge information Ki1. Control unit 20 adjusts the parameters of convolutional layer 35 and fully connected layer 36 so that overall error L, which is the sum of error L1 and error L2 at an arbitrary ratio α, is as small as possible (see equation 1).

[0180] The secondary biological information distillation model 34 trained using the primary knowledge information Ki1 in this way can inherit information obtained by training by the primary model 100 that is not included in the electrocardiogram signal Bi1. Therefore, even when the secondary biological information distillation model 34 is trained based on fewer types of electrocardiogram signals Bi1 than the five types of biological information used when obtaining the primary knowledge information Ki1 using the primary model 100, namely the electrocardiogram signal Bi1, pulse wave Bi2, electroencephalogram Bi3, sweating Bi4, and respiration Bi5, the secondary biological information distillation model 34 has estimation accuracy close to that of the primary model 100. As a result, by using the electrocardiogram signal Bi1, which can be easily obtained from the subject U, for emotion estimation, it is possible to improve the convenience of obtaining bioinformation Bi when estimating emotions while also improving the accuracy of emotion estimation.

[0181] [Fourth Embodiment] <Emotion Estimation Using Tertiary Biometric Information Distillation Model> A feeling estimation device 1B according to a fourth embodiment of the present invention will be described with reference to Fig. 10. Fig. 9 shows a configuration when learning of a third-order biometric information distillation model 37 included in an emotion estimation model 30 in the feeling estimation device 1B according to the second embodiment of the present invention is performed using the output of a second-order biometric information distillation model 34.

[0182] As shown in FIG. 10 , the feeling estimation device 1B according to the fourth embodiment of the present invention uses the feeling estimation model 30 included in the control unit 20 to create emotion information Ei related to the emotion of the subject U from the biological information Bi related to the electrocardiogram signal Bi1 of the subject U.

[0183] The emotion estimation model 30 possessed by the control unit 20 is a neural network model that has been trained based on training data that is a combination of the electrocardiogram signal Bi1, which is learning biometric information BiL, and learning emotion information EiL self-reported by the subject T. The emotion estimation model 30 is configured to be able to input learning biometric information BiL related to the electrocardiogram signal Bi1 of the subject U acquired by the sensor 11.

[0184] <Creation of Tertiary Biometric Information Distillation Model 37> Next, we will explain the creation of the emotion estimation model 30. The emotion estimation model 30 includes a tertiary biometric information distillation model 37 that is created by learning secondary knowledge information Ki2 obtained based on the multidimensional vector V of the training biometric information BiL using a trained secondary biometric information distillation model 34. In this embodiment, the tertiary biometric information distillation model 37 is trained using, as secondary knowledge information Ki2, secondary estimated emotion information Ei2 created by the trained secondary biometric information distillation model 34 based on the multidimensional vector V of the training biometric information BiL.

[0185] The tertiary biometric information distillation model 37 is used to estimate emotions based on the number of types of learning biometric information BiL equal to or less than the number of types of learning biometric information BiL used when obtaining the secondary knowledge information Ki2 using the secondary biometric information distillation model 34. The tertiary biometric information distillation model 37 includes a convolutional layer 38 used to calculate a multidimensional vector V of the biometric information Bi, and a fully connected layer 39 used to calculate tertiary estimated emotion information Ei3 including an arousal value Ar and a valence value Va of the emotion estimated based on the multidimensional vector V.

[0186] The control unit 20 adjusts the parameters of the convolutional layer 38 and the fully connected layer 39 so that the sum of the error L1 of the tertiary estimated emotion information Ei3 relative to the learning emotion information EiL contained in the teacher data input to the convolutional layer 38 and the error L2 of the tertiary estimated emotion information Ei3 relative to the secondary estimated emotion information Ei2, which is the secondary knowledge information Ki2 of the secondary biological information distillation model 34, at an arbitrary ratio α is as small as possible (see equation 1).

[0187] The tertiary biometric information distillation model 37 configured in this manner is trained using secondary inferred emotion information Ei2 obtained based on the multidimensional vector V of the training biometric information BiL, which is the training data, using the trained secondary biometric information distillation model 34, tertiary inferred emotion information Ei3 obtained based on the training biometric information BiL using the tertiary biometric information distillation model 37, and the training emotion information EiL of the training data.

[0188] In this embodiment, the secondary biological information distillation model 34 is a model that has been trained based on the primary knowledge information Ki1 created using the primary model 100 and the electrocardiogram signals Bi1 of multiple subjects T (see FIG. 9 ). When training the tertiary biological information distillation model 37, the electrocardiogram signals Bi1, which are training biological information BiL, are used in the secondary biological information distillation model 34. In addition, the tertiary biological information distillation model 37 uses the electrocardiogram signals Bi1 of the subjects T, which are training data.

[0189] When learning is performed on the third-order biometric information distillation model, the electrocardiogram signal Bi1 is input to the convolutional layer 35 of the second-order biometric information distillation model 34. In the second-order biometric information distillation model 34, a multidimensional vector V related to the electrocardiogram signal Bi1 is created based on the electrocardiogram signal Bi1 using the convolutional layer 35. The second-order biometric information distillation model 34 outputs the multidimensional vector V to the fully connected layer 36. In the second-order biometric information distillation model 34, secondary estimated emotion information Ei2 is created based on the multidimensional vector V using the fully connected layer 36.

[0190] In the third-order biometric information distillation model 37, an electrocardiogram signal Bi1 that is the same as the electrocardiogram signal Bi1 input to the second-order biometric information distillation model 34 is input to a convolutional layer 38. In the third-order biometric information distillation model 37, a multidimensional vector V is created based on the electrocardiogram signal Bi1 using the convolutional layer 38. The third-order biometric information distillation model 37 outputs the multidimensional vector V to a fully connected layer 39. In the third-order biometric information distillation model 37, third-order estimated emotion information Ei3 is created based on the multidimensional vector V using the fully connected layer 39.

[0191] The control unit 20 calculates the error L1 of the tertiary estimated emotion information Ei3 relative to the learning emotion information EiL contained in the teacher data. Furthermore, the control unit 20 calculates the error L2 of the tertiary estimated emotion information Ei3 relative to the secondary estimated emotion information Ei2, which is secondary knowledge information Ki2. The control unit 20 adjusts the parameters of the convolutional layer 38 and fully connected layer 39 so that the overall error L, which is the sum of errors L1 and L2 at an arbitrary ratio α, is as small as possible. At this time, the parameters of the encoder unit and fully connected layer of the trained secondary biological information distillation model 34 are not adjusted.

[0192] The secondary knowledge information Ki2 is information for estimating emotions based on the electrocardiogram signal Bi1 of the subject U, which is obtained by the secondary biological information distillation model 34 through learning of the electrocardiogram signal Bi1 and the primary knowledge information Ki1. In other words, the secondary knowledge information Ki2 includes information (knowledge) obtained through learning by the primary model 100 and the secondary biological information distillation model 34. The tertiary biological information distillation model 37 trained using the secondary knowledge information Ki2 can inherit information not included in the electrocardiogram signal Bi1.

[0193] The third-order biological information distillation model 37 trained using the secondary knowledge information Ki2 in this way can obtain estimation accuracy close to the emotion estimation accuracy of the secondary biological information distillation model 34, even when trained based on the electrocardiogram signals Bi1 of the subject U, which have fewer types than the number of types of learning biological information BiL used when obtaining the primary knowledge information Ki1 using the secondary biological information distillation model 31. As a result, by using the biological information Bi that can be easily acquired from the subject U for emotion estimation, it is possible to improve the convenience of acquiring the biological information Bi when estimating emotions while also improving the emotion estimation accuracy.

[0194] (Other Embodiments) In each of the above-described embodiments, the control unit 20 estimates the emotion of the subject U or the subject T using at least one type of biometric information Bi from among the electrocardiogram signal Bi1, pulse wave Bi2, electroencephalogram Bi3, sweating Bi4, and respiration Bi5 of the subject U or the subject T. However, the biometric information used by the control unit may be biometric information other than the biometric information related to the electrocardiogram signal Bi1, pulse wave Bi2, electroencephalogram Bi3, sweating Bi4, and respiration Bi5. For example, the control unit may estimate the emotion from electromyography, body surface temperature, eye movement, pupil state, triaxial acceleration of the body, etc.

[0195] Furthermore, in each of the above-described embodiments, the control unit 20 generates the emotional arousal value Ar and valence value Va, which are information related to the emotions of the subject U or subject T, as secondary estimated emotion information Ei2 or tertiary estimated emotion information Ei3. That is, the emotion estimation model 30 indicates the emotion of the subject U or subject T by combining the emotional arousal value Ar and the valence value Va. However, the emotion estimation model may also indicate the emotions of the subject U or subject T as a ratio of each emotion. For example, the emotion estimation model may indicate the emotions of the subject U or subject T as 0.85 for joy, 0.12 for pleasure, 0.02 for sadness, 0.01 for anger, etc.

[0196] Furthermore, in each of the above-described embodiments, the emotion estimation model 30 indicates the valence value Va of the subject U or the test subject T as a degree between a positive emotion and a negative emotion, and indicates the arousal value Ar as a degree between an activated emotional state and an inactivated emotional state. However, the emotion estimation model may also indicate the valence value Va and the arousal value Ar of the subject U or the test subject T by classifying them into two levels: high level and low level.

[0197] Furthermore, in each of the above-described embodiments, ensemble learning is performed in the primary model 100 to create primary estimated emotion information Ei1, which is the average of first estimated emotion information Eia, second estimated emotion information Eib, and third estimated emotion information Eic, which are created using the first internal model 100a, second internal model 100b, and third internal model 100c. However, other types of ensemble learning may be performed for the primary model. For example, ensemble learning may be performed in the primary model to create second estimated emotion information using a second internal model based on the first estimated emotion information created by the first internal model, and to create third estimated emotion information using a third internal model based on the second estimated emotion information.

[0198] Furthermore, in each of the above-described embodiments, the secondary biometric information distillation model 31 uses the primary estimated emotion information Ei1 created using the trained primary model 100 as the primary knowledge information Ki1. However, the secondary biometric information distillation model may be trained using intermediate layer parameters of the trained primary model as the primary knowledge information. Similarly, the tertiary biometric information distillation model may use intermediate layer parameters of the trained secondary biometric information distillation model as the secondary knowledge information.

[0199] Furthermore, in each of the above-described embodiments, the primary model 100 receives as input a multidimensional vector V of the training biometric information BiL. However, the primary model may also receive training biometric information. In this case, the primary model is configured as a convolutional neural network model having a convolutional layer. Furthermore, the secondary biometric information distillation model 34 and the tertiary biometric information distillation model 37 receive as input the training biometric information BiL. However, the secondary biometric information distillation model and the tertiary biometric information distillation model may also receive as input a multidimensional vector of the training biometric information.

[0200] Furthermore, in each of the above-described embodiments, the primary model 100 has a first encoder unit 111, a second encoder unit 112, and a third encoder unit 113. Furthermore, the primary model 100 has a first head unit 121, a second head unit 122, and a third head unit 123. However, the primary model does not have to be a model that has at least one of an encoder or a head unit.

[0201] In each of the above-described embodiments, the primary model 100, the secondary biometric information distillation models 31 and 34, and the tertiary biometric information distillation model 37 are configured by neural network models. However, the primary model, the secondary biometric information distillation model, and the tertiary biometric information distillation model may be configured by a machine learning model such as a support vector machine (SVM), a decision tree, a k-nearest neighbor method, a naive Bayes algorithm, a logistic regression, a linear regression, a nonlinear regression, or a stepwise regression.

[0202] In each of the above-described embodiments, the feeling estimation devices 1, 1A, and 1B are configured with a mobile device Pt such as a smartphone carried by the subject U and a wearable sensor or the like. The feeling estimation devices 1, 1A, and 1B estimate the subject U's feeling using a processor and memory of the mobile device Pt based on biometric information Bi acquired by the wearable sensor or the like. However, the feeling estimation device may be configured as part of a feeling estimation system in which multiple feeling estimation devices are connected to a server via the Internet or the like. The feeling estimation system may be connected to multiple feeling estimation devices via the Internet or the like. The feeling estimation system may be configured to be able to collect estimated emotion information of each subject U from multiple feeling estimation devices.

[0203] In each of the above-described embodiments, the emotion estimation model 30 includes a secondary biometric information distillation model 31 that has learned primary knowledge information Ki1 created using a trained primary model 100, or a tertiary biometric information distillation model 37 that has learned secondary knowledge information Ki2 created using a trained secondary biometric information distillation model 34. However, the emotion estimation model may also include a fourth-order model that has learned tertiary knowledge information using information created using a trained third-order biometric information distillation model as tertiary knowledge information. In other words, the emotion estimation model may include a model that has learned information created using a trained model. Furthermore, the emotion estimation model may include models, programs, etc. other than models that estimate emotions.

[0204] In each of the above-described embodiments, the n-th order biometric information distillation model is configured as the second order biometric information distillation models 31 and 34 and the third order biometric information distillation model 37. The n-th order biometric information distillation model may also include a fourth order biometric information distillation model and a biometric information distillation model of a higher order than the fourth order biometric information distillation model.

[0205] Although the embodiments of the present invention have been described above, the above-described embodiments are merely examples for carrying out the present invention. Therefore, the present invention is not limited to the above-described embodiments, and it is possible to appropriately modify the above-described embodiments within the scope of the spirit of the present invention.

[0206] 1, 1A, 1B Emotion estimation device 10 Biometric information acquisition unit 11 Sensor 12 Sensor communication device 20 Control unit 21 Control unit communication device 30 Emotion estimation model 31, 34 Secondary biometric information distillation model 32, 35 Convolutional layer 33, 36 Fully connected layer 37 Third biometric information distillation model 38 Convolutional layer 39 Fully connected layer 40 Memory unit 50 Output unit Bi Biometric information BiL Learning biometric information Bi1 ECG signal Bi2 Pulse wave Bi3 Electroencephalogram Bi4 Sweating Bi5 Respiration Bif First biometric information Bis Second biometric information Bit Third biometric information EiL Learning emotion information Ei1 Primary estimated emotion information Ei2 Second estimated emotion information Ei3 Third estimated emotion information Eia First estimated emotion information Eib Second estimated emotion information Eic Third estimated emotion information V Multidimensional vector Vca First compressed multidimensional vector Vcb Second compressed multidimensional vector Vcc Third compressed multidimensional vector Vra First restored multidimensional vector Vrb Second restored multidimensional vector Vrc Third restored multidimensional vector Ki1 Primary knowledge information Ki2 Secondary knowledge information 100 Primary model 100a First internal model 100b Second internal model 100c Third internal model 110 Autoencoder 111 First encoder unit 112 Second encoder unit 113 Third encoder unit 114 First decoder unit 115 Second decoder unit 116 Third decoder unit 121 First head unit 122 Second head unit 123 Third head unit L, L1, L2 Error Pt Mobile terminal

Claims

1. A storage unit that stores biometric information; a control unit having an emotion estimation model used for estimating an emotion based on the biometric information, acquiring biometric information of a subject and storing it in the storage unit, estimating an emotion of the subject based on the biometric information of the subject stored in the storage unit by using the emotion estimation model, and outputting information related to the estimated emotion as emotion information; An emotion estimation device having: When a model used for estimating emotions based on a plurality of types of biometric information is defined as a primary model, and a model created using n-1-order knowledge information obtained by an n-1-order biometric information distillation model, such as a secondary biometric information distillation model created using primary knowledge information obtained by the primary model, or a tertiary biometric information distillation model created using secondary knowledge information obtained by the secondary biometric information distillation model, is defined as an n-order biometric information distillation model (n is an integer of 2 or greater), The emotion estimation model is The n-th order biometric information distillation model is used to estimate the subject's emotion based on the biometric information that is included in the types of biometric information required for emotion estimation in the first model and that is fewer than the number of types of biometric information required for emotion estimation in the first model, The control unit is Using the n-th order biometric information distillation model, an emotion of the subject is estimated based on biometric information of the subject, the number of types of which is included in the types of biometric information necessary for estimating emotion in the first model and is less than the number of types of biometric information necessary for estimating emotion in the first model, and information related to the estimated emotion is output as emotion information. Emotion estimation device.

2. The emotion estimation device according to claim 1 , The n-th order biological information distillation model is the first knowledge information obtained by the first model created based on a plurality of types of bioinformation including first bioinformation, which is at least one of the bioinformation that is not easily influenced by the environment around the subject, the work of the subject, or the motion of the subject, the bioinformation that can be easily obtained from the subject, or the bioinformation that is not easily influenced by the environment around the subject, the work of the subject, or the motion of the subject and can be easily obtained from the subject, or the n-1-order knowledge information obtained by the n-1-order bioinformation distillation model created based on the bioinformation including the first bioinformation, The control unit is Using the n-th order biometric information distillation model, an emotion of the subject is estimated based on the first biometric information, and information regarding the estimated emotion is output as the emotion information. Emotion estimation device.

3. The emotion estimation device according to claim 1 , The n-th order biological information distillation model is the first-order knowledge information obtained by the first-order model created based on a plurality of types of bioinformation including the bioinformation that is easily influenced by the environment around the subject, the work of the subject, or the motion of the subject, the bioinformation that is difficult to easily obtain from the subject, or the second bioinformation that is at least one of the bioinformation that is easily influenced by the environment around the subject, the work of the subject, or the motion of the subject and that is difficult to easily obtain from the subject, or the n-1-order knowledge information obtained by the n-1-order bioinformation distillation model created based on the bioinformation including the second bioinformation, The control unit is Using the n-th order biometric information distillation model, an emotion of the subject is estimated based on the biometric information of the subject not including the second biometric information, and information regarding the estimated emotion is output as the emotion information. Emotion estimation device.

4. The emotion estimation device according to claim 1 , The n-th order biological information distillation model is The method is created using the first-order knowledge information obtained by the first-order model created based on a plurality of types of bioinformation including third bioinformation, which is at least two of brain waves, sweating, respiration, pulse waves, and electrocardiogram signals, or the n-1-order knowledge information obtained by the n-1-order bioinformation distillation model created based on the bioinformation including the third bioinformation, The control unit is Using the n-th order biometric information distillation model, an emotion of the subject is estimated based on the biometric information including a part of the plurality of types of biometric information included in the third biometric information and having a small number of types, and information regarding the estimated emotion is output as emotion information. Emotion estimation device.

5. The feeling estimation device according to claim 1 , The control unit is using the emotion estimation model, estimating an emotion of the subject based on bio-information related to at least one of an electrocardiogram signal, respiration, or a pulse wave among bio-information related to an electroencephalogram, sweating, respiration, an electrocardiogram signal, or a pulse wave of the subject, and outputting information related to the estimated emotion as the emotion information; Emotion estimation device.

6. The feeling estimation device according to claim 1 , The n-1 order bioinformation distillation model is A plurality of internal models are included that are used to estimate emotions based on the plurality of types of bioinformation, The emotion estimation model is The n-th order biometric information distillation model is created using the n-1 order knowledge information obtained based on the plurality of types of biometric information using the plurality of internal models. Emotion estimation device.

7. The emotion estimation device according to claim 1 , The emotion estimation model is The n-th order biological information distillation model is created using the n-1th order knowledge information obtained based on a plurality of types of biological information including biological information regarding at least one of an electrocardiogram signal, breathing, or a pulse wave among biological information regarding an electroencephalogram, sweating, breathing, a pulse wave, or an electrocardiogram signal, and the biological information including biological information regarding at least one of an electrocardiogram signal, breathing, or a pulse wave, using the n-1th order biological information distillation model. Emotion estimation device.

8. The emotion estimation device according to claim 1 , The emotion estimation model is The n-th order bioinformation distillation model is a neural network model created using the n-th order knowledge information obtained based on the n-th order bioinformation distillation model, which is a neural network model; Emotion estimation device.

9. A mobile terminal that acquires biometric information of a subject to be used for estimating the subject's emotion in the emotion estimation device according to any one of claims 1 to 8, A portable terminal memory that stores the measured biological information of the subject; A processor for a mobile terminal, The mobile terminal processor includes: acquiring biometric information of the subject and storing it in the memory for the portable terminal; and outputting the biometric information of the subject stored in the memory for the portable terminal. Mobile device.

10. A mobile terminal as claimed in claim 9, The mobile terminal includes: The device is configured to be capable of communicating with the outside world, The mobile terminal processor includes: outputting the biological information of the subject stored in the portable device memory to the feeling estimation device outside the portable device; Mobile device.

11. An emotion estimation device according to any one of claims 1 to 8, which estimates an emotion of the subject based on the bioinformation acquired by the mobile terminal according to claim 10, The emotion estimation device includes: The device is configured to be capable of communicating with the outside world, The control unit is acquiring the biometric information from the mobile terminal; Emotion estimation device.

12. A portable emotion estimation device which is the emotion estimation device according to claim 1 and also the mobile terminal according to claim 9, The mobile terminal processor functions as the control unit, The portable terminal memory functions as the storage unit, The mobile terminal processor includes: a feeling estimation model including the n-th order biometric information distillation model, acquiring biometric information of the subject output from a biometric information acquisition unit and storing the acquired information in the storage unit, estimating the feeling of the subject based on the biometric information of the subject that is included in the types of the biometric information required for estimating feelings in the primary model and that is a number of types less than the number of types of the biometric information required for estimating feelings in the primary model, using the n-th order biometric information distillation model, and outputting information related to the estimated feeling as feeling information; Emotion estimation device.

13. A program for controlling the emotion estimation device according to any one of claims 1 to 8, The program is storing biological information of the subject in the storage unit; and causing the control unit to utilize the n-th order biometric information distillation model to estimate an emotion of the subject based on biometric information of the subject, the number of types of which is included in the types of biometric information required for estimating an emotion in the first model and is less than the number of types of the biometric information required for estimating an emotion in the first model, and to output information related to the estimated emotion as emotion information. program.