Information processing device and information processing method

Abstract

To enable future human feelings to be predicted with high accuracy on the basis of time change of a plurality of environment data.SOLUTION: An information processing device includes a learning part for constructing a prediction mode for predicting feelings at a future prediction time by machine learning on the basis of teacher data showing a correspondence relation between at least a plurality of time changes of environment and time changes of feelings within a first period, an acquisition part for acquiring data of at least the plurality of time changes of the environment within a second period just before the prediction time, and a prediction part for inputting the data of at least the plurality of time changes of the environment acquired by the acquisition part to the prediction model, and predicting feelings of the prediction time.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] With recent changes in learning and working environments, research into grasping and estimating human mental states (hereinafter referred to as emotions), such as stress, fatigue, comfort, and emotional arousal caused by these environments, has been attracting attention. Methods for grasping and estimating emotions are generally subjective analyses such as psychological experiments and questionnaires, but on the other hand, objective and quantitative analyses based on physiological data and psychological indicators such as eye movement, body temperature, and electroencephalograms are also being conducted, and in recent years, research has been conducted to clarify the correspondence between physiological data and psychological indicators using machine learning and deep learning. For example, a method has been proposed for estimating the emotions of people in an indoor environment from indoor environmental data obtained from wireless sensors, etc. (see Non-Patent Documents 1 and 2). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Sensing mental states (1)(http: / / sensait.jp / 22706 / ) [Non-Patent Document 2] Sensing mental states (2)(http: / / sensait.jp / 22709 / ) Summary of the Invention [Problem to be solved by the invention]

[0004] However, the disclosures of Non-Patent Documents 1 and 2 are limited to estimating current human emotions based on time-dependent changes in multiple environmental data, and do not disclose a method for predicting future human emotions.

[0005] Therefore, an embodiment of the present invention provides an information processing device and an information processing method that can accurately predict future human emotions based on time-dependent changes in a plurality of environmental data. [Means for solving the problem]

[0006] In order to solve the above problems, according to an embodiment of the present invention, there is provided a method and apparatus for predicting emotions at a future prediction time based on teacher data indicating a correspondence relationship between at least a plurality of environmental changes over time and emotion changes over time within a first time period; an acquisition unit that acquires data of time changes of at least the plurality of environments within a second period immediately before the predicted time; a prediction unit that inputs data on at least the plurality of environmental changes over time acquired by the acquisition unit into the prediction model and predicts emotions at the prediction time, An information processing device is provided. Effect of the Invention

[0007] Recently, in response to the spread of new infectious diseases, educational work environments have become more diverse due to infection prevention measures. In order to manage the mental and physical health of workers in diversifying educational work environments, it is becoming more important than ever to understand the mental state of people (mental health), such as emotional valence including stress, fatigue, and comfort, and each emotional system, as well as to understand and improve the work environment.

[0008] The inventor has thus far built a system that collects environmental information and biological data obtained from non-contact sensor data, and can objectively and non-invasively specify and grasp the mental and physical health conditions of people, and discovered a method for comprehensively evaluating the educational work environment. Note that a non-contact environmental sensor is an environmental sensor that can measure without being worn on the human body.

[0009] However, the current technology of real-time emotion estimation cannot prevent accidents while driving a car or in a factory, nor can it improve the efficiency of work or study in the office.

[0010] According to an embodiment of the present invention, by using a prediction model constructed by machine learning based on training data showing the correspondence between at least multiple environmental changes over time and emotions over time, emotions (mental state) at a predicted time in the future can be estimated.

[0011] According to the embodiment of the present invention, by predicting future emotions rather than in real time, it is possible to prevent accidents while driving a car or in a factory, which is a conventional technology that estimates emotions in real time, and to greatly improve the effectiveness of accident prevention. In addition, it is possible to more efficiently improve the effectiveness of work and study in the office.

[0012] In addition, the embodiment of the present invention may be applied to a non-contact environment using a non-contact sensor. The non-contact environment does not have restrictions on use such as the need to wear a wearable terminal continuously, and the embodiment of the present invention can be applied in various situations. In addition, since there is no stress on the subject, estimation can be performed in a natural state.

[0013] In this way, the embodiment of the present invention is characterized by obtaining data on time changes in a plurality of environments and analyzing the data in a time series to predict future emotions. Biometric data directly affects emotions and is invasive and subjective in nature, but the embodiment of the present invention focuses on data on non-invasive time changes in the environment that affect the biometric data, and can predict emotions from the data non-invasively, objectively, and with high accuracy, without using such biometric data.

[0014] The embodiment of the present invention can predict future changes in emotional states based on the environmental conditions to which a person is currently exposed. Based on this prediction, it is expected that the educational and working environment can be improved to be more comfortable, less stressful, and with cleaner air, thereby maintaining physical and mental health. This can be expected to be applied to various fields, such as improving mental health by controlling the living environment, preventive medicine, safe driving support, preventing accidents in factories, improving work efficiency in offices, and improving the effectiveness of learning in schools. [Brief description of the drawings]

[0015] [Figure 1] 1 is a block diagram showing a schematic configuration of an information processing device according to an embodiment of the present invention; [Diagram 2] FIG. 4 is a diagram showing emotion data according to the embodiment of the present invention. [Figure 3A] 1 is a graph showing the correlation between temperature and emotions. [Figure 3B] 1 is a graph showing the correlation between blue light and emotions. [Figure 3C] 1 is a graph showing the correlation between volume and emotion. [Figure 3D] 1 is a graph showing the correlation between CO2 concentration and emotions. [Figure 4] 4 is a flowchart showing a process of the information processing apparatus according to the embodiment of the present invention. [Figure 5A] FIG. 1 is a block diagram showing a prediction calculation process using a CNN model. [Figure 5B] FIG. 1 is a block diagram showing a prediction calculation process using an LSTM model. [Figure 6] FIG. 1 is a block diagram illustrating an LSTM model. [Figure 7] 13 is a graph showing the accuracy of emotion prediction for each number of sensors. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of an information processing apparatus and an information processing method will be described with reference to the drawings.

[0017] (Embodiments of the present invention) 1 is a block diagram showing a schematic configuration of an information processing device 1 according to an embodiment of the present invention. The information processing device 1 measures the state of a given environment (e.g., in a laboratory) and predicts the future emotions of a subject in the environment. The information processing device 1 includes an acquisition unit 2 and an information processing unit 3.

[0018] The acquisition unit 2 receives input of multiple environmental data of the above-mentioned environment and the subject's emotions, and provides them to the information processing unit 3. The acquisition unit 2 has an emotion input unit 11 and an environmental sensor unit 13. The acquisition unit 2 may also have a biosensor unit 12 to receive input of biodata of the subject in the environment. In the following embodiment, a form in which the acquisition unit 2 has the biosensor unit 12 will be described.

[0019] The emotion input unit 11 accepts input of the subject's emotion, and is composed of an input terminal such as a computer. The emotion input unit 11 is connected to the information processing unit 3, for example, via a wired or wireless network. The emotion input unit 11 may also be included in the information processing unit 3. Emotions input to the emotion input unit 11 include, for example, Happy, Stress, Relax, and Sad. Details of emotions will be described later. The emotions input to the emotion input unit can also be obtained, for example, from measured values ​​of biological data (brain waves, etc.).

[0020] The biosensor unit 12 measures data related to the subject's body in the environment, and is composed of, for example, a contact or non-contact sensor, a wearable device, etc. In this specification, the data acquired by the biosensor unit 12 is called biodata. The biosensor unit 12 acquires multiple biodata such as the subject's body temperature, skin temperature (skin temperature), blood pressure, body movement, and pulse rate.

[0021] The environmental sensor unit 13 measures data related to a predetermined environment (for example, in a laboratory) and is composed of, for example, sensors and cameras installed in the laboratory. In this specification, data acquired by the environmental sensor unit 13 is called environmental data. The environmental sensor unit 13 acquires a plurality of environmental data, for example, indoor temperature, visible light intensity (illuminance), odor (odor intensity), humidity, blue light intensity, dust (particulate concentration), sound volume, CO2 concentration, VOC (Volatile Organic Compounds), and air pressure. The environmental sensor unit 13 may also measure the distance between subjects by, for example, infrared rays. The environmental sensor unit 13 may also have a human sensor that measures the distance between the subject and a sensor constituting the environmental sensor unit 13.

[0022] The data obtained from the biosensor unit 12 and the environmental sensor unit 13 is transferred to the information processing unit 3, for example, via a wireless network.

[0023] The information processing unit 3 learns the information input from the acquisition unit 2, and predicts the subject's future environment based on the information input from the acquisition unit 2. The information processing unit 3 has a model construction unit 21, a neural network unit 22, and a prediction unit 23. The neural network unit 22 and the model construction unit 21 constitute a learning unit 24.

[0024] The model construction unit 21 acquires data indicating time changes of multiple emotions during a predetermined learning period (hereinafter referred to as the first period) from the emotion input unit 11, and acquires data indicating time changes of multiple biological data and multiple environmental data during the first period from the biological sensor unit 12 and the environmental sensor unit 13, and associates these data with time information.

[0025] In this specification, the data constructed in the model construction unit 21 during the first period is referred to as teacher data. The teacher data includes a correspondence relationship between the time changes of a plurality of pieces of environmental data and the time changes of a plurality of pieces of emotion data, and a correspondence relationship between the time changes of a plurality of pieces of biological data and the time changes of a plurality of pieces of emotion data. The model construction unit 21 supplies this teacher data to the neural network unit 22, thereby constructing a prediction model that predicts emotions at a future prediction time by machine learning.

[0026] The neural network unit 22 holds the prediction model obtained by the above machine learning.

[0027] The prediction unit 23 acquires data indicating time changes of a plurality of environmental data during a predetermined measurement period (hereinafter referred to as the second period) from the environmental sensor unit 13, and inputs the data to a prediction model in the neural network unit 22. In the second period, at least a part of the plurality of environmental data measured in the first period is measured. The prediction unit 23 may input only data indicating time changes of the plurality of environmental data acquired from the environmental sensor unit 13 to the prediction model. In addition to data indicating time changes of the plurality of environmental data, the prediction unit 23 may input data indicating time changes of the plurality of biological data acquired from the biological sensor unit 23 to the prediction model. In the second period, the prediction unit 23 predicts emotions at a future prediction time based on the data input to the prediction model. The prediction unit 23 outputs the predicted emotion data to, for example, an external device (not shown). In this specification, data input from the biological sensor unit 12 and the environmental sensor unit 13 during the second period is referred to as input data.

[0028] The first period for accumulating the teacher data is sufficiently longer than the second period, which is the measurement period. For example, the first period is 80 hours, and the second period is 3 minutes. The above-mentioned future predicted time refers to a time after the second period. For example, the prediction unit 23 predicts the emotion one minute after the second period.

[0029] FIG. 2 is a diagram showing emotion data according to an embodiment of the present invention. The emotion data has two scales, arousal level and comfort level. The emotion data has a Happy region (first region) where the arousal level is higher than a predetermined reference value (first reference value) and the comfort level is higher than a predetermined reference value (second reference value), a Stress region (second region) where the arousal level is higher than the predetermined reference value and the comfort level is equal to or lower than the predetermined reference value, a Relax region (third region) where the arousal level is equal to or lower than the predetermined reference value and the comfort level is higher than the predetermined reference value, and a Sad region (fourth region) where the arousal level is equal to or lower than the predetermined reference value and the comfort level is also equal to or lower than the predetermined reference value. The Stress region is sometimes called an Angry region.

[0030] To the emotion input unit 11, information on which of the above four regions the subject's emotion falls into is input. Furthermore, the prediction unit 23 predicts to which of the above four regions the emotion at a future predicted time falls. Note that the emotion input unit 11 may also receive an arousal level and a comfort level, and the prediction unit 23 may predict the arousal level and the comfort level based on this.

[0031] A subject's biometric data (e.g., heart rate, skin temperature) directly affects the emotional data, causing the subject's emotional data to fluctuate. In addition, for example, a plurality of environmental data (e.g., temperature, humidity, light intensity, odor) in the laboratory in which the subject is present affects the subject's biometric data, and the subject's emotional data can be objectively estimated by using the plurality of environmental data.

[0032] When multiple pieces of environmental data change over time, emotion data also changes over time. It is possible to predict future emotion data based on multiple pieces of environmental data that change over time. In this way, in an embodiment of the present invention, it is possible to predict future emotion data by analyzing environmental data and time information in a time series.

[0033] 3A to 3D are graphs showing the correlation, or correspondence, between representative environmental data among a plurality of environmental data and emotion data. FIG. 3A to 3D show, as time-series data, the change over time (change over time) of each environmental parameter in a room in which a subject is present, and the emotion of the subject at the measurement time. In each of FIG. 3A to 3D, the emotion of the subject at each measurement time is displayed as a plot Pem. The plot Pem is indicated by an x ​​mark on FIG. 3A to 3D. Furthermore, the relationship between the measurement time and emotion is the same in FIG. 3A to 3D.

[0034] In Figures 3A to 3D, the horizontal axis is the measurement time, and the vertical axis is temperature, blue light, sound volume, and CO2 concentration. In Figures 3A, 3B, 3C, and 3D, the changes over time of temperature, blue light, sound volume, and CO2 concentration are shown by solid line graphs W1, W2, W3, and W4, respectively. Figures 3A to 3D show that the subject's emotions change when there is an environmental change that can stimulate the subject.

[0035] As shown in Figs. 3A to 3D, a change in the subject's emotions is not caused by only one specific environmental data, but the subject's emotions may change in response to changes in multiple environmental data. If the environment, such as temperature, changes suddenly, it seems intuitive that the emotional data will become Stress, but for example, in Fig. 3B, a subject whose emotional data is Happy is also observed. This is because other environmental data have an influence. In this way, depending on the type of environmental data, it may not be possible to say that one environmental data alone is correlated with any emotional data. Unlike biological data that directly influences emotional data, environmental data has the characteristic that the affecting emotional data changes depending on the combination with other environmental data.

[0036] For this reason, prediction unit 23 predicts emotion data based on multiple pieces of environmental data. To improve the emotion data prediction accuracy in prediction unit 23, it is desirable that a larger amount of environmental data be input to model construction unit 21 and prediction unit 23. Also, it is desirable that each piece of environmental data be weighted according to the strength of its influence on the emotion data.

[0037] 4 is a flowchart showing a processing procedure of the information processing device 1 according to the embodiment of the present invention. First, teacher data is generated by the emotion input unit 11, the biosensor unit 12, and the environment sensor unit 13 (step S1). The teacher data is generated, for example, by placing one or more subjects in a laboratory and continuing for, for example, 80 hours. One or more environment sensor units 13 are placed in the laboratory, and one or more subjects are fitted with the biosensor unit 12. The emotions of the subjects are input to the emotion input unit 11 at regular intervals.

[0038] Incidentally, the frequency of occurrence of emotions varies from subject to subject, and an imbalance may occur. It is desirable for the emotion input unit 11 to equalize the frequency of occurrence of emotions by oversampling the emotions of the subjects, thereby eliminating this imbalance. In this specification, an example will be described in which the learning unit 24 performs oversampling processing of emotion data in the teacher data.

[0039] The teacher data is data in which time-series input data such as temperature, blue light, and sound volume in a room in which one or more subjects are placed, output from the environmental sensor unit 13, and time-series emotion data of multiple subjects input from the emotion input unit 11 are associated in chronological order. The teacher data may include biometric data in addition to environmental data, and in that case, the teacher data is data including a correspondence relationship between the time change of the environmental data and the time change of the emotion data, and a correspondence relationship between the time change of the biometric data and the time change of the emotion data. The teacher data is data generated by measuring these correspondence relationships for, for example, 80 hours.

[0040] The teacher data generated using the emotion input unit 11, the biosensor unit 12, and the environmental sensor unit 13 is input to the model construction unit 21. The model construction unit 21 constructs a prediction model for predicting future emotions using the neural network unit 22, based on the teacher data relating to the time change of emotions of one or more subjects in step S1 (step S2).

[0041] Various deep learning models can be applied to the prediction model. In this specification, an example using a CNN-LSTM model that combines a CNN (Convolutional Neural Network) model and a LSTM (Long Short-Term Memory) model will be described. Note that even if a recurrent neural network or a neural network is used instead of the CNN-LSTM model, it is possible to predict future emotions with the same level of accuracy as the CNN-LSTM model. In addition, a CART (Classification and Regression Tree) may be used as the prediction model.

[0042] The prediction model is constructed based on training data. As described above, the prediction model is a model constructed by machine learning based on training data for, for example, 80 hours.

[0043] Thereafter, immediately before predicting the subject's future emotions (second period), the input data is measured by the environment sensor unit 13 (step S3). The input data is measured for, for example, 1 to 5 minutes. If the input data measurement time is too short, there will be insufficient data required for prediction, and if it is too long, the prediction accuracy will decrease due to overlearning. In order to predict future emotions with high accuracy, it is desirable to measure the input data for about 3 minutes. An example in which the input data is measured for 3 minutes will be described below. The input data is measured in the same laboratory as in step S1.

[0044] The input data acquired by the environmental sensor unit 13 is input to the prediction unit 23. The prediction unit 23 inputs the input data into the prediction model constructed in step S2, and predicts the future emotion of the subject who performed the measurement in step S3 (step S4). The future emotion is, for example, the emotion 15 seconds to 3 minutes after the measurement in step S3.

[0045] The subject in step S3 may be the same subject as in step S1, or may be a different subject. Note that the information processing device 1 that generates teacher data for one specific subject in step S1 and is adjusted so as to be able to predict the emotions of the specific subject with high accuracy is called a dedicated machine. Also, the information processing device 1 that generates teacher data for multiple unspecified subjects in step S1 and is adjusted so as to be able to predict the emotions of multiple unspecified subjects is called a general-purpose machine. Below, an example in which the information processing device 1 is a dedicated machine will be described.

[0046] Furthermore, the information processing device 1 according to the embodiment of the present invention can also measure emotions in a room different from the room in which the teacher data was measured.

[0047] In the following, an example of predicting emotions one minute later will be described. Note that the input data measured by the biosensor unit 12 may be input to the prediction unit 23 together with the input data acquired by the environment sensor unit 13.

[0048] As described above, the generation of teacher data in step S1 uses, for example, a plurality of biometric data of the subject measured by the biometric sensor unit 12 and a plurality of environmental data measured by the environmental sensor unit 13. Furthermore, input data for the prediction unit 23 in step S3 uses at least a plurality of environmental data measured by the environmental sensor unit 13. Furthermore, in steps S3 and S4, it is also possible to predict the subject's future emotions from only a plurality of environmental data.

[0049] In the following, an example of predicting a subject's future emotion based only on a plurality of environmental data will be described. In addition to data measured by the environmental sensor unit 13, data measured by the biosensor unit 12 or other sensors (for example, a distance sensor that measures the distance between subjects) may be used to generate the teacher data.

[0050] 5A and 5B are block diagrams showing the procedure of prediction calculation by the CNN-LSTM model. FIG. 5A shows the processing of the CNN model. First, input data is input to the prediction unit 23 (step S11). The input data includes multiple environmental data. The input data may also include multiple biological data, or may only include multiple environmental data. In this specification, each of the environmental data and biological data contained in the input data is also called a parameter. The input data has at least n parameters corresponding to the n environmental data obtained from the environmental sensor unit 13.

[0051] The parameters of the input data include, for example, temperature and blue light as shown in Figs. 3A to 3D. As described in Figs. 3A to 3D, it is desirable that the data input to the model construction unit 21 and the prediction unit 23 have many parameters. It is also desirable that each parameter is weighted according to the strength of the influence it has on the emotion data. Note that, when there are many parameters, they may be folded into a small number of parameters.

[0052] The parameters are measured u times at a predetermined time interval during the input data measurement period (step S3 in FIG. 4). That is, n×u data is input to the prediction unit 23. The number of parameters is also called the number of dimensions. In addition, n parameters are also called n-dimensional parameters. That is, the input data is represented by n-dimensional parameters that change over time.

[0053] Next, the parameters are convoluted (step S12). The prediction unit 23 filters the input parameters by the convolution. As a result, features are extracted. In step S12, the n-dimensional parameters are reduced to, for example, n-k1 dimensions.

[0054] The parameters reduced in step S12 are further convoluted (step S13). In step S13, the same convolution as in step S12 is performed, and the n-k1-dimensional parameters are reduced to n-k1-k2 dimensions. This allows more features to be extracted.

[0055] As shown in steps S12 and S13, in the CNN model, the number of dimensions is compressed stepwise to generate feature data. Although Fig. 5A shows an example of parameter compression in two steps, the present invention is not limited to this, and parameter reduction in three or more steps may be performed. Moreover, parameter compression may be performed in only one step.

[0056] Next, parameter pooling is performed (step S14). The prediction unit 23 aggregates the convoluted parameters by pooling, and extracts parameters that have a greater effect on emotions as feature data. In step S14, n-k1-k2 parameters are compressed to, for example, 1 parameter. Note that, although FIG. 5A shows an example in which n×u data input in step S11 is compressed to 1×u data, the present invention is not limited to this, and the CNN model can extract feature data with fewer dimensions than n.

[0057] Fig. 5B shows the process of the LSTM model. Following Fig. 5A, feature data extracted by the CNN model is acquired as time-series data (step S21). The prediction unit 23 inputs each data of this time-series data in chronological order to the LSTM model (step S22). Using the LSTM model, the prediction unit 23 calculates and outputs one piece of emotion data (step S23).

[0058] Fig. 6 is a block diagram showing an LSTM model. Input data 31, hidden state data 32a, and cell state data 33a are input to the LSTM model in Fig. 6. The LSTM model outputs hidden state data 32b, cell state data 33b, and output data 34. The LSTM model also has sigmoid functions 41a, 41b, and 41c, hyperbolic tangent functions 42a and 42b, multipliers 43a, 43b, and 43c, and an adder 44.

[0059] For example, feature amount data at time t from among the features extracted in step S13 of Fig. 5A is input to the input data (Xt) 31. For example, an emotion predicted from the feature amount data at time t is output from the output data (ht) 34. This emotion data is also output as hidden state data (ht) 32b. An emotion predicted from the feature amount data at time t-1, which is before time t, is input to the hidden state data (ht-1) 32a.

[0060] The LSTM model also stores the output results in a predetermined storage cell for a long term. The LSTM model outputs a value calculated based on the feature amount data at time t as cell state data (Ct) 33b for long term storage. The LSTM model also receives a value calculated based on the feature amount data at the previous time t-1 and stored for a long term storage as cell state data (Ct-1) 33a.

[0061] The sigmoid function 41a is also called a forget gate and is used to forget a memory cell. The sigmoid function 41a calculates a forget gate activation vector ft based on the input data 31 and the hidden state data 32a, and inputs it to the multiplier 43a. The multiplier 43a multiplies the forget gate activation vector ft by the cell state data 33a.

[0062] The sigmoid function 41b is also called an input gate and is used to input a new memory to the memory cell. The sigmoid function 41b calculates an input gate activation vector it based on the input data 31 and the hidden state data 32a, and inputs it to the multiplier 43b. The input data 31 and the hidden state data 32a normalized by the hyperbolic tangent function 42a are also input to the multiplier 43b. The multiplier 43b multiplies these input data and outputs the calculation result to the adder 44. The adder 44 adds the output value of the multiplier 43a and the output value of the multiplier 43b. The output result of the adder 44 is output as the cell state data 33b.

[0063] The sigmoid function 41c, also called an output gate, calculates the output data of the LSTM model. The sigmoid function 41c calculates an output gate activation vector Ot based on the input data 31 and the hidden state data 32a, and inputs it to the multiplier 43c. The multiplier 43c also receives the output result of the adder 44 normalized by the hyperbolic tangent function 42b. The multiplier 43c multiplies these input data and outputs them as the output data 34 and the hidden state data 32b.

[0064] The LSTM model has a feature that time series data at a specific time is input, and output results from times before the specific time are recursively input. That is, the next output result is calculated based on past output results. In addition, since the LSTM model has a long-term memory that retains output results for a long period of time, long-term fluctuations in the output result are reflected in the next output result.

[0065] The LSTM model according to the embodiment of the present invention outputs emotion data in chronological order. The LSTM model also receives feature data (i.e., compressed input data) in chronological order, as well as emotion data previously output. That is, the LSTM model predicts future emotions based on a plurality of environmental data at a predetermined time and past emotion data up to the predetermined time. Even when exposed to the same environmental stimuli, the LSTM model can predict emotion fluctuations based on fluctuations in past emotion data, such as a subject who was previously in a Relaxed state changing to a Happy state, and a subject who was previously in a Sad state changing to a Stress state.

[0066] An example of the training data input to the model construction unit 21 is shown in Table 1.

[0067] [Table 1]

[0068] In the teacher data of Table 1, environmental data such as the temperature, blue light, and sound volume in a room where multiple subjects are placed are stored in chronological order from the environmental sensor unit 13. In addition, the emotions of the subjects are input from the emotion input unit 11 and linked to the environmental data in chronological order. When there are multiple subjects, the teacher data stores the emotions of the multiple subjects for each time period in Table 1. For example, 80 hours' worth of teacher data is input.

[0069] The input data input to the prediction unit 23 is shown in Table 2.

[0070] [Table 2]

[0071] The input data in Table 2 has parameters in common with the environmental data stored in the training data. For example, the training data in Table 1 and the input data in Table 2 store the parameters of temperature, blue light, and volume in common.

[0072] Although Tables 1 and 2 show data at one-minute intervals, in reality, environmental data (and biological data) are measured at intervals of, for example, 10 seconds. The measurement interval is determined depending on the environment in which the subject is located and the time to be predicted, for example between 10 seconds and 1 minute. In situations such as in-vehicle environments where even a brief lapse in concentration can lead to fatal results, measurements at 10-second intervals are required. On the other hand, in indoor environments where real-time performance is not as important as in-vehicle environments, measurement intervals of about 1 minute may be sufficient.

[0073] The output data output by the prediction unit 23 is shown in Table 3.

[0074] [Table 3]

[0075] Table 3 shows the data one minute after the input data in Table 2.

[0076] The prediction unit 23 predicts the emotion one minute after the measurement period (second period) from the data input to the prediction model. As shown in FIG. 6, the prediction unit 23 inputs the input data in chronological order and outputs the predicted emotion in chronological order. The prediction unit 23 searches the teacher data for data showing the same or similar time change as the input data (Table 2), and outputs the emotion one minute after the teacher data as the emotion one minute after the measurement period (second period) assuming that the emotion in the measurement period (second period) also changes in the same way as the emotion data corresponding to the teacher data searched for (Table 3). That is, the prediction model can output the emotion at the predicted time based on the emotion change in the teacher data that has the same or similar tendency as the time change of the multiple environmental data acquired by the acquisition unit 2.

[0077] Specifically, the transition of the environmental input data in the input data in Table 2 is similar to the transition of the environmental input data from 16:31 to 16:33 in the teacher data in Table 1. In this case, the emotion data predicted by the prediction unit 23 is output in chronological order with a transition similar to the emotion from 16:32 to 16:34 in the teacher data in Table 1.

[0078] The prediction unit 23 ultimately outputs emotion Sad, which is the same as the data at 16:34 in the teacher data, as shown in Table 3, based on the teacher data in Table 1 and the input data in Table 2.

[0079] The inventor confirmed through an experiment that the more parameters are input, the more accurate the prediction by the prediction unit 23 is. First, the inventor prepared a dedicated machine for each of the four subjects, and generated teacher data based on the pulse rate, skin temperature, CO2 concentration, illuminance (visible light), and temperature measured during a 72-hour learning period (first period). In addition, environmental data of the CO2 concentration, illuminance (visible light), and temperature measured during a three-minute measurement period (second period) were input to the prediction unit 23, and an experiment was conducted to predict the emotions one minute later. As a result, the accuracy of emotion prediction by the prediction unit 23 was 60%, 66%, 63%, and 85% for each subject.

[0080] The inventor also generated teacher data based on the pulse rate, skin temperature, CO2 concentration, illuminance (visible light), temperature, blue light, infrared (distance), human presence, volume, odor intensity, air pressure, and particulate concentration measured during the same first period as above, and inputted environmental data of CO2 concentration, illuminance (visible light), temperature, blue light, infrared (distance), human presence, volume, odor intensity, air pressure, and particulate concentration measured during the same second period as above, and performed another experiment to predict emotions one minute later. As a result, the accuracy of emotion prediction by the prediction unit 23 was 78%, 81%, 80%, and 86% for each subject, and the accuracy was improved for all subjects when the number of types of study data was larger.

[0081] Next, the inventor conducted an experiment on two subjects, inputting two types of biological data, namely skin temperature and heart rate during a three-minute measurement period, into the prediction unit 23 to predict the emotions one minute later. Table 4 shows the results of predictions made using skin temperature and heart rate as input data.

[0082] [Table 4]

[0083] Table 4 shows the Happy, Relaxed, Stressed, and Sad ratings for subjects 1 and 2 at a given experimental time. Table 4 shows that the accuracy of prediction for subject 1 is 75.6%. For subject 2, the accuracy is 91.6%.

[0084] Table 5 shows the prediction results when environmental data parameters are added to biometric data as input data.

[0085] [Table 5]

[0086] In Table 5, predictions are made using skin temperature, visible light, odor, humidity, blue light, and dust as input data. In this case, the prediction accuracy for subject 1 is 81.1%. Furthermore, the prediction accuracy for subject 2 is 94.4%. Table 4 shows that by using biological data that directly affect emotions as input data, a certain degree of accuracy can be achieved even with as few as two parameters, but as mentioned above, Table 5, which adds multiple environmental data to the input data, has even higher accuracy than Table 4.

[0087] Furthermore, the inventors confirmed the accuracy of emotion prediction for three subjects when parameters were added in order of importance in the emotion data. Table 6 shows the importance of each parameter in the emotion data. The importance indicates the degree of contribution to emotion prediction. The sum of the importance of all parameters was adjusted to be 100%.

[0088] [Table 6]

[0089] Figure 7 is a graph showing the emotion prediction accuracy for each number of sensors for three subjects. The horizontal axis shows the number of parameters, and the vertical axis shows the emotion prediction accuracy. Figure 7 also shows three line graphs corresponding to the three subjects.

[0090] The parameters are added in descending order in Table 6. For example, when the number of parameters is three, the parameters of CO2 concentration, temperature, and odor intensity are measured and emotion prediction is performed. As shown in Figure 7, the accuracy of emotion prediction increases as the number of input parameters increases. In particular, the accuracy of emotion prediction increases monotonically for all subjects up to four parameters. Furthermore, when the number of parameters is five or more, the accuracy of emotion prediction increases to approximately 80%.

[0091] As can be seen from these experimental results, the greater the number of input parameters, or more specifically, the greater the types of environmental data measured during the measurement period in step S3 of FIG. 5, the more accurate the emotion prediction by prediction unit 23 can be.

[0092] As described above, the embodiment of the present invention successfully predicts emotion data by analyzing environmental data that influences biometric data in a time series, using environmental data that does not show any correlation with emotion data in individual data. As a result, the embodiment of the present invention can predict emotion non-invasively, objectively, and with high accuracy by analyzing environmental data that influences biometric data in a time series, without using invasive biometric data that is directly linked to emotion.

[0093] In this way, the information processing device 1 according to the embodiment of the present invention can predict the future (for example, one minute later) emotions of the subject based on the time change of the environment by using at least environmental data as input data. In addition, by increasing the number of dimensions of the input parameters (input data) (i.e., the type of environmental data measured within the measurement period), the prediction accuracy of the future emotions can be improved. Furthermore, in the information processing device 1 according to the embodiment of the present invention, a prediction model for predicting future emotions by compressing the number of dimensions of multidimensional parameters by machine learning is constructed, so that future emotions can be predicted easily and accurately by using the constructed prediction model. In addition, just before predicting future emotions, data on time changes in the environment for a short period of time is acquired to predict future emotions, so future emotions can be predicted accurately with data acquired for a short period of time. In addition, in the embodiment of the present invention, future emotions can be predicted accurately using only environmental data without using biological data that is directly correlated with emotional data. In this case, as described above, there may be cases where the correlation with emotional data cannot be specified using only one environmental data, so it is necessary to prepare a model that has previously learned the correlation between multiple environmental data and emotional data, and input multiple environmental data into this model to predict emotional data.

[0094] At least a part of the information processing device 1 described in the above embodiment may be configured with hardware or software. When configured with software, a program that realizes at least a part of the functions of the information processing device 1 may be stored in a recording medium such as a flexible disk or a CD-ROM, and may be read and executed by a computer. The recording medium is not limited to a removable one such as a magnetic disk or an optical disk, and may be a fixed recording medium such as a hard disk device or a memory.

[0095] Also, a program that realizes at least a part of the functions of the information processing device 1 may be distributed via a communication line (including wireless communication) such as the Internet. Furthermore, the program may be encrypted, modulated, or compressed and distributed via a wired line or wireless line such as the Internet, or stored in a recording medium.

[0096] The embodiment of the present invention can be used for various purposes. For example, it is possible to predict future changes in emotional states based on the environmental conditions to which a person is currently exposed. The embodiment of the present invention is a technology that can be expected to be applied to various fields, such as improving mental health by controlling the living environment, preventive medicine, supporting safe driving of automobiles, preventing accidents in factories, improving work efficiency in offices, and improving the effectiveness of learning in schools.

[0097] (Additional Note) The present technology can be configured as follows. [Item 1] a learning unit that constructs, by machine learning, a prediction model that predicts emotions at a future prediction time based on teacher data that indicates a correspondence relationship between at least a plurality of environmental changes over time and emotion changes over time within a first time period; an acquisition unit that acquires data of time changes of at least the plurality of environments within a second period immediately before the predicted time; a prediction unit that inputs data on at least the plurality of environmental changes over time acquired by the acquisition unit into the prediction model and predicts an emotion at the prediction time, Information processing device. [Item 2] In the prediction unit, only data of time changes of the plurality of environments is input to the prediction model. Item 1. An information processing device according to item 1. [Item 3] In the learning unit, the teacher data is subjected to an oversampling process for equalizing the occurrence frequency of the emotions. 3. The information processing device according to item 1 or 2. [Item 4] constructing a prediction model by machine learning to predict emotions at a future prediction time based on training data indicating correspondence relationships between at least a plurality of environmental changes over time and emotion changes over time within a first time period; acquiring data of time changes of at least the plurality of environments within a second period immediately preceding the predicted time; and inputting at least the data of the plurality of environmental changes over time acquired by the acquisition unit into the prediction model to predict emotions at the prediction time. Information processing methods. [Explanation of symbols]

[0098] 1 Information processing device, 2 Acquisition unit, 3 Information processing unit, 11 Emotion input unit, 12 Biosensor unit, 13 Environmental sensor unit, 21 Model construction unit, 22 Neural network unit, 23 Prediction unit, 24 Learning unit, 31 Input data, 32a, 32b State data, 33a, 33b Cell state data, 34 Output data, 41a, 41b, 41c Sigmoid function, 42a, 42b Hyperbolic tangent function, 43a, 43b, 43c Multiplier, 44 Adder

Claims

1. a learning unit that constructs a prediction model by machine learning to predict emotions at a future prediction time based on teacher data indicating correspondence relationships between at least a plurality of environmental changes over time and emotion changes over time within a first time period; an acquisition unit that acquires data of time changes of at least the plurality of environments within a second period immediately before the predicted time; a prediction unit that inputs data on at least the plurality of environmental changes over time acquired by the acquisition unit into the prediction model and predicts emotions at the prediction time, Information processing device.

2. In the prediction unit, only data of time changes of the plurality of environments is input to the prediction model. The information processing device according to claim 1 .

3. The learning unit performs an oversampling process to equalize the frequency of occurrence of the emotions in the training data. The information processing device according to claim 1 .

4. constructing a prediction model by machine learning to predict emotions at a future prediction time based on teacher data indicating a correspondence relationship between at least a plurality of environmental changes over time and emotion changes over time within a first time period; acquiring data of time changes of at least the plurality of environments during a second period immediately preceding the predicted time; and inputting at least the data of the plurality of environmental changes over time acquired by the acquisition unit into the prediction model to predict emotions at the prediction time. Information processing methods.

Images