Information processing system and information processing method

Abstract

An information processing system according to one aspect of the present disclosure comprises one or more processing units. The one or more processing units can derive a correction coefficient on the basis of the transition of context included in first time-series data of the context of a biological subject. The one or more processing units can calculate a normalized feature by normalizing a feature obtained from second time-series data of biological information of the biological subject corresponding to the first time-series data on the basis of the correction coefficient, a first baseline feature set in advance for the pre-transition context, and a second baseline feature set in advance for the post-transition context.

Description

Information processing system and information processing method

[0001] The present disclosure relates to an information processing system and an information processing method.

[0002] When a person's emotions change, physiological responses such as brain waves, heart rate, and sweating are expressed on the body surface. These physiological responses can be read as biosignals using a sensor device, making it possible to estimate a person's emotions. For example, by performing predetermined signal processing on the read biosignals to obtain feature quantities such as physiological indices that contribute to emotional responses, the feature quantities can be input into a model formula calculated by machine learning, thereby enabling estimation of a person's emotions (see, for example, Patent Documents 1 and 2).

[0003] International Publication WO2012 / 056546 JP 2016-106689 A

[0004] However, when constructing a physiologically valid emotion estimation model in a laboratory environment and applying the constructed model to a real environment, context dependency, in which a person's physiological response to emotion depends on the context (the context of the person's environment and behavior), affects the emotion estimation model. In particular, because people exercise in their daily lives, it is difficult to distinguish whether the accompanying changes in arousal level are due to exercise or mental changes. In particular, when recovering immediately after exercise, even if a person is resting, the effects of exercise make it difficult to accurately estimate changes in arousal level. Therefore, it is desirable to provide an information processing system and information processing method that can reduce context dependency and accurately estimate arousal levels.

[0005] An information processing system according to one aspect of the present disclosure includes a data acquisition unit and one or more processing units. The data acquisition unit is capable of acquiring first time-series data of a context of a target living organism and second time-series data of biometric information of the target living organism corresponding to the first time-series data. The one or more processing units are capable of performing processing using the first time-series data and the second time-series data. The one or more processing units are capable of performing a correction coefficient derivation process that derives a correction coefficient based on a context transition included in the first time-series data. The one or more processing units are capable of performing a feature normalization process that calculates normalized features by normalizing features obtained from the second time-series data based on the correction coefficient, a first baseline feature that is preset for the context before the transition, and a second baseline feature that is preset for the context after the transition. The one or more processing units are capable of performing an emotion estimation process that estimates an emotion of the target living organism based on the normalized feature.

[0006] An information processing method according to an aspect of the present disclosure includes the following four steps: (1) acquiring first time-series data of a context of a target living organism and second time-series data of biometric information of the target living organism corresponding to the first time-series data; (2) deriving a correction coefficient based on a transition of a context included in the first time-series data; (3) calculating normalized features by normalizing features obtained from the second time-series data based on the correction coefficient, a first baseline feature set in advance for the context before the transition, and a second baseline feature set in advance for the context after the transition; and (4) estimating emotions of the target living organism based on the normalized features.

[0007] FIG. 1 is a diagram illustrating an example of a schematic configuration of an information processing system according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of a data table of baseline features. FIG. 3 is a diagram illustrating an example of a data table of transition parameters. FIG. 4 is a diagram illustrating an example of a data table of correction gains. FIG. 5 is a diagram illustrating an example of the correction gains of FIG. 4. FIG. 6 is a diagram illustrating an example of transition of features when a context transitions. FIG. 7 is a diagram illustrating an example of a procedure for deriving baseline features. FIG. 8 is a diagram illustrating an example of a procedure for normalizing features. FIG. 9 is a diagram illustrating a specific example of the information processing system of FIG. 1. FIG. 10 is a diagram illustrating an example of a schematic configuration of the wristwatch of FIG. 9. FIG. 11 is a diagram illustrating an example of a schematic configuration of the mobile terminal of FIG. 9. FIG. 12 is a diagram illustrating a specific example of the information processing system of FIG. 1. FIG. 13 is a diagram illustrating an example of a schematic configuration of the mobile terminal of FIG. 12. FIG. 14 is a diagram illustrating an example of a schematic configuration of the server device of FIG. 12. FIG. 15 is a diagram illustrating a modified example of the schematic configuration of a portion of the information processing system.

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0009] <1. About Arousal Level> A person's arousal level is closely related to their ability to concentrate. When a person is concentrating, they have a high level of interest in the object of their concentration. Therefore, by knowing a person's arousal level, it is possible to estimate the person's objective level of interest (emotion). A person's arousal level can be derived based on biometric information obtained from, for example, a person working in an office environment or practicing yoga (hereinafter referred to as the "target organism"). Examples of biometric information from which the arousal level of the target organism can be derived include electroencephalograms, sweating, pulse waves, electrocardiograms, blood flow, skin temperature, facial myoelectric potentials, electrooculography, or information on specific components contained in saliva. Below, an embodiment of an information processing system that estimates emotions using biometric information is described.

[0010] 2. Embodiment [Configuration] An information processing system 1 according to an embodiment of the present disclosure will be described. FIG. 1 shows a schematic configuration example of the information processing system 1. The information processing system 1 is a system capable of estimating the emotions of a target living organism based on biometric information obtained from the target living organism. In this embodiment, the target living organism is a human. Note that in the information processing system 1, the target living organism is not limited to a human.

[0011] The information processing system 1 may be realized by a single electronic device, or may be realized by multiple electronic devices capable of transmitting and receiving data to and from each other via a communication network. When the information processing system 1 is configured as in-ear headphones or a headband, the biometric information is obtained, for example, from the ear. When the information processing system 1 is configured as VR (Virtual Reality) goggles, the biometric information is obtained, for example, from the forehead. When the information processing system 1 is configured as a band, the biometric information is obtained, for example, from the arm or leg to which the band is attached. The information processing system 1 may include a sensor that acquires the biometric information, or may include an acquisition unit that acquires a biometric signal obtained by the sensor that acquires the biometric information.

[0012] The information processing system 1 includes, for example, a data acquisition unit 10, a preprocessing unit 20, a feature extraction unit 30, a context discrimination unit 40, a baseline calculation unit 50, a correction coefficient derivation unit 60, a feature normalization unit 70, and an emotion estimation unit 80. The information processing system 1 further includes, for example, a baseline feature table 51, a normalization parameter table 71, and a model parameter table 81. The data acquisition unit 10, the preprocessing unit 20, the feature extraction unit 30, the context discrimination unit 40, the baseline calculation unit 50, the correction coefficient derivation unit 60, the feature normalization unit 70, and the emotion estimation unit 80 are configured, for example, by one or more arithmetic processing chips. The arithmetic processing chip includes, for example, at least one of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and a GPU (Graphics Processing Unit). The baseline feature table 51, the normalization parameter table 71, and the model parameter table 81 are configured, for example, by non-volatile memory, such as an EEPROM (Electrically Erasable Programmable Read-Only Memory), a flash memory, or a resistance change memory.

[0013] The data acquisition unit 10 is capable of acquiring biological information (biological signal Da) detected by various sensor units. The "various sensor units" may be, for example, sensors that contact the target living body or sensors that do not contact the target living body. The "various sensor units" are, for example, sensors that can detect information on at least one of electroencephalograms, sweating, pulse waves, electrocardiograms, blood flow, skin temperature, facial myoelectric potentials, electrooculography, and specific components contained in saliva. The biological signal Da is time-series data (second time-series data) output from the "various sensor units." The "various sensor units" are capable of outputting the detected biological signal Da to the data acquisition unit 10. The data acquisition unit 10 is capable of outputting the acquired biological signal Da to the preprocessing unit 20.

[0014] The data acquisition unit 10 is capable of acquiring information (context information Db) suggesting the context of the target living body. The context information Db is, for example, time-series data (first time-series data) about at least one of information obtained by an inertial sensor (e.g., translational motion information, rotational motion information), an audio signal obtained by a microphone, location information obtained by a GPS (Global Positioning System), and usage history information of a mobile device. The data acquisition unit 10 is capable of acquiring, for example, input data Din including a biosignal Da and the context information Db. The data acquisition unit 10 is capable of outputting the acquired context information Db to the context determination unit 40.

[0015] The pre-processing unit 20 is capable of performing pre-processing such as band-pass filtering and noise removal on the biosignal Da acquired from the data acquisition unit 10. The pre-processing unit 22 is capable of outputting the biosignal Da′ obtained by the pre-processing to the feature extraction unit 30.

[0016] The feature extraction unit 30 is capable of extracting feature quantities X(t) as model input variables for estimating emotions from the biosignal Da′ acquired from the preprocessing unit 20. The feature quantities X(t) are time-series data. The feature extraction unit 30 is capable of outputting the extracted feature quantities X(t) to the feature quantity normalization unit 70 and the baseline calculation unit 50. The feature quantities X(t) are not limited to physiologically known feature quantities. The feature extraction unit 30 may be configured to perform signal processing to extract feature quantities that contribute to emotions in a data-driven manner, for example, by deep learning, an autoencoder, or the like.

[0017] The context determination unit 40 is capable of determining the context of the target living organism based on the context information Db acquired from the data acquisition unit 10. The context determination unit 40 is capable of outputting the context (context Dc) of the target living organism obtained by the determination to the baseline calculation unit 50 and the correction coefficient derivation unit 60. The context determination unit 40 is capable of determining the behavior and state of the target living organism based on information about the position and movement of the target living organism included in the context information Db. The context determination unit 40 is capable of determining, for example, whether the target living organism is resting or exercising, or whether the target living organism has transitioned from exercising to resting, based on the information about the position and movement of the target living organism included in the context information Db. The context determination unit 40 may be capable of determining the behavior and state of the target living organism by also taking into account time information included in the context information Db. The context determination unit 40 may be capable of determining whether the target living organism is sleeping by also taking into account time information included in the context information Db.

[0018] The baseline calculation unit 50 is capable of calculating a baseline feature Xbase corresponding to the context Dc acquired from the context determination unit 40, based on the feature X(t) acquired from the feature extraction unit 30. The baseline calculation unit 50 is capable of setting, for example, a representative value (e.g., average value, mode, or median) of the feature X(t) obtained during the period up to the transition of the context Dc acquired from the context determination unit 40 as the baseline feature Xbase.

[0019] Each time the context Dc acquired from the context determination unit 40 transitions, the baseline calculation unit 50 is capable of calculating a baseline feature Xbase corresponding to the context Dc before the transition (immediately before the transition). The baseline calculation unit 50 is capable of storing the calculated baseline feature Xbase in a baseline feature data table 51 in association with the context Dc. In the baseline feature data table 51, for example, as shown in FIG. 2 , a baseline feature Xbase is associated with each context Dc. FIG. 2 illustrates an example in which resting, walking, running, and sleeping are stored as contexts Dc, and a baseline feature XbaseB at rest, a baseline feature XbaseA at walking, a baseline feature XbaseD at running, and a baseline feature XbaseE at sleeping are stored as baseline feature Xbases.

[0020] Here, suppose that when the baseline calculation unit 50 attempts to store a baseline feature Xbase corresponding to a certain context Dc in the baseline feature data table 51, a baseline feature Xbase corresponding to the same context Dc is already stored in the baseline feature data table 51. In this case, the baseline calculation unit 50 may update the baseline feature Xbase by, for example, performing a weighted addition of the baseline feature Xbase already stored in the baseline feature data table 51 and the baseline feature Xbase that is to be stored.

[0021] Furthermore, suppose that when the baseline calculation unit 50 attempts to store a baseline feature Xbase corresponding to a certain context Dc in the baseline feature data table 51, no baseline feature Xbase corresponding to the same context Dc is stored in the baseline feature data table 51. In this case, the baseline calculation unit 50 may, for example, newly register the calculated baseline feature Xbase in the baseline feature data table 51.

[0022] The correction coefficient derivation unit 60 is capable of deriving a correction coefficient for taking into account the motor influence of the physiological response at the time of the transition of the context Dc when normalizing the feature quantity X(t). As shown in FIG. 1, the correction coefficient derivation unit 60 has, for example, a transition parameter selection unit 61, a correction gain selection unit 62, a transition parameter table 63, and a correction gain table 64. The transition parameter table 63 and the correction gain table 64 are configured, for example, by nonvolatile memory, such as an EEPROM, a flash memory, or a resistive memory.

[0023] In the transition parameter table 63, for example, as shown in Fig. 3, the time constant (transition parameter τ) of the recovery transition of the physiological response when the context Dc transitions is defined for each type of transition of the context Dc. Fig. 3 illustrates an example in which the following transition parameters τ are defined: Transition parameter τ when the context Dc transitions from resting to walking １２ The transition parameter τ when the context Dc transitions from resting to running １３ The transition parameter τ when the context Dc transitions from rest to sleep １４ The transition parameter τ when the context Dc transitions from walking to resting ２１ The transition parameter τ when the context Dc transitions from walking to running ２３ The transition parameter τ when the context Dc transitions from walking to sleeping ２４ The transition parameter τ when the context Dc transitions from running to resting ３１ The transition parameter τ when the context Dc transitions from running to walking ３２ The transition parameter τ when the context Dc transitions from running to sleeping ３４ The transition parameter τ when the context Dc transitions from sleep to rest ４１ The transition parameter τ when the context Dc transitions from sleep to walking ４２ The transition parameter τ when the context Dc transitions from sleep to running ４３

[0024] In the correction gain data table 64, for example, as shown in FIG. 4, a correction gain g that defines the degree of recovery of the sensitivity of the physiological response according to the elapsed time of recovery of the physiological response at the time of transition of the context Dc is defined for each type of transition of the context Dc. For example, as shown in FIG. 5, the correction gain g is a function that decreases linearly over time, becomes 1.0 when the physiological response is completely recovered, and becomes zero thereafter. The function of the correction gain g differs depending on the type of transition of the context Dc. FIG. 4 illustrates an example in which the following correction gains g are defined. Correction gain g when the context Dc transitions from resting to walking １２ Correction gain g when the context Dc transitions from resting to running １３ Correction gain g when the context Dc transitions from rest to sleep １４ Correction gain g when the context Dc transitions from walking to resting ２１ Correction gain g when the context Dc transitions from walking to running ２３ Correction gain g when the context Dc transitions from walking to sleeping ２４ Correction gain g when the context Dc transitions from running to resting ３１ Correction gain g when the context Dc transitions from running to walking ３２ Correction gain g when the context Dc transitions from running to sleeping ３４ Correction gain g when the context Dc transitions from sleep to rest ４１ Correction gain g when the context Dc transitions from sleep to walking ４２ Correction gain g when the context Dc transitions from sleep to running ４３

[0025] When the context Dc input from the context determination unit 40 transitions, the transition parameter selection unit 61 can obtain a transition parameter τ corresponding to the transition from the transition parameter data table 63. Upon obtaining the transition parameter τ from the transition parameter data table 63, the transition parameter selection unit 61 can output the obtained transition parameter τ to the feature normalization unit 70. When the context Dc input from the context determination unit 40 transitions, the correction gain selection unit 62 can obtain a correction gain g corresponding to the transition from the correction gain data table 64. Upon obtaining the correction gain g from the correction gain data table 64, the correction gain selection unit 62 can output the obtained correction gain g to the feature normalization unit 70.

[0026] The feature normalization unit 70 is capable of calculating the normalized feature Xnorm(t) by normalizing the feature X(t) based on the correction coefficient (transition parameter τ), the baseline feature Xbase that is set in advance for the context Dc before the transition, and the baseline feature Xbase that is set in advance for the context Dc after the transition.

[0027] 6 shows an example of the transition of the feature quantity X(t) when the context Dc transitions. FIG. 6 shows an example of the transition of the feature quantity X(t) when the context Dc transitions from context A (walking) to context B (resting) or context C (reading). The feature quantity X(t) when the context A (walking) transitions to context B (resting) corresponds to the dashed-dotted line graph in the figure and is composed of a motor arousal recovery component. Meanwhile, the feature quantity X(t) when the context A (walking) transitions to context C (reading) corresponds to the dashed-dotted line graph in the figure and is composed of a motor arousal recovery component and a mental arousal component. Therefore, the difference between the feature value X(t) when there is a transition from context A (walking) to context C (reading) and the feature value X(t) when there is a transition from context A (walking) to context B (resting) is the component of mental arousal.

[0028] Immediately after the transition from context A (walking) to context B (resting), the mental arousal response is sluggish. Therefore, the dashed-dotted line graph in the figure roughly overlaps with the dashed-dotted line graph in the figure. However, as time passes after the transition from context A (walking) to context B (resting), the difference between the dashed-dotted line graph and the dashed-dotted line graph in the figure widens. This suggests that the sensitivity of the physiological response associated with emotional changes recovers over time in response to physiological recovery from the exercise state. For example, the feature normalization unit 70 estimates the progression of recovery of the physiological response immediately after exercise and removes the exercise arousal component from the feature immediately before exercise, thereby correcting the sensitivity of the physiological response associated with emotional changes.

[0029] The feature normalization unit 70 calculates the difference ΔX by subtracting a baseline feature XbaseA that is preset for the context A (walking) before the transition from a baseline feature XbaseB that is preset for the context B (resting) after the transition (Equation (1)). The feature normalization unit 70 calculates the difference ΔX by subtracting the baseline feature XbaseA, the baseline feature XbaseB that is preset for the context B (resting) after the transition, from the difference ΔX, the baseline feature XbaseA, and a transition parameter τ ２１ and calculate the recovery transition estimation result Xpa(t) (Equation (2)). The recovery transition estimation result Xpa(t) corresponds to the transition of the baseline feature immediately after the context transition. The feature normalization unit 70 normalizes the feature X(t) obtained after the transition, for example, by subtracting the recovery transition estimation result Xpa(t) from the feature X(t) obtained after the transition, thereby obtaining the normalized feature Xnorm(t) (Equation (3)). This makes it possible to obtain a feature (normalized feature Xnorm(t)) from which the motor arousal component has been removed.

[0030] ΔX=XbaseB−XbaseA…(1) Xpa(t)=XbaseA+ΔX・exp(−t / τ ２１ )...(2) Xnorm(t)=X(t)-Xpa(t)...(3)

[0031] The feature normalization unit 70 calculates corrected normalized feature g·Xnorm(t) by multiplying the time-series feature (x1, x2, ..., xk) within the sliding window, out of the normalized feature Xnorm(t), by a correction coefficient (correction gain g). The feature normalization unit 70 is capable of outputting the calculated corrected normalized feature g·Xnorm(t) to the emotion estimation unit 80.

[0032] The emotion estimation unit 80 is capable of analyzing the data pattern of the corrected normalized feature g·Xnorm(t) and estimating the user's emotion. The emotion estimation unit 80 is capable of outputting an estimation result of the user's emotion (emotion estimation result). The emotion estimation unit 80 has a pre-constructed reference model and is capable of reading various model parameters used in the reference model from a model parameter table 81. The model parameter table 81 stores various model parameters used in the reference model.

[0033] The emotion estimation unit 80 is capable of inputting the time-series feature quantities (g·x1, g·x2, ..., g·xk) within the sliding window, out of the corrected normalized feature quantity g·Xnorm(t), to a reference model. By inputting the feature quantities (g·x1, g·x2, ..., g·xk) to the reference model, the emotion estimation unit 80 is capable of obtaining predicted labels (y1, y2, ..., yk) of the time-series emotional states and their reliability from the reference model. The emotion estimation unit 80 is capable of obtaining the reliability of the representative value of the predicted labels (y1, y2, ..., yk) by weighting and adding the predicted labels (y1, y2, ..., yk) of the time-series emotional states by the reliability. The emotion estimation unit 80 performs threshold processing on the reliability of the representative value of the predicted labels (y1, y2, ..., yk), thereby making it possible to obtain the representative value z of the predicted labels (y1, y2, ..., yk) as the emotion estimation result.

[0034] Next, a procedure for deriving the baseline feature Xbase will be described. Fig. 7 shows an example of the procedure for deriving the baseline feature Xbase.

[0035] First, the baseline calculation unit 50 acquires context information Db from the context determination unit 40 (step S101). Next, the baseline calculation unit 50 determines a context Dc of the target organism based on the acquired context information Db (step S102). Next, the baseline calculation unit 50 acquires a feature X(t) corresponding to the context Dc obtained by the determination from the feature extraction unit 30 and buffers it (step S103).

[0036] Next, the baseline calculation unit 50 determines whether or not a transition of the context Dc has occurred in the time-series data of the context Dc obtained by the determination based on the acquired context information Db (step S104). If there is no transition of the context Dc (step S104; N), the baseline calculation unit 50 returns to step S103 and continues buffering. On the other hand, if there is a transition of the context Dc (step S104; Y), the baseline calculation unit 50 stops buffering the feature X(t) of the context Dc immediately before the transition (step S105). At this time, the baseline calculation unit 50 starts buffering the feature X(t) corresponding to the context Dc after the transition.

[0037] The baseline calculation unit 50 calculates a baseline feature Xbase in the context Dc immediately before the transition (step S106). The baseline calculation unit 50 calculates a representative value (e.g., an average value, a mode value, or a median value) of the time-series data of the feature X(t) obtained by buffering, for example.

[0038] Next, the baseline calculation unit 50 determines whether the baseline feature Xbase corresponding to the context Dc immediately before the transition is stored in the baseline feature data table 51 (storage unit) (step S107). If the baseline feature Xbase corresponding to the context Dc immediately before the transition is stored in the baseline feature data table 51 (storage unit), the baseline calculation unit 50 stores the calculated baseline feature Xbase in the baseline feature data table 51 and updates the baseline feature Xbase corresponding to the context Dc immediately before the transition (step S108). Then, the baseline calculation unit 50 outputs the calculated (updated) baseline feature Xbase to the feature normalization unit 70 (step S109). On the other hand, if the baseline feature Xbase corresponding to the context Dc immediately before the transition is not stored in the baseline feature data table 51 (storage unit), the baseline calculation unit 50 stores the calculated baseline feature Xbase in the baseline feature data table 51 and newly registers the baseline feature Xbase corresponding to the context Dc immediately before the transition (step S110).The baseline calculation unit 50 then outputs the calculated (newly registered) baseline feature Xbase to the feature normalization unit 70 (step S109).

[0039] Next, a procedure for normalizing the feature quantity X(t) will be described. Fig. 8 shows an example of the procedure for normalizing the feature quantity X(t).

[0040] The data acquisition unit 10 acquires biological information (biological signal Da) detected by various sensors (step S201). The data acquisition unit 10 further acquires information suggesting the context of the target biological body (context information Db) (step S201). The data acquisition unit 10 outputs the acquired biological signal Da to the preprocessing unit 20. The data acquisition unit 10 outputs the acquired context information Db to the context determination unit 40.

[0041] The preprocessing unit 20 performs preprocessing such as band-pass filtering and noise removal on the biological signal Da (step S202). The preprocessing unit 20 outputs the biological signal Da' obtained by the preprocessing to the feature extraction unit 30. The context determination unit 40 determines the context of the target biological body based on the context information Db. The context determination unit 40 outputs the context of the target biological body (context Dc) obtained by the determination to the baseline calculation unit 50 and the correction coefficient derivation unit 60.

[0042] The feature extraction unit 30 extracts feature X(t) from the preprocessed biological signal Da′ as a model input variable for estimating emotion (step S203). The feature extraction unit 30 outputs the extracted feature X(t) to the feature normalization unit 70 and the baseline calculation unit 50.

[0043] The baseline calculation unit 50 determines whether or not there is a transition in the context Dc (step S205). If there is no transition in the context Dc, the information processing system 1 repeatedly performs steps S201 to S204. If there is a transition in the context Dc, the baseline calculation unit 50 calculates a baseline feature Xbase corresponding to the context Dc before the transition (immediately before the transition) (step S205). The baseline calculation unit 50 sets, for example, a representative value (e.g., average, mode, or median) of the feature X(t) obtained until the transition of the context Dc as the baseline feature Xbase.

[0044] The baseline calculation unit 50 stores the calculated baseline feature Xbase in the baseline feature data table 51 in association with the context Dc.

[0045] Here, suppose that when the baseline calculation unit 50 attempts to store a baseline feature Xbase corresponding to a certain context Dc in the baseline feature data table 51, a baseline feature Xbase corresponding to the same context Dc is already stored in the baseline feature data table 51. In this case, the baseline calculation unit 50 updates the baseline feature Xbase by, for example, performing a weighted addition of the baseline feature Xbase already stored in the baseline feature data table 51 and the baseline feature Xbase to be stored. The baseline calculation unit 50 outputs the updated baseline feature Xbase to the feature normalization unit 70.

[0046] Furthermore, suppose that when the baseline calculation unit 50 attempts to store a baseline feature Xbase corresponding to a certain context Dc in the baseline feature data table 51, no baseline feature Xbase corresponding to the same context Dc is stored in the baseline feature data table 51. In this case, the baseline calculation unit 50, for example, newly registers the calculated baseline feature Xbase in the baseline feature data table 51. The baseline calculation unit 50 outputs the newly registered baseline feature Xbase to the feature normalization unit 70.

[0047] When the context Dc input from the context determination unit 40 transitions, the correction coefficient derivation unit 60 acquires the correction coefficients (transition parameter τ and correction gain g) corresponding to the transition from the transition parameter data table 63 and the correction gain data table 64 (step S206). The correction coefficient derivation unit 60 outputs the acquired correction coefficients (transition parameter τ and correction gain g) to the feature normalization unit 70.

[0048] The feature normalization unit 70 estimates the recovery transition of the physiological response immediately after the transition of the context Dc (step S207). The feature normalization unit 70 normalizes the feature X(t) using the result obtained by the estimation (recovery transition estimation result) (step S208). This results in the normalized feature Xnorm(t).

[0049] The emotion estimation unit 80 analyzes the data pattern of the normalized feature Xnorm(t) and estimates the user's emotion (step S209). The emotion estimation unit 80 inputs, for example, the normalized feature Xnorm(t) obtained by multiplying the time-series feature (x1, x2, ..., xk) within the sliding window by a correction coefficient (correction gain g) (corrected feature g·Xnorm(t)) into the reference model. For example, by inputting the corrected feature g·Xnorm(t) into the reference model, the emotion estimation unit 80 obtains predicted labels (y1, y2, ..., yk) of the time-series emotional states and their reliability from the reference model. For example, the emotion estimation unit 80 obtains the reliability of the representative value of the predicted labels (y1, y2, ..., yk) by weighting and adding the predicted labels (y1, y2, ..., yk) of the time-series emotional states by the reliability. The emotion estimation unit 80 performs threshold processing on the reliability of the representative value of the predicted labels (y1, y2, ..., yk), for example, and thereby obtains a representative value z of the predicted labels (y1, y2, ..., yk) as the emotion estimation result.

[0050] [Effects] Next, the effects of the information processing system 1 will be described.

[0051] When a person's emotions change, physiological responses such as brain waves, heart rate, and sweating are expressed on the body surface. These physiological responses can be read as biosignals using a sensor device, making it possible to estimate a person's emotions. For example, by performing predetermined signal processing on the read biosignals to obtain feature quantities such as physiological indices that contribute to emotional responses, a person's emotions can be estimated by inputting these feature quantities into a model formula derived through machine learning.

[0052] However, when constructing a physiologically valid emotion estimation model in a laboratory environment and applying the model to a real environment, the context-dependence of human emotional physiological responses affects the emotion estimation model. In particular, because people exercise in their daily lives, it is difficult to distinguish whether the accompanying changes in arousal are due to exercise or mental changes. In particular, when recovering immediately after exercise, even if a person is resting, the effects of exercise can make it difficult to accurately estimate changes in arousal.

[0053] On the other hand, in this embodiment, a correction coefficient (transition parameter τ, correction gain g) is derived based on the context transition included in the time-series data (context information Db) of the context of the target organism. Furthermore, in this embodiment, the feature (t) obtained from the time-series data (biological signal Da) of the target organism's biological information corresponding to the context information Db is normalized based on the correction coefficient (transition parameter τ, correction gain g), a baseline feature Xbase (XbaseA) preset for the context before the transition, and a baseline feature Xbase (XbaseB) preset for the context after the transition, thereby calculating the corrected normalized feature g·Xnorm(t). As a result, for example, when the context transitions from context A (walking) to context B (resting), a feature (corrected normalized feature g·Xnorm(t)) obtained by removing the motor arousal component from the feature X(t) after the context transition can be obtained. As a result, it is possible to estimate the level of arousal with high accuracy and reduced context dependency.

[0054] In this embodiment, the transition parameter τ and the correction gain g are used as correction coefficients to calculate the corrected normalized feature g·Xnorm(t). This allows the motor arousal component to be accurately removed from the feature X(t) after the context transition. As a result, the arousal level can be estimated with high accuracy with reduced context dependency.

[0055] Furthermore, in this embodiment, a baseline feature Xbase (XbaseA) before the transition and a baseline feature Xbase (XbaseB) after the transition are calculated based on the feature X(t) obtained from the biosignal Da obtained before the biosignal Da corresponding to the context information Db was acquired, and these are stored in the baseline feature data table 51. This allows the corrected normalized feature g·Xnorm(t) to be obtained using the highly accurate baseline feature Xbase. As a result, the arousal level can be estimated with high accuracy and with reduced context dependency.

[0056] 3. Modifications Next, a description will be given of modifications of the information processing system 1. The information processing system 1 may be configured by a single device, or may be configured by a plurality of devices that can communicate with each other.

[0057] [Variation A] In the above embodiment, information processing system 1 may be configured with wristwatch 100 and mobile terminal 200 that can communicate with each other, as shown in Fig. 9, for example. Wristwatch 100 includes, for example, main body 110 and belt 120 that supports main body 110 and can be wrapped around a person's wrist. Main body 110 includes, for example, sensor 130, control unit 140, storage unit 150, and communication unit 160, as shown in Fig. 10.

[0058] The sensor unit 130 is configured to include the above-mentioned "various sensor units." The sensor unit 130 is capable of outputting a biological signal Da obtained by detection to the control unit 140. The control unit 140 is capable of outputting input data Din including the biological signal Da input from the sensor unit 130 to the communication unit 160. The control unit 140 is configured to include, for example, a CPU.

[0059] The storage unit 150 stores, for example, a processing program 151. The processing program 151 is a program describing a series of processes for controlling the sensor unit 130 and outputting input data Din including a biosignal Da obtained from the sensor unit 130 to the mobile terminal 200 via the communication unit 160. By loading the processing program 151, the control unit 140 is able to control the sensor unit 130 and execute a series of processes for outputting the biosignal Da obtained from the sensor unit 130 to the mobile terminal 200 via the communication unit 160.

[0060] The storage unit 150 stores sensor data 152 including, for example, a biological signal Da obtained by the sensor unit 130. The storage unit 150 is configured, for example, by a non-volatile memory such as an EEPROM, a flash memory, or a resistance change memory. The communication unit 160 is a communication interface capable of communicating with the mobile terminal 200 via a network. The communication unit 160 is capable of outputting input data Din input from the control unit 140 to the mobile terminal 200 via the network.

[0061] 11, the mobile terminal 200 includes, for example, a data acquisition unit 10, a preprocessing unit 20, a feature extraction unit 30, a context determination unit 40, a baseline calculation unit 50, a correction coefficient derivation unit 60, a feature normalization unit 70, and an emotion estimation unit 80. The mobile terminal 200 further includes, for example, a communication unit 210 and a display unit 220, as shown in FIG.

[0062] The communication unit 210 is a communication interface capable of communicating with the wristwatch 100 via a network. The communication unit 210 is capable of receiving input data Din from the wristwatch 100 and outputting it to the data acquisition unit 10. The display unit 220 is capable of displaying data input from the emotion estimation unit 80 (for example, the estimation results of the user's emotion). The display unit 220 is configured, for example, by a liquid crystal display panel or an organic EL display panel.

[0063] In this modification, processes with a high computational load (for example, processes executed by the data acquisition unit 10, preprocessing unit 20, feature extraction unit 30, context determination unit 40, baseline calculation unit 50, correction coefficient derivation unit 60, feature normalization unit 70, and emotion estimation unit 80) are executed by the mobile terminal 200. This minimizes the computational processing in the wristwatch 100, making it possible to reduce the size of the wristwatch 100. Note that a wearable device without a watch function may be used instead of the wristwatch 100. Examples of wearable devices without a watch function include earphones, eyeglass-type devices, necklace-type devices, and belt-type devices.

[0064] [Variation B] In the above embodiment, information processing system 1 may include, for example, wristwatch 100, mobile terminal 300, and server device 400, as shown in Fig. 12. Wristwatch 100 and mobile terminal 300 are configured to be able to communicate with each other via a network. Mobile terminal 300 and server device 400 are configured to be able to communicate with each other via network 500. Network 500 is, for example, the Internet.

[0065] Wristwatch 100, for example, comprises a main body 110 and a belt 120. As shown in Fig. 10, main body 110 comprises a sensor 130, a control unit 140, a storage unit 150, and a communication unit 160. Communication unit 160 is capable of outputting input data Din to mobile terminal 300 via a network.

[0066] 13, the mobile terminal 300 includes a data acquisition unit 10, a preprocessing unit 20, a feature extraction unit 30, a context determination unit 40, and a baseline calculation unit 50. The mobile terminal 300 further includes a communication unit 210, a display unit 220, a data acquisition unit 310, and a communication unit 320, for example, as shown in FIG.

[0067] The data acquisition unit 310 is capable of, for example, acquiring the feature X(t) from the feature extraction unit 30, acquiring the context Dc from the context determination unit 40, and acquiring the baseline feature Xbase from the baseline calculation unit 50. The data acquisition unit 310 is capable of, for example, outputting the acquired feature X(t), context Dc, and baseline feature Xbase to the communication unit 320. The data acquisition unit 310 is capable of, for example, acquiring an estimation result of the user's emotion (representative value z) from the communication unit 320 and outputting it to the display unit 220. The display unit 220 is capable of, for example, displaying the data input from the data acquisition unit 310 (the estimation result of the user's emotion (representative value z)).

[0068] The communication unit 320 is a communication interface capable of communicating with the server device 400 via the network 500. The communication unit 320 is capable of acquiring data (e.g., feature X(t), context Dc, and baseline feature Xbase) from the data acquisition unit 310 and transmitting the data to the server device 400. The communication unit 320 is capable of receiving data (e.g., an estimation result of the user's emotion (representative value z)) from the server device 400 and outputting the data to the data acquisition unit 310.

[0069] 14, the server device 400 includes a correction coefficient derivation unit 60, a feature normalization unit 70, an emotion estimation unit 80, a normalization parameter table 71, and a model parameter table 81. The server device 400 further includes a communication unit 410 and a data acquisition unit 420, for example, as shown in FIG.

[0070] The communication unit 410 is a communication interface capable of communicating with the mobile device 300 via the network 500. The communication unit 410 is capable of receiving data (e.g., feature X(t), context Dc, and baseline feature Xbase) from the mobile device 300 and outputting the data to the data acquisition unit 420. The communication unit 410 is capable of outputting data input from the data acquisition unit 420 (e.g., an estimation result of the user's emotion (representative value z)) to the mobile device 300 via the network 500. The data acquisition unit 420 is capable of outputting data acquired from the communication unit 410 (e.g., feature X(t), context Dc, and baseline feature Xbase) to the correction coefficient derivation unit 60 and the feature normalization unit 70. The emotion estimation unit 80 is capable of outputting generated data (e.g., an estimation result of the user's emotion (representative value z)) to the data acquisition unit 420.

[0071] In this modification, processes with particularly high computational loads (for example, processes executed by correction coefficient derivation unit 60, feature normalization unit 70, and emotion estimation unit 80) are executed by server device 400. This minimizes the computational processing in wristwatch 100 and mobile terminal 300, making it possible to miniaturize wristwatch 100 and use mobile terminals 300 with low specifications. Note that a wearable device without a watch function may be used instead of wristwatch 100. Examples of wearable devices without a watch function include earphones, eyeglass-type devices, necklace-type devices, and belt-type devices.

[0072] [Variation C] In the above-described embodiment and variations thereof, the emotion estimation unit 80 may include, for example, as shown in FIG. 15 , multiple machine learning models (e.g., three reference models 82, 83, and 84) having different arousal baselines, and an output unit 85. The machine learning models include, for example, a regression model or a discriminative model. That is, in this variation, the information processing system 1 estimates the emotion of a target living organism using multiple machine learning models provided in the emotion estimation unit 80.

[0073] The alertness baseline refers to the average value of the training information used in training each machine learning model before normalization. For example, the alertness baseline of reference model 82 is lower than the alertness baselines of reference models 83 and 84, and the alertness baseline of reference model 83 is lower than the alertness baseline of reference model 84.

[0074] "Information before normalization of the training information used for training each machine learning model" refers to, for example, feature quantities (non-normalized feature quantities) generated based on time-series data of biological information obtained by the sensor unit 10 or the like. These feature quantities may include multiple classes of different arousal levels. This means, for example, that while a target organism is at a predetermined arousal level in a certain environment, the arousal level may change depending on the time and situation. Note that the target organism when generating the training information is usually different from the target organism whose emotion is estimated by the information processing system 1.

[0075] The reference models 82, 83, and 84 are models generated using data acquired under environments with different arousal levels. The reference model 82 is a model generated using normalized information on the features of data acquired at a relatively low arousal baseline and arousal information. The model parameters obtained by this learning are stored in the model parameter table 81. The reference model 84 is a model generated using normalized information on the features of data acquired at a relatively high arousal baseline and arousal information. The model parameters obtained by this learning are stored in the model parameter table 81. The reference model 83 is a model generated using normalized information on the features of data acquired at a wakefulness baseline between the wakefulness baseline of the reference model 82 and the wakefulness baseline of the reference model 84 and arousal information. The model parameters obtained by this learning are stored in the model parameter table 81.

[0076] The correction coefficient conversion unit 90 is capable of selecting one machine learning model from among the multiple machine learning models included in the emotion estimation unit 80. The correction coefficient conversion unit 90 selects, for example, the reference model 82. The correction coefficient conversion unit 90 is capable of selecting, for example, from among the multiple machine learning models included in the emotion estimation unit 80, the machine learning model having an arousal baseline that is closest to the arousal baseline of the feature quantity X(t) obtained based on the biological signal Da. This is to perform mapping, which will be described later, with high accuracy.

[0077] The correction coefficient conversion unit 90 is capable of converting the correction coefficients (transition parameter τ, correction gain g) into correction coefficients (transition parameter τ', correction gain g') corresponding to the selected machine learning model based on the feature X(t) input from the feature extraction unit 30 and the feature used in training the selected machine learning model. The correction coefficient conversion unit 90 is capable of deriving a conversion gain that maps the feature used in training the selected machine learning model to the feature X(t) input from the feature extraction unit 30. The correction coefficient conversion unit 90 is capable of converting the correction coefficients (transition parameter τ, correction gain g) into correction coefficients (transition parameter τ', correction gain g') corresponding to the selected machine learning model using the derived conversion gain. The correction coefficient conversion unit 90 is capable of outputting the converted correction coefficients (transition parameter τ', correction gain g') to the feature normalization unit 70.

[0078] The feature normalization unit 70 has multiple normalization units (e.g., three normalization units 72, 73, and 74), one for each machine learning model (e.g., three reference models 82, 83, and 84) included in the emotion estimation unit 80. The normalization unit 72 is provided corresponding to the reference model 82. The normalization unit 73 is provided corresponding to the reference model 83. The normalization unit 74 is provided corresponding to the reference model 84.

[0079] The normalization unit (72, 73, or 74) corresponding to the machine learning model selected by the correction coefficient conversion unit 90 is capable of calculating the normalized feature Xnorm(t)′ by normalizing the feature X(t) based on the converted correction coefficient (transition parameter τ′), the baseline feature Xbase previously set for the context Dc before the transition, and the baseline feature Xbase previously set for the context Dc after the transition.

[0080] The emotion estimation unit 80 is capable of inputting the normalized feature Xnorm(t)' multiplied by the converted correction coefficient (correction gain g') (corrected feature g'·Xnorm(t)') to the machine learning model (reference model 82, 83, or 84) selected by the correction coefficient conversion unit 90. When the corrected feature g'·Xnorm(t)' is input, the machine learning model (reference model 82, 83, or 84) selected by the correction coefficient conversion unit 90 is capable of outputting predicted labels (y1', y2', ..., yk') and reliability of the time-series emotional state according to the input corrected feature g'·Xnorm(t)'.

[0081] The output unit 85 performs weighted addition on the predicted labels (y1', y2', ..., yk') of the emotional states in time series by the reliability, thereby obtaining the reliability of the representative value of the predicted labels (y1', y2', ..., yk'). For example, the output unit 85 performs threshold processing on the reliability of the representative value of the predicted labels (y1', y2', ..., yk'), thereby obtaining the representative value z of the predicted labels (y1', y2', ..., yk') as the emotion estimation result.

[0082] In this modification, the correction coefficient conversion unit 90 converts the correction coefficients (transition parameter τ, correction gain g) into correction coefficients (transition parameter τ', correction gain g') corresponding to the selected machine learning model, and estimates the emotion based on the converted correction coefficients (transition parameter τ', correction gain g') and the selected machine learning model. This makes it possible to accurately determine the emotion of the target living organism even if the arousal baseline assumed during model learning does not match the arousal baseline of the user.

[0083] Furthermore, for example, the present disclosure can be configured as follows: (1) An information processing system including: a data acquisition unit capable of acquiring first time series data of a context of a target living organism and second time series data of biometric information of the target living organism corresponding to the first time series data; and one or more processing units capable of performing processing using the first time series data and the second time series data, wherein the one or more processing units are capable of performing correction coefficient derivation processing to derive correction coefficients based on a transition of the context included in the first time series data, feature normalization processing to calculate normalized features by normalizing features obtained from the second time series data based on the correction coefficients, first baseline features set in advance for the context before the transition, and second baseline features set in advance for the context after the transition, and emotion estimation processing to estimate emotions of the target living organism based on the normalized features. (2) The information processing system according to (1), wherein the one or more processing units are capable of calculating the normalized feature amount by using, as the correction coefficient, a transition parameter that defines a transition of recovery of a physiological response at the time of a context transition and a correction gain that defines a recovery degree of the sensitivity of the physiological response according to an elapsed time of recovery of the physiological response. (3) The information processing system according to (1) or (2), wherein the one or more processing units are capable of performing a baseline calculation process to calculate the first baseline feature amount and the second baseline feature amount based on feature amounts obtained from third time-series data of biological information of the target living organism that was obtained before the acquisition of the second time-series data, and store the calculated feature amounts in a storage unit.(4) An information processing method including: acquiring first time series data of a context of a target organism and second time series data of biometric information of the target organism corresponding to the first time series data; deriving a correction coefficient based on a transition of the context included in the first time series data; calculating normalized features by normalizing features obtained from the second time series data based on the correction coefficient, a first baseline feature preset for the context before the transition, and a second baseline feature preset for the context after the transition; and estimating emotions of the target organism based on the normalized features.

[0084] In an information processing system and an information processing method according to an aspect of the present disclosure, a feature obtained from second time-series data of biological information of a target living organism is normalized based on a correction coefficient derived based on a transition of a context of the target living organism, a first baseline feature preset for the context before the transition, and a second baseline feature preset for the context after the transition. This makes it possible to reduce feature change caused by the context before the transition from feature change in the context after the transition. As a result, a change in arousal level in the context after the transition can be obtained with reduced influence of arousal level change caused by the context before the transition. Therefore, context dependency can be reduced, and arousal level can be estimated with high accuracy.

[0085] This application claims priority based on Japanese Patent Application No. 2024-010466, filed on January 26, 2024, in the Japan Patent Office, the entire contents of which are incorporated herein by reference.

[0086] Those skilled in the art will recognize that various modifications, combinations, subcombinations, and variations may occur depending on design requirements and other factors, and are intended to be within the scope of the appended claims and their equivalents.

Claims

1. A data acquisition unit capable of acquiring first time-series data of the context of a target living body and second time-series data of the biological information of the target living body corresponding to the first time-series data, and one or more processing units capable of performing processing using the first time-series data and the second time-series data. The one or more processing units perform a correction coefficient derivation process for deriving a correction coefficient based on the transition of the context included in the first time-series data, and based on the correction coefficient, a first baseline feature amount preset for the context before the transition, and a second baseline feature amount preset for the context after the transition, perform a feature amount normalization process for calculating a normalized feature amount by normalizing the feature amount obtained from the second time-series data, and perform an emotion estimation process for estimating the emotion of the target living body based on the normalized feature amount. An information processing system that can do this.

2. In the feature amount normalization process, the one or more processing units can calculate the normalized feature amount using, as the correction coefficient, a transition parameter that defines the recovery transition of the physiological reaction at the time of the transition of the context and a correction gain that defines the degree of recovery of the sensitivity of the physiological reaction according to the elapsed time of the recovery of the physiological reaction. The information processing system according to claim 1.

3. The one or more processing units can perform a baseline calculation process for calculating the first baseline feature amount and the second baseline feature amount based on the feature amount obtained from the third time-series data of the biological information of the target living body obtained before the acquisition of the second time-series data, and storing them in a storage unit. The information processing system according to claim 1.

4. Obtaining first time-series data of the context of a target living body and second time-series data of the biological information of the target living body corresponding to the first time-series data; deriving a correction coefficient based on the transition of the context included in the first time-series data; calculating a normalized feature amount by normalizing the feature amount obtained from the second time-series data based on the correction coefficient, a first baseline feature amount preset for the context before the transition, and a second baseline feature amount preset for the context after the transition; and estimating the emotion of the target living body based on the normalized feature amount. An information processing method including the above steps.

Images