Information processing system

Abstract

An information processing system according to the present disclosure comprises one or a plurality of information processing devices. The information processing devices each include: an estimation unit that estimates an internal state with respect to a stimulus or a situation and calculates a time series indicating temporal changes of the internal state; and a calculation unit that calculates an approximation line of the time-series and calculates an intervention effect in accordance with the stimulus or the situation on the basis of the approximation line.

Description

Information Processing Systems

[0001] The present disclosure relates to an information processing system.

[0002] Methods for calculating the intervention effect of stimuli or situations have been proposed (see Patent Documents 1 to 3). However, because changes in a person's internal state generally fluctuate, it is difficult to define the intervention effect on the internal state due to the endpoint at which stimuli or situations change, such as entertainment content enjoyed at a certain time.

[0003] JP 2020-531982 A JP 2022-94751 A JP 2023-4225 A

[0004] For example, when one is in a state of high arousal, one may listen to music to relax, but one may want to know the relaxation effect from the dynamics of arousal after one starts listening to music. However, the state of arousal fluctuates, making it difficult to quantitatively evaluate the relaxation effect.

[0005] Therefore, it is desirable to provide an information processing system that can quantitatively evaluate the effects of interventions based on stimuli or situations.

[0006] An information processing system according to one embodiment of the present disclosure includes one or more information processing devices, and the information processing devices have an estimation unit that estimates an internal state in response to a stimulus or situation and calculates a time series showing changes in the internal state over time, and a calculation unit that calculates an approximation line of the time series and calculates the intervention effect of the stimulus or situation based on the approximation line.

[0007] In an information processing system according to one embodiment of the present disclosure, a time series approximation line showing the temporal change in the internal state in response to a stimulus or situation is calculated, and the intervention effect of the stimulus or situation is calculated based on the calculated approximation line.

[0008] FIG. 1 is a block diagram schematically illustrating an example configuration of an information processing system according to an embodiment of the present disclosure. FIG. 2 is a block diagram schematically illustrating an example configuration of an information processing unit according to an embodiment. FIG. 3 is an external view illustrating an example of a head-mounted display equipped with a biosensor. FIG. 4 is an external view illustrating an example of a headband equipped with a biosensor. FIG. 5 is an external view illustrating an example of headphones equipped with a biosensor. FIG. 6 is an external view illustrating an example of earphones equipped with a biosensor. FIG. 7 is an external view illustrating an example of a watch equipped with a biosensor. FIG. 8 is an external view illustrating an example of eyeglasses equipped with a biosensor. FIG. 9 is a flowchart illustrating an example operation of an information processing system according to an embodiment. FIG. 10 is a flowchart illustrating an example of a process for calculating an approximation line in an information processing system according to an embodiment. FIG. 11 is an explanatory diagram illustrating an example calculation of an intervention effect in an information processing system according to an embodiment. FIG. 12 is an explanatory diagram illustrating an example of a calculation method of an intervention effect in an information processing system according to an embodiment. FIG. 13 is an explanatory diagram illustrating MAE (Mean Absolute Error) as an example of a method for calculating an error. FIG. 14 is an explanatory diagram showing a modified example of calculation of an intervention effect in an information processing system according to an embodiment. FIG. 15 is an explanatory diagram showing a modified example of calculation of an intervention effect in an information processing system according to an embodiment. FIG. 16 is an explanatory diagram showing an example of a UI screen using information on the evaluation of the intervention effect calculated by an information processing system according to an embodiment. FIG. 17 is an explanatory diagram showing an example of a UI screen using information on the evaluation of the intervention effect calculated by an information processing system according to an embodiment. FIG. 18 is an explanatory diagram showing an example of a UI screen using information on the evaluation of the intervention effect calculated by an information processing system according to an embodiment. FIG. 19 is an explanatory diagram showing an example of a UI screen using information on the evaluation of the intervention effect calculated by an information processing system according to an embodiment. FIG. 20 is an explanatory diagram showing an example of a UI screen using information on the evaluation of the intervention effect calculated by an information processing system according to an embodiment. FIG. 21 is an explanatory diagram showing an example of a UI screen using information on the evaluation of the intervention effect calculated by an information processing system according to an embodiment.FIG. 22 is an explanatory diagram showing an example of a UI screen using information on the evaluation of intervention effect calculated by an information processing system according to an embodiment. FIG. 23 is an explanatory diagram showing an example of a UI screen using information on the evaluation of intervention effect calculated by an information processing system according to an embodiment. FIG. 24 is an explanatory diagram showing an example of a UI screen using information on the evaluation of intervention effect calculated by an information processing system according to an embodiment. FIG. 25 is an explanatory diagram showing an example of a UI screen using information on the evaluation of intervention effect calculated by an information processing system according to an embodiment. FIG. 26 is an explanatory diagram showing an example of a UI screen using information on the evaluation of intervention effect calculated by an information processing system according to an embodiment. FIG. 27 is an explanatory diagram showing an example of a UI screen using information on the evaluation of intervention effect calculated by an information processing system according to an embodiment. FIG. 28 is an explanatory diagram showing an example of a UI screen using information on the evaluation of intervention effect calculated by an information processing system according to an embodiment. FIG. 29 is an explanatory diagram showing an example of a UI screen using information on the evaluation of intervention effect calculated by an information processing system according to an embodiment. FIG. 30 is an explanatory diagram showing an example of a UI screen using information on the evaluation of intervention effect calculated by an information processing system according to an embodiment. Fig. 31 is an explanatory diagram showing an example of a UI screen using information on the evaluation of the intervention effect calculated by an information processing system according to an embodiment. Fig. 32 is an explanatory diagram showing an example of a UI screen using information on the evaluation of the intervention effect calculated by an information processing system according to an embodiment. Fig. 33 is an explanatory diagram showing an example of a UI screen using information on the evaluation of the intervention effect calculated by an information processing system according to an embodiment.

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The description will be made in the following order: 1. One embodiment 1.1 Method for deriving internal state 1.1.1 Regarding arousal level 1.1.2 Regarding pleasantness / unpleasantness 1.2 Configuration of information processing system 1.3 Operation of information processing system 1.4 Effects 2. Other embodiments

[0010] 1. One embodiment 1.1 Method for deriving internal states Before describing an information processing system according to one embodiment, a general method for deriving internal states will be described. Examples of a person's internal states include arousal level and pleasant / unpleasant feelings. Arousal level is related to concentration, drowsiness, attention, craving, etc. Pleasure / unpleasant feelings are related to enjoyment, joy, sadness, etc.

[0011] [1.1.1 About Arousal Level] A person's arousal level is closely related to their ability to concentrate. When a person is concentrating, they have a high level of interest in the object of their concentration. Therefore, by knowing a person's arousal level, it is possible to estimate the person's objective level of interest (emotion). A person's arousal level can be derived based on biometric information or behavioral information obtained from the person or the communication partner (hereinafter referred to as the "target organism") during a conversation with the communication partner.

[0012] Examples of biological information from which the level of arousal of a subject living body can be derived include electroencephalograms, sweating, pulse waves, electrocardiograms, blood flow, skin temperature, facial myoelectric potentials, electrooculography, or information on specific components contained in saliva.

[0013] (Electroencephalograms) It is known that alpha waves contained in electroencephalograms increase when one is relaxed, such as at rest, and beta waves contained in electroencephalograms increase when one is actively thinking or concentrating. Therefore, for example, when the power spectrum area of ​​the frequency band of alpha waves contained in electroencephalograms is smaller than a predetermined threshold value th1 and the power spectrum area of ​​the frequency band of beta waves contained in electroencephalograms is larger than a predetermined threshold value th2, it is possible to estimate that the level of arousal of the target living organism is high.

[0014] Furthermore, when estimating the arousal level of a target living organism using electroencephalograms, an estimation model such as machine learning can be used instead of the thresholds th1 and th2. This estimation model is, for example, a model trained using the power spectrum of electroencephalograms when the arousal level is clearly high as training data. When the power spectrum of electroencephalograms is input, for example, this estimation model estimates the arousal level of the target living organism based on the input power spectrum of the electroencephalograms. This estimation model includes, for example, a neural network. This learning model may include, for example, a deep neural network such as a convolutional neural network (CNN).

[0015] Alternatively, the electroencephalogram may be divided into a plurality of segments on the time axis, a power spectrum may be derived for each divided segment, and the power spectrum area of ​​the α wave frequency band may be derived for each derived power spectrum. In this case, for example, if the derived power spectrum area is smaller than a predetermined threshold tha, it is possible to estimate that the level of arousal of the target living organism is high.

[0016] Furthermore, it is also possible to estimate the level of arousal of a target living organism using, for example, an estimation model that estimates the level of arousal of a target living organism based on the derived power spectrum area. This estimation model is, for example, a model that is trained using power spectrum areas when the level of arousal is clearly high as training data. When, for example, a power spectrum area is input, this estimation model estimates the level of arousal of the target living organism based on the input power spectrum area. This estimation model includes, for example, a neural network. This learning model may include, for example, a deep neural network such as a convolutional neural network (CNN).

[0017] (Sweating) Psychological sweating is sweating released from eccrine glands during sympathetic nervous tension due to mental or psychological issues such as stress, tension, and anxiety. For example, by attaching a sweat meter probe to the palms or soles of the feet and measuring sweating (psychological sweating) on ​​the palms or soles induced by various stress stimuli, the sympathetic sweating response (SSwR) can be obtained as a signal voltage. When the values ​​of a predetermined high-frequency component or a predetermined low-frequency component in this signal voltage are higher than a predetermined threshold, it is possible to estimate that the target living organism has a high level of arousal.

[0018] Furthermore, it is also possible to estimate the level of arousal of a target living organism using an estimation model that estimates the level of arousal of a target living organism based on a predetermined high-frequency component or a predetermined low-frequency component contained in the signal voltage. This estimation model is, for example, a model trained using a predetermined high-frequency component or a predetermined low-frequency component contained in the signal voltage when the level of arousal is clearly high as training data. When a predetermined high-frequency component or a predetermined low-frequency component is input, this estimation model estimates the level of arousal of the target living organism based on the input predetermined high-frequency component or predetermined low-frequency component. This estimation model includes, for example, a neural network. This learning model may include, for example, a deep neural network such as a convolutional neural network (CNN).

[0019] (Pulse wave, electrocardiogram, blood flow) A high heart rate is generally said to indicate a high level of arousal. The heart rate can be derived from the pulse wave, electrocardiogram, or blood flow velocity. Therefore, for example, if the heart rate is derived from the pulse wave, electrocardiogram, or blood flow velocity, and the derived heart rate is greater than a predetermined threshold, it can be estimated that the level of arousal of the target living organism is high.

[0020] It is also possible to estimate the level of arousal of a target living organism using an estimation model that estimates the level of arousal of a target living organism based on, for example, a heart rate derived from a pulse wave, an electrocardiogram, or a blood flow velocity. This estimation model is, for example, a model trained using heart rates at which the level of arousal is clearly high as training data. When, for example, a heart rate derived from a pulse wave, an electrocardiogram, or a blood flow velocity is input, this estimation model estimates the level of arousal of the target living organism based on the input heart rate. This estimation model includes, for example, a neural network. This learning model may include, for example, a deep neural network such as a convolutional neural network (CNN).

[0021] Furthermore, when the heart rate variability (HRV) is small, it is generally said that the parasympathetic nervous system is subordinated and the level of arousal is high. Therefore, for example, by deriving the heart rate variability (HRV) from the pulse wave, electrocardiogram, or blood flow velocity, it is possible to estimate that the level of arousal of the target living body is high when the derived heart rate variability (HRV) is smaller than a predetermined threshold.

[0022] Furthermore, it is also possible to estimate the level of arousal of a target living organism using an estimation model that estimates the level of arousal of a target living organism based on, for example, heart rate variability (HRV) derived from a pulse wave, an electrocardiogram, or blood flow velocity. This estimation model is, for example, a model trained using heart rate variability (HRV) at a clearly high level of arousal as training data. When, for example, heart rate variability (HRV) derived from a pulse wave, an electrocardiogram, or blood flow velocity is input, this estimation model estimates the level of arousal of the target living organism based on the input heart rate variability (HRV). This estimation model includes, for example, a neural network. This learning model may include, for example, a deep neural network such as a convolutional neural network (CNN).

[0023] (Skin Temperature) A high skin temperature is generally said to indicate a high level of arousal. Skin temperature can be measured, for example, by thermography. Therefore, for example, when the skin temperature measured by thermography is higher than a predetermined threshold, it is possible to estimate that the level of arousal of the target living body is high.

[0024] It is also possible to estimate the arousal level of a target living organism using, for example, an estimation model that estimates the arousal level of a target living organism based on skin temperature. This estimation model is, for example, a model that is trained using skin temperatures at times when the arousal level is clearly high as training data. When, for example, a skin temperature is input, this estimation model estimates the arousal level of the target living organism based on the input skin temperature. This estimation model includes, for example, a neural network. This learning model may include, for example, a deep neural network such as a convolutional neural network (CNN).

[0025] (Facial myoelectric potential) It is known that the corrugator supercilii, which frowns when thinking, shows high activity. It is also known that the zygomaticus major muscle does not change much when imagining happy things. In this way, it is possible to estimate emotion and arousal level according to the parts of the face. Therefore, for example, by measuring the facial myoelectric potential of a specific part, if the measured value is higher than a specific threshold, it is possible to estimate the level of arousal level of the target living organism.

[0026] It is also possible to estimate the level of arousal of a target organism using, for example, an estimation model that estimates the level of arousal of a target organism based on facial myoelectric potentials. This estimation model is, for example, a model trained using, as training data, facial myoelectric potentials obtained when the level of arousal is clearly high. When, for example, facial myoelectric potentials are input, this estimation model estimates the level of arousal of the target organism based on the input facial myoelectric potentials. This estimation model includes, for example, a neural network. This learning model may include, for example, a deep neural network such as a convolutional neural network (CNN).

[0027] (Electrooculography) A method for measuring eye movement is known, taking advantage of the fact that the corneal side of the eye is positively charged and the retina side is negatively charged. The measurement value obtained using this measurement method is an electrooculogram. For example, it is possible to estimate eye movement from the acquired electrooculogram, and if the estimated eye movement follows a predetermined trend, it is possible to estimate the level of arousal of the target living organism.

[0028] It is also possible to estimate the level of arousal of a target living organism using, for example, an estimation model that estimates the level of arousal of a target living organism based on an electrooculogram. This estimation model is, for example, a model that is trained using electrooculograms obtained when the level of arousal is clearly high as training data. When, for example, an electrooculogram is input, this estimation model estimates the level of arousal of the target living organism based on the input electrooculogram. This estimation model includes, for example, a neural network. This learning model may include, for example, a deep neural network such as a convolutional neural network (CNN).

[0029] (Saliva) Saliva contains cortisol, a type of stress hormone. It is known that the amount of cortisol in saliva increases when a subject is under stress. Therefore, for example, when the amount of cortisol in saliva is higher than a predetermined threshold, it can be estimated that the subject's level of arousal is high.

[0030] It is also possible to estimate the level of arousal of a target organism using, for example, an estimation model that estimates the level of arousal of a target organism based on the amount of cortisol contained in saliva. This estimation model is, for example, a model trained using the amount of cortisol contained in saliva when the level of arousal is clearly high as training data. When the amount of cortisol contained in saliva is input, for example, this estimation model estimates the level of arousal of the target organism based on the input amount of cortisol. This estimation model includes, for example, a neural network. This learning model may include, for example, a deep neural network such as a convolutional neural network (CNN).

[0031] On the other hand, examples of the motion information from which the arousal level of the target living organism can be derived include information on facial expressions, voice, blinking, breathing, or reaction times of actions.

[0032] (Facial Expression) It is known that the zygomatic major muscle does not change much when frowning while thinking, or when imagining happy things. In this way, it is possible to estimate emotions and arousal levels based on facial expressions. For example, it is possible to photograph a person's face with a camera, estimate facial expressions based on the resulting video data, and estimate the level of arousal of the target living organism based on the estimated facial expressions.

[0033] Furthermore, it is also possible to estimate the level of arousal of a target living organism using an estimation model that estimates the level of arousal of a target living organism based on video data of facial expressions. This estimation model is, for example, a model trained using video data of facial expressions captured when the level of arousal is clearly high as training data. When video data of facial expressions is input, for example, this estimation model estimates the level of arousal of the target living organism based on the input video data. This estimation model includes, for example, a neural network. This learning model may include, for example, a deep neural network such as a convolutional neural network (CNN).

[0034] (Voice) It is known that voice, like facial expressions, changes depending on emotions and arousal levels. Therefore, for example, it is possible to acquire voice data using a microphone and estimate the level of arousal level of a target living organism based on the acquired voice data.

[0035] It is also possible to estimate the level of arousal of a target living organism using, for example, an estimation model that estimates the level of arousal of a target living organism based on audio data. This estimation model is, for example, a model trained using audio data from a time when the level of arousal is clearly high as training data. When audio data is input, for example, this estimation model estimates the level of arousal of the target living organism based on the input audio data. This estimation model includes, for example, a neural network. This learning model may include, for example, a deep neural network such as a convolutional neural network (CNN).

[0036] (Blinking) It is known that blinking, like facial expressions, changes depending on emotions and arousal levels. Therefore, for example, blinking can be captured with a camera, the blinking frequency can be measured based on the resulting video data, and the level of arousal level of the target living organism can be estimated based on the measured blinking frequency. Furthermore, for example, blinking frequency can be measured using an electrooculogram, and the level of arousal level of the target living organism can be estimated based on the measured blinking frequency.

[0037] Furthermore, it is also possible to estimate the level of arousal of a target living organism using an estimation model that estimates the level of arousal of a target living organism based on, for example, video data of blinking or an electrooculogram. This estimation model is a model that is trained using, for example, video data of blinking captured when the level of arousal is clearly high, or an electrooculogram, as training data. When, for example, video data of blinking or an electrooculogram is input, this estimation model estimates the level of arousal of the target living organism based on the input video data or electrooculogram. This estimation model includes, for example, a neural network. This learning model may include, for example, a deep neural network such as a convolutional neural network (CNN).

[0038] (Respiration) It is known that respiration, like facial expressions, changes depending on emotions and arousal levels. Therefore, for example, by measuring the respiration volume or respiration rate, it is possible to estimate the level of arousal of a target living body based on the measurement data obtained thereby.

[0039] It is also possible to estimate the arousal level of a target living organism using an estimation model that estimates the arousal level of a target living organism based on, for example, a respiration volume or a respiration rate. This estimation model is, for example, a model trained using a respiration volume or a respiration rate when the arousal level is clearly high as training data. When, for example, a respiration volume or a respiration rate is input, this estimation model estimates the arousal level of the target living organism based on the input respiration volume or respiration rate. This estimation model includes, for example, a neural network. This learning model may include, for example, a deep neural network such as a convolutional neural network (CNN).

[0040] (Reaction time of actions) It is known that the processing time (reaction time) when a person sequentially processes multiple tasks and the variability of the processing time (reaction time) depend on the person's level of arousal. Therefore, for example, by measuring the processing time (reaction time) and the variability of the processing time (reaction time), it is possible to estimate the level of arousal of the target living organism based on the measurement data obtained thereby.

[0041] [1.1.2 Pleasure and Displeasure] A person's pleasure and displeasure, like their level of alertness, are closely related to their ability to concentrate. When a person is concentrating, they have a high level of interest in the object of their concentration. Therefore, by knowing a person's pleasure and displeasure, it is possible to estimate the person's objective level of interest (emotion). A person's pleasure and displeasure can be derived based on biological information or behavioral information obtained from the person or the communication partner (hereinafter referred to as the "target organism") during a conversation with the communication partner.

[0042] Examples of biological information from which the comfort / discomfort of a target living body can be derived include information on brain waves and sweating, and examples of motion information from which the comfort / discomfort of a target living body can be derived include facial expressions.

[0043] (Electroencephalograms) It is known that a person's comfort or discomfort can be estimated from the difference in alpha waves contained in electroencephalograms between the left and right sides of the frontal lobe. For example, suppose that the alpha waves contained in electroencephalograms obtained from the left side of the frontal lobe (hereinafter referred to as "left alpha waves") are compared with the alpha waves contained in electroencephalograms obtained from the right side of the frontal lobe (hereinafter referred to as "right alpha waves"). When the left alpha waves are lower than the right alpha waves, it can be estimated that the subject is feeling comfort, and when the left alpha waves are higher than the right alpha waves, it can be estimated that the subject is feeling discomfort.

[0044] Furthermore, when estimating the pleasure / discomfort of a target living organism using electroencephalograms, instead of deriving the difference between the left and right sides of the frontal lobe in alpha waves contained in electroencephalograms, it is also possible to use an estimation model such as machine learning. This estimation model is, for example, a model trained using alpha waves or beta waves contained in electroencephalograms obtained when the target living organism is clearly experiencing pleasure as training data. When alpha waves or beta waves contained in electroencephalograms are input, for example, this estimation model estimates the pleasure / discomfort of the target living organism based on the input alpha waves or beta waves. This estimation model includes, for example, a neural network. This learning model may include, for example, a deep neural network such as a convolutional neural network (CNN).

[0045] (Sweating) Psychological sweating is sweating released from eccrine glands during sympathetic nervous tension due to mental or psychological issues such as stress, tension, and anxiety. For example, by attaching a sweat meter probe to the palms or soles of the feet and measuring palm or sole sweating (psychological sweating) induced by various stress stimuli, the sympathetic sweating response (SSwR) can be obtained as a signal voltage. When the values ​​of predetermined high-frequency components and predetermined low-frequency components obtained from the left hand in this signal voltage are higher than the values ​​of predetermined high-frequency components and predetermined low-frequency components obtained from the right hand, it can be inferred that the subject is experiencing pleasure. When the values ​​of predetermined high-frequency components and predetermined low-frequency components obtained from the left hand in this signal voltage are lower than the values ​​of predetermined high-frequency components and predetermined low-frequency components obtained from the right hand, it can be inferred that the subject is experiencing discomfort. When the amplitude value obtained from the left hand in this signal voltage is higher than the amplitude value obtained from the right hand, it can be inferred that the subject is experiencing pleasure. Furthermore, when the amplitude value obtained from the left hand is lower than the amplitude value obtained from the right hand in the signal voltage, it can be estimated that the subject living body is feeling discomfort.

[0046] Furthermore, it is also possible to estimate the level of arousal of a target living organism using an estimation model that estimates the level of arousal of a target living organism based on a predetermined high-frequency component or a predetermined low-frequency component contained in the signal voltage. This estimation model is, for example, a model trained using a predetermined high-frequency component or a predetermined low-frequency component contained in the signal voltage when the level of arousal is clearly high as training data. When a predetermined high-frequency component or a predetermined low-frequency component is input, this estimation model estimates the level of arousal of the target living organism based on the input predetermined high-frequency component or predetermined low-frequency component. This estimation model includes, for example, a neural network. This learning model may include, for example, a deep neural network such as a convolutional neural network (CNN).

[0047] (Facial Expression) It is known that when a person feels uncomfortable, they frown, while when a person feels comfortable, the zygomatic major muscle does not change much. In this way, it is possible to estimate comfort or discomfort based on facial expression. Therefore, for example, it is possible to photograph the face with a camera, estimate facial expression based on the obtained video data, and estimate the comfort or discomfort of the target living organism based on the estimated facial expression.

[0048] It is also possible to estimate the comfort / discomfort of a target organism using an estimation model that estimates the comfort / discomfort of a target organism based on video data of captured facial expressions. This estimation model is a model that is trained using video data of captured facial expressions at a clearly high level of arousal as training data. When video data of captured facial expressions is input, this estimation model estimates the comfort / discomfort of the target organism based on the input video data. This estimation model includes, for example, a neural network. This learning model may include, for example, a deep neural network such as a convolutional neural network (CNN).

[0049] ・The frequency components of EEGs are described, for example, in the following document: Wang, Xiao-Wei, Dan Nie, and Bao-Liang Lu. "EEG-based emotion recognition using frequency domain features and support vector machines." International conference on neural information processing. Springer, Berlin, Heidelberg, 2011. ・An estimation model using EEGs is described, for example, in the following document: Patent Application No. 2020-203058. ・Sweating is described, for example, in the following document: Jing Zhai, AB Barreto, C. Chin and Chao Li, "Realization of stress detection using psychophysiological signals for improvement of human-computer interactions," Proceedings. IEEE SoutheastCon, 2005., Ft. Lauderdale, FL, USA, 2005, pp. 415-420, doi: 10.1109 / SECON.2005.1423280. Boucsein, Wolfram. Electrodermal activity. Springer Science & Business Media, 2012. ・Heart rate is described, for example, in the following literature: Veltman, JA, and AWK Gaillard. "Physiological indices of workload in a simulated flight task." Biological psychology 42.3 (1996): 323-342. ・Heart rate variability intervals are described, for example, in the following literature:Appelhans, Bradley M., and Linda J. Luecken. "Heart rate variability as an index of regulated emotional responding." Review of general psychology 10.3 (2006): 229-240. - Salivary cortisol levels are described, for example, in the following literature: Lam, Suman, et al. "Emotion regulation and cortisol reactivity to a social-evaluative speech task." Psychoneuroendocrinology 34.9 (2009): 1355-1362. - Facial expressions are described, for example, in the following literature: Lyons, Michael J., Julien Budynek, and Shigeru Akamatsu. "Automatic classification of single facial images." IEEE transactions on pattern analysis and machine intelligence 21.12 (1999): 1357-1362. - Facial muscles are described, for example, in the following literature: Ekman, Paul. "Facial action coding system." (1977). ・Blink frequency is described, for example, in the following literature: Chen, Siyuan, and Julien Epps. "Automatic classification of eye activity for cognitive load measurement with emotion interference." Computer methods and programs in biomedicine 110.2 (2013): 111-124. ・Respiration volume / respiration rate is described, for example, in the following literature:Zhang Q., Chen X., Zhan Q., Yang T., Xia S. Respiration-based emotion recognition with deep learning. Comput. Ind. 2017;92-93:84-90. doi: 10.1016 / j.compind.2017.04.005. ・Skin surface temperature is described, for example, in the following literature: Nakanishi R., Imai-Matsumura K. Facial skin temperature decreases in infants with joyful expression. Infant Behav. Dev. 2008;31:137-144. doi: 10.1016 / j.infbeh.2007.09.001. ・Multimodal models are described, for example, in the following literature: Choi J.-S., Bang J., Heo H., Park K. Evaluation of Fear Using Nonintrusive Measurement of Multimodal Sensors. Sensors. 2015;15:17507-17533. doi: 10.3390 / s150717507.

[0050] [1.2 Configuration of Information Processing System] FIG. 1 schematically illustrates an example configuration of an information processing system 1 according to an embodiment of the present disclosure.

[0051] An information processing system 1 according to one embodiment includes an intervention effect quantification device 10 as an information processing device, and a server device 20. A plurality of intervention effect quantification devices 10 may be provided, for example, to quantitatively evaluate the intervention effect of each of a plurality of users (individual a, individual b, individual c, individual d, ...). The server device 20 aggregates information on the intervention effect from the plurality of intervention effect quantification devices 10.

[0052] The intervention effect quantification device 10 includes a stimulus presentation device 11, a biosensor 12, an information processing unit 13, a history recording unit 14, an analysis unit 15, and an information presentation unit 16. The intervention effect quantification device 10 may be provided with a plurality of stimulus presentation devices 11 and a plurality of biosensors 12.

[0053] The main parts of the intervention effect quantification device 10 according to one embodiment (the information processing unit 13, the history recording unit 14, the analysis unit 15, and the information presentation unit 16) may be configured by a computer including, for example, one or more central processing units (CPUs), one or more read-only memories (ROMs), and one or more random access memories (RAMs). In this case, the processing of the main parts of the intervention effect quantification device 10 according to one embodiment may be realized by one or more CPUs executing processing based on programs stored in one or more ROMs or RAMs. Alternatively, the processing of the main parts of the intervention effect quantification device 10 according to one embodiment may be realized by one or more CPUs executing processing based on programs supplied from an external source via a wired or wireless network, for example.

[0054] The stimulus presentation device 11 presents a stimulus that can change an internal state. The stimulus presentation device 11 may be, for example, a smartphone, a PC (personal computer), a tablet terminal, a smart watch, an eyeglass display, a television, a theater screen, a theater speaker, or an in-vehicle device (an in-vehicle tablet, an in-vehicle speaker, etc.). The stimulus presentation device 11 outputs information on the stimulus start timing at which the stimulus is presented to the information processing unit 13.

[0055] The biosensor 12 detects bioinformation such as the user's brain waves, sweating state, pulse wave, electromyography, blood pressure, blood flow, or body temperature as a biosignal.

[0056] 3 to 8 show examples of devices equipped with the biosensor 12. For example, biosensor 12 can be placed on a finger, ear, head, arm, or chest to measure biometric information.

[0057] The biosensor 12 can be mounted on, for example, a head-mounted display (HMD) 200 as shown in Fig. 3. In the head-mounted display 200, the detection electrodes 203 of the biosensor 12 can be provided on, for example, the inner surfaces of the pad portion 201 and the band portion 202.

[0058] The biosensor 12 can be mounted on, for example, a headband 300 as shown in Fig. 4. In the headband 300, the detection electrodes 303 of the biosensor 12 can be provided on, for example, the inner surfaces of the band portions 301 and 302 that come into contact with the head.

[0059] The biosensor 12 can also be mounted on headphones 400 such as that shown in Fig. 5. In the headphones 400, the detection electrodes 403 of the biosensor 12 can be provided on the inner surface of a band 401 that comes into contact with the head, ear pads 402, or the like.

[0060] The biosensor 12 can be mounted on, for example, an earphone 500 as shown in Fig. 6. In the earphone 500, for example, a detection electrode 502 of the biosensor 12 can be provided on an earpiece 501 that is inserted into the ear.

[0061] The biosensor 12 can also be mounted on, for example, a watch 600 as shown in Fig. 7. In the watch 600, the detection electrodes 604 of the biosensor 12 can be provided on, for example, the inner surface of a display unit 601 that displays the time, or the inner surface of a band unit 602 (for example, the inner surface of a buckle unit 603).

[0062] The biosensor 12 can also be mounted on, for example, eyeglasses 700 as shown in Fig. 8. In the eyeglasses 700, the detection electrodes 702 of the biosensor 12 can be provided on the inner surfaces of temples 701, for example.

[0063] The biosensor 12 may also be mounted on, for example, a glove, a ring, a pencil, a pen, or a game console controller.

[0064] The information processing unit 13 estimates the internal state based on the biological signal, and calculates and quantifies the intervention effect due to the stimulus or situation.

[0065] FIG. 2 shows a schematic configuration example of the information processing unit 13 according to an embodiment.

[0066] The information processing unit 13 includes a data acquisition unit 31 , an internal state estimation unit 32 , and an intervention effect calculation unit 33 .

[0067] The data acquisition unit 31 acquires information on the stimulus start timing from the stimulus presentation device 11 and data on the biological signal from the biological sensor 12 .

[0068] The internal state estimation unit 32 is an estimation unit that estimates an internal state (for example, arousal level, pleasantness / unpleasantness) in response to a stimulus or situation, and calculates a time series indicating temporal changes in the internal state.

[0069] The internal state estimation unit 32 can easily determine whether the internal state is a comfortable or uncomfortable state from, for example, information about the person's body movement. In this case, the biosensor 12 detects, for example, the movement of the person's head as the body movement of the person.

[0070] The biosensor 12 may be, for example, a sensor that generates sensor data capable of detecting the user's body movement (amount of body movement). The biosensor 12 may be, for example, a wearable sensor worn on the user's head and configured as an acceleration sensor that detects acceleration. In this case, the biosensor 12 outputs a biosignal based on the detected acceleration data. Furthermore, for example, the biosensor 12 may be configured as an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. In this case, the biosensor 12 outputs a biosignal based on image data captured of the user. For example, when detecting body movement, the biosensor 12 may be any sensor capable of detecting the amount of body movement of the user, such as a speed sensor or a gyro sensor (angular velocity sensor).

[0071] The internal state estimation unit 32 determines the user's comfort / discomfort state based on, for example, the acceleration of the user's head as a biosignal. For example, if a single threshold is used to determine the comfort / discomfort state, the internal state estimation unit 32 determines the user's comfort state when the acceleration is greater than a predetermined threshold Tha, and determines the user's discomfort state when the acceleration is equal to or less than the predetermined threshold Tha. Alternatively, if a first threshold Thb1 for determining whether the user is in a comfort state and a second threshold Thb2 (Thb1>Thb2) for determining whether the user is in an discomfort state are separately set, the internal state estimation unit 32 determines the user's comfort state when the acceleration is greater than the first threshold Thb1, and determines the user's discomfort state when the acceleration is equal to or less than the second threshold Thb2. The predetermined threshold Tha, the first threshold Thb1, and the second threshold Thb2 may be determined based on past data obtained, for example, through experiments.

[0072] The intervention effect calculation unit 33 is a calculation unit that calculates a time series approximation line (see Figures 11, 14, and 15 described below) that shows the temporal change in the internal state calculated by the internal state estimation unit 32, and calculates the intervention effect due to the stimulus or situation based on the approximation line.

[0073] The intervention effect calculation unit 33 calculates a reference point on the approximation line (point P1 in Figure 11, point P2 in Figure 14, and point P3 in Figure 15, which will be described later) that minimizes the error between the time series and the approximation line, and uses the approximation line in the period from the stimulation start point or the end point of the situation change to the reference point (0≦t≦X^, which will be described later) as the approximation line for calculating the intervention effect.

[0074] The intervention effect calculation unit 33 calculates the intervention effect based on at least one of the value of the slope of the approximation line during the period from the start point of the stimulation or the end point of the situation change to the reference point, the amount of change in the internal state, and the ratio between the slope of the calculated approximation line and the reference approximation line.

[0075] The intervention effect calculation unit 33 may calculate the intervention effect for each of a plurality of stimuli or a plurality of situations.

[0076] The intervention effect calculation unit 33 sets an attention period (t=0 to T) from the stimulation start point or the end point of the situation change to a predetermined end time (t=T), and calculates an approximate line of the time series in the attention period.

[0077] The intervention effect calculation unit 33 may calculate an approximation line during the attention period, using a straight line including a first line A and a second line B (described later) (see FIG. 12 ). In this case, the intervention effect calculation unit 33 sets an arbitrary time during the attention period as a reference time X, sets a first line A in the period from the stimulation start point or the endpoint of the situation change to the reference time X, and sets a second line B in the period from the reference time X to a predetermined end time T. The intervention effect calculation unit 33 then calculates a first error δA between the time series and the first line A and a second error δB between the time series and the second line B, and sets the reference time X at which the sum of the first error δA and the second error δB is minimum as the reference time X^ for defining the intervention effect. The intervention effect calculation unit 33 sets the first line A^ in the period from the stimulation start point or the endpoint of the situation change to the reference time X^ as the approximation line for calculating the intervention effect.

[0078] Alternatively, the intervention effect calculation unit 33 may calculate a curved approximation line as the approximation line for the period of interest (see FIGS. 14 and 15 described below). The intervention effect calculation unit 33 may calculate, as a reference point, an inflection point on the curved approximation line that minimizes the error between the time series and the approximation line (point P2 in FIG. 14 and point P3 in FIG. 15 described below).

[0079] The history recording unit 14 is a recording unit that records information on the calculated intervention effects. When there are multiple calculated intervention effects, the history recording unit 14 may be configured to record information on multiple intervention effects.

[0080] The analysis unit 15 ranks the plurality of stimuli or the plurality of situations (intra-personal ranking) based on the plurality of pieces of information on the intervention effects. Furthermore, the analysis unit 15 may create recommended information for the user (such as information on stimuli or situations with high intervention effects) based on the plurality of pieces of information on the intervention effects.

[0081] The information presenting unit 16 is a presenting unit that presents information indicating the calculated intervention effect. The information presenting unit 16 presents information such as the evaluation of the intervention effect to the user via a UI (user interface) using a display device (see FIGS. 16 to 33, which will be described later). The information presenting unit 16 may present a list of ranked stimuli or situations together with the information indicating the intervention effect (see, for example, FIGS. 18, 21, and 24, which will be described later).

[0082] The server device 20 includes a cloud recording unit 21 , an analysis unit 22 , and an information presentation unit 23 .

[0083] The cloud recording unit 21 aggregates information on the intervention effects from the multiple intervention effect quantification devices 10 and records it as information on the intervention effect of the group.

[0084] The analysis unit 22 ranks a plurality of stimuli or a plurality of situations (ranks the intervention effects of a group) based on the plurality of pieces of information on the intervention effects from the plurality of intervention effect quantification devices 10. Furthermore, the analysis unit 15 may create recommended information for the user (such as information on stimuli or situations with high intervention effects) based on the plurality of pieces of information on the intervention effects.

[0085] The information presenting unit 23 provides the information presenting unit 16 with information on the UI to be presented on each of the information presenting units 16 of the multiple intervention effect quantification devices 10 .

[0086] [1.3 Operation of Information Processing System] Fig. 9 is a flowchart showing an example of operation of the information processing system 1 according to one embodiment. Fig. 10 is a flowchart showing an example of calculation processing of an approximation line in the information processing system 1. Figs. 11 and 12 are explanatory diagrams showing an example of calculation of an intervention effect in the information processing system 1.

[0087] First, the information processing unit 13 of the intervention effect quantification device 10 acquires information for estimating the internal state (step S101). The data acquisition unit 31 of the information processing unit 13 acquires information for estimating the internal state, including information on the start timing of a stimulus from the stimulus presentation device 11 and data on a biosignal from the biosensor 12. The data acquisition unit 31 acquires information for estimating the internal state from at least a point immediately before the start of a stimulus or an end point of a situation change. The data acquisition unit 31 acquires the start of a stimulus or an end point of a situation change, for example, by time-synchronizing the operation timing of the stimulus presentation device 11 with the detection timing of the biosensor 12. For example, when music is presented as a stimulus, the data acquisition unit 31 acquires the information by time-synchronizing the operation timing of a music player serving as the stimulus presentation device 11 with the detection timing of the biosensor 12. The operation timing of the music player may be voice recognition or, analogically, the timing when the user directly presses a button on the UI. Furthermore, when calculating the intervention effect of a life log, the start point of the stimulation or the end point of the situation change may be, for example, the timing when the user presses the UI of the stimulation presentation device 11 or the timing when a photo is taken with a smartphone serving as the stimulation presentation device 11. Furthermore, if the stimulation presentation device 11 is equipped with a GPS (Global Positioning System), the start point of the stimulation or the end point of the situation change may be a timing linked to the GPS (such as the timing when the user moves to a specific location).

[0088] Next, the internal state estimation unit 32 of the information processing unit 13 estimates the internal state in response to the stimulus or situation and calculates a time series indicating the change in the internal state over time (step S102). The method for estimating the internal state by the internal state estimation unit 32 is not particularly limited.

[0089] Next, the intervention effect calculation unit 33 of the information processing unit 13 sets an attention period (step S103). The intervention effect calculation unit 33 sets the attention period from the start point of the stimulus or the end point of the situation change (t=0) to a predetermined end time (t=T) when attention to the effect ends (see FIG. 12). The attention period may be determined from the history of the same type of stimulus, or an approximate time may be set in advance.

[0090] Next, the intervention effect calculation unit 33 of the information processing unit 13 calculates an approximate line of the time series (step S104).

[0091] 10 and 12 show an example of the process of calculating an approximate line by using a straight line.

[0092] First, the intervention effect calculation unit 33 sets the reference time as X, determines the period 0≦X≦T, and defines a first line A in the period 0≦t≦X and a second line B in the period X≦t≦T (step S201).

[0093] Next, the intervention effect calculation unit 33 defines a first line A^ that minimizes the first error δA between the time series indicating the internal state and the first line A over the period 0≦t≦X (step S202). Next, the intervention effect calculation unit 33 defines a second line B^ that minimizes the second error δB between the time series indicating the internal state and the second line B over the period X≦t≦T (step S203). The first error δA and the second error δB may be calculated using, for example, the mean absolute error (MAE) or the root mean squared error (RMSE), or any appropriate metric may be selected. FIG. 13 shows MAE as an example of a method for calculating the error.

[0094] Next, the intervention effect calculation unit 33 adds the first error δA and the second error δB (step S204). Next, the intervention effect calculation unit 33 moves X within the period 0≦X≦T and selects a time X = X^ at which the sum of the two errors, the first error δA and the second error δB, is minimized (step S205). The intervention effect calculation unit 33 sets this time X^ as the reference time (the time of reference point P1) for defining the intervention effect. Note that if there is no X = X^ at which the sum of the two errors is minimized, the intervention effect calculation unit 33 may increase the period of interest (end time T), for example by doubling it, and recalculate. Alternatively, the processing may be terminated as the intervention effect cannot be defined. The end time T defining the period of interest may be arbitrarily selectable.

[0095] Next, the intervention effect calculation unit 33 sets the first straight line A^ in the period 0≦t≦X^ as an approximation line for calculating the intervention effect (step S206). The intervention effect calculation unit 33 calculates the intervention effect based on the calculated approximation line.

[0096] Next, the history recording unit 14 of the intervention effect quantification device 10 records information on the calculated intervention effect (step S105). The history recording unit 14 records, for example, the slope of the approximation line (e.g., the first line A^) during the period 0≦t≦X^ (see FIG. 11 ) and the amount of change in the internal state during the period 0≦t≦X^ (see FIG. 11 ) as the intervention effect. Instead of the slope of the approximation line, the history recording unit 14 may record, as the intervention effect, the ratio of the slope of the reference approximation line to the slope of a reference approximation line obtained when white noise is used as the reference stimulus. For example, the intervention effect calculation unit 33 defines the absolute value of the slope as the intervention effect. If there are intervention effects with the same slope, the intervention effect calculation unit 33 defines the intervention effect as the amount of change in the internal state. For example, in FIG. 11 , the intervention effect calculation unit 33 may define the range of values ​​indicating the internal state on the vertical axis as 0 to 1, for example, and may define the unit as any unit. The intervention effect calculation unit 33 may define, for example, the level of arousal as an internal state from 0 to 1. When indicating pleasantness and unpleasantness as internal states, the intervention effect calculation unit 33 may define only pleasantness from 0 to 1, only unpleasantness from 0 to 1, or both pleasantness and unpleasantness from 0 to 1. The intervention effect calculation unit 33 may also define other internal states from 0 to 1. For example, in FIG. 11 , the unit of time on the horizontal axis may be appropriately selectable from seconds, minutes, hours, or days. Minutes have fewer digits and are easier to read.

[0097] Next, the analysis unit 15 of the intervention effect quantification device 10 analyzes the intervention effect (step S106). Next, the information presentation unit 16 of the intervention effect quantification device 10 presents the intervention effect via a UI (step S107).

[0098] In addition, depending on the type of intervention effect, the intervention effect quantification device 10 may select whether to use the statistical value of the intervention effect based on the state immediately before the start of stimulation or the change in situation. For example, if the intervention effect is a relaxation effect, the state of high arousal immediately before the start of stimulation or the change in situation may be used as the statistical value of the intervention effect. Also, the cumulative arousal level may be used when it is equal to or greater than a certain value.

[0099] (Modification of Approximation Line) FIGS. 14 and 15 show modifications of calculation of intervention effect in the information processing system 1 according to one embodiment.

[0100] The intervention effect calculation unit 33 of the information processing unit 13 may calculate an approximate line (approximate curve C) using a curve (for example, spline approximation or polynomial approximation) as the approximate line. As in the case of calculating the approximate line using a straight line described above, the intervention effect calculation unit 33 determines the period 0≦t≦T and calculates the approximate curve C using, for example, spline approximation or polynomial approximation.

[0101] As shown in Figure 14, when evaluating a downward point of a relaxation effect or the like, the point among the inflection points on the approximate curve C where the value indicating the internal state on the vertical axis is the lowest is selected as reference point P2. As shown in Figure 15, when evaluating an upward point of an excitement effect or the like, the point among the inflection points on the approximate curve C where the value indicating the internal state on the vertical axis is the highest is selected as reference point P3. The intervention effect calculation unit 33 calculates the intervention effect based on at least one of the value of the slope of the approximate curve C during the period from the stimulation start point or the endpoint of the situation change to reference point P2 (or reference point P3), the amount of change in the internal state, and the ratio of the slope of the calculated approximate curve C to the reference approximate line.

[0102] (Examples of UI Presenting Information Indicating Intervention Effect) Figures 16 to 33 show examples of UI screens using evaluation information of the intervention effect calculated by the information processing system 1 according to one embodiment. The UI screens are displayed by the information presentation unit 16. Figures 16 to 33 show examples in which evaluation information of the intervention effect is displayed when music is presented as a stimulus by the stimulus presentation device 11. Figures 16 to 33 also show examples in which the intervention effect is a relaxation effect.

[0103] 16 shows an example of a UI screen displaying information 51 about a song presented as a stimulus, information 52 about a slope value indicating the intervention effect, and information 53 about a time axis of the intervention effect. The song information 51 may include, for example, the song title, the artist name, and information about the date and time when the song was listened to. In the example of FIG. 16, the time axis of the intervention effect can be selected between "minutes" and "seconds," but a longer time axis such as "hours" or "days" may also be selected as the time axis of the intervention effect.

[0104] FIG. 17 shows an example in which a graph 54 showing the intervention effect is displayed on the UI screen in addition to the example of FIG.

[0105] 18 shows an example of a UI screen displaying a ranking list 55 of intervention effects by music pieces when gradient is used as the intervention effect. In the example of Fig. 18, the ranking list 55 of intervention effects by music pieces is displayed as a ranking list of intervention effects by music pieces that the user has personally listened to in the past.

[0106] 19 shows an example of a UI screen displaying information 51 about the music piece presented as a stimulus and information about a ratio value indicating the intervention effect 56. Here, information about the ratio value indicating the intervention effect 56 indicates, for example, the ratio value (S2 / S1) of a slope S2 indicating the intervention effect when the music piece to be actually evaluated is used to a slope S1 indicating the intervention effect when white noise is used as the reference music piece.

[0107] FIG. 20 shows an example in which a graph 54 showing the intervention effect is displayed on the UI screen in addition to the example of FIG.

[0108] 21 shows an example of a UI screen displaying a ranking list 57 of intervention effects by music pieces when the ratio to the intervention effect of white noise is used as the value indicating the intervention effect. In the example of FIG. 21, the ranking list 57 of intervention effects by music pieces is displayed as a ranking list of intervention effects by music pieces that the user has personally listened to in the past.

[0109] 22 shows an example in which, in addition to the example in FIG. 16, both slope value information 52 and change amount value information 58 are displayed on the UI screen as intervention effects. Note that, instead of the slope value information 52, the ratio of the white noise to the intervention effect (see FIG. 19 ) may be displayed together with the change amount value information 58.

[0110] FIG. 23 shows an example in which a graph 54 showing the intervention effect is displayed on the UI screen in addition to the example of FIG.

[0111] 24 shows an example of a UI screen displaying a ranking list 59 of intervention effects by music pieces when both slope and amount of change are used as the intervention effect. In the example of FIG. 24, a ranking list of intervention effects by music pieces that the user has personally listened to in the past is displayed as the ranking list 59 of intervention effects by music pieces. Note that instead of slope, the ratio to the intervention effect of white noise may be used as the intervention effect.

[0112] FIG. 25 shows an example of a UI screen displaying a ranking list 60 of intervention effects by music pieces when a slope is used as the intervention effect. In the example of FIG. 25, a ranking list of intervention effects by music pieces that the user has listened to in the past is displayed as the ranking list 60 of intervention effects by music pieces. In addition, in the example of FIG. 25, any music piece that the user wants to listen to is displayed in a selectable manner using a check box from the ranking list 60 of intervention effects by music pieces. Note that the amount of change or the ratio of the intervention effect of white noise to the intervention effect may be used as the intervention effect. Furthermore, instead of a ranking list for the individual user, a ranking list of a group whose intervention effect is similar to that of the user may be displayed in a selectable manner by the user.

[0113] FIG. 26 shows an example of a UI screen for creating a list of intervention effects equal to or greater than a predetermined value. In the example of FIG. 26, a slope value 61 indicating the intervention effect is displayed on the UI screen so that it can be changed to any value. Note that the value indicating the intervention effect may also be displayed as a change amount or a ratio of the intervention effect of white noise, so that it can be changed to any value. The example of FIG. 26 shows an example of creating a user's personal relaxation song list by selecting songs with a slope value of 0.5 or greater from among multiple songs that the user has personally listened to in the past.

[0114] 27 shows an example of a UI screen displaying information indicating what kind of music has the greatest intervention effect for other users whose intervention effects are similar to those of the user. This allows so-called recommendations to be made to the user.

[0115] FIG. 28 shows an example of a UI screen displaying a ranking list 62 of intervention effects by music when gradient is used as the intervention effect. In the example of FIG. 28, a ranking list of intervention effects for multiple users (groups) is displayed as the ranking list 62 of intervention effects by music. Note that the amount of change or the ratio to the intervention effect of white noise may be used as the intervention effect. The value indicating the intervention effect may be, for example, the average or median of the intervention effects of multiple users. Note that information on the intervention effect of the group can be acquired from the server device 20.

[0116] FIG. 29 shows an example of a UI screen for creating a list of intervention effects equal to or greater than a predetermined value. In the example of FIG. 29, the UI screen displays both a slope value 63 and a change amount value 64 as intervention effects, which can be changed to any value. Note that, instead of the slope value 63, a value indicating the intervention effect may be displayed that is changeable as the ratio of the intervention effect of white noise to the slope value 63, along with the change amount value 64. The example of FIG. 29 shows an example of creating a user's personal relaxation song list by selecting songs with a slope value of 0.5 or greater and a change amount of 0.40 or greater from among multiple songs that the user has personally listened to in the past.

[0117] FIG. 30 shows an example of a UI screen displaying a ranking list 66 of song intervention effects when both slope and amount of change are used as the intervention effect. In the example of FIG. 30 , the ranking list 66 of song intervention effects displays a ranking list of intervention effects for multiple users (groups) with a certain attribute. Note that the amount of change or the ratio to the intervention effect of white noise may be used as the intervention effect. The value indicating the intervention effect may be, for example, the average or median of the intervention effect for multiple users. Note that information on the intervention effect of the group can be acquired from the server device 20. In the example of FIG. 30 , group attribute information 65 is displayed in a selectable manner. For example, gender and age are displayed as the group attribute information 65. This makes it possible to display a list of multiple users (groups) with attributes selected as the group attribute information 65 as the ranking list 66 of song intervention effects. In the example of FIG. 30 , both male and female gender and teenagers are selected as group attributes for age.

[0118] FIG. 31 shows an example of a UI screen displaying information 68 on the intervention effect of a piece of music presented as a stimulus. In the example of FIG. 31 , information on a plurality of users (groups) having a certain attribute is displayed as the information 68 on the intervention effect of the piece of music. Note that the information on the intervention effect of the group can be acquired from the server device 20. In the example of FIG. 31 , a graph showing a slope and the intervention effect is displayed as the information 68 on the intervention effect of the piece of music. In the example of FIG. 31 , group attribute information 67 is displayed in a selectable manner. For example, gender and age are displayed as the group attribute information 67. This makes it possible to display information on a plurality of users (groups) having attributes selected as the group attribute information 67 as the information 68 on the intervention effect of the piece of music. In the example of FIG. 31 , both male and female gender and teenagers are selected as the group attributes for age. In the example of FIG. 31 , the intervention effect information 68 indicates that 75% of the group having the selected attribute experienced a relaxation effect with a slope of 0.5 or more.

[0119] Fig. 32 shows an example of a UI screen displaying a ranking list 71 of intervention effects by songs when both slope and amount of change are used as the intervention effect. In the example of Fig. 32, the ranking list 71 of intervention effects by songs is displayed two-dimensionally, with the horizontal axis representing amount of change and the vertical axis representing slope, as a ranking list of intervention effects by songs that the user has personally listened to in the past. This makes it easier for the user to visually evaluate multiple songs. Note that instead of slope, the ratio to the intervention effect of white noise may be used as the intervention effect.

[0120] FIG. 33 shows an example of a UI screen displaying a ranking list 72 of song intervention effects when both slope and amount of change are used as the intervention effect. In the example of FIG. 33 , the ranking list 72 of song intervention effects is a two-dimensional display of the intervention effects for multiple users (groups) with a certain attribute, with the amount of change on the horizontal axis and the slope on the vertical axis. This makes it easier for users to visually evaluate multiple songs. The amount of change or the ratio of the intervention effect to white noise may be used as the intervention effect. The value indicating the intervention effect may be, for example, the average or median of the intervention effects for multiple users. Information on the intervention effect of the group can be acquired from the server device 20. In the example of FIG. 33 , group attribute information 65 is selectably displayed. For example, gender and age are displayed as the group attribute information 65. This allows the ranking list 72 of song intervention effects to display a list for multiple users (groups) with an attribute selected as the group attribute information 65. In the example of FIG. 33, both male and female gender and teenagers age are selected as attributes of the group.

[0121] [1.4 Effects] As described above, the information processing system 1 according to one embodiment calculates a time-series approximation line that indicates a temporal change in an internal state in response to a stimulus or a situation, and calculates an intervention effect due to the stimulus or situation based on the calculated approximation line. This makes it possible to quantitatively evaluate the intervention effect due to the stimulus or situation.

[0122] (Examples of Use of Evaluation of Intervention Effect) According to the information processing system 1 of one embodiment, for example, it is possible to quantitatively define and evaluate the intervention effect from the endpoint of a stimulus such as entertainment content or an environmental change (change in situation). Furthermore, according to the information processing system 1 of one embodiment, it is possible to quantitatively define and evaluate the intervention effect of music, the intervention effect of a game scene, the intervention effect of a movie, TV (television), and streaming video content, the effect of switching between automated driving and manual driving, the effect of an event in a life log, the effect of a drug or anesthesia (including the effect of a sleeping pill), etc.

[0123] Specifically, the intervention effect can be evaluated as follows:

[0124] Entertainment intervention effects (content creation, user enjoyment purposes) For example, the intervention effects of music can be evaluated to assess the effects of relaxation / excitement, and pleasant / unpleasant feelings after listening to music. The intervention effects of games can be evaluated to assess the impact of specific scenes in horror, shooting, and racing games. The intervention effects of video content such as movies, TV, and streaming can be evaluated to assess reactions to specific scenes.

[0125] - Driving intervention effect (safety purpose) For example, when autonomous driving ends, it will be possible to check whether the level of alertness increases, and if it does not increase suddenly, it will be possible to notify (sound, dashboard UI, light, vibration, etc.) that the driver is not paying enough attention. It will also be possible to notify if the level of alertness does not increase when switching to a situation other than driving that requires attention.

[0126] - Life log event intervention effects (purpose of use for user enjoyment) It is possible to record as intervention effects whether the effects of an event (a beautiful view, a meal, etc.) were sudden or gradual, and whether the pleasantness, unpleasantness, or arousal level changed.

[0127] - Effects of drug and anesthetic interventions (including sleeping pills, opioids, cannabis, stimulants, etc., for use by doctors for follow-up observation) It will be possible to record whether the effects of drugs on awakening, relaxation, cravings, and pain are sudden or gradual.

[0128] According to the information processing system 1 of one embodiment, as the number of intervention effect cases increases in all of the above examples, individuals can create objective rankings of stimuli based on the magnitude of the effect on their internal state (e.g., a personal relaxing music list for music). This can be utilized by both music producers and users. For example, in the case of music, users can create a ranking list of songs above a certain value to listen to with the aim of listening to effective songs. Furthermore, users can select songs they want to listen to from a ranking list created based on their personal history. For example, they can select songs ranked higher than a certain level as the songs they want to listen to. Furthermore, producers can understand individual preferences and use this information for marketing purposes. Furthermore, not only users but also content producers (creators) such as artists can check the results of the intervention effect and confirm whether they were able to intervene in emotions as intended. The producers can then reflect these results in their content production (composition, etc.).

[0129] Furthermore, according to the information processing system 1 of one embodiment, by collecting group information in the server device 20, it is possible to evaluate the impact of an intervention on a group, to understand individual differences due to the intervention effect, and to identify similar user groups. In this case, utilization is possible both on the music production side and the user side. For example, users can use rankings of similar individuals or groups. Based on the group results, producers can present recommended songs that have not yet been listened to by people with similar ranking history. When used for marketing, it is possible to create a relaxing music album based on the objective history of intervention effects for a group.

[0130] The effects described in this specification are merely examples and are not limiting, and other effects may also be achieved. The same applies to the effects of other embodiments described below.

[0131] 2. Other Embodiments The technology according to the present disclosure is not limited to the description of the above embodiment, and various modifications are possible.

[0132] For example, the present technology can be configured as follows. According to the present technology configured as follows, an approximate line of a time series showing a temporal change in an internal state in response to a stimulus or a situation is calculated, and an intervention effect due to the stimulus or situation is calculated based on the calculated approximate line. This makes it possible to provide an information processing system capable of quantitatively evaluating the intervention effect due to a stimulus or a situation.

[0133] (1) An information processing system comprising one or more information processing devices, wherein the information processing devices have: an estimation unit that estimates an internal state in response to a stimulus or a situation and calculates a time series showing a temporal change in the internal state; and a calculation unit that calculates an approximation line of the time series and calculates an intervention effect due to the stimulus or the situation based on the approximation line. (2) The calculation unit calculates a reference point on the approximation line that minimizes an error between the time series and the approximation line, and uses the approximation line in a period from a stimulus start point or an endpoint of the situation change to the reference point as the approximation line for calculating the intervention effect. (3) The information processing system described in (2) above, wherein the calculation unit calculates the intervention effect based on at least one of a value of a slope of the approximation line in a period from the stimulus start point or endpoint of the situation change to the reference point, an amount of change in the internal state, and a ratio of a slope of the calculated approximation line to a reference approximation line. (4) The information processing system according to any one of (1) to (3) above, wherein the information processing device further includes a recording unit that records information on the calculated intervention effect. (5) The information processing system according to (4) above, wherein the calculation unit calculates an intervention effect for each of the plurality of stimuli or the plurality of situations, and the recording unit records information on the calculated plurality of intervention effects. (6) The information processing system according to (5) above, further including an analysis unit that ranks the plurality of stimuli or the plurality of situations based on information on the plurality of intervention effects. (7) The information processing system according to (6) above, wherein the information processing device further includes a presentation unit that presents information indicating the calculated intervention effect. (8) The information processing system according to (7) above, wherein the presentation unit presents a list of the ranked plurality of stimuli or the plurality of situations together with information indicating the intervention effect. (9) The information processing system according to (7) or (8) above, further comprising: a server device that includes a plurality of the information processing devices, and that aggregates the plurality of pieces of information on the intervention effect from the plurality of the information processing devices and records the information on the intervention effect of a group.(10) The information processing system according to any one of (1) to (9), wherein the presentation unit presents information on the intervention effect of the group aggregated in the server device. (11) The information processing system according to any one of (2) to (10), wherein the calculation unit sets an attention period from the stimulation start point or an endpoint of the situation change to a predetermined end time, and calculates the approximation line of the time series in the attention period. (12) The calculation unit calculates, as the approximation line in the attention period, an approximation line formed by a straight line including a first straight line and a second straight line. (13) The information processing system according to (12), wherein the calculation unit sets an arbitrary time in the attention period as a reference time, sets the first straight line in a period from the stimulation start point or an endpoint of the situation change to the reference time, and sets the second straight line in a period from the reference time to the predetermined end time. (14) The information processing system described in (13) above, wherein the calculation unit calculates a first error between the time series and the first straight line and a second error between the time series and the second straight line, and sets the base time at which the sum of the first error and the second error is minimum as a reference time for defining the intervention effect. (15) The calculation unit sets the first straight line in a period from the stimulation start point or an end point of the situation change to the reference time as an approximation line for calculating the intervention effect. (16) The information processing system described in (11) above, wherein the calculation unit calculates a curved approximation line as the approximation line during the attention period. (17) The information processing system described in (16) above, wherein the calculation unit calculates an inflection point on the curved approximation line that minimizes the error between the time series and the approximation line as the reference point.

[0134] This application claims priority based on Japanese Patent Application No. 2023-173496, filed on October 5, 2023, in the Japan Patent Office, the entire contents of which are incorporated herein by reference.

[0135] Those skilled in the art will recognize that various modifications, combinations, subcombinations, and variations may occur depending on design requirements and other factors, and are intended to be within the scope of the appended claims and their equivalents.

Claims

1. An information processing system comprising one or more information processing devices, the information processing devices having an estimation unit that estimates an internal state in response to a stimulus or situation and calculates a time series showing changes in the internal state over time, and a calculation unit that calculates an approximation line of the time series and calculates an intervention effect due to the stimulus or situation based on the approximation line.

2. The information processing system of claim 1, wherein the calculation unit calculates a reference point on the approximation line that minimizes the error between the time series and the approximation line, and the approximation line in the period from the start point of the stimulation or the end point of the situation change to the reference point is used as the approximation line for calculating the intervention effect.

3. The information processing system of claim 2, wherein the calculation unit calculates the intervention effect based on at least one of the value of the slope of the approximation line in the period from the stimulation start point or the end point of the situation change to the reference point, the amount of change in the internal state, and the ratio between the slope of the calculated approximation line and a reference approximation line.

4. The information processing system according to claim 1, wherein the information processing device further comprises a recording unit for recording information on the calculated intervention effect.

5. The information processing system according to claim 4, wherein the calculation unit calculates the intervention effect for each of the plurality of stimuli or the plurality of situations, and the recording unit records information on the calculated intervention effects.

6. The information processing system according to claim 5, further comprising an analysis unit that ranks the plurality of stimuli or the plurality of situations based on information on the plurality of intervention effects.

7. The information processing system according to claim 6, wherein the information processing device further comprises a presentation unit that presents information indicating the calculated intervention effect.

8. The information processing system according to claim 7, wherein the presentation unit presents a list of the ranked stimuli or situations together with information indicating the intervention effect.

9. The information processing system according to claim 7, further comprising a server device that includes a plurality of the information processing devices, and that aggregates the plurality of pieces of intervention effect information from the plurality of the information processing devices and records the information as the intervention effect information for a population.

10. The information processing system according to claim 9, wherein the presentation unit presents information on the intervention effect of the group aggregated in the server device.

11. The information processing system according to claim 2, wherein the calculation unit sets an attention period from the start point of the stimulus or the end point of the situation change to a predetermined end time, and calculates the approximation line of the time series during the attention period.

12. The information processing system according to claim 11, wherein the calculation unit calculates, during the period of interest, an approximation line based on straight lines including a first straight line and a second straight line as the approximation line.

13. The information processing system of claim 12, wherein the calculation unit sets an arbitrary time in the attention period as a reference time, sets the first straight line in the period from the start point of the stimulus or the end point of the situation change to the reference time, and sets the second straight line in the period from the reference time to the specified end time.

14. The information processing system of claim 13, wherein the calculation unit calculates a first error between the time series and the first line and a second error between the time series and the second line, and sets the base time at which the sum of the first error and the second error is minimum as a reference time for defining the intervention effect.

15. The information processing system according to claim 14, wherein the calculation unit regards the first straight line in the period from the stimulation start point or the end point of the situation change to the reference time as an approximation line for calculating the intervention effect.

16. The information processing system according to claim 11, wherein the calculation unit calculates a curved approximation line as the approximation line during the period of interest.

17. The information processing system according to claim 16, wherein the calculation unit calculates, as the reference point, an inflection point on the approximation line of the curve that minimizes an error between the time series and the approximation line.

Images