Information processing device, information processing method, and information processing program

Abstract

The objective of the present invention is to provide an information processing device, an information processing method, and an information processing program which can estimate an inner state of a subject from biometric information and guide the subject to an appropriate inner state.　One aspect of the present invention provides an information processing device comprising: an information acquiring unit that acquires subject biometric information including at least one of a biometric indicator and a behavioral indicator of a subject who is utilizing first processing with application software; an inner state estimating unit that estimates an inner state of the subject on the basis of the subject biometric information acquired by the information acquiring unit, and that obtains an inner state value quantitatively representing the inner state; and a processing execution unit that causes second processing different from the first processing to be executed on the basis of the inner state value obtained by the inner state estimating unit.

Description

Information processing device, information processing method, and information processing program

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program for determining psychological conditions such as emotions, arousal levels, and pleasant and unpleasant states based on biological information.

[0002] In recent years, methods for estimating emotions by combining biometric information and a psychological model have been proposed. Patent Literature 1 discloses an emotion estimation system for estimating the emotion of a subject, the system including: an input interface that receives a first biosignal corresponding to the subject's pulse and a second biosignal corresponding to the subject's electroencephalogram; a processor that converts a first dataset including a first value corresponding to the first biosignal and a second value corresponding to the second biosignal into a second dataset including a plurality of emotion values ​​corresponding to the amounts of a plurality of emotions arranged on a Russell circumplex model; a storage that stores colors associated with each of the plurality of emotions; and a display device that displays a coordinate plane formed by a first coordinate axis corresponding to the first biosignal and a second coordinate axis corresponding to the second biosignal, wherein the processor displays a marker at a position on the coordinate plane determined by the first dataset and a color associated with one of the plurality of emotion values ​​having the maximum value on the display device.

[0003] Patent Literature 2 discloses an emotion improving device that can instantly improve a user's emotions. This emotion improving device includes a stimulus determining unit that determines a predetermined stimulus to improve the user's emotions based on the user's emotions analyzed using physiological indices including the user's pulse rate and activity levels including the user's number of steps and amount of conversation, and a terminal control unit that controls a terminal device to provide the determined stimulus to the user.

[0004] Among the methods for estimating emotions from biological information, a method for estimating emotions in real time by interpreting emotions by associating heart rate variability indices, which represent a person's autonomic nervous activity, and electroencephalogram indices, which represent central nervous activity, with the Russell Circle model has been applied to various services (Non-Patent Documents 1 to 3). This method is believed to make it possible to detect anger or a state of concentration, such as brain activation, from the combination of electroencephalograms and heart rate, even without the person being aware of it.

[0005] On the other hand, there is a technology called biofeedback, which acquires a person's biological information, converts it into physical information, and then returns that physical information to the person. Biofeedback technology has been studied for many years and is widely applied in the field of medical treatment (Non-Patent Documents 4 and 5). Training using biofeedback technology allows people to consciously control their physiological state, such as regulating the autonomic nervous system.

[0006] By combining these technologies, it will be possible to estimate a person's emotions, pleasure / discomfort, level of arousal and concentration, and then communicate this information to the person through biofeedback, making it possible to control emotions, pleasure / discomfort, level of arousal and concentration.

[0007] JP 2020-185138 A JP 2022-169828 A

[0008] Takada, N., Laohakangvalvit, T., & Sugaya, M. (2022). Human Error Prediction Using Heart Rate Variability and Electroencephalography. Sensors, 22(23), 9194.Jadram, N., Anuardi, M. N. A. M., Laohakangvalvit, T., & Sugaya, M. (2021, June). Preliminary Experiment for Driver's Comfortable State Using EEG and HRV During Semi-autonomous Driving. In Advances in Human Aspects of Transportation: Proceedings of the AHFE 2021 Virtual Conference on Human Aspects of Transportation, July 25-29, 2021, USA (pp. 373-381). Cham: Springer International Publishing.Sripian, P., Mohd Anuardi, M. N. A., Kajihara, Y., & Sugaya, M. (2021). Empathetic robot evaluation through emotion estimation analysis and facial expression synchronization from biological information. Artificial Life and Robotics, 26, 379-389.Schoenberg, P. L., & David, A. S. (2014). Biofeedback for psychiatric disorders: a systematic review. Applied psychophysiology and biofeedback, 39(2), 109-135.Frank DL, Khorshid L, Kiffer JF, Moravec CS, McKee MG. Biofeedback in medicine: who, when, why and how? Ment Health Fam Med. 2010 Jun;7(2):85-91.

[0009] In modern society, controlling emotions and feelings such as anger is an extremely important issue. Inability to control anger leads to various social problems, such as troubles at nursing care and welfare facilities, accidents caused by aggressive driving, and online outrages. Anger management training to control anger is widely available, but most of it is symptomatic treatment that is left up to the individual.

[0010] When emotions such as anger arise, the individual may not be aware of them, so symptomatic treatment methods that are left to the individual may not be effective enough. Therefore, there is a need for appropriate solutions that can be easily used every day and that allow people to control emotions such as anger and their own state.

[0011] On the other hand, concentration is a biological state that involves the same brain functions. It is surprisingly difficult to concentrate when you want to concentrate and relax when you want to relax. For example, you may find yourself unable to concentrate on studying when you should. For this reason, there is a need to influence the appropriate internal state, such as concentrating when you want to concentrate and resting when you want to rest.

[0012] An object of the present invention is to provide an information processing device, an information processing method, and an information processing program that can estimate an inner state of a subject from biological information and lead the subject to an appropriate inner state.

[0013] One aspect of the present invention is an information processing device that includes an information acquisition unit that acquires subject biometric information including at least one of biometric indicators and behavioral indicators of a subject who is using a first process with application software; an internal state estimation unit that estimates the internal state of the subject based on the subject biometric information acquired by the information acquisition unit and obtains an internal state value that indicates the internal state; and a processing execution unit that executes a second process different from the first process based on the internal state value obtained by the internal state estimation unit.

[0014] According to this configuration, application software acquires subject information including at least one of biometric information and body movement information of a subject using a first process, and estimates the inner state of the subject using the first process. Based on an inner state value indicating this estimated inner state, a second process different from the first process is executed, making the subject aware of their current inner state and encouraging them to change or maintain their inner state.

[0015] In the information processing device, the second process may be a process for changing the inner state value of the subject who is using the first process, or a process for maintaining the inner state of the subject, whereby the application software guides the subject who is using the first process to change the inner state if it is better to change it, or to maintain the inner state if it is better not to change it.

[0016] In the information processing device, the first process may be a game, and the second process may be a process of increasing the difficulty level of the game in the first process when an internal state value indicating "anger" or "stress" among the internal states of the subject using the first process is equal to or greater than a predetermined threshold. As a result, if the subject feels "anger" or "stress" while playing the game, the difficulty level of the game is increased, causing the subject to feel even more "anger" or "stressed," and allowing the subject to quickly become aware of his or her internal state of "anger" or "stress."

[0017] In the information processing device, the second process may be a process of alerting a subject using the first process based on the inner state value, thereby proactively informing the subject of his or her own inner state and raising awareness of the subject.

[0018] In the information processing device, the second process may be a process of temporarily stopping the execution of the first process, thereby giving the subject time to objectively view their own inner state.

[0019] The information processing device may further include an external terminal that can be used by persons other than the subject, and the second process may be a process of outputting information based on the inner state value to the external terminal, thereby notifying persons other than the subject of the subject's inner state, which is difficult to notice, and allowing the persons other than the subject to take action based on the inner state.

[0020] In the information processing device, the processing execution unit may change the second processing based on a temporal change in the inner state value of the subject using the first processing, thereby dynamically performing the second processing in accordance with the temporal change in the inner state value.

[0021] Another aspect of the present invention is an information processing device that includes an information acquisition unit that acquires the acceleration of a writing action of a subject who is using a first process involving writing in application software; an internal state estimation unit that estimates the internal state of the subject based on the acceleration of the writing action acquired by the information acquisition unit and obtains an internal state value that indicates the internal state by a value; and a processing execution unit that executes at least one of encouraging concentration, encouraging a break, outputting a sound, outputting an image, outputting light, and outputting a scent as a second process different from the first process based on the internal state value acquired by the internal state estimation unit.

[0022] According to this configuration, the acceleration of the writing action of a subject using a first process involving writing in application software is acquired, and the internal state of the subject using the first process is estimated based on the acceleration of the writing action. Based on an internal state value indicating the estimated internal state, a second process different from the first process is executed, and at least one of encouraging the subject to concentrate, encouraging the subject to take a break, outputting a sound, outputting an image, outputting light, and outputting a scent is executed.

[0023] Another aspect of the present invention is an information processing method including an information acquisition step of acquiring subject biometric information including at least one of biometric indicators and behavioral indicators of a subject using a first process with application software; an internal state estimation step of estimating the internal state of the subject based on the subject information acquired in the information acquisition step and obtaining an internal state value indicating the internal state; and a processing execution step of executing a second process different from the first process based on the internal state value obtained in the internal state estimation step.

[0024] According to this configuration, a second process different from the first process is executed based on the inner state value obtained from the subject's biometric information, making the subject aware of his or her current inner state and encouraging the subject to change or maintain that inner state on his or her own.

[0025] Another aspect of the present invention is an information processing program that causes a computer to execute an information acquisition step of acquiring subject biometric information including at least one of biometric indicators and behavioral indicators of a subject who is using a first process with application software; an internal state estimation step of estimating the internal state of the subject based on the subject biometric information acquired in the information acquisition step and obtaining an internal state value that indicates the internal state in numerical terms; and a process execution step of executing a second process that is different from the first process based on the internal state value obtained in the internal state estimation step.

[0026] According to this configuration, a second process different from the first process is executed based on the inner state value obtained from the subject's biometric information, making the subject aware of his or her current inner state and encouraging the subject to change or maintain that inner state on his or her own.

[0027] According to the present invention, it is possible to provide an information processing device, an information processing method, and an information processing program that can estimate the inner state of a subject from biological information and lead the subject to an appropriate inner state.

[0028] 1 is a block diagram illustrating the configuration of an information processing device according to the present embodiment. FIG. 2 is a schematic diagram illustrating a specific example of an information processing device according to the present embodiment. FIG. 3 is a diagram illustrating Russell's circular model. FIG. 4 is a diagram illustrating Russell's circular model. FIG. 5 is a schematic diagram illustrating an example of application software executed on an external terminal. FIG. 6 is a schematic diagram illustrating cycles of game progress, difficulty level changes, and game resumption from pause. FIG. 7 is a flowchart illustrating an information processing method according to the present embodiment. FIG. 8 is a schematic diagram illustrating an example of acquiring behavioral indicators using a camera. FIG. 9 is a schematic diagram illustrating an example of acquiring behavioral indicators using a device having an acceleration sensor. FIG. 10 is a flowchart illustrating an information processing method using writing actions. FIG. 11 is a diagram illustrating an experimental state from measurement of biological information to feedback. FIG. 12 is a diagram illustrating changes in the rate of correct answers. FIG. 13 is a diagram illustrating changes in electroencephalograms (γ waves in the lateral frontal region AF8) before and after stimulation.

[0029] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following description, the same components are designated by the same reference numerals, and the description of components that have already been described will be omitted as appropriate.

[0030] 1 is a block diagram illustrating the configuration of an information processing device according to this embodiment. The information processing device 1 according to this embodiment has a function of estimating the inner state (psychological condition) of a subject using application software and guiding the subject to an appropriate inner state.

[0031] 1 , the information processing device 1 includes an information acquisition unit 10, an inner state estimation unit 20, and a processing execution unit 30. In addition to these, the information processing device 1 may also include an input unit 40, an output unit 50, a communication unit 60, and a control unit 70. The units of the information processing device 1 may be provided within the same housing, or at least one of them may be provided externally or connected via a network.

[0032] The information acquisition unit 10 acquires biometric information of the subject (subject biometric information) including at least one of biometric indicators and behavioral indicators of the subject. Here, the biometric indicators include values ​​indicating physiological phenomena such as heart rate, brain waves, respiratory rate, blood oxygen concentration, blood pressure, body temperature, and sweating. The body movement information includes values ​​indicating human behavior such as head and face movement, blinking, and facial expressions, as well as human body movement (body part motion such as hands and feet).

[0033] The internal state estimation unit 20 estimates the internal state of the subject based on the subject's biometric information acquired by the information acquisition unit 10, and obtains an internal state value that indicates the internal state as a value. The internal state of the subject includes the subject's emotions, arousal level, pleasantness / unpleasantness, stress, etc., and the internal state value includes these internal states quantified as a single number, coordinate values, a multidimensional data set, etc. The internal state of the subject may be estimated using technologies such as AI (artificial intelligence), machine learning, and deep learning.

[0034] The processing execution unit 30 performs processing to cause the application software to execute a second processing different from the first processing based on the inner state value obtained by the inner state estimation unit 20. The first processing executed by the application software executes processing that is not directly affected by the subject's inner state. The subject's inner state is estimated while using the first processing of the application software, and the second processing acts on the first processing based on this estimated inner state. In other words, the second processing executes processing that is directly affected by the subject's inner state. Note that the second processing may appear to be configured as part of the same application software as the first processing, or may be configured as software independent of the first processing.

[0035] The input unit 40 is, for example, a switch, a button, a touch panel, a camera, a microphone, etc. The output unit 50 is, for example, a speaker, a lamp, a vibration device, etc. The communication unit 60 includes a transmission unit 61 and a reception unit 62. The communication unit 60 transmits and receives information to and from external devices. The communication unit 60 may transmit and receive information via wireless communication or wired communication. The control unit 70 is, for example, a CPU that controls each unit.

[0036] (Specific Example of Information Processing Device) FIG. 2 is a schematic diagram showing a specific example of an information processing device according to the present embodiment. In the example shown in FIG. 2, the information processing device 1 is configured as an information processing system including a subject-side terminal 2 and an external terminal 3. The subject-side terminal 2 is a device including an information acquisition unit 10 that acquires subject biometric information, and examples of such devices include a heart rate / pulse monitor and an electroencephalograph that are attached to the subject. A specific example of the subject-side terminal 2 is a wristwatch-type portable information terminal (smart watch). The external terminal 3 is a device including an inner state estimation unit 20 and a processing execution unit 30, and examples of such devices include a mobile terminal, a tablet terminal, and a personal computer. A specific example of the external terminal 3 is a smartphone. Information is communicated between the subject-side terminal 2 and the external terminal 3 via, for example, wireless communication (such as Bluetooth (registered trademark) or Wi-Fi (registered trademark)).

[0037] A first process of application software is executed in the external terminal 3, and the subject uses the first process. A specific example of the first process is a game. In the information processing device 1, biometric information of the subject using the first process is acquired by the subject's terminal 2. For example, the subject's terminal 2 acquires biometric information of the subject using the first process, such as the heart rate, pulse wave, and brain wave, and transmits the acquired information to the external terminal 3 via wireless communication.

[0038] The external terminal 3 estimates the subject's internal state based on the subject's biometric information transmitted from the subject-side terminal 2. The internal state estimation unit 20 of the external terminal 3 predicts values ​​such as emotion, arousal level, pleasantness / unpleasantness, and stress from the subject's biometric information. This prediction uses a Russell circumplex model, as shown in FIGS. 3 and 4 . The Russell circumplex model shown in FIGS. 3 and 4 has a vertical axis corresponding to arousal level (Arousal) obtained from electroencephalograms and a horizontal axis corresponding to pleasantness / unpleasantness (Valence) obtained from heart rate, and represents a person's emotions and feelings (joy, anger, sadness, and happiness) using quadrants. As shown in FIG. 4 , in the Russell circumplex model, psychological conditions are associated with the direction and magnitude of vectors on the model. The internal state estimation unit 20 of the external terminal 3 may, for example, obtain an internal state value by quantifying the subject's internal state using the quadrant of the Russell circumplex model and the distance from the center.

[0039] Furthermore, it is also possible to distinguish, predict, and estimate internal state values ​​such as emotion, arousal level, pleasantness / unpleasantness, and concentration level, as well as trust, compatibility, emotion, relaxation, arousal, behrens, excitement, anger, and tension. This can be predicted using Russell's circumplex model based on the subject's biological information, or it can be predicted from the subject's direct behavior and facial expressions.

[0040] 5 is a schematic diagram showing an example of application software executed on an external terminal. An example of a first process of the application software executed on the external terminal 3 is a shooting game. The process execution unit 30 of the external terminal 3 executes, as a second process, a process of changing the difficulty level of the shooting game, which is an example of the first process. That is, as an example of the second process, the process execution unit 30 changes the difficulty level (game level) of the shooting game based on the internal state value of the subject (the subject playing the shooting game, which is the first process) obtained by the internal state estimation unit 20. For example, as the numerical value of anger, which is the internal state of the subject, increases, the enemy's movements become faster and the difficulty level increases.

[0041] When the difficulty of the game increases and the subject collides with an enemy, resulting in a game over, or when the subject wants to lower the difficulty of the game because they are about to collide with an enemy, they can use the pause function, which gives the subject time to objectively reflect on their own internal state.

[0042] While the game is paused, the subject calms down and performs an action (behavior) that reduces the value representing the emotion of anger. This action may be, for example, a calming action such as staying still, taking a deep breath, or talking to someone. The information acquisition unit 10 of the subject's terminal 2 sequentially acquires the subject's biometric information and transmits it to the external terminal 3. Even while the game is paused, the subject's biometric information is transmitted from the subject's terminal 2 to the external terminal 3, and the internal state estimation unit 20 of the external terminal 3 obtains the subject's internal state value. If the subject calms down while the game is paused and the internal state value decreases, the processing execution unit 30 performs a process to lower the game difficulty as a second process. If the subject resumes the game with the game difficulty reduced, the subject can more easily progress in the game.

[0043] Figure 6 is a schematic diagram illustrating the cycle of game progression, difficulty change, pause, and restart. As the subject progresses through the game, the time remaining until game over decreases, and if the subject does not progress well, stress builds up. This internal state can be estimated, for example, using Russell's circular model. As the anger level increases, the game's difficulty level increases. As the game's difficulty level increases, the subject is able to understand their own emotions and pause the game. By taking actions to calm themselves while the game is paused, the game's difficulty level decreases, and the subject is able to resume the game. When anger levels rise, the subject is able to understand how to lower their anger levels. By repeating this cycle, the subject can understand their own emotions and learn to control them when they are not good, becoming aware of and able to control their own emotions and feelings even without playing games in their daily lives.

[0044] In this example, the second process increases the game difficulty as the subject's emotional state, such as anger or stress, increases. In the Russell Circumplex model shown in FIG. 3 , if the subject's emotional state is in the "anger" quadrant, the second process increases the level of "anger" (for example, by increasing the game difficulty). This increases the subject's emotional "anger," making it easier for the subject to quickly recognize their own "anger" state. In other words, when performing anger management using the information processing device 1 according to this embodiment, the subject can quickly grasp their own "anger" state, thereby shortening the training time.

[0045] If the subject pauses the game and takes action to calm down, the level of "anger" in the internal state will decrease, and the second process will be executed to lower the difficulty of the game. If the subject resumes the game in this state, it will be easier to complete than before, and the subject will be able to become aware through the game that their internal state value of "anger" has decreased.

[0046] On the other hand, if the subject's internal state is in the "joy" or "pleasure" quadrant, a process for maintaining the game difficulty is executed as a second process in order to maintain that internal state. The subject can recognize that their own internal state is also stable because the game difficulty is stable. Here, if the subject's internal state is in the "joy" or "pleasure" quadrant, the game difficulty may be temporarily lowered. By temporarily lowering the game difficulty, the subject can more easily recognize that their own internal state is in the "joy" or "pleasure" quadrant. Note that after the temporary lowering of the difficulty, it is preferable to return the difficulty to its original level.

[0047] Nowadays, people have been letting their anger get the better of them, leading to trouble at nursing care and welfare facilities and accidents caused by aggressive driving, but by repeatedly undergoing this type of training, people will be able to become aware of and control their own emotions and feelings, which will help prevent trouble from occurring in the first place.

[0048] (Information Processing Method) FIG. 7 is a flowchart illustrating an information processing method according to this embodiment. Note that the flowchart shown in FIG. 7 illustrates an example in which a game is being played as the first process. First, as shown in step S101, subject biometric information is acquired (information acquisition process). For example, the subject wears a device that acquires biometric information. The device acquires the subject biometric information. Next, as shown in step S102, the subject biometric information is analyzed (internal state estimation process). Here, based on the subject biometric information, numerical values ​​indicating the subject's internal state, such as emotions, feelings, pleasantness / unpleasantness, arousal level, and stress, are obtained.

[0049] Next, as shown in step S103, it is determined whether the numerical value indicating the inner state is equal to or greater than a preset threshold. If the numerical value indicating the inner state is not equal to or greater than the threshold, a process of maintaining or lowering the game difficulty is executed as a second process (step S104). Next, it is determined whether the game has been cleared (step S105). If the game has been cleared, the process ends, and if the game has not been cleared, the process returns to step S101.

[0050] On the other hand, if it is determined in step S103 that the numerical value indicating the inner state is equal to or greater than the threshold value, a process for increasing the game difficulty level is executed as a second process (step S106). Here, the process for maintaining or decreasing the game difficulty level (step S104) and the process for increasing the game difficulty level (step S106) are process execution steps.

[0051] After the difficulty level of the game is increased in the process of step S106, it is determined whether or not the game has been cleared (step S107), and if it has been cleared, the process is terminated, and if it has not been cleared, it is determined whether or not an instruction to end the game or stop has been given (step S108).

[0052] If the game is over as determined in step S107 or if a stop command is given, the game is stopped (step S109); if the game is not over or if a stop command is not given, the process returns to step S101. After proceeding to step S109 and stopping the game, the subject takes an action to improve their emotions or feelings (step S110). Thereafter, as shown in step S111, the subject's biological information is acquired and analyzed. Then, as shown in step S112, it is determined whether the numerical value indicating the inner state is equal to or greater than a predetermined certain value or whether a restart command has been given. If the numerical value indicating the inner state is not equal to or greater than the certain value or if a restart command has not been given, the process returns to step S110. On the other hand, if the numerical value indicating the inner state is equal to or greater than the certain value or if a restart command has been given, the process returns to step S103.

[0053] In this way, taking a game as an example, the subject wears a device that acquires biometric information, and the subject's biometric information is acquired during the game. The information processing method analyzes the subject's biometric information and quantifies it into indicators such as emotions, feelings, pleasantness / unpleasantness, arousal level, and stress. If the numerical value (internal state value) is smaller than a threshold, the difficulty level of the game is maintained or decreased. On the other hand, if the numerical value is equal to or greater than the threshold, the difficulty level of the game is increased. In this case, the threshold may be changed according to the subject's individual state, or may be variable based on past numerical values. Furthermore, a mechanism may be incorporated in which the difficulty level increases over time or irregularly to inflict more stress.

[0054] The information processing method has a function of temporarily stopping the game when the player is unable to clear the game and receives some kind of game instruction, such as colliding with an enemy or being hit by a ball hit by an enemy, or when the subject directly instructs the player to stop. While the game is stopped, the subject takes an action to improve their emotions, feelings, stress, etc. This action may include staying still, taking a deep breath, talking to someone, etc., and the information processing method may output this as a recommendation from the output means of the device.

[0055] The game resumes when the numerical value (internal state value) falls below a certain value or when the subject issues a restart command. By repeating this process using the information processing method, the subject becomes aware of their own emotions and stress, and is able to control those emotions and stress themselves. The game itself ends (stops) when the game is cleared or when the subject issues an end command.

[0056] (Device Configuration Examples) Examples of device configurations applicable to the information processing device 1 and information processing method according to this embodiment include the following: Examples of electronic devices that acquire biometric information of a subject (subject biometric information) include smart watches, patch-type heart rate sensors worn on the arm or chest such as "MyBeat" (registered trademark), headband-type electroencephalogram devices worn on the head such as "Muse2" (product name), head-mounted display-type, and head-patch-type electroencephalogram devices. The electronic device that acquires the subject biometric information may be any one of these, or a combination of these.

[0057] The subject's biometric information acquired by the electronic device is processed by the electronic device itself or an external device such as a smartphone, tablet device, or personal computer, and is quantified as information such as emotions, feelings, stress, arousal level, and pleasantness / unpleasantness. Furthermore, the electronic device has an electronic device that performs a second process (e.g., adjusting the difficulty level of a game) for a first process (e.g., a game) based on the numerical information. This electronic device may be the same as the electronic device that acquires the subject's biometric information, or may be the same as the electronic device that executes the first process.

[0058] By using these components to process information, it becomes possible to implement gamification, for example, by encouraging people to change their behavior through games. Gamification can also enable more effective anger and stress management, at a level that is difficult or impossible for the target person to notice on their own.

[0059] (Other means for acquiring subject's biological information) In the above example, the bioindicators were heart rate and brain waves, but other bioindicators can also be used, such as pulse wave, electrocardiogram, skin conductance level, skin temperature, sweat rate, respiratory rate, pupil diameter, electromyogram, etc. These bioindicators are acquired from the subject using the first process and become the subject's biological information.

[0060] The above biomarkers are explained in detail below. Brain waves: Increased alertness increases brain wave frequency, particularly in the high-frequency range (such as beta waves). Brain waves reflect brain activity and alertness. Brain wave frequencies range from 0.1 Hz to 100 Hz, and these frequencies are known as alpha, beta, theta, delta, and gamma waves. Some examples of brain waves are listed below. Alpha waves (alpha waves): Frequency range is 8 to 13 hertz (Hz). Alpha waves are typically observed in relaxed states or with eyes closed and are associated with a relaxed, alert state. Beta waves (beta waves): Frequency range is 13 to 30 Hz. Beta waves are typically observed in states of heightened alertness, concentration, and cognitive activity. Gamma waves (gamma waves): Frequency range is typically above 30 Hz. Gamma waves are associated with higher brain functions such as cognition, learning, and information processing.

[0061] Skin Conductance Level (SCR): Increased arousal leads to an increase in the electrical skin response, which is a change in the electrical resistance of the skin and reflects increased sympathetic nervous activity.

[0062] Skin temperature: Increased arousal is usually accompanied by an increase in skin temperature. Sympathetic nervous activity causes vasoconstriction, lowering skin temperature.

[0063] Respiratory rate: Increased arousal leads to increased respiratory rate. Respiratory activity is closely related to the body's state of arousal.

[0064] Pupil diameter: As arousal increases, pupil diameter increases.

[0065] Electromyogram (EMG): Increased arousal leads to increased muscle tone and increased EMG activity, which reflects the degree of muscle tone in the body.

[0066] Furthermore, with regard to the heart rate exemplified above, indices such as pNN and HF / LF can be used to quantify the heart rate interval. pNN is the percentage of heart rates in which the difference between consecutive adjacent RR intervals exceeds a certain value. HF refers to the power spectrum of the frequency band from 0.15 to 0.4 Hz. FL refers to the power spectrum of the frequency band from 0.004 to 0.15 Hz. The LF / HF ratio refers to the ratio of the power of LF (low frequency) to HF (high frequency).

[0067] (Means for making a subject aware of internal state) One means for making a subject aware of values ​​(internal state values) that indicate their own internal state, such as their emotions, arousal level, pleasantness / unpleasantness, and stress, is to display or generate them via information on an electronic device. Specific examples include characters, images, colors, illustrations, sounds, music, scents, and vibrations. The characters, images, colors, and illustrations may be displayed on the same device that acquires the biometric information, or may be displayed on a separate device, such as a smartwatch, smartphone, tablet, personal computer, or television.

[0068] Regarding color, the color of the entire environment can be changed using a light-emitting device that can change its brightness and color in response to an external signal. Regarding sound and music, they can be played from the same device that acquires biometric information, or from a separate device such as a smartwatch, smartphone, tablet, personal computer, or television. This sound or music refers to the change in air vibrations from a silent state, and includes not only typical classical or pop music, but also white noise and sounds with frequencies inaudible to the human ear.

[0069] The vibrations may be emitted from the same device that acquires the biometric information, or from a separate device such as a smartwatch, smartphone, tablet, or personal computer.

[0070] Regarding scent, external devices such as aroma diffusers can be used to change the scent in response to external signals, but the scent can also be changed manually by the subject or another person who notices a change in the subject.

[0071] The means for making the subject aware of these internal state values ​​is also a process in which the second process prompts the subject to pay attention when the subject is using the first process, thereby proactively informing the subject of his or her internal state and making the subject aware of his or her internal state.

[0072] (Estimation of Inner State Based on Behavioral Indicators) When the information acquisition unit acquires the subject's biological information and the inner state estimation unit obtains the values ​​of emotion, pleasantness / unpleasantness, arousal level, and stress (inner state values), the inner state may be estimated from behavioral indices such as a person's behavior and facial expressions, in addition to directly measuring biological indices such as brain waves and heart rate. The estimation of the inner state based on the hardness index may use technologies such as AI, machine learning, and deep learning.

[0073] FIG. 8 is a schematic diagram illustrating an example of acquiring behavioral indicators using a camera. In the example shown in FIG. 8 , the subject's behavior and facial expressions are captured using, for example, the camera 4c of the mobile device 4, and the subject's internal state is predicted based on the captured image (video). The mobile device 4 is configured with a function for executing a first process using application software (information acquisition unit 10, information acquisition step), a function for predicting the subject's internal state from the subject's behavior and display captured by the camera 4c (internal state estimation unit 20, internal state estimation step), and a function for executing a second process different from the first process based on the internal state value (process execution unit 30, processing execution step). The second process may include outputting images, illustrations, videos, or text on the screen of the mobile device 4, outputting sounds, or controlling the game being played on the mobile device 4, for example, by adjusting the difficulty level.

[0074] In addition to the above, other devices for acquiring behavioral indicators of a subject include sensing devices equipped with an acceleration sensor that can be attached to the subject's body, such as ring- or bracelet-type devices.

[0075] 9 is a schematic diagram showing an example of acquiring a behavioral index using a device having an acceleration sensor. The device for acquiring the subject's behavioral index is a writing instrument 100 equipped with an acceleration sensor or an IoT (Internet of Things) device 110 attached to the writing instrument. A function for estimating the subject's internal state based on the behavioral index (e.g., an internal state estimation unit or an internal state estimation step) is provided in the writing instrument itself or in an external device 200 such as a smartphone, tablet terminal, or personal computer, where information is processed to obtain internal state values ​​such as concentration and brain arousal level.

[0076] Furthermore, the device that performs an operation (second process) such as playing music based on the internal state value may be the same as the device that acquires the behavioral index, or the same as the device that performs information processing to estimate the internal state, or may be the same as both of these.

[0077] Other means for acquiring writing actions include sensing devices equipped with an acceleration sensor attached to the body that moves when writing, such as a ring or bracelet-type device, or acquiring information from the trajectory of writing (direction, distance, speed, acceleration, etc.) when writing on a touch panel. Furthermore, the means for acquiring writing actions may be a method of capturing images of the writing with a camera to determine the writing action, or a method of acquiring pressure changes during writing using a pressure sensor such as a pressure sensor attached to a writing implement or a touch panel to determine the writing action.

[0078] For example, by using AI that uses the correlation between acceleration information in writing actions and the subject's behavioral indices as a learning model, it is possible to predict behavioral indices that represent the subject's concentration, such as head movement, eye movement, and blinking, from the acceleration information in writing actions.The subject's internal state can be estimated based on these predicted behavioral indices.

[0079] As a specific example, in the example shown in FIG. 9 , a behavioral index is acquired using an attachment-type IoT device 110 attached to a writing instrument 100. The attachment-type IoT device 110 attached to the writing instrument 100 has an acceleration sensor built in, and can acquire the acceleration of the writing action when the subject writes using the writing instrument 100 (including the acceleration when actually writing characters, etc., as well as the acceleration when holding the writing instrument 100 without writing characters, etc.). Information based on the acceleration of the writing action acquired by the IoT device 110 is transmitted to an external device 200 such as a mobile terminal via wireless communication, etc. The external device 200 predicts the subject's internal state from the information based on the acceleration sent from the IoT device 110 and obtains an internal state value. The external device 200 executes a first process as application software, executes a second process based on the internal state value, and executes a process according to the internal state.

[0080] An example of estimating internal states from information based on the acceleration of writing movements is to analyze the acceleration of writing movements while studying while writing, and determine internal states such as concentration, cognitive load, thinking ability, and logic from the acceleration of writing movements, or internal states based on the correlation between writing acceleration and brain waves. Here, concentration level is used as an example. Based on the determined concentration level, when the subject is concentrating on studying, nothing in particular is done, and when not concentrating, music that helps the subject concentrate is played from an external device such as a smartphone. For example, white noise, babbling brook, or classical music is played from the external device. This allows the subject to concentrate better. Furthermore, when the subject is concentrating for an excessively long period of time, the external device 200 such as a smartphone may prompt the subject to take a break and take measures to prevent fatigue (such as displaying text or images on the screen). This continues until the subject finishes studying.

[0081] Such an information processing device 1 can make the subject aware of their current state of concentration in learning or studying, encourage concentration, and lead them to effective learning or studying by displaying the concentration time or study time on the screen of the external device 200 or outputting images or music that encourage concentration. Also, if the concentration time exceeds a predetermined time, images or music that encourages a break or relaxation can be output, allowing the subject to study or study effectively with a balance between concentration and relaxation.

[0082] (Information Processing Method Using Writing Behavior) FIG. 10 is a flowchart illustrating an information processing method using writing behavior. First, as shown in step S201, behavioral indicators of writing behavior are acquired using an acceleration sensor, pressure sensor, etc., while the subject is studying, etc. Next, as shown in step S202, the subject's internal state is analyzed based on the acquired behavioral indicators of writing behavior. For example, technologies such as AI, machine learning, and deep learning are used to predict internal states such as concentration and brain arousal level, and an internal state value is obtained.

[0083] Next, as shown in step S203, it is determined whether the internal state value is equal to or greater than a threshold value. If the internal state value is equal to or greater than the threshold value, the operation of the external device is not changed (step S204). This makes it possible to avoid interrupting the state of concentration. Next, it is determined whether or not to continue the writing operation (step S205). If the writing operation is to be continued, the process returns to step S201; if not, the process ends.

[0084] On the other hand, if the determination in step S203 is that the internal state value is less than the threshold, the operation of the external device is changed (step S206). For example, the operation of the external device is changed to promote better concentration and brain awakening. One example of this operation is to play music that promotes concentration (white noise, babbling, classical music, etc.). By doing so, it is possible to change from a state of not concentrating to an operation that promotes concentration.

[0085] Next, it is determined whether or not to continue the writing action (step S207). If the writing action is to be continued, the process returns to step S201; if not, the process ends. That is, when the writing action is to be stopped (stop studying), this action is stopped, but at other times, the process of acquiring the writing action, making this determination, and maintaining or changing the operation of the external device is repeated. Note that when performing this process, it is also possible to perform a process of not changing the same operation even if the internal state value changes for a certain period of time so as not to change the operation of the external device too frequently. This may be a process of leveling out the numerical value by using a moving average of the change in the internal state value, or the like. This process continues until the end of studying.

[0086] (Other Means of Obtaining Behavioral Indicators) Other examples of means of obtaining behavioral indices are given below. Obtaining hand movements using a smartwatch Obtaining the subject's whole-body movements using a camera Obtaining the subject's facial expressions using a camera Obtaining information such as walking and running using acceleration information from a smartphone By applying the behavioral indices obtained in these ways to AI or the like, it becomes possible to infer and predict internal states such as emotion, arousal level, pleasantness / unpleasantness, concentration level, trust level, compatibility level, affect, relaxation, arousal, berenescence, excitement, anger, and tension.

[0087] (How to make people aware of their internal state) Methods for encouraging changes in internal state and behavioral changes, such as increasing concentration, include the following. Appealing to the visual sense: Methods for appealing to the visual sense include displaying the current internal state as text, images, illustrations, videos, and animations on the screen of a smartwatch, smartphone, tablet device, or personal computer. Another method is to change the color or brightness of the screen. Another method is to change the color or brightness of electricity or lights using light-emitting devices that can change the intensity and color of light based on external information. Another method is to understand people's preferences and display images, text, videos, illustrations, colors, etc. that they like or dislike.

[0088] Auditory stimulation: Methods for stimulating the auditory sense include playing sounds or music from a smartwatch, smartphone, tablet, personal computer, music player, headphones, or speakers. Other methods include sounds or music that promote concentration or relaxation, such as classical music, white noise, or rippling sounds. Favorite music or intense music is expected to have an effect of waking you up. It is also possible to combine a system that allows you to select music according to your preferences.

[0089] Stimulating the sense of smell: Citrus scents and pungent smells are known to wake you up. Scents such as lavender are also known to have a relaxing effect. The scents can be generated using an aroma diffuser. Alternatively, the subject can be instructed via a smartphone or tablet device to choose a perfume or air freshener suitable for changing their internal state. The subject can prepare it themselves by following the instructions on the screen.

[0090] Influencing the sense of taste: Food textures and tastes can wake you up. For example, chewing gum can wake you up. Another example is using a favorite food to raise your energy. To provide food textures, devices that stimulate the inside of the mouth can be used. Additionally, the subject can be instructed via a smartphone or tablet device to choose foods suitable for changing their internal state. The subject can prepare the food themselves by following instructions on the screen.

[0091] Influencing the sense of touch: One way to influence the sense of touch is to generate vibrations to stimulate, promote concentration, wake people up, and increase brain alertness. In this case, for example, vibrations can be applied via a smartwatch, smartphone, tablet, etc. However, even without vibration, electrical stimulation or devices that inflict pain can also be used to encourage behavioral change. The sense of touch includes stimulation of the skin and stimulation inside the mouth and ears.

[0092] Directly influencing the brain: One way to directly influence the brain is to insert electrodes into the brain and give direct stimulation. Stimulating the brain can change the subject's behavior. Electrical stimulation is one example of how to influence the brain.

[0093] (Other Examples of Device Operation Changes Based on Internal State Values) The above examples show how the device's operation remains unchanged depending on whether the internal state value is equal to or greater than a threshold, while changing (e.g., playing a sound) depending on whether the internal state value is equal to or greater than a threshold. However, the present invention is not limited to these examples and includes the following: The device may have a function for selecting different operations (different second processes) depending on whether the threshold is exceeded or not. There may be multiple thresholds, and the device may have a function for selecting different operations (different second processes) for each threshold. The different operations may include not changing (maintaining) the device's operation. The threshold may be a different value for each individual, or it may change dynamically based on changes in predicted values ​​including previous biometric information when the individual uses the device. This includes the threshold being optimized for each individual. For example, if an operation performed to promote concentration disrupts concentration, the threshold may be changed from the next time and no operation is performed when the previous threshold is reached. The device may have a mechanism for optimizing the device's operation, such as to promote concentration, for each individual. For example, if music played to promote concentration disrupts concentration or does not result in any change, the system may choose not to use that music in the future.

[0094] When it is assumed that these biometric and behavioral indicators are changed (second process) during learning / studying, examples of the subject's internal state to be grasped include the state of concentration, cognitive load, thinking ability, logical thinking, comfortable or uncomfortable, and stressed. Furthermore, examples of behavioral indicators during learning include the behavior / action of writing, the behavior / action of reading a book, the behavior / action of viewing learning content, thinking behavior / action such as solving a problem, memorizing behavior / action, typing or other input behavior / action, reading aloud or other vocal behavior / action, and listening to audio. These behaviors / actions can be obtained by capturing the physical quantities of the subject's tools (study tools, etc.) and the subject's appearance using input devices such as a camera or microphone. The subject's internal state during learning / studying can be grasped by correlating the acquired behaviors / actions with the internal state.

[0095] Furthermore, the processing execution unit 30 of the information processing device 1 or the processing execution step of the information processing method may change the second processing based on a change over time in the inner state value of the subject who is using the first processing. For example, the inner state value of the subject is continuously obtained, and the second processing is changed depending on whether the change over time in the inner state value is a vector based on Russell's ring model that changes rapidly or not. Furthermore, the degree of change in the second processing may be changed depending on the quadrant, such as which quadrant of Russell's ring model the change is in, or whether the change crosses quadrants.

[0096] (Example of Involving a Person Other Than the Subject) In the present embodiment, when the subject's internal state is estimated based on the subject's biometric information, the internal state or the internal state value may be notified to a person other than the subject (a third party). That is, the information processing device 1 and the information processing method may be configured so that a third party other than the subject intervenes when changing the estimated internal state of the subject (emotion, arousal level, pleasantness / unpleasantness, concentration level, trust level, compatibility level, emotion, relaxation, excitement, anger, tension, etc.). The information processing device 1 includes an external terminal that can be used by a third party other than the subject. The second process in the information processing device 1 is to output information based on the subject's internal state value to the external terminal that can be used by the third party.

[0097] As a specific example, a subject may study or work in a cafe while wearing a smartwatch or a small electroencephalograph. The subject's internal state, or changes in the internal state, are transmitted from the subject's smartwatch or other device and notified to an external device accessible to the cafe staff. If the subject begins to lose concentration or become sleepy, the cafe staff can grasp the subject's internal state based on the information notified to the external device and speak to the subject or offer them a drink. This can have the effect of restoring the subject's concentration or waking them up, thereby improving the efficiency of their studies or work. Note that while the above example uses a relationship between a cafe staff member and a customer, this could also be a parent-child relationship. This allows the subject's internal state, which may be difficult to notice, to be made known to someone other than the subject, allowing those other than the subject to take action based on the internal state.

[0098] (Experimental Example) Figure 11 shows the state of the experiment from measurement of biological information to feedback. In this experiment, the biological information obtained was the subject's brain waves and heart rate. In the experiment, auditory and visual stimuli were applied when the subject began to lose concentration while studying / learning, and it was confirmed whether this increased brain waves (beta waves and gamma waves) that indicate concentration and whether the rate of correct answers to the questions changed. The specific experimental method was to present calculation problems in English using sound, and to measure brain waves when answering the problems and examine changes in the rate of correct answers.

[0099] Fig. 12 is a diagram illustrating the change in the correct answer rate. Fig. 12 shows the change in the correct answer rate after concentration waned (later part of the questions) when no stimulus was given (No Stimuli) and when a sound stimulus (River Sound: babbling) was given. It can be seen that the correct answer rate decreased when no stimulus was given, whereas the correct answer rate increased when a sound stimulus (River Sound: babbling) was given.

[0100] Figure 13 shows an example of changes in electroencephalograms (γ waves in the lateral frontal region AF8) before and after stimulation. γ waves are most active when the brain is activated, that is, when the subject is concentrating. Without stimulation, γ waves decrease due to fatigue, making it difficult to concentrate. However, when sound stimulation (river sound: babbling) is given, γ waves increase, indicating greater concentration.

[0101] The experimental results shown in Figures 12 and 13 show that by identifying when concentration is waning during study and providing a stimulus in the form of the sound of a stream at that time, concentration improves and the rate of correct answers to questions also increases.

[0102] Note that this experimental data was statistically processed from data from multiple people, and the dots in Figure 12 and the error bars in Figure 13 represent variations among subjects. Furthermore, in the experiment, changes in EEG were continuously acquired by comparing them with previous data in a time series. The acquisition period for EEG comparison data can be effectively set between 0.01 seconds and 3 hours. Furthermore, the threshold for the amount of change in EEG can be averaged using a moving average or the like.

[0103] (Information Processing Program) The information processing program (application software) according to this embodiment has the following configuration.

[0104] An information processing program executed by a computer causes the computer to execute an information acquisition step of acquiring subject biometric information including at least one of biometric indicators and behavioral indicators of a subject who is using a first process with application software; an internal state estimation step of estimating an internal state of the subject based on the subject biometric information acquired in the information acquisition step and obtaining an internal state value indicating the internal state; and a process execution step of executing a second process different from the first process based on the internal state value obtained in the internal state estimation step.

[0105] The information processing program may be recorded on a recording medium or distributed via a network.

[0106] The information processing device 1, information processing method, and information processing program according to the present embodiment described above can be applied to various systems, such as anger management and stress management using games as the first processing, as well as a learning support system that notifies users when they are losing concentration, when high-frequency brain waves are decreasing, when they are becoming too relaxed, when they are becoming sleepy, etc., to encourage them to improve their learning efficiency.

[0107] Furthermore, the information processing device 1, information processing method, and information processing program that acquire the subject's behavioral index, estimate the subject's internal state, and guide the subject to an appropriate internal state are not limited to the acceleration of writing actions as a behavioral index, and can be applied to the acceleration of any subject's physical movements. For example, the acceleration of moving a toothbrush, the acceleration of the movements of body parts when playing a musical instrument or playing sports, the acceleration of the movements of various tools, the acceleration of the body movements when operating a vehicle such as an automobile, the acceleration of the movements of the steering, pedals, levers, etc. that are the objects of operation, the acceleration of the movements of elements such as the vehicle body, suspension, and links, etc. can be applied as behavioral indexes of the subject.

[0108] As described above, according to this embodiment, it is possible to provide an information processing device 1, an information processing method, and an information processing program that can estimate the inner state of a subject who is using application software and guide the subject to an appropriate inner state.

[0109] Although the present embodiment and its application examples have been described above, the present invention is not limited to these examples. For example, although a shooting game has been described as the first process, other games such as puzzles may also be used. Furthermore, those skilled in the art may appropriately add, delete, or modify components of the above-described embodiments or their application examples, or appropriately combine features of the embodiments, as long as they include the gist of the present invention. These additions, deletions, and design changes are also encompassed within the scope of the present invention.

[0110] REFERENCE SIGNS LIST 1 information processing device 2 subject-side terminal 3 external terminal 4 mobile terminal 4c camera 10 information acquisition unit 20 inner state estimation unit 30 processing execution unit 40 input unit 50 output unit 60 communication unit 61 transmission unit 62 reception unit 70 control unit 100 writing implement 110 IoT device 200 external device

Claims

1. An information processing device comprising: an information acquisition unit that acquires subject biometric information including at least one of biometric indicators and behavioral indicators of a subject using a first process with application software; an inner state estimation unit that estimates an inner state of the subject based on the subject biometric information acquired by the information acquisition unit and obtains an inner state value indicating the inner state; and a process execution unit that executes a second process different from the first process based on the inner state value obtained by the inner state estimation unit.

2. An information processing device according to claim 1, wherein the second process is a process for changing the inner state value of the subject who is using the first process.

3. An information processing device as described in claim 1, wherein the first processing is a game, and the second processing is processing for increasing the difficulty level of the game in the first processing when the inner state value indicating "anger" or "stress" among the inner states of the subject using the first processing is equal to or greater than a predetermined threshold value.

4. An information processing device according to claim 1, wherein the second processing is processing for maintaining the inner state value of the subject who is using the first processing.

5. An information processing device according to claim 1, wherein the second process is a process for alerting the subject using the first process based on the inner state value.

6. The information processing device according to claim 1, wherein the second process is a process for temporarily stopping the execution of the first process.

7. An information processing device as described in claim 1, further comprising an external terminal that can be used by persons other than the subject person, and the second process is a process of outputting information based on the inner state value to the external terminal.

8. An information processing device according to claim 1, wherein said process execution section changes said second process based on a time change in said inner state value of said subject utilizing said first process.

9. An information processing device comprising: an information acquisition unit that acquires the acceleration of a writing action of a subject who is using a first process involving writing on application software; an inner state estimation unit that estimates an inner state of the subject based on the acceleration of the writing action acquired by the information acquisition unit and obtains an inner state value indicating the inner state by a value; and a process execution unit that executes at least one of encouraging concentration, encouraging a break, outputting a sound, outputting an image, outputting light, and outputting a scent as a second process different from the first process based on the inner state value acquired by the inner state estimation unit.

10. An information processing method comprising: an information acquisition step of acquiring subject biometric information including at least one of biometric indicators and behavioral indicators of a subject using a first process with application software; an inner state estimation step of estimating an inner state of the subject based on the subject biometric information acquired in the information acquisition step and obtaining an inner state value indicating the inner state; and a process execution step of executing a second process different from the first process based on the inner state value obtained in the inner state estimation step.

11. An information processing program that causes a computer to execute the following steps: an information acquisition step of acquiring subject biometric information including at least one of biometric indicators and behavioral indicators of a subject using a first process with application software; an inner state estimation step of estimating an inner state of the subject based on the subject biometric information acquired in the information acquisition step, and obtaining an inner state value indicating the inner state in terms of a value; and a process execution step of executing a second process different from the first process based on the inner state value obtained in the inner state estimation step.

Images