Information processing device, information processing method, and information processing program

Abstract

To provide an information processing device, an information processing method, and an information processing program capable of leading an inner surface state to an appropriate one by estimating the inner surface state from biological information on an object person.SOLUTION: An information processing device includes: an information acquisition unit for acquiring object person biological information including at least either a biological index or a behavior index of an object person using first processing in application software; an inner surface state estimation unit for estimating an inner surface state of the object person on the basis of the object person biological information acquired by the information acquisition unit, and acquiring an inner surface state value showing the inner surface state by a value; and a processing execution unit for causing second processing difference from the first processing to be executed on the basis of the inner surface state value acquired by the inner surface state estimation unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program for determining psychological conditions such as emotions, arousal level, and comfort / discomfort based on biological information. [Background technology]

[0002] In recent years, a method for estimating emotions by combining bioinformation and a psychology model has been proposed. Patent Document 1 discloses an emotion estimation system for estimating the emotion of a subject, which includes an input interface for receiving a first biosignal corresponding to the pulse of the subject and a second biosignal corresponding to the brainwaves of the subject, a processor for converting a first dataset including a first value corresponding to the first biosignal and a second value corresponding to the second biosignal into a second dataset including a plurality of emotion values ​​corresponding to the amounts of a plurality of emotions arranged on a Russell ring model, a storage for storing colors associated with each of the plurality of emotions, and a display device for displaying a coordinate plane formed by a first coordinate axis corresponding to the first biosignal and a second coordinate axis corresponding to the second biosignal, and the processor displays a marker at a position on the coordinate plane determined by the first dataset and displays a color associated with one of the plurality of emotion values ​​having a maximum value on the display device.

[0003] Patent Document 2 discloses an emotion improving device that can instantly improve a user's emotion. This emotion improving device includes a stimulus determining unit that determines a predetermined stimulus to improve the user's emotion based on the user's emotion analyzed using physiological indices including the user's pulse rate and the amount of activity including the user's number of steps and conversation volume, and a terminal control unit that controls a terminal device to give the determined stimulus to the user.

[0004] Among the methods for estimating emotions from biological information, a method for estimating emotions in real time by interpreting emotions by associating heart rate variability indexes, which indicate a person's autonomic nervous activity, and electroencephalogram indexes, which indicate central nervous activity, with the Russell Circle Model has been applied to various services (Non-Patent Documents 1 to 3). With this method, it is believed possible to detect anger or a state of concentration, such as brain activation, from a combination of electroencephalograms and heart rate, even without the person being aware of it.

[0005] On the other hand, there is a technology called biofeedback that acquires a person's biological information, converts it into physical information, and returns the physical information to the person. Biofeedback technology has been researched for many years and is widely applied in the field of treatment (Non-Patent Documents 4, 5). By training using biofeedback technology, people can consciously control their physiological state, such as regulating the autonomic nervous system.

[0006] By combining these technologies, it will be possible to estimate a person's emotions, pleasure / displeasure, level of alertness and concentration, and then inform the person of this through biofeedback, making it possible to control those emotions, pleasure / displeasure, level of alertness and concentration. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2020-185138 A [Patent Document 2] JP 2022-169828 A [Non-patent literature]

[0008] [Non-Patent Document 1] Takada, N., Laohakangvalvit, T., & Sugaya, M. (2022). Human Error Prediction Using Heart Rate Variability and Electroencephalography. Sensors, 22(23), 9194. [Non-Patent Document 2] Jadram, N., Anuardi, M. N. A. M., Laohakangvalvit, T., & Sugaya, M. (2021, June). Preliminary Experiment for Driver's Comfortable State Using EEG and HRV During Semi-autonomous Driving. In Advances in Human Aspects of Transportation: Proceedings of the AHFE 2021 Virtual Conference on Human Aspects of Transportation, July 25-29, 2021, USA (pp. 373-381). Cham: Springer International Publishing. [Non-Patent Document 3] Sripian, P., Mohd Anuardi, M. N. A., Kajihara, Y., & Sugaya, M. (2021). Empathetic robot evaluation through emotion estimation analysis and facial expression synchronization from biological information. Artificial Life and Robotics, 26, 379-389. [Non-Patent Document 4] Schoenberg, PL, & David, AS (2014). Biofeedback for psychiatric disorders: a systematic review. Applied psychophysiology and biofeedback, 39(2), 109-135. [Non-Patent Document 5] Frank DL, Khorshid L, Kiffer JF, Moravec CS, McKee MG. Biofeedback in medicine: who, when, why and how? Ment Health Fam Med. 2010 Jun;7(2):85-91 Summary of the Invention [Problem to be solved by the invention]

[0009] In modern society, controlling emotions and feelings such as anger is a very important issue. Inability to control anger leads to various social problems such as troubles in nursing and welfare facilities, accidents caused by aggressive driving, and flame wars on social media sites. Anger management training to control anger is widely conducted, but most of it is just symptomatic treatment that is left to the individual.

[0010] When emotions such as anger occur, the individual may not be aware of it, so symptomatic treatment methods that are left to the individual may not be effective enough. Therefore, there is a need for appropriate solutions that can be easily used every day and that allow people to control emotions such as anger and their own state.

[0011] On the other hand, there is also concentration, which is a state that comes from the living body that includes the same brain function. It is surprisingly difficult to concentrate when you want to concentrate and relax when you want to relax. For example, you may find yourself unable to concentrate on your studies when you should. For this reason, there is a need to work on the appropriate internal state, such as concentrating when you want to concentrate and resting when you want to rest.

[0012] An object of the present invention is to provide an information processing device, an information processing method, and an information processing program capable of estimating an inner state of a subject from biological information and guiding the subject to an appropriate inner state. [Means for solving the problem]

[0013] One aspect of the present invention is an information processing device that includes an information acquisition unit that acquires subject biometric information including at least one of bioindicators and behavioral indices of a subject who is using a first process in application software, an inner state estimation unit that estimates an inner state of the subject based on the subject biometric information acquired by the information acquisition unit and obtains an inner state value that indicates the inner state by a value, and a process execution unit that executes a second process different from the first process based on the inner state value obtained by the inner state estimation unit.

[0014] According to such a configuration, subject information including at least one of biometric information and body movement information of a subject using a first process is acquired by the application software, and an inner state of the subject using the first process is estimated. Based on an inner state value indicating this estimated inner state, a second process different from the first process is executed, making the subject aware of his / her current inner state and encouraging him / her to change or maintain his / her inner state.

[0015] In the above information processing device, the second process may be a process for changing the inner state value of the subject who is using the first process, or a process for maintaining the inner state of the subject. This allows the application software to guide the subject who is using the first process to change the inner state when it is better to change the inner state, and to maintain the inner state when it is better not to change the inner state.

[0016] In the above information processing device, the first process may be a game, and the second process may be a process of increasing the difficulty of the game in the first process when an inner state value indicating "anger" or "stress" among the inner states of the subject using the first process is equal to or greater than a predetermined threshold. In this way, if the subject feels "anger" or "stress" while playing the game, the difficulty of the game increases, causing the subject to feel even more "anger" or "stressed," and allowing the subject to quickly become aware of his or her inner states of "anger" or "stress."

[0017] In the above information processing device, the second process may be a process of alerting a subject using the first process based on the inner state value, thereby proactively informing the subject of his / her own inner state and raising the subject's awareness.

[0018] In the information processing device, the second process may be a process of temporarily stopping the execution of the first process, thereby giving the subject time to objectively view his / her own inner state.

[0019] The information processing device may further include an external terminal that can be used by persons other than the subject, and the second process may be a process of outputting information based on the inner state value to the external terminal, whereby the inner state of the subject, which is difficult to notice, is made known to persons other than the subject, and a response based on the inner state is made by the persons other than the subject.

[0020] In the information processing device, the process execution unit may change the second process based on a change over time in the inner-state value of the subject who is using the first process, thereby dynamically performing the second process in response to the change over time in the inner-state value.

[0021] Another aspect of the present invention is an information processing device that includes an information acquisition unit that acquires an acceleration of a writing action of a subject who is using a first process involving writing on application software, an inner state estimation unit that estimates an inner state of the subject based on the acceleration of the writing action acquired by the information acquisition unit and obtains an inner state value indicating the inner state by a value, and a process execution unit that executes at least one of encouraging concentration, encouraging a break, outputting a sound, outputting an image, outputting light, and outputting a scent as a second process different from the first process based on the inner state value acquired by the inner state estimation unit.

[0022] According to this configuration, the acceleration of the writing action of a subject using a first process involving writing in application software is acquired, and the inner state of the subject using the first process is estimated based on the acceleration of the writing action. Based on an inner state value indicating the estimated inner state, a second process different from the first process is executed, and at least one of encouraging the subject to concentrate, encouraging the subject to take a break, outputting a sound, outputting an image, outputting a light, and outputting a scent is executed.

[0023] Another aspect of the present invention is an information processing method comprising an information acquisition step of acquiring subject biometric information including at least one of biometric indicators and behavioral indicators of a subject using a first process with application software, an inner state estimation step of estimating an inner state of the subject based on the subject information acquired in the information acquisition step and obtaining an inner state value indicating the inner state by a value, and a process execution step of executing a second process different from the first process based on the inner state value obtained in the inner state estimation step.

[0024] According to such a configuration, a second process different from the first process is executed based on the inner state value obtained from the subject's biometric information, making the subject aware of his / her current inner state and encouraging him / her to change or maintain his / her inner state.

[0025] Another aspect of the present invention is an information processing program that causes a computer to execute an information acquisition step of acquiring subject biometric information including at least one of bioindicators and behavioral indices of a subject who is using a first process with application software, an inner state estimation step of estimating an inner state of the subject based on the subject biometric information acquired in the information acquisition step and obtaining an inner state value indicating the inner state by a value, and a process execution step of executing a second process different from the first process based on the inner state value obtained in the inner state estimation step.

[0026] According to such a configuration, a second process different from the first process is executed based on the inner state value obtained from the subject's biometric information, making the subject aware of his / her current inner state and encouraging him / her to change or maintain his / her inner state. Effect of the Invention

[0027] According to the present invention, it is possible to provide an information processing device, an information processing method, and an information processing program capable of estimating an inner state from biological information of a subject and guiding the subject to an appropriate inner state. [Brief description of the drawings]

[0028] [Figure 1] 1 is a block diagram illustrating a configuration of an information processing device according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a schematic diagram showing a specific example of an information processing device according to an embodiment of the present invention. [Diagram 3] FIG. 1 is a diagram showing Russell's ring model. [Figure 4] FIG. 1 is a diagram showing Russell's ring model. [Diagram 5] FIG. 2 is a schematic diagram showing an example of application software executed on an external terminal. [Figure 6] FIG. 1 is a schematic diagram illustrating a cycle of game progress, difficulty level change, and game resumption after pausing. [Figure 7] 1 is a flowchart illustrating an information processing method according to the present embodiment. [Figure 8]FIG. 13 is a schematic diagram showing an example of acquiring a behavior index by a camera. [Figure 9] 1 is a schematic diagram showing an example of acquiring a behavior index by a device having an acceleration sensor; [Figure 10] 11 is a flowchart illustrating an information processing method using a writing action. [Figure 11] FIG. 13 is a diagram showing an experimental procedure from measurement of biological information to feedback. [Figure 12] FIG. 13 is a diagram illustrating an example of a change in the rate of correct answers. [Figure 13] FIG. 13 is a diagram illustrating an example of changes in electroencephalogram (γ waves in the frontal lateral AF8) before and after stimulation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following description, the same members are designated by the same reference numerals, and the description of members that have already been described will be omitted as appropriate.

[0030] (Configuration of information processing device) FIG. 1 is a block diagram illustrating the configuration of an information processing device according to this embodiment. The information processing device 1 according to the present embodiment has a function of estimating the inner state (psychological condition) of a subject who is using application software, and guiding the subject to an appropriate inner state.

[0031] 1, the information processing device 1 includes an information acquisition unit 10, an inner state estimation unit 20, and a process execution unit 30. In addition to these, the information processing device 1 may also include an input unit 40, an output unit 50, a communication unit 60, and a control unit 70. The units of the information processing device 1 may be provided within the same housing, or at least one of them may be provided externally or connected via a network.

[0032] The information acquiring unit 10 acquires biometric information of the subject (subject biometric information) including at least one of the biometric indicators and behavioral indicators of the subject. Here, the biometric indicators include values ​​indicating physiological phenomena such as heart rate, brain waves, respiratory rate, blood oxygen concentration, blood pressure, body temperature, and sweating. The body movement information includes values ​​indicating human behavior such as head and face movements, blinking, and facial expressions, and human body movements (body part motions such as hands and feet).

[0033] The inner state estimation unit 20 estimates the inner state of the subject based on the subject's biometric information acquired by the information acquisition unit 10, and obtains an inner state value indicating the inner state as a value. The inner state of the subject includes the subject's emotions, arousal level, pleasantness / unpleasantness, stress, etc., and the inner state value includes these inner states quantified as a single number, coordinate values, a multidimensional data set, etc. The inner state of the subject may be estimated using technologies such as AI (Artificial Intelligence), machine learning, and deep learning.

[0034] The process execution unit 30 performs a process of causing the application software to execute a second process different from the first process based on the inner state value obtained by the inner state estimation unit 20. The first process executed by the application software executes a process that is not directly affected by the inner state of the subject. The inner state of the subject is estimated while using the first process of the application software, and the second process acts on the first process based on the estimated inner state. That is, the second process executes a process that is directly affected by the inner state of the subject. Note that the second process may be configured as a part of the same application software as the first process, or may be configured as software independent of the first process.

[0035] The input unit 40 is, for example, a switch, a button, a touch panel, a camera, a microphone, etc. The output unit 50 is, for example, a speaker, a lamp, a vibration device, etc. The communication unit 60 includes a transmission unit 61 and a reception unit 62. The communication unit 60 transmits and receives information to and from an external device. The communication unit 60 may transmit and receive information via wireless communication or wired communication. The control unit 70 is, for example, a CPU that controls each unit.

[0036] (One specific example of an information processing device) FIG. 2 is a schematic diagram showing a specific example of the information processing device according to the present embodiment. In the example shown in FIG. 2, the information processing device 1 is configured as an information processing system including a subject-side terminal 2 and an external terminal 3. The subject-side terminal 2 is a device including an information acquisition unit 10 that acquires subject biometric information, and examples of such devices include a heart rate / pulse meter and an electroencephalograph that are attached to the subject. A specific example of the subject-side terminal 2 is a wristwatch-type portable information terminal (smart watch). The external terminal 3 is a device including an inner state estimation unit 20 and a processing execution unit 30, and examples of such devices include a portable terminal, a tablet terminal, and a personal computer. A specific example of the external terminal 3 is a smartphone. Between the subject-side terminal 2 and the external terminal 3, information is mutually communicated, for example, by wireless communication (such as Bluetooth (registered trademark) or Wi-Fi (registered trademark)).

[0037] A first process of the application software is executed in the external terminal 3, and the subject uses the first process. A specific example of the first process is a game. In the information processing device 1, the subject's terminal 2 acquires subject's biometric information while the first process is being used. For example, the subject's terminal 2 acquires subject's biometric information such as heart rate, pulse waves, and brain waves of the subject while the first process is being used, and transmits it to the external terminal 3 by wireless communication.

[0038] The external terminal 3 estimates the inner state of the subject based on the subject's biological information transmitted from the subject's terminal 2. The inner state estimation unit 20 of the external terminal 3 predicts values ​​such as emotion, arousal, pleasantness / unpleasantness, and stress from the subject's biological information. For this prediction, the Russell's ring model shown in Fig. 3 and Fig. 4 is used. In the Russell's ring model shown in Fig. 3 and Fig. 4, the vertical axis corresponds to arousal (Arousal) obtained from electroencephalograms, and the horizontal axis corresponds to pleasantness / unpleasantness (Valence) obtained from heartbeats, and the model represents human emotions and feelings (joy, anger, sadness, and happiness) by quadrants. As shown in Fig. 4, in the Russell's ring model, psychological conditions are associated with the direction and magnitude of a vector on the ring model. The inner state estimation unit 20 of the external terminal 3 may, for example, obtain an inner state value by quantifying the inner state of the subject based on the quadrant of the Russell's ring model and the distance from the center.

[0039] In addition, it is also possible to distinguish, predict, or estimate inner state values ​​such as emotion, arousal level, pleasantness / unpleasantness, and concentration level, as well as trust, compatibility, emotion, relaxation, arousal, behrens, excitement, anger, tension, etc. This can be predicted using Russell's circumplex model from the subject's biological information, or from the subject's direct behavior or facial expression.

[0040] FIG. 5 is a schematic diagram showing an example of application software executed on an external terminal. An example of a first process of application software executed on the external terminal 3 is a shooting game. The process execution unit 30 of the external terminal 3 executes, as a second process, a process of changing the difficulty level of the shooting game, which is an example of the first process. That is, the process execution unit 30 changes the difficulty level (game level) of the shooting game, as an example of the second process, based on the inner state value of the subject (the subject playing the shooting game, which is the first process) obtained by the inner state estimation unit 20. For example, the higher the numerical value of the emotion of anger, which is the inner state of the subject, the faster the enemy's movements become, and the higher the difficulty level becomes.

[0041] When the game becomes too difficult and the subject collides with an enemy, resulting in a game over, or when the subject is about to collide with an enemy and wants to lower the game's difficulty, the subject can use the pause function, which gives the subject time to objectively view his or her own internal state.

[0042] While the game is paused, the subject calms down and performs an action (behavior) that reduces the value that represents the anger emotion. This action is, for example, a calming action such as staying still, taking a deep breath, or talking to someone. The information acquisition unit 10 of the subject's terminal 2 sequentially acquires the subject's biological information and transmits it to the external terminal 3. Even while the game is paused, the subject's biological information is transmitted from the subject's terminal 2 to the external terminal 3, and the internal state estimation unit 20 of the external terminal 3 obtains the subject's internal state value. If the subject calms down during the game pause and the internal state value decreases, the process execution unit 30 performs a process of lowering the difficulty level of the game as a second process. If the subject resumes the game with the difficulty level lowered, the subject can easily progress in the game.

[0043] FIG. 6 is a schematic diagram illustrating a cycle of game progress, difficulty level change, and game resumption after pausing. As the subject plays the game, the time remaining until game over decreases, and if they do not progress well, they become stressed. This internal state can be estimated, for example, using Russell's circumplex model, and as the anger value increases, the difficulty of the game increases. As the difficulty of the game increases, the subject is able to understand his or her own emotions and pauses the game. While the game is paused, the subject can take action to calm himself or herself, which reduces the difficulty of the game and allows the subject to resume the game. When feelings of anger increase, the subject is able to understand how to reduce those values. By repeating this cycle, the subject can understand his or her own emotions and train to control them when they are not good, and become able to recognize and control his or her own emotions and feelings even without playing games in his or her everyday life.

[0044] In this example, the second process is a process of increasing the difficulty of the game as the numerical value of emotions such as anger and stress increases as the inner state of the subject. In the Russell Circumplex model shown in FIG. 3, when the inner state of the subject is in the "anger" quadrant, the second process is performed in a direction to further increase the magnitude of "anger" (for example, to increase the difficulty of the game). This causes the subject's emotion of "anger" to further increase, and the subject is more likely to become aware of his / her inner state of "anger" in a short time. In other words, when anger management is performed with the information processing device 1 according to this embodiment, the subject is able to grasp his / her inner state of "anger" in a short time, and the trading time can be shortened.

[0045] If the subject pauses the game and takes action to calm down, and the level of the "anger" in the internal state decreases, the second process is executed to lower the difficulty of the game. If the subject resumes the game in this state, it becomes easier to complete, and the subject becomes aware through the game that the internal state value of his or her "anger" has decreased.

[0046] On the other hand, when the subject's inner state is in the "happy" or "pleasant" quadrant, a process for maintaining the game difficulty is executed as a second process in order to maintain the inner state. The subject can recognize that his / her own inner state is also stable because the game difficulty is stable. Here, when the subject's inner state is in the "happy" or "pleasant" quadrant, the game difficulty may be temporarily lowered. By temporarily lowering the game difficulty, the subject can easily recognize that his / her own inner state is in the "happy" or "pleasant" quadrant. Note that after the temporary lowering of the difficulty, it is preferable to return the difficulty to the original level.

[0047] Nowadays, there have been cases of people letting their anger get the better of them, leading to trouble at nursing care or welfare facilities, or accidents caused by aggressive driving, but by repeatedly undergoing this type of training, people will be able to become aware of and control their own feelings and emotions, which will have the effect of preventing trouble from occurring in the first place.

[0048] (Information processing method) FIG. 7 is a flowchart illustrating an information processing method according to this embodiment. In the flowchart shown in FIG. 7, a case where a game is executed as the first process is taken as an example. First, as shown in step S101, subject biometric information is acquired (information acquisition step). For example, the subject wears a device that acquires biometric information. The device acquires the subject biometric information. Next, as shown in step S102, the subject biometric information is analyzed (internal state estimation step). Here, based on the subject biometric information, numerical values ​​are obtained that indicate the subject's internal state, such as emotions, feelings, pleasantness / unpleasantness, arousal level, and stress.

[0049] Next, as shown in step S103, it is determined whether the numerical value indicating the inner state is equal to or greater than a preset threshold. If the numerical value indicating the inner state is not equal to or greater than the threshold, a process of maintaining or lowering the game difficulty is executed as a second process (step S104). Next, it is determined whether the game has been cleared (step S105), and if cleared, the process ends, and if not, the process returns to step S101.

[0050] On the other hand, if it is determined in step S103 that the numerical value indicating the inner state is equal to or greater than the threshold value, a process for increasing the game difficulty is executed as a second process (step S106). Here, the process for maintaining or decreasing the game difficulty (step S104) and the process for increasing the game difficulty (step S106) are process execution steps.

[0051] After the game difficulty level is increased in the process of step S106, it is determined whether or not the game has been cleared (step S107), and if the game has been cleared, the process is terminated, and if the game has not been cleared, it is determined whether or not there is an instruction to end the game or stop (step S108).

[0052] If the game is over as determined in step S107 or if an instruction to stop is given, the game is stopped (step S109), and if the game is not over or if an instruction to stop is not given, the process returns to step S101. After proceeding to step S109 and stopping the game, the subject takes an action to improve his / her emotions and feelings (step S110). Thereafter, as shown in step S111, the subject's biological information is acquired and analyzed. Then, as shown in step S112, it is determined whether the numerical value indicating the inner state is equal to or greater than a preset certain value or whether a restart instruction has been given. If the numerical value indicating the inner state is not equal to or greater than the certain value or if a restart instruction has not been given, the process returns to step S110. On the other hand, if the numerical value indicating the inner state is equal to or greater than the certain value or if a restart instruction has been given, the process returns to step S103.

[0053] Thus, in the case of a game as an example, the subject wears a device for acquiring biometric information, and the subject's biometric information is acquired during the game. The information processing method analyzes the subject's biometric information and quantifies it into indicators such as feelings, emotions, pleasantness / unpleasantness, arousal level, and stress. If the numerical value (internal state value) is smaller than a threshold, the difficulty level of the game is maintained or decreased. On the other hand, if the numerical value is equal to or greater than the threshold, the difficulty level of the game is increased. In this case, the threshold may be changed according to the subject's individual state, or may be variable based on past numerical values. In addition, a mechanism may be included in which the difficulty level increases over time or increases irregularly to make it easier to inflict stress.

[0054] The information processing method has a function of temporarily stopping the game when the game cannot be cleared and there is some kind of game instruction, such as colliding with an enemy or being hit by a ball hit by an enemy, or when there is a direct instruction to stop from the subject. The subject takes an action to improve emotions, feelings, stress, etc. while the game is stopped. This action indicates staying still, taking a deep breath, talking to someone, etc., and as the information processing method, it may be output as a recommendation from the output means of the device.

[0055] The game is restarted when the numerical value (internal state value) falls below a certain value or when the subject issues a restart command. By repeating this process using the information processing method, the subject becomes aware of his or her own emotions and stress, and is able to control those emotions and stress on his or her own. The game itself ends (stops) when the game is cleared or when the subject issues an end command.

[0056] (Example of equipment configuration) Examples of the configuration of devices to which the information processing device 1 and information processing method according to this embodiment are applied include the following. Examples of electronic devices that acquire the subject's biometric information (subject biometric information) include smart watches, patch-type heart rate sensors worn on the arm or chest such as "MyBeat" (registered trademark), headband-type brainwave devices worn on the head such as "Muse2" (product name), head-mounted display-type, and head-patch-type brainwave devices. The electronic device that acquires the subject's biometric information may be any one of these, or a combination of multiple devices.

[0057] The subject's biological information acquired by the electronic device is processed by the electronic device itself or an external device such as a smartphone, tablet terminal, or personal computer, and is quantified as information such as feelings, emotions, stress, arousal level, and pleasantness / unpleasantness. Furthermore, the electronic device has an electronic device that performs a second process (e.g., adjusting the difficulty level of a game) for a first process (e.g., a game) based on the numerical information. This electronic device may be the same as the electronic device that acquires the subject's biological information, or may be the same as the electronic device that executes the first process.

[0058] By using these components to process information, it becomes possible to carry out gamification, for example, by encouraging people to change their behavior through games. Gamification can also make anger management and stress management more effective and deeper, at a level that the subject himself does not or finds difficult to notice.

[0059] (Other means of acquiring subject biometric information) In the above example, the bioindicators were heart rate and brain waves, but other bioindicators can also be used, such as pulse wave, electrocardiogram, skin conductance level, skin temperature, sweat rate, respiratory rate, pupil diameter, electromyogram, etc. These bioindicators are acquired from the subject using the first process, and become the subject's bioinformation.

[0060] The above biomarkers will now be described in detail. Brain waves: As the level of arousal increases, the frequency of brain waves increases, especially in the high-frequency band (such as beta waves). Brain waves reflect the activity and arousal level of the brain. The frequency of brain waves is 0.1Hz to 100Hz, and this frequency range is called alpha waves, beta waves, theta waves, delta waves, and gamma waves. Some examples of brain waves are given below. Alpha waves (α waves): Frequency ranges from 8 to 13 Hertz (Hz). Alpha waves are typically observed in a relaxed state or with eyes closed and are associated with an alert yet relaxed state. Beta waves (β waves): Frequency ranges from 13 to 30 Hz. Beta waves are typically observed in states of heightened alertness, concentration, and cognitive activity. Gamma waves: Frequency ranges typically above 30 Hz. Gamma waves are associated with higher brain functions such as cognition, learning, and information processing.

[0061] Skin Conductance Level (SCR): Increased arousal leads to an increase in the electrical skin response. Skin conductance response is a change in the electrical resistance of the skin, which reflects increased sympathetic nerve activity.

[0062] Skin temperature (skin temperature): An increase in arousal is usually accompanied by an increase in skin temperature. Sympathetic nervous activity causes vasoconstriction, lowering skin temperature.

[0063] Respiratory rate: Increased arousal leads to increased respiratory rate. Respiratory activity is closely related to the body's state of arousal.

[0064] Pupil diameter: As arousal increases, pupil diameter increases.

[0065] Electromyogram (EMG): Increased arousal leads to increased muscle tone and increased EMG activity, which reflects the degree of muscle tone in the body.

[0066] In addition, with regard to the heart rate exemplified above, an index that quantifies the interval between heart rates, such as pNN or HF / LF, can be used. pNN is the percentage of beats in which the difference between successive adjacent RR intervals exceeds a certain value. HF refers to the power spectrum of the frequency band from 0.15 to 0.4 Hz. FL refers to the power spectrum in the frequency band of 0.004 to 0.15 Hz. The LF / HF ratio refers to the ratio of LF (low frequency) to HF (high frequency) power.

[0067] (Method of making you aware of your inner state) As a means of making a subject aware of values ​​indicating their own inner state such as their emotions, arousal level, pleasure / pain, and stress, there are methods that display or generate information on electronic devices. Specific examples include characters, images, colors, illustrations, sounds, music, scents, and vibrations. The text, images, colors, and illustrations may be displayed on the same device that acquires the biometric information, or on a separate device, such as a smart watch, smartphone, tablet, personal computer, or television.

[0068] Regarding color, the color of the entire environment may be changed by using light-emitting devices that can change brightness and color in response to external signals. The sound or music may be played from the same device that acquires the biometric information, or from a different device, such as a smartwatch, smartphone, tablet, personal computer, television, etc. This sound or music refers to the change in air vibration from a silent state, and includes not only regular classical or pop music, but also white noise and sounds with frequencies that cannot be heard by the human ear.

[0069] The vibration may be emitted from the same device that acquires the biometric information, or from a separate device, such as a smart watch, smartphone, tablet terminal, or personal computer.

[0070] With regard to scent, an external device such as an aroma diffuser can be used to change the scent in response to an external signal, but the scent can also be changed manually by another person who notices a change in the subject or the person themselves.

[0071] The means for making the subject aware of these inner state values ​​is also a process in which the second process prompts the subject to pay attention when the subject is using the first process, thereby making the subject aware of his or her inner state by actively informing the subject of the inner state.

[0072] (Estimation of inner states based on behavioral indicators) When the information acquisition unit acquires the subject's biological information and the inner state estimation unit obtains the values ​​of emotion, pleasantness / unpleasantness, arousal level, and stress (inner state values), the inner state may be estimated from behavioral indices such as human behavior and facial expressions, in addition to directly measuring biological indices such as brain waves and heart rate. The estimation of the inner state based on the hardness index may use technologies such as AI, machine learning, and deep learning.

[0073] FIG. 8 is a schematic diagram showing an example of acquiring a behavior index by a camera. In the example shown in Fig. 8, for example, the camera 4c of the mobile terminal 4 is used to capture the behavior and facial expression of the subject, and the inner state is predicted based on the captured image (video). The mobile terminal 4 is configured with a function of executing a first process using application software (information acquisition unit 10, information acquisition step), a function of predicting the inner state of the subject from the behavior and display of the subject captured by the camera 4c (inner state estimation unit 20, inner state estimation step), and a function of executing a second process different from the first process based on the inner state value (process execution unit 30, process execution step). The second process may include outputting images, illustrations, videos, and text on the screen of the mobile terminal 4, outputting sounds, and controlling the game being executed on the mobile terminal 4, for example, by adjusting the difficulty level of the game.

[0074] In addition to the above, other devices for acquiring behavioral indicators of a subject include sensing devices equipped with an acceleration sensor that are attached to the subject's body, such as ring- or bracelet-type devices.

[0075] FIG. 9 is a schematic diagram showing an example of acquiring a behavior index by a device having an acceleration sensor. The device for acquiring the behavioral index of the subject is a writing implement 100 equipped with an acceleration sensor or an IoT (Internet of Things) device 110 attached to the writing implement. A function for estimating the inner state of the subject based on the behavioral index (e.g., an inner state estimation unit, an inner state estimation step) is provided in the device itself or an external device 200 such as a smartphone, tablet terminal, or personal computer, where information is processed to obtain inner state values ​​such as concentration and brain arousal level.

[0076] Furthermore, the device that performs an operation (second process) such as playing music based on the inner state value may be the same as the device that acquires the behavior index, or may be the same as the device that performs information processing to estimate the inner state, or may be the same as both of these.

[0077] Other means for acquiring writing actions include a sensing device equipped with an acceleration sensor attached to the body that moves when writing, such as a ring or bracelet-type device, or acquiring information from the trajectory of writing (direction, distance, speed, acceleration, etc.) when writing on a touch panel. In addition, the means for acquiring writing actions may be a method of photographing the writing state with a camera to determine the writing action, or a method of acquiring pressure changes during writing using a pressure sensor such as a pressure sensor attached to a writing instrument or a touch panel to determine the writing action.

[0078] For example, by using AI that uses the correlation between acceleration information in writing actions and the behavioral indices of the subject as a learning model, it is possible to predict behavioral indices that indicate the subject's concentration, such as head movement, eye movement, and blinking, from the acceleration information in writing actions.The subject's internal state can be estimated based on these predicted behavioral indices.

[0079] As a specific example, in the example shown in FIG. 9, a behavior index is acquired using an attachment-type IoT device 110 attached to a writing instrument 100. An acceleration sensor is built into the attachment-type IoT device 110 attached to the writing instrument 100, and the acceleration of the writing action when the subject writes using the writing instrument 100 (including the acceleration when actually writing characters, etc., as well as the acceleration when holding the writing instrument 100 without writing characters, etc.) can be acquired. Information based on the acceleration of the writing action captured by the IoT device 110 is transmitted to an external device 200 such as a mobile terminal by wireless communication or the like. The external device 200 predicts the subject's inner state from the information based on the acceleration sent from the IoT device 110 and obtains an inner state value. The external device 200 executes a first process as application software, executes a second process based on the inner state value, and executes a process according to the inner state.

[0080] An example of estimating the internal state from information based on the acceleration of the writing action is, for example, analyzing the acceleration of the writing action when studying with writing action, and determining the internal state obtained from the acceleration of the writing action, such as the state of concentration, the state of cognitive load, the state of thinking ability, logic, etc., and the internal state obtained based on the correlation between the acceleration of writing and brain waves. Here, the concentration level is taken as an example. From the determined concentration level, when the subject is concentrating on studying, nothing in particular is done, and when not concentrating, music that helps the subject concentrate is played from an external device such as a smartphone. For example, white noise, babbling, classical music, etc. are played from the external device. By doing so, the subject can concentrate better. Also, when the subject is concentrating for an excessively long time, the external device 200 such as a smartphone may take measures to encourage a break and not get too tired (such as displaying text or images on the screen). This is continued until the study is finished.

[0081] In such an information processing device 1, the concentration time or study time is displayed on the screen of the external device 200, and images or music that encourage concentration are output, so that the subject can recognize the current state of concentration in learning or studying, and can be guided to effective learning or studying by encouraging concentration. Also, when the concentration time becomes longer than a predetermined time, images or music that encourages a break or relaxation can be output, making it possible to achieve effective learning or studying with a balance between concentration and relaxation.

[0082] (Information processing method using writing actions) FIG. 10 is a flowchart illustrating an information processing method using a writing action. First, as shown in step S201, an action index of a writing action is obtained using an acceleration sensor, a pressure sensor, etc., when writing in a situation such as studying. Next, as shown in step S202, the subject's inner state is analyzed based on the obtained action index of the writing action. For example, the inner state such as concentration and brain arousal is predicted using technologies such as AI, machine learning, and deep learning, and an inner state value is obtained.

[0083] Next, as shown in step S203, it is determined whether the inner state value is equal to or greater than a threshold value. If the inner state value is equal to or greater than the threshold value, the operation of the external device is not changed (step S204). This makes it possible to avoid interrupting the state of concentration. Next, it is determined whether or not to continue the writing action (step S205), and if the writing action is to be continued, the process returns to step S201, and if not, the process ends.

[0084] On the other hand, if the determination in step S203 is that the internal state value is less than the threshold value, the operation of the external device is changed (step S206). For example, the operation of the external device is changed to promote concentration and brain awakening. One example of this operation is playing music that promotes concentration (white noise, babbling, classical music, etc.). In this way, it is possible to change from a state of not concentrating to an operation that promotes concentration.

[0085] Next, it is determined whether or not to continue the writing action (step S207), and if the writing action is to be continued, the process returns to step S201, and if not, the process ends. That is, when the writing action is to be stopped (studying is to be stopped), this action is stopped, but otherwise, acquisition of the writing action and this determination, and the process of maintaining or changing the operation of the external device are repeated. When performing this process, it is possible to perform a process of not changing the same operation even if the inner state value changes for a certain period of time so as not to change the operation of the external device too frequently. This may be a process of leveling out the numerical value by using a moving average of the change in the inner state value, or the like. This is continued until the end of studying.

[0086] (Other means of obtaining behavioral indicators) Other examples of means for acquiring the behavioral index are given below. - Capture hand movements using a smartwatch - Capture the subject's whole body movements using a camera - Capture facial expressions of the subject using a camera - Obtain information on walking, running, etc. using acceleration information from your smartphone By applying the behavioral indicators obtained through this to AI, it becomes possible to infer and predict internal states such as emotions, level of arousal, pleasure / displeasure, concentration, trust, compatibility, affect, relaxation, arousal, behrens, excitement, anger, and tension.

[0087] (How to become aware of your inner state) The following are some ways to promote changes in your internal state and behavior, such as improving your concentration: Appealing to the sense of sight: Methods of appealing to the sense of sight include displaying the current internal state on the screen of a smartwatch, smartphone, tablet device, or personal computer using text, images, illustrations, videos, or animations. Another method is to change the color or brightness of the screen. Another method is to change the color or brightness of electricity or lights using light-emitting devices that can change the intensity or color of light based on external information. Another method is to understand people's preferences and display images, text, videos, illustrations, colors, etc. that they like or dislike.

[0088] Stimulating the sense of hearing: Methods of stimulating the sense of hearing include playing sounds or music from a smartwatch, smartphone, tablet device, personal computer, music player, headphones, speakers, etc. Also, sounds or music that encourage concentration, sounds or music that encourage relaxation, etc., such as classical music, white noise, and rippling sounds, can be used. Favorite music or intense music is expected to have an effect of waking you up. It is also possible to combine a system that allows you to select music according to your preferences.

[0089] Stimulating the sense of smell: Citrus scents and pungent smells are known to wake people up. Also, scents such as lavender are known to have a relaxing effect. The scents can be generated using an aroma diffuser. Also, the subject can be instructed via a smartphone or tablet device to choose a suitable perfume or air freshener to change their internal state. The subject can prepare it themselves by following the instructions on the screen.

[0090] Influencing the sense of taste: Food textures and tastes can wake you up. For example, chewing gum can wake you up. You can also raise your spirits by eating your favorite foods. To impart food textures, devices that stimulate the inside of the mouth can be used. You can also instruct the subject via a smartphone or tablet device on foods that are suitable for changing their internal state. The subject can prepare the food themselves by following instructions on the screen.

[0091] Influencing the sense of touch: Methods of influencing the sense of touch include stimulating the sense of touch by generating vibrations, encouraging concentration, waking people up, and increasing the level of brain arousal. In this case, for example, vibrations can be given via a smartwatch, smartphone, tablet, etc. In addition, it is also possible to encourage behavioral change by using devices that do not use vibrations, such as electrical stimulation or pain. The sense of touch includes stimulation of the skin, as well as stimulation inside the mouth and ears.

[0092] Directly influencing the brain: One way to directly influencing the brain is to insert electrodes into the brain and directly stimulate it. Stimulating the brain can change the subject's behavior. Electrical stimulation is one example of influencing the brain.

[0093] (Another example of device behavior change due to internal state values) In the above, we have shown examples of behavioral changes in which, depending on whether the internal state value is equal to or greater than a threshold, one device does not change its operation, while the other device changes its operation (such as playing a sound), but this is not limited to these and the following types of behavioral changes are also included. The device may have a function for selecting different operations (different second processes) depending on whether the threshold is exceeded or not. There may be multiple thresholds, and the device may have a function of selecting a different operation (different second process) for each threshold. The different operation may include not changing (maintaining) the operation of the device. The threshold may be a different value for each individual, or may change dynamically according to changes in predicted values, including biometric information, when an individual uses the device. This includes the threshold being optimized for each individual. For example, if an action taken to encourage concentration disrupts concentration, the threshold may be changed from the next time and nothing may be done when the threshold is at the previous value. The device may have a mechanism for optimizing the operation of the device to promote concentration for each individual. For example, if the music played to promote concentration disrupts concentration or there is no change, the device may not use that music next time.

[0094] When it is assumed that these bio-indicators and behavioral indices are changed during learning / studying (second processing), examples of the internal state of the subject to be grasped include the state of concentration, the state of cognitive load, the state of thinking ability, the state of logical thinking, the state of comfort or discomfort, and the state of stress. In addition, examples of behavioral indices during learning include the behavior / action of writing, the behavior / action of reading a book, the behavior / action of looking at learning content, the behavior / action of thinking such as solving a problem, the behavior / action of memorizing something, the behavior / action of inputting by typing, the behavior / action of speaking out loud such as reading aloud, and the behavior / action of listening to audio. These behaviors / actions can be obtained by acquiring the physical quantities of the tools (study tools, etc.) used by the subject and the state of the subject using input devices such as a camera or microphone. The internal state of the subject during learning / studying can be grasped by the correlation between the acquired behavior / action and the internal state.

[0095] Furthermore, the process execution unit 30 of the information processing device 1 or the process execution step of the information processing method may change the second process based on a time change in the inner state value of the subject who is using the first process. For example, the inner state value of the subject is continuously obtained, and the second process is changed depending on whether the time change in the inner state value is a vector based on Russell's ring model that changes suddenly or does not change much. Furthermore, the degree of change in the second process may be changed depending on the quadrant, such as in which quadrant of Russell's ring model the change is occurring, or whether the change crosses quadrants.

[0096] (Example of having someone other than the intended recipient intervene) In this embodiment, when the inner state of the subject is estimated based on the subject's biological information, the inner state or the inner state value may be notified to a person other than the subject (a third party). That is, the information processing device 1 and the information processing method may be configured so that a third party other than the subject intervenes when changing the estimated inner state of the subject (emotion, arousal level, pleasantness / unpleasantness, concentration level, trust level, compatibility level, emotion, relaxation, excitement, anger, tension, etc.). The information processing device 1 includes an external terminal that can be used by a third party other than the subject. The second process in the information processing device 1 performs a process of outputting information based on the inner state value of the subject to the external terminal that can be used by the third party.

[0097] As a specific example, a subject wears a smartwatch or a small electroencephalograph while studying or working in a cafe. The subject's inner state or changes in the inner state are transmitted from the subject's device, such as a smartwatch, and are notified to an external terminal that can be used by the cafe staff. If the subject loses concentration or becomes sleepy, the cafe staff will grasp the subject's inner state based on the information notified to the external terminal, talk to the subject, offer the subject something to drink, etc. This will have the effect of increasing the subject's concentration again and waking up from sleepiness, and will increase the efficiency of studying or working. Note that the above example shows the relationship between the cafe staff and the customer, but this may also be a parent-child relationship. As a result, the subject's inner state, which is difficult to notice, can be made known to someone other than the subject, and the person other than the subject can take action based on the inner state.

[0098] (Experimental Example) FIG. 11 is a diagram showing the experimental process from measurement of biological information to feedback. In this experiment, the subject's brain waves and heart rate were obtained as biological information. In the experiment, auditory and visual stimuli were given when the subjects began to lose concentration while studying / learning, and it was confirmed whether this increased the brain waves (beta waves and gamma waves) that indicate concentration, and whether the rate of correct answers changed. The specific experimental method was to present calculation problems in English using sounds, and to measure the brain waves when answering the problems, and to examine the change in the rate of correct answers.

[0099] FIG. 12 is a diagram illustrating an example of changes in the rate of correct answers. Figure 12 shows the change in the rate of correct answers after concentration waned (later part of the questions) when no stimulus was given (NoStimuli) and when a sound stimulus (RiverSound: babbling brook) was given. It can be seen that when no stimulus was given, the rate of correct answers decreased, whereas when a sound stimulus (River Sound: babbling) was given, the rate of correct answers increased.

[0100] FIG. 13 is a diagram illustrating changes in electroencephalogram (γ waves in the frontal lateral region AF8) before and after stimulation. Gamma waves are most active when the brain is activated, that is, when one is concentrating. Without any stimulation, gamma waves drop due to fatigue, making it difficult to concentrate, whereas when sound stimulation (River Sound: babbling) is given, gamma waves rise, indicating greater concentration.

[0101] The experimental results shown in Figures 12 and 13 show that by identifying when the participant's concentration begins to wane during study and providing a stimulus in the form of the sound of a stream at that time, the participant's concentration improves and the rate at which they answer questions correctly also increases.

[0102] The experimental data was statistically processed from data from multiple people, and the dots in Fig. 12 and the error bars in Fig. 13 represent the variation between subjects. In the experiment, the changes in brain waves were continuously obtained by comparing them with the previous data in a time series. The period for obtaining the comparison data of brain waves can be effectively set between 0.01 seconds and 3 hours. The threshold for the amount of change in brain waves can also be averaged using a moving average or the like.

[0103] (Information Processing Program) The information processing program (application software) according to this embodiment has the following configuration.

[0104] The information processing program executed by the computer On the computer, an information acquisition step of acquiring subject biometric information including at least one of a biometric indicator and a behavioral indicator of a subject who is using a first process with the application software; an inner state estimating step of estimating an inner state of the subject based on the subject's biological information acquired in the information acquiring step, and obtaining an inner state value indicating the inner state by a value; a processing execution step of executing a second processing different from the first processing based on the inner surface state value obtained in the inner surface state estimation step; Execute the command.

[0105] The information processing program may be recorded on a recording medium or distributed via a network.

[0106] The information processing device 1, information processing method, and information processing program according to the present embodiment described above can be applied to various systems, such as anger management and stress management using games as a first process, as well as a learning support system that notifies users when they are losing concentration, when high-frequency brain waves are decreasing, when they are becoming too relaxed, when they are feeling sleepy, etc., to encourage improved learning efficiency.

[0107] In addition, the information processing device 1, information processing method, and information processing program that obtain the subject's behavior index, estimate the subject's inner state, and lead the subject to an appropriate inner state are not limited to the acceleration of writing motion as a behavior index, and can be applied to the acceleration of any subject's body motion. For example, the acceleration of moving a toothbrush, the acceleration of body part motion when playing a musical instrument or playing sports, the acceleration of various tools, the acceleration of body motion when operating a vehicle such as a car, the acceleration of the steering, pedals, levers, etc. that are the objects of operation, the acceleration of the motion of elements such as the vehicle body, suspension, and links, etc. can be applied as the subject's behavior index.

[0108] As described above, according to this embodiment, it is possible to provide an information processing device 1, an information processing method, and an information processing program that can estimate the inner state of a subject who is using application software and guide the subject to an appropriate inner state.

[0109] Although the present embodiment and its application examples have been described above, the present invention is not limited to these examples. For example, a shooting game has been shown as an example of the first process, but other games such as puzzles may also be used. In addition, those in which a person skilled in the art appropriately adds, deletes, or modifies components of the above-mentioned embodiments or their application examples, or those in which the features of the embodiments are appropriately combined, are also included in the scope of the present invention as long as they include the gist of the present invention. [Explanation of symbols]

[0110] 1...Information processing device 2. Target device 3…External terminal 4. Mobile devices 4c…Camera 10…Information acquisition department 20…Inner state estimation unit 30...Processing execution unit 40...Input section 50…Output section 60…Communications Department 61...Transmitter 62...Receiver 70...Control unit 100...Writing implements 110…IoT equipment 200...External device

Claims

1. an information acquisition unit that acquires subject biometric information including at least one of a biometric indicator and a behavioral indicator of a subject who is using a first process with application software; an inner state estimation unit that estimates an inner state of the subject based on the subject's biological information acquired by the information acquisition unit, and obtains an inner state value that indicates the inner state; a process execution unit that executes a second process different from the first process based on the inner surface state value obtained by the inner surface state estimation unit; An information processing device comprising:

2. The information processing apparatus according to claim 1 , wherein the second process is a process for changing the inner state value of the subject who is using the first process.

3. the first process is a game, 2. The information processing device according to claim 1, wherein the second process is a process of increasing a difficulty level of the game in the first process when the inner state value indicating "anger" or "stress" among the inner states of the subject using the first process is equal to or greater than a predetermined threshold value.

4. The information processing apparatus according to claim 1 , wherein the second process is a process for maintaining the inner state value of the subject who is using the first process.

5. The information processing device according to claim 1 , wherein the second process is a process for prompting the subject using the first process to pay attention based on the inner state value.

6. The information processing apparatus according to claim 1 , wherein the second process is a process of temporarily stopping the execution of the first process.

7. Further comprising an external terminal that can be used by persons other than the target person, The information processing apparatus according to claim 1 , wherein the second process is a process of outputting information based on the inner surface state value to the external terminal.

8. The information processing apparatus according to claim 1 , wherein the process execution unit changes the second process based on a time change in the inner state value of the subject utilizing the first process.

9. an information acquisition unit that acquires an acceleration of a writing action of a subject who is using a first process involving writing in application software; an inner state estimation unit that estimates an inner state of the subject based on the acceleration of the writing action acquired by the information acquisition unit, and obtains an inner state value that indicates the inner state; and a process execution unit that executes at least one of encouraging concentration, encouraging a break, outputting a sound, outputting an image, outputting light, and outputting a scent as a second process different from the first process based on the inner state value obtained by the inner state estimation unit.

10. an information acquiring step of acquiring subject biometric information including at least one of a biometric indicator and a behavioral indicator of a subject who is using a first process with the application software; an inner state estimating step of estimating an inner state of the subject based on the subject's biological information acquired in the information acquiring step, and obtaining an inner state value indicating the inner state; a processing execution step of executing a second processing step different from the first processing step based on the inner surface state value obtained in the inner surface state estimation step; An information processing method comprising:

11. On the computer, an information acquisition step of acquiring subject biometric information including at least one of a biometric indicator and a behavioral indicator of a subject who is using a first process with the application software; an inner state estimating step of estimating an inner state of the subject based on the subject's biological information acquired in the information acquiring step, and obtaining an inner state value indicating the inner state by a value; a processing execution step of executing a second processing different from the first processing based on the inner surface state value obtained in the inner surface state estimation step; An information processing program that executes the above.

Images