Reaction generation device, reaction generation method, virtual person presentation system, and reaction generation program

Abstract

To provide a device for acquiring object person input information and object person brain wave information on the object person when the object person input information is input, and outputting object person state information.SOLUTION: A reaction generation device includes: an information acquisition unit for acquiring object person input information input to an object person and object person brain wave information on the object person when the object person input information is input; a state generation unit for generating object person state information indicating the state of the object person on the basis of the object person brain wave information; and a reaction generation unit for generating reaction information indicating the reaction of the object person on the basis of the object person input information and the state of the object person. The information acquisition unit may further acquire biological information on the object person when the object person input information is input. The state generation unit may generate object person state information on the basis of the object person brain wave information and the biological information.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a reaction generation device, a reaction generation method, a virtual character presentation system, and a reaction generation program.

Background Art

[0002] Non-Patent Document 1 describes "an attempt to revive the deceased using artificial intelligence (AI) or humanoid robot technology." [Prior Art Document] [Non-Patent Document] [Non-Patent Document 1] Information and Communication Policy Research, Vol. 5, No. 1, 2021, pp. 131-144

Summary of the Invention

[0003] In a first aspect of the present invention, a reaction generation device is provided. The reaction generation device includes an information acquisition unit that acquires subject input information input by a subject and subject brain wave information of the subject when the subject input information is input, a state generation unit that generates subject state information indicating the state of the subject based on the subject brain wave information, and a reaction generation unit that generates reaction information indicating the reaction of the subject based on the subject input information and the state of the subject.

[0004] The information acquisition unit may further acquire biometric information of the subject when the subject input information is input. The state generation unit may generate subject state information based on the subject brain wave information and the biometric information.

[0005] In any of the above reaction generation devices, the information acquisition unit may acquire the subject brain wave information before and after the subject input information is input. The state generation unit may generate subject state information based on the change from the subject brain wave information before the subject input information is input to the subject brain wave information after the input and the biometric information.

[0006] In any of the above reaction generation devices, the state generation unit may generate subject state information based on the change from the ratio of the amplitude of brain waves in a predetermined frequency band in the subject's brain wave information before the subject input information is input to the ratio of the amplitude of brain waves in the predetermined frequency band in the subject's brain wave information after the subject input information is input, and the ratio of the magnitude of the first power spectrum to the magnitude of the second power spectrum in the subject's heartbeat. The overall amplitude is the sum of the amplitudes of alpha waves, beta waves, theta waves, gamma waves, and delta waves. The frequency band of the second power spectrum is a higher frequency band than the frequency band of the first power spectrum.

[0007] In any of the above reaction generation devices, the state generation unit may generate subject state information based on the magnitude relationship between the ratio of the magnitude of the first power spectrum to the magnitude of the second power spectrum and a predetermined threshold value of the ratio of the magnitude of the first power spectrum to the magnitude of the second power spectrum, and the change from the ratio of the amplitude of brain waves in a predetermined frequency band before the subject input information is input to the ratio of the amplitude of brain waves in the predetermined frequency band after the subject input information is input.

[0008] In any of the above reaction generation devices, the subject state information may include information related to a plurality of states of the subject. The state generation unit may generate subject state information related to one of the plurality of states based on the ratio of the magnitude of the first power spectrum to the magnitude of the second power spectrum and the change from the ratio of the amplitude of brain waves in a predetermined frequency band before the subject input information is input to the ratio of the amplitude of brain waves in the predetermined frequency band after the subject input information is input.

[0009] In any of the above reaction generation devices, the brain waves in a predetermined frequency band may be at least one of delta waves, theta waves, low alpha waves, and medium alpha waves, or may be at least one of high alpha waves, low beta waves, high beta waves, and gamma waves.

[0010] In any of the above reaction generation devices, the reaction of the subject may be presented by a presentation unit that presents a virtual human model. The information acquisition unit may further acquire the user's brain wave information of the user who has come into contact with the reaction presented by the virtual human model. The state generation unit may generate user state information indicating the state of the user based on the user's brain wave information. When the user state information is in a predetermined sense of discomfort state, the information acquisition unit may acquire feedback from the user regarding the reaction presented by the virtual human model. The reaction generation unit may correct at least one of the subject state information and the reaction information based on the feedback.

[0011] In any of the above reaction generation devices, the reaction of the subject may be presented by a presentation unit that presents a virtual human model. The information acquisition unit may further acquire the reaction of the subject presented by the virtual human model. Any of the above reaction generation devices may further include a reaction learning unit that generates a reaction inference model for inferring the reaction of the subject based on the subject state information by machine learning the relationship between the subject state information and the reaction of the subject acquired by the information acquisition unit.

[0012] In any of the above reaction generation devices, the information acquisition unit may further acquire the user's brain wave information of the user who has come into contact with the reaction presented by the virtual human model. The state generation unit may generate user state information indicating the state of the user based on the user's brain wave information. When the user state information is in a predetermined sense of discomfort state, the information acquisition unit may acquire feedback from the user regarding the reaction presented by the virtual human model. The reaction learning unit may correct the reaction inference model based on the feedback.

[0013] Any of the above reaction generation devices may further include a state learning unit that generates a state inference model for inferring the state of a subject based on subject input information by machine learning the relationship between the subject input information and the subject state information. The reaction learning unit may correct the reaction inference model based on the state of the subject inferred by the state inference model.

[0014] Any of the above reaction generation devices may further include a state learning unit that generates a state inference model for inferring the state of a subject based on subject input information by machine learning the relationship between the subject input information and the subject state information. The reaction learning unit may generate a reaction inference model for each user. The state learning unit may generate a state inference model common to a plurality of users.

[0015] In any of the above reaction generation devices, the reaction learning unit may correct the reaction inference model for each user. The state learning unit does not have to correct the common state inference model.

[0016] Any of the above reaction generation devices may further include a discomfort learning unit that generates a discomfort state inference model for inferring the discomfort state of a user based on subject input information and the reaction of the subject by machine learning the relationship between the subject input information and the reaction of the subject and the discomfort state of the user.

[0017] In any of the above reaction generation devices, the information acquisition unit may further acquire a first reaction of the subject inferred by the reaction inference model based on one piece of subject input information and a second reaction of the subject when the one piece of subject input information is input to the subject. The reaction learning unit may correct the reaction inference model based on the first reaction and the second reaction.

[0018] Any of the above reaction generation devices may further include a state learning unit that generates a state inference model for inferring the state of a subject based on subject input information by machine learning the relationship between the subject input information and the subject state information.

[0019] In any of the above reaction generation devices, the reaction of the subject may be presented by a presentation unit that presents a virtual human model. The information acquisition unit may further acquire the user's electroencephalogram information of the user in contact with the reaction presented by the virtual human model. The state generation unit may generate user state information indicating the state of the user based on the user's electroencephalogram information. When the user state information is in a predetermined uncomfortable state, the information acquisition unit may acquire feedback from the user on the reaction presented by the virtual human model. The state learning unit may correct the state inference model based on the feedback.

[0020] In a second aspect of the present invention, a virtual human presentation system is provided. The virtual human presentation system includes a reaction generation device and a model generation device that generates a virtual human model.

[0021] In a third aspect of the present invention, a reaction generation method is provided. The reaction generation method includes an information acquisition step in which an information acquisition unit acquires subject input information input by a subject and the subject's electroencephalogram information of the subject when the subject input information is input, a state generation step in which a state generation unit generates subject state information indicating the state of the subject based on the subject's electroencephalogram information, and a reaction generation step in which a reaction generation unit generates reaction information indicating the reaction of the subject based on the subject input information and the state of the subject.

[0022] In a fourth aspect of the present invention, a reaction generation program is provided. The reaction generation program causes a computer to execute an information acquisition step of acquiring subject input information input by a subject and the subject's electroencephalogram information of the subject when the subject input information is input, a state generation step of generating subject state information indicating the state of the subject based on the subject's electroencephalogram information, and a reaction generation step of generating reaction information indicating the reaction of the subject based on the subject input information and the state of the subject.

[0023] Note that the above summary of the invention does not list all the features of the present invention. Also, sub-combinations of these feature groups can also be inventions.

Brief Description of the Drawings

[0024]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Embodiments for Carrying Out the Invention

[0025] Hereinafter, the present invention will be described through embodiments of the invention. However, the following embodiments do not limit the invention according to the claims. Also, not all combinations of features described in the embodiments are essential for the solution means of the invention.

[0026] FIG. 1 is a diagram showing an example of a situation in which target person input information 114 is input to a target person 110. The target person input information 114 is information input to the target person 110 and can affect the state S1 (described later) of the target person 110. The target person input information 114 may be voice, may be an action of a communication target 112 that the target person 110 visually recognizes, or may be video. The video may be a moving image or a still image. The target person input information 114 may be a scenery, landscape, scene or scenario.

[0027] In this example, the target person 110 is having a conversation with the communication target 112. In this example, the communication target 112 is a human. The communication target 112 may be a robot having artificial intelligence. In this example, the target person input information 114 is the voice of the utterance "It's a Mother's Day present" of the communication target 112, and the action of the communication target 112 of passing a bouquet of flowers. In this example, the target person 110 is in a state S1 (described later) of being happy due to the target person input information 114 of "It's a Mother's Day present".

[0028] FIG. 2 is a diagram showing another example of a situation in which the subject input information 114 is input to the subject 110. In this example, the subject 110 is listening to sad news such as disasters reported on TV, radio, etc. In this example, the subject input information 114 is the voice of TV, radio, etc. In this example, the subject 110 is in a state S1 (described later) of being sad by listening to the voice reporting the disaster on TV, radio, etc.

[0029] FIG. 3 is a block diagram showing an example of a reaction generation device 100 according to an embodiment of the present invention. The reaction generation device 100 includes an information acquisition unit 10, a state generation unit 20, and a reaction generation unit 30. The reaction generation device 100 may include a presentation unit 40, a storage unit 50, a reaction learning unit 60, a state learning unit 64, a discomfort learning unit 68, and a control unit 90.

[0030] Part or all of the reaction generation device 100 may be realized by a computer. The control unit 90 may be a CPU (Central Processing Unit) of the computer. When the reaction generation device 100 is realized by a computer, a reaction generation program for causing the computer to function as the reaction generation device 100 may be installed in the computer, and an information processing program for causing the computer to execute the information processing method described later may also be installed.

[0031] The information acquisition unit 10 acquires the subject input information 114 and the subject brain wave information of the subject 110 when the subject input information 114 is input. Let the subject brain wave information be the subject brain wave information Ib1. The information acquisition unit 10 may acquire the subject brain wave information Ib1 before and after the subject input information 114 is input. The subject brain wave information Ib1 after the subject input information 114 is input may refer to the subject brain wave information Ib1 when the subject input information 114 is input.

[0032] The information acquisition unit 10 may acquire the user brain wave information of the user 120 (described later). Let the user brain wave information be the user brain wave information Ib2.

[0033] The subject brain wave information Ib1 may be information that reproduces at least a part of the time waveform of the brain waves of the subject 110. The subject brain wave information Ib1 may include data obtained by sampling the time waveform of the brain waves, may include data indicating the magnitudes of the frequency components of the brain waves at one or more frequencies, and may also include other data. The user brain wave information Ib2 may be the same. The subject brain wave information Ib1 may include data indicating the magnitude of at least one component of delta waves (less than 4 Hz), theta waves (4 Hz or more and less than 8 Hz), alpha waves (8 Hz or more and less than 14 Hz), beta waves (14 Hz or more and less than 26 Hz), and gamma waves (26 Hz or more and less than 40 Hz). The user brain wave information Ib2 may be the same.

[0034] Alpha waves may be further classified into lower alpha waves (8 Hz or more and less than 10 Hz), middle alpha waves (10 Hz or more and less than 12 Hz), and higher alpha waves (12 Hz or more and less than 14 Hz) according to the frequency band. The subject brain wave information Ib1 and the user brain wave information Ib2 may also include data indicating the magnitude of at least one of the lower alpha waves, middle alpha waves, and higher alpha waves. Beta waves may be further classified into lower beta waves (14 Hz or more and less than 18 Hz) and higher beta waves (18 Hz or more and less than 26 Hz) according to the frequency band. The subject brain wave information Ib1 and the user brain wave information Ib2 may also include data indicating the magnitude of at least one of the lower beta waves and higher beta waves.

[0035] The subject brain wave information Ib1 may include information on the time waveforms of one or more brain waves measured at one or more positions in the head of the subject 110 including the head and face. The same may apply to the user brain wave information Ib2. For example, the subject brain wave information Ib may be obtained by measuring the time waveforms of the potentials of electrodes arranged at equal intervals near the scalp of the subject 110, such as the international 10-20 method, or may be obtained by other methods. The same may apply to the user brain wave information Ib2. The plurality of electrodes arranged on the scalp may not be at equal intervals. The electrodes may be provided in wearable devices worn on the head of the subject 110, such as a headgear, headphones, earphones, glasses, etc. The subject brain wave information Ib1 may be information obtained by wireless communication of an electrical signal in an electrode embedded in the body of the subject 110. The same may apply to the user brain wave information Ib2.

[0036] Let the sum of the amplitudes of alpha waves, beta waves, theta waves, gamma waves, and delta waves at a certain timing be the overall amplitude As. As an example, if the ratio of the amplitude of the delta wave of the subject 110 to the overall amplitude As is larger than any of the ratios of the amplitude of the alpha wave to the overall amplitude As, the ratio of the amplitude of the beta wave to the overall amplitude As, the ratio of the amplitude of the theta wave to the overall amplitude As, and the ratio of the amplitude of the gamma wave to the overall amplitude As, the subject 110 may be presumed to be in a sleeping state. The same applies to the user 120.

[0037] Let the state of the subject 110 be the state S1. Let the information indicating the state S1 of the subject 110 be the subject state information Is1. The state generation unit 20 generates the subject state information Is1 based on the subject brain wave information Ib1. Let the state of the user 120 (described later) be the state S2. Let the information indicating the state S2 of the user 120 be the user state information Is2. The state generation unit 20 may generate the user state information Is2 based on the user brain wave information Ib2.

[0038] As an example, when the ratio of the amplitude of the theta wave of the subject 110 to the overall amplitude As after the subject input information 114 is input is greater than the ratio of the amplitude of the theta wave of the subject 110 to the overall amplitude As before the subject input information 114 is input, the subject 110 may be in a state S1 where fatigue and drowsiness are increasing. The same may apply to the state S2 of the user 120 (described later).

[0039] As an example, when the ratio of the amplitude of the gamma wave of the subject 110 to the overall amplitude As after the subject input information 114 is input is greater than the ratio of the amplitude of the gamma wave of the subject 110 to the overall amplitude As before the subject input information 114 is input, the subject 110 may be in a state S1 where it is receiving a lot of stimuli. The same may apply to the state S2 of the user 120 (described later).

[0040] As an example, when the ratio of the sum of the amplitudes of the low alpha wave and the middle alpha wave of the subject 110 to the overall amplitude As after the subject input information 114 is input is greater than the ratio of the sum of the amplitudes of the low alpha wave and the middle alpha wave of the subject 110 to the overall amplitude As before the subject input information 114 is input, the subject 110 may be in a state S1 where the degree of relaxation is increasing. The same may apply to the state S2 of the user 120 (described later).

[0041] As an example, when the ratio of the sum of the amplitudes of the high alpha wave and the low beta wave of the subject 110 to the overall amplitude As after the subject input information 114 is input is greater than the ratio of the sum of the amplitudes of the high alpha wave and the low beta wave of the subject 110 to the overall amplitude As before the subject input information 114 is input, the subject 110 may be in a state S1 where the balance between relaxation and concentration is good. The state where the balance between relaxation and concentration is good is the so-called state of being fully absorbed. The same may apply to the state S2 of the user 120 (described later).

[0042] The reaction generation unit 30 generates reaction information indicating the reaction of the subject 110 based on the subject input information 114 and the state S1 of the subject 110. Let the reaction of the subject 110 be the reaction R. Let the reaction information indicating the reaction R be the reaction information Ir.

[0043] The subject input information 114, the subject brain wave information Ib1 when the subject input information 114 is input, the state S1 of the subject 110, and the reaction information Ir may be associated with each other. A plurality of subject input information 114, the respective subject brain wave information Ib1 when each of the plurality of subject input information 114 is input, the respective states S1 of the subject 110, and the respective reaction information Ir of the subject 110 may be associated with each other. The associated subject input information 114, subject brain wave information Ib1, state S1, and reaction information Ir may be stored in the storage unit 50.

[0044] FIG. 4 is a diagram showing an example of a conversation between the user 120 and the virtual character model 130. The virtual character model 130 is a virtual character model that simulates the subject 110. In this example, the virtual character model 130 is presented on the presentation unit 40. The virtual character model 130 may be a robot. The user 120 is a person who converses with the virtual character model 130. In this example, the user input information 124 of the user 120 is input to the reaction generation device 100 (see FIG. 3).

[0045] Similar to the subject input information 114, the user input information 124 may be voice, an action of the user 120, or video. The user input information 124 may be a scenery, landscape, scene, or situation. In this example, the user input information 124 is the voice of the user 120's utterance "It's a Mother's Day present" and the action of the user 120 presenting a bouquet of flowers to the virtual character model 130.

[0046] The reaction generation unit 30 may generate reaction information Ir corresponding to the user input information 124 based on the associated subject input information 114 and reaction information Ir, and the user input information 124. The reaction R related to the reaction information Ir may be presented by the virtual character model 130. In the example of FIG. 1, the subject input information 114 of "It's a present for Mother's Day" and the reaction information Ir of the subject 110 of "Thank you. I'm happy" may be associated and stored in the storage unit 50. In this example, based on the subject input information 114 and the reaction information Ir stored in the storage unit 50 and the above-mentioned speech of the user 120, the virtual character model 130 presents the reaction R of "Thank you. I'm happy."

[0047] The information acquisition unit 10 may acquire the user brain wave information Ib2 of the user 120 in contact with the reaction R presented by the virtual character model 130. In this example, the information acquisition unit 10 acquires the user brain wave information Ib2 of the user 120 in contact with the reaction R of "Thank you. I'm happy." of the virtual character model 130.

[0048] When the user 120 and the virtual character model 130 have a conversation, the subject 110 may be a deceased person or may be alive. The user 120 and the communication target 112 (FIG. 1) may be the same person or may be different persons.

[0049] FIG. 5 is a diagram showing an example of an electroencephalograph 14 capable of measuring the subject brain wave information Ib1 or the user brain wave information Ib2. The electroencephalograph 14 in this example is of a headgear type. The electroencephalograph 14 may be of an earphone type. The subject brain wave information Ib1 or the user brain wave information Ib2 measured by the electroencephalograph 14 may be wirelessly transmitted to the reaction generation device 100 and may be transmitted to the control unit 90 (see FIG. 3) of the reaction generation device 100.

[0050] Subject 110 may input subject input information 114 while wearing a headgear-type or earphone-type electroencephalograph 14. Thereby, information acquisition unit 10 acquires subject electroencephalogram information Ib1 when subject input information 114 is input. Similarly, user 120 (described later) may input user input information 124 (described later) while wearing a headgear-type or earphone-type electroencephalograph 14. Thereby, information acquisition unit 10 acquires user electroencephalogram information Ib2 when user input information 124 (described later) is input.

[0051] Information acquisition unit 10 may further acquire biometric information of subject 110 when subject input information 114 is input. Let the biometric information be biometric information Ig1. Biometric information Ig1 may include at least one of heartbeat information, sweating amount information, and body temperature information of subject 110. Biometric information Ig1 of subject 110 may be acquired by a sensor provided in a wearable device worn by subject 110. State generation unit 20 may generate subject state information Is1 based on subject electroencephalogram information Ib1 and biometric information Ig1.

[0052] Information acquisition unit 10 may further acquire biometric information of user 120 when user input information 124 is input. Let the biometric information be biometric information Ig2. Biometric information Ig2 may include at least one of heartbeat information, sweating amount information, and body temperature information of user 120. Biometric information Ig2 of user 120 may be acquired by a sensor provided in a wearable device worn by user 120. State generation unit 20 may generate user state information Is2 based on user electroencephalogram information Ib2 and biometric information Ig2.

[0053] Let the magnitude of the first power spectrum in the heartbeat of subject 110 be LF1, and the magnitude of the second power spectrum be HF1. Let the magnitude of the first power spectrum in the heartbeat of user 120 be LF2, and the magnitude of the second power spectrum be HF2. The frequency band of the second power spectrum is a higher-frequency band than the frequency band of the first power spectrum. The frequency band of the first power spectrum and the frequency band of the second power spectrum do not have to overlap. The frequency band of the first power spectrum is, for example, 0.04 - 0.15 Hz. The frequency band of the second power spectrum is, for example, 0.15 - 0.4 Hz.

[0054] The state generation unit 20 may generate subject state information Is1 based on the change from the subject brain wave information Ib1 before the input of the subject input information 114 to the subject brain wave information Ib1 after the input of the subject input information 114, and the biological information Ig1. Let the change from the ratio of the amplitude of the brain waves in a predetermined frequency band in the subject brain wave information Ib1 before the input of the subject input information 114 to the total amplitude As of the brain waves in the predetermined frequency band in the subject brain wave information Ib1 after the input of the subject input information 114 be change C1. The state generation unit 20 may generate subject state information Is1 based on change C1 and the ratio of LF1 to HF1 (LF1 / HF1).

[0055] Let the ratio of LF1 to HF1 (LF1 / HF1) after the input of the subject input information 114 be ratio Rag1. Let the predetermined threshold of ratio Rag1 be threshold Pth1. Let the ratio of LF2 to HF2 (LF2 / HF2) after the user 120 contacts the reaction R be ratio Rag2. Let the predetermined threshold of ratio Rag2 be threshold Pth2.

[0056] As an example, when the ratio of the sum of the amplitudes of the high beta waves and gamma waves of the subject 110 after the subject input information 114 is input to the total amplitude As is greater than the ratio of the sum of the amplitudes of the high beta waves and gamma waves of the subject 110 before the subject input information 114 is input to the total amplitude As, and the ratio of LF1 to HF1 (LF1 / HF1) after the subject input information 114 is input is equal to or greater than the threshold value Pth1, it can be inferred that the irritable state, nervous state, or stress state of the subject 110 is increasing. When the ratio of LF1 to HF1 (LF1 / HF1) is equal to or greater than the threshold value Pth1, the subject 110 may be determined to be in a state where the sympathetic nerve is dominant over the parasympathetic nerve. When the ratio of LF1 to HF1 (LF1 / HF1) is less than the threshold value Pth1, the subject 110 may be determined to be in a state where the parasympathetic nerve is dominant over the sympathetic nerve. The threshold value Pth1 may be 2, may be 3, may be 4, or may be 5.

[0057] As an example, when the ratio of the sum of the amplitudes of the high beta waves and gamma waves of the subject 110 after the subject input information 114 is input to the total amplitude As is greater than the ratio of the sum of the amplitudes of the high beta waves and gamma waves of the subject 110 before the subject input information 114 is input to the total amplitude As, and the ratio of LF1 to HF1 (LF1 / HF1) after the subject input information 114 is input is less than the threshold value Pth1, it can be inferred that the excitement state of the subject 110 is increasing.

[0058] Based on the ratio of the amplitude of the brain waves in a predetermined frequency band in the user brain wave information Ib2 before the user 120 contacts the reaction R to the total amplitude As, the change to the ratio of the amplitude of the brain waves in a predetermined frequency band in the user brain wave information Ib2 after contacting the reaction R to the total amplitude As is defined as the change C2. The state generation unit 20 may generate the user state information Is2 based on the change C2 and the ratio of LF2 to HF2 (LF2 / HF2). The state generation unit 20 may generate the subject state information Is2 based on the change C2 and the ratio of LF2 to HF2 (LF2 / HF2).

[0059] The state generation unit 20 may generate subject state information Is1 based on the magnitude relationship between the ratio Rag1 and the threshold Pth1, and the change C1. The state generation unit 20 may generate user state information Is2 based on the magnitude relationship between the ratio Rag2 and the threshold Pth2, and the change C2.

[0060] FIG. 6 is a diagram showing an example of the subject state information Is1. The subject state information Is1 may include information related to a plurality of states (the first state Is1-1 to the nth state Is1-n) of the subject 110. In this example, the subject state information Is1 includes information related to four states (the first state Is1-1 to the fourth state Is1-4) of the subject 110. In FIG. 6, the brain wave of the low frequency f1 refers to at least one of a delta wave, a theta wave, a low alpha wave, and a middle alpha wave, and the brain wave of the high frequency f2 refers to at least one of a high alpha wave, a low beta wave, a high beta wave, and a gamma wave.

[0061] The amplitude of the brain wave of the subject 110, that is, the amplitude of the brain wave in a predetermined frequency band, is set as the amplitude Af1. The amplitude Af1 of the brain wave of the subject 110 before the subject input information 114 is input is set as the amplitude Af1-1. The amplitude Af1 of the brain wave of the subject 110 after the subject input information 114 is input is set as the amplitude Af1-2. The brain wave in the predetermined frequency band may be at least one of a low alpha wave, a middle alpha wave, a high alpha wave, a low beta wave, a high beta wave, a gamma wave, and a theta wave.

[0062] The state generation unit 20 may generate the subject state information Is1 based on the change from the ratio of the amplitude Af1-2 to the total amplitude As to the ratio of the amplitude Af1-1 to the total amplitude As, and the ratio of LF1 to HF1 (LF1 / HF1). The subject state information Is1 may be state information Is1 related to one of the plurality of states of the subject 110 (any one of the first state Is1-1 to the nth state Is1-n).

[0063] In this example, the first state Is1-1 is a state of the subject 110 when, in the brain wave of the low frequency f1, the ratio of the amplitude Af1-2 to the overall amplitude As is larger than the ratio of the amplitude Af1-1 to the overall amplitude As, and the ratio (LF1 / HF1) of LF1 to HF1 after the subject input information 114 is input is equal to or higher than a threshold value. When the subject 110 is in the first state Is1-1, it can be presumed that the fatigue state and drowsiness state of the subject 110 are increasing. When the subject 110 is in the first state Is1-1, the state generation unit 20 may generate a state S1 indicating that the degree of interest of the subject 110 in at least one of the subject input information 114 and the communication target 112 is decreasing.

[0064] In this example, the second state Is1-2 is a state of the subject 110 when, in the brain wave of the low frequency f1, the ratio of the amplitude Af1-2 to the overall amplitude As is larger than the ratio of the amplitude Af1-1 to the overall amplitude As, and the ratio (LF1 / HF1) of LF1 to HF1 after the subject input information 114 is input is less than the threshold value. When the subject 110 is in the second state Is1-2, it can be presumed that the relaxation state of the subject 110 is increasing. When the subject 110 is in the second state Is1-2, the state generation unit 20 may generate a state S1 indicating that the degree of security of the subject 110 with respect to at least one of the subject input information 114 and the communication target 112 is increasing.

[0065] In this example, the third state Is1-3 is a state of the subject 110 when, in the brain wave of the high frequency f2, the ratio of the amplitude Af1-2 to the overall amplitude As is greater than the ratio of the brain wave Af1-1 to the overall amplitude As, and the ratio of LF1 to HF1 (LF1 / HF1) after the subject input information 114 is input is equal to or greater than the threshold value. When the subject 110 is in the third state Is1-3, it can be presumed that the irritation state, hypersensitive state, or stress state of the subject 110 has increased. When the subject 110 is in the third state Is1-3, the state generation unit 20 may generate a state S1 indicating that the alertness of the subject 110 with respect to at least one of the subject input information 114 and the communication target 112 has increased.

[0066] In this example, the fourth state Is1-4 is a state of the subject 110 when, in the brain wave of the high frequency f2, the ratio of the amplitude Af1-2 to the overall amplitude As is greater than the ratio of the brain wave Af1-1 to the overall amplitude As, and the ratio of LF1 to HF1 (LF1 / HF1) after the subject input information 114 is input is less than the threshold value. When the subject 110 is in the fourth state Is1-4, it can be presumed that the state of concentration of the subject 110 has increased. When the subject 110 is in the fourth state Is1-4, the state generation unit 20 may generate a state S1 indicating that the degree of interest of the subject 110 with respect to at least one of the subject input information 114 and the communication target 112 has increased.

[0067] FIG. 7 is a diagram showing an example of the user state information Is2. Similar to the subject state information Is1, the user state information Is2 may include information related to a plurality of states (the first state Is2-1 to the nth state Is2-n) of the user 120. In this example, the user state information Is2 includes information related to four states (the first state Is2-1 to the fourth state Is2-4) of the user 120.

[0068] The amplitude of the brain wave of user 120, which is the amplitude of the brain wave in a predetermined frequency band, is set as amplitude Af2. The amplitude Af2 of the brain wave of user 120 before contacting reaction R (see FIG. 4) is set as amplitude Af2-1. The amplitude Af2 of the brain wave of user 120 after contacting reaction R is set as amplitude Af2-2.

[0069] Similar to the subject state information Is1, the state generation unit 20 may generate user state information Is2 based on the change from the ratio of amplitude Af2-1 in the overall amplitude As to the ratio of amplitude Af2-2 in the overall amplitude As, and the ratio of LF2 to HF2 (LF2 / HF2). The user state information Is2 may be state information Is2 (any one of the first state Is2-1 to the nth state Is2-n) related to one of the multiple states of user 120.

[0070] Similar to the case of FIG. 6, when user 120 is in the first state Is2-1, the state generation unit 20 may generate a state S2 indicating that the degree of interest of user 120 in reaction R is decreasing. When user 120 is in the second state Is2-2, the state generation unit 20 may generate a state S2 indicating that the degree of comfort of user 120 in reaction R is increasing. When user 120 is in the third state Is2-3, the state generation unit 20 may generate a state S2 indicating that the degree of alertness of user 120 in reaction R is increasing. When user 120 is in the fourth state Is2-4, the state generation unit 20 may generate a state S2 indicating that the degree of interest of user 120 in reaction R is increasing.

[0071] Figures 8 and 9 are diagrams showing another example of a conversation between user 120 and virtual character model 130. Figures 8 and 9 are diagrams showing an example of the state S2 of user 120 after contacting the reaction R presented by virtual character model 130. In the example of Figure 8, similar to the example of Figure 4, user 120 is inputting user input information 124 of "It's a present for Mother's Day" into reaction generation device 100 (see Figure 3). In this example, similar to the example of Figure 4, virtual character model 130 is presenting a speech of "Thank you. I'm happy." However, in the example of Figure 8, virtual character model 130 is presenting a reaction R with a blank expression as compared to the example of Figure 4. In the example of Figure 8, user 120 is in a sense of discomfort state of "Huh, is something wrong?" with respect to this reaction R.

[0072] In the example of Figure 9, user 120 is inputting user input information 124 into reaction generation device 100 (see Figure 3). Virtual character model 130 is presenting a reaction R to user input information 124. In the example of Figure 9, user input information 124 is an utterance to virtual character model 130. In the example of Figure 9, user 120 who has contacted the reaction R is in a sense of discomfort state.

[0073] In the examples of Figures 8 and 9, information acquisition unit 10 acquires user brain wave information Ib2 of user 120 who has contacted the reaction R presented by virtual character model 130. State generation unit 20 generates user state information Is2 indicating the state S2 of user 120 based on user brain wave information Ib2. In the examples of Figures 8 and 9, state generation unit 20 generates user state information Is2 indicating that the sense of discomfort with respect to reaction R is increasing based on the user brain wave information Ib2. State generation unit 20 may generate user state information Is2 indicating that the sense of discomfort is increasing when user 120 is in the first state Is2-1 or the third state Is2-3 shown in Figure 7.

[0074] When the user status information Is2 is in a predetermined uncomfortable state, the information acquisition unit 10 may acquire feedback from the user 120 on the reaction R presented by the virtual character model 130. Let the feedback be the feedback Fb. As shown in FIG. 9, the user 120 in contact with the virtual character model 130 may be asked about the presence or absence of discomfort with respect to the reaction R and the content of the discomfort. The feedback Fb may be the answer of the user 120 to the question. The user 120 may answer the question by speaking or in writing. The information acquisition unit 10 may acquire the answer by speaking or in writing.

[0075] The reaction generation unit 30 may correct the reaction information Ir based on the feedback Fb. The reaction generation unit 30 may correct the reaction information Ir based on the feedback Fb so that the uncomfortable state of the user 120 is reduced. The reaction generation unit 30 may correct the reaction information Ir based on the feedback Fb so that the state S2 of the user 120 becomes the second state Is2-2 or the fourth state Is2-4 shown in FIG. 7. Thereby, the reaction generation device 100 can reduce the uncomfortable state of the user 120.

[0076] FIG. 10 is a diagram showing an example of a situation where the subject 110 and the virtual character model 130 are communicating with the communication target 112. The virtual character model 130 is a virtual character model that simulates the person of the subject 110. In this example, the communication target 112 is speaking to the subject 110 and the virtual character model 130 at the same timing. In this example, the speech of the communication target 112 to the subject 110 and the virtual character model 130 is the subject input information 114. In this example, in response to the speech, the subject 110 feels "happy", but the virtual character model 130 presents a non-expressive reaction R compared to the subject 110. In this example, the subject 110 is in an uncomfortable state with respect to this reaction R.

[0077] In this example, the information acquisition unit 10 acquires the subject brain wave information Ib1 of the subject 110 who has come into contact with the reaction R presented by the virtual human model 130. In this example, the state generation unit 20 generates subject state information Is1 indicating that the sense of discomfort with respect to the reaction R is increasing, based on the subject brain wave information Ib1. The state generation unit 20 may generate the subject state information Is1 indicating that the sense of discomfort is increasing when the subject 110 is in the first state Is1-1 or the third state Is1-3 shown in FIG. 6.

[0078] When the subject state information Is1 is in a predetermined sense of discomfort state, the information acquisition unit 10 may acquire feedback Fb regarding the reaction R from the subject 110. The feedback Fb may be the answer of the subject 110 when questioned about the presence or absence of a sense of discomfort with respect to the reaction R. The subject 110 may answer the question in writing or by speech. The information acquisition unit 10 may acquire the answer in writing or by speech.

[0079] The reaction generation unit 30 may correct the subject state information Is1 based on the feedback Fb. The reaction generation unit 30 may correct the subject state information Is1 based on the feedback Fb so that the sense of discomfort state of the subject 110 decreases. The reaction generation unit 30 may correct the subject state information Is1 based on the feedback Fb so that the state S1 of the subject 110 becomes the second state Is1-2 or the fourth state Is1-4 shown in FIG. 6. Thereby, the reaction generation device 100 can reduce the sense of discomfort state of the subject 110.

[0080] FIG. 11 is a diagram showing an example of the reaction inference model 62. The information acquisition unit 10 may acquire the reaction R of the target person 110 presented by the virtual person model 130. The information acquisition unit 10 acquires, for example, the reaction R of "Thank you. I'm happy." of the virtual person model 130 in the example of FIG. 4. In this example, the reaction learning unit 60 (see FIG. 3) machine-learns the relationship between the target person state information Is1 and the reaction R of the target person 110. The reaction learning unit 60 generates the reaction inference model 62 by machine-learning the relationship between the target person state information Is1 and the reaction R. The reaction inference model 62 infers the reaction R of the target person 110 based on the target person state information Is1. Since the reaction inference model 62 machine-learns the relationship between the target person state information Is1 and the reaction R, the reaction R can be inferred based on the target person state information Is1.

[0081] FIG. 12 is a diagram showing another example of the reaction inference model 62. In this example, the reaction learning unit 60 machine-learns the relationship between the target person input information 114 (see FIG. 1), the target person state information Is1, and the reaction R of the target person 110. The reaction learning unit 60 generates the reaction inference model 62 by machine-learning the relationship between the target person input information 114, the target person state information Is1, and the reaction R. The reaction inference model 62 infers the reaction R of the target person 110 based on the target person input information 114 and the target person state information Is1. Since the reaction inference model 62 machine-learns the relationship between the target person input information 114, the target person state information Is1, and the reaction R, the reaction R can be inferred based on the target person input information 114 and the target person state information Is1.

[0082] The reaction learning unit 60 may correct the reaction inference model 62 based on the feedback Fb. The reaction learning unit 60 may correct the reaction inference model 62 based on the feedback Fb so that the discomfort state of the subject 110 or the user 120 is reduced. The reaction learning unit 60 may correct the reaction inference model 62 based on the feedback Fb so that the state S1 of the subject 110 becomes the second state Is1-2 or the fourth state Is1-4 shown in FIG. 6. The reaction learning unit 60 may correct the reaction inference model 62 based on the feedback Fb so that the state S2 of the user 120 becomes the second state Is2-2 or the fourth state Is2-4 shown in FIG. 7. Thereby, the reaction generation device 100 can reduce the discomfort state of the subject 110 or the user 120.

[0083] When the reaction inference model 62 infers the reaction R of the subject 110 based on one piece of subject input information 114 and one piece of subject state information Is1, and the feedback Fb indicating that there is discomfort with the reaction R is obtained by the information acquisition unit 10 from the subject 110 or the user 120, if the reaction learning unit 60 has learned the relationship between one piece of subject input information 114 and one piece of subject state information Is1 and the reaction R, the reaction learning unit 60 may update the reaction inference model 62 based on the feedback Fb. Thereby, the reaction inference model 62 can infer the reaction R with higher accuracy based on one piece of subject input information 114 and one piece of subject state information Is1.

[0084] When the reaction learning unit 60 has not learned the relationship between one piece of subject input information 114 and one piece of subject state information Is1 and the reaction R, the reaction learning unit 60 may add one piece of subject input information 114 and one piece of subject state information Is1 and the reaction R based on one piece of subject input information 114 and one piece of subject state information Is1 as new teacher data to the reaction inference model 62. That the reaction learning unit 60 corrects the reaction inference model may refer to the reaction learning unit 60 updating the reaction inference model 62 and adding new teacher data to the reaction inference model 62.

[0085] In the example of FIG. 11 or FIG. 12, the information acquisition unit 10 may acquire the first reaction of the subject 110 inferred by the reaction inference model 62 based on a piece of subject input information 114, and the second reaction of the subject 110 when the piece of subject input information 114 is input to the subject 110. Let the first reaction of the subject 110 be the first reaction R1, and the second reaction be the second reaction R2. The second reaction R2 is, for example, the reaction R of the subject 110 when the subject input information 114 (the utterance in the example of FIG. 10) is input to the subject 110 in the example of FIG. 10.

[0086] The reaction inference model 62 may correct the reaction inference model 62 based on the first reaction R1 and the second reaction R2. The reaction inference model 62 may correct the reaction inference model 62 based on the difference between the first reaction R1 and the second reaction R2. The first reaction R1 inferred by the reaction inference model 62 and the second reaction R2 of the subject 110 when the subject input information 114 is input to the subject 110 may be different. The reaction inference model 62 may correct the reaction inference model 62 so that the first reaction R1 approaches the second reaction R2. Thereby, the reaction inference model 62 can infer the first reaction R1 with higher accuracy.

[0087] FIG. 13 is a diagram showing an example of the state inference model 66. The state learning unit 64 (see FIG. 3) performs machine learning on the relationship between the subject input information 114 and the subject state information Is1. The state learning unit 64 generates the state inference model 66 by performing machine learning on the relationship between the subject input information 114 and the subject state information Is1. The state inference model 66 infers the state S1 of the subject 110 based on the subject input information 114. Since the state inference model 66 performs machine learning on the relationship between the subject input information 114 and the subject state information Is1, the state S1 can be inferred based on the subject input information 114.

[0088] The state learning unit 64 may correct the state inference model 66 based on the feedback Fb. The state learning unit 64 may correct the state inference model 66 so that the discomfort state of the subject 110 or the user 120 decreases based on the feedback Fb. The state learning unit 64 may correct the state inference model 66 so that the state S1 of the subject 110 becomes the second state Is1-2 or the fourth state Is1-4 shown in FIG. 6 based on the feedback Fb. The state learning unit 64 may correct the state inference model 66 so that the state S2 of the user 120 becomes the second state Is2-2 or the fourth state Is2-4 shown in FIG. 7 based on the feedback Fb. Thereby, the reaction generation device 100 can reduce the discomfort state of the subject 110 or the user 120.

[0089] When the state inference model 66 infers the state information Is1 of one subject based on the input information 114 of one subject, the state learning unit 64 may update the state inference model 66 using the input information 114 of one subject and the state information Is1 of one subject as new learning data. Thereby, the state inference model 66 can infer the state information Is1 of one subject with higher accuracy based on the input information 114 of one subject.

[0090] When the state learning unit 64 has not learned the relationship between the input information 114 of one subject and the state information Is1 of one subject, the state learning unit 64 may add the input information 114 of one subject and the state information Is1 of one subject to the state inference model 66 as new teacher data.

[0091] The reaction learning unit 60 (see FIG. 3) may correct the reaction inference model 62 based on the state S of the subject 110 inferred by the state inference model 66. The reaction learning unit 60 may correct the reaction inference model 62 so that the state S1 of the subject 110 becomes the second state Is1-2 or the fourth state Is1-4 shown in FIG. 6 based on the inferred state S. Thereby, the reaction generation device 100 can reduce the discomfort state of the subject 110 or the user 120.

[0092] FIG. 14 is a diagram showing an example of the relationship between the virtual character model 130 and a plurality of users 120. In this example, each of the plurality of users 120 inputs user input information 124 (see FIG. 8) to the reaction generation device 100. The reaction generation unit 30 generates respective reactions R corresponding to the respective user input information 124.

[0093] The reaction learning unit 60 (see FIG. 3) may generate a reaction inference model 62 for each user 120. The information acquisition unit 10 may have a face authentication unit that authenticates the face of the user 120. The face authentication unit may identify one user 120 by face authentication. The reaction learning unit 60 may generate a reaction inference model 62 corresponding to the one user 120.

[0094] The information acquisition unit 10 may acquire the attributes of the user 120. The attributes of the user 120 refer to, for example, the degree of psychological security of one user 120 (e.g., user 120-1) with respect to another user 120 (e.g., user 120-2). The degree of psychological security may be divided into a plurality of stages. The degree of psychological security may be divided into, for example, two stages of "high" and "low". When the psychological security is high, it means that one user 120 can communicate with the virtual character model 130 without worrying about another user 120. In this case, the content of the speech of one user 120 has a high probability of reflecting the true intention of one user 120. Other users 120 with high psychological security are, for example, the family members, close friends, etc. of one user 120. When the psychological security is low, it means that one user 120 communicates with the virtual character model 130 while worrying about another user 120. In this case, the content of the speech of one user 120 has a high probability of not reflecting the true intention of one user 120. Other users 120 with low psychological security are, for example, others who are not acquaintances of one user 120.

[0095] The attributes of other users (e.g., users 120-2 to 120-n) as seen from one user 120 (e.g., user 120-1) may be acquired in advance and stored in the storage unit 50. The reaction learning unit 60 may generate a reaction inference model 62 based on the attributes of the user 120. When the reaction learning unit 60 generates a reaction inference model 62 for each of two or more users 120, if there are two or more other users 120 with low psychological safety as seen from one of the two or more users 120, a reaction inference model 62 corresponding to the other user 120 may be generated. Thereby, the reaction inference model 62 can infer a safe reaction R for one user 120 when there are other users 120.

[0096] When the reaction learning unit 60 generates a reaction inference model 62 for each user 120, the state learning unit 64 (see FIG. 3) may generate a common state inference model 66 for a plurality of users 120. The state inference model 66 infers a common state S1 regardless of the user 120, and the reaction learning unit 60 machine-learns the relationship between the target state information Is1 related to the state S1 and the reaction R, whereby the reaction learning unit 60 can more easily generate a reaction inference model 62 for each user 120 with higher accuracy.

[0097] The reaction learning unit 60 may correct the reaction inference model 62 for each user 120. When the reaction learning unit 60 corrects the reaction inference model 62 for each user 120, the state learning unit 64 does not have to correct the common state inference model 66. The common state inference model 66 machine-learns the relationship between a plurality of target input information 114 (see FIG. 1) and the target state information Is1 corresponding to each of the plurality of target input information 114. Therefore, the common state inference model 66 can infer the state S1 in which the target 110 is most likely to become for the target input information 114. Therefore, by the reaction learning unit 60 correcting the reaction inference model 62 for each user 120 and the state learning unit 64 not correcting the common state inference model 66, the reaction learning unit 60 can more easily correct the reaction inference model 62 for each user 120 with higher accuracy.

[0098] FIG. 15 is a diagram showing an example of the discomfort state inference model 69. In this example, the discomfort learning unit 68 (see FIG. 3) machine-learns the relationship between the subject input information 114 (see FIG. 1), the reaction R of the subject 110, and the discomfort state of the user 120. When the discomfort learning unit 68 machine-learns the relationship between the subject input information 114, the reaction R of the subject 110, and the discomfort state of the user 120, the reaction R of the subject 110 may be the reaction R inferred by the reaction inference model 62 in the example of FIG. 11 or FIG. 12.

[0099] The discomfort learning unit 68 generates a discomfort state inference model 69 by machine-learning the relationship between the subject input information 114, the reaction R, and the discomfort state. The discomfort state inference model 69 infers the discomfort state of the user 120 based on the subject input information 114 and the reaction R. Since the discomfort state inference model 69 machine-learns the relationship between the subject input information 114, the reaction R, and the discomfort state, the discomfort state of the user 120 can be inferred based on the subject input information 114 and the reaction R.

[0100] FIG. 16 is a block diagram showing an example of the virtual character presentation system 200. The virtual character presentation system 200 includes a reaction generation device 100 and a model generation device 140. The model generation device 140 generates a virtual character model 130 (see FIGS. 4, 8 to 10, and 14).

[0101] FIG. 17 is a flowchart showing an example of the reaction presentation method according to an embodiment of the present invention. The reaction presentation method includes an information acquisition step S100, a state generation step S102, and a reaction generation step S104. The reaction presentation method may include an information acquisition step S110, an information acquisition step S120, a state generation step S122, an information acquisition step S124, a reaction learning step S130, an information acquisition step S132, a state learning step S140, and a discomfort learning step S150. The reaction presentation method according to an embodiment of the present invention will be described by taking the reaction generation device 100 shown in FIG. 3 as an example.

[0102] The information acquisition step S100 is a step in which the information acquisition unit 10 acquires the subject input information 114 input by the subject 110 and the subject brain wave information Ib1 of the subject 110 when the subject input information 114 is input. The state generation step S102 is a step in which the state generation unit 20 generates subject state information Is1 indicating the state S1 of the subject 110 based on the subject brain wave information Ib1. The reaction generation step S104 is a step in which the reaction generation unit 30 generates reaction information Ir indicating the reaction R of the subject 110 based on the subject input information 114 and the state S1 of the subject 110.

[0103] The information acquisition step S100 may be a step in which the information acquisition unit 10 further acquires the biological information Ig1 of the subject 110 when the subject input information 114 is input. The state generation step S102 may be a step in which the state generation unit 20 generates the subject state information Is1 based on the subject brain wave information Ib1 and the biological information Ig1.

[0104] The information acquisition step S100 may be a step in which the information acquisition unit 10 acquires the subject brain wave information Ib1 before and after the subject input information 114 is input. The state generation step S102 may be a step in which the state generation unit 20 generates the subject state information Is1 based on the change from the subject brain wave information Ib1 before the subject input information 114 is input to the subject brain wave information Ib1 after the input and the biological information Ig1.

[0105] The state generation step S102 may be a step in which the state generation unit 20 generates the subject state information Is1 based on the change C1 and the ratio of LF1 to HF1 (LF1 / HF1). The state generation step S102 may be a step in which the state generation unit 20 generates the subject state information Is1 based on the magnitude relationship between the ratio Rag1 and the threshold value Pth1 and the change C1. The state generation step S102 may be a step in which the state generation unit 20 generates the subject state information Is1 based on the change from the ratio of the overall amplitude As occupied by the amplitude Af1-1 to the ratio of the overall amplitude As occupied by the amplitude Af1-2 and the ratio of LF1 to HF1 (LF1 / HF1).

[0106] The information acquisition step S120 is a step in which the information acquisition unit 10 acquires the user brain wave information Ib2 of the user 120 in contact with the reaction R presented by the virtual character model 130. The state generation step S122 is a step in which the state generation unit 20 generates the user state information Is2 indicating the state S2 of the user 120 based on the user brain wave information Ib2. The information acquisition step S124 is a step in which, when the user state information Is2 is in a predetermined sense of discomfort state, the information acquisition unit 10 acquires the feedback Fb from the user 120 with respect to the reaction R presented by the virtual character model 130. The reaction generation step S104 is a step in which the reaction generation unit 30 corrects at least one of the subject state information Is1 and the reaction information Ir based on the feedback Fb.

[0107] The information acquisition step S110 is a step in which the information acquisition unit 10 acquires the reaction R of the target person 110 presented by the virtual person model 130. The reaction learning step S130 is a step in which the reaction learning unit 60 generates a reaction inference model 62 that infers the reaction R of the target person 110 based on the target person state information Is1 by machine learning the relationship between the target person state information Is1 and the reaction R. The reaction learning step S130 may be a step in which the reaction learning unit 60 generates a reaction inference model 62 that infers the reaction R of the target person 110 based on the target person input information 114 and the target person state information Is1 by machine learning the relationship between the target person input information 114, the target person state information Is1, and the reaction R. The reaction learning step S130 may be a step in which the reaction learning unit 60 corrects the reaction inference model 62 based on the feedback Fb acquired in the information acquisition step S124.

[0108] The state learning step S140 is a step in which the state learning unit 64 generates a state inference model 66 that infers the state S1 of the target person 110 based on the target person input information 114 by machine learning the relationship between the target person input information 114 and the target person state information Is1. The state learning step S140 may be a step in which the state learning unit 64 corrects the state inference model 66 based on the feedback Fb acquired in the information acquisition step S124.

[0109] The reaction learning step S130 may be a step in which the reaction learning unit 60 corrects the reaction inference model 62 based on the state S1 of the target person 110 inferred in the state learning step S140. The reaction learning step S130 may be a step in which the reaction learning unit 60 generates a reaction inference model 62 for each user 120. The state learning step S140 may be a step in which the state learning unit 64 generates a state inference model 66 common to a plurality of users 120.

[0110] The reaction learning step S130 may be a step in which the reaction learning unit 60 corrects the reaction inference model 62 for each user 120. The state learning step S140 may be a step in which the state learning unit 64 does not correct the state inference model 66 common to a plurality of users 120.

[0111] The information acquisition step S132 is a step in which the information acquisition unit 10 acquires the first reaction R1 of the target person 110 inferred by the reaction inference model 62 based on the one target person input information 114 and the second reaction R2 of the target person 110 when the one target person input information 114 is input to the target person 110. The reaction learning step S130 may be a step in which the reaction learning unit 60 corrects the reaction inference model 62 based on the first reaction R1 and the second reaction R2. The reaction generation step S104 may be a step in which the reaction generation unit 30 generates the reaction information Ir related to the reaction R inferred by the reaction inference model 62.

[0112] The discomfort learning step S150 is a step in which the discomfort learning unit 68 generates a discomfort state inference model 69 that infers the discomfort state of the user 120 based on the target person input information 114 and the reaction R by machine learning the relationship between the target person input information 114 and the reaction R of the target person 110 and the discomfort state of the user 120. The information acquisition step S124 may be a step in which, when the discomfort state is inferred by the discomfort state inference model 69, the information acquisition unit 10 acquires the feedback Fb from the user 120 for the reaction R presented by the virtual character model 130.

[0113] FIG. 18 is a diagram showing an example of a computer 2200 in which the reaction generation device 100 or the virtual character presentation system 200 according to one embodiment of the present invention may be embodied in whole or in part. The program installed in the computer 2200 can cause the computer 2200 to perform operations associated with the reaction generation device 100 or the virtual character presentation system 200 according to the embodiments of the present invention, or function as one or more sections of the reaction generation device 100 or the virtual character presentation system 200, or execute the operations or the one or more sections, or cause the computer 2200 to execute each stage (see FIG. 17) according to the method of the present invention. The program may be executed by the CPU 2212 to cause the computer 2200 to perform specific operations associated with some or all of the blocks in the flowcharts (FIG. 17) and block diagrams (FIG. 3) described herein.

[0114] A computer 2200 according to one embodiment of the present invention includes a CPU 2212, a RAM 2214, a graphic controller 2216, and a display device 2218. The CPU 2212, the RAM 2214, the graphic controller 2216, and the display device 2218 are interconnected by a host controller 2210. The computer 2200 further includes input / output units such as a communication interface 2222, a hard disk drive 2224, a DVD-ROM drive 2226, and an IC card drive. The communication interface 2222, the hard disk drive 2224, the DVD-ROM drive 2226, and the IC card drive are connected to the host controller 2210 via an input / output controller 2220. The computer further includes legacy input / output units such as a ROM 2230 and a keyboard 2242. The ROM 2230 and the keyboard 2242 are connected to the input / output controller 2220 via an input / output chip 2240.

[0115] The CPU 2212 controls each unit by operating according to the programs stored in the ROM 2230 and the RAM 2214. The graphic controller 2216 causes the image data generated by the CPU 2212 to be displayed on the display device 2218 by acquiring the image data in a frame buffer or the like provided in the RAM 2214 or in the RAM 2214.

[0116] The communication interface 2222 communicates with other electronic devices via a network. The hard disk drive 2224 stores the programs and data used by the CPU 2212 in the computer 2200. The DVD-ROM drive 2226 reads a program or data from the DVD-ROM 2201 and provides the read program or data to the hard disk drive 2224 via the RAM 2214. The IC card drive reads programs and data from an IC card or writes programs and data to an IC card.

[0117] The ROM 2230 stores a boot program or the like executed by the computer 2200 at activation or a program dependent on the hardware of the computer 2200. The input / output chip 2240 may be connected to the input / output controller 2220 via various input / output units through a parallel port, a serial port, a keyboard port, a mouse port, or the like.

[0118] The program is provided by a computer-readable medium such as a DVD-ROM 2201 or an IC card. The program is read from the computer-readable medium, installed in a hard disk drive 2224, a RAM 2214, or a ROM 2230, which are also examples of computer-readable media, and executed by a CPU 2212. The information processing described in these programs is read by the computer 2200, resulting in the cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be configured by realizing the operation or processing of information according to the use of the computer 2200.

[0119] For example, when communication is executed between the computer 2200 and an external device, the CPU 2212 may execute a communication program loaded in the RAM 2214 and instruct the communication interface 2222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 2212, the communication interface 2222 reads the transmission data stored in a transmission buffer processing area provided in a recording medium such as the RAM 2214, the hard disk drive 2224, the DVD-ROM 2201, or the IC card, transmits the read transmission data to the network, or writes the received data received from the network to a reception buffer processing area provided on the recording medium.

[0120] The CPU 2212 may cause all or a necessary part of a file or database stored in an external recording medium such as the hard disk drive 2224, a DVD-ROM drive 2226 (DVD-ROM 2201), or an IC card to be read into the RAM 2214. The CPU 2212 may perform various types of processing on the data on the RAM 2214. The CPU 2212 may then write back the processed data to the external recording medium.

[0121] Various types of information such as various types of programs, data, tables, and databases may be stored in a recording medium and processed. The CPU 2212 may perform various types of processing on the data read from the RAM 2214, including various types of operations, information processing, conditional judgment, conditional branch, unconditional branch, information search or replacement, etc., specified by the instruction sequence of the program described in this disclosure. The CPU 2212 may write back the result to the RAM 2214.

[0122] The CPU 2212 may search for information in files, databases, etc. within the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 2212 searches for an entry that matches the condition where the attribute value of the first attribute is specified from among the plurality of entries, reads the attribute value of the second attribute stored in the entry, and by reading the second attribute value, may obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0123] The above-described program or software module may be stored on the computer 2200 or on a computer-readable medium of the computer 2200. A recording medium such as a hard disk or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as a computer-readable medium. The program may be provided to the computer 2200 by the recording medium.

[0124] As described above, the present invention has been described using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that forms with such changes or improvements can also be included in the technical scope of the present invention.

[0125] In the claims, the specification, and the drawings, for the operations, procedures, steps, stages, and other processes in the apparatus, system, program, and method shown, the execution order of each process, such as the operations, procedures, steps, and stages, is not explicitly stated as "earlier" or "preceding" etc. in particular. It should be noted that, unless the output of the previous process is used in the subsequent process, it can be realized in any order. Regarding the operation flows in the claims, the specification, and the drawings, even if, for convenience, descriptions are made using "first," "next," etc., it does not mean that it is essential to implement in this order.

Explanation of Reference Signs

[0126] 10 ··· Information acquisition unit, 14 ··· Electroencephalograph, 20 ··· State generation unit, 30 ··· Reaction generation unit, 40 ··· Presentation unit, 50 ··· Memory unit, 60 ··· Reaction learning unit, 62 ··· Reaction inference model, 64 ··· State learning unit, 66 ··· State inference model, 68 ··· Discomfort learning unit, 69 ··· Discomfort state inference model, 90 ··· Control unit, 100 ··· Reaction generation device, 110 ··· Subject, 112 ··· Communication target, 114 ··· Subject input information, 120 ··· User, 124 ··· User input information, 130 ··· Virtual human model, 140 ··· Model generation device, 200 ··· Virtual human presentation system

Claims

1. An information acquisition unit that acquires the subject input information input by the subject and the subject brain wave information of the subject when the subject input information is input; A state generation unit that generates subject state information indicating the state of the subject based on the subject brain wave information; A reaction generation unit that generates reaction information indicating the reaction of the subject based on the subject input information and the state of the subject; A reaction generation device comprising the above.

2. The information acquisition unit further acquires the biological information of the subject when the subject input information is input; The state generation unit generates the subject state information based on the subject brain wave information and the biological information; The reaction generation device according to Claim 1.

3. The information acquisition unit acquires the subject brain wave information before and after the subject input information is input; The state generation unit generates the subject state information based on the change from the subject brain wave information before the subject input information is input to the subject brain wave information after input and the biological information; The reaction generation device according to Claim 2.

4. The state generation unit calculates, from the ratio of the amplitude of the brain wave in a predetermined frequency band in the subject brain wave information before the subject input information is input to the total amplitude of the brain wave, the ratio of the amplitude of the brain wave in the frequency band in the subject brain wave information after the subject input information is input to the total amplitude, and the ratio of the magnitude of the first power spectrum in the subject's heartbeat to the magnitude of the second power spectrum, and generates the subject state information based on these ratios; The total amplitude is the sum of the amplitudes of alpha waves, beta waves, theta waves, gamma waves, and delta waves; The frequency band of the second power spectrum is a higher frequency band than the frequency band of the first power spectrum; The reaction generation device according to Claim 3.

5. The state generation unit generates the subject state information based on the ratio of the magnitude of the first power spectrum to the magnitude of the second power spectrum, the magnitude relationship between the ratio and a predetermined threshold of the magnitude of the first power spectrum to the magnitude of the second power spectrum, and the change from the ratio of the amplitude of the brain wave in the frequency band to the overall amplitude before the subject input information is input to the ratio of the amplitude of the brain wave in the frequency band to the overall amplitude after the subject input information is input. The reaction generation device according to claim 4.

6. The subject state information includes information related to a plurality of states of the subject. The state generation unit generates the subject state information related to one of the plurality of states based on the ratio of the magnitude of the first power spectrum to the magnitude of the second power spectrum and the change from the ratio of the amplitude of the brain wave in the frequency band to the overall amplitude before the subject input information is input to the ratio of the amplitude of the brain wave in the frequency band to the overall amplitude after the subject input information is input. The reaction generation device according to claim 4.

7. The brain wave in the frequency band is at least one of delta wave, theta wave, low alpha wave and medium alpha wave, or at least one of high alpha wave, low beta wave, high beta wave and gamma wave. The reaction generation device according to claim 6.

8. The reaction of the subject is presented by a presentation unit that presents a virtual character model. The information acquisition unit further acquires user brain wave information of a user who has contacted the reaction presented by the virtual character model. The state generation unit generates user state information indicating the state of the user based on the user brain wave information. When the user state information is in a predetermined sense of discomfort state, the information acquisition unit acquires feedback from the user on the reaction presented by the virtual character model. The reaction generation unit corrects at least one of the subject state information and the reaction information based on the feedback. The reaction generation device according to any one of claims 2 to 7.

9. The reaction of the subject is presented by a presentation unit that presents a virtual character model. The information acquisition unit further acquires the reaction of the subject presented by the virtual character model. A reaction learning unit is further provided for generating a reaction inference model that infers the reaction of the subject based on the subject state information by machine learning the relationship between the subject state information and the reaction of the subject obtained by the information acquisition unit. The reaction generation device according to any one of claims 2 to 7.

10. The information acquisition unit further acquires user electroencephalogram information of a user who has come into contact with the reaction presented by the virtual character model. The state generation unit generates user state information indicating the state of the user based on the user electroencephalogram information. When the user state information is in a predetermined sense of incongruity state, the information acquisition unit acquires feedback on the reaction presented by the virtual character model from the user. The reaction learning unit corrects the reaction inference model based on the feedback. The reaction generation device according to claim 9.

11. A state learning unit is further provided for generating a state inference model that infers the state of the subject based on the subject input information by machine learning the relationship between the subject input information and the subject state information. The reaction learning unit corrects the reaction inference model based on the state of the subject inferred by the state inference model. The reaction generation device according to claim 9.

12. A state learning unit is further provided for generating a state inference model that infers the state of the subject based on the subject input information by machine learning the relationship between the subject input information and the subject state information. The reaction learning unit generates the reaction inference model for each user. The state learning unit generates a state inference model common to a plurality of the users. The reaction generation device according to claim 10.

13. The reaction learning unit corrects the reaction inference model for each user. The state learning unit does not correct the common state inference model. The reaction generation device according to claim 12.

14. The reaction generation device according to claim 8, further comprising a sense of incongruity learning unit for generating a sense of incongruity state inference model that infers the sense of incongruity state of the user based on the subject input information and the reaction of the subject and the sense of incongruity state of the user by machine learning the relationship therebetween.

15. The information acquisition unit further acquires a first reaction of the subject inferred by the reaction inference model based on one piece of the subject input information, and a second reaction of the subject when one piece of the subject input information is input to the subject, The reaction learning unit corrects the reaction inference model based on the first reaction and the second reaction, The reaction generation device according to claim 9.

16. The reaction generation device according to any one of claims 2 to 7, further comprising a state learning unit that generates a state inference model for inferring the state of the subject based on the subject input information by machine learning the relationship between the subject input information and the subject state information.

17. The reaction of the subject is presented by a presentation unit that presents a virtual human model, The information acquisition unit further acquires user brain wave information of a user who has come into contact with the reaction presented by the virtual human model, The state generation unit generates user state information indicating the state of the user based on the user brain wave information, When the user state information is in a predetermined sense of discomfort state, the information acquisition unit acquires feedback from the user on the reaction presented by the virtual human model, The state learning unit corrects the state inference model based on the feedback, The reaction generation device according to claim 16.

18. A virtual human presentation system including the reaction generation device according to claim 8 and a model generation device that generates the virtual human model.

19. An information acquisition step in which an information acquisition unit acquires subject input information input to a subject and subject brain wave information of the subject when the subject input information is input; A state generation step in which a state generation unit generates subject state information indicating the state of the subject based on the subject brain wave information; A reaction generation step in which a reaction generation unit generates reaction information indicating the reaction of the subject based on the subject input information and the state of the subject; A reaction generation method comprising:

20. On a computer, An information acquisition step of acquiring subject input information input to a subject and subject brain wave information of the subject when the subject input information is input; A state generation step of generating subject state information indicating the state of the subject based on the subject brain wave information; A reaction generation step of generating reaction information indicating the reaction of the subject based on the subject input information and the state of the subject; A reaction generation program for causing the above to be executed.

Images