Augmented space construction system and augmented space construction method

Abstract

To realize an augmented space construction system and augmented space construction method that enable multiple participants participating in an information space to deepen relationships with each other while engaging in communication activities.SOLUTION: While adaptively estimating internal states of corresponding participants based on human and environment information corresponding to each participant, an augmented space construction system evaluates, when each participant participates in a communication activity, the overall and interpersonal human relationships between the participants while integrating internal states of all participants participating in the communication activity. The system successively generates, based on evaluation results, proactive information representing positive words, actions, and attitudes aimed at empathetic understanding between the participants in an information space, and reflects this information to respective participants collectively or individually so as to intervene in communication activities of each participant.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an augmented space construction system and an augmented space construction method, which are suitable for application to an augmented space including a three-dimensional virtual space and a real space constructed on a network, for example. [Background technology]

[0002] In recent years, the term metaverse has come into widespread use to describe a fully immersive, three-dimensional virtual space that includes a virtual environment in which humans are represented by avatars (characters that are the user's alter-ego). One example of such a metaverse is a virtual space experience system that allows a user to view an image of a three-dimensional virtual space constructed on a network, while reflecting the user's human activities in the real world in the virtual space, allowing the user to feel as if he or she is actually present in the virtual space.

[0003] Speech recognition and response generation systems for robot dialogue have also been developed, and for example, a dialogue robot has been proposed that outputs responses to user utterances by voice via a server (see Patent Document 1). In this dialogue robot, the server analyzes the meaning of the user's utterance, and while referring to a response sentence database, selects a response sentence corresponding to the meaning, and outputs the response in the form of voice data.

[0004] Expanding on this, a conference support device has also been proposed that indirectly supports the progress of a conference by supporting a facilitator who manages the progress of the conference, or directly supports the progress of a conference by taking on the role of a facilitator (see Patent Document 2). This conference support device calculates an activity level that indicates the degree of activity of the discussion in the conference based on the biological information of the conference participants, and outputs activity level information that indicates a change in activity level based on the activity level within a predetermined period after the facilitator recognizes the timing of intervention to encourage the participants to speak. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2022-67709 A [Patent Document 2] Patent Publication No. 2022-68904 Summary of the Invention [Problem to be solved by the invention]

[0006] However, when using the conversational robot of Patent Document 1 mentioned above to realize the conference support system of Patent Document 2, there was a problem that even though multiple participants participating in the metaverse were engaged in communication activities, the mere intervention of a facilitator was insufficient in practical terms to deepen the relationships between the participants.

[0007] The present invention has been made in consideration of the above points, and aims to propose an augmented space construction system and an augmented space construction method that enable multiple participants taking part in an information space to deepen their relationships with each other while engaging in communication activities. [Means for solving the problem]

[0008] In order to solve such problems, the present invention provides a human and environment information input / output unit provided for each of a plurality of participants existing in a real space, which acquires human and environment information including brain and nervous system information, musculoskeletal system information, physiological information, psychological information, motion information, behavioral information and environmental information of each participant, a virtual activity construction unit which virtually constructs communication activities of each participant in the real space in real time in an information space based on the human and environment information corresponding to each participant output from the human and environment information input / output unit, an adaptive state estimation unit provided for each participant, which adaptively estimates an internal state of the corresponding participant based on the human and environment information output from the human and environment information input / output unit, and The information space communication system is provided with a human relationship evaluation section which, when participating in a communication activity by a user, evaluates the human relationships among the participants as a whole and between individuals while integrating the internal states of all the participants participating in the communication activity; an action information generation section which generates action information representing positive words, actions and attitudes aimed at empathetic understanding among the participants in the information space based on the evaluation results by the human relationship evaluation section; and an activity reflection section which reflects the action information generated by the action information generation section to all or each participant individually so as to intervene in the communication activities of each participant constructed in the information space.

[0009] As a result, in the augmented space construction system, multiple participants taking part in the information space can deepen their relationships with each other by sharing the content of their activities with each other while reflecting encouraging information to guide communication activities in a positive direction in the information space.

[0010] In addition, in the present invention, the adaptive state estimation unit sets an identification model in which the internal state of each participant is modeled as a state variable, and comprises an identification model estimation unit which estimates internal parameters which are variable characteristics of the identification model based on human and environmental information, a difference estimation unit which estimates the difference as to how close the identification model relating to the internal parameters estimated by the identification model estimation unit has become to the human and environmental information output in real time from the human and environmental information input / output unit, and a parameter identification mechanism unit which corrects the internal parameters constituting the identification model so that the difference estimated by the difference estimation unit becomes smaller, and the action information generation unit changes the content of the action information based on the internal parameters estimated by the identification model estimation unit so that the correction by the parameter identification mechanism unit is gradually reduced.

[0011] As a result, the augmented space construction system facilitates mutual empathy among participants during their interactions in the information space, and by promoting active interaction, it becomes possible to share the internal environment of the information space, experience physical and psychological connections with others, and improve empathy for others.

[0012] Furthermore, in the present invention, an evaluation learning unit is provided which constructs an effectiveness evaluation model which is trained to evaluate the effectiveness of the action information generated by the action information generation unit based on the integrated results of the human and environmental information of each participant, and the action information generation unit adaptively modifies the action information so that it always satisfies the evaluation criteria based on the effectiveness evaluation model constructed by the evaluation learning unit.

[0013] As a result, the augmented space construction system makes it possible to maintain a relatively high quality of interactive information (such as expert knowledge from clinical psychologists) that influences the internal state of each participant while they are active in the information space.

[0014] Furthermore, the present invention is provided with a goodness evaluation learning unit that constructs a goodness evaluation model that is trained to evaluate the goodness of the environmental state of the selected virtual space based on the integrated results of the human and environmental information of each participant, and an environmental state correction unit that adaptively corrects the environmental state of the information space so as to always satisfy the evaluation criteria based on the goodness evaluation model constructed by the goodness evaluation learning unit.

[0015] As a result, the augmented space construction system makes it possible to always maintain the environmental state of the information space, which affects the internal state of each participant, in an optimal state while the participant is active in the information space.

[0016] Furthermore, in the present invention, the action information generation unit includes a correlation quantification unit that quantifies the correlations between the internal states of the participants in a time series manner, and a transition state evaluation unit that evaluates the transition states of the internal states of each participant based on the correlations quantified by the correlation quantification unit, and is adapted to adaptively adjust the action information according to the evaluation results by the transition state evaluation unit.

[0017] As a result, the augmented space construction system is able to grasp, as a whole system, the process by which each participant's internal state changes as it is influenced by the words, actions, and attitudes of other participants, making it possible to respond (interact) appropriately to each participant.

[0018] Furthermore, in the present invention, when each participant engages in communication activities in the information space, a spatial layer construction unit is provided that constructs a spatial layer for expanding the information space in cooperation with the participants in the information space.

[0019] As a result, in the augmented space construction system, it becomes possible for participants to cooperate with each other within the information space and expand the information space.

[0020] Furthermore, the present invention includes a learning model training unit that trains a learning model constructed by statistical and mathematical processing to map the content of collaboration between participants to a spatial extension of the information space based on the spatial layer construction process constructed by the spatial layer construction unit, and the spatial layer construction unit automatically constructs a spatial layer based on the learning model trained by the learning model training unit, even without collaboration between the participants.

[0021] As a result, the augmented space construction system is allowed to learn the process of expanding the information space by learning, and it becomes possible to continue the expansion process automatically without the cooperation of each participant.

[0022] Furthermore, in the present invention, when the learning model is trained by the learning model training unit, a layer management unit is provided that manages the spatial layers constructed in the information space to be expanded for the purpose of maintaining order.

[0023] As a result, in the extended space construction system, when continuing the expansion process of the information space, it is possible to prevent each participant from suffering any disadvantage by carrying out management aimed at maintaining order.

[0024] Furthermore, in the present invention, the system includes a first step of acquiring human and environmental information, which is provided for each of a plurality of participants existing in the real space and includes brain and nervous system information, musculoskeletal system information, physiological information, psychological information, motion information, behavioral information, and environmental information of each participant; a second step of virtually constructing a communication activity of each participant in the real space in an information space in real time based on the human and environmental information corresponding to each participant acquired in the first step; a third step of adaptively estimating an internal state of the corresponding participant for each participant based on the human and environmental information output from the human and environmental information input / output unit; a fourth step of evaluating, when each participant participates in the communication activity, the overall and interpersonal relationships among the participants while integrating the internal states of all the participants participating in the communication activity; a fifth step of successively generating encouraging information representing positive words, actions, and attitudes aimed at empathetic understanding between the participants in the information space based on the evaluation results in the fourth step; and a sixth step of reflecting the encouraging information generated in the fifth step on all or each participant individually so as to intervene in the communication activity of each participant constructed in the information space.

[0025] As a result, in the augmented space construction system, multiple participants taking part in the information space can deepen their relationships with each other by sharing the content of their activities with each other while reflecting encouraging information to guide communication activities in a positive direction in the information space. Effect of the Invention

[0026] According to the present invention, it is possible to realize an augmented space construction system and an augmented space construction method that enable multiple participants taking part in an information space to deepen relationships with each other while engaging in communication activities. [Brief description of the drawings]

[0027] [Figure 1] FIG. 1 is a conceptual diagram illustrating an augmented space construction system according to an embodiment of the present invention. [Diagram 2] 2 is a conceptual diagram showing an internal configuration of the interface device shown in FIG. [Diagram 3] 1A and 1B are a perspective view and six-view diagrams showing the external configuration of a vital sign measuring device. [Figure 4] FIG. 2 is a conceptual diagram showing the configuration of a brain activity measuring unit. [Diagram 5] FIG. 2 is a conceptual diagram showing the configuration of the biological signal measurement wearing tool. [Figure 6] 10 is a conceptual diagram for explaining a state in which the biosignal measurement wearing device is worn. FIG. [Figure 7] FIG. 13 is a conceptual diagram showing an additional configuration of a vital sign measuring device. [Figure 8] 1 is a graph showing pulse waveforms before and after application of a filter. [Figure 9] 1 is a conceptual diagram showing an ROI of a near-infrared image and an ROI of a far-infrared image. [Figure 10] FIG. 1 is a conceptual diagram illustrating nose breathing and mouth breathing. [Figure 11] FIG. 1 is a conceptual diagram for explaining temperature conditions when breathing through the nose and through the mouth. [Figure 12] 1 is a graph showing changes in nostril temperature caused by nasal breathing and changes in oral cavity temperature caused by mouth breathing. [Figure 13] This is a graph showing a change in breathing technique from nose breathing to mouth breathing and superimposing each state content. [Figure 14] FIG. 1 is a conceptual diagram illustrating non-contact measurement of oxygen saturation using light. [Figure 15] FIG. 2 is a block diagram showing the internal configuration of a motion capture device. [Figure 16] FIG. 1 is a conceptual diagram for explaining a method for acquiring three-dimensional skeletal information regarding a participant's entire body. [Figure 17] FIG. 2 is a conceptual diagram illustrating a structure of a behavior recognition model. [Figure 18] FIG. 11 is a conceptual diagram mainly showing the internal configuration of an interface device according to another embodiment. [Figure 19] FIG. 11 is a conceptual diagram mainly showing the internal configuration of an interface device according to another embodiment. [Figure 20] FIG. 11 is a conceptual diagram mainly showing the internal configuration of an interface device according to another embodiment. [Figure 21] FIG. 11 is a conceptual diagram mainly showing the internal configuration of an interface device according to another embodiment. [Figure 22] FIG. 11 is a conceptual diagram mainly showing the internal configuration of an interface device according to another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] An embodiment of the present invention will now be described in detail with reference to the drawings.

[0029] (1) Configuration of the extended space construction system according to this embodiment 1 shows an augmented space construction system 1 according to this embodiment. In the augmented space construction system 1, an interface device 2 is provided for each participant, which acquires various information from the participant himself / herself and the surrounding environment existing in the real space, and the various information transmitted from each interface device 2 is reflected in an information space constructed in a cloud 3.

[0030] As shown in FIG. 2, each interface device 2 includes a human / environment information input / output unit 10, a virtual activity construction unit 11, an adaptive state estimation unit 12, a human relationship evaluation unit 13, an approach information generation unit 14, and an activity reflection unit 15.

[0031] The human / environment information input / output unit 10 has vital sign measuring devices (measuring means for electrocardiogram, pulse rate, pulse pressure, oxygen saturation, body temperature, brain waves, blood sugar level, blood pressure, and electromyogram, etc.) 20, a facial imaging camera 21, a sound collecting microphone 22, motion capture devices (optical, inertial sensor, mechanical and magnetic types, or a combination of these) 23, and peripheral measuring devices (thermometer, hygrometer, audible and ultrasonic ambient sound collectors, etc.) 24, and an information detection unit 25 that detects human / environment information, which will be described later, based on the output of these sensors and devices, etc.

[0032] In this human / environmental information input / output unit 10, information detection unit 25 detects the subject's brain / nervous system information, musculoskeletal system information, and physiological information based on the detection results from vital sign measuring device 20, and detects psychological information by analyzing the subject's facial expressions and voice using face imaging camera 21 and sound collecting microphone 22, and further detects the subject's movement information and behavior information using motion capture device 23, and detects surrounding environmental information centered on the subject using peripheral measuring device 24.

[0033] The virtual activity construction unit 11 virtually constructs the human activity of the subject in the real space in the information space in real time based on the human and environmental information of the subject output from the human and environmental information input / output unit 10.

[0034] Here, the information space is a virtual space generated by a computer, and is typified by a three-dimensional virtual space constructed on a network (cloud 3) and the metaverse, which is a service for the virtual space. The functionality required for this information space is that it must be capable of two-way communication with the real space via a network. When using the information space as a versatile platform, it is desirable that the system that provides the information space is open and that it has user management and access restriction functions. Examples of information spaces that satisfy such functions include Second Life (registered trademark) provided by Linden Lab, Inc., and OpenSim, which is provided under the BSD License (Berkeley Software Distribution License).

[0035] The adaptive state estimation unit 12 adaptively estimates the internal state of the corresponding participant for each participant based on the human and environmental information output from the human and environmental information input / output unit 10. This internal state is information that contributes to empathy between participants sharing the same space, and is a precursor state to the emission of utterances, facial expressions, behaviors, etc. that lead to the force promoting the empathy.

[0036] When each participant participates in a communication activity, the human relationship evaluation unit 13 integrates the internal states of all the participants participating in the communication activity and evaluates the intra-group and inter-personal human relationships among the participants.

[0037] The outreach information generation unit 14 successively generates outreach information expressing positive words, actions, and attitudes aimed at empathetic understanding between participants in the information space, based on the evaluation results by the human relationship evaluation unit 13. The activity reflection unit 15 reflects the outreach information generated by the outreach information generation unit 14 to all or individual participants, so as to intervene in the communication activities of each participant constructed in the information space.

[0038] As a result, in the augmented space construction system 1, multiple participants taking part in the information space can deepen their relationships with each other by sharing the content of their activities with each other while reflecting encouraging information to guide communication activities in a positive direction in the information space.

[0039] (2) Detailed configuration and functions of the human and environmental information input / output unit (2-1) Configuration of vital sign measuring devices Specifically, as shown in Fig. 3(A), the vital sign measuring device 20 has an electrode terminal group 30 and a photoelectric probe window 31 arranged in a predetermined pattern on a measurement surface 20X of a device body 20A, and measures the subject's biological information through the measurement surface 20X. The biological information targeted by the vital sign measuring device 20 is at least one of the measurement results of electrocardiogram, pulse, pulse pressure, oxygen saturation, body temperature, brain waves, blood glucose level, blood pressure, and electromyogram. In other words, the vital sign measuring device 20 is configured so that the same device body can be used as a measuring means for all kinds of biological information.

[0040] 3(B), the vital sign measuring device 20 has a sound collecting microphone 22 detachably connected to a connection terminal 32 provided on the side of the device body 20A, and when connected, can collect vibrations generated from the subject's larynx and pharynx as acoustic signals. As a result, it is possible to obtain information on vibrations (inaudible range) associated with the subject's heart and breathing that cannot be heard by the ears.

[0041] (2-2) Method for detecting brain and nervous system information This human / environment information input / output unit 10 has a brain activity measurement unit 40 that is attached to the subject's head and measures brain waves and cerebral blood flow in a measured area of ​​the head as an additional component of the vital sign measuring device 20. The information detection unit 25 converts the transmission state of the central nervous system centered on the head based on the measurement data of the brain waves and cerebral blood flow obtained from the brain activity measurement unit 40 into quantitative nervous system data.

[0042] The brain activity measuring section 40 according to this embodiment is in line with the contents described in Japanese Patent Application Laid-Open No. 5283700 by the present inventor, and uses the same principle.

[0043] 4(A), the brain activity measuring section 40 has a blood flow measuring section 41, an electrocardiograph 42, a first control section 43, and a wireless communication device 44. The blood flow measuring section 41 measures the state of blood flow from changes in the optical path and transmitted light amount of laser light caused by blood flow, and further measures the state of brain activity from the state of blood flow in the brain.

[0044] That is, when the laser light irradiated to the blood enters the blood layer, it passes through the blood as both light components, the normal light component reflected and scattered by red blood cells and the light component reflected and scattered by the attached thrombus. Since the influence of the laser light during the process of passing through the blood layer changes from moment to moment depending on the state of the blood, it is possible to observe various changes in the state of the blood by continuously measuring the amount of transmitted light (amount of reflected light) and observing the change in the amount of light.

[0045] The blood flow measuring unit 41 is composed of a net-like base 45 formed in a hemispherical shape corresponding to the outer shape of the head so as to be attached to the head, and a number of sensor units 46A to 46N. Each of the sensor units 46A to 46N is supported at a predetermined interval by the net-like base 45, and outputs a detection signal of the amount of transmitted light measured at each measurement point on the head to a first control unit 43. The electrocardiograph 42 measures cardiac potential generated in response to heart movement by electrodes E attached to the subject's skin.

[0046] The first control unit 43 derives the displacement of blood vessels and tissues around the blood vessels due to blood flow based on the light intensity when the light (FIG. 4(B)) emitted from the light-emitting unit 47 of each sensor unit 46A-46N is received by the light-receiving unit 48, and measures the brain activity state (distribution of red blood cells). The first control unit 43 also stores a control program that performs calculation processing to cancel components due to oxygen saturation included in the signals obtained from at least two or more light-receiving units 48.

[0047] Furthermore, the first control unit 43 derives the displacement of the inner wall of the blood vessel based on the displacement of the blood vessel and the tissue surrounding the blood vessel. Furthermore, the first control unit 43 determines the pulse wave velocity at each measurement position from the phase difference between the waveform of the electrocardiogram signal from the electrocardiograph 42 and the waveform of the detection signal obtained from the light receiving unit, and derives the displacement state of the inner wall of the blood vessel from the pulse wave velocity.

[0048] The wireless communication device 44 wirelessly transmits the measurement results (blood flow data) output from the first control unit 43 to the vital sign measuring device 20. The brain activity measuring unit 40 has optical sensor units 46A-46N arranged on a net-like base 45, and therefore can simultaneously measure blood flow throughout the entire head.

[0049] When measuring cerebral blood vessel characteristics, as shown in Fig. 4(B), the first control unit 43 selects an arbitrary sensor unit 46D from the multiple arranged sensor units 46A to 46N, and causes the light-emitting unit 47 of the sensor unit 46D to emit laser light. At this time, the laser light emitted from the light-emitting unit 47 is output at a wavelength λ (λ ≈ 805 nm) that is not affected by oxygen saturation.

[0050] When the blood flow measuring unit 41 of the brain activity measuring unit 40 is attached to the subject's head, the elasticity of the net-like base 45 positions the multiple sensor units 46A-46N at each measurement point on the head, and holds each measurement surface facing the head surface. With the blood flow measuring unit 41 attached to the head, the multiple sensor units 46A-46N can measure the vascular characteristics of the middle cerebral artery and anterior cerebral artery (percentage of vascular elasticity, amount of plaque in blood vessels, percentage of arteriosclerosis) by measuring the pulse wave velocity in the cerebral arteries from the change in the amount of light received after irradiating the brain surface with light and propagating inside the brain.

[0051] The principle of measuring the pulse wave velocity of the cerebral artery will now be described. For example, the pulse wave velocity due to blood flowing through the middle cerebral artery and the anterior cerebral artery is detected by each of the sensor units 46A-46N arranged corresponding to the measurement positions that receive light propagating through the brain when light is irradiated. The measurement method is a method of comparing the waveform of the cardiac potential obtained from the electrocardiograph 42 with the waveform of the signal output from each of the sensor units 46A-46N at the measurement positions, finding the pulse wave velocity from the phase difference, and deriving the vascular characteristics corresponding to the pulse wave velocity.

[0052] Next, the principle of detecting blood vessel characteristics from cerebral blood flow will be described. As shown in FIG. 4(B), the brain BR is covered with cerebrospinal fluid BR1, skull BR2, and scalp BR3. A part of the laser light emitted from the light emitting unit 47 of each sensor unit 46A to 46N of the blood flow measuring unit 41 is reflected by the scalp BR3, but the remaining light passes through the scalp BR3, skull BR2, and cerebrospinal fluid BR1 and proceeds into the brain BR. The light irradiated to the head and proceeds to the brain propagates in the radial direction (depth direction and radial direction) in an arc-shaped pattern CP as shown by the dashed line in the figure.

[0053] The light propagation path through the brain becomes longer as it moves away from the base point BP where the laser light is irradiated in the radial direction, and the light transmittance decreases. Therefore, the light receiving level (amount of transmitted light) of the sensor unit 46B adjacent to the sensor unit 46A on the light emitting side at a predetermined distance is detected as strong. The light receiving level (amount of transmitted light) of the sensor unit 46C installed next to the sensor unit 46B at a predetermined distance is detected as weaker than the light receiving level of the sensor unit 46B. The light receiving section 48 of the sensor unit 46A on the light emitting side also receives light from the brain.

[0054] The first control unit 43 uses detection signals corresponding to the light intensities received by the multiple sensor units 46A to 46N as measurement data, and derives vascular characteristics at each measurement position by comparing the waveform with the waveform of the cardiac potential signal from the electrocardiograph 42. It is also possible to obtain graphic data showing the distribution of arteriosclerosis according to the pulse wave velocity by mapping the detection results.

[0055] Therefore, it becomes possible to measure changes in blood flow through the middle cerebral artery and the anterior cerebral artery based on the detection signal waveforms of the sensor units 46A to 46N, and to detect the pulse wave velocity in the brain based on the measurement data of the changes in blood flow.

[0056] Thus, the vital sign measuring device 20 is equipped with a brain activity measuring unit 40 that is attached to the subject's head and measures the brain waves and cerebral blood flow in the measured area of ​​the head, and the information detecting unit 25 adds the transmission state of the central nervous system centered on the head based on the measurement data of the brain waves and cerebral blood flow obtained from the brain activity measuring unit 40 as nervous system data. In this way, the transmission state of the central nervous system from the subject's brain can be quantitatively converted into data and acquired, making it possible to monitor the subject's progress and transition state.

[0057] As a result, the state of transmission from the subject's brain to the central nervous system can be quantitatively converted into data and the subject's progress and transition state can be monitored.

[0058] (2-3) Method for detecting musculoskeletal system information The human / environment information input / output unit 10 has a biological signal measurement garment 50 worn by the subject and a joint circumference detection unit (not shown) as additional components of the vital sign measuring device 20. The joint circumference detection unit has an actuator that is actively or passively driven in conjunction with the limb movement of the subject, and detects physical quantities around the joints associated with the limb movement of the subject based on the output signal from the actuator.

[0059] The physical quantities around the joints include the muscle forces generated in the musculoskeletal system, the range of motion of each joint, the speed of motion and reaction speed, the autonomous control characteristics against disturbances, the parameter identification results of at least one of the frame's moment of inertia, mass and center of gravity, the impedance adjustment results of the joint system including flexor and extensor muscles (viscosity characteristics due to friction), electrical physical quantities (command signals), etc.

[0060] The biosignal measurement wearing device 50 worn by the subject is consistent with the contents and uses the same principle as those described in Japanese Patent Application Publication No. 5409637 by the present inventor. Figure 5(A) shows the biosignal measurement wearing device 50 in this embodiment. This biosignal measurement wearing device 50 is formed so as to cover the subject's body surface and has a wearing device main body 50A worn by the subject, and the inner surface of this wearing device main body 50A (the surface that comes into contact with the subject's body surface when worn) has a biosignal sensor 53 for detecting a biopotential signal at at least one position where a biosignal can be measured from the subject's body.

[0061] A biosignal is a signal caused by electricity generated in the subject's body, which is a measurable signal from the body and changes over time with the movement of the body and signals from the brain. For example, a biosignal means a signal generated by the activity of a living body, such as a nerve potential, a muscle potential, an electroencephalogram, a cardiac potential, and furthermore a potential generated by motion artifacts (influence of movement), a potential generated by a biochemical reaction, and a vibration such as a pulse wave generated by the beating of the heart. The generated potential (amplitude, density, time delay) and the generation site of the biosignal are also taken into account in the quantitative data conversion.

[0062] In Fig. 5(A), the biosignal measurement garment 50 of this embodiment has a symmetrical structure, so the left half of the figure shows the structure of the back side of the left leg, and the right half of the figure shows the structure of the front side of the left leg. The garment main body 50A is provided with a group of biosignal sensors 54A to 54F consisting of a plurality of biosignal sensors 53 that detect the biosignals of the subject. The biosignal sensors 53 are arranged at equal intervals along the flow of the muscles in the subject's leg.

[0063] In the wearing device main body 50A, a plurality of biosignal sensors 53 are arranged in a portion corresponding to the buttocks of the subject (biological signal sensor group 54A) along the flow of the gluteus maximus muscle. Similarly, in a portion corresponding to the back of the thigh of the subject (biological signal sensor group 54B), the sensors are arranged along the flow of the biceps femoris muscle, the semimembranosus muscle, and the semitendinosus muscle, and in a portion corresponding to the calf (biological signal sensor group 54C), the sensors are arranged along the flow of the triceps surae muscle. In a portion corresponding to the front of the hip joint (biological signal sensor group 54D), the sensors are arranged along the flow of the adductor longus muscle and the iliopsoas muscle, in a portion corresponding to the front of the thigh (biological signal sensor group 54E), the sensors are arranged along the flow of the quadriceps femoris muscle, and in a portion corresponding to the shin (biological signal sensor group 54F), the sensors are arranged along the flow of the tibialis anterior muscle, the soleus muscle, and the extensor digitorum longus muscle.

[0064] 5(B) is a diagram showing an example of a measurement module connected to a group of biosignal sensors provided on the wearing device main body 50A. Note that, since all of the groups of biosignal sensors 54A to 54F provided on the wearing device main body 50A are connected to the measurement module 55 in the same manner, the group of biosignal sensors 54A will be described as an example here.

[0065] 5(B) is a schematic diagram showing a biosignal sensor group 54A provided on the wearing device main body 50A and a measurement module 55 connected to the biosignal sensor group 54A. The biosignal sensors 53 constituting the biosignal sensor group 54A are provided insulated from each other, and are each connected to the measurement module 55 via a conductive wiring. An address is assigned to each of the biosignal sensors 53.

[0066] The measurement module 55 is connected to a plurality of biosignal sensors 53 constituting the biosignal sensor group 54A, and includes a measurement module controller 56 that selects at least two of the biosignal sensors 53 and acquires a biosignal by calculating a difference between detection signals detected by the selected biosignal sensors 53, a memory 57 that records the acquired biosignals, and a communication unit 58 that transmits to the outside the sequentially acquired biosignals and / or the biosignals recorded in the memory 57. The measurement module 55 is provided for each of the biosignal sensor groups 54A to 54F, and each biosignal sensor 53 is connected thereto.

[0067] This measurement module controller 56 has an electronic circuit that can sequentially select at least two biosignal sensors 53 from the multiple connected biosignal sensors 53 in response to a command signal input via the communication unit 58, and acquire the biosignals between these two selected biosignal sensors 53.

[0068] The measurement module controller 56 further includes signal processing means such as a filter for removing or extracting a predetermined frequency component from the thus acquired biosignal, and an amplifier for amplifying the thus acquired biosignal. The thus acquired biosignal is output from the measurement module controller 56 to the communication unit 58 and / or the memory 57. When selecting the biosignal sensor 53, the measurement module controller 56 may operate each biosignal sensor 53 in a predetermined order, or may select the biosignal sensor 53 at the address designated by a designation signal input via the communication unit 58.

[0069] The communication unit 58 is composed of a thin antenna and a communication circuit connected to the antenna. The communication unit 58 transmits, via the antenna, to the second control unit 59, measurement information including a signal including a biosignal output from the measurement module controller 56 and information such as an address indicating the position information of the biosignal sensor 53 that detected the biosignal, and / or a signal including a biosignal read from the memory 57 and information such as an address indicating the position information of the biosignal sensor 53 that detected the biosignal.

[0070] This second control unit 59 generates a designation signal that specifies which biosignal sensor 53 to select from the group of biosignal sensors 54A to 54F provided on the wearing device main body 50A, as well as signals for starting and ending data acquisition, and selects the designated biosignal sensor 53 in accordance with the signal to measure the biosignal.

[0071] In this way, according to the biosignal measurement wearing device 50 of the present invention, the wearing device main body 50A is provided with the biosignal sensor group 54A-54F consisting of the multiple biosignal sensors 53, so that multiple biosignal sensors 53 can be placed at predetermined locations at once just by wearing the wearing device main body 50A, and it becomes possible to detect biosignals at each point by adhering them to the skin surface. In this way, even when monitoring biosignals at multiple points on the subject's body surface, it is possible to save the trouble of attaching and peeling off the biosignal sensors 53 one by one, and the biosignals can be easily measured.

[0072] In addition, by detecting bio-signals using the plurality of bio-signal sensors 53 in this manner, it is possible to measure bio-signals at a plurality of points within the area in which the bio-signal sensor groups 54A to 54F are arranged. Then, by mapping the measurement data at each point according to the address assigned to each bio-signal sensor 53, it is possible to measure the distribution of bio-signals in the subject's body.

[0073] The second control unit 59 obtains, from the biosignals obtained by the measurement module 55, a myoelectric potential signal associated with muscle activity of the subject and a nerve transmission signal for operating the musculoskeletal system of the subject.

[0074] The information detection unit 25 converts the activity state of the musculoskeletal system centered on the joint and / or the transmission state of the peripheral nervous system into quantitative musculoskeletal system data and / or nervous system data based on the physical quantities around the joint described above and either or both of the myoelectric potential signal and the nerve transmission signal obtained from the second control unit 59, and converts the transmission state of either or both of the central nervous system and the peripheral nervous system centered on a desired location into quantitative nervous system data.

[0075] As a result, by having the subject perform limb movements, the activity state of the musculoskeletal system and the transmission state of the peripheral nervous system can be quantitatively converted into data based on the physical quantities around the joints and the biological signals obtained from the surface parts of the body relative to the joints, making it possible to monitor the subject's progress and transitional state.

[0076] In addition, biosignals can be obtained from body surface areas other than the areas referenced by the subject's joints, and the transmission state of not only the peripheral nervous system around the joints but also the central nervous system and peripheral nervous system from desired areas can be quantitatively converted into data and obtained, making it possible to monitor the subject's progress and transitional state.

[0077] For example, as shown in Fig. 6, by disposing biosignal sensors 54A-54F on the whole body of the subject, it becomes possible to visualize the state transition of the information transmission pathway of the brain and nervous system. In Fig. 6, by dividing the whole body of the subject into predetermined regions (for example, regions controlled by sensory nerves such as dermatomes), it becomes possible to grasp the neural connections of the whole body from a bird's-eye view and for each region. Therefore, if it is possible to detect which region has a problem with the neural transmission, it becomes possible to grasp not only the defective location but also the degree of influence on either the central side or the peripheral side.

[0078] (2-4) Method for detecting the activity state of physiological systems In the human / environment information input / output unit 10, as additional components of the vital sign measuring device 20, a near-infrared detection unit 60 and a far-infrared detection unit 61 are provided, as shown in FIG. 7, and the activity state of the subject's physiological system and nervous system is detected using a non-contact method.

[0079] That is, the human / environment information input / output unit 10 uses the near-infrared detection unit 60 and the far-infrared detection unit 61 to irradiate infrared rays mainly onto the face of the subject, thereby simultaneously and non-invasively measuring fluctuations in the subject's pulse rate and changes in skin temperature in daily life.

[0080] The near-infrared detection unit 60 irradiates the region of interest of the subject's face, including the cheeks, with near-infrared light and receives the light reflected from the face to generate a near-infrared image. The far-infrared detection unit 61 irradiates the region of interest of the subject's face with far-infrared light and detects the skin temperature of the face.

[0081] The third control unit 62 estimates the subject's pulse rate based on the period of change in the intensity waveform in the region of interest of the reflected light received by the near-infrared detection unit 60. The third control unit 62 extracts areas of the subject's face that are susceptible to and less susceptible to the activity of the autonomic nervous system from the near-infrared image generated by the near-infrared detection unit 60, and calculates the temperature difference in the skin temperature of each area from the detection result of the far-infrared detection unit 61.

[0082] A. Pulse measurement method based on infrared light The light-absorbing substances in biological tissues are water and hemoglobin in blood. Water has strong absorption properties for infrared rays with wavelengths longer than 1350 [nm], while hemoglobin has strong absorption properties for visible light with wavelengths shorter than 650 [nm]. In non-invasive biological diagnosis using light, infrared to near-infrared light with wavelengths between 650 [nm] and 1350 [nm], which has high biological penetration, is often irradiated and the reflected light and transmitted light containing biological information are measured.

[0083] Pulse is a physiological phenomenon in which the volume of blood vessels changes due to the work of blood pressure caused by cardiac output. Changes in blood vessel volume are brought about by changes in blood flow. Changes in blood flow change the amount of light absorbed by hemoglobin in the blood, so the pulse rate can be estimated from the period of change in the transmitted and reflected light of light irradiated onto a living body.

[0084] In the present invention, the subject is not required to wear a measuring device and can be measured day or night. The pulse rate is estimated by measuring the period of change in reflectance of near-infrared light irradiated on the living body using a near-infrared camera. Since near-infrared light is invisible, the measurement can be performed at night without burdening the subject. In order to estimate the stress state from the pulse rate, it is necessary to measure the pulse rate, which fluctuates minutely due to the activity of the autonomic nervous system.

[0085] The third control unit 62 implements an algorithm for extracting a waveform period required for calculating the pulse rate from the waveform data obtained from the near-infrared detection unit 60.

[0086] The near-infrared detection unit 60 measures the intensity of the irradiated near-infrared light reflected by biological tissue. This near-infrared detection unit 60 measures the subject at rest, but minute body movements and facial movements due to breathing are superimposed on the pulse waveform as motion artifacts that indicate changes in the amount of reflected light.

[0087] To remove these motion artifacts, a digital filter is first applied to the frequency band of the pulse rate. The range of the resting pulse rate is set to 35 to 180 [bpm], and motion artifacts may also be superimposed on the measured waveform in the 0.5 to 3.0 [Hz] band, but the amplitude of the waveform superimposed on the pulse waveform is significantly large.

[0088] Therefore, when the variance value of the waveform data within a certain period of time exceeds a certain threshold, the third control unit 62 recognizes that the subject is moving and removes the data during that period as unreliable. The waveform after the bandpass filter has a bimodal feature due to the arterial blood flow. To make it easier to extract feature points, a moving average filter is applied to make the two-peak waveform into a single peak.

[0089] Figures 8(A) and (B) show the waveforms before and after applying a band-pass filter and a moving average filter to the measured waveform, respectively. It can be seen that the motion artifacts superimposed on the pulse waveform have been removed by filtering.

[0090] Next, there are two main methods known for calculating pulse rate from the light intensity waveform measured by a near-infrared camera.

[0091] In the first method, the cheek area of ​​the face was set as the region of interest (ROI), and the reflected light intensity data for 30 seconds from the cheek area was obtained by measuring with an RGB camera. The frequency of the frequency spectrum obtained by a spectrum analyzer was compared with the frequency determined from the pulse rate indicated by the pulse oximeter. However, the noise caused by body movement was significantly large in the pulse wave waveform. When attempting to calculate the pulse rate using this type of frequency analysis, it is necessary to greatly restrict the movement of the person being measured.

[0092] On the other hand, in the second method, the time required for one pulse beat is calculated from the period of the characteristic points of the waveform detected from the time series data of the measured waveform, and the pulse rate is calculated by dividing that time by 60. Compared to the method of calculating the pulse rate from frequency analysis, which requires continuous measurement data for a certain period of time, the method of calculating the pulse rate from the period of the measured characteristic points can remove motion artifacts by removing the period of the characteristic points that are erroneously detected due to noise caused by body movement. In addition, the minimum measurement time required is shorter than that of frequency analysis, and the pulse rate can be calculated from several beats, making it possible to calculate the pulse rate at shorter intervals.

[0093] However, because the resting pulse rate of a healthy person is 60-80 [bpm], the period of the characteristic points obtained from a 5-second measurement is 5-7 times, and there is a problem that the amount of data obtained is insufficient to calculate a more accurate pulse rate. Also, as shown in Figure 8(A), the waveform of the intensity change has two peaks in sync with the arterial blood flow rate, which causes erroneous detection of the period of the characteristic points, which is a factor that causes errors in the calculated pulse rate.

[0094] For daily measurements, the body part to be measured needs to be exposed regardless of day or night. Therefore, in the vital sign measuring device 20 according to the present invention, the ROI is set on the cheek of a person. The location of the set ROI is shown in FIG. 9(A). The average of the intensity values ​​measured at the pixels within the set ROI was taken as the measurement value, and the change in the measurement value over time was recorded as pulse waveform data. The interval t between the upper ends of the waveform was used as the period of the characteristic points of the waveform data. tp Not only the bottom end spacing t bp By adding as another feature point, the number of feature points that serve as reference data for calculating the pulse rate is increased.

[0095] Next, to extract the feature points necessary for calculating the pulse rate, the detected t tp ,t bp Among them, there are false positives that do not indicate a pulse cycle. tp , t bp The algorithm for removing is shown in (a) to (f) below. Here, the set before removal is denoted as S0.

[0096] (a) t tp ,t bp For each of these, there is a peak interval t i is the previous peak interval t i-1 If there is a fluctuation of 30% or more, the peak interval t i is removed from the set S0. The set S1 of selected peak intervals is expressed by the following equation (1).

number

[0097] (b) Calculate the standard deviation σ0 of the set S0 and find the peak interval t that varies within the range of 2σ0. i is removed from the set S1. The set S2 of selected peak intervals is expressed by the following equation (2).

number

[0098] (c) Calculate the standard deviation σ2 of the set S2 and find the peak interval t i is removed from the set S2. The set S3 of selected peak intervals is expressed by the following formula (3).

number

[0099] (d) Calculate the standard deviation σ3 of the set S3 and find the peak interval t i is removed from S3. The set S4 of selected peak intervals is expressed by the following formula (4).

number

[0100] (e) The peak interval blood remaining in set S4 is converted to pulse rate, and a histogram is created in increments of 5 bpm. At this time, the pulse rate width in one section of the histogram is the same, but the width of the peak intervals corresponding to that pulse rate is different. Therefore, a weighted histogram is created using the following formula (5).

number

[0101] Here, t 30-35 is the time interval between peaks of pulse rate 30 [bpm] and 35 [bpm], t i-i+5 is the time interval between the peaks of pulse rates i (bpm) and i+5 (bpm), and h i-i+5 is the frequency of peak intervals from pulse rate i (bpm) to i+5 (bpm) before correction, and h'i-i+5 represents the correction value of the frequency of the peak interval from pulse rate i [bpm] to i+5 [bpm].

[0102] (f) For the weighted histogram values, a window of width 20 [bpm] is set so that the sum of the frequencies of the four adjacent intervals is maximized. i Remove.

[0103] According to the above algorithm, the average value of the extracted peak intervals is set as the peak width t within the measurement time, and the pulse rate is calculated by dividing 60 seconds by the peak interval t.

[0104] In this way, the third control unit 62 sets feature points at the upper and lower ends of the intensity waveform of the reflected light obtained from the near-infrared detection unit 60, and estimates the pulse rate based on the peak intervals between the upper ends and between the lower ends, which are the periods of the feature points. The third control unit 62 also calculates the standard deviation of the multiple peak intervals, and removes peak intervals that do not indicate a pulse period using the standard deviation as a reference. Furthermore, the third control unit 62 creates a weighted histogram for a predetermined time unit from the pulse rate converted based on the remaining removed peak intervals, and corrects the frequency of the peak intervals based on the histogram.

[0105] Thus, the third control unit 62 can calculate an accurate pulse rate from the measured waveform of the pulse waveform data for a relatively short period of time based on the pixel group within the set ROI.

[0106] B. Skin temperature measurement using far-infrared detector Heat generated inside the body is transported to the body surface by conduction and convection. However, since biological tissue itself has poor thermal conductivity and acts as a thermal barrier, most of the heat transport to the skin is by skin blood flow. Skin blood flow changes depending on the activity of the autonomic nervous system, mainly the contraction and expansion of blood vessels by the sympathetic and parasympathetic nervous systems.

[0107] Previous studies have attempted to estimate the activity of the autonomic nervous system by measuring changes in skin temperature. However, since the heat distribution on the face is significantly affected by the subject's hairstyle and whether or not they are wearing glasses, the human / environment information input / output unit of the present invention sets the measurement site appropriate for estimating the activity of the autonomic nervous system from near-infrared images.

[0108] In order to estimate stress state from changes in skin temperature, it is necessary to set the ROI for measurement to the parts of the face where temperature changes due to stress are prominent. Among the parts of the face, the nose in particular has a high concentration of arteriovenous anastomoses (AVA), which are easily affected by the activity of the autonomic nervous system, making it suitable for measuring skin temperature to estimate stress state.

[0109] In addition, because skin temperature is affected by the outside temperature, changes in skin temperature must be recorded as relative temperature. In this case, the ROI used to measure the standard for skin temperature changes in the nose must be set in a part of the face that is less affected by autonomic nervous system activity. The forehead is one part of the face that has low AVAs and is less affected by autonomic nervous system activity.

[0110] From the above, the third control unit 62 according to the present invention records the change in skin temperature as the temperature difference between the region of interest ROI_n in the nose and the region of interest ROI_fh in the forehead. The locations of the ROIs are shown in FIG. 9(B). The equation for calculating the temperature change is shown in equation (6).

number

[0111] Here, T n is the average skin temperature measured at all pixels in ROI_n, T fh is the average skin temperature measured at all pixels in ROI_fh, T r is the relative temperature of the forehead and nose. When setting each ROI, the coordinates of the nose and forehead in the near-infrared image are converted to coordinates on the far-infrared image.

[0112] Specifically, when converting coordinates from a near-infrared (NIR) image to a far-infrared (FIR) image, the resolution and angle of view differ between near-infrared and far-infrared cameras, so the coordinate information of the near-infrared image is made to correspond to the pixels of the far-infrared image.

[0113] That is, the x coordinate on the far-infrared image is X fir The horizontal angle of view of the near-infrared camera is θ nir_h , the horizontal angle of view of the far-infrared camera is θ fir_h Then, it can be expressed as the following equation (7).

number

[0114] The y coordinate on the far-infrared image, Yfir, is the vertical angle of view of the near-infrared camera, θ nir_v , the vertical angle of view of the far-infrared camera is θ fir_v Then, it can be expressed as the following equation (8): where d represents the difference in the optical axis, and L represents the distance to the object.

number

[0115] Thus, following the automatic detection of the measurement target area using image information from the near-infrared detection unit 60, the third control unit 62 can accurately calculate the relative temperature difference between the skin temperatures of the subject's forehead and nose based on the image information from the far-infrared detection unit 61.

[0116] C. Respirometry based on Skin Temperature Breathing is related to the functioning of the autonomic nervous system and the immune system, and is used to diagnose sleep disorders. There are two types of breathing: nasal breathing (Fig. 10(A)) and mouth breathing (Fig. 10(B)). Nasal breathing is the natural way to breathe. When breathing through the mouth, the inhaled air does not pass through the cilia and mucous membranes of the nasal cavity. Mouth breathing can also cause various diseases, including sleep apnea syndrome (SAS).

[0117] As shown in Figures 11(A) and (B), the nostrils and oral cavity are warmed by exhaled air (35-36°C) and cooled by inhaled air (23-28°C). For this reason, regions of interest (ROIs) are set at the subject's nostrils and oral cavity, and the temperature change at the nostrils and oral cavity due to breathing is measured by a far-infrared detection unit (far-infrared camera). Since the surface temperature of the skin is affected by the outside air temperature, it is measured as the relative temperature between the nostrils and oral cavity.

[0118] In practice, since it is difficult to automatically detect the measurement target area using a skin temperature distribution map, the measurement target area (nostrils and oral cavity) is automatically detected using image information from the near-infrared detection unit (camera) as described above, and the region of interest (ROI) is set. The nostril temperature is relatively high in nasal breathing, whereas the oral cavity temperature is relatively low in mouth breathing.

[0119] After measuring the subject's nasal breathing and mouth breathing for 30 seconds each, we took a one-minute break and then repeated the same breathing measurement four times. As a result, as shown in Figure 12(A), we confirmed that the temperature of the nostrils decreased due to inhalation during nasal breathing. In addition, as shown in Figure 12(B), we confirmed that the temperature of the oral cavity also decreased in accordance with mouth breathing.

[0120] Furthermore, as shown in Fig. 13(A), the opening and closing of the mouth due to the change in breathing method from nose breathing to mouth breathing was also confirmed based on the distance between the upper and lower lips. When the graphs of Fig. 12(A), (B) and Fig. 13(A) are superimposed, it is shown as Fig. 13(B).

[0121] The method for extracting the inhalation timing involves first using a moving average filter to remove high frequency noise from the measurement results of the relative temperatures of the nostrils and oral cavity, and then performing first differentiation to extract the timing of the temperature drop due to exhalation.

[0122] Furthermore, if the nostril temperature and oral temperature drop at the same time, the intra-alveolar pressure becomes negative relative to atmospheric pressure, causing air to flow into the nasal passages, making it possible to distinguish this as mouth breathing.

[0123] In this way, it is possible to calculate the respiration rate and distinguish the respiration method using a non-contact measurement method. It is possible to measure nose breathing and mouth breathing simultaneously, and by executing an algorithm that takes into account the characteristics of temperature changes during each breath, it is possible to accurately calculate the respiration rate and distinguish the respiration method.

[0124] D. Oxygen saturation using two near-infrared wavelengths A pulse oximeter is a medical device that uses the light absorption characteristics of hemoglobin in blood. The pulse oximeter is a medical device that is attached to the fingertip to measure blood oxygen saturation and pulse rate.

[0125] In the present invention, since blood oxygen saturation varies depending on respiratory rate, the degree of binding between hemoglobin and oxygen, and cardiac output, we focused on the difference in the light absorption characteristics between oxygenated hemoglobin (HbO2) and reduced hemoglobin (Hb).

[0126] In the conventional measurement method, infrared light (wavelength around 660 [nm]) and near-infrared light (wavelength around 900 [nm]) are irradiated onto the fingertip from a light-emitting element, and the blood oxygen saturation is estimated from the light intensity ratio of each transmitted light measured by a light-receiving element, while the pulse rate is also estimated using the periodic change in the light intensity of the transmitted light. Infrared light, which is visible light, has been used because there is a large difference in the light absorption coefficient between oxygenated hemoglobin (HbO2) and reduced hemoglobin (Hb).

[0127] However, in the present invention, non-contact measurement is performed using only invisible near-infrared (NIR) light (FIG. 14). As a result, by alternately turning on and off light-emitting elements that irradiate two types of near-infrared light, 800 (or 760) [nm] and 900 [nm], and photographing the region of interest (ROI) irradiated by each light-emitting element with a single near-infrared camera (near-infrared detection unit 60), it is possible to measure the blood oxygen saturation of the subject centered on the ROI in a non-contact manner even in a dark environment.

[0128] (2-5) Method for detecting nervous system activity The autonomic nervous system consists of the sympathetic nervous system, which functions when the body is in a state of tension or activity, and the parasympathetic nervous system, which functions when the body is at rest. When the sympathetic nervous system is dominant, blood pressure and pulse rate increase, whereas when the parasympathetic nervous system is dominant, blood pressure and pulse rate decrease, so autonomic nervous function and cardiac function are highly correlated.

[0129] One method for measuring autonomic nervous function is to measure cardiac sympathetic nervous function using heart rate variability. Specifically, there is the CVR-R (Coefficient of Variation of RR intervals) method, which uses the RR intervals of the electrocardiogram waveform to calculate the coefficient of variation for evaluation. Secondly, there is a method in which the fluctuation power ratio of the frequency components (high frequency components and low frequency components) of heart rate variability is used as an index of sympathetic nervous activity.

[0130] There is also a method for measuring autonomic nervous function by measuring the sympathetic nervous function of the vascular system using pulse waves. One such method is to calculate the magnitude of amplitude fluctuation of the photoplethysmogram (PPG) waveform as an evaluation value of autonomic nervous function.

[0131] In the present invention, the third control unit 62 can contribute to the evaluation of the subject's autonomic nervous function (presence or absence of disturbance, etc.) based on all or a combination of the subject's pulse rate detected using the near-infrared detection unit 60 and the far-infrared detection unit 61 constituting the human / environment information input / output unit 10, the relative temperature difference between the skin temperatures of the subject's forehead and nose, the subject's respiratory rate and breathing method, and the subject's oxygen saturation.

[0132] (2-6) Psychological information detection method In the human / environment information input / output unit 10, the information detection unit 25 analyzes the facial expression and voice of the subject using the face image pickup camera 21 and the sound collection microphone 22 to detect psychological information.

[0133] Specifically, the information detection unit 25 executes a face analysis algorithm to detect the facial expressions of the target person (including not only expressions such as sadness, joy, anger, sympathy, and frivolity, but also flushed skin, goose bumps, frequency of opening and closing of eyelids, tears, etc.), and at the same time, executes a voice analysis algorithm to detect the tone and pitch of the target person's voice (tone of voice, elation, etc.).

[0134] This facial analysis algorithm is designed to approximate human interpretation (experience, preconceptions and expectations) and is derived from the accumulated results of how emotions and psychology are reflected in facial expressions, based on the state and changes of the human face.

[0135] Similarly, voice analysis algorithms are designed to approximate human interpretation (experience, biases and expectations) and are derived from the cumulative results of how emotions and psychology are reflected in the human voice based on tone and pitch.

[0136] Using this voice analysis algorithm, it is possible to estimate changes in the subject's psychological state through the subject's everyday communication and narrative analysis, based on keywords used in the subject's speech, the intonation of the speech, the volume of the voice, the structure of the narrative, the continuity / discontinuity of communication, etc., as well as differences with these accumulated data.

[0137] In addition, the information detection unit 25 is also capable of estimating the psychological state of the subject based on the subject's personal information (heart rate, electrocardiogram, pulse, blood pressure, body temperature, skin blood flow distribution, skin temperature distribution, sweat, odorous substances, etc.) detected by the vital sign measuring device 20 described above, and the subject's environmental information detected by the peripheral measuring device 24 described below.

[0138] In this way, the information detection unit 25 is able to estimate and grasp a person's psychological information comprehensively from both the person information obtained by sensing and the person information obtained by communication analysis and narrative analysis.

[0139] (2-7) Method for detecting the activity state of the motor system (motion information and behavior information) In the human / environment information input / output unit 10, the information detection unit 25 detects the motion data recognized by the motion capture device 23 as the active state of the motor system of the subject.

[0140] 15, the motion capture device 23 is arranged in indoor equipment such as the subject's home, and is composed of a control unit 70 that controls the entire device, and an RGB-D sensor 71 and an IMU (Inertial Measurement Unit) sensor 72 that are connected to the control unit 70. The IMU sensor 72 is composed of a three-axis acceleration sensor, a three-axis angular velocity sensor, and a geomagnetic sensor, and is capable of measuring three-axis acceleration, angular velocity, and geomagnetism.

[0141] The RGB-D sensor 71 has a depth sensor that can measure the distance to an object as seen from the camera in addition to the RGB color camera function, and can perform 3D scanning of the object. When the LiDAR camera L515 from RealSense (a trademark of Microsoft Corporation) is used as the RGB-D sensor 71, for example, the depth sensor is made up of a LiDAR sensor, and measures the time it takes for laser light to be emitted, hit an object, and bounce back, thereby measuring the distance and direction to the object.

[0142] In fact, the RGB-D sensor 71 is disposed so as to capture an image of the entire body of the subject based on the standing position of the subject in the indoor equipment, and sequentially obtains an RGB image and a depth image as the imaging result.

[0143] The control unit 70 has a fourth control unit 73 consisting of a CPU (Central Processing Unit) that controls the entire system, and a data storage unit 74 in which various data is stored in a database that can be read and written in response to instructions from the fourth control unit 73.

[0144] The fourth control unit 73 is composed of an image coordinate setting unit 80, a coordinate group extraction unit 81, a skeletal information acquisition unit 82, and a behavior recognition unit 83. The image coordinate setting unit 80 estimates the posture of the entire body of the subject based on the RGB images sequentially acquired from the RGB-D sensor 71, and sets a plurality of skeletal representative points attached to the entire body of the subject as image coordinates.

[0145] Specifically, the image coordinate setting unit 80 sets image coordinates in a two-dimensional coordinate system in which the origin of coordinates is either the left or right shoulder of the subject, the direction of the line connecting the left and right shoulders is the X-axis, and the direction perpendicular to the X-axis is the Z-axis, based on the origin of coordinates. That is, the origin of coordinates is moved to the right or left shoulder, and the XZ plane is rotated so as to match the line connecting the left and right shoulders. As a result, it becomes possible to recognize actions with increased robustness regardless of the position of the RGB-D sensor 71.

[0146] The coordinate group extraction unit 81 sequentially extracts image coordinate groups related to the subject's actions based on the transition state of the subject's posture from the image coordinate groups set by the image coordinate setting unit 80. In reality, the coordinate group extraction unit 81 needs to extract key parts of the whole body from the actions, assuming that the subject's clothing and surrounding environment may change.

[0147] Therefore, the coordinate group extraction unit 81 adopts a lightweight OpenPose architecture (a system that estimates a person's skeleton by deep learning) using a trained model to estimate the posture of the entire body and obtain image coordinates of the skeleton representative points of the fingers of both hands. For example, for the upper body of the subject, the coordinate group extraction unit 81 sequentially extracts a total of 60 skeleton representative points, consisting of 18 points on the torso of the upper body of the subject and 42 points on the fingers of both hands (21 points on each finger), as image coordinate groups. Note that by using only stages 1 and 2 out of the six stages of the OpenPose architecture, the amount of calculation can be reduced by 32.4%.

[0148] The skeletal information acquisition unit 82 acquires temporally continuous three-dimensional skeletal information centered on the fingers of both hands of the subject by time-synchronizing the image coordinate groups sequentially extracted by the coordinate group extraction unit 81 with the depth images sequentially acquired from the RGB-D sensor 71.

[0149] Specifically, the skeletal information acquisition unit 82 constructs a three-dimensional coordinate system by combining the depth direction in the depth images sequentially acquired from the RGB-D sensor 71 with the two-dimensional coordinate system of the image coordinate groups sequentially extracted by the coordinate group extraction unit 81, and acquires three-dimensional skeletal information.

[0150] The conversion equation from the two-dimensional image coordinate system to the three-dimensional world coordinate system is expressed as the following equation (1).

number

[0151] K, R, and t were calculated by camera calibration using Zhang's method ("A flexible new technique for camera calibration". IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000.). For the calibration to calculate K, a calibration board with circular dots printed in a grid pattern was used. The central coordinates of each dot on the calibration board were calibrated to 1 / 1,000.

[0152] As a result, in the motion capture device 23, as shown in FIG. 16, the relationship between the position in three-dimensional space and the position of the captured image for the entire body of the subject can be clarified, making it possible to acquire three-dimensional skeletal information while maintaining high robustness regardless of the imaging position of the RGB-D sensor 71.

[0153] The behavior recognition unit 83 recognizes a behavior, which is a connection of a plurality of local actions, from a transition state of the subject's posture based on the three-dimensional skeletal information sequentially acquired by the skeletal information acquisition unit 82.

[0154] The behavior recognition unit 83 uses the behavior recognition pattern set for each local behavior as teacher data, and sequentially recognizes corresponding local behaviors from the three-dimensional skeletal information sequentially acquired by the skeletal information acquisition unit 82, while referring to the hand-washing behavior recognition model constructed by deep learning.

[0155] Specifically, in order to recognize multiple local actions, the behavior recognition unit 83 applies a behavior recognition model consisting of three modules, a convolutional neural network (CNN) layer, a batch normalization layer (Batch Norm), and an activation function layer (tanh function), as well as a fully connected layer, as shown in FIG. 17.

[0156] In the action recognition model, it is possible to input skeletal information from the current frame and the frame two frames before it and calculate the likelihood for each of multiple types of local movements related to actions.

[0157] Furthermore, the behavior recognition unit 83 performs post-processing of the most frequent behavior among the behaviors observed in 15 frames (the current frame and the 14 frames before it) as the current local behavior as the recognition result. In this way, the behavior recognition unit 83 trained the behavior recognition model by supervised learning using a unique dataset. For optimization, a cross-entropy loss function and Adam (adaptive movement estimation) with a learning rate of 0.001 were used.

[0158] As a result, the motion capture device 23 can significantly improve the accuracy and speed of recognizing local movements based on three-dimensional skeletal information.

[0159] (2-8) Method for detecting environmental information In the human / environment information input / output unit 10, the surrounding measuring device 24 has a thermometer, a hygrometer, a barometer, an audible and ultrasonic ambient sound collector, etc., and acquires environmental information around the subject. This environmental information is mainly the temperature, humidity, and ambient sound around the subject.

[0160] (3) Adaptive identification of internal states In the augmented space construction system 1 of the present invention, when multiple participants taking part in an information space take part in a communication activity in that information space, they are allowed to share the content of their activities with each other while reflecting encouraging information to guide the communication activity in a positive direction.

[0161] The adaptive state estimation unit 12 in the interface device 2 for each participant adaptively estimates the internal state of the corresponding participant based on the human and environment information output from the human and environment information input / output unit 10. As shown in Fig. 18 in which the same reference numerals are used for parts corresponding to those in Fig. 2, the adaptive state estimation unit 12 has an identification model estimation unit 90, a difference estimation unit 91, and a parameter identification mechanism unit 92.

[0162] The identification model estimation unit 90 sets up an identification model in which the internal state of each participant is modeled as a state variable, and estimates internal parameters, which are variable characteristics of the identification model, based on the human and environmental information output from the human and environmental information input / output unit 10.

[0163] The difference estimation unit 91 estimates the difference as to how close the identification model related to the internal parameters estimated by the identification model estimation unit 90 has become to the human and environmental information output in real time from the human and environmental information input / output unit 10.

[0164] The parameter identification mechanism unit 92 corrects the internal parameters constituting the identification model so that the difference estimated by the difference estimation unit 91 becomes smaller.

[0165] In this way, the adaptive state estimation unit 12 estimates the internal parameters corrected by the parameter identification mechanism unit 92 as the internal state of the participant.

[0166] As mentioned above, "internal states" are information that contributes to empathy between participants who are sharing the same space. Therefore, in understanding this empathy, it is often explained as a basic counseling attitude (an active listening attitude) using the three principles of Carl Rogers' theory.

[0167] The three principles of Carl Rogers' theory are "Empathic Understanding," "Unconditional Positive Regard," and "Congruence," which he positions as the core conditions for therapeutic personality change to occur.

[0168] "Empathic understanding" means sensing the other person's inner world as if it were you, and understanding what they are trying to say as if you were in their position. It is also necessary to make an effort to communicate this to the other person. The above-mentioned "as if" means that it is not actually possible to grasp the other person's inner world, so it is necessary to virtually identify (estimate) it in information space. By providing this as feedback to the other person and clarifying the difference, further feedback can bring the person closer to the next stage of self-concordance.

[0169] "Unconditional positive consideration" means listening to what the other person is saying without judging it as good or bad, or judging what you like or dislike. It also means not denying what the other person is saying, but listening with positive interest to find out why they think that way. Genuine interest in the other person's problems makes them feel at ease when talking. While smoothly acquiring human and environmental information from the other person, be conscious of system identification (identification of the other person's model) and try to continue transmitting information as much as possible. It is a good idea to increase the accuracy of system identification by applying identification input for system identification as appropriate.

[0170] Furthermore, "self-concordance" means experiencing and accepting one's own emotions as they are, and being authentic as a human being (what one says and what one does are the same). In order to improve the accuracy of one's understanding of the other person, it is a good idea to provide feedback to a virtually prepared identification model of the other person (system identification).

[0171] In this case, it is important not only to identify the other person's model, but also for the other person to identify their own model. Therefore, instead of reacting through an idealized model of yourself, you feed back your own natural reaction. If your false ideal self matches your true self, your and the other person's models will identify each other, and while receiving feedback from each other, you will adaptively identify each other's models and get closer to each other. In this way, it leads to the "empathic understanding" mentioned above as a form of mutual understanding.

[0172] In the end, "empathy" is defined as "empathic understanding" on the premise that complete empathy with others cannot be achieved. Furthermore, in order to promote "empathy," we must constantly work to have the other person accept the input necessary for model identification with a positive attitude so that we can empathize, and based on the information obtained from the other person, we must provide feedback so that the difference between the information obtained from the other person from the results of identification and state estimation and the information that can be considered empathic understanding becomes smaller, while matching the model with reality so that the difference between the other person's state obtained as empathic understanding and their actual reactions and narration becomes smaller.

[0173] Next, when each participant participates in a communication activity, the human relationship assessment unit 13 evaluates the overall and interpersonal human relationships among the participants while integrating the internal states of all the participants participating in the communication activity.

[0174] These human relationships represent the correlation of intellect, emotion, and will among the participants as a whole and among individuals, and refer to the relationship between people that integrates psychology, emotions, personality, and attitudes. These include relationships between specific participants (one-to-one), relationships between a specific participant and multiple participants (one-to-many), and relationships between multiple participants and multiple participants (many-to-many).

[0175] The encouraging information generating unit 14 successively generates encouraging information representing positive words, actions and attitudes aimed at empathetic understanding between participants in the information space, based on the evaluation results by the human relationship evaluating unit 13.

[0176] That is, the influencing information is information that represents expressive behaviors and attitudes that enable the participant to more freely experience his or her inner feelings, perceptions, and personal meanings in order to convey an empathetic understanding that is accurate with moment-to-moment sensitivity in the here and now, based on the results of the evaluation of the participants' intrapersonal and interpersonal relationships.

[0177] Furthermore, the action information generating unit 14 changes the content of the action information based on the internal parameters estimated by the identification model estimating unit 90 so that the correction by the parameter identification mechanism unit 92 is gradually reduced.

[0178] The activity reflecting unit 15 reflects the encouraging information generated by the encouraging information generating unit 14 to all or each participant individually so as to intervene in the communication activities of each participant constructed in the information space.

[0179] In this case, the activity reflection unit 15 reflects not only conversation but also non-verbal body language, facial expressions, tone of voice, breathing pattern, behavior, etc. as means of intervention in communication activities based on the initiation information, thereby fully promoting empathy between participants while playing the role of a facilitator.

[0180] What the facilitator does is to understand the participants and environmental information sharing the information space, and then determine the intention of what state the facilitator wants to bring about, and then change the input needed to facilitate according to that intention.

[0181] Specifically, there are two points to keep in mind when facilitating. First, if you are able to understand the participants sharing the space and the environmental information, you can use identification input (language, behavior, facial expressions, etc.) to understand the internal state of the participants sharing the space (current state), and provide information to those involved to facilitate moving the current state toward the goal (the desired state to be reached: consensus building, empathy, etc.).

[0182] Secondly, by obtaining information from a situation that has already been formed by environmental information and human information from participants (for example, when interaction is already occurring between participants), the facilitator can use that information as input for identification and perform identification without providing any information himself (or by adding a little), and then provide information to those involved to facilitate the process in the desired direction.

[0183] The reason why a facilitator intervenes is to change the characteristics that a person or the space that people make up at that time has, and to change them to different characteristics. To explain it more semantically, there is a simple process where a facilitator intervenes and the problem is solved on the spot, but if this is the extent of it, the situation will return to the original state when the facilitator leaves.

[0184] For this reason, advanced facilitators aim to change the characteristics of the people and the space that make up the space through multiple, overlapping interventions. In this way, even without feedback, the system itself can exist as a system that has the characteristics of a problem being solved, and advanced facilitation is realized (in reality, the model itself has changed).

[0185] This "system itself can exist as a system with problem-solving characteristics" means that this is the result of adjustments to the synaptic connections in the nervous system (gradual change due to plasticity), or that the memory of one or several keywords can instantly change the characteristics of the system itself (a mechanism that forces you to skip a problem internally).

[0186] According to the above configuration, the augmented space construction system 1 facilitates mutual empathy during interactions between participants in the information space, promoting active interaction, making it possible to share the internal environment of the information space, experience physical and psychological connections with others, and improve empathy for others.

[0187] When applying this kind of augmented space construction system 1 to counseling, the "active listening" required of a counselor is interpreted in terms of systems theory. In other words, in terms of systems theory, it can be explained by the identification process and state observation.

[0188] In this systems-theoretic interpretation, while providing moderate input that affects the participants' characteristics, the internal state is inferred by identifying the participants' characteristic parameters from information obtained from the participants and their environment, thereby understanding the participant's state.

[0189] Going a step further, good counseling can be said to have taken place when, by providing internal and external feedback to the participant, it is possible to modify the participant's characteristic parameters or change the dynamic equilibrium point, gradually shifting the participant to another stage and correcting them to a new, better state.

[0190] (4) Other embodiments (4-1) Improving the accuracy of exchange content through learning In Fig. 19, in which parts corresponding to those in Fig. 18 are given the same reference numerals, an interface device 100 for each participant has an evaluation learning unit 101 interposed between the effort information generation unit 14 and the activity reflection unit 15. The evaluation learning unit 101 constructs an effectiveness evaluation model that has been trained to evaluate the effectiveness of the effort information generated by the effort information generation unit 14, based on the integration result of the person and environment information of each participant.

[0191] The effort information generating unit 14 adaptively corrects the effort information so that the effort information always satisfies the evaluation criteria based on the effectiveness evaluation model constructed by the evaluation learning unit 101.

[0192] As a result, the augmented space construction system 1 makes it possible to maintain a relatively high quality of interactive information (such as expert knowledge from clinical psychologists) that influences the internal state of each participant during their activities in the information space.

[0193] (4-2) Improving the information space environment through learning In FIG. 20, in which the same reference numerals are used to denote parts corresponding to those in FIG. 18, an interface device 110 for each participant is further provided with a goodness evaluation learning unit 111 and an environmental state correction unit 112.

[0194] The goodness evaluation learning unit 111 constructs a goodness evaluation model that is trained to evaluate the goodness of the environmental state (atmosphere, background sound, brightness, saturation, color harmony, etc.) of the selected information space based on the integrated results of the human and environmental information of each participant.

[0195] The environmental state correction unit 112 adaptively corrects the environmental state of the information space so as to always satisfy an evaluation criterion based on the goodness evaluation model constructed by the goodness evaluation learning unit 111 .

[0196] As a result, in the augmented space construction system 1, it becomes possible to always maintain the environmental state of the information space, which affects the internal state of each participant, in an optimal state while the participant is active in the information space.

[0197] (4-3) How to integrate the internal states of participants In Figure 21, in which parts corresponding to those in Figure 18 are given the same symbols, an action information generation unit 121 in an interface device 120 for each participant has a correlation quantification unit 122 and a transition state evaluation unit 123, and is configured to adaptively adjust the action information.

[0198] The correlation quantification unit 122 quantifies the correlations between the internal states of the participants in a time series manner. The transition state evaluation unit 123 evaluates the transition state of the internal state of each participant based on the correlations quantified by the correlation quantification unit 122. The action information generation unit 121 adaptively adjusts the action information according to the evaluation result by the transition state evaluation unit 123.

[0199] As a result, the augmented space construction system 1 can grasp, as a whole system, the process by which each participant's internal state changes as it is influenced by the words, actions, and attitudes of other participants, making it possible to respond (interact) appropriately to each participant.

[0200] (4-4) Expansion of activity areas in information space In FIG. 22, in which parts corresponding to those in FIG. 18 are given the same reference numerals, an interface device 130 for each participant is provided with a spatial layer construction unit 131, a learning model training unit 132, and a layer management unit 133.

[0201] When each participant is engaged in a communication activity in an information space, the space layer construction unit 131 constructs a space layer for expanding the information space while the participants in the information space cooperate with each other.

[0202] As a result, in the expanded space construction system 1, it becomes possible for participants to expand the information space in cooperation with each other within the information space.

[0203] In addition, the learning model training unit 132 trains a learning model constructed by statistical and mathematical processing to map the collaboration content between participants to a spatial extension of the information space based on the spatial layer construction process constructed by the spatial layer construction unit 131.

[0204] Then, the spatial layer construction unit 131 automatically constructs a spatial layer based on the learning model trained by the learning model training unit 132 without the cooperation of each participant.

[0205] As a result, in the augmented space construction system 1, the process of expanding the information space is learned by learning, so that the expansion process can be continued automatically without the cooperation of each participant.

[0206] Furthermore, when the learning model is trained by the learning model training unit 132, the layer management unit 133 manages the spatial layers constructed in the information space to be expanded in order to maintain order.

[0207] As a result, in the extended space construction system 1, by carrying out management aimed at maintaining order when continuing the expansion process of the information space, it is possible to prevent each participant from suffering any disadvantage. [Explanation of symbols]

[0208] 1...Augmented space construction system, 2, 100, 110, 120, 130...Interface device, 3...Cloud, 10...Human and environment information input / output unit, 11...Virtual activity construction unit, 12...Adaptation state estimation unit, 13...Human relationship evaluation unit, 14, 121...Initiative information generation unit, 15...Activity reflection unit, 20...Vital sign measurement device, 21...Facial imaging camera, 22...Sound collection microphone, 23...Motion capture device, 24...Peripheral measurement device, 25...Information detection unit, 30...Electrode terminal group, 31...Photoelectric probe window, 32...Connection terminal, 40...Brain activity measurement unit, 41...Blood flow measurement unit, 42...Electrocardiograph, 43...First control unit, 44...Wireless communication device, 45...Net-like base, 46A to 46N...Sensor unit, 47...Light emitter, 48...Light receiver, 50...Biosignal measurement wearable device, 51...Drive unit, 52...Joint circumference detection unit, 5 3...biological signal sensor, 54A to 54F...biological signal sensor group, 55...measurement module, 56...measurement module controller, 57...memory, 58...communication unit, 59...second control unit, 60...near-infrared detection unit, 61...far-infrared detection unit, 62...third control unit, 70...control unit, 71...RGB-D sensor, 72...IMU sensor, 73...fourth control unit, 74...data storage unit, 80...image coordinate setting unit, 81...coordinate group extraction unit, 82...skeletal information acquisition unit, 83...behavior recognition unit, 90...identification model estimation unit, 91...difference estimation unit, 92...parameter identification mechanism unit, 101...evaluation learning unit, 111...goodness evaluation learning unit, 112...environmental state correction unit, 122...correlation quantification unit, 123...transition state evaluation unit, 131...spatial layer construction unit, 132...learning model training unit, 133...layer management unit.

Claims

1. a person / environment information input / output unit provided for each of a plurality of participants present in a real space, for acquiring person / environment information including brain / nervous system information, musculoskeletal system information, physiological information, psychological information, motion information, behavior information, and environmental information of each of the participants; a virtual activity construction unit that virtually constructs communication activities of each of the participants in the real space in an information space in real time based on the person and environment information corresponding to each of the participants output from the person and environment information input / output unit; an adaptive state estimation unit provided for each of the participants, the adaptive state estimation unit estimating an internal state of the corresponding participant based on the human and environmental information output from the human and environmental information input / output unit; a human relationship evaluation unit that evaluates the overall and interpersonal human relationships among the participants while integrating internal states of all the participants participating in the communication activity when each of the participants participates in the communication activity; an encouragement information generating unit that generates encouragement information representing positive behaviors and attitudes aimed at empathic understanding between the participants in the information space based on the evaluation results by the human relationship evaluation unit; an activity reflection unit that reflects the approach information generated by the approach information generation unit to all or each of the participants so as to intervene in the communication activities of the participants constructed in the information space; An augmented space construction system comprising:

2. The adaptive state estimation unit an identification model estimation unit that sets an identification model in which the internal state of each participant is modeled as a state variable, and estimates internal parameters, which are variable characteristics of the identification model, based on the person and environment information; a difference estimation unit that estimates a difference indicating how close the identification model related to the internal parameters estimated by the identification model estimation unit is to the human and environmental information output in real time from the human and environmental information input / output unit; a parameter identification mechanism unit that corrects internal parameters constituting the identification model so that the difference estimated by the difference estimation unit becomes small; The action information generating unit changes the content of the action information based on the internal parameters estimated by the identification model estimating unit so that the correction by the parameter identification mechanism unit is gradually reduced.

2. The augmented space construction system according to claim 1 .

3. an evaluation learning unit that constructs an effectiveness evaluation model that is trained to evaluate the effectiveness of the engagement information generated by the engagement information generation unit based on the integration result of the person and environment information of each participant; The action information generation unit adaptively corrects the action information so as to always satisfy an evaluation criterion based on the effectiveness evaluation model constructed by the evaluation learning unit.

3. The augmented space construction system according to claim 2.

4. a goodness evaluation learning unit that constructs a goodness evaluation model that is trained to evaluate the goodness of the environmental state of the selected virtual space based on the integrated result of the human and environmental information of each participant; an environmental state correction unit that adaptively corrects an environmental state of the information space so as to always satisfy an evaluation criterion based on the goodness evaluation model constructed by the goodness evaluation learning unit; 3. The extended space construction system according to claim 2, further comprising:

5. The approach information generation unit, a correlation quantification unit that quantifies the correlations of the internal states of the participants in a time series manner; a transition state evaluation unit that evaluates a transition state of an internal state of each of the participants based on the correlations quantified by the correlation quantification units; and adaptively adjusting the action information according to the evaluation result by the transition state evaluation unit.

3. The augmented space construction system according to claim 2.

6. a space layer construction unit that constructs a space layer for expanding the information space while the participants cooperate with each other in the information space when the participants engage in communication activities in the information space; 2. The augmented space construction system according to claim 1, further comprising:

7. A learning model training unit that trains a learning model constructed by statistical and mathematical processing for mapping the collaboration content between the participants to a spatial extension of the information space based on the construction process of the spatial layer constructed by the spatial layer construction unit. The spatial layer construction unit automatically constructs the spatial layer without the cooperation of each of the participants based on the learning model trained by the learning model training unit.

7. The extended space construction system according to claim 6.

8. A layer management unit that manages the spatial layer constructed in the information space to be expanded for the purpose of maintaining order when the learning model is trained by the learning model training unit.

8. The extended space construction system according to claim 7, further comprising:

9. a first step of acquiring human and environmental information including brain and nervous system information, musculoskeletal system information, physiological information, psychological information, motion information, behavioral information, and environmental information of each of the participants, the first step being provided for each of the participants in a real space; a second step of virtually constructing communication activities of each of the participants in the real space in an information space in real time based on the human and environmental information corresponding to each of the participants acquired in the first step; a third step of adaptively estimating, for each of the participants, an internal state of the corresponding participant based on the human and environmental information output from the human and environmental information input / output unit; a fourth step of evaluating the intra- and inter-personal relationships among the participants while integrating the internal states of all the participants participating in the communication activity during the participation of each of the participants in the communication activity; A fifth step of successively generating encouraging information representing positive behaviors and attitudes aimed at empathic understanding between the participants in the information space based on the evaluation results obtained in the fourth step; A sixth step of reflecting the approach information generated in the fifth step on all or each of the participants so as to intervene in the communication activities of the participants constructed in the information space; An extended space construction method comprising:

10. In the third step, An identification model is set in which the internal state of each participant is modeled as a state variable, and internal parameters, which are variable characteristics of the identification model, are estimated based on the person and environment information. A difference is estimated as to how close the identification model related to the estimated internal parameters has come to the person and environment information acquired in real time by the first step, and the internal parameters constituting the identification model are corrected so that the estimated difference becomes smaller. In the fifth step, The content of the encouraging information is changed so that the modification of the internal parameter is gradually reduced based on the internal parameter estimated based on the human and environment information.

10. The method for constructing an extended space according to claim 9.

11. A seventh step of constructing an effectiveness evaluation model that is trained to evaluate the effectiveness of the engagement information generated by the engagement information generation unit based on the integration result of the person and environment information of each participant, In the fifth step, the approach information is adaptively corrected so as to always satisfy an evaluation criterion based on the effectiveness evaluation model constructed in the seventh step. The method for constructing an extended space according to claim 10.

12. An eighth step of constructing a goodness evaluation model that is trained to evaluate the goodness of the environmental state of the selected virtual space based on the integrated result of the human and environmental information of each participant; a ninth step of adaptively correcting the environmental state of the information space so as to always satisfy an evaluation criterion based on the goodness evaluation model constructed in the eighth step; The method for constructing an extended space according to claim 10, further comprising:

13. In the fifth step, A transition state of the internal state of each participant is evaluated based on the correlation between the internal states of the participants, which is obtained by quantifying the correlation in a time series, and then the approach information is adaptively adjusted according to the result of the evaluation. The method for constructing an extended space according to claim 10.

14. A tenth step of constructing a space layer for expanding the information space while the participants cooperate with each other in the information space when the participants engage in communication activities in the information space. The method for constructing an extended space according to claim 9, further comprising:

15. An eleventh step of training a learning model constructed by statistical and mathematical processing for mapping the collaboration content between the participants to a spatial extension of the information space based on the construction process of the spatial layer constructed by the tenth step. In the tenth step, the spatial layer is automatically constructed without the cooperation of each participant based on the learning model trained in the eleventh step.

15. The method for constructing an extended space according to claim 14.

16. A 12th step of managing the spatial layer constructed in the information space to be expanded when the learning model is trained in the 11th step, for the purpose of maintaining order. The method for constructing an extended space according to claim 15, further comprising:

Images