Augmented space construction system and augmented space construction method

Abstract

According to the present invention, when participants are participating in a communication activity, the internal states of corresponding participants are adaptively estimated on the basis of human and environmental information corresponding to each participant, collective and individual human relationships between the participants are evaluated while the internal states of all the participants participating in the communication activity are being integrated, and, on the basis of the evaluation results, proactive information representing positive words, actions, and attitudes aimed at empathetic understanding between the participants in an information space is successively generated on the basis of the evaluation results and is reflected to the participants collectively or individually so as to intervene in the communication activity of each participant.

Description

Augmented space construction system and augmented space construction method

[0001] The present invention relates to an augmented space construction system and an augmented space construction method, and is suitable for application to an augmented space including a three-dimensional virtual space and a real space constructed on a network, for example.

[0002] In recent years, the term "metaverse" has come into widespread use to describe fully immersive, three-dimensional virtual spaces that include virtual environments in which people are represented by avatars (characters that represent the user's avatar). One example of such a metaverse is a virtual space experience system that allows users to perceive an image of a three-dimensional virtual space constructed on a network, while reflecting their activities in the virtual space, allowing the users to experience the sensation of being present in the virtual space.

[0003] Speech recognition and response generation systems for robot dialogue have also been developed, and for example, a dialogue robot that outputs voice responses to user utterances via a server has been proposed (see Patent Document 1). In this dialogue robot, the server analyzes the meaning of the user's utterance, and while referring to a response sentence database, selects a response sentence corresponding to the meaning and outputs it in the form of voice data.

[0004] Expanding on this, a conference support device has also been proposed that indirectly supports the progress of a conference by supporting a facilitator who manages the progress of the conference, or directly supports the progress of a conference by taking on the role of a facilitator (see Patent Document 2). This conference support device calculates activity levels that represent the degree of activity in the discussion at the conference based on the biometric information of the conference participants, and outputs activity level information that represents changes in activity levels based on the activity levels within a predetermined period after the facilitator recognizes the timing to intervene and encourage the participants to speak.

[0005] JP 2022-67709 A JP 2022-68904 A

[0006] However, when using the conversational robot of Patent Document 1 to realize the conference support system of Patent Document 2, there was a problem that even though multiple participants participating in the metaverse were engaged in communication activities, the mere intervention of a facilitator was insufficient in practice to deepen the relationships between the participants.

[0007] The present invention has been made in consideration of the above points, and aims to propose an augmented space construction system and an augmented space construction method that enable multiple participants in an information space to deepen their relationships with each other while engaging in communication activities.

[0008] In order to solve this problem, the present invention provides a person / environment information input / output unit provided for each of a plurality of participants present in a real space, which acquires person / environment information including brain / nervous system information, musculoskeletal system information, physiological information, psychological information, movement information, behavioral information, and environmental information of each participant; a virtual activity construction unit that virtually constructs communication activities of each participant in the real space in real time in an information space based on the person / environment information corresponding to each participant output from the person / environment information input / output unit; an adaptive state estimation unit provided for each participant, which adaptively estimates the internal state of the corresponding participant based on the person / environment information output from the person / environment information input / output unit; The information space communication system is provided with a human relations evaluation section that, when participating in a communication activity by a group of participants, evaluates the human relations among the participants as a whole and among individuals while integrating the internal states of all the participants participating in the communication activity; an action information generation section that successively generates action information representing positive speech, actions, and attitudes aimed at empathetic understanding among the participants in the information space based on the evaluation results by the human relations evaluation section; and an activity reflection section that reflects the action information generated by the action information generation section to all or each participant individually so as to intervene in the communication activities of each participant constructed in the information space.

[0009] As a result, in the augmented space construction system, multiple participants taking part in an information space can deepen their relationships with each other by sharing the details of their activities with each other while reflecting influence information to guide communication activities in a positive direction in the information space.

[0010] In addition, in the present invention, the adaptive state estimation unit is composed of an identification model estimation unit that sets up an identification model in which the internal state of each participant is modeled as a state variable, and estimates internal parameters that are variable characteristics of the identification model based on person / environment information; a difference estimation unit that estimates the difference in how close the identification model related to the internal parameters estimated by the identification model estimation unit has come to the person / environment information that is output in real time from the person / environment information input / output unit; and a parameter identification mechanism unit that corrects the internal parameters that constitute the identification model so that the difference estimated by the difference estimation unit becomes smaller, and the action information generation unit changes the content of the action information based on the internal parameters estimated by the identification model estimation unit so that the corrections by the parameter identification mechanism unit are gradually reduced.

[0011] As a result, the augmented space construction system facilitates mutual empathy during interactions between participants in the information space, promoting active interaction, making it possible to share the internal environment of the information space, experience physical and psychological connections with others, and improve empathy for others.

[0012] Furthermore, the present invention includes an evaluation learning unit that constructs an effectiveness evaluation model that is trained to evaluate the effectiveness of the encouragement information generated by the encouragement information generation unit based on the integrated results of the human and environmental information of each participant, and the encouragement information generation unit adaptively modifies the encouragement information so that it always meets the evaluation criteria based on the effectiveness evaluation model constructed by the evaluation learning unit.

[0013] As a result, the augmented space construction system makes it possible to maintain a relatively high level of influence information (such as expert knowledge from clinical psychologists) that influences the internal state of each participant during their activities in the information space.

[0014] Furthermore, the present invention is provided with a goodness evaluation learning unit that constructs a goodness evaluation model that is trained to evaluate the goodness of the environmental state of the selected virtual space based on the integrated results of the human and environmental information of each participant, and an environmental state correction unit that adaptively corrects the environmental state of the information space so that it always satisfies the evaluation criteria based on the goodness evaluation model constructed by the goodness evaluation learning unit.

[0015] As a result, the augmented space construction system can maintain the environmental state of the information space, which affects the internal state of each participant, in an optimal state at all times while the participant is active in the information space.

[0016] Furthermore, in the present invention, the action information generation unit includes a correlation quantification unit that chronologically quantifies the correlations between the internal states of the participants, and a transition state evaluation unit that evaluates the transition state of the internal state of each participant based on the correlations quantified by the correlation quantification unit, and adaptively adjusts the action information according to the evaluation results by the transition state evaluation unit.

[0017] As a result, the augmented space construction system can grasp the process by which each participant's internal state changes as it is influenced by the words, actions, and attitudes of other participants, making it possible to respond (interact) appropriately to each participant.

[0018] Furthermore, in the present invention, when each participant engages in communication activities in the information space, a spatial layer construction unit is provided that constructs a spatial layer to expand the information space in cooperation with the participants in the information space.

[0019] As a result, in the augmented space construction system, it becomes possible for participants to expand the information space in cooperation with each other within the information space.

[0020] Furthermore, the present invention is provided with a learning model training unit that trains a learning model constructed by statistical and mathematical processing to map the content of collaboration between participants to a spatial extension of the information space based on the spatial layer construction process constructed by the spatial layer construction unit, and the spatial layer construction unit automatically constructs a spatial layer based on the learning model trained by the learning model training unit, even without collaboration between each participant.

[0021] As a result, the augmented space construction system is able to learn the process of expanding the information space by learning, and can continue the expansion process automatically without the cooperation of each participant.

[0022] Furthermore, in the present invention, when the learning model is trained by the learning model training unit, a layer management unit is provided that manages the spatial layers constructed in the information space to be expanded for the purpose of maintaining order.

[0023] As a result, in the augmented space construction system, when continuing the expansion process of the information space, management aimed at maintaining order is carried out, making it possible to prevent each participant from suffering any disadvantages.

[0024] Furthermore, the present invention includes a first step of acquiring person-environment information, which is provided for each of a plurality of participants existing in the real space and includes brain-nervous system information, musculoskeletal system information, physiological information, psychological information, movement information, behavioral information, and environmental information for each participant; a second step of virtually constructing in real time in the information space the communication activities of each participant in the real space based on the person-environment information corresponding to each participant acquired in the first step; a third step of adaptively estimating the internal state of each participant based on the person-environment information output from the person-environment information input / output unit; a fourth step of evaluating, when each participant participates in the communication activity, the interpersonal relationships among the participants as a whole and between individuals while integrating the internal states of all the participants participating in the communication activity; a fifth step of successively generating, based on the evaluation results in the fourth step, encouragement information representing positive words, actions, and attitudes aimed at empathetic understanding among the participants in the information space; and a sixth step of reflecting the encouragement information generated in the fifth step on each participant, either together or individually, so as to intervene in the communication activities of each participant constructed in the information space.

[0025] As a result, in the augmented space construction system, multiple participants taking part in an information space can deepen their relationships with each other by sharing the details of their activities with each other while reflecting influence information to guide communication activities in a positive direction in the information space.

[0026] According to the present invention, it is possible to realize an augmented space construction system and an augmented space construction method that enable multiple participants in an information space to deepen their relationships with each other while engaging in communication activities.

[0027] 1 is a conceptual diagram illustrating an augmented space creation system according to the present embodiment. FIG. 1 is a conceptual diagram illustrating the internal configuration of the interface device shown in FIG. 1. FIG. 2 is a perspective view and six-view diagram illustrating the external configuration of the vital sign measuring device. FIG. 3 is a conceptual diagram illustrating the configuration of a brain activity measuring unit. FIG. 4 is a conceptual diagram illustrating the configuration of a biosignal measurement garment. FIG. 5 is a conceptual diagram illustrating the wearing state of the biosignal measurement garment. FIG. 6 is a conceptual diagram illustrating an additional configuration of the vital sign measuring device. FIG. 7 is a graph illustrating pulse waveforms before and after filter application. FIG. 8 is a conceptual diagram illustrating an ROI of a near-infrared image and an ROI of a far-infrared image. FIG. 9 is a conceptual diagram illustrating nasal breathing and mouth breathing. FIG. 10 is a conceptual diagram illustrating the temperature state of nasal breathing and mouth breathing. FIG. 11 is a graph illustrating the change in nostril temperature due to nasal breathing and the change in oral temperature due to mouth breathing. FIG. 12 is a graph superimposing the change in breathing technique from nasal breathing to mouth breathing and the details of each state. FIG. 13 is a conceptual diagram illustrating non-contact measurement of oxygen saturation using light. FIG. 14 is a block diagram illustrating the internal configuration of a motion capture device. FIG. 15 is a conceptual diagram illustrating a method for acquiring three-dimensional skeletal information related to a participant's entire body. FIG. 16 is a conceptual diagram illustrating the structure of a behavior recognition model. Fig. 1 is a conceptual diagram mainly showing the internal configuration of an interface device according to another embodiment. Fig. 2 is a conceptual diagram mainly showing the internal configuration of an interface device according to another embodiment. Fig. 3 is a conceptual diagram mainly showing the internal configuration of an interface device according to another embodiment. Fig. 4 is a conceptual diagram mainly showing the internal configuration of an interface device according to another embodiment. Fig. 5 is a conceptual diagram mainly showing the internal configuration of an interface device according to another embodiment.

[0028] An embodiment of the present invention will be described in detail below with reference to the drawings.

[0029] (1) Configuration of the Augmented Space Construction System According to the Present Embodiment Fig. 1 shows an augmented space construction system 1 according to the present embodiment. In the augmented space construction system 1, an interface device 2 is provided for each participant, which acquires various information obtained from the participant himself or herself and the surrounding environment in the real space, and the various information transmitted from each interface device 2 is reflected in an information space constructed in the cloud 3.

[0030] As shown in FIG. 2, each interface device 2 includes a person / environment information input / output unit 10, a virtual activity construction unit 11, an adaptive state estimation unit 12, a human relationship evaluation unit 13, an approach information generation unit 14, and an activity reflection unit 15.

[0031] The human and environmental information input / output unit 10 has vital sign measuring devices (measuring means for electrocardiogram, pulse rate, pulse pressure, oxygen saturation, body temperature, brain waves, blood sugar level, blood pressure, and electromyography, etc.) 20, a facial imaging camera 21, a sound collection microphone 22, motion capture devices (optical, inertial sensor, mechanical, and magnetic types, or a combination of these) 23, and peripheral measuring devices (thermometer, hygrometer, audible and ultrasonic ambient sound collectors, etc.) 24, and an information detection unit 25 that detects human and environmental information, which will be described later, based on the output of these sensors and devices, etc.

[0032] In this human / environment information input / output unit 10, the information detection unit 25 detects the subject's brain / nervous system information, musculoskeletal system information, and physiological information based on the detection results from the vital sign measuring device 20, and also detects psychological information by analyzing the subject's facial expressions and voice using the facial imaging camera 21 and sound collecting microphone 22, and further detects the subject's movement information and behavior information using the motion capture device 23, and detects surrounding environmental information centered on the subject using the peripheral measuring device 24.

[0033] The virtual activity construction unit 11 virtually constructs the human activities of the subject in real space in real time in the information space based on the subject's human and environmental information output from the human and environmental information input / output unit 10 .

[0034] Here, information space refers to a virtual space generated by a computer, typified by a three-dimensional virtual space constructed on a network (cloud 3) and the metaverse, which is a service for such a virtual space. The functionality required for this information space requires two-way communication with the real world via a network. When using an information space as a versatile platform, it is desirable for the system that provides the information space to be open and to include user management and access restriction functions. Examples of information spaces that satisfy these functions include Second Life (registered trademark) provided by Linden Lab, Inc., and OpenSim, which is provided under the BSD License (Berkeley Software Distribution License).

[0035] The adaptive state estimation unit 12 adaptively estimates the internal state of each participant based on the person and environment information output from the person and environment information input / output unit 10. This internal state is information that contributes to empathy between participants sharing a space, and is a precursor state to the production of statements, facial expressions, behaviors, etc. that will lead to the promotion of empathy.

[0036] When each participant joins a communication activity, the human relationship evaluation unit 13 integrates the internal states of all participants participating in the communication activity and evaluates the overall and interpersonal human relationships among the participants.

[0037] The encouragement information generation unit 14 successively generates encouragement information representing positive words, actions, and attitudes aimed at fostering empathetic understanding among participants in the information space, based on the evaluation results by the human relationship evaluation unit 13. The activity reflection unit 15 reflects the encouragement information generated by the encouragement information generation unit 14 on all or individual participants, so as to intervene in the communication activities of each participant established in the information space.

[0038] As a result, in the augmented space construction system 1, multiple participants participating in the information space can deepen their relationships with each other by sharing the content of their activities with each other while reflecting influence information to guide communication activities in a positive direction in the information space.

[0039] (2) Detailed Configuration and Function of the Person / Environment Information Input / Output Unit (2-1) Configuration of the Vital Sign Measuring Device Specifically, as shown in FIG. 3(A), the vital sign measuring device 20 has a device body 20A on a measurement surface 20X on which an electrode terminal group 30 and a photoelectric probe window 31 are arranged in a predetermined pattern, and measures the subject's biological information via the measurement surface 20X. The biological information measured by the vital sign measuring device 20 is at least one of the measurement results of electrocardiogram, pulse rate, pulse pressure, oxygen saturation, body temperature, electroencephalogram, blood glucose level, blood pressure, and electromyogram. In other words, the vital sign measuring device 20 is configured so that the same device body can be used as a means for measuring all types of biological information.

[0040] 3B, the vital sign measuring device 20 has a sound collection microphone 22 detachably connected to a connection terminal 32 provided on the side of the device body 20A, and when connected, can collect vibrations generated from the subject's larynx and pharynx as acoustic signals. As a result, it is possible to obtain information about the subject's vibrations (inaudible range) associated with the heart and breathing that cannot be heard by the human ear.

[0041] (2-2) Method for detecting brain and nervous system information This human and environment information input / output unit 10 has a brain activity measurement unit 40 that is worn on the subject's head and measures brain waves and cerebral blood flow in a measurement area of ​​the head, as an additional component of the vital sign measuring device 20. The information detection unit 25 then converts the transmission state of the central nervous system centered on the head into quantitative nervous system data based on the measurement data of brain waves and cerebral blood flow obtained from the brain activity measurement unit 40.

[0042] The brain activity measuring section 40 according to this embodiment is in line with the content described in Japanese Patent Application Laid-Open No. 5283700 by the inventor of the present application, and uses the same principle.

[0043] 4A, the brain activity measuring unit 40 has a blood flow measuring unit 41, an electrocardiograph 42, a first control unit 43, and a wireless communication device 44. The blood flow measuring unit 41 measures the state of blood flow from changes in the optical path and transmitted light amount of laser light caused by blood flow, and further measures the state of brain activity from the state of blood flow in the brain.

[0044] That is, when the laser light irradiated onto the blood enters the blood layer, it passes through the blood as both light components: a reflected and scattered light component by normal red blood cells and a reflected and scattered light component by adherent thrombi. The influence of the laser light on the process of passing through the blood layer changes from moment to moment depending on the state of the blood. Therefore, by continuously measuring the amount of transmitted light (amount of reflected light) and observing the change in the amount of light, it is possible to observe various changes in the state of the blood.

[0045] The blood flow measurement unit 41 comprises a net-like base 45 formed in a hemispherical shape corresponding to the external shape of the head so as to be worn on the head, and a number of sensor units 46A to 46N. Each of the sensor units 46A to 46N is supported at predetermined intervals by the net-like base 45, and outputs a detection signal of the amount of transmitted light measured at each measurement point on the head to a first control unit 43. The electrocardiograph 42 measures cardiac potentials generated in response to cardiac movement using electrodes E attached to the subject's skin.

[0046] The first control unit 43 derives the displacement of blood vessels and tissues surrounding the blood vessels due to blood flow and measures the state of brain activity (distribution of red blood cells) based on the light intensity when light (FIG. 4B) emitted from the light-emitting unit 47 of each sensor unit 46A to 46N is received by the light-receiving unit 48. The first control unit 43 also stores a control program that performs arithmetic processing to cancel components due to oxygen saturation included in signals obtained from at least two or more light-receiving units 48.

[0047] The first control unit 43 further derives the displacement of the inner wall of the blood vessel based on the displacement of the blood vessel and the tissue surrounding the blood vessel. The first control unit 43 further calculates the pulse wave velocity at each measurement position from the phase difference between the waveform of the electrocardiogram signal from the electrocardiograph 42 and the waveform of the detection signal obtained from the light receiving unit, and derives the displacement state of the inner wall of the blood vessel from the pulse wave velocity.

[0048] The wireless communication device 44 wirelessly transmits the measurement results (blood flow data) output from the first control unit 43 to the vital sign measuring device 20. The brain activity measuring unit 40 has optical sensor units 46A to 46N arranged on a net-like base 45, and therefore can simultaneously measure blood flow throughout the entire head.

[0049] 4(B), when measuring cerebral blood vessel characteristics, the first control unit 43 selects an arbitrary sensor unit 46D from the array of multiple sensor units 46A to 46N and causes the light-emitting unit 47 of the selected sensor unit 46D to emit laser light. At this time, the laser light emitted from the light-emitting unit 47 has a wavelength λ (λ≈805 nm) that is not affected by oxygen saturation.

[0050] When the blood flow measurement unit 41 of the brain activity measurement unit 40 is attached to the subject's head, the elasticity of the net-like base 45 positions the multiple sensor units 46A to 46N at each measurement point on the head and holds each measurement surface facing the head surface. With the blood flow measurement unit 41 attached to the head, the multiple sensor units 46A to 46N can measure the vascular characteristics of the middle cerebral artery and anterior cerebral artery (the rate of vascular elasticity, the amount of plaque in the blood vessels, and the rate of arteriosclerosis) by measuring the pulse wave velocity in the cerebral arteries from changes in the amount of light received after irradiating the surface of the brain with light and propagating within the brain.

[0051] The principle of measuring the pulse wave velocity of cerebral arteries will now be described. For example, the pulse wave velocity due to blood flowing through the middle cerebral artery and the anterior cerebral artery is detected by each of the sensor units 46A to 46N, which are arranged corresponding to the measurement positions and receive light that has propagated through the brain when light is irradiated. The measurement method involves comparing the waveform of the cardiac potential obtained from the electrocardiograph 42 with the waveform of the signal output from each of the sensor units 46A to 46N at the measurement positions, determining the pulse wave velocity from the phase difference, and deriving the vascular characteristics corresponding to the pulse wave velocity.

[0052] Next, the principle of detecting vascular characteristics from cerebral blood flow will be described. As shown in Figure 4(B), the brain BR is covered by cerebrospinal fluid BR1, skull BR2, and scalp BR3. A portion of the laser light emitted from the light-emitting unit 47 of each sensor unit 46A to 46N of the blood flow measurement unit 41 is reflected by the scalp BR3, while the remaining light passes through the scalp BR3, skull BR2, and cerebrospinal fluid BR1 and travels into the brain BR. The light irradiated onto the head and traveling into the brain propagates in the radial direction (depth direction and radial direction) along an arc-shaped pattern CP as indicated by the dashed lines in the figure.

[0053] The propagation of light through the brain becomes longer the farther radially from the base point BP where the laser light is irradiated, and the lower the light transmittance. Therefore, the light reception level (amount of transmitted light) of sensor unit 46B, which is adjacent to sensor unit 46A on the light-emitting side at a predetermined distance, is detected as being strong. The light reception level (amount of transmitted light) of sensor unit 46C, which is located a predetermined distance next to sensor unit 46B, is detected as being weaker than the light reception level of sensor unit 46B. The light-receiving unit 48 of sensor unit 46A on the light-emitting side also receives light from the brain.

[0054] The first control unit 43 uses detection signals corresponding to the light intensities received by the plurality of sensor units 46A to 46N as measurement data, and derives the vascular characteristics of each measurement position by comparing the waveforms of the detection signals with the waveform of the cardiac potential signal from the electrocardiograph 42. Furthermore, by performing mapping processing on these detection results, it is possible to obtain graphic data showing the distribution of arteriosclerosis according to the pulse wave velocity.

[0055] Therefore, it is possible to measure changes in blood flow through the middle cerebral artery and anterior cerebral artery based on the detection signal waveforms of the sensor units 46A to 46N, and to detect the pulse wave velocity in the brain from the measurement data of the changes in blood flow.

[0056] As described above, the vital sign measuring device 20 is equipped with a brain activity measuring unit 40 that is attached to the subject's head and measures brain waves and cerebral blood flow in the measurement area of ​​the head, and the information detecting unit 25 also adds as nervous system data the transmission state of the central nervous system centered on the head based on the measurement data of brain waves and cerebral blood flow obtained from the brain activity measuring unit 40. In this way, the transmission state of the central nervous system from the subject's brain can also be quantitatively converted into data and acquired, making it possible to monitor the subject's progress and transition state.

[0057] As a result, the state of transmission from the subject's brain to the central nervous system can be quantitatively converted into data and acquired, making it possible to monitor the subject's progress and transitional state.

[0058] (2-3) Method for detecting musculoskeletal system information The human and environment information input / output unit 10 has a biological signal measurement garment 50 worn by the subject and a joint circumference detection unit (not shown) as additional components of the vital sign measuring device 20. The joint circumference detection unit has an actuator that is driven actively or passively in conjunction with the limb movement of the subject, and detects physical quantities around the joints that accompany the limb movement of the subject based on the output signal from the actuator.

[0059] The physical quantities around the joints include the muscle forces generated in the musculoskeletal system, the range of motion of each joint, the speed of motion and reaction speed, the autonomous control characteristics against disturbances, the parameter identification results of at least one of the frame's moment of inertia, mass and center of gravity, the impedance adjustment results of the joint system including flexor and extensor muscles (viscous characteristics due to friction), electrical physical quantities (command signals), etc.

[0060] The biosignal measurement wearing device 50 worn by the subject is consistent with and uses the same principles as the device described in Japanese Patent Application Publication No. 5409637 filed by the present inventor. Fig. 5(A) shows the biosignal measurement wearing device 50 of this embodiment. This biosignal measurement wearing device 50 is formed to cover the subject's body surface and has a wearing device main body 50A to be worn by the subject. The inner surface of this wearing device main body 50A (the surface that comes into contact with the subject's body surface when worn) has a biosignal sensor 53 for detecting biopotential signals at at least one position where a biosignal can be measured from the subject's body.

[0061] Biosignals are signals generated by electricity within a subject's body. They are measurable signals that change over time in response to bodily movements and signals from the brain. For example, biosignals refer to signals generated by biological activity, such as nerve potentials, muscle potentials, electroencephalograms, cardiac potentials, and potentials generated by motion artifacts (movement effects), potentials generated by biochemical reactions, and vibrations such as pulse waves generated by heartbeats. The potentials (amplitude, density, time delay) and locations of biosignals are also taken into account when quantifying the data.

[0062] In Figure 5(A), the biosignal measurement garment 50 of this embodiment has a bilaterally symmetrical structure, so the left half of the figure shows the structure of the back of the left leg, and the right half of the figure shows the structure of the front of the left leg. The garment main body 50A is provided with a group of biosignal sensors 54A to 54F consisting of multiple biosignal sensors 53 that detect the subject's biosignals. The biosignal sensors 53 are arranged at equal intervals along the flow of the subject's leg muscles.

[0063] The wearing device main body 50A has multiple biosignal sensors 53 arranged along the flow of the gluteus maximus in the area corresponding to the subject's buttocks (biosignal sensor group 54A). Similarly, the biosignal sensors 53 are arranged along the flow of the biceps femoris, semimembranosus, and semitendinosus in the area corresponding to the back of the subject's thighs (biosignal sensor group 54B). The biosignal sensors 53 are arranged along the flow of the triceps surae in the area corresponding to the calves (biosignal sensor group 54C). The biosignal sensors 53 are arranged along the flow of the adductor longus and iliopsoas in the area corresponding to the anterior hip joints (biosignal sensor group 54D). The biosignal sensors 53 are arranged along the flow of the quadriceps femoris in the area corresponding to the anterior thighs (biosignal sensor group 54E). The biosignal sensors 53 are arranged along the flow of the tibialis anterior, soleus, and extensor digitorum longus in the area corresponding to the shins (biosignal sensor group 54F).

[0064] 5(B) is a diagram showing an example of a measurement module connected to a group of biosignal sensors provided on the wearing device main body 50A. Note that all of the biosignal sensor groups 54A to 54F provided on the wearing device main body 50A are connected to the measurement module 55 in the same way, so the description here will be made using the biosignal sensor group 54A as an example.

[0065] 5(B) is a diagram schematically illustrating a biosignal sensor group 54A provided on the wearing device main body 50A and a measurement module 55 connected to the biosignal sensor group 54A. The biosignal sensors 53 constituting the biosignal sensor group 54A are provided insulated from one another and connected to the measurement module 55 via conductive wiring. Each of the biosignal sensors 53 is assigned an address.

[0066] The measurement module 55 is connected to the plurality of biosignal sensors 53 that make up the biosignal sensor group 54A, and includes a measurement module controller 56 that selects at least two of the biosignal sensors 53 and acquires biosignals by calculating the difference between the detection signals detected by the selected biosignal sensors 53, a memory 57 that records the acquired biosignals, and a communication unit 58 that transmits to the outside the sequentially acquired biosignals and / or the biosignals recorded in the memory 57. Note that a measurement module 55 is provided for each of the biosignal sensor groups 54A to 54F, and each biosignal sensor 53 is connected to the measurement module 55.

[0067] This measurement module controller 56 has an electronic circuit that can sequentially select at least two biosignal sensors 53 from the multiple connected biosignal sensors 53 in response to a command signal input via the communication unit 58, and acquire the biosignals between these two selected biosignal sensors 53.

[0068] The measurement module controller 56 further includes signal processing means such as a filter that removes or extracts predetermined frequency components from the acquired biological signal, and an amplifier that amplifies the acquired biological signal. The acquired biological signal is output from the measurement module controller 56 to the communication unit 58 and / or the memory 57. When selecting a biological signal sensor 53, the measurement module controller 56 may operate each biological signal sensor 53 sequentially in a predetermined order, or may select the biological signal sensor 53 at the address specified by a designation signal input via the communication unit 58.

[0069] The communication unit 58 includes a thin antenna and a communication circuit connected to the antenna, and transmits, via the antenna to the second control unit 59, measurement information including a signal containing the biosignal output from the measurement module controller 56 and information such as an address indicating the positional information of the biosignal sensor 53 that detected the biosignal, and / or a signal containing the biosignal read from the memory 57 and information such as an address indicating the positional information of the biosignal sensor 53 that detected the biosignal.

[0070] This second control unit 59 generates a designation signal that specifies which biosignal sensor 53 to select from the group of biosignal sensors 54A to 54F provided on the attachment main body 50A, as well as signals such as the start and end of data acquisition, and selects the designated biosignal sensor 53 to measure the biosignal in accordance with the signal.

[0071] As described above, according to the biosignal measurement garment 50 of the present invention, the garment main body 50A is provided with biosignal sensor groups 54A-54F each consisting of a plurality of biosignal sensors 53, so that simply by wearing the garment main body 50A, the plurality of biosignal sensors 53 can be placed at predetermined locations at once, and can be brought into close contact with the skin surface to detect biosignals at each point. In this way, even when monitoring biosignals at multiple points on the subject's body surface, the effort of attaching and removing the biosignal sensors 53 one by one can be eliminated, and biosignals can be easily measured.

[0072] Furthermore, by detecting biosignals using the plurality of biosignal sensors 53 in this manner, biosignals can be measured at a plurality of points within the area where the biosignal sensors 54A to 54F are arranged. Then, by mapping the measurement data at each point according to the address assigned to each biosignal sensor 53, it is possible to measure the distribution of biosignals in the subject's body.

[0073] The second control unit 59 acquires, from the biological signals acquired by the measurement module 55, myoelectric potential signals associated with the subject's muscle activity and nerve transmission signals for operating the subject's musculoskeletal system.

[0074] The information detection unit 25 converts the activity state of the musculoskeletal system centered on the joint and / or the transmission state of the peripheral nervous system into quantitative musculoskeletal system data and / or nervous system data based on the physical quantities around the joint described above and either or both of the myoelectric potential signal and the nerve transmission signal obtained from the second control unit 59, and also converts the transmission state of either or both of the central nervous system and the peripheral nervous system centered on the desired location into quantitative nervous system data.

[0075] As a result, by having the subject perform limb movements, the activity state of the musculoskeletal system and the transmission state of the peripheral nervous system can be quantitatively converted into data based on the physical quantities around the joints and the biological signals obtained from the surface parts of the body relative to the joints, and the subject's progress and transition state can be monitored.

[0076] In addition, biological signals can be obtained from areas on the body surface other than those based on the subject's joints, and quantitative data can be obtained not only on the peripheral nervous system around the joints, but also on the transmission state of the central nervous system and peripheral nervous system from desired areas, making it possible to monitor the subject's progress and transitional state.

[0077] For example, as shown in Figure 6, by disposing biosignal sensors 54A-54F all over the subject's body, it is possible to visualize the state transitions of the information transmission pathways in the cranial nervous system. In Figure 6, by dividing the subject's entire body into predetermined regions (e.g., regions innervated by sensory nerves such as dermatomes), it is possible to grasp the neural connections of the entire body from a bird's-eye view and for each region. Therefore, if it is possible to detect in which region a problem has occurred in neural transmission, it is possible to grasp not only the location of the problem but also the degree of impact on either the central or peripheral side.

[0078] (2-4) Method for detecting the activity state of the physiological system In the human / environment information input / output unit 10, as shown in FIG. 7, a near-infrared detection unit 60 and a far-infrared detection unit 61 are provided as additional components of the vital sign measuring device 20, and the activity state of the physiological system and nervous system of the subject is detected using a non-contact method.

[0079] That is, the human / environment information input / output unit 10 uses the near-infrared detection unit 60 and the far-infrared detection unit 61 to irradiate infrared rays mainly onto the face of the subject, thereby simultaneously and non-invasively measuring fluctuations in the subject's pulse rate and skin temperature in daily life.

[0080] The near-infrared detector 60 irradiates a region of interest on the face of the subject, including the cheeks, with near-infrared light and receives light reflected from the face to generate a near-infrared image. The far-infrared detector 61 irradiates a region of interest on the face of the subject with far-infrared light and detects the skin temperature of the face.

[0081] The third control unit 62 estimates the pulse rate of the subject based on the period of change in the intensity waveform in the region of interest of the reflected light received by the near-infrared detection unit 60. The third control unit 62 extracts areas of the subject's face that are susceptible to and less susceptible to the activity of the autonomic nervous system from the near-infrared image generated by the near-infrared detection unit 60, and calculates the temperature difference in the skin temperature of each area from the detection result of the far-infrared detection unit 61.

[0082] A. Pulse measurement method based on infrared light The light-absorbing substances in biological tissue are water and hemoglobin in the blood. Water has strong absorption properties for infrared light with wavelengths longer than 1350 nm, while hemoglobin has strong absorption properties for visible light with wavelengths shorter than 650 nm. Non-invasive biomedical diagnosis using light often involves irradiating the body with light in the infrared to near-infrared range with wavelengths between 650 nm and 1350 nm, which has high biological penetration, and measuring the reflected and transmitted light, which contains biological information.

[0083] Pulse is a physiological phenomenon in which the volume of blood vessels changes due to the work of blood pressure caused by cardiac output. Changes in blood vessel volume are brought about by changes in blood flow. Changes in blood flow change the amount of light absorbed by hemoglobin in the blood, so the pulse rate can be estimated from the periodic changes in transmitted and reflected light when light is irradiated onto a living body.

[0084] In this invention, the subject is not required to wear a measuring device and measurements can be performed day or night. Therefore, the pulse rate is estimated by measuring the periodic change in reflectance of near-infrared light irradiated onto the living body using a near-infrared camera. Since near-infrared light is invisible, measurements can be made without burdening the subject, even at night. In order to estimate stress state from the pulse rate, it is necessary to measure the pulse rate, which fluctuates minutely due to activity of the autonomic nervous system.

[0085] The third control unit 62 implements an algorithm for extracting the waveform period required for calculating the pulse rate from the waveform data obtained from the near-infrared detection unit 60 .

[0086] The near-infrared detector 60 measures the intensity of the light reflected by biological tissue when the irradiated near-infrared light is reflected. This near-infrared detector 60 measures the subject at rest, but slight body movements and facial movements due to breathing are superimposed on the pulse waveform as motion artifacts that represent changes in the amount of reflected light.

[0087] To remove these motion artifacts, a digital filter is first applied to the pulse rate frequency band. The resting pulse rate range is set to 35 to 180 bpm, and motion artifacts may also be superimposed on the measured waveform in the 0.5 to 3.0 Hz band, but the amplitude of the waveform superimposed on the pulse waveform is significantly large.

[0088] Therefore, if the variance value of the waveform data within a certain period of time exceeds a certain threshold, the third control unit 62 recognizes that the subject is moving and removes the data during that period as unreliable. The waveform after the bandpass filter has a bimodal feature due to arterial blood flow. To facilitate feature point extraction, a moving average filter is applied to reduce the two-peak waveform to a single peak.

[0089] 8A and 8B show the waveforms before and after applying a band-pass filter and a moving average filter to the measured waveform, respectively. It can be seen that the motion artifacts superimposed on the pulse waveform have been removed by filtering.

[0090] Next, there are two main known methods for calculating the pulse rate from the light intensity waveform measured by a near-infrared camera.

[0091] In the first method, the cheek area of ​​the face was set as the region of interest (ROI), and the intensity data of the reflected light from the cheek area for 30 seconds obtained by measurement with an RGB camera was used to obtain a frequency spectrum using a spectrum analyzer. The frequency that is thought to be the pulse wave component was then compared with the frequency determined from the pulse rate indicated by a pulse oximeter. However, the noise caused by body movement is significantly large in the pulse waveform. When attempting to calculate the pulse rate using this type of frequency analysis, it is necessary to significantly restrict the movement of the person being measured.

[0092] On the other hand, the second method calculates the time required for one pulse beat from the period of the waveform feature points detected from the time-series data of the measured waveform, and then calculates the pulse rate by dividing that time by 60. Compared to methods that calculate the pulse rate from frequency analysis, which requires continuous measurement data for a certain period of time, methods that calculate the pulse rate from the period of the measured feature points can remove motion artifacts by removing feature point periods that are falsely detected due to noise caused by body movement. In addition, the minimum measurement time required is shorter than that required by frequency analysis, and the pulse rate can be calculated from several beats, making it possible to calculate the pulse rate over shorter intervals.

[0093] However, since the resting pulse rate of a healthy person is 60 to 80 bpm, the period of the characteristic points obtained in a 5-second measurement is 5 to 7 times, which is a problem in that the amount of data obtained is insufficient to calculate a more accurate pulse rate. Also, as shown in Figure 8(A), the waveform of the intensity change has two peaks synchronized with the arterial blood flow, which can cause erroneous detection of the period of the characteristic points, resulting in an error in the calculated pulse rate.

[0094] For daily measurements, the body part to be measured needs to be exposed day and night. Therefore, in the vital sign measuring device 20 according to the present invention, the ROI is set on the cheek of a person. The location of the set ROI is shown in Figure 9(A). The average of the intensity values ​​measured at the pixels within the set ROI was taken as the measurement value, and the time change of the measurement value was recorded as pulse waveform data. The period of the characteristic points of the waveform data was determined by the interval t between the top ends of the waveform. tp Not only that, but also the bottom interval t bp is added as another feature point, the number of feature points that can be used as reference data for calculating the pulse rate is increased.

[0095] Next, in order to extract the characteristic point period necessary for calculating the pulse rate, the detected t tp , t bp Among them, there are falsely detected t tp , t bp The algorithm for removing S is shown below in (a) to (f). ０ Let's say.

[0096] (a) ttp , t bp For each of the peaks, a certain peak interval t i is the previous peak interval t i-1 If there is a fluctuation of 30% or more, the peak interval t i Set S ０ The set of selected peak intervals S １ is expressed by the following equation (1).

[0097] (b) Set S ０ Standard deviation σ ０ Calculate 2σ ０ The peak interval t varies in the range i Set S １ The set of selected peak intervals S ２ is expressed by the following equation (2).

[0098] (c) Set S ２ Standard deviation σ ２ Calculate and σ ２ The peak interval t varies in the range i Set S ２ The set of selected peak intervals S ３ is expressed by the following equation (3).

[0099] (d) Set S ３ Standard deviation σ ３ Calculate and σ ３ The peak interval t varies in the range i S ３ The set of selected peak intervals S ４ is expressed by the following equation (4).

[0100] (e) Set S ４ The remaining peak interval blood pressure is converted to pulse rate, and a histogram is created with pulse rate intervals of 5 bpm. In this case, the pulse rate width for each section of the histogram is the same, but the width of the peak intervals corresponding to that pulse rate varies. Therefore, a weighted histogram is created using the following equation (5).

[0101] Here, t 30-35 is the time interval between peaks of pulse rate 30 [bpm] and 35 [bpm], t i-i+5 is the time interval between peaks of pulse rate i [bpm] to i+5 [bpm], h i-i+5 is the frequency of the peak interval between pulse rates i [bpm] and i+5 [bpm] before correction, and h' i-i+5 represents the correction value of the frequency of the peak intervals from pulse rate i [bpm] to i+5 [bpm].

[0102] (f) Regarding the values ​​of the weighted histogram, a window of width 20 [bpm] is set so that the sum of the frequencies of the four adjacent sections is maximized, and the peak interval t outside that range is i Remove.

[0103] According to the above algorithm, the average value of the extracted peak intervals is set as the peak width t within the measurement time, and the pulse rate is calculated by dividing 60 seconds by the peak interval t.

[0104] In this way, the third control unit 62 sets feature points at the upper and lower ends of the intensity waveform of the reflected light obtained from the near-infrared detection unit 60, and estimates the pulse rate based on the peak intervals between the upper and lower ends, which are the periods of the feature points. The third control unit 62 also calculates the standard deviation of the multiple peak intervals and removes peak intervals that do not represent a pulse period based on the standard deviation. The third control unit 62 then creates a weighted histogram for a predetermined time unit from the pulse rate converted based on the remaining peak intervals after the removal, and corrects the frequency of the peak intervals based on the histogram.

[0105] Thus, the third control unit 62 can calculate an accurate pulse rate from the measured waveform of pulse waveform data for a relatively short period based on the pixel group within the set ROI.

[0106] B. Skin temperature measurement using far-infrared detector Heat generated inside the body is carried to the body surface by conduction and convection. However, since biological tissue itself has poor thermal conductivity and acts as a heat insulator, most of the heat transport to the skin is thought to be due to skin blood flow. Skin blood flow changes depending on the activity of the autonomic nervous system, which is centered on the contraction and expansion of blood vessels by the sympathetic and parasympathetic nervous systems.

[0107] Previous studies have attempted to estimate the activity of the autonomic nervous system by measuring changes in skin temperature. However, because the heat distribution on the face is significantly affected by the subject's hairstyle and whether or not they are wearing glasses, the human and environment information input / output unit of the present invention uses near-infrared images to set the measurement site appropriate for estimating the activity of the autonomic nervous system.

[0108] In order to estimate stress levels from changes in skin temperature, it is necessary to set the ROI for measurement to a region of the face where temperature changes due to stress are most noticeable. Among the facial regions, the nose in particular is home to a high concentration of arteriovenous anastomoses (AVAs), which are susceptible to the activity of the autonomic nervous system, making it suitable for measuring skin temperature to estimate stress levels.

[0109] Furthermore, because skin temperature is affected by the ambient temperature, changes in skin temperature must be recorded as relative temperatures. In this case, the ROI used to measure the nasal skin temperature change as a reference must be set in a region that is less susceptible to the activity of the autonomic nervous system. The forehead is an example of a region of the face where AVAs are less dense and less susceptible to the activity of the autonomic nervous system.

[0110] Based on the above, the third control unit 62 according to the present invention records the change in skin temperature as the temperature difference between the region of interest ROI_n in the nose and the region of interest ROI_fh in the forehead. The locations of the ROIs are shown in FIG. 9B. The equation for calculating the temperature change is shown in Equation (6).

[0111] Here, T n is the average skin temperature measured at all pixels in ROI_n, T fh is the average skin temperature measured at all pixels in ROI_fh, T r is the relative temperature of the forehead and nose. When setting each ROI, the coordinates of the nose and forehead in the near-infrared image are converted to coordinates on the far-infrared image.

[0112] Specifically, when converting coordinates from a near-infrared (NIR) image to a far-infrared (FIR) image, the resolution and angle of view differ between near-infrared and far-infrared cameras, so the coordinate information of the near-infrared image is made to correspond to the pixels of the far-infrared image.

[0113] That is, the x-coordinate on the far-infrared image is X fir is the horizontal angle of view of the near-infrared camera, θ nir_h , the horizontal angle of view of the far-infrared camera is θ fir_h Then, it is expressed as the following equation (7).

[0114] The y coordinate on the far-infrared image, Yfir, is the vertical angle of view of the near-infrared camera, θ nir_v , the vertical angle of view of the far-infrared camera is θ fir_v Then, it is expressed as the following equation (8): where d represents the difference between the optical axes, and L represents the distance to the object.

[0115] Thus, the third control unit 62 can accurately calculate the relative temperature difference between the skin temperatures of the subject's forehead and nose based on the image information from the far-infrared detection unit 61, following the automatic detection result of the measurement target area using the image information from the near-infrared detection unit 60.

[0116] C. Respiration measurement based on skin temperature Respiration is related to the function of the autonomic nervous system and the immune system, and is used to diagnose sleep disorders. There are two types of breathing: nasal breathing (Figure 10(A)) and mouth breathing (Figure 10(B)), with nasal breathing being the preferred form of breathing. When breathing through the mouth, the inhaled air does not pass through the cilia and mucous membranes of the nasal cavity. Mouth breathing can also cause various diseases, including sleep apnea syndrome (SAS).

[0117] As shown in Figures 11(A) and (B), the nostrils and oral cavity are warmed by exhaled air (35-36°C) and cooled by inhaled air (23-28°C). Therefore, regions of interest (ROIs) are set at the subject's nostrils and oral cavity, and the temperature changes at the nostrils and oral cavity due to breathing are measured using a far-infrared detector (far-infrared camera). Because the surface temperature of the skin is affected by the ambient air temperature, it is measured as the relative temperature between the nostrils and oral cavity.

[0118] In practice, because it is difficult to automatically detect the measurement target area using a skin temperature distribution map, the measurement target areas (nostrils and oral cavity) are automatically detected using image information from the near-infrared detector (camera) as described above, and the region of interest (ROI) is set. The nostril temperature is relatively high during nasal breathing, while the oral cavity temperature is relatively low during mouth breathing.

[0119] We measured the subject's nasal breathing and mouth breathing for 30 seconds each, then took a one-minute break and repeated the same respiration measurement four times. As a result, as shown in Figure 12(A), we confirmed that the temperature of the nostrils decreased due to inhalation during nasal breathing. Furthermore, as shown in Figure 12(B), we confirmed that the temperature of the oral cavity also decreased due to mouth breathing.

[0120] Furthermore, as shown in Figure 13(A), the opening and closing of the mouth due to a change in breathing method from nose breathing to mouth breathing was also confirmed based on the distance between the upper and lower lips. When the graphs in Figures 12(A), 12(B), and 13(A) are superimposed, it is shown in Figure 13(B).

[0121] The method for extracting the timing of inhalation is to first use a moving average filter to remove high frequency noise from the measurement results of the relative temperatures of the nostrils and oral cavity, and then perform first differentiation to extract the timing of the temperature drop due to exhalation.

[0122] Furthermore, if the nostril temperature and oral temperature drop at the same time, the intra-alveolar pressure becomes negative relative to atmospheric pressure, causing air to flow into the nasal passages, making it possible to distinguish this as mouth breathing.

[0123] In this way, it is possible to calculate the respiration rate and distinguish the respiration method using a non-contact measurement method. It is possible to measure nasal breathing and mouth breathing simultaneously, and by executing an algorithm that takes into account the characteristics of temperature changes during each breath, it is possible to accurately calculate the respiration rate and distinguish the respiration method.

[0124] D. Oxygen saturation using two types of near-infrared wavelengths A pulse oximeter is a medical device that uses the light absorption characteristics of hemoglobin in the blood. A pulse oximeter is attached to the fingertip and measures blood oxygen saturation and pulse rate.

[0125] In this invention, we focused on the difference in absorption characteristics between oxygenated hemoglobin (HbO2) and reduced hemoglobin (Hb), since blood oxygen saturation varies depending on respiratory rate, the degree of hemoglobin-oxygen binding, and cardiac output.

[0126] In conventional measurement methods, infrared light (wavelength around 660 [nm]) and near-infrared light (wavelength around 900 [nm]) are irradiated onto the fingertip from a light-emitting element, and the blood oxygen saturation is estimated from the ratio of the light intensity of each transmitted light measured by a light-receiving element, while the pulse rate is also estimated using the periodic change in the light intensity of the transmitted light.Infrared light, which is visible light, has been used because there is a large difference in the light absorption coefficient between oxygenated hemoglobin (HbO2) and reduced hemoglobin (Hb).

[0127] However, in the present invention, non-contact measurement is performed using only near-infrared (NIR) light, which is invisible light (Fig. 14). As a result, by alternately turning on and off light-emitting elements that emit two types of near-infrared light, one with a wavelength of 800 (or 760) [nm] and the other with a wavelength of 900 [nm], and photographing the region of interest (ROI) irradiated by each light-emitting element with a single near-infrared camera (near-infrared detection unit 60), it is possible to measure the blood oxygen saturation of the subject centered on the ROI without contact and even in a dark environment.

[0128] (2-5) Method for detecting the activity state of the nervous system The autonomic nervous system includes the sympathetic nervous system, which functions when the living body is in a state of tension or activity, and the parasympathetic nervous system, which functions when the living body is at rest. When the sympathetic nervous system is dominant, blood pressure and pulse rate increase, while when the parasympathetic nervous system is dominant, blood pressure and pulse rate decrease, so there is a high correlation between autonomic nervous function and cardiac function.

[0129] As a method for measuring autonomic nervous function, there is a method for measuring cardiac sympathetic nervous function using heart rate fluctuations. Specifically, there is the CVR-R (Coefficient of Variation of R-R intervals) method, which uses the R-R intervals of an electrocardiogram waveform to calculate and evaluate the coefficient of variation. Secondly, there is a method in which the fluctuation power ratio of the frequency components (high frequency components and low frequency components) of heart rate fluctuations is used as an index of sympathetic nervous activity.

[0130] There is also a method for measuring autonomic nervous function by measuring the function of the sympathetic nerves in the vascular system using pulse waves, particularly a method for calculating the magnitude of amplitude fluctuation in the photoplethysmogram (PPG) waveform as an evaluation value of autonomic nervous function.

[0131] In the present invention, the third control unit 62 can contribute to the evaluation of the subject's autonomic nervous function (presence or absence of disturbance, etc.) based on all or a combination of the subject's pulse rate detected using the near-infrared detection unit 60 and far-infrared detection unit 61 that constitute the human / environment information input / output unit 10, the relative temperature difference between the skin temperatures of the subject's forehead and nose, the subject's respiratory rate and breathing method, and the subject's oxygen saturation.

[0132] (2-6) Method for Detecting Psychological Information In the person and environment information input / output unit 10, the information detection unit 25 detects psychological information by analyzing the facial expression and voice of the subject using the face imaging camera 21 and the sound collecting microphone 22.

[0133] Specifically, the information detection unit 25 executes a face analysis algorithm to detect the facial expressions of the subject (including not only expressions such as sadness, joy, anger, empathy, and frivolity, but also flushed skin, goosebumps, frequency of eyelid opening and closing, tears, etc.), and at the same time executes a voice analysis algorithm to detect the tone and pitch of the subject's voice (tone of voice, elation, etc.).

[0134] This facial analysis algorithm is designed to approximate human interpretation (experience, preconceptions, and expectations) and is derived from the accumulated results of how emotions and psychology are reflected in facial expressions based on the state and changes of the human face.

[0135] Similarly, voice analysis algorithms are designed to approximate human interpretation (experience, biases, and expectations) and are derived from the accumulated results of how emotions and psychology are reflected in the voice based on human tone and pitch.

[0136] Using this voice analysis algorithm, it is possible to estimate changes in the subject's psychological state through the subject's everyday communication and narrative analysis, based on the keywords used in the subject's speech, the intonation of the speech, the volume of the voice, the structure of the narrative, the continuity / discontinuity of communication, and other accumulated data.

[0137] In addition, the information detection unit 25 can also estimate the psychological state of the subject based on the subject's personal information (heart rate, electrocardiogram, pulse, blood pressure, body temperature, skin blood flow distribution, skin temperature distribution, sweating, odorous substances, etc.) detected by the vital sign measuring device 20 described above, and the subject's environmental information detected by the peripheral measuring device 24 described below.

[0138] In this way, the information detection unit 25 can comprehensively estimate and grasp a person's psychological information from both the person information obtained by sensing and the person information obtained by communication analysis and narrative analysis.

[0139] (2-7) Method for detecting the activity state of the motor system (movement information and behavior information) In the human / environment information input / output unit 10, the information detection unit 25 detects the movement data recognized by the motion capture device 23 as the activity state of the subject's motor system.

[0140] 15, the motion capture device 23 is placed in an indoor facility such as the subject's home, and is composed of a control unit 70 that controls the entire device, and an RGB-D sensor 71 and an IMU (Inertial Measurement Unit) sensor 72 that are connected to the control unit 70. The IMU sensor 72 is composed of a triaxial acceleration sensor, a triaxial angular velocity sensor, and a geomagnetic sensor, and is capable of measuring triaxial acceleration, angular velocity, and geomagnetism.

[0141] In addition to its RGB color camera function, the RGB-D sensor 71 also has a depth sensor that can measure the distance to an object as seen from the camera, enabling it to perform 3D scanning of the object. If, for example, a LiDAR camera L515 from RealSense (a trademark of Microsoft Corporation) is used as the RGB-D sensor 71, the depth sensor is made up of a LiDAR sensor that measures the time it takes for laser light to irradiate the object, hit it, and bounce back, thereby measuring the distance and direction to the object.

[0142] In practice, the RGB-D sensor 71 is positioned so as to capture an image of the subject's entire body based on the subject's standing position in the indoor equipment, and sequentially acquires an RGB image and a depth image as the imaging results.

[0143] The control unit 70 has a fourth control unit 73 consisting of a CPU (Central Processing Unit) that controls the entire system, and a data storage unit 74 in which various data is stored in a database that can be read and written in response to instructions from the fourth control unit 73.

[0144] The fourth control unit 73 is composed of an image coordinate setting unit 80, a coordinate group extraction unit 81, a skeletal information acquisition unit 82, and a behavior recognition unit 83. The image coordinate setting unit 80 estimates the posture of the entire body of the subject based on the RGB images sequentially acquired from the RGB-D sensor 71, and sets a plurality of skeletal representative points attached to the entire body of the subject as image coordinates.

[0145] Specifically, the image coordinate setting unit 80 sets a group of image coordinates in a two-dimensional coordinate system in which the coordinate origin is either the left or right shoulder of the subject, the line connecting the left and right shoulders is the X-axis, and the direction perpendicular to the X-axis is the Z-axis, based on the coordinate origin. That is, the coordinate origin is moved to the right or left shoulder, and the XZ plane is rotated so that it coincides with the line connecting the left and right shoulders. As a result, it becomes possible to recognize movements with increased robustness regardless of the position of the RGB-D sensor 71.

[0146] The coordinate group extraction unit 81 sequentially extracts image coordinate groups related to the subject's actions based on the transition of the subject's posture from the image coordinate groups set by the image coordinate setting unit 80. In practice, the coordinate group extraction unit 81 needs to extract key points of the entire body from the actions, taking into account changes in the subject's clothing and surrounding environment.

[0147] Therefore, the coordinate group extraction unit 81 employs a lightweight OpenPose architecture (a system that estimates a person's skeleton using deep learning) that uses a trained model to estimate the posture of the entire body and obtain image coordinates of the skeletal representative points of the fingers of both hands. For example, for the subject's upper body, the coordinate group extraction unit 81 sequentially extracts a total of 60 skeletal representative points as image coordinate groups: 18 points on the torso of the subject's upper body and 42 points on the fingers of both hands (21 points on each finger). Furthermore, by using only stages 1 and 2 of the six stages of the OpenPose architecture, the amount of calculation can be reduced by 32.4%.

[0148] The skeletal information acquisition unit 82 acquires three-dimensional skeletal information that is continuous over time, centered on the fingers of both hands of the subject, by temporally synchronizing the image coordinate groups sequentially extracted by the coordinate group extraction unit 81 with the depth images sequentially acquired from the RGB-D sensor 71.

[0149] Specifically, the skeletal information acquisition unit 82 constructs a three-dimensional coordinate system by combining the depth direction in the depth images sequentially acquired from the RGB-D sensor 71 as the Y-axis with the two-dimensional coordinate system of the image coordinate groups sequentially extracted by the coordinate group extraction unit 81, and acquires three-dimensional skeletal information.

[0150] The transformation equation from the image coordinate system, which is a two-dimensional coordinate system, to the world coordinate system, which is a three-dimensional coordinate system, is expressed as the following equation (1). Here, X Ｗ = ｔ [X Ｗ, Y Ｗ, Z Ｗ ] indicates a coordinate in the world coordinate system, and x Ｉ = ｔ [x Ｉ ,y Ｉ] indicates coordinates in the image coordinate system. K (3 × 3 matrix) is an intrinsic parameter of the image sensor, R (3 × 3 matrix) is a rotation matrix that performs coordinate transformation to the image coordinate system, t (3 × 1 vector) is a translation vector, Z Ｃ represents depth information corresponding to two-dimensional coordinates in the image coordinate system.

[0151] K, R, and t were calculated by camera calibration using Zhang's method ("A flexible new technique for camera calibration", IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000.). A calibration board with circular dots printed in a grid pattern was used for the calibration to calculate K. The central coordinates of each dot on the calibration board were calibrated to 1 / 1,000.

[0152] As a result, the motion capture device 23 clarifies the relationship between the position in three-dimensional space and the position of the captured image for the entire body of the subject, as shown in Figure 16, and is able to acquire three-dimensional skeletal information while maintaining high robustness regardless of the imaging position taken by the RGB-D sensor 71.

[0153] The behavior recognition unit 83 recognizes behavior, which is a connection of multiple local movements, from the transition state of the subject's posture based on the three-dimensional skeletal information sequentially acquired by the skeletal information acquisition unit 82.

[0154] The action recognition unit 83 uses the action recognition pattern set for each local action as training data, and while referring to the hand-washing action recognition model constructed by deep learning, sequentially recognizes the corresponding local action from the three-dimensional skeletal information sequentially acquired by the skeletal information acquisition unit 82.

[0155] Specifically, in order to recognize multiple local actions, the action recognition unit 83 applies an action recognition model consisting of three modules, namely, a convolutional neural network (CNN) layer, a batch normalization layer (Batch Norm), and an activation function layer (tanh function), as well as a fully connected layer, as shown in FIG. 17 .

[0156] In the behavior recognition model, it is possible to input skeletal information from the current frame and the frame two frames before it and calculate the likelihood of each of multiple types of local motions related to behavior.

[0157] Furthermore, the behavior recognition unit 83 performs post-processing on the most frequently occurring behavior among the behaviors observed in 15 frames (the current frame and the 14 frames preceding it) as the current local behavior as a recognition result. In this way, the behavior recognition unit 83 trained the behavior recognition model through supervised learning using a unique dataset. For optimization, a cross-entropy loss function and Adam (adaptive movement estimation) with a learning rate of 0.001 were used.

[0158] As a result, the motion capture device 23 can significantly improve the accuracy and speed of recognizing local movements based on three-dimensional skeletal information.

[0159] (2-8) Method for detecting environmental information In the person / environment information input / output unit 10, the peripheral measuring device 24 includes a thermometer, a hygrometer, a barometer, an audible and ultrasonic ambient sound collector, etc., and acquires environmental information around the subject. This environmental information mainly includes the temperature, humidity, and ambient sound around the subject.

[0160] (3) Adaptive identification method for internal state In the augmented space construction system 1 of the present invention, when multiple participants taking part in an information space take part in a communication activity in the information space, they are allowed to share the content of their activities with each other while reflecting influence information to guide the communication activity in a positive direction.

[0161] The adaptive state estimation unit 12 in each participant's interface device 2 adaptively estimates the internal state of the corresponding participant based on the person and environment information output from the person and environment information input / output unit 10. As shown in Fig. 18 , in which the same reference numerals are used for parts corresponding to those in Fig. 2 , the adaptive state estimation unit 12 has an identification model estimation unit 90, a difference estimation unit 91, and a parameter identification mechanism unit 92.

[0162] The identification model estimation unit 90 sets up an identification model in which the internal state of each participant is modeled as a state variable, and estimates internal parameters, which are variable characteristics of the identification model, based on the human and environmental information output from the human and environmental information input / output unit 10.

[0163] The difference estimation unit 91 estimates the difference in how close the identification model related to the internal parameters estimated by the identification model estimation unit 90 is to the human and environmental information output in real time from the human and environmental information input / output unit 10.

[0164] The parameter identification mechanism unit 92 corrects the internal parameters that constitute the identification model so that the difference estimated by the difference estimation unit 91 becomes smaller.

[0165] In this way, the adaptive state estimation unit 12 estimates the internal parameters corrected by the parameter identification mechanism unit 92 as the internal state of the participant.

[0166] As mentioned above, "internal states" are information that contributes to empathy among participants who share the same space, and therefore, in order to understand this empathy, they are often explained as a basic counseling attitude (an attitude of active listening) using the three principles of Carl Rogers' theory.

[0167] The three principles of Carl Rogers' theory are "Empathic Understanding," "Unconditional Positive Regard," and "Congruence," which he positions as the core conditions for therapeutic personality change.

[0168] "Empathic understanding" means perceiving the other person's inner world as if it were your own, and understanding what they are trying to say as if you were in their position. It is also necessary to make an effort to communicate this perception to them. The "as if" mentioned above means that it is not possible to actually grasp the other person's inner world, so it is necessary to virtually identify (estimate) it in an information space. By providing feedback to the other person and clarifying the differences, further feedback can bring them closer to the next stage of self-concordance.

[0169] "Unconditional positive consideration" also means listening to what the other person is saying without judging it as good or bad, or liking or disliking it, and listening with positive interest, without denying what the other person is saying, and asking why they think that way. Genuine interest in the other person's problems makes them feel at ease when talking. While smoothly acquiring person and environmental information from the other person, be conscious of system identification (identifying the other person's model) and try to continue transmitting information as much as possible. It is good to increase the accuracy of system identification by applying identification input for system identification as appropriate.

[0170] Furthermore, "self-concordance" means experiencing and accepting one's own true emotions, and authenticity as a human being (what one says and what one does). To improve the accuracy of one's understanding of the other person, it is a good idea to provide feedback to a virtually prepared identification model of the other person (system identification).

[0171] In this case, it is important not only to identify the other person's model, but also for the other person to identify their own model. Therefore, rather than reacting through an idealized model of oneself, one provides feedback based on one's own true reaction. If one's false ideal self is consistent with one's true self, one's and the other person's models will mutually identify, and while receiving feedback from one another, one will adaptively identify and approach each other's models. In this way, mutual understanding leads to the "empathic understanding" mentioned above.

[0172] Ultimately, "empathy" is defined as "empathic understanding," assuming that complete empathy with others cannot be achieved. Furthermore, in order to promote "empathy," we must constantly work to have the other person accept the input necessary for model identification with a positive attitude so that we can empathize, and based on the information obtained from the other person, we must provide feedback to minimize the difference between the information obtained from the other person from the results of identification and state estimation and the information that can be considered empathic understanding, while aligning the model with reality so that the difference between the other person's state obtained as empathic understanding and their actual reactions and narratives becomes smaller.

[0173] Next, when each participant participates in a communication activity, the human relationship evaluation unit 13 integrates the internal states of all participants participating in the communication activity and evaluates the overall and interpersonal human relationships among the participants.

[0174] These human relationships represent the intellectual, emotional, and willful correlations between participants as a whole and between individuals, and refer to relationships between people that integrate psychology, emotions, personality, and attitudes. Examples include relationships between specific participants (one-to-one), relationships between a specific participant and multiple participants (one-to-many), and relationships between multiple participants (many-to-many).

[0175] Based on the evaluation results by the human relationship evaluation unit 13, the encouragement information generation unit 14 successively generates encouragement information representing positive words, actions, and attitudes aimed at fostering empathetic understanding among participants in the information space.

[0176] In other words, the influencing information is information that represents the words, actions, and attitudes that allow the participant to more freely experience their inner emotions, perceptions, and personal meanings in order to convey an empathetic understanding that is accurate and sensitive to the present moment, based on the evaluation of the participants' intra- and interpersonal relationships.

[0177] Furthermore, the working information generating unit 14 changes the content of the working information based on the internal parameters estimated by the identification model estimating unit 90 so that the correction by the parameter identifying mechanism unit 92 is gradually reduced.

[0178] The activity reflecting unit 15 reflects the encouraging information generated by the encouraging information generating unit 14 to all or each participant individually so as to intervene in the communication activities of each participant constructed in the information space.

[0179] In this case, the activity reflection unit 15 reflects not only the conversation but also non-verbal body language, facial expressions, tone of voice, breathing patterns, behavior, etc. as means of intervention in the communication activity based on the influence information, thereby fully promoting empathy between participants while playing the role of a facilitator.

[0180] What this facilitator does is to understand the participants and environmental information sharing the information space, and then determine the state they want to bring to, while changing the input necessary to facilitate according to that intention.

[0181] Specifically, there are two points to keep in mind when facilitating. First, if you understand the participants and environmental information sharing the space, you can use identifying input (language, behavior, facial expressions, etc.) to understand the internal state of the participants sharing the space (current state), and provide information to those involved to facilitate moving the current state toward the goal (the desired state to be reached: consensus building, empathy, etc.).

[0182] Secondly, by acquiring information from a situation that has already been formed by environmental information and personal information from participants (such as when interaction has already occurred between participants), the facilitator can use that information as input for identification, and perform identification without providing any information themselves (or by adding a little), and then provide information to those involved that will facilitate the process in the desired direction.

[0183] The reason why a facilitator intervenes is to change the characteristics that a person or the space that people make up has at that time, and to change them into different characteristics. Explaining this semantically, there is a simple process where a facilitator intervenes and the problem is solved on the spot, but if this is the extent of it, the situation will return to its original state once the facilitator leaves.

[0184] For this reason, advanced facilitators aim to change the characteristics of the people and the space that make up the space through multiple, overlapping interventions. In this way, even without feedback, "the system itself can exist as a system with the characteristics of a problem being solved," and advanced facilitation is realized (in fact, the model itself has changed).

[0185] This "system itself can exist as a system with problem-solving properties" means that this is the result of the synaptic connections in the nervous system being adjusted (gradual change due to plasticity), or that the memory of one or several keywords can instantly change the properties of the system itself (a mechanism that forces you to skip something internally).

[0186] According to the above configuration, the augmented space construction system 1 facilitates mutual empathy during interactions between participants in the information space, promoting active interaction, thereby enabling participants to share the internal environment of the information space, experience physical and psychological connections with others, and improve empathy for others.

[0187] When applying this augmented space construction system 1 to counseling, the requirement of a counselor, "active listening," is interpreted from a systems theory perspective. In other words, from a systems theory perspective, it can be explained by an identification process and state observation.

[0188] In this systems theory interpretation, input that affects the participants' characteristics is given in moderation, and the internal state is estimated by identifying the participants' characteristic parameters from information obtained from the participants and the environment, thereby understanding the participant's state.

[0189] Going a step further, good counseling can be considered to have taken place when, by providing internal and external feedback to the participant, it is possible to modify the participant's characteristic parameters or change their dynamic equilibrium point, gradually shifting them to another stage and correcting them to a new, better state.

[0190] (4) Other Embodiments (4-1) Improving Accuracy of Exchange Content Through Learning In Fig. 19, in which parts corresponding to those in Fig. 18 are assigned the same reference numerals, an interface device 100 for each participant has an evaluation learning unit 101 interposed between the encouragement information generation unit 14 and the activity reflection unit 15. The evaluation learning unit 101 constructs an effectiveness evaluation model that has been trained to evaluate the effectiveness of the encouragement information generated by the encouragement information generation unit 14, based on the integration result of the person and environment information of each participant.

[0191] The action information generating unit 14 adaptively corrects the action information so that the action information always satisfies the evaluation criteria based on the effectiveness evaluation model constructed by the evaluation learning unit 101 .

[0192] As a result, the augmented space construction system 1 makes it possible to maintain a relatively high level of influence information (such as expert knowledge from clinical psychologists) that influences the internal state of each participant during their activities in the information space.

[0193] (4-2) Improving the Information Space Environment through Learning In FIG. 20, in which the same reference numerals are used to denote parts corresponding to those in FIG. 18, the interface device 110 for each participant further includes a goodness evaluation learning unit 111 and an environmental state correction unit 112.

[0194] The goodness evaluation learning unit 111 constructs a goodness evaluation model that is trained to evaluate the goodness of the environmental state (atmosphere, background sounds, harmony of brightness, saturation, color tone, etc.) of the selected information space based on the integrated results of each participant's human and environmental information.

[0195] The environmental state correcting unit 112 adaptively corrects the environmental state of the information space so as to always satisfy the evaluation criteria based on the goodness evaluation model constructed by the goodness evaluation learning unit 111 .

[0196] As a result, the augmented space construction system 1 can always maintain the environmental state of the information space, which affects the internal state of each participant, in an optimal state while the participant is active in the information space.

[0197] (4-3) Method of integrating internal states of participants In Figure 21, in which the same symbols are used for parts corresponding to those in Figure 18, the action information generation unit 121 in the interface device 120 for each participant has a correlation quantification unit 122 and a transition state evaluation unit 123, and is configured to adaptively adjust the action information.

[0198] The correlation quantification unit 122 chronologically quantifies the correlations between the internal states of the participants. The transition state evaluation unit 123 evaluates the transition state of the internal state of each participant based on the correlations quantified by the correlation quantification unit 122. The action information generation unit 121 then adaptively adjusts the action information in accordance with the evaluation result by the transition state evaluation unit 123.

[0199] As a result, the augmented space construction system 1 can grasp, as a whole system, the process by which each participant's internal state changes as it is influenced by the words, actions, and attitudes of other participants, making it possible to respond (interact) appropriately to each participant.

[0200] (4-4) Expanding the activity area in the information space In Figure 22, in which the same symbols are used for parts corresponding to those in Figure 18, the interface device 130 for each participant is provided with a spatial layer construction unit 131, a learning model training unit 132, and a layer management unit 133.

[0201] When each participant is engaged in a communication activity in the information space, the space layer construction unit 131 constructs a space layer for expanding the information space while the participants in the information space cooperate with each other.

[0202] As a result, in the augmented space construction system 1, it becomes possible for participants to expand the information space in cooperation with each other within the information space.

[0203] In addition, the learning model training unit 132 trains a learning model constructed by statistical and mathematical processing to map the collaboration content between participants into a spatial extension of the information space based on the spatial layer construction process constructed by the spatial layer construction unit 131.

[0204] Then, the spatial layer construction unit 131 automatically constructs a spatial layer based on the learning model trained by the learning model training unit 132 without the cooperation of each participant.

[0205] As a result, the augmented space construction system 1 is able to learn the process of expanding the information space by learning, and can continue the expansion process automatically without the cooperation of each participant.

[0206] Furthermore, when the learning model training unit 132 trains the learning model, the layer management unit 133 manages the spatial layers constructed in the information space to be expanded in order to maintain order.

[0207] As a result, in the extended space construction system 1, when continuing the expansion process of the information space, management aimed at maintaining order is carried out, making it possible to prevent each participant from suffering any disadvantages.

[0208] 1...Augmented space construction system, 2, 100, 110, 120, 130...Interface device, 3...Cloud, 10...Human and environment information input / output unit, 11...Virtual activity construction unit, 12...Adaptation state estimation unit, 13...Human relationship evaluation unit, 14, 121...Influence information generation unit, 15...Activity reflection unit, 20...Vital measurement device, 21...Facial imaging camera, 22...Sound collection microphone, 23...Motion capture device, 24...Peripheral measurement device, 25...Information detection unit, 30...Electrode terminal group, 31...Photoelectric probe window, 32...Connection terminal, 40...Brain activity measurement unit, 41...Blood flow measurement unit, 42...Electrocardiograph, 43...First control unit, 44...Wireless communication device, 45...Net-like base, 46A to 46N...Sensor unit, 47...Light emitter, 48...Light receiver, 50...Biological signal measurement wearing device, 51...Drive unit, 52...Joint circumference detection unit, 5 3...biological signal sensor, 54A to 54F...biological signal sensor group, 55...measurement module, 56...measurement module controller, 57...memory, 58...communication unit, 59...second control unit, 60...near-infrared detection unit, 61...far-infrared detection unit, 62...third control unit, 70...control unit, 71...RGB-D sensor, 72...IMU sensor, 73...fourth control unit, 74...data storage unit, 80...image coordinate setting unit, 81...coordinate group extraction unit, 82...skeletal information acquisition unit, 83...behavior recognition unit, 90...identification model estimation unit, 91...difference estimation unit, 92...parameter identification mechanism unit, 101...evaluation learning unit, 111...goodness evaluation learning unit, 112...environmental state correction unit, 122...correlation quantification unit, 123...transition state evaluation unit, 131...spatial layer construction unit, 132...learning model training unit, 133...layer management unit.

Claims

1. A human and environment information input / output unit provided for each of a plurality of participants existing in a real space, which acquires human and environment information including brain and nervous system information, musculoskeletal system information, physiological information, psychological information, motion information, behavioral information and environmental information of each participant; a virtual activity construction unit which virtually constructs a communication activity of each participant in the real space in real time in an information space based on the human and environment information corresponding to each participant output from the human and environment information input / output unit; an adaptive state estimation unit provided for each participant, which adaptively estimates the internal state of the corresponding participant based on the human and environment information output from the human and environment information input / output unit; a human relationship evaluation unit which, when each participant participates in the communication activity, evaluates the intra-participant and inter-personal human relationships among the participants while integrating the internal states of all the participants participating in the communication activity; and an encouraging information generation unit which successively generates encouraging information representing positive words, actions and attitudes aimed at empathetic understanding between the participants in the information space based on the evaluation results by the human relationship evaluation unit. and an activity reflection unit that reflects the action information generated by the action information generation unit to all or each of the participants individually so as to intervene in the communication activities of each of the participants constructed in the information space.

2. The augmented space construction system described in claim 1, characterized in that the adaptive state estimation unit comprises: an identification model estimation unit that sets up an identification model in which the internal state of each participant is modeled as a state variable, and estimates internal parameters, which are variable characteristics of the identification model, based on the person and environment information; a difference estimation unit that estimates the difference of how close the identification model related to the internal parameters estimated by the identification model estimation unit has come to the person and environment information output in real time from the person and environment information input / output unit; and a parameter identification mechanism unit that modifies the internal parameters that constitute the identification model so that the difference estimated by the difference estimation unit becomes smaller; and the action information generation unit changes the content of the action information based on the internal parameters estimated by the identification model estimation unit so that the modification by the parameter identification mechanism unit is gradually reduced.

3. The augmented space construction system described in claim 2, further comprising an evaluation learning unit that constructs an effectiveness evaluation model that is trained to evaluate the effectiveness of the action information generated by the action information generation unit based on the integrated result of the human and environmental information of each participant, and the action information generation unit adaptively modifies the action information so that it always satisfies the evaluation criteria based on the effectiveness evaluation model constructed by the evaluation learning unit.

4. The augmented space construction system described in claim 2, characterized in that it comprises: a goodness evaluation learning unit that constructs a goodness evaluation model that is trained to evaluate the goodness of the environmental state of the selected virtual space based on the integrated result of the human and environmental information of each participant; and an environmental state correction unit that adaptively corrects the environmental state of the information space so as to always satisfy the evaluation criteria based on the goodness evaluation model constructed by the goodness evaluation learning unit.

5. The augmented space construction system described in claim 2, characterized in that the action information generation unit comprises a correlation quantification unit that quantifies the correlations between the internal states of the participants in a time series manner, and a transition state evaluation unit that evaluates the transition states of the internal states of the participants based on the correlations quantified by the correlation quantification unit, and adaptively adjusts the action information in accordance with the evaluation results by the transition state evaluation unit.

6. The extended space construction system according to claim 1, further comprising a spatial layer construction unit that constructs a spatial layer for expanding the information space in cooperation with the participants in the information space when the participants engage in communication activities in the information space.

7. The extended space construction system of claim 6, further comprising a learning model training unit that trains a learning model constructed by statistical and mathematical processing to map the content of collaboration between the participants to a spatial extension of the information space based on the construction process of the spatial layer constructed by the spatial layer construction unit, wherein the spatial layer construction unit automatically constructs the spatial layer based on the learning model trained by the learning model training unit even without collaboration between the participants.

8. The augmented space construction system described in claim 7, characterized in that it is provided with a layer management unit that manages the spatial layers constructed in the information space to be expanded for the purpose of maintaining order when the learning model is trained by the learning model training unit.

9. A first step of acquiring human and environmental information, including brain and nervous system information, musculoskeletal system information, physiological information, psychological information, motion information, behavioral information, and environmental information, for each of a plurality of participants existing in a real space; a second step of virtually constructing in real time in an information space a communication activity of each of the participants in the real space based on the human and environmental information corresponding to each of the participants acquired by the first step; a third step of adaptively estimating an internal state of the corresponding participant based on the human and environmental information output from the human and environmental information input / output unit for each of the participants; a fourth step of evaluating, when each of the participants participates in the communication activity, the overall and interpersonal relationships between the participants while integrating the internal states of all the participants participating in the communication activity; and a fifth step of successively generating encouraging information representing positive words, actions, and attitudes aimed at empathic understanding between the participants in the information space based on the evaluation results from the fourth step. and a sixth step of reflecting the interaction information generated in the fifth step to all or each of the participants individually so as to intervene in the communication activities of each of the participants constructed in the information space.

10. The method for constructing an augmented space as described in claim 9, characterized in that in the third step, an identification model is set in which the internal state of each participant is modeled as a state variable, internal parameters which are variable characteristics of the identification model are estimated based on the person and environment information, and a difference in how close the identification model related to the estimated internal parameters has come to the person and environment information acquired in real time in the first step is estimated, while modifying the internal parameters constituting the identification model so as to reduce the estimated difference; and in the fifth step, the content of the influence information is changed so that the modification of the internal parameters is gradually reduced based on the internal parameters estimated based on the person and environment information.

11. The augmented space construction method described in claim 10, further comprising a seventh step of constructing an effectiveness evaluation model that is trained to evaluate the effectiveness of the influence information generated by the influence information generation unit based on the integrated result of the human and environmental information of each participant, and in the fifth step, the influence information is adaptively modified so as to always satisfy evaluation criteria based on the effectiveness evaluation model constructed in the seventh step.

12. The augmented space construction method described in claim 10, characterized in that it comprises: an eighth step of constructing a goodness evaluation model that is trained to evaluate the goodness of the environmental state of the selected virtual space based on the integrated result of the human and environmental information of each participant; and a ninth step of adaptively modifying the environmental state of the information space so as to always satisfy an evaluation criterion based on the goodness evaluation model constructed by the eighth step.

13. The method for constructing an augmented space as described in claim 10, characterized in that in the fifth step, the transition state of the internal state of each participant is evaluated based on the correlation between the internal states of the participants obtained by quantifying them over time, and then the influence information is adaptively adjusted according to the result of the evaluation.

14. The method for constructing an extended space as described in claim 9, further comprising a tenth step of constructing a spatial layer for expanding the information space in cooperation with the participants in the information space when the participants engage in communication activities in the information space.

15. The extended space construction method of claim 14, further comprising: an eleventh step of training a learning model constructed by statistical and mathematical processing for mapping the content of collaboration between the participants to a spatial extension of the information space based on the construction process of the spatial layer constructed in the tenth step; wherein in the tenth step, the spatial layer is automatically constructed based on the learning model trained in the eleventh step, even without collaboration between the participants.

16. The method for constructing an augmented space as described in claim 15, characterized in that it comprises a 12th step of managing the spatial layer constructed in the information space to be expanded for the purpose of maintaining order when the learning model is trained by the 11th step.

Images