Space sharing system and space sharing method

Abstract

According to the present invention, human and environmental information relating to a subject existing in a real space, including brain and nervous system information, muscle and skeletal system information, physiological information, psychological information, motion information, behavior information and environmental information, are acquired, after which human activities of the subject in the real space are constructed virtually in real time in an information space on the basis of the acquired human and environmental information relating to the subject, and human characteristic information specific to the subject is created while analyzing the human activities of the subject constructed in the information space.

Description

Space sharing system and space sharing method

[0001] The present invention relates to a space sharing system and a space sharing method, and is suitable for application to an extended space including a three-dimensional virtual space and a real space constructed on a network, for example.

[0002] In recent years, the term metaverse has been widely used to describe a fully immersive, three-dimensional virtual space containing virtual environments in which humans are represented by avatars.

[0003] One example of such a metaverse is a virtual space experience system that allows users to view an image of a three-dimensional virtual space constructed on a network, while reflecting the user's human activities in the real world in the virtual space, allowing the user to feel as if they are actually present in that virtual space.

[0004] One proposed virtual space experience system of this type uses a motion capture device to recognize changes in the user's state in real space (e.g., body movements, coordinate movements, changes in posture, etc.), and then changes the state of an avatar in the virtual space in response to the recognized changes in state (see Patent Document 1).

[0005] However, in this virtual space experience system, the user only controls an avatar that exists in the virtual space, so an interface system has been proposed in which the virtual space is used as an interface between two real spaces, and a second avatar is operated through the movements of a first avatar that corresponds to the user in the virtual space, thereby controlling the movements of a controlled object such as a robot that corresponds to the second avatar in a real space that is different in at least one of time and location from the real space in which the user exists (see Patent Document 2).

[0006] JP 2018-49629 A Japanese Patent No. 6933849

[0007] In the interface system of Patent Document 2, a user can control the behavior of a control object that exists in an area different from the area in which the user exists in real space via a virtual space, in accordance with the different area, without taking into account the constraints in that different area.

[0008] However, this interface system only allows the user to relatively freely control the behavior of a control object located in a remote location via a virtual space, and is insufficient in practical terms for providing support to or cooperating with people in remote locations while reflecting information about the user and the environment.

[0009] The present invention has been made taking the above points into consideration, and aims to propose a space-sharing system and a space-sharing method that enable a subject to directly express their individuality when providing support to or collaborating with others in a location different from their current location.

[0010] In order to solve such problems, the present invention comprises a human and environment information input / output unit that acquires human and environment information including brain and nervous system information, musculoskeletal system information, physiological information, psychological information, movement information, behavioral information and environmental information of a subject existing in real space, a virtual activity construction unit that virtually constructs the human activities of the subject in real space in information space in real time based on the human and environment information of the subject output from the human and environment information input / output unit, and an information creation unit that creates human characteristic information specific to the subject while analyzing the human activities of the subject constructed in information space by the virtual activity construction unit.

[0011] As a result, the space-sharing system virtually constructs the subject's human activities in an information space based on the subject's human and environmental information, and creates personal characteristic information specific to the subject from the analysis results of that human activity.This makes it possible for the subject's personal characteristic information to be reflected in the content of their activities when they provide support to others or engage in collaborative activities with others in a location other than their current location.

[0012] In addition, the present invention is provided with an activity transcription unit that, when a request for connection with a subject is received from an external output device, transcribes all or part of the subject's human activity constructed in the information space by the virtual activity construction unit in accordance with the output format of the output device, and the activity transcription unit reflects the human characteristic information specific to the subject created by the information creation unit in the output operation by the output device.

[0013] In this way, in a space-sharing system, when a subject connects to an external output device, all or part of the subject's human activity is transferred according to the output format of the output device, and the subject's unique personal characteristics are reflected in the output operation of the output device, allowing the output device to directly express the subject's individuality.

[0014] Furthermore, the present invention is provided with an activity status transmission unit that transmits feedback of the operating status of the output device to the subject in real space in real time via the information space.

[0015] As a result, in the space sharing system, when a subject operates using an output device, the subject can share the physical sensations of the human body through the output device from interactions related to the subject's activity.

[0016] Furthermore, in the present invention, the activity state transmission unit includes a sensory estimation unit that, when the output device is operating, estimates the physical and psychological sensations of a human being from the action results related to the operation content of the output device and the sensory results of the environmental state in which the output device is located; a sensory feedback generation unit that generates the physical and psychological sensations estimated by the sensory estimation unit as feedback information; and a sensory transmission unit that transmits the feedback information generated by the sensory feedback generation unit to a subject in real space.

[0017] As a result, in the space-sharing system, when a subject operates an output device, the subject can share, through the output device, the human physical and psychological sensations that arise as a result of interactions related to the subject's activity.

[0018] Furthermore, the present invention includes an estimation model construction unit that constructs an estimation model that has been trained to estimate functions with characteristics unique to the subject by statistically analyzing the correlation between the subject's activity results transferred to the output device and the subject's human and environmental information, and an autonomous avatar generation unit that generates an autonomous avatar that is capable of self-expression including the subject's functions in information space based on the estimation model estimated by the estimation model construction unit, and the virtual activity construction unit virtually constructs the human activities of a substitute for the subject in real space while reflecting the self-expression of the autonomous avatar in information space, and the information creation unit creates human characteristic information unique to the substitute while analyzing the human activities of the substitute represented as an autonomous avatar constructed in information space by the virtual activity construction unit.

[0019] As a result, in the space-sharing system, when a substitute who is different from the target person in the real space acts in place of the target person, an entity similar to the target person can be reflected as an autonomous avatar via the information space from the cloud.

[0020] Furthermore, the present invention further includes a personality trait inference unit that infers personality traits related to the subject's self-expression from the operation results of the output device, and when the acquisition of the subject's person and environment information by the person and environment information input / output unit stops for a predetermined period of time, the virtual activity construction unit reflects the personality traits inferred by the personality trait inference unit in the self-expression of the autonomous avatar.

[0021] As a result, in the space-sharing system, even if a target person in the real space dies, it is possible to reflect a presence similar to that target person as an autonomous avatar via the information space from the cloud.

[0022] Furthermore, the present invention includes a first step of acquiring human and environmental information including brain and nervous system information, musculoskeletal system information, physiological information, psychological information, movement information, behavioral information, and environmental information of a subject existing in real space; a second step of virtually constructing the human activities of the subject in real space in an information space in real time based on the human and environmental information of the subject acquired in the first step; and a third step of creating human characteristic information specific to the subject while analyzing the human activities of the subject constructed in the information space in the second step.

[0023] As a result, the space sharing method virtually constructs the subject's human activities in an information space based on the subject's human and environmental information, and creates human characteristic information specific to the subject from the analysis results of the human activities.This makes it possible for the subject's unique human characteristic information to be reflected in the content of their activities when they provide support to people or engage in collaborative activities with people in remote locations.

[0024] According to the present invention, it is possible to realize a space sharing system and a space sharing method that enable a subject to directly express the subject's individuality when providing support to or engaging in collaborative activities with others in a location different from their current location.

[0025] FIG. 1 is a conceptual diagram illustrating the space-sharing system according to the present embodiment. FIG. 1 is a conceptual diagram illustrating the internal configuration of the first interface device shown in FIG. 1. FIG. 1 is a conceptual diagram illustrating a body fluid system model. FIG. 1 is a conceptual diagram illustrating a virtual human body model. FIG. 2 is a perspective view and six-view diagram illustrating the external configuration of a vital sign measuring device. FIG. 3 is a conceptual diagram illustrating the configuration of a brain activity measuring unit. FIG. 4 is a conceptual diagram illustrating the configuration of a biosignal measurement garment. FIG. 5 is a conceptual diagram illustrating the wearing state of the biosignal measurement garment. FIG. 6 is a conceptual diagram illustrating additional configurations of a vital sign measuring device. FIG. 7 is a graph illustrating pulse waveforms before and after filter application. FIG. 8 is a conceptual diagram illustrating the ROI of a near-infrared image and the ROI of a far-infrared image. FIG. 9 is a conceptual diagram illustrating nasal breathing and mouth breathing. FIG. 10 is a conceptual diagram illustrating the temperature state of nasal breathing and mouth breathing. FIG. 11 is a graph illustrating the change in nostril temperature due to nasal breathing and the change in oral cavity temperature due to mouth breathing. FIG. 12 is a graph superimposing the change in breathing technique from nasal breathing to mouth breathing and the details of each state. FIG. 13 is a conceptual diagram illustrating non-contact measurement of oxygen saturation using light. FIG. 14 is a block diagram illustrating the internal configuration of a motion capture device. 20 is a conceptual diagram illustrating a method for acquiring three-dimensional skeletal information about the entire body of a subject. FIG. 21 is a conceptual diagram illustrating the structure of a behavior recognition model. FIG. 22 is a conceptual diagram mainly illustrating the internal configuration of the second interface device shown in FIG. 1. FIG. 23 is a conceptual diagram illustrating a specific example of the output form of an output device. FIG. 24 is a conceptual diagram relating to the internal configuration of a sensation estimation unit and the structure of a load sensor. FIG. 25 is a conceptual diagram including additional configurations of the second interface device shown in FIG.

[0026] An embodiment of the present invention will be described in detail below with reference to the drawings.

[0027] (1) Configuration of the space-sharing system according to this embodiment Fig. 1 shows a space-sharing system 1 according to this embodiment. The space-sharing system 1 has a first interface device 2 that acquires various information obtained from the subject himself / herself and the surrounding environment present in the real space, and the various information transmitted from the first interface device 2 is reflected in an information space constructed in the cloud (information space) 3.

[0028] In this space-sharing system 1, when an external output device 4 is installed in real space together with a corresponding second interface device 5 for the purpose of transcribing all or part of the human activity of a subject, the first interface device 2 corresponding to the subject can be connected to the output device 4 via the cloud (information space) 3 and the second interface device 5.

[0029] 2, the first interface device 2 includes a person / environment information input / output unit 10, a virtual activity construction unit 11, and an information creation unit 12. The person / environment information input / output unit 10 includes vital sign measuring devices (measuring means for electrocardiogram, pulse rate, pulse pressure, oxygen saturation, body temperature, electroencephalogram, blood glucose level, blood pressure, electromyogram, etc.) 20, a facial imaging camera 21, a sound collection microphone 22, motion capture devices (optical, inertial sensor, mechanical, and magnetic types, or a combination thereof) 23, and peripheral measuring devices (thermometer, hygrometer, audible and ultrasonic ambient sound collectors, etc.) 24, as well as an information detection unit 25 that detects person / environment information (described later) based on the outputs of these sensors and devices.

[0030] In this human / environment information input / output unit 10, the information detection unit 25 detects the subject's brain / nervous system information, musculoskeletal system information, and physiological information based on the detection results from the vital sign measuring device 20, and also detects psychological information by analyzing the subject's facial expressions and voice using the facial imaging camera 21 and sound collecting microphone 22, and further detects the subject's movement information and behavior information using the motion capture device 23, and detects surrounding environmental information centered on the subject using the peripheral measuring device 24.

[0031] In addition to the various components of the human / environment information input / output unit 10, the first interface device 2 not only acquires various information from the subject, but also allows the subject, who is wearing a head-mounted display (not shown), to recognize images and sounds acquired from the second interface device 5 corresponding to the output device 4.

[0032] The virtual activity construction unit 11 virtually constructs the human activities of the subject in real space in real time in the information space based on the subject's human and environmental information output from the human and environmental information input / output unit 10 .

[0033] Here, information space refers to a virtual space generated by a computer, typified by a three-dimensional virtual space constructed on a network (cloud 3) and the metaverse, which is a service for such a virtual space. The functionality required for this information space requires two-way communication with the real world via a network. When using an information space as a versatile platform, it is desirable for the system that provides the information space to be open and to include user management and access restriction functions. Examples of information spaces that satisfy these functions include Second Life (registered trademark) provided by Linden Lab, Inc., and OpenSim, which is provided under the BSD License (Berkeley Software Distribution License).

[0034] The information creation unit 12 creates human characteristic information specific to the subject while analyzing the human activities of the subject constructed in the information space by the virtual activity construction unit 11. This human characteristic information is a specific property obtained from the analysis results of the human activities of the subject, and is treated as the subject's biological parameters (neural, physiological, physical, and skeletal systems) and the subject's state variables (transiently changing biological states) formed by environmental information applied to the subject.

[0035] The subject's specific human characteristic information can be expressed as a compartment model of the body fluid system described by matrices, state variables, environmental information (including disturbances), parameters, and eigenvalues.

[0036] Compartment models are not only used in pharmacodynamics research (research into the transfer of substances and water between body fluids) to quantitatively evaluate the absorption, distribution, metabolism, and excretion of tracers (the amount and concentration of substances such as ions and proteins that move with the water that makes up the body fluids that act as their solvent), but are also widely used to predict pharmacokinetics from experimental animals to humans, and play a central role in the issues of bioavailability and bioequivalence in tracer evaluation (including water transfer).

[0037] In general, the blood concentration of a tracer is an issue when it is administered. This is because the blood concentration profile of a tracer reflects its distribution to organs and tissues and its ability to disappear through metabolism and excretion within the body. The blood concentration profile of a tracer is defined by the administered dose, the average blood concentration (systemic clearance), and a value indicating the rate of decline (biological half-life), and these values ​​are estimated based on the physiological characteristics of the living body.

[0038] Additionally, the distribution volume, which shows the relationship between the dose of a tracer and its concentration in the body, is generally proportional to body weight and is said to determine the total body clearance and biological half-life. In this way, examining the blood concentration of a tracer reveals its pharmacokinetics. In tracer kinetics, the living body (as a whole) is represented by a differential equation system based on the balance of the tracer. The components (processing areas) displayed in this differential equation are called compartments.

[0039] For example, a three-compartment model of the body fluid system, consisting of intracellular fluid, interstitial fluid, and blood, will be described as shown in Figure 3. In this compartment model, a therapeutic operation can be initiated through the blood compartment.

[0040] When the distribution volume of the intracellular fluid compartment is Vc, the tracer concentration is Cc, the distribution volume of the interstitial fluid compartment is Vi, the tracer concentration is Ci, and the distribution volume of the blood compartment is Vb, the tracer concentration is Cb, the subject's biological characteristics (body fluid system) can be expressed as a matrix representing the connection pattern between compartments as shown in the following equation (1).

[0041] Here, Kci, Kib, and Kbm respectively represent mass transfer coefficients between compartments, Rci, Rib, and Rbm respectively represent filtration specific resistances, and Sci, Sib, and Sbm respectively represent sieving coefficients.

[0042] In this way, the biological characteristics of the subject (here, the body fluid system) are determined by the properties of the eigenvalues ​​of the matrix expressed by the above-mentioned formula (1). Note that, although the case has been described where human characteristic information specific to the subject is expressed as a model explained by a matrix, state variables, environmental information (including disturbances), parameters, and eigenvalues ​​when creating the information, in reality, it may also be expressed as a model composed of a learning system / function system such as a multidimensional neural network.

[0043] In addition, the subject's unique information regarding their personal characteristics is treated as the subject's state variables (transiently changing biological states) formed by biological parameters (neurological, physiological, physical, and skeletal systems) and environmental information (environmental information applied to the subject).

[0044] As described above, the subject's human characteristics can be expressed using matrices and state variables, and the basic characteristics of a person can be treated as eigenvalues. A class composed of the idea of ​​characteristics unique to such a subject represents the unique characteristics of a person, and as these change, the response changes.

[0045] By labeling this with abstract concepts such as "getting sick," "getting better," "getting worse," "being tenacious," or "being healthy," it becomes easier for the general public to understand by connecting it to everyday meanings.

[0046] By developing the above-mentioned body fluid system model, a virtual human model such as that shown in Figure 4 is constructed. This virtual human model is an architecture in which organs and functions are classified (classes are constructed as units for which parameter identification and state observation are possible), and the behavior of the virtual human is calculated through the interaction of information. By adding classes in a separately constructed mental model to this virtual human model, it becomes possible to construct human characteristics that reflect the individual human characteristics of the system as a whole, based on each person's parameters and observed state.

[0047] As a result, the space sharing system 1 virtually constructs the subject's human activities in an information space based on the subject's personal and environmental information, and creates personal characteristic information specific to the subject from the analysis results of that human activity.This makes it possible for the subject's personal characteristic information to be reflected in the content of their activities when they provide support to others or engage in collaborative activities with others in a location other than their current location.

[0048] (2) Detailed Configuration and Function of the Person / Environment Information Input / Output Unit (2-1) Configuration of the Vital Sign Measuring Device Specifically, as shown in FIG. 5A, the vital sign measuring device 20 has a device body 20A on a measurement surface 20X on which an electrode terminal group 30 and a photoelectric probe window 31 are arranged in a predetermined pattern, and measures the subject's biological information via the measurement surface 20X. The biological information measured by the vital sign measuring device 20 is at least one of the measurement results of electrocardiogram, pulse rate, pulse pressure, oxygen saturation, body temperature, electroencephalogram, blood glucose level, blood pressure, and electromyogram. In other words, the vital sign measuring device 20 is configured so that the same device body can be used as a means for measuring all types of biological information.

[0049] 5B, the vital sign measuring device 20 has a sound collection microphone 22 detachably connected to a connection terminal 32 provided on the side of the device body 20A, and when connected, can collect vibrations generated from the subject's larynx and pharynx as acoustic signals. As a result, it is possible to obtain information about the subject's vibrations (inaudible range) associated with the heart and breathing that cannot be heard by the human ear.

[0050] (2-2) Method for detecting brain and nervous system information This human and environment information input / output unit 10 has a brain activity measurement unit 40 that is worn on the subject's head and measures brain waves and cerebral blood flow in a measurement area of ​​the head, as an additional component of the vital sign measuring device 20. The information detection unit 25 then converts the transmission state of the central nervous system centered on the head into quantitative nervous system data based on the measurement data of brain waves and cerebral blood flow obtained from the brain activity measurement unit 40.

[0051] The brain activity measuring section 40 according to this embodiment is in line with the content described in Japanese Patent Application Laid-Open No. 5283700 by the inventor of the present application, and uses the same principle.

[0052] 6(A), the brain activity measuring unit 40 has a blood flow measuring unit 41, an electrocardiograph 42, a first control unit 43, and a wireless communication device 44. The blood flow measuring unit 41 measures the state of blood flow from changes in the optical path and transmitted light amount of laser light caused by blood flow, and further measures the state of brain activity from the state of blood flow in the brain.

[0053] That is, when the laser light irradiated onto the blood enters the blood layer, it passes through the blood as both light components: a reflected and scattered light component by normal red blood cells and a reflected and scattered light component by adherent thrombi. The influence of the laser light on the process of passing through the blood layer changes from moment to moment depending on the state of the blood. Therefore, by continuously measuring the amount of transmitted light (amount of reflected light) and observing the change in the amount of light, it is possible to observe various changes in the state of the blood.

[0054] The blood flow measurement unit 41 comprises a net-like base 45 formed in a hemispherical shape corresponding to the external shape of the head so as to be worn on the head, and a number of sensor units 46A to 46N. Each of the sensor units 46A to 46N is supported at predetermined intervals by the net-like base 45, and outputs a detection signal of the amount of transmitted light measured at each measurement point on the head to a first control unit 43. The electrocardiograph 42 measures cardiac potentials generated in response to cardiac movement using electrodes E attached to the subject's skin.

[0055] The first control unit 43 derives the displacement of blood vessels and tissues surrounding the blood vessels due to blood flow and measures the state of brain activity (distribution of red blood cells) based on the light intensity when light (FIG. 6B) emitted from the light-emitting unit 47 of each sensor unit 46A to 46N is received by the light-receiving unit 48. The first control unit 43 also stores a control program that performs arithmetic processing to cancel components due to oxygen saturation included in signals obtained from at least two or more light-receiving units 48.

[0056] The first control unit 43 further derives the displacement of the inner wall of the blood vessel based on the displacement of the blood vessel and the tissue surrounding the blood vessel. The first control unit 43 further calculates the pulse wave velocity at each measurement position from the phase difference between the waveform of the electrocardiogram signal from the electrocardiograph 42 and the waveform of the detection signal obtained from the light receiving unit 48, and derives the displacement state of the inner wall of the blood vessel from the pulse wave velocity.

[0057] The wireless communication device 44 wirelessly transmits the measurement results (blood flow data) output from the first control unit 43 to the vital sign measuring device 20. The brain activity measuring unit 40 has optical sensor units 46A to 46N arranged on a net-like base 45, and therefore can simultaneously measure blood flow throughout the entire head.

[0058] 6(B), when measuring cerebral blood vessel characteristics, the first control unit 43 selects an arbitrary sensor unit 46D from the array of multiple sensor units 46A to 46N and causes the light-emitting unit 47 of the selected sensor unit 46D to emit laser light. At this time, the laser light emitted from the light-emitting unit 47 has a wavelength λ (λ≈805 nm) that is not affected by oxygen saturation.

[0059] When the blood flow measurement unit 41 of the brain activity measurement unit 40 is attached to the subject's head, the elasticity of the net-like base 45 positions the multiple sensor units 46A to 46N at each measurement point on the head and holds each measurement surface facing the head surface. With the blood flow measurement unit 41 attached to the head, the multiple sensor units 46A to 46N can measure the vascular characteristics of the middle cerebral artery and anterior cerebral artery (the rate of vascular elasticity, the amount of plaque in the blood vessels, and the rate of arteriosclerosis) by measuring the pulse wave velocity in the cerebral arteries from changes in the amount of light received after irradiating the surface of the brain with light and propagating within the brain.

[0060] The principle of measuring the pulse wave velocity of cerebral arteries will now be described. For example, the pulse wave velocity due to blood flowing through the middle cerebral artery and the anterior cerebral artery is detected by each of the sensor units 46A to 46N, which are arranged corresponding to the measurement positions and receive light that has propagated through the brain when light is irradiated. The measurement method involves comparing the waveform of the cardiac potential obtained from the electrocardiograph 42 with the waveform of the signal output from each of the sensor units 46A to 46N at the measurement positions, determining the pulse wave velocity from the phase difference, and deriving the vascular characteristics corresponding to the pulse wave velocity.

[0061] Next, the principle of detecting vascular characteristics from cerebral blood flow will be described. As shown in Figure 6(B), the brain BR is covered by cerebrospinal fluid BR1, skull BR2, and scalp BR3. A portion of the laser light emitted from the light-emitting unit 47 of each sensor unit 46A to 46N of the blood flow measurement unit 41 is reflected by the scalp BR3, while the remaining light passes through the scalp BR3, skull BR2, and cerebrospinal fluid BR1 and travels into the brain BR. The light irradiated onto the head and traveling into the brain propagates in the radial direction (depth direction and radial direction) along an arc-shaped pattern CP as shown by the dashed lines in the figure.

[0062] The propagation of light through the brain becomes longer the farther radially from the base point BP where the laser light is irradiated, and the lower the light transmittance. Therefore, the light reception level (amount of transmitted light) of sensor unit 46B, which is adjacent to sensor unit 46A on the light-emitting side at a predetermined distance, is detected as being strong. The light reception level (amount of transmitted light) of sensor unit 46C, which is located a predetermined distance next to sensor unit 46B, is detected as being weaker than the light reception level of sensor unit 46B. The light-receiving unit 48 of sensor unit 46A on the light-emitting side also receives light from the brain.

[0063] The first control unit 43 uses detection signals corresponding to the light intensities received by the plurality of sensor units 46A to 46N as measurement data, and derives the vascular characteristics of each measurement position by comparing the waveforms of the detection signals with the waveform of the cardiac potential signal from the electrocardiograph 42. Furthermore, by performing mapping processing on these detection results, it is possible to obtain graphic data showing the distribution of arteriosclerosis according to the pulse wave velocity.

[0064] Therefore, it is possible to measure changes in blood flow through the middle cerebral artery and anterior cerebral artery based on the detection signal waveforms of the sensor units 46A to 46N, and to detect the pulse wave velocity in the brain from the measurement data of the changes in blood flow.

[0065] Thus, the vital sign measuring device 20 is equipped with a brain activity measuring unit 40 that is attached to the subject's head and measures brain waves and cerebral blood flow in the measurement area of ​​the head, and the information detecting unit 25 also adds as nervous system data the state of communication in the central nervous system centered on the head based on the measurement data of brain waves and cerebral blood flow obtained from the brain activity measuring unit 40. In this way, the state of communication in the central nervous system from the subject's brain can also be quantitatively converted into data and acquired, making it possible to monitor the subject's progress and transitional state.

[0066] As a result, the state of transmission from the subject's brain to the central nervous system can be quantitatively converted into data and acquired, making it possible to monitor the subject's progress and transitional state.

[0067] (2-3) Method for detecting musculoskeletal system information The human and environment information input / output unit 10 has a biological signal measurement garment 50 worn by the subject and a joint circumference detection unit (not shown) as additional components of the vital sign measuring device 20. The joint circumference detection unit has an actuator that is driven actively or passively in conjunction with the limb movement of the subject, and detects physical quantities around the joints that accompany the limb movement of the subject based on the output signal from the actuator.

[0068] The physical quantities around the joints include the muscle forces generated in the musculoskeletal system, the range of motion of each joint, the speed of motion and reaction speed, the autonomous control characteristics against disturbances, the parameter identification results of at least one of the frame's moment of inertia, mass and center of gravity, the impedance adjustment results of the joint system including flexor and extensor muscles (viscous characteristics due to friction), electrical physical quantities (command signals), etc.

[0069] The biosignal measurement wearing device 50 worn by the subject is consistent with and uses the same principles as the device described in Japanese Patent Application Publication No. 5409637 filed by the present inventor. Fig. 7(A) shows the biosignal measurement wearing device 50 of this embodiment. This biosignal measurement wearing device 50 is formed to cover the subject's body surface and has a wearing device main body 50A to be worn by the subject. The inner surface of this wearing device main body 50A (the surface that comes into contact with the subject's body surface when worn) has a biosignal sensor 53 for detecting biopotential signals at at least one position where a biosignal can be measured from the subject's body.

[0070] Biosignals are signals generated by electricity within a subject's body. They are measurable signals that change over time in response to bodily movements and signals from the brain. For example, biosignals refer to signals generated by biological activity, such as nerve potentials, muscle potentials, electroencephalograms, cardiac potentials, and potentials generated by motion artifacts (movement effects), potentials generated by biochemical reactions, and vibrations such as pulse waves generated by heartbeats. The potentials (amplitude, density, time delay) and locations of biosignals are also taken into account when quantifying the data.

[0071] 7A, the biosignal measurement garment 50 of this embodiment has a bilaterally symmetrical structure, so the left half of the figure shows the structure of the back of the left leg, and the right half of the figure shows the structure of the front of the left leg. The garment main body 50A is provided with a group of biosignal sensors 54A to 54F consisting of multiple biosignal sensors 53 that detect the subject's biosignals. The biosignal sensors 53 are arranged at equal intervals along the flow of the subject's leg muscles.

[0072] The wearing device main body 50A has multiple biosignal sensors 53 arranged along the flow of the gluteus maximus in the area corresponding to the subject's buttocks (biosignal sensor group 54A). Similarly, the biosignal sensors 53 are arranged along the flow of the biceps femoris, semimembranosus, and semitendinosus in the area corresponding to the back of the subject's thighs (biosignal sensor group 54B). The biosignal sensors 53 are arranged along the flow of the triceps surae in the area corresponding to the calves (biosignal sensor group 54C). The biosignal sensors 53 are arranged along the flow of the adductor longus and iliopsoas in the area corresponding to the anterior hip joints (biosignal sensor group 54D). The biosignal sensors 53 are arranged along the flow of the quadriceps femoris in the area corresponding to the anterior thighs (biosignal sensor group 54E). The biosignal sensors 53 are arranged along the flow of the tibialis anterior, soleus, and extensor digitorum longus in the area corresponding to the shins (biosignal sensor group 54F).

[0073] 7(B) is a diagram showing an example of a measurement module connected to a group of biosignal sensors provided on the wearing device main body 50A. Note that all of the groups of biosignal sensors 54A to 54F provided on the wearing device main body 50A are connected to the measurement module 55 in the same way, so the description here will be made using the group of biosignal sensors 54A as an example.

[0074] 7(B) is a diagram showing a schematic diagram of a biosignal sensor group 54A provided on the wearing device main body 50A and a measurement module connected to this biosignal sensor group 54A. The biosignal sensors 53 constituting the biosignal sensor group 54A are provided insulated from each other and connected to the measurement module 55 via conductive wiring. Each of these biosignal sensors 53 is assigned an address.

[0075] The measurement module 55 is connected to the plurality of biosignal sensors 53 that make up the biosignal sensor group 54A, and includes a measurement module controller 56 that selects at least two of the biosignal sensors 53 and acquires biosignals by calculating the difference between the detection signals detected by the selected biosignal sensors 53, a memory 57 that records the acquired biosignals, and a communication unit 58 that transmits to the outside the sequentially acquired biosignals and / or the biosignals recorded in the memory 57. Note that a measurement module 55 is provided for each of the biosignal sensor groups 54A to 54F, and each biosignal sensor 53 is connected to the measurement module 55.

[0076] This measurement module controller 56 has an electronic circuit that can sequentially select at least two biosignal sensors 53 from the multiple connected biosignal sensors 53 in response to a command signal input via the communication unit 58, and acquire the biosignals between these two selected biosignal sensors 53.

[0077] The measurement module controller 56 further includes signal processing means such as a filter that removes or extracts predetermined frequency components from the acquired biological signal, and an amplifier that amplifies the acquired biological signal. The acquired biological signal is output from the measurement module controller 56 to the communication unit 58 and / or the memory 57. When selecting a biological signal sensor 53, the measurement module controller 56 may operate each biological signal sensor 53 sequentially in a predetermined order, or may select the biological signal sensor 53 at the address specified by a designation signal input via the communication unit 58.

[0078] The communication unit 58 includes a thin antenna and a communication circuit connected to the antenna, and transmits, via the antenna to the second control unit 59, measurement information including a signal containing the biosignal output from the measurement module controller 56 and information such as an address indicating the positional information of the biosignal sensor 53 that detected the biosignal, and / or a signal containing the biosignal read from the memory 57 and information such as an address indicating the positional information of the biosignal sensor 53 that detected the biosignal.

[0079] This second control unit 59 generates a designation signal that specifies which biosignal sensor 53 to select from the group of biosignal sensors 54A to 54F provided on the attachment main body 50A, as well as signals such as the start and end of data acquisition, and selects the designated biosignal sensor 53 to measure the biosignal in accordance with the signal.

[0080] As described above, according to the biosignal measurement garment 50 of the present invention, the garment main body 50A is provided with biosignal sensor groups 54A-54F each consisting of a plurality of biosignal sensors 53, so that simply by wearing the garment main body 50A, the plurality of biosignal sensors 53 can be placed at predetermined locations at once, and can be brought into close contact with the skin surface to detect biosignals at each point. In this way, even when monitoring biosignals at multiple points on the subject's body surface, the effort of attaching and removing the biosignal sensors 53 one by one can be eliminated, and biosignals can be easily measured.

[0081] Furthermore, by detecting biosignals using the plurality of biosignal sensors 53 in this manner, biosignals can be measured at a plurality of points within the area where the biosignal sensors 54A to 54F are arranged. Then, by mapping the measurement data at each point according to the address assigned to each biosignal sensor 53, it is possible to measure the distribution of biosignals in the subject's body.

[0082] The second control unit 59 acquires, from the biosignals acquired by the measurement module 55, myoelectric potential signals associated with the subject's muscle activity and nerve transmission signals for operating the subject's musculoskeletal system.

[0083] The information detection unit 25 converts the activity state of the musculoskeletal system centered on the joint and / or the transmission state of the peripheral nervous system into quantitative musculoskeletal system data and / or nervous system data based on the physical quantities around the joint described above and either or both of the myoelectric potential signal and the nerve transmission signal obtained from the second control unit 59, and also converts the transmission state of either or both of the central nervous system and the peripheral nervous system centered on the desired location into quantitative nervous system data.

[0084] As a result, by having the subject perform limb movements, the activity state of the musculoskeletal system and the transmission state of the peripheral nervous system can be quantitatively converted into data based on the physical quantities around the joints and the biological signals obtained from the body surface areas relative to the joints, and the subject's progress and transition state can be monitored.

[0085] In addition, biological signals can be obtained from body surface areas other than those based on the subject's joints, and quantitative data can be obtained not only on the peripheral nervous system around the joints, but also on the transmission state of the central nervous system and peripheral nervous system from desired areas, making it possible to monitor the subject's progress and transitional state.

[0086] For example, as shown in Figure 8, by disposing biosignal sensors 54A-54F all over the subject's body, it is possible to visualize the state transitions of the information transmission pathways in the cranial nervous system. In Figure 8, by dividing the subject's entire body into predetermined regions (e.g., regions innervated by sensory nerves such as dermatomes), it is possible to grasp the neural connections of the entire body from a bird's-eye view and for each region. Therefore, if it is possible to detect in which region a problem has occurred in neural transmission, it is possible to grasp not only the location of the problem but also the degree of impact on either the central or peripheral side.

[0087] (2-4) Method for detecting the activity state of the physiological system In the human / environment information input / output unit 10, as additional components of the vital sign measuring device 20, a near-infrared detection unit 60 and a far-infrared detection unit 61 are provided, as shown in FIG. 9, and the activity state of the physiological system and nervous system of the subject is detected using a non-contact method.

[0088] That is, the human / environment information input / output unit 10 uses the near-infrared detection unit 60 and the far-infrared detection unit 61 to irradiate infrared rays mainly onto the face of the subject, thereby simultaneously and non-invasively measuring fluctuations in the subject's pulse rate and skin temperature in daily life.

[0089] The near-infrared detector 60 irradiates a region of interest on the face of the subject, including the cheeks, with near-infrared light and receives light reflected from the face to generate a near-infrared image. The far-infrared detector 61 irradiates a region of interest on the face of the subject with far-infrared light and detects the skin temperature of the face.

[0090] The third control unit 62 estimates the pulse rate of the subject based on the period of change in the intensity waveform in the region of interest of the reflected light received by the near-infrared detection unit 60. The third control unit 62 extracts areas of the subject's face that are susceptible to and less susceptible to the activity of the autonomic nervous system from the near-infrared image generated by the near-infrared detection unit 60, and calculates the temperature difference in the skin temperature of each area from the detection result of the far-infrared detection unit 61.

[0091] A. Pulse measurement method based on infrared light The light-absorbing substances in biological tissue are water and hemoglobin in the blood. Water has strong absorption properties for infrared light with wavelengths longer than 1350 nm, while hemoglobin has strong absorption properties for visible light with wavelengths shorter than 650 nm. Non-invasive biomedical diagnosis using light often involves irradiating the body with light in the infrared to near-infrared range with wavelengths between 650 nm and 1350 nm, which has high biological penetration, and measuring the reflected and transmitted light, which contains biological information.

[0092] Pulse is a physiological phenomenon in which the volume of blood vessels changes due to the work of blood pressure caused by cardiac output. Changes in blood vessel volume are brought about by changes in blood flow. Changes in blood flow change the amount of light absorbed by hemoglobin in the blood, so the pulse rate can be estimated from the periodic changes in transmitted and reflected light when light is irradiated onto a living body.

[0093] In this invention, the subject is not required to wear a measuring device and measurements can be performed day or night. Therefore, the pulse rate is estimated by measuring the periodic change in reflectance of near-infrared light irradiated onto the living body using a near-infrared camera. Since near-infrared light is invisible, measurements can be made without burdening the subject, even at night. In order to estimate stress state from the pulse rate, it is necessary to measure the pulse rate, which fluctuates minutely due to activity of the autonomic nervous system.

[0094] The third control unit 62 implements an algorithm for extracting the waveform period required for calculating the pulse rate from the waveform data obtained from the near-infrared detection unit 60 .

[0095] The near-infrared detector 60 measures the intensity of the light reflected by biological tissue when the irradiated near-infrared light is reflected. This near-infrared detector 60 measures the subject at rest, but slight body movements and facial movements due to breathing are superimposed on the pulse waveform as motion artifacts that represent changes in the amount of reflected light.

[0096] To remove these motion artifacts, a digital filter is first applied to the pulse rate frequency band. The resting pulse rate range is set to 35 to 180 bpm, and motion artifacts may also be superimposed on the measured waveform in the 0.5 to 3.0 Hz band, but the amplitude of the waveform superimposed on the pulse waveform is significantly large.

[0097] Therefore, if the variance value of the waveform data within a certain period of time exceeds a certain threshold, the third control unit 62 recognizes that the subject is moving and removes the data during that period as unreliable. The waveform after the bandpass filter has a bimodal feature due to arterial blood flow. To facilitate feature point extraction, a moving average filter is applied to reduce the two-peak waveform to a single peak.

[0098] 10A and 10B show the waveforms before and after applying a band-pass filter and a moving average filter to the measured waveform, respectively. It can be seen that the motion artifacts superimposed on the pulse waveform have been removed by filtering.

[0099] Next, generally, there are two known methods for calculating the pulse rate from the light intensity waveform measured by a near-infrared camera.

[0100] In the first method, the cheek area of ​​the face was set as the region of interest (ROI), and the intensity data of the reflected light from the cheek area for 30 seconds obtained by measurement with an RGB camera was used to obtain a frequency spectrum using a spectrum analyzer. The frequency that is thought to be the pulse wave component was then compared with the frequency determined from the pulse rate indicated by a pulse oximeter. However, the noise caused by body movement is significantly large in the pulse waveform. When attempting to calculate the pulse rate using this type of frequency analysis, it is necessary to significantly restrict the movement of the person being measured.

[0101] On the other hand, the second method calculates the time required for one pulse beat from the period of the waveform feature points detected from the time-series data of the measured waveform, and then calculates the pulse rate by dividing that time by 60. Compared to methods that calculate the pulse rate from frequency analysis, which requires continuous measurement data for a certain period of time, methods that calculate the pulse rate from the period of the measured feature points can remove motion artifacts by removing feature point periods that are falsely detected due to noise caused by body movement. In addition, the minimum measurement time required is shorter than that required by frequency analysis, and the pulse rate can be calculated from several beats, making it possible to calculate the pulse rate over shorter intervals.

[0102] However, since the pulse rate of a healthy person at rest is 60 to 80 bpm, the period of the characteristic points obtained in a 5-second measurement is 5 to 7 times, which is a problem in that the amount of data obtained is insufficient to calculate a more accurate pulse rate. Also, as shown in Figure 10(A), the waveform of the intensity change has two peaks synchronized with the arterial blood flow, which can cause erroneous detection of the period of the characteristic points, resulting in an error in the calculated pulse rate.

[0103] For daily measurements, the body part to be measured needs to be exposed day and night. Therefore, in the vital sign measuring device 20 according to the present invention, the ROI is set on the cheek of a person. The location of the set ROI is shown in Figure 11 (A). The average of the intensity values ​​measured at the pixels within the set ROI was taken as the measurement value, and the time change of the measurement value was recorded as pulse waveform data. The period of the characteristic points of the waveform data was determined by the interval t between the top ends of the waveform. tp Not only that, but also the bottom interval t bp is added as another feature point, the number of feature points that can be used as reference data for calculating the pulse rate is increased.

[0104] Next, in order to extract the characteristic point period necessary for calculating the pulse rate, the detected t tp , t bp Among them, there are falsely detected t tp , t bp The algorithm for removing S is shown below in (a) to (f). ０ Let's say.

[0105] (a) ttp , t bp For each of the peaks, a certain peak interval t i is the previous peak interval t i-1 If there is a fluctuation of 30% or more, the peak interval t i Set S ０ The set of selected peak intervals S １ is expressed by the following equation (2).

[0106] (b) Set S ０ Standard deviation σ ０ Calculate 2σ ０ The peak interval t varies in the range i Set S １ The set of selected peak intervals S ２ is expressed by the following equation (3).

[0107] (c) Set S ２ Standard deviation σ ２ Calculate and σ ２ The peak interval t varies in the range i Set S ２ The set of selected peak intervals S ３ is expressed by the following equation (4).

[0108] (d) Set S ３ Standard deviation σ ３ Calculate and σ ３ The peak interval t varies in the range i S ３ The set of selected peak intervals S ４ is expressed by the following equation (5).

[0109] (e) Set S ４ The remaining peak interval blood pressure is converted to pulse rate, and a histogram is created with pulse rate intervals of 5 bpm. In this case, the pulse rate width for each section of the histogram is the same, but the width of the peak intervals corresponding to that pulse rate varies. Therefore, a weighted histogram is created using the following equation (6).

[0110] Here, t 30-35 is the time interval between peaks of pulse rate 30 [bpm] and 35 [bpm], t i-i+5 is the time interval between peaks of pulse rate i [bpm] to i+5 [bpm], h i-i+5 is the frequency of the peak interval between pulse rates i [bpm] and i+5 [bpm] before correction, and h' i-i+5 represents the correction value of the frequency of the peak intervals from pulse rate i [bpm] to i+5 [bpm].

[0111] (f) Regarding the values ​​of the weighted histogram, a window of width 20 [bpm] is set so that the sum of the frequencies of the four adjacent sections is maximized, and the peak interval t outside that range is i Remove.

[0112] According to the above algorithm, the average value of the extracted peak intervals is set as the peak width t within the measurement time, and the pulse rate is calculated by dividing 60 seconds by the peak interval t.

[0113] In this way, the third control unit 62 sets feature points at the upper and lower ends of the intensity waveform of the reflected light obtained from the near-infrared detection unit 60, and estimates the pulse rate based on the peak intervals between the upper and lower ends, which are the periods of the feature points. The third control unit 62 also calculates the standard deviation of the multiple peak intervals and removes peak intervals that do not represent a pulse period based on the standard deviation. The third control unit 62 then creates a weighted histogram for a predetermined time unit from the pulse rate converted based on the remaining peak intervals after the removal, and corrects the frequency of the peak intervals based on the histogram.

[0114] Thus, the third control unit 62 can calculate an accurate pulse rate from the measured waveform of pulse waveform data for a relatively short period based on the pixel group within the set ROI.

[0115] B. Skin temperature measurement using far-infrared detector Heat generated inside the body is carried to the body surface by conduction and convection. However, since biological tissue itself has poor thermal conductivity and acts as a heat insulator, most of the heat transport to the skin is thought to be due to skin blood flow. Skin blood flow changes depending on the activity of the autonomic nervous system, which is centered on the contraction and expansion of blood vessels by the sympathetic and parasympathetic nervous systems.

[0116] Previous studies have attempted to estimate the activity of the autonomic nervous system by measuring changes in skin temperature. However, because the heat distribution on the face is significantly affected by the subject's hairstyle and whether or not they are wearing glasses, the human and environment information input / output unit of the present invention uses near-infrared images to set the measurement site appropriate for estimating the activity of the autonomic nervous system.

[0117] In order to estimate stress levels from changes in skin temperature, it is necessary to set the ROI for measurement to a region of the face where temperature changes due to stress are most noticeable. Among the facial regions, the nose in particular is home to a high concentration of arteriovenous anastomoses (AVAs), which are susceptible to the activity of the autonomic nervous system, making it suitable for measuring skin temperature to estimate stress levels.

[0118] Furthermore, because skin temperature is affected by the ambient temperature, changes in skin temperature must be recorded as relative temperatures. In this case, the ROI used to measure the nasal skin temperature change as a reference must be set in a region that is less susceptible to the activity of the autonomic nervous system. The forehead is an example of a region of the face where AVAs are less dense and less susceptible to the activity of the autonomic nervous system.

[0119] Based on the above, the third control unit 62 according to the present invention records the change in skin temperature as the temperature difference between the region of interest ROI_n in the nose and the region of interest ROI_fh in the forehead. The locations of the ROIs are shown in FIG. 11(B). The equation for calculating the temperature change is shown in Equation (7).

[0120] Here, T n is the average skin temperature measured at all pixels in ROI_n, T fh is the average skin temperature measured at all pixels in ROI_fh, T r is the relative temperature of the forehead and nose. When setting each ROI, the coordinates of the nose and forehead in the near-infrared image are converted to coordinates on the far-infrared image.

[0121] Specifically, when converting coordinates from a near-infrared (NIR) image to a far-infrared (FIR) image, the resolution and angle of view differ between near-infrared and far-infrared cameras, so the coordinate information of the near-infrared image is made to correspond to the pixels of the far-infrared image.

[0122] That is, the x-coordinate on the far-infrared image is X fir is the horizontal angle of view of the near-infrared camera, θ nir_h , the horizontal angle of view of the far-infrared camera is θ fir_h Then, it is expressed as the following equation (8).

[0123] The y coordinate on the far-infrared image, Yfir, is the vertical angle of view of the near-infrared camera, θ nir_v , the vertical angle of view of the far-infrared camera is θ fir_v Then, it is expressed as the following equation (9): where d represents the difference between the optical axes, and L represents the distance to the object.

[0124] Thus, the third control unit 62 can accurately calculate the relative temperature difference between the skin temperatures of the subject's forehead and nose based on the image information from the far-infrared detection unit 61, following the automatic detection result of the measurement target area using the image information from the near-infrared detection unit 60.

[0125] C. Respiration measurement based on skin temperature Respiration is related to the function of the autonomic nervous system and the immune system, and is used to diagnose sleep disorders. There are two types of breathing: nasal breathing (Figure 12 (A)) and mouth breathing (Figure 12 (B)), with nasal breathing being the preferred form of breathing. When breathing through the mouth, the inhaled air does not pass through the cilia and mucous membranes of the nasal cavity. Mouth breathing can also cause various diseases, including sleep apnea syndrome (SAS).

[0126] As shown in Figures 13(A) and (B), the nostrils and oral cavity are warmed by exhaled air (35-36°C) and cooled by inhaled air (23-28°C). Therefore, regions of interest (ROIs) are set at the subject's nostrils and oral cavity, and the temperature changes at the nostrils and oral cavity due to breathing are measured using a far-infrared detector (far-infrared camera). Because the surface temperature of the skin is affected by the ambient air temperature, it is measured as the relative temperature between the nostrils and oral cavity.

[0127] In practice, because it is difficult to automatically detect the measurement target area using a skin temperature distribution map, the measurement target areas (nostrils and oral cavity) are automatically detected using image information from the near-infrared detector (camera) as described above, and the region of interest (ROI) is set. The nostril temperature is relatively high during nasal breathing, while the oral cavity temperature is relatively low during mouth breathing.

[0128] We measured the subject's nasal breathing and mouth breathing for 30 seconds each, then took a one-minute break and repeated the same respiration measurement four times. As a result, as shown in Figure 14(A), we confirmed that the temperature of the nostrils decreased due to inhalation during nasal breathing. Furthermore, as shown in Figure 14(B), we confirmed that the temperature of the oral cavity also decreased due to mouth breathing.

[0129] Furthermore, as shown in Figure 15(A), the opening and closing of the mouth due to a change in breathing method from nose breathing to mouth breathing was also confirmed based on the distance between the upper and lower lips. When the graphs in Figures 14(A), (B), and 15(A) are superimposed, it is shown as in Figure 15(B).

[0130] The method for extracting the timing of inhalation is to first use a moving average filter to remove high frequency noise from the measurement results of the relative temperatures of the nostrils and oral cavity, and then perform first differentiation to extract the timing of the temperature drop due to exhalation.

[0131] Furthermore, if the nostril temperature and oral temperature drop at the same time, the intra-alveolar pressure becomes negative relative to atmospheric pressure, causing air to flow into the nasal passages, making it possible to distinguish this as mouth breathing.

[0132] In this way, it is possible to calculate the respiration rate and distinguish the respiration method using a non-contact measurement method. It is possible to measure nasal breathing and mouth breathing simultaneously, and by executing an algorithm that takes into account the characteristics of temperature changes during each breath, it is possible to accurately calculate the respiration rate and distinguish the respiration method.

[0133] D. Oxygen saturation using two types of near-infrared wavelengths A pulse oximeter is a medical device that uses the light absorption characteristics of hemoglobin in the blood. A pulse oximeter is attached to the fingertip and measures blood oxygen saturation and pulse rate.

[0134] In this invention, we focused on the difference in absorption characteristics between oxygenated hemoglobin (HbO2) and reduced hemoglobin (Hb), since blood oxygen saturation varies depending on respiratory rate, the degree of hemoglobin-oxygen binding, and cardiac output.

[0135] In conventional measurement methods, infrared light (wavelength around 660 [nm]) and near-infrared light (wavelength around 900 [nm]) are irradiated onto the fingertip from a light-emitting element, and the blood oxygen saturation is estimated from the ratio of the light intensity of each transmitted light measured by a light-receiving element, while the pulse rate is also estimated using the periodic change in the light intensity of the transmitted light.Infrared light, which is visible light, has been used because there is a large difference in the light absorption coefficient between oxygenated hemoglobin (HbO2) and reduced hemoglobin (Hb).

[0136] However, in the present invention, non-contact measurement is performed using only near-infrared (NIR) light, which is invisible light (FIG. 16). As a result, by alternately turning on and off light-emitting elements that emit two types of near-infrared light, one with a wavelength of 800 (or 760) [nm] and the other with a wavelength of 900 [nm], and photographing the region of interest (ROI) irradiated by each light-emitting element with a single near-infrared camera (near-infrared detection unit 60), it is possible to measure the blood oxygen saturation of the subject centered on the ROI without contact and even in a dark environment.

[0137] (2-5) Method for detecting the activity state of the nervous system The autonomic nervous system includes the sympathetic nervous system, which functions when the living body is in a state of tension or activity, and the parasympathetic nervous system, which functions when the living body is at rest. When the sympathetic nervous system is dominant, blood pressure and pulse rate increase, while when the parasympathetic nervous system is dominant, blood pressure and pulse rate decrease, so there is a high correlation between autonomic nervous function and cardiac function.

[0138] As a method for measuring autonomic nervous function, there is a method for measuring cardiac sympathetic nervous function using heart rate fluctuations. Specifically, there is the CVR-R (Coefficient of Variation of R-R intervals) method, which uses the R-R intervals of an electrocardiogram waveform to calculate and evaluate the coefficient of variation. Secondly, there is a method in which the fluctuation power ratio of the frequency components (high frequency components and low frequency components) of heart rate fluctuations is used as an index of sympathetic nervous activity.

[0139] There is also a method for measuring autonomic nervous function by measuring the function of the sympathetic nerves in the vascular system using pulse waves, particularly a method for calculating the magnitude of amplitude fluctuation in the photoplethysmogram (PPG) waveform as an evaluation value of autonomic nervous function.

[0140] In the present invention, the third control unit 62 can contribute to the evaluation of the subject's autonomic nervous function (presence or absence of disturbance, etc.) based on all or a combination of the subject's pulse rate detected using the near-infrared detection unit 60 and far-infrared detection unit 61 that constitute the human / environment information input / output unit 10, the relative temperature difference between the skin temperatures of the subject's forehead and nose, the subject's respiratory rate and breathing method, and the subject's oxygen saturation.

[0141] (2-6) Method for Detecting Psychological Information In the person and environment information input / output unit 10, the information detection unit 25 detects psychological information by analyzing the facial expression and voice of the subject using the face imaging camera 21 and the sound collecting microphone 22.

[0142] Specifically, the information detection unit 25 executes a face analysis algorithm to detect the facial expressions of the subject (including not only expressions such as sadness, joy, anger, empathy, and frivolity, but also flushed skin, goosebumps, frequency of eyelid opening and closing, tears, etc.), and at the same time executes a voice analysis algorithm to detect the tone and pitch of the subject's voice (tone of voice, elation, etc.).

[0143] This facial analysis algorithm is designed to approximate human interpretation (experience, preconceptions, and expectations) and is derived from the accumulated results of how emotions and psychology are reflected in facial expressions based on the state and changes of the human face.

[0144] Similarly, voice analysis algorithms are designed to approximate human interpretation (experience, biases, and expectations) and are derived from the accumulated results of how emotions and psychology are reflected in the voice based on human tone and pitch.

[0145] Using this voice analysis algorithm, it is possible to estimate changes in the subject's psychological state through the subject's everyday communication and narrative analysis, based on the keywords used in the subject's speech, the intonation of the speech, the volume of the voice, the structure of the narrative, the continuity / discontinuity of communication, and other accumulated data.

[0146] In addition, the information detection unit 25 can also estimate the psychological state of the subject based on the subject's personal information (heart rate, electrocardiogram, pulse, blood pressure, body temperature, skin blood flow distribution, skin temperature distribution, sweating, odorous substances, etc.) detected by the vital sign measuring device 20 described above, and the subject's environmental information detected by the peripheral measuring device 24 described below.

[0147] In this way, the information detection unit 25 can comprehensively estimate and grasp a person's psychological information from both the person information obtained by sensing and the person information obtained by communication analysis and narrative analysis.

[0148] (2-7) Method for detecting the activity state of the motor system (movement information and behavior information) In the human / environment information input / output unit 10, the information detection unit 25 detects the movement data recognized by the motion capture device 23 as the activity state of the subject's motor system.

[0149] 17, the motion capture device 23 is placed in an indoor facility such as the subject's home, and is composed of a control unit 70 that controls the entire device, and an RGB-D sensor 71 and an IMU (Inertial Measurement Unit) sensor 72 that are connected to the control unit 70. The IMU sensor 72 is composed of a triaxial acceleration sensor, a triaxial angular velocity sensor, and a geomagnetic sensor, and is capable of measuring triaxial acceleration, angular velocity, and geomagnetism.

[0150] In addition to its RGB color camera function, the RGB-D sensor 71 also has a depth sensor that can measure the distance to an object as seen from the camera, enabling it to perform 3D scanning of the object. If, for example, the RealSense (a trademark of Microsoft Corporation) LiDAR camera L515 is used as the RGB-D sensor 71, the depth sensor is made up of a LiDAR sensor that measures the time it takes for laser light to irradiate the object, hit it, and bounce back, thereby measuring the distance and direction to the object.

[0151] In practice, the RGB-D sensor 71 is positioned so as to capture an image of the subject's entire body based on the subject's standing position in the indoor equipment, and sequentially acquires an RGB image and a depth image as the imaging results.

[0152] The control unit 70 has a fourth control unit 73 consisting of a CPU (Central Processing Unit) that controls the entire system, and a data storage unit 74 in which various data is stored in a database that can be read and written in response to instructions from the fourth control unit 73.

[0153] The fourth control unit 73 is composed of an image coordinate setting unit 80, a coordinate group extraction unit 81, a skeletal information acquisition unit 82, and a behavior recognition unit 83. The image coordinate setting unit 80 estimates the posture of the entire body of the subject based on the RGB images sequentially acquired from the RGB-D sensor 71, and sets a plurality of skeletal representative points attached to the entire body of the subject as image coordinates.

[0154] Specifically, the image coordinate setting unit 80 sets a group of image coordinates in a two-dimensional coordinate system in which the coordinate origin is either the left or right shoulder of the subject, the line connecting the left and right shoulders is the X-axis, and the direction perpendicular to the X-axis is the Z-axis, based on the coordinate origin. That is, the coordinate origin is moved to the right or left shoulder, and the XZ plane is rotated so that it coincides with the line connecting the left and right shoulders. As a result, it becomes possible to recognize movements with increased robustness regardless of the position of the RGB-D sensor 71.

[0155] The coordinate group extraction unit 81 sequentially extracts image coordinate groups related to the subject's actions based on the transition of the subject's posture from the image coordinate groups set by the image coordinate setting unit 80. In practice, the coordinate group extraction unit 81 needs to extract key points of the entire body from the actions, taking into account changes in the subject's clothing and surrounding environment.

[0156] Therefore, the coordinate group extraction unit 81 employs a lightweight OpenPose architecture (a system that estimates a person's skeleton using deep learning) that uses a trained model to estimate the posture of the entire body and obtain image coordinates of the skeletal representative points of the fingers of both hands. For example, for the subject's upper body, the coordinate group extraction unit 81 sequentially extracts a total of 60 skeletal representative points as image coordinate groups: 18 points on the torso of the subject's upper body and 42 points on the fingers of both hands (21 points on each finger). Furthermore, by using only stages 1 and 2 of the six stages of the OpenPose architecture, the amount of calculation can be reduced by 32.4%.

[0157] The skeletal information acquisition unit 82 acquires three-dimensional skeletal information that is continuous over time, centered on the fingers of both hands of the subject, by temporally synchronizing the image coordinate groups sequentially extracted by the coordinate group extraction unit 81 with the depth images sequentially acquired from the RGB-D sensor 71.

[0158] Specifically, the skeletal information acquisition unit 82 constructs a three-dimensional coordinate system by combining the depth direction in the depth images sequentially acquired from the RGB-D sensor 71 as the Y-axis with the two-dimensional coordinate system of the image coordinate groups sequentially extracted by the coordinate group extraction unit 81, and acquires three-dimensional skeletal information.

[0159] The transformation equation from the image coordinate system, which is a two-dimensional coordinate system, to the world coordinate system, which is a three-dimensional coordinate system, is expressed as the following equation (10).

[0160] Here, X Ｗ = ｔ [X Ｗ, Y Ｗ, Z Ｗ ] indicates a coordinate in the world coordinate system, and x Ｉ = ｔ [x Ｉ ,y Ｉ] indicates coordinates in the image coordinate system. K (3 × 3 matrix) is an intrinsic parameter of the image sensor, R (3 × 3 matrix) is a rotation matrix that performs coordinate transformation to the image coordinate system, t (3 × 1 vector) is a translation vector, Z Ｃ represents depth information corresponding to two-dimensional coordinates in the image coordinate system.

[0161] K, R, and t were calculated by camera calibration using Zhang's method ("A flexible new technique for camera calibration" IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000.). For the calibration to calculate K, a calibration board with circular dots printed in a grid pattern was used. The central coordinates of each dot on the calibration board were calibrated to 1 / 1,000.

[0162] As a result, the motion capture device 23 clarifies the relationship between the position in three-dimensional space and the position of the captured image for the entire body of the subject, as shown in Figure 18, and is able to acquire three-dimensional skeletal information while maintaining high robustness regardless of the imaging position taken by the RGB-D sensor 71.

[0163] The behavior recognition unit 83 recognizes behavior, which is a connection of multiple local movements, from the transition state of the subject's posture based on the three-dimensional skeletal information sequentially acquired by the skeletal information acquisition unit 82.

[0164] The action recognition unit 83 uses the action recognition pattern set for each local action as training data, and while referring to the hand-washing action recognition model constructed by deep learning, sequentially recognizes the corresponding local action from the three-dimensional skeletal information sequentially acquired by the skeletal information acquisition unit 82.

[0165] Specifically, in order to recognize multiple local actions, the action recognition unit 83 applies an action recognition model consisting of three modules, namely, a convolutional neural network (CNN) layer, a batch normalization layer (Batch Norm), and an activation function layer (tanh function), as well as a fully connected layer, as shown in FIG. 19 .

[0166] In the behavior recognition model, it is possible to input skeletal information from the current frame and the frame two frames before it and calculate the likelihood of each of multiple types of local motions related to behavior.

[0167] Furthermore, the behavior recognition unit 83 performs post-processing on the most frequently occurring behavior among the behaviors observed in 15 frames (the current frame and the 14 frames preceding it) as the current local behavior as a recognition result. In this way, the behavior recognition unit 83 trained the behavior recognition model through supervised learning using a unique dataset. For optimization, a cross-entropy loss function and Adam (Adaptive Movement Estimation) with a learning rate of 0.001 were used.

[0168] As a result, the motion capture device 23 can significantly improve the accuracy and speed of recognizing local movements based on three-dimensional skeletal information.

[0169] (2-8) Method for detecting environmental information In the person / environment information input / output unit 10, the peripheral measuring device 24 includes a thermometer, a hygrometer, a barometer, an audible and ultrasonic ambient sound collector, etc., and acquires environmental information around the subject. This environmental information mainly includes the temperature, humidity, and ambient sound around the subject.

[0170] (2-9) Gaze Detection Method In the human and environmental information input / output unit 10, the subject wears a head-mounted display (not shown) and experiences his / her own human activities constructed in the information space by the virtual activity construction unit as video and audio, and also experiences the output operation of the output device 4 transcribed by the activity transcription unit described below as video and audio.

[0171] This head-mounted display is equipped with a monitor for displaying images to the wearer and a speaker for audio recognition. The housing of this head-mounted display is made of a transparent, light-transmitting material so that there is no interference with the detection of the subject's facial expression, facial surface temperature, etc. during imaging by the facial imaging camera 21 or non-contact detection by the vital sign measuring device 20.

[0172] The head-mounted display also captures an image of the space corresponding to the subject's field of vision, performs image recognition of the movement of the subject's eyeballs within the imaging range, detects the subject's line of sight, and detects the position of the subject's line of sight within the imaging range using the spatial imaging unit.

[0173] The configuration and method for this gaze detection are not particularly limited, but in addition to the imaging method, a method using infrared difference images, a gaze identification method using a glasses-like sensor equipped with a half mirror, etc. may also be applied. For example, the method using infrared difference images utilizes the fact that a strong reflected image is obtained from the eye when a light source is placed on the line of sight, and is a method in which two infrared rays with different wavelengths are irradiated onto the subject's eyes, a reflected image of the iris is extracted from the difference between the reflected images, and the subject's gaze is identified from this reflected image.

[0174] (3) Method of Reflecting on Output Device (3-1) Reflecting on Output Device In the space sharing system 1 of the present invention, when a subject connects to an external output device 4 via the cloud 3, the subject's human activities in the real space that is virtually constructed in real time in the information space are reflected as much as possible on the output device 4 via the second interface device 5.

[0175] 20 shows the configuration of the second interface device 5. This second interface device 5 has an activity transcription unit 90 and a part (sensation estimation unit 101) of the activity state transmission unit 100 described later. Fig. 20 also shows the configuration of the first interface device 2, which includes the other parts of the activity state transmission unit 100 (sensory feedback generation unit 102 and sensation transmission unit 103).

[0176] When the activity transcription unit 90 receives a request to connect with the subject from an external output device 4, it transcribes all or part of the subject's human activity constructed in the information space by the virtual activity construction unit 11 (Figure 2) in the first interface device 2 according to the output format of the output device 4.

[0177] In this case, the activity transcription unit 90 determines the consistency and differences between all of the subject's human activities in real space and the operations that the output device 4 can perform, based on the output form and function of the output device 4, and then transcribes all or part of the subject's human activities to the output device 4 within the constraints and range of the output operations according to the determination result.

[0178] At the same time, the activity transcription unit 90 reflects the personal characteristic information specific to the subject, created by the information creation unit 12, in the output operation of the output device 4.

[0179] In this way, in the space sharing system 1, when a subject connects to an external output device 4, all or part of the subject's human activity is transcribed according to the output format of the output device 4, making it possible to reflect the subject's unique human characteristic information in the output operation by the output device 4.

[0180] 21(A) to 21(F) show specific examples of output forms of the output device 4. The output forms of the output device 4 are mainly assumed to be support activities for people and collaborative activities with people. 21(A) and 21(B) show cases in which the output device 4 as an avatar of a subject performs work. Examples include tomato harvesting in the agricultural field ( FIG. 21(A) ) and material transportation work in the construction field ( FIG. 21(B) ).

[0181] 21(C) and 21(D) , the output device 4 as an avatar of the subject is assumed to perform a proxy action at a remote site, such as visiting a patient hospitalized in a hospital ( FIG. 21(C) ) or babysitting for a mother suffering from hemiplegia ( FIG. 21(D) ).

[0182] 21(E) and (F) assume cases in which remote medical care is performed using the output device 4. For example, there are cases in which a subject who is a doctor performs surgery on a patient waiting in an operating room at a hospital in a remote location (FIG. 21(E)), and cases in which a subject who is a patient receives a diagnosis from a doctor at a hospital in a remote location (FIG. 21(F)).

[0183] 20, the activity state transmission unit 100 transmits feedback of the operating state of the output device 4 to the subject in real space in real time via the information space. The activity state transmission unit 100 is composed of a sensation estimation unit 101 in the second interface device 5, and a sensation feedback generation unit 102 and a sensation transmission unit 103 in the first interface device 2.

[0184] The sensation estimation unit 101 estimates the physical sensation and psychological sensation of a person from the action result related to the operation content of the output device 4 and the sensory result of the environmental state in which the output device 4 exists when the output device 4 is operating.

[0185] (3-2-1) Estimation of Human Physical Senses The sense estimation unit 101 is configured to detect the tactile sense, which is the result of the action of the output device 4, among the human physical senses, and in particular, is configured to apply a tactile sense detection means that can accurately measure the magnitude of a load applied from the outside, regardless of the area on which the load acts.

[0186] As shown in FIG. 22A, the sensation estimation unit 101 serving as the tactile sense detection means includes a load sensor 110, a capacitance detection unit 111, a distributed load measurement unit 112, and a load information calculation unit 113.

[0187] As shown in Figures 22(B) to (D), the load sensor 110 is of a capacitance type, and forms multiple capacitors by inserting cylinders 116 of a predetermined shape made of a viscoelastic body in the same number as the number of positive electrodes between a first electrode plate 114 on which multiple positive electrodes are arranged in an array on the same plane and a second electrode plate 115 on which a single negative electrode is arranged.

[0188] Specifically, the load sensor 110 is entirely covered with a shielding material made of a non-conductive material. As a result, it is possible to prevent the formation of a capacitor between the sensor itself and other dielectrics and to eliminate external noise. The capacitance detection unit 111 detects the capacitance of each capacitor, which changes in response to an external force applied to the second electrode plate 115 of the load sensor 110.

[0189] Specifically, the capacitance detection unit 111 calibrates and corrects the measurement error of the load applied to the corresponding cylinder 116 according to the material properties of the cylinder 116 and the response delay according to the hardness of the cylinder 116 based on the increase in capacitance from the time the load is applied until a predetermined time has elapsed for each capacitor.

[0190] As a result, the cylinder 116, which is a viscoelastic body, experiences a response delay due to the gradual viscous deformation that occurs after instantaneous elastic deformation, so the response delay increases in proportion to the low hardness.In addition, the lower the hardness, the greater the amount of expansion and contraction of the cylinder 116 in response to load.As the distance between the electrode plates becomes closer, the capacitance, which is inversely proportional to the distance between the electrode plates, increases, so the increase in capacitance increases in proportion to the low hardness.However, factors based on such material properties can be eliminated.

[0191] The distributed load measuring unit 112 measures a distributed load that indicates the distribution of the load applied to each cylinder 116, based on the amount of change in capacitance of each capacitor detected by the capacitance detecting unit 111. The load information calculating unit 113 calculates the total load of the external force applied to the second electrode plate 115 of the load sensor 110 and the position of the load center, based on the relationship between the amount of expansion and contraction of each cylinder 116 and the pattern of the distributed load measured by the distributed load measuring unit 112.

[0192] Specifically, the load information calculation unit 113 calculates an error compensation amount for a non-uniform load in which the load center position is outside the center of the second electrode plate 115 based on the total load of external forces applied to the second electrode plate 115 of the load sensor 110, the ratio of the distributed load in the x-axis direction with the first electrode plate 114 as a plane, and the ratio of the distributed load in the y-axis direction with the first electrode plate 114 as a plane, and calibrates and corrects the load center position based on the error compensation amount.

[0193] As a result, even for uneven loads in which the load center position is outside the center of the second electrode plate 115, it is possible to calculate the error compensation amount through simulation and calibrate the load center position at the same time as correcting the size error of each first electrode plate 114.

[0194] In this way, the sensation estimation unit 101 can accurately measure the magnitude of the load applied from outside, regardless of the area of ​​the load acting on the second electrode plate 115, because the distributed load applied to each cylinder 116 is uniquely determined depending on the total load of the external force applied to the second electrode plate 115 of the load sensor 110 and the position of the load center.

[0195] (3-2-2) Estimation of human psychological sensations The sensation estimation unit 101 applies, as an effect related to the operation content of the output device 4, keywords used in speech, intonation when speaking, loudness of voice, narrative structure, continuity / discontinuity of communication, etc., through the subject's daily communication and analysis thereof and narrative analysis, as well as changes in psychological state estimated based on differences with these accumulated data, and also applies, as a result of sensing the environmental state where the output device is located, not only the ambient environment of the output device (temperature, humidity, air pressure, ambient sounds, etc.), but also personal information of the subject acquired by sensing technology (heart rate, electrocardiogram, pulse rate, blood pressure, body temperature, skin blood flow distribution, skin temperature distribution, sweating, facial expressions (facial expression recognition), odor substances, etc.) as reference objects.

[0196] The sensation estimation unit 101 estimates a change in the human psychological state as a psychological sensation based on the effect related to the operation content of the output device 4 and the sensory results of the environmental state in which the output device is located. That is, the sensation estimation unit 101 can comprehensively estimate and grasp human psychological information from both human information obtained by communication analysis and narrative analysis and human information obtained by sensing.

[0197] Next, the sensory feedback generation unit 102 generates, as feedback information, the psychological sensation estimated by the sensation estimation unit 101. The sensation transfer unit 103 transfers the feedback information generated by the sensory feedback generation unit 102 to the subject in real space.

[0198] As a result, in the space sharing system 1, when a subject operates using the output device 4, the subject can share human psychological sensations through the output device 4 from interactions related to the subject's activity.

[0199] (4) Method for generating an autonomous avatar In addition to the above configuration, in the space-sharing system 1 according to the present invention, when a substitute different from the subject in real space acts in place of the subject, an entity similar to the subject in real space is represented as an autonomous avatar in the information space using an estimated model construction unit 120 and an autonomous avatar generation unit 121 provided in the second interface device 5, as shown in Figure 23, in which the same symbols are used for parts corresponding to those in Figure 20.

[0200] The estimation model construction unit 120 constructs an estimation model that has been trained to estimate functions with characteristics unique to the subject by statistically analyzing the correlation between the subject's activity results transcribed to the output device 4 and the subject's personal and environmental information.

[0201] As a premise, when a subject performs a predetermined behavioral task, the estimation model construction unit 120 divides the behavioral task in chronological order into action phases corresponding to the correlation of the subject's activity state based on person and environment information. That is, the subject performs the behavioral task along the time axis, using basic action phases (action patterns having spatial information and temporal information) necessary for performing the predetermined behavioral task as unit criteria.

[0202] For example, when performing care work as a behavioral task, the action phases include bathing assistance, meal assistance, toilet assistance, mobility assistance, sleep assistance, and recreation. Input assistance requires not only keeping the care recipient's body clean, but also actions such as improving metabolism and loosening muscles. Meal assistance requires actions to support the care recipient so that they can eat safely and deliciously.

[0203] Toilet assistance requires constant attention because the timing varies depending on the care recipient, and it is also physically and mentally stressful, so it is necessary to build a relationship of mutual trust. Mobility assistance requires safely supporting mobility actions such as getting up, standing, walking, and sitting. Sleeping assistance requires not only bedmaking and guiding the care recipient to the bed, but also actions such as responding if the care recipient wakes up in the middle of the night to go to the toilet.

[0204] Next, when the subject habitually performs a behavioral task, the estimation model construction unit 120 reads out from the cloud 3 activity status data (and environmental data, if necessary) corresponding to the behavioral task for multiple times, including at least the most recent one, and performs a cluster analysis based on similarity for each operation phase to create a group.

[0205] The estimation model construction unit 120 clusters together (i.e., groups) of similar action phases that make up a behavioral task. Each action phase has multiple parameters expressed by multiple values, such as the subject's attributes, gender, age, and proficiency level.

[0206] The estimation model construction unit 120 performs cluster analysis by a nonparametric Bayesian method using a Dirichlet process mixture distribution having these multi-value parameters. Note that the cluster analysis method may be a method (k-means method) that classifies each data into multiple groups based on the similarity between data calculated using a preset objective function.

[0207] The estimation model construction unit 120 compares the collected movement phases of different times and calculates the deviation of the subject's activity state in each movement phase as an evaluation function. That is, for each collected movement phase, the estimation model construction unit 120 prepares the deviation between a movement pattern set as a reference and a movement pattern corresponding to the movement phase as a first evaluation function.

[0208] At the same time, the estimation model construction unit 120 prepares, for each grouped operation phase, the deviation between the work time required for the operation pattern set as the standard and the work time required for the operation pattern corresponding to the operation phase in question as a second evaluation function.

[0209] Furthermore, the estimation model construction unit 120 prepares a third evaluation function for each of the collected motion phases based on the physiological system information set as a reference, along with the first and second evaluation functions. To evaluate the appropriateness of this physiological system information, the third evaluation function is set using the state of nervous system activity, such as the sympathetic nervous system, based on heart rate variability, as well as body temperature, blood pressure, heart sounds, etc. as indicators. Note that this heart rate variability may be calculated not only from pulse wave variability from the skin surface, but also from pulse waves, electrocardiograms, ballistic waves associated with cardiac output, or cardiac activity obtained by microwaves, etc.

[0210] Furthermore, the estimation model construction unit 120 prepares, for each grouped operation phase, environmental information (room temperature, humidity, air pressure, noise, etc.) set as a standard as a fourth evaluation function in addition to the first and second evaluation functions described above.

[0211] The estimation model construction unit 120 estimates the transition of the subject's activity state for each action phase in the behavioral task based on the first to fourth evaluation functions, and evaluates the reproducibility, efficiency, and error rate of the subject's behavioral task.

[0212] In this way, the estimation model construction unit 120 constructs an estimation model that has been trained to estimate functions with characteristics unique to the subject by statistically analyzing the correlation between the subject's activity results transcribed to the output device and the subject's personal and environmental information, based on the evaluation results for the subject.

[0213] The autonomous avatar generation unit 121 generates an autonomous avatar that can express itself in information space, including the functions of the subject, based on the estimated model estimated by the estimated model construction unit 120.

[0214] The virtual activity construction unit 11 virtually constructs the human activities of a substitute for the subject in real space by reflecting the self-expression of the autonomous avatar in the information space. At the same time, the information creation unit 12 creates personal characteristic information specific to the substitute while analyzing the human activities of the substitute represented as the autonomous avatar constructed in the information space by the virtual activity construction unit 11.

[0215] As a result, in the space sharing system 1, when a substitute different from the target person in the real space acts in place of the target person, a presence similar to the target person can be reflected as an autonomous avatar via the information space from the cloud 3.

[0216] (5) Method for enabling virtual existence of subject even after death Furthermore, in the space-sharing system 1 according to the present invention, even if a subject in real space dies, as shown in FIG. 23, a presence similar to the subject is represented as an autonomous avatar in the information space using the personality trait analogy unit 130 provided in the second interface device 5.

[0217] The personality trait inference unit 130 infers personality traits related to self-expression of the subject from the operation results of the output device 4. The personality traits related to self-expression of the subject are traits inferred as the subject's personality by associating and accumulating the estimation results of the human psychological sensations (changes in psychological state) by the sensation estimation unit 101 described above for each operation phase constituting the operation content of the output device 4.

[0218] When the acquisition of the subject's human and environmental information by the human and environmental information input / output unit 10 stops for a predetermined period of time, the virtual activity construction unit 11 reflects the personality characteristics inferred by the personality characteristic inference unit 130 in the self-expression of the autonomous avatar.

[0219] As a result, in the space sharing system 1, even if a target person in the real space dies, it is possible to reflect a presence similar to the target person as an autonomous avatar from the cloud 3 via the information space.

[0220] (6) Other Embodiments As described above, in this embodiment, the configuration of the space-sharing system 1 has been described as being configured only in a one-to-one relationship between the first interface device 2 corresponding to the target person and the second interface device 5 corresponding to the output device 4. However, the present invention is not limited to this. It is also possible to provide a plurality of first interface devices 2 corresponding to a plurality of target people, and a plurality of second interface devices 5 corresponding to a plurality of types of output devices 4 according to the purpose.

[0221] 1...Space sharing system, 2...First interface device, 3...Cloud, 4...Output device, 5...Second interface device, 10...Human and environment information input / output section, 11...Virtual activity construction section, 12...Information creation section, 20...Vital sign measurement device, 21...Facial imaging camera, 22...Sound collection microphone, 23...Motion capture device, 24...Peripheral measurement device, 25...Information detection section, 30...Electrode terminal group, 31...Photoelectric probe window, 32...Connection terminal, 40...Brain activity measurement section, 41...Blood flow measurement section, 42...Electrocardiograph, 43...First control section, 44...Wireless communication device, 45...Net-like base, 46A to 46N...Sensor unit, 47...Light emitter, 48...Light receiver, 50...Biological signal measurement wearing device, 53...Biological signal sensor, 54A to 54F...Biological signal sensor group, 55...Measurement module, 56...Measurement module module controller, 57...memory, 58...communication unit, 59...second control unit, 60...near-infrared detection unit, 61...far-infrared detection unit, 62...third control unit, 70...control unit, 71...RGB-D sensor, 72...IMU sensor, 73...fourth control unit, 74...data storage unit, 80...image coordinate setting unit, 81...coordinate group extraction unit, 82...skeleton information acquisition unit, 83...behavior recognition unit, 90...activity transcription unit, 100...activity state transmission unit, 101...sensation estimation unit, 102...sensation feedback generation unit, 103...sensation transmission unit, 110...load sensor, 111...capacitance detection unit, 112...distributed load measurement unit, 113...load information calculation unit, 114...first electrode plate, 115...second electrode plate, 116...cylinder, 120...estimated model construction unit, 121...autonomous avatar generation unit, 130...personality feature analogy unit.

Claims

1. A space sharing system comprising: a human and environment information input / output unit that acquires human and environment information including brain and nervous system information, musculoskeletal system information, physiological information, psychological information, movement information, behavioral information and environmental information of a subject existing in real space; a virtual activity construction unit that virtually constructs the human activities of the subject in the real space in an information space in real time based on the human and environment information of the subject output from the human and environment information input / output unit; and an information creation unit that creates human characteristic information specific to the subject while analyzing the human activities of the subject constructed in the information space by the virtual activity construction unit.

2. The space sharing system described in claim 1, further comprising an activity transcription unit which, upon receiving a request to connect with the subject from an external output device, transcribes all or part of the subject's human activity constructed in the information space by the virtual activity construction unit in accordance with the output format of the output device, and the activity transcription unit reflects human characteristic information specific to the subject created by the information creation unit in the output operation by the output device.

3. The space sharing system according to claim 2, further comprising an activity status transmission unit that transmits feedback of the operating status of the output device to the subject in the real space in real time via the information space.

4. The space sharing system described in claim 3, characterized in that the activity state transmission unit comprises: a sensory estimation unit which, when the output device is operating, estimates human physical and psychological sensations from action results related to the operation content of the output device and sensory results of the environmental state in which the output device is located; a sensory feedback generation unit which generates the physical and psychological sensations estimated by the sensory estimation unit as feedback information; and a sensory transmission unit which transmits the feedback information generated by the sensory feedback generation unit to the subject in the real space.

5. A space sharing system as described in claim 1, comprising: an estimation model construction unit that constructs an estimation model that has been learned to estimate functions having characteristics unique to the subject by statistically analyzing the correlation between the subject's activity results transferred to the output device and the subject's human and environmental information; and an autonomous avatar generation unit that generates an autonomous avatar that is capable of self-expression including the subject's functions in the information space based on the estimation model estimated by the estimation model construction unit, wherein the virtual activity construction unit virtually constructs human activities of a substitute for the subject in the real space while reflecting the self-expression of the autonomous avatar in the information space, and the information creation unit creates human characteristic information unique to the substitute while analyzing the human activity of the substitute represented as the autonomous avatar constructed in the information space by the virtual activity construction unit.

6. The space sharing system described in claim 3, further comprising a personality trait inference unit that infers personality traits related to the subject's self-expression from the operation results of the output device, and wherein the virtual activity construction unit reflects the personality traits inferred by the personality trait inference unit in the self-expression of the autonomous avatar when the acquisition of the subject's human and environmental information by the human and environmental information input / output unit stops for a predetermined period of time.

7. A space sharing method comprising: a first step of acquiring human and environmental information including brain and nervous system information, musculoskeletal system information, physiological information, psychological information, movement information, behavioral information and environmental information of a subject existing in real space; a second step of virtually constructing the human activities of the subject in the real space in an information space in real time based on the human and environmental information of the subject acquired in the first step; and a third step of creating human characteristic information specific to the subject while analyzing the human activities of the subject constructed in the information space by the second step.

8. A space sharing method as described in claim 7, further comprising a fourth step of transcribing, upon receiving a request to connect with the subject from an external output device, all or part of the human activity of the subject constructed in the information space by the second step in accordance with the output format of the output device, wherein in the fourth step, human characteristic information specific to the subject created by the third step is reflected in the output operation by the output device.

9. The space sharing method according to claim 8, further comprising a fifth step of feeding back and transmitting the operating status of said output device to said target person in said real space in real time via said information space.

10. The space sharing method described in claim 9, characterized in that in the fifth step, when the output device is operating, human physical and psychological sensations are estimated from the action results related to the operation content of the output device and the sensory results of the environmental state in which the output device is present, the estimated physical and psychological sensations are generated as feedback information, and then the feedback information is transmitted to the target person in the real space.

11. A space sharing method as described in claim 7, comprising: a sixth step of constructing an estimation model that has been trained to estimate functions having characteristics unique to the subject by statistically analyzing the correlation between the subject's activity results transferred to the output device and the subject's human and environmental information; and a seventh step of generating an autonomous avatar capable of self-expression including the subject's functions in the information space based on the estimation model estimated in the sixth step, wherein in the second step, human activities of a substitute for the subject in the real space are virtually constructed while reflecting the self-expression of the autonomous avatar in the information space, and in the third step, human characteristic information unique to the substitute is created while analyzing the human activity of the substitute represented as the autonomous avatar constructed in the information space by the virtual activity construction unit.

12. The space sharing method of claim 9, further comprising an eighth step of inferring personality traits related to the subject's self-expression from the operation results of the output device, wherein in the second step, when the acquisition of the subject's human and environmental information by the human and environmental information input / output unit is stopped for a predetermined period of time, the personality traits inferred by the personality trait inference unit are reflected in the self-expression of the autonomous avatar.

Images