System for detecting physical function relative to cognitive function

Abstract

To provide a system for detecting a physical function relative to a cognitive function.SOLUTION: A system includes a measurement device and a terminal provided with a display. The measurement device includes posture detection means that detects posture information of a user, and posture transmission means that transmits the posture information of the user to the terminal. The terminal includes: model display means that displays, on the display, a sample human-shaped model representing predetermined posture information and instructs the user to take the same posture motion as the sample human-shaped model; and body schema display means that displays, on the display as body schema information, a difference between the received posture information of the user and posture information of the sample human-shaped model. The display of the terminal may be a head-mounted type that can be worn on a head.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technology for a system that serves as a user interface for a virtual reality space, and in particular to a technology for manipulating the movement and posture of a virtual object (avatar or character) in a virtual space or a game space. [Background technology]

[0002] Virtual reality is a technology that projects a three-dimensional virtual space into the user's field of vision. While viewing the virtual space, the user can generally move within the virtual space by holding and operating a stick device or game controller. In particular, the user can experience a sense of immersion through their field of vision by wearing a head-mounted display (HMD) device on their head. This HMD device may access a virtual space server via a network or may be equipped with a virtual space server itself. The HMD device detects the user's head movement using a posture sensor and controls the user's field of vision and position in the virtual space based on the measured values. In addition, a depth camera capturing the user's field of vision can detect the user's hand movements and reflect those operations in the virtual space.

[0003] Virtual space servers include servers deployed on the Internet (Dedicated Servers) and locally deployed servers (Listen Servers). Services that provide virtual spaces also include social VR platforms such as VRChat (registered trademark), Metaverse (registered trademark), and Second Life (registered trademark).

[0004] Conventionally, there is a technology that reflects the posture of a user's legs in real space in the posture of a virtual character object in virtual space (see, for example, Non-Patent Document 1). According to this technology, an IMU (Inertial Measurement Unit) sensor unit is attached to at least the user's torso, above each knee, and below each knee. Then, posture information of the user's torso and legs estimated using the IMU measurement values ​​is transmitted to a terminal running a virtual reality application. This allows the user's posture in real space to be reflected in the posture of the user's virtual object in the virtual reality application.

[0005] There is also technology that uses an RGB camera and a depth sensor to detect the user's posture and control the game (see, for example, Non-Patent Document 2). This technology uses an RGB camera, a depth sensor, a multi-array microphone, and a processor running dedicated software to recognize the player's position, movement, voice, and face. This allows the user to intuitively play video games using their own body. This technology also uses motion capture, which primarily reads and synthesizes the player's movements. However, unlike typical motion capture, it does not require a special suit with markers or a tracker to detect the markers. By capturing the subject on camera, the player's skeletal movements are detected and reflected in the character's movements in the game in real time.

[0006] There is also a chair-type device that detects the user's posture in real space (see, for example, Patent Document 1). This technology is based on a controller chair that generates signals corresponding to the movements of the user's torso, left arm, right arm, left leg, and right leg. This chair-type device can change the posture of a virtual character by transmitting these rotation angle signals to a terminal running a virtual reality application.

[0007] In recent years, with the increase in dementia patients, training that simultaneously trains cognitive and physical functions has been attracting attention. For example, "Cognicise (registered trademark)" (see Non-Patent Document 6) combines various cognitive tasks that place a cognitive load on the brain with various motor tasks. By implementing this training at the stage of MCI (a state that is not dementia but cannot be said to be normal), it is possible to suppress the decline in cognitive function.

[0008] Another type of dementia is body schema impairment, which can result in a variety of neurological conditions, including difficulty planning movements (apraxia), body neglect (inability to pay attention to one side of the body), and phantom limb syndrome (perception of an amputated limb). This leaves the patient unable to understand how they perceive, navigate, and interact with the world around them. The term "body schema" refers to the sensorimotor representation of the body that primarily functions in coordination of movement and maintaining posture. It is an unconscious, sensorimotor representation of the body's position and movement in space, and its function is usually not consciously understood during everyday activities. Therefore, if the movements and postures based on the body schema differ from the actual movements and postures, the user needs to adjust them. This mainly involves the parietal lobe and motor cortex of the brain. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Patent No. 7189811 [Patent Document 2] Patent No. 6886141 [Non-patent literature]

[0010] [Non-Patent Document 1] Shiftall, [online], [searched April 27, 2024], Internet<URL: https: / / ja.shiftall.net / products / haritorax> [Non-patent document 2] Wikipedia "Kinect", [online], [searched April 27, 2024], Internet<URL: https: / / en.wikipedia.org / wiki / Kinect> [Non-patent document 3] Wikipedia "Eular angles", [online], [Retrieved April 27, 2024], Internet<URL: https: / / en.wikipedia.org / wiki / Euler_angles> [Non-patent document 4] Wikipedia "Quaternion", [online], [Retrieved April 27, 2024], Internet<URL: https: / / en.wikipedia.org / wiki / Quaternion> [Non-patent document 5] Wikipedia "Axis-angle representation", [online], [Retrieved April 27, 2024], Internet<URL: https: / / en.wikipedia.org / wiki / Axis-angle_representation> [Non-patent document 6] National Center for Geriatrics and Gerontology, "Cognisize: Exercise for Dementia Prevention," [online], [Retrieved April 27, 2024], Internet<URL: https: / / www.ncgg.go.jp / hospital / kenshu / kenshu / 27-4.html> [Non-Patent Document 7] "Inverse Kinematics: Analytical Solution for 6-DOF Robot Arms (Geometric Solution)", [online], [Retrieved April 27, 2024], Internet<URL:https: / / qiita.com / Conny_Brown_jp / items / 4433ae9c70e8d60ec084> [Non-patent document 8] "New Generation VR Interactive Gloves," [online], [Retrieved April 27, 2024], Internet<URL:https: / / www.aiuto-jp.co.jp / products / product_4102.php> Summary of the Invention [Problem to be solved by the invention]

[0011] However, the techniques such as those in Patent Document 1 and Non-Patent Documents 1 and 2 mentioned above do not detect physical functions relative to cognitive functions.

[0012] Therefore, an object of the present invention is to provide a system for detecting physical functions relative to cognitive functions. [Means for solving the problem]

[0013] According to the present invention, there is provided a system including a measuring device and a terminal equipped with a display, The measuring device is posture detection means for detecting posture information of a user; posture transmission means for transmitting posture information of the user to the terminal; and The terminal is a model display means for displaying a human model representing predetermined posture information on a display and instructing the user to take the same posture and movement as the human model; a body schema display means for displaying a difference between the received posture information of the user and the posture information of the person-to-person model on a display as body schema information; The present invention is characterized by having the following.

[0014] According to another embodiment of the system of the present invention, The display of the device is a head-mounted type that can be worn on the head. It is also preferable.

[0015] According to another embodiment of the system of the present invention, The body schema display means of the terminal displays a human model representing the predetermined posture information and a real human model representing the received posture information of the user side by side. It is also preferable.

[0016] According to another embodiment of the system of the present invention, The posture detection means of the measurement device detects posture information of the user's legs using a sensor installed in a position close to the user's legs. It is also preferable.

[0017] According to another embodiment of the system of the present invention, The posture detection means of the measurement device is one or more load sensors that measure the right load value of the right foot and the left load value of the left foot in time series, The terminal is a step period determination means for determining the step period of the user's stepping based on the time series of right load values ​​and left load values; and The model display means displays the human model as if it is walking at a predetermined walking cycle; The body schema display means displays the difference between the walking cycle of the user and the walking cycle of the model as body schema information. It is also preferable.

[0018] According to another embodiment of the system of the present invention, As a measuring device, One or more load sensors that measure a right load value of the right foot and a left load value of the left foot in time series; a step period determination means for determining the step period of the user's stepping based on the time series of right load values ​​and left load values; and the posture transmitting means transmits the step period as posture information of the user; As a terminal, The model display means displays the human model as if it is walking at a predetermined walking cycle; The body schema display means displays the difference between the walking cycle of the user and the walking cycle of the model as body schema information. It is also preferable.

[0019] According to another embodiment of the system of the present invention, The measuring device is a high-pass filter that passes only frequencies equal to or higher than a first predetermined frequency with respect to the time-series right and left weight values ​​and outputs left high-pass weight values ​​and right high-pass weight values; a load difference calculation means for calculating a load difference between the left high-pass load value and the right high-pass load value; It is also preferred that the

[0020] According to another embodiment of the system of the present invention, A non-walking determination means is provided in front of the step period determination means, which determines that the user is in a non-walking state when both the right load value and the left load value are simultaneously equal to or greater than a third predetermined threshold value or equal to or less than a fourth predetermined threshold value. and The load difference calculation means determines that the state is walking when the absolute value of the load difference from the non-walking state exceeds a predetermined threshold value (for example, 15 N), or when the absolute value of the difference between the value at the "peak" of the load difference and the value at the "valley" from the non-walking state exceeds a predetermined threshold value. It is also preferable.

[0021] According to another embodiment of the system of the present invention, The posture detection means of the measurement device is one or more load sensors that measure the right load value of the right foot and the left load value of the left foot in time series, The terminal is a low-pass filter that passes only frequencies equal to or lower than a second predetermined frequency with respect to the right weight value and the left weight value over time, and outputs a right low-pass weight value and a left low-pass weight value; a reference load value storage means for storing right and left load values ​​in a seated state in which the user has balanced the left and right sides as right and left reference load values; an applied load value calculation means for calculating a right applied load value which is the difference between the right low-pass load value and the right reference load value, and a left applied load value which is the difference between the left low-pass load value and the left reference load value; a center-of-gravity position specifying means for specifying the left and right center-of-gravity positions based on the difference between the right applied load value and the left applied load value; and The model display means displays the pose of the pose model with a predetermined center of gravity; The body schema display means displays the difference between the center of gravity of the user and the center of gravity of the representative model as body schema information. It is also preferable.

[0022] According to another embodiment of the system of the present invention, The measuring device is The posture detection means includes one or more load sensors that measure the right load value of the right foot and the left load value of the left foot in time series, a low-pass filter that passes only frequencies equal to or lower than a second predetermined frequency with respect to the right weight value and the left weight value over time, and outputs a right low-pass weight value and a left low-pass weight value; a reference load value storage means for storing right and left load values ​​in a seated state in which the user has balanced the left and right sides as right and left reference load values; an applied load value calculation means for calculating a right applied load value which is the difference between the right low-pass load value and the right reference load value, and a left applied load value which is the difference between the left low-pass load value and the left reference load value; a center-of-gravity position specifying means for specifying the left and right center-of-gravity positions based on the difference between the right applied load value and the left applied load value; and the posture transmitting means transmits the center of gravity position as posture information of the user; As a terminal, The model display means displays the pose of the pose model with a predetermined center of gravity; The body schema display means displays the difference between the center of gravity of the user and the center of gravity of the representative model as body schema information. It is also preferable.

[0023] According to another embodiment of the system of the present invention, The center-of-gravity position specifying means specifies the left and right center-of-gravity positions from a subtraction weight value obtained by subtracting the left applied weight value from the right applied weight value or subtracting the right applied weight value from the left applied weight value. It is also preferable.

[0024] According to another embodiment of the system of the present invention, The center of gravity position specifying means specifies the front and rear center of gravity positions by adding the right applied load value and the left applied load value. It is also preferable.

[0025] According to another embodiment of the system of the present invention, The center of gravity position specifying means uses the following coordinate axes: As both the right and left applied load values ​​increase, the center of gravity position moves forward, and as both decrease, the center of gravity position moves backward. The greater the right load value is compared to the left load value, the more the center of gravity position moves to the right, and the greater the left load value is compared to the right load value, the more the center of gravity position moves to the left. It is also preferable.

[0026] According to another embodiment of the system of the present invention, The measurement device is a chair-like piece of furniture that comes into contact with the user's body, The load sensor is placed on the seat of the furniture-type device, where the pressure of the user's thighs is transmitted, and measures the right and left load values ​​based on the user's footsteps while seated. It is also preferable.

[0027] According to another embodiment of the system of the present invention, The measuring device is a plate-type device or a shoe-type device on which the user stands and presses the sole of the foot, The load sensor is placed in the plate-type device or the shoe-type device at a portion where the pressure of the sole of the user's foot is transmitted, and measures the right load value and the left load value based on the stepping in the standing state. It is also preferable.

[0028] According to another embodiment of the system of the present invention, The attitude detection means of the measurement device includes: a depth sensor disposed at a position below the seat of the measurement device, for measuring a planar depth distribution around the lower legs of a seated user; a leg angle estimation means for estimating the posture of the lower leg and the posture of the upper leg based on the depth distribution and estimating the direction and / or angle of the lower leg and the upper leg; and outputs the orientation and / or angle of the lower leg and upper leg as posture information of the user. It is also preferable.

[0029] According to another embodiment of the system of the present invention, the measurement device is a controller that the user holds in his / her hand or wears on his / her arm, The attitude detection means detects a spatial position of the controller; The attitude transmitting means transmits the spatial position of the controller to the terminal; As a terminal, further comprising a posture estimation means for estimating posture information of the user from the spatial position of the controller and the spatial position of the terminal; the model display means displays the arm of the human model in a predetermined direction and / or angle; The body schema display means displays the difference between the posture information of the user's arm and the posture information of the human model's arm as body schema information. It is also preferable to

[0030] According to another embodiment of the system of the present invention, The measuring device is the posture detection means is a camera that captures a posture image of the user, The posture transmission means transmits the posture image to the terminal; As a terminal, The apparatus further includes a posture estimation means for estimating posture information of the user from a posture image of the camera, the model display means displays the arm of the human model in a predetermined direction and / or angle; The body schema display means displays the difference between the posture information of the user's arm and the posture information of the human model's arm as body schema information. It is also preferable to [Effects of the Invention]

[0031] The system of the present invention aims to provide a system for detecting physical functions relative to cognitive functions. [Brief explanation of the drawings]

[0032] [Figure 1] 1 is a configuration diagram of a system including a measuring device and a terminal according to the present invention. [Figure 2] FIG. 2 is an external view illustrating a state in which a user is seated relative to the measurement device. [Figure 3] FIG. 2 is a diagram illustrating the basic functional configuration of a measurement device and a terminal according to the present invention. [Figure 4] FIG. 2 is a layout diagram of an attitude detection unit in the measurement device of the present invention. [Figure 5] 5 is a graph showing a change in load value in the load sensor of FIG. 4. [Figure 6] FIG. 2 is a functional configuration diagram of a terminal according to the present invention. [Figure 7] 10 is a graph showing a change in a load value in a high-pass filter. [Figure 8] 10 is a graph showing a change in a load value in a load difference calculation unit. [Figure 9] 10 is a graph showing absolute values ​​relative to load differences. [Figure 10] 10 is a graph showing a change in a load value in a low-pass filter. [Figure 11] 10A and 10B are explanatory diagrams showing changes in the center of gravity position in the center of gravity position identifying unit; [Figure 12] FIG. 2 is a functional configuration diagram of a measurement device according to the present invention. [Figure 13] FIG. 1 is an external view showing the arrangement of depth sensors in a measurement device of the present invention. [Figure 14]FIG. 10 is an explanatory diagram showing each vector in the right leg of a user seated on the measurement device. [Figure 15] 1 is an explanatory diagram showing the detection of a position vector j of the ankle and a direction vector l of the lower leg when the measurement device is viewed from the side. [Figure 16] FIG. 2 is an explanatory diagram of a vector estimated by the present invention. [Figure 17] FIG. 2 is a functional configuration diagram of a controller and a terminal for detecting the posture of an arm. [Figure 18] FIG. 10 is an explanatory diagram showing detection of the posture of the arm by inverse kinematics. [Figure 19] FIG. 10 is a view diagram displayed by a training application on the body schema display. DETAILED DESCRIPTION OF THE INVENTION

[0033] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0034] FIG. 1 is a diagram showing the configuration of a system comprising a measuring device and a terminal according to the present invention.

[0035] 1 includes a measuring device 1 and a terminal 2 equipped with a display. As the terminal 2, for example, Meta Quest 3 (registered trademark) manufactured by Meta Inc. can also be used.

[0036] 1(a), the measurement device is shown as a chair-like piece of furniture-type equipment that the user can sit on and step on, as a first embodiment.

[0037] 1(b), the measurement device 1 is shown as a plate-shaped device on which the user stands as a second embodiment. The user stands with the soles of their feet pressed against the plate-shaped portion of the measurement device 1, and can step on it. Of course, the measurement device 1 is not limited to a plate-shaped device, and may also be a shoe-shaped device for each foot.

[0038] Furthermore, terminal 2 is represented as a head-mounted type (HMD (Head Mounted Display) terminal) that can be worn on the user's head. This allows the user to be immersed in a virtual reality space. At this time, the user cannot see the real space, and cannot visually see their own body. Of course, this is not limited to an HMD terminal, and any terminal connected to a display that the user can visually recognize can be used.

[0039] FIG. 2 is an external view showing a state in which a user is seated relative to the measurement device.

[0040] 2, a user is seated on a chair-like measuring device 1, wearing a head-mounted terminal 2 on their head, and is immersed in a virtual reality space that appears before their eyes. In this state, the user can step on their feet, move the center of gravity of their body (forward, backward, left, and right), and move their leg joints. Furthermore, according to the present invention, the user may hold the controller 3 in both hands or wear it on both arms. The user can move the joints of the hands and arms. As will be described later, the controller 3 can detect the spatial position of the user's hands using a built-in sensor. Although FIG. 2 shows a chair-shaped piece of furniture-type equipment and a board-shaped stationary equipment together, it goes without saying that each piece of equipment may be used independently.

[0041] In other embodiments, the measuring device 1 is not limited to a chair-like shape, but may be a cushion-like or mat-like shape. In this case, the measuring device 1 can be placed on the seat of an existing chair. Of course, the measuring device 1 may also be a wheelchair.

[0042] According to the present invention, for example, a virtual space of a training application is displayed on the display of terminal 2, and the user moves their body accordingly. For example, the user is instructed to step, bend and stretch their legs, shift the center of gravity of their buttocks, and move their arms forward, backward, left and right. It is also preferable that the user themselves act as an avatar in the virtual space.

[0043] FIG. 3 is a diagram showing the basic functional configuration of the measuring device and terminal according to the present invention.

[0044] The system shown in Figure 3 uses virtual reality technology to allow users to adjust their own movements and postures based on their body schema. In other words, it allows users to recognize that the movements and postures based on their body schema differ from real movements and postures. This makes it possible to provide elderly people with training to correct their body schema using healthcare equipment.

[0045] In the present invention, communication between the measuring device 1 and the terminal 2 is possible by short-distance wireless (for example, Bluetooth (registered trademark) or wireless LAN). Of course, communication may also be possible by wire.

[0046] <Measurement device 1> 3, the measurement device 1 of the present invention has an attitude detection unit 11 and an attitude transmission unit 12. These functional components are realized by executing a program that causes a computer installed in the measurement device 1 to function.

[0047] According to the embodiment of the present invention, the attitude detection unit 11 of the measurement device 1 detects the following attitude information. <<Posture information of the user's legs detected by the load sensor of the measurement device 1>> <When identifying the walking cycle> <When identifying the center of gravity> <<Posture information of the user's legs detected by the depth sensor of the measurement device 1>> In another embodiment, the posture detection unit 11 may be detected by a sensor of another device that serves as a separate measurement apparatus 1. <<Posture information of the arm position detected by the sensor of controller 3>> <<User posture information captured by a camera>> Then, the attitude transmitting unit 12 transmits the detected attitude information to the terminal 2.

[0048] <Terminal 2> 3, the terminal 2 of the present invention has a model display unit 21 and a body schema display unit 22. These functional components are realized by a computer installed in the terminal 2 executing a program that causes the computer to function.

[0049] The model display unit 21 displays a "personalized model" that expresses predetermined posture information on the display, and instructs the user to assume the same posture and movement as the personalized model. Here, particularly in the case of terminal 2 equipped with a head-mounted display, the user cannot see his / her own legs, hands, and arms and must move them using only his / her senses, which can easily cause a mismatch between cognitive function and physical function.

[0050] The body schema display unit 22 displays the difference between the received posture information of the user and the posture information of the human model on the display as body schema information. This functions as a training application. Specifically, Unity (registered trademark), a game engine by Unity Technologies, Inc., can be used.

[0051] According to the present invention, the "real-life model" walks with a predetermined walking cycle, leans its body to a predetermined center of gravity position, and bends and straightens its legs at a predetermined angle. In response, the user also takes the same posture and movements as the real-life model displayed on the display of terminal 2. The user steps in time with the walking cycle of the real-life model, moves the center of gravity of the buttocks in the same way, and bends and straightens its legs in the same way. In this case, even though the user subjectively thinks that they have adopted the same posture and movement as the person-to-person model, the posture and movement may differ from the actual posture and movement when viewed objectively. This means that there is a discrepancy between the posture and movement of the unconscious, sensorimotor "body schema" and the actual posture and movement. For this reason, terminal 2 needs to make the user aware that the movements and postures based on the body schema differ from those of the person-to-person model.

[0052] The body schema display unit 22 of the terminal 2 may display a phantom human model representing predetermined posture information and a real human model representing the received posture information of the user side by side or overlaid on each other. Of course, the body schema display unit 22 of the terminal 2 may also display the difference between the posture and movement of the user and the real posture and movement numerically. For example, when a user moves their legs or arms, they can see their movements from a bird's-eye view in the virtual space. By comparing their real-life model with the model of their actual self and viewing it from above, the user can recognize their own body schema.

[0053] The body schema display unit 22 of the terminal 2 has the following two display methods. (Display Example 1) When the user is moving their body (when the terminal 2 is receiving posture information from the measurement device 1), only the visibly posed model is displayed on the display, and posture information is accumulated. At this time, the user can only see the visibly posed model and cannot recognize the posture they are moving. After that, in order to replay using the accumulated posture information, the visibly posed model and their own real-life model are displayed side by side or overlaid (or a numerical value indicating the difference is displayed) to the user. (Display Example 2) Even when the user is moving their body (when the terminal 2 is receiving posture information from the measurement device 1), the display shows the visibly-appearing model and the user's actual human model side by side or overlaid (or displays a numerical value indicating the difference). At this time, the user can adjust their own posture in real time while recognizing the difference between the visibly-appearing model and the posture they are moving.

[0054] In one embodiment of the present invention, it is also preferable to display a real human model on a display as if it is walking, even though the user is seated on a chair-like piece of furniture and tapping their feet. The model walks in accordance with the user's stepping cycle, and the state of the user walking is displayed in the virtual space. At this time, the forward movement amount per unit time is calculated according to the step cycle. For the user, this change in field of view serves as rehabilitation for the walking nerves in the brain. By immersing the user in a virtual space and training them as if they were playing a game, the user is able to focus on achieving their goal and activate their brain.

[0055] In another embodiment of the present invention, the body schema display unit 22 may display the user's whole body posture as a virtual object by combining the user's posture movements detected by the measurement device 1 with a hand tracking function using the controller 3.

[0056] <<Posture information of the user's legs detected by the load sensor of the measurement device 1>> FIG. 4 is a layout diagram of the attitude detection unit in the measurement device of the present invention.

[0057] 4 shows the seat portion of the chair-like furniture-type measuring device 1 as seen from above. The posture detecting unit 11 and the posture transmitting unit 12 are embedded in the furniture-type device (measuring device 1). The posture detection unit 11 detects posture information of the user's legs using sensors installed in proximity to the user's legs. The posture detection unit 11 may be, for example, one or more "load sensors" that measure the right load value of the right foot and the left load value of the left foot in time series. A load sensor is a sensor that converts load into an electrical signal.

[0058] According to FIG. 4, when the measuring device 1 is a furniture-type device (see FIG. 1(a) described above), the load sensor is placed on the seat portion where the pressure of the user's thighs is transmitted, and measures the right load value and the left load value based on the user's footsteps while seated. Furthermore, even if the measuring device 1 is a plate-type (see Figure 1(b) above) or shoe-type device, the load sensor is placed in the part of the measuring device 1 where the pressure of the user's sole is transmitted, and measures the right load value and left load value based on the stepping in the standing state. Then, the attitude transmitting unit 12 of the measurement device 1 transmits the right load value and the left load value to the terminal 2 as attitude information.

[0059] FIG. 5 is a graph showing changes in the load value in the load sensor of FIG.

[0060] 5 shows the right load value and left load value detected by the load sensor of the posture detection unit 11 according to the elapsed time when the user steps. The load value increases when the user swings the foot down, and decreases when the user swings the foot up. The load value changes cyclically when the user steps, and the phase difference between the left and right load values ​​is approximately 180 degrees.

[0061] FIG. 6 is a functional configuration diagram of a terminal according to the present invention.

[0062] 6, the functional units realized by the programs executed by the installed computer are configured within the terminal 2. Of course, they may also be configured within the measurement device 1, as shown in FIG. 12, which will be described later.

[0063] <When identifying the walking cycle> 6, the terminal 2 has at least a step period identification unit 136. The terminal 2 may further have a high-pass filter 133, a load difference calculation unit 134, and a non-walking determination unit 135. These functional components are realized by executing a program that causes a computer installed in the terminal 2 to function. The processing flow of these functional components can also be understood as a step period identification method.

[0064] [High Pass Filter 133] The high-pass filter 133 is located in the front stage of the load difference calculation unit 134, and passes only frequencies equal to or higher than a first predetermined frequency with respect to the right load value and the left load value over time, and outputs a left high-pass load value and a right high-pass load value.

[0065] FIG. 7 is a graph showing changes in the weight value in the high-pass filter. 7, the high-pass filter performs zero point correction on the load value while leaving the stepping component. In the case of walking, the cutoff frequency (first predetermined frequency) may be set to, for example, 0.08 Hz.

[0066] [Load difference calculation unit 134] The load difference calculation unit 134 calculates the load difference between the right load value and the left load value.

[0067] FIG. 8 is a graph showing the change in the load value in the load difference calculation unit.

[0068] Figure 8 shows the load difference when stepping, and the graph is close to a sine wave. The graph shows a "peak" when the left foot is swung down and the right foot is swung up, and a "valley" when the left foot is swung up and the right foot is swung down.

[0069] It is also preferable to judge the stepping state by dividing it into a "walking state" and a "non-walking state." According to Figure 8, when stepping starts with the left foot, the load difference first becomes a "peak." Conversely, when stepping starts with the right foot, the load difference first becomes a "valley." Therefore, for example, it may be possible to determine that the "walking state" has been reached when either of the following two conditions is met: (i) When the absolute value of the load difference from the non-walking state exceeds a predetermined threshold (e.g., 15 N). (ii) When the absolute value of the difference between the load difference at the "peak" and the load difference at the "valley" exceeds a predetermined threshold value from the non-walking state. The period from when the state becomes walking until when the state becomes non-walking is determined as the walking state.

[0070] [Non-walking determination unit 135] The non-walking determining unit 135 determines the "non-walking state" when both the right load value and the left load value are simultaneously equal to or greater than the third predetermined threshold value or equal to or less than the fourth predetermined threshold value.

[0071] For example, it may be determined that the user has entered a non-walking state when any of the following three conditions is met: (i) When the left and right load values ​​simultaneously exceed a predetermined threshold (e.g., 20 N) while walking. (ii) When the left and right load values ​​simultaneously fall below a predetermined threshold (e.g., −20 N) while walking. (iii) When the time during which the absolute value of the load difference is equal to or less than a predetermined threshold (for example, 10 N) from the walking state exceeds a threshold value thr_time calculated by the following formula, for example: thr_time=0.30-(speed_ave×0.08) speed_ave: Moving average of walking speed in virtual space (e.g., the previous 20 times) In (i) and (ii), the state is judged as non-walking when both feet are stepping simultaneously, and in (iii), the state is judged as non-walking when stepping stops.

[0072] FIG. 9 is a graph showing absolute values ​​versus load differences.

[0073] FIG. 9 shows the reason why (iii) is set, in which the state is determined to be non-walking when the stepping stops. When the user is stepping, the time during which the absolute value of the load difference is, for example, 10 N or less is very short (e.g., FIG. 9A). On the other hand, when the user stops stepping, the time during which the absolute value of the load difference is 10 N or less becomes longer (e.g., FIG. 9B). Here, the slower the pace of stepping, the longer the time during which the absolute value of the load difference remains below 10 N (for example, FIG. 9A). For this reason, thr_time must be sufficiently large. On the other hand, the faster the pace of stepping, the faster the walking speed in the virtual space. Therefore, if the user does not immediately determine that they are not walking when they stop stepping, it will give the user a strange feeling. In other words, when stepping at a fast pace, thr_time must be small. Therefore, as shown in the formula above, the speed_ave element is added to thr_time, which makes thr_time smaller the faster the stepping pace, preventing the user from feeling uncomfortable. In addition, the coefficients and constants in the above-mentioned formula (iii) may be adjusted adaptively by the system based on the time in Figures 9A and 9B by monitoring the measured values ​​of the user's stepping data. Also, other judgment formulas may be used that adaptively adjust the non-walking state based on the walking speed or load value.

[0074] [Gait cycle determination unit 136] The step period determination unit 136 determines the step period of the user's stepping from the time series of the load difference. Specifically, the time from when the load difference exceeds a predetermined threshold (e.g., +15N) to when the load difference again exceeds the predetermined threshold (e.g., +15N) is defined as the left foot stepping cycle, and its inverse is detected as the left foot stepping frequency.Furthermore, the time from when the load difference falls below a predetermined threshold (e.g., -15N) to when the load difference again falls below the predetermined threshold (e.g., -15N) is defined as the right foot stepping cycle, and its inverse is detected as the right foot stepping frequency. The detected stepping frequency is multiplied by a gain to determine the "walking speed" in the virtual space. This allows the walking speed to change according to the pace of the stepping. Specifically, for example, the walking cycle of an adult is about 3 Hz (per leg), while the walking cycle of an elderly person tends to be about 1 Hz (per leg). The leg load value of an adult is about 30 to 50 N, while the leg load value of an elderly person tends to be about 10 to 15 N. Of course, it is preferable to be able to change the threshold value of the load difference depending on the user's attributes such as age, sex, and physique.

[0075] <When identifying the center of gravity> 6, the terminal 2 has a low-pass filter 143, a reference load value storage unit 144, an applied load value calculation unit 145, and a center-of-gravity position identification unit 146. These functional components are realized by executing a program that causes a computer installed in the terminal 2 to function. The processing flow of these functional components can also be understood as a center-of-gravity position identification method.

[0076] [Low-pass filter 143] The low-pass filter 143 passes only frequencies equal to or lower than a second predetermined frequency with respect to the right weight value and the left weight value that change over time, and outputs a right low-pass weight value and a left low-pass weight value.

[0077] FIG. 10 is a graph showing changes in the load value in the low-pass filter.

[0078] 10, the low-pass filter 143 removes stepping components from the right load value and the left load value to extract the center of gravity shift component. This cutoff frequency (second predetermined frequency) is, for example, 0.07 Hz. For example, when the user leans to the right, the right low-pass weight value increases and the left low-pass weight value decreases, whereas when the user leans to the left, the left low-pass weight value increases and the right low-pass weight value decreases.

[0079] According to Figure 10, the following changes have occurred: (58 seconds to 60 seconds) The left and right low-pass load values ​​increase in the same way. This can be understood as the center of gravity moving forward. (63 seconds) The difference between the left and right low-pass load values ​​is small. This can be understood as meaning that the center of gravity is close to the origin, i.e., the user is sitting with their left and right balance in mind. (64 seconds to 66 seconds) The left low-pass load value increases, and the right low-pass load value decreases. This can be understood as the center of gravity moving to the left. (69 seconds) The difference between the left and right low-pass load values ​​is large. This indicates that the center of gravity is far from the origin, i.e., the user is leaning significantly to the left or right.

[0080] [Reference load value storage unit 144] The reference load value storage unit 144 stores the right and left load values ​​in a seated state where the user has balanced the left and right sides as the right reference load value and the left reference load value. The right and left reference load values ​​become the origins of the coordinate system of the center of gravity position.

[0081] [Applied load value calculation unit 145] The applied weight value calculation unit 145 calculates a right applied weight value which is the difference between the right low-pass weight value and the right reference weight value, and a left applied weight value which is the difference between the left low-pass weight value and the left reference weight value. Right load value R = Right low pass load value - Right reference load value Left applied load value L = Left low pass load value - Left reference load value

[0082] [Center of gravity position identification unit 146] The center of gravity position specifying unit 146 specifies the center of gravity position from the right applied load value R and the left applied load value L. (Identify the center of gravity on the left and right) The center of gravity position specifying unit 146 subtracts the left applied load value L from the right applied load value R, or specifies the left and right center of gravity positions from the subtracted load value obtained by subtracting the right applied load value R from the left applied load value L. (Identify the front and rear center of gravity) The center of gravity position specifying unit 146 specifies the front and rear center of gravity positions by adding the right applied load value R and the left applied load value L. Center of gravity position (RL, R+L) or (LR, L+R)

[0083] FIG. 11 is an explanatory diagram showing a change in the center of gravity position in the center of gravity position specifying unit.

[0084] 11, the positive side of the y-axis coordinate is the front side of the chair seat (the thighs), and the negative side is the rear side of the chair seat (the backrest). First, when the user is seated with their left and right balance in mind, the right and left load values ​​are stored as the right and left reference load values ​​so that the origin coordinates are (0,0).

[0085] The center of gravity position specifying unit 146 specifies the center of gravity position on the coordinate axes as follows. (1a) When both the right load value R and the left load value L increase over time (when R+L is large in the positive direction), the center of gravity is shifting forward (posture is leaning forward). According to Figure 11, the center of gravity is shifting in the positive direction of the y-axis. (1b) When both the right load value R and the left load value L decrease over time (when R+L is large in the negative direction), the center of gravity is shifting toward the rear of the seat (posture is leaning backward). According to Figure 11, the center of gravity is shifting in the negative direction of the y-axis. (2a) When the right load value R is larger than the left load value L (when RL is large in a positive direction or when LR is large in a negative direction), the center of gravity position moves to the right of the seat (posture tilts to the right). According to Figure 11, the center of gravity position moves in the positive direction of the x-axis. (2b) When the left applied load value L is larger than the right applied load value R (when LR is large in a positive direction or when LR is large in a negative direction), the center of gravity position moves to the left of the seat (posture tilts to the left). According to Figure 11, the center of gravity position moves in the negative direction of the x-axis.

[0086] [Model display section 21] The model display unit 21 displays the human model walking with a predetermined step cycle and / or tilting to a predetermined center of gravity position.

[0087] [Body Schema Display 22] The body schema display unit 22 displays the difference between the walking cycle and / or the center of gravity position of the user and the walking cycle and / or the center of gravity position of the representative model as body schema information.

[0088] FIG. 12 is a functional configuration diagram of the measurement device according to the present invention.

[0089] 12, in comparison with FIG. 6, the "posture information" transmitted from the measuring device 1 to the terminal 2 is the step period and / or the center of gravity position. When transmitting the walking period to the terminal 2, the measuring device 1 has, in addition to the posture detection unit 11 (left load sensor, right load sensor), a high-pass filter 133, a load difference calculation unit 134, a non-walking determination unit 135, and a walking period identification unit 136, as in the above-mentioned Figure 6. In addition, when the measuring device 1 transmits the center of gravity position to the terminal 2, it has a low-pass filter 143, a reference load value memory unit 144, an applied load value calculation unit 145, and a center of gravity position identification unit 146, as shown in Figure 6 above. In this way, the measuring device 1 may be integrally built therein, or, of course, a plurality of functional units of the measuring device 1 may be configured as an external device. Then, the posture transmitting unit 12 of the measurement device 1 transmits the stride period and / or the center of gravity position to the terminal 2 as posture information of the user.

[0090] 12, the terminal 2 receives the stride period and / or the center of gravity position as posture information from the measurement device 1. Then, the terminal 2 causes only the model display unit 21 and the body schema display unit 22 described above to function.

[0091] <<Posture information of the user's legs detected by the depth sensor of the measurement device 1>> FIG. 13 is an external view showing the arrangement of depth sensors in the measurement device of the present invention.

[0092] 13, the posture detection unit 11 of the measurement device 1 includes a depth sensor 113 and a leg angle estimation unit 114. The leg angle estimation unit 114 is realized by a program that causes a computer installed in the measurement device to function.

[0093] [Depth Sensor 113] The depth sensor 113 is placed at a position below the seat of the measurement device, and measures the planar depth distribution around the lower legs of a seated user.

[0094] 13(a), one depth sensor 113 is installed on a relay axis extending from the seat of the measurement device 1 toward the floor surface, and is directed toward the area around the lower legs of a seated user. The depth sensor 113 may be, for example, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) device or a depth camera. This allows the planar depth distribution around the lower legs of a seated user to be measured. For example, a stereo camera or an IR (infrared) camera can be used to measure the planar depth distribution.

[0095] According to FIG. 13(b), 30 distance measuring sensors are arranged in 5 rows and 6 columns on the front curtain surface below the seat of the measurement device 1, facing the area around the lower legs of a seated user. Distance sensors measure a single distance using light, ultrasound, changes in capacitance, etc. For example, by arranging them in an array of two or more rows and two or more columns, it is possible to measure a planar depth distribution.

[0096] According to the present invention, it is sufficient to measure a planar depth distribution using one or more depth sensors 113. Here, a planar depth distribution is a two-dimensional array of multiple depth values, or a three-dimensional point cloud based on those depth values.

[0097] [Leg angle estimation unit 114] The leg angle estimation unit 114 estimates the posture of the lower leg and the posture of the upper leg based on the depth distribution, and estimates the direction and / or angle of the lower leg and the upper leg.

[0098] FIG. 14 is an explanatory diagram showing each vector in the right leg of a user seated on the measurement device.

[0099] According to FIG. 14, the leg angle estimation unit 114 represents the following five vectors in a preset coordinate system. Ankle position vector j: posture of the lower leg · Direction vector l of the lower leg: Posture of the lower leg (from the knee to the ankle) Knee position vector k: Upper thigh posture Upper thigh direction vector u: Upper thigh posture (from the hip joint to the knee) Hip joint position vector h The ankle in the present invention means the area around the ankle or heel.

[0100] Here, the position means a three-dimensional real vector. These vectors may be converted into a predetermined coordinate system before being output. The coordinate system may be one or more arbitrary coordinate systems set during the manufacture of the measurement device 1, or may be a coordinate system that is dynamically set according to the posture of the seated user. The output values ​​may also be corrected by scaling, rotating, offsetting, or limiting the range of values.

[0101] FIG. 15 is an explanatory diagram showing the detection of the position vector j of the ankle and the direction vector l of the lower leg when the measurement device is viewed from the side.

[0102] The leg angle estimation unit 114 detects, as the posture, a position vector i of the user's ankle and a direction vector l of the user's lower leg based on the depth distribution. 15, five depths are measured when the front curtain surface below the seat of the measurement device 1 is viewed from the side. Here, there is a large difference in depth between the top two depths and the bottom three depths. This can be understood as the top two depths representing the distance to the user's lower legs. The "ankle position vector j" can be considered as the position of the lowest depth of the two upper depths. The "lower leg direction vector l" can be calculated from the difference between the positions of the two upper depths. In this way, the ankle position vector i and the lower leg direction vector l can be analytically detected based on the depth distribution detected by the depth sensor 113, by taking the difference between adjacent depth values, or by extracting feature points based on a threshold or gradient.

[0103] For simplicity, Fig. 15 shows the case of one leg in two dimensions, but of course it can be expanded to two legs in three dimensions as well. For example, if high-resolution depth distribution can be obtained by LiDAR or a depth camera, estimation can be performed with high accuracy.

[0104] [Detection of ankle position vector i and lower leg direction vector l using a machine learning engine] A machine learning engine may be used to estimate the ankle position vector i and the lower leg direction vector l from the depth distribution detected by the depth sensor 113. In this case, the training data may use a plurality of depth distributions as explanatory variables and the ankle position vector i and the lower leg direction vector l as objective variables. The machine learning engine is trained in advance using this training data to build a learning model. Of course, techniques such as segmentation and clustering may also be used, or a neural network may also be used.

[0105] FIG. 16 is an explanatory diagram of vectors estimated by the present invention.

[0106] 16(a), the leg angle estimation unit 114 estimates the knee position vector k as the posture from the user's ankle position vector i and lower leg direction vector l, using a predetermined parameter t. Specifically, this is expressed by the following equation: k=-tl+j (t>0)

[0107] According to FIG. 16(b), the leg angle estimation unit 114 estimates the posture, ie, the length x of the upper leg and the length y of the lower leg, from the position vector i of the user's ankle, the direction vector l of the lower leg, and the position vector h of the hip joint that is set in advance or detected by a sensor, using a predetermined parameter t. The length x of the user's upper leg and the length y of the user's lower leg are expressed by the following equations: x=||kh||=||-tl+jh|| y=||jk||=||tl||

[0108] Here, the hip joint position vector h is determined based on the position and orientation of the seated user's waist, with the upper leg serving as the center of rotation. This may be set in advance when the measurement device is designed. The measurement device 1 may also include one or more seat pressure sensors for measuring pressure distribution, which are arranged on the seat (see, for example, Patent Document 2). The leg angle estimation unit 114 can estimate the hip joint position vector h based on the pressure distribution measured by the seat pressure sensor. In other words, it can estimate the position and orientation of the seated user's waist. For example, the built-in pressure distribution sensor sheet is a film-like sensor sheet with a thickness of approximately 0.1 mm, and can detect applied pressure in real time.

[0109] Here, the leg angle estimation unit 114 determines a predetermined parameter t based on the ratio α between the length x of the upper leg and the length y of the lower leg. The ratio α may be set in advance based on statistical information on human body dimensions. Alternatively, it may be approximately set to α=1, or may be changeable by the user. This is expressed by the following formula: x=αy (α>0) ||-tl+jh|| 2 =α 2 ||tl|| 2 This equation is a linear equation with respect to t when α=1, and a quadratic equation when α≠1, so the following solution can be analytically derived.

number

[0110] 16(c), the leg angle estimation unit 114 estimates the posture of the user as the direction vector u of the upper leg from the position vector i of the user's ankle, the direction vector l of the lower leg, and the position vector h of the hip joint that is set in advance or detected by a sensor, using a predetermined parameter t. As a result, it is expressed by the following equation. u=kh=-tl+jh

[0111] The orientation of the upper leg direction vector u and the lower leg direction vector l is calculated based on a predetermined coordinate system. Here, the orientation refers to information that represents rotation, such as Euler angles, quaternions, and axis-angle representations (see, for example, Non-Patent Documents 3, 4, and 5).

[0112] As shown in Figures 16(a) to (c), according to the present invention, the depth sensor 113 arranged in the measurement device 1 can detect the position of the knee and the posture of the upper leg by simply observing only a part of the user's lower leg.

[0113] <About local coordinate systems> The orientation of a body part can be calculated by setting a reference coordinate system and a local coordinate system of that body part (see, for example, Non-Patent Document 3). For example, the local coordinate system of the upper thigh is set with the seat surface or the lower back as the reference coordinate system. x-axis: direction vector u of the upper thigh y-axis: A vector that is perpendicular to the upper leg direction vector u and parallel to the plane defined by the upper leg direction vector u and the lower leg direction vector l z-axis: vector perpendicular to the x-axis and y-axis Furthermore, the local coordinate system of the lower leg is set with the previously determined local coordinate system of the upper leg as the reference coordinate system. x-axis: direction vector of the lower leg y-axis: A vector that is perpendicular to the direction vector l of the lower leg and parallel to the plane defined by the direction vector u of the upper leg and the direction vector l of the lower leg z-axis: vector perpendicular to the x-axis and y-axis However, since the setting of the local coordinate system for the body part differs depending on the system, the method for calculating the orientation of the body part is not limited to this.

[0114] When outputting the orientation of the upper leg or lower leg as Euler angles, the units of the values ​​may be converted to degrees, radians, or the like. Also, taking into account the degrees of freedom of the joints, only some of the Euler angles may be output. For example, since the rotation axis of the knee joint has one degree of freedom, when outputting the orientation of the lower leg with the local coordinate system of the upper leg as the reference coordinate system, only the pitch angle of the Euler angles may be output. Furthermore, corrections may be made to the output orientation values, such as by applying a scaling factor or rotation, adding an offset, or limiting the range of values. For example, by correcting the pitch angle of the orientation of the lower leg with the local coordinate system of the upper leg as the reference coordinate system, it is possible to output it as the flexion angle of the knee joint.

[0115] The following are examples of attitudes that can be output by the measurement device. Upper thigh posture: position of the center of rotation of the upper thigh (hip joint), knee position, upper thigh direction, knee direction Lower leg posture: Position of the center of rotation of the lower leg (knee), position of the ankle, direction of the lower leg, direction of the ankle Note that this posture information has redundancy. That is, the orientations of the knee, ankle, upper leg, and lower leg can be calculated from the position of the rotation center of the user's upper leg, the position of the knee, and the position of the ankle. Furthermore, if the lengths of the upper leg and lower leg are set in advance, the position and orientation of the knee and ankle can also be calculated from the orientation of the upper leg and the orientation of the lower leg. As a result, the measurement device does not need to output all of the above postures, and may output only a portion of them.

[0116] Furthermore, in another embodiment, the leg angle estimation unit 114 estimates the user's height and / or eye level from the upper leg length x and the lower leg length y using statistical information on human body dimensions. A table may be provided that preliminarily associates the user's height and / or eye level with the upper leg length x and the lower leg length y.

[0117] <<Posture information of the arm position detected by the sensor of controller 3>> FIG. 17 is a functional configuration diagram for detecting the posture of the arm by the controller and the terminal.

[0118] The system shown in Fig. 17 further includes a (hand) controller 3 that the user holds in their hand or wears on their arm. Specifically, it may be, for example, a MetaQuest Touch Pro (registered trademark) controller. It may also be, for example, a glove-type controller that is worn on the arm (see, for example, Non-Patent Document 8). The controller 3 in FIG. 17 also includes an attitude detection unit 31 and an attitude transmission unit 32, similar to the controller 3 in FIG. The attitude detection unit 31 detects the spatial position of the controller 3. Specifically, for example, the position of the controller itself is determined from images captured by a plurality of cameras and tracked. The attitude transmitting unit 32 transmits the spatial position of the controller 3 itself to the terminal 2.

[0119] In contrast to this, the terminal 2 further includes a posture estimation unit 23. The posture estimation unit 23 detects posture information of the user's arm from the spatial position of the controller 3 itself and the spatial position of the terminal. The model display unit 21 displays the arms of the human model in a predetermined direction and / or angle. Furthermore, the body schema display unit 22 displays the difference between the posture information of the user's arms and the posture information of the human model's arms as body schema information.

[0120] FIG. 18 is an explanatory diagram showing detection of the posture of the arm by inverse kinematics.

[0121] 18, the controller 3 (or a hand capture or a sensor that can be attached to the upper limb) detects the position and posture of the user's arm as seen from the head-mounted display terminal 2. This makes it possible to estimate the posture (pose) of the user's upper limb. According to FIG. 18, the coordinate system is expressed as follows: Σ0: Coordinate system based on terminal 2 Σlh: coordinate system fixed to the user's left hand Σrh: coordinate system fixed to the user's right hand The relationship between the position and posture of the coordinate systems Σlh and Σlh is acquired by the controllers 3 held in the left and right hands or attached to the arms. Then, the following attitude can be determined by inverse kinematic calculation (see, for example, Non-Patent Document 7). Wrist (Σlw,Σrw) Elbow (Σle,Σre) Shoulders (Σls,Σrs)

[0122] The length of each part of the arm and the positional relationship between the terminal 2 and the joints differ depending on the physique of the user, but can be estimated, for example, by the following method. Specifically, while measuring the position and posture of the user's arms using the controller 3, the user moves each part of the upper limb to obtain a characteristic trajectory (the relationship between the coordinate system Σ0 and the coordinate systems Σlh and Σrh) according to the user's physique, such as the length of each part. The length of each part can then be estimated by comparing it with a human skeletal model that assumes the length of each arm part and the length of the wrist. One of the simplest and most effective methods involves moving the wrist, elbow, and shoulder joints independently and sequentially up, down, left, and right. Since the trajectory of the hand traces a simple arc, the position of the joints (i.e., the position of each part's coordinate system) can be easily estimated as the center point of the arc. Once the length of each part is determined, the pose of the user's upper limbs can be calculated and measured using inverse kinematics calculations in robotics. In addition, the position and orientation of the terminal 2 (head-mounted display) in real space can also be detected by the built-in sensor, making it possible to measure the movement of the upper body, including the head, as a body schema. The length of each part may be estimated only once. By recording it as user information, it can be recalled and used from the second time onwards, without impairing convenience.

[0123] <<User posture information captured by a camera>> In another embodiment of the present invention, the measurement device 1 may be a camera (not shown) that captures a posture image of the user. The posture image captured by the camera is transmitted to the terminal 2. The camera may be mounted on the measurement device 1 or the terminal 2, or may be provided externally. The posture estimation unit 23 of the terminal 2 detects posture information (arms and / or legs) of the user from the posture image captured by the camera. The model display unit 21 of the terminal 2 displays the arms and / or legs of the person-to-be model in a predetermined direction and / or angle. The body schema display unit 22 displays the difference between the received posture information of the user and the posture information of the person-to-be model as body schema information.

[0124] FIG. 19 is a view diagram displayed by the training application on the body schema display.

[0125] In FIG. 19, the human figure in the shaded area represents the imitation model, and the human figure in the white area represents the real human model that represents the user's own posture, and they are displayed superimposed on each other. The user can visually recognize the body schema by observing the discrepancy between the models. The position of the arms is detected by the controller 3 that the user holds in their hand or wears on their arm. The user moves their arms so that they are in the same position as the imitation model.

[0126] Examples of training applications include the following games:

[0127] (1) Walking Game The user steps in sync with the dummy model walking in the virtual space. For example, a high score can be obtained by stepping with the same walking cycle as the dummy model. Such continuous stepping exercise is expected to be effective as exercise for the user's lower body. In addition, changes in the surrounding scenery in the virtual space (e.g., the seasons) can also be expected to have a positive effect on the user's cognitive function.

[0128] (2) Balance ball game The moving ball placed on the board is carried to the target area in conjunction with the user's center of gravity. Here, the virtual model is the ball on the board, and the user balances the center of gravity of their body to prevent the ball from rolling off the board. Before the game starts, the user sits upright and calibrates the board (measuring the reference load value mentioned above) so that the ball is placed in the center of the board. The user then tilts their body's center of gravity, causing the ball to roll on the board. Points are determined based on the time the ball is kept on the board. Since the user tilts their body back and forth and left and right to balance their center of gravity, this game is expected to be effective as a training for core balance. In particular, since the user must imagine the ball's movement and change the center of gravity of their body, it can simultaneously train both cognitive and motor functions.

[0129] (3) Posture Matching Game The user tries to match their posture to the posture of the human model as closely as possible. The longer the posture matches, the higher the score. Posture includes stride cycle, center of gravity, leg angle, and hand and arm angle. For example, in elderly rehabilitation, a human model is operated as follows to train the shoulder joint to move widely. (s1) Before starting, stand with both arms extended in front of you. (s2) With both arms outstretched, slowly move them horizontally. When the arms are opened to a predetermined angle, they remain in that position for a predetermined period of time. (s3) With both arms extended, slowly return both arms to an extended position. (s4) Next, with both arms extended, slowly move them upwards. (s5) When both arms are raised to a predetermined angle, they remain in that position for a predetermined period of time. (s6) With both arms extended, slowly return both arms to an extended position.

[0130] As described above in detail, the system of the present invention can detect the relationship between physical function and cognitive function.

[0131] Based on the various embodiments of the present invention described above, those skilled in the art can easily make various changes, modifications, and omissions within the scope of the technical spirit and scope of the present invention. The above description is for illustrative purposes only and is not intended to be limiting in any way. The present invention is limited only by the claims and their equivalents. [Explanation of symbols]

[0132] 1. Measuring equipment 11 Attitude detection unit 111 Left load sensor 112 Right load sensor 113 Depth Sensor 114 Leg angle estimation section 12 Attitude transmitter 133 High-pass filter 134 Load difference calculation section 135 Non-walking determination unit 136 Step cycle determination unit 143 Low-pass filter 144 Reference load value memory unit 145 Load value calculation section 146 Center of gravity location identification part 2. Terminal 21 Model display section 22 Body schema display 23 Posture estimation section 3 Controller 31 Attitude detection unit 32 Attitude transmitter

Claims

1. A system having a measurement device and a terminal equipped with a display, The measuring device is posture detection means for detecting posture information of a user; posture transmission means for transmitting posture information of the user to the terminal; and The terminal is a model display means for displaying a human model representing predetermined posture information on a display and instructing the user to take the same posture and movement as the human model; a body schema display means for displaying a difference between the received posture information of the user and the posture information of the person-to-person model on a display as body schema information; A system comprising:

2. The display of the device is a head-mounted type that can be worn on the head.

2. The system of claim 1.

3. The body schema display means of the terminal displays the imitation human model expressing the predetermined posture information and the real human model expressing the received posture information of the user side by side or overlaid. The system of claim 1 .

4. The posture detection means of the measurement device detects posture information of the user's legs using a sensor installed in a position close to the user's legs.

2. The system of claim 1.

5. The posture detection means of the measurement device is one or more load sensors that measure a right load value of the right foot and a left load value of the left foot in time series, The terminal is a step period determination means for determining the step period of the user's stepping based on the time series of right load values ​​and left load values; and The model display means displays the human model as if it is walking with a predetermined walking cycle; The body schema display means displays the difference between the walking cycle of the user and the walking cycle of the model as body schema information.

5. The system of claim 4.

6. As a measuring device, one or more load sensors that measure a right load value of the right foot and a left load value of the left foot in time series; a step period determination means for determining the step period of the user's stepping based on the time series of right load values ​​and left load values; and the posture transmitting means transmits the step period as posture information of the user; As a terminal, The model display means displays the human model as if it is walking with a predetermined walking cycle; The body schema display means displays the difference between the walking cycle of the user and the walking cycle of the model as body schema information.

5. The system of claim 4.

7. The measuring device is a high-pass filter that passes only frequencies equal to or higher than a first predetermined frequency with respect to the time-series right and left weight values ​​and outputs left high-pass weight values ​​and right high-pass weight values; a load difference calculation means for calculating a load difference between the left high-pass load value and the right high-pass load value; 7. The system according to claim 5, further comprising:

8. a non-walking determination means, disposed before the step period determination means, for determining that the subject is in a non-walking state when both the right load value and the left load value are equal to or greater than a third predetermined threshold value or equal to or less than a fourth predetermined threshold value at the same time; and The load difference calculation means determines that the state is walking when the absolute value of the load difference from the non-walking state exceeds a predetermined threshold value (for example, 15 N), or when the absolute value of the difference between the value at the "peak" of the load difference and the value at the "valley" from the non-walking state exceeds a predetermined threshold value.

7. The system according to claim 5 or 6.

9. The posture detection means of the measurement device is one or more load sensors that measure a right load value of the right foot and a left load value of the left foot in time series, The terminal is a low-pass filter that passes only frequencies equal to or lower than a second predetermined frequency with respect to the right weight value and the left weight value over time, and outputs a right low-pass weight value and a left low-pass weight value; a reference load value storage means for storing right and left load values ​​in a seated state in which the user has balanced the left and right sides as right and left reference load values; an applied load value calculation means for calculating a right applied load value which is the difference between the right low-pass load value and the right reference load value, and a left applied load value which is the difference between the left low-pass load value and the left reference load value; a center-of-gravity position specifying means for specifying the left and right center-of-gravity positions based on the difference between the right applied load value and the left applied load value; and The model display means displays the pose of the pose model with a predetermined center of gravity; The body schema display means displays the difference between the center of gravity of the user and the center of gravity of the representative model as body schema information.

5. The system of claim 4.

10. The measuring device is The posture detection means includes one or more load sensors that measure the right load value of the right foot and the left load value of the left foot in time series, a low-pass filter that passes only frequencies equal to or lower than a second predetermined frequency with respect to the right weight value and the left weight value over time, and outputs a right low-pass weight value and a left low-pass weight value; a reference load value storage means for storing right and left load values ​​in a seated state in which the user has balanced the left and right sides as right and left reference load values; an applied load value calculation means for calculating a right applied load value which is the difference between the right low-pass load value and the right reference load value, and a left applied load value which is the difference between the left low-pass load value and the left reference load value; a center-of-gravity position specifying means for specifying the left and right center-of-gravity positions based on the difference between the right applied load value and the left applied load value; and The posture transmitting means transmits the center of gravity position as posture information of the user; As a terminal, The model display means displays the pose of the pose model with a predetermined center of gravity; The body schema display means displays the difference between the center of gravity of the user and the center of gravity of the representative model as body schema information.

5. The system of claim 4.

11. The center-of-gravity position specifying means specifies the left and right center-of-gravity positions from a subtraction weight value obtained by subtracting the left applied weight value from the right applied weight value or subtracting the right applied weight value from the left applied weight value.

11. The system according to claim 9 or 10.

12. The center of gravity position specifying means specifies the front and rear center of gravity positions by adding the right applied load value and the left applied load value. The system of claim 11 .

13. The center of gravity position specifying means uses the following coordinate axes: As both the right and left applied load values ​​increase, the center of gravity position moves forward, and as both decrease, the center of gravity position moves backward. The greater the right load value is compared to the left load value, the more the center of gravity position moves to the right, and the greater the left load value is compared to the right load value, the more the center of gravity position moves to the left. The system of claim 11 .

14. The measurement device is a chair-like piece of furniture that comes into contact with the user's body, The load sensor is placed on the seat of the furniture-type device, where the pressure of the user's thighs is transmitted, and measures the right and left load values ​​based on the user's footsteps while seated.

11. The system according to claim 5, 6, 9 or 10.

15. The measuring device is a plate-type device or a shoe-type device on which the user stands and presses the sole of the foot, The load sensor is placed in the plate-type device or the shoe-type device at a portion where the pressure of the sole of the user's foot is transmitted, and measures the right load value and the left load value based on the stepping in the standing state.

11. The system according to claim 5, 6, 9 or 10.

16. The attitude detection means of the measurement device includes: a depth sensor disposed at a position below the seat of the measurement device, for measuring a planar depth distribution around the lower legs of a seated user; a leg angle estimation means for estimating the posture of the lower leg and the posture of the upper leg based on the depth distribution and estimating the direction and / or angle of the lower leg and the upper leg; and outputs the orientation and / or angle of the lower leg and upper leg as posture information of the user.

5. The system of claim 4.

17. the measurement device is a controller that the user holds in his / her hand or wears on his / her arm, The attitude detection means detects a spatial position of the controller; The attitude transmitting means transmits the spatial position of the controller to the terminal; As a terminal, further comprising a posture estimation means for estimating posture information of the user from the spatial position of the controller and the spatial position of the terminal; the model display means displays the arms of the human model in a predetermined direction and / or angle; The body schema display means displays the difference between the posture information of the user's arm and the posture information of the human model's arm as body schema information.

2. The system according to claim 1, wherein:

18. The measuring device is the posture detection means is a camera that captures a posture image of the user, The posture transmission means transmits the posture image to the terminal; As a terminal, The apparatus further includes a posture estimation means for estimating posture information of the user from a posture image of the camera, the model display means displays the arms of the human model in a predetermined direction and / or angle; The body schema display means displays the difference between the posture information of the user's arm and the posture information of the human model's arm as body schema information.

2. The system according to claim 1, wherein:

Images