Information processing system, information processing method and information processing program

Abstract

To effectively improve a user's cognitive ability and athletic ability.SOLUTION: An information processing system includes a blocking part for preventing all or a part of a user's body from being seen, a request part for requesting, from the user, such a body motion that overlaps the part of the user's body or an object operated by the user and a target object, and a control part for controlling a degree of difficulty of the body motion in accordance with the movement of a joint different from a body part performing the body motion under an environment in which an abnormal motion of a joint generated by the blocking part and the request part is revealed.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing method, and an information processing program. [Background technology]

[0002] In the above technical field, Patent Document 1 discloses a system for treating hemiplegic patients caused by stroke or the like. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-228957 A [Non-patent literature]

[0004] [Non-Patent Document 1] Nature "A somato-cognitive action network alternates with effector regions in motor cortex" by Evan M. Gordon et al. Published online:19 04 2023 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology described in the above document does not provide an indication of what kind of movement or what kind of movement the user should make to proceed to the next step. On the other hand, as disclosed in Non-Patent Document 1, it has been discovered that there is a brain nerve region in the neural circuit in the brain that controls coordinated movements that link multiple joints.

[0006] An object of the present invention is to provide a technique for solving the above-mentioned problems. [Means for solving the problem]

[0007] In order to achieve the above object, the system according to the present invention comprises: a blocking portion that blocks all or part of a user's body from view; a request unit that requests the user to perform a body motion such that a part of the user's body or an object operated by the user overlaps with a target object; a control unit for controlling a difficulty level of the physical movement in accordance with a movement of a joint different from a body part performing the physical movement in an environment that makes abnormal movements of the joints apparent, the environment being generated by the blocking unit and the request unit; and It is an information processing system equipped with the above. In order to achieve the above object, the method according to the present invention comprises: A blocking step of blocking all or part of the user's body from view by a blocking portion; a request step of requesting the user to perform a body motion such that a part of the user's body or an object operated by the user overlaps with a target object by a request unit; a control step in which a control unit controls a difficulty level of the physical movement in accordance with a movement of a joint different from a body part performing the physical movement under an environment that makes abnormal movements of the joints apparent, the environment being generated by the blocking step and the request step; The present invention relates to an information processing method. In order to achieve the above object, the program according to the present invention comprises: a blocking step of obscuring all or part of the user's body from view; a request step of requesting the user to perform a body motion such that a part of the user's body or an object operated by the user overlaps with a target object; a control step for controlling a difficulty level of the physical movement in accordance with a movement of a joint different from a body part performing the physical movement in an environment that makes abnormal movements of the joints apparent, the environment being generated by the blocking step and the request step; It is an information processing program that causes a computer to execute the above. Effect of the Invention

[0008] According to the present invention, it is possible to effectively improve the cognitive or motor skills of a user. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration of an information processing system according to a first embodiment. [Figure 2A] FIG. 11 is a block diagram showing the configuration of a rehabilitation support system according to a second embodiment. [Figure 2B] FIG. 11 is a diagram for explaining the configuration of a rehabilitation support system according to a second embodiment. [Figure 2C] FIG. 11 is a diagram for explaining the configuration of a rehabilitation support system according to a second embodiment. [Figure 3A] FIG. 11 is a diagram showing an example of an operation panel screen of the rehabilitation support system according to the second embodiment. [Figure 3B] FIG. 11 is a diagram showing another example of the operation panel screen of the rehabilitation support system according to the second embodiment. [Figure 4A] FIG. 11 is a diagram showing an example of a task data table of the rehabilitation support system according to the second embodiment. [Figure 4B] FIG. 11 is a diagram showing another example of the task data table of the rehabilitation support system according to the second embodiment. [Diagram 5] FIG. 11 is a diagram showing an example of a display screen on a head mounted display of the rehabilitation support system according to the second embodiment. [Figure 6] FIG. 11 is a diagram showing an example of a display screen on a head mounted display of the rehabilitation support system according to the second embodiment. [Figure 7] FIG. 11 is a diagram showing an example of a display screen on a head mounted display of the rehabilitation support system according to the second embodiment. [Figure 8] FIG. 11 is a diagram showing an example of a display screen on a head mounted display of the rehabilitation support system according to the second embodiment. [Figure 9] 13 is a flowchart showing a processing flow of the rehabilitation support system according to the second embodiment. [Figure 10] 13 is data showing the results of rehabilitation performed by the rehabilitation support system according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. However, the components described in the following embodiment are merely examples, and are not intended to limit the technical scope of the present invention.

[0011] [First embodiment] An information processing system 100 according to a first embodiment of the present invention will be described with reference to Fig. 1. The information processing system 100 is a system that provides an environment for making abnormal movements of joints apparent.

[0012] 1, the information processing system 100 includes a blocking unit 101, a request unit 102, and a control unit 103. The blocking unit 101 blocks all or part of the body of a user 110 from view. The request unit 102 requests the user 110 to perform a physical movement such as overlapping a part of the body of the user 110 or an object 121 operated by the user with a target object 111. The control unit 103 controls the difficulty of the physical movement according to a movement 131 of a joint different from the body part where the user 110 performs the physical movement.

[0013] With the above configuration, it is possible to reveal abnormal movements of joints other than the body part where the user performs physical movements, and ultimately to use the abnormal movements as an indicator to effectively improve the user's cognitive or motor abilities.

[0014] [Second embodiment] A rehabilitation support system 200 according to a second embodiment of the present invention will be described with reference to FIG. 2A. The rehabilitation support system 200 is an example of a system that provides an environment that makes abnormal joint movements apparent, and the present invention is not limited thereto. In other words, the present invention should not be limited to the concept of "rehabilitation", and the present invention also includes a treatment system that actively treats physical disorders such as paralysis and ataxia, dementia and schizophrenia, and cognitive and mental disorders. Furthermore, the present invention also includes a motion verification system, ability improvement system, and improvement system for improving the motor and cognitive abilities of healthy individuals who do not require any treatment or rehabilitation from their current state. 2A is a diagram for explaining the configuration of a rehabilitation support system 200 according to this embodiment. The rehabilitation support system 200 according to this embodiment improves at least one of physical functions such as upper limb function, walking function, trunk function, and balance function, cognitive function (including spatial cognition, attention function, and higher brain function), and sensory function (including the inner ear, vestibular system, touch, thermo-pain sensation, position sense, and deep sensation).

[0015] As shown in FIG. 2A, the rehabilitation support system 200 includes a rehabilitation support device 210 as a request unit and a control unit, two base stations 231 and 232, a head-mounted display 233 as a blocking unit, and two controllers 234 and 235. A user 220 wears the head-mounted display 233 while sitting on a chair 221, and moves his / her body in accordance with the display on the head-mounted display 233. In this embodiment, the description is based on the premise of verifying an action performed while sitting on a chair, but the present invention is not limited to this. The action may be performed while standing, while walking, while on a bed, while lying on one's back or on one's back, while running, or while performing other specific actions. In this embodiment, the user 220 holds the controllers 234 and 235 with both hands, but the present invention is not limited to this. The controllers may be held or worn on parts of the body other than the hands, such as the feet or the trunk.

[0016] The two base stations 231, 232 detect the movement of the head mounted display 233 and the movements of the controllers 234, 235 and send them to the rehabilitation support device 210. The rehabilitation support device 210 controls the display on the head mounted display 233 based on the movement of the head mounted display 233. Specifically, the rehabilitation support device 210 changes the position and direction of the viewpoint in the virtual space according to the position and direction of the head mounted display 233 and determines the image to be displayed on the head mounted display 233. In addition, the rehabilitation movement of the user 220 is evaluated based on the movements of the controllers 234, 235. The head mounted display 233 may be a non-transparent type, a video see-through type, an optical see-through type, or a glasses type. In this embodiment, a virtual space of VR (Virtual Reality) is presented to the user, but the real space and the virtual space may be superimposed as in AR (Augmented Reality), real information may be reflected in the virtual space as in MR (Mixed Reality), or hologram technology may be used as an alternative. What is important here is that the head mounted display 233 makes all or part of the user's body invisible. Usually, in real space, humans visually recognize the movement of their body while unconsciously fine-tuning the movement in their brain and moving it to the target. In this system, the user 220 cannot directly see his / her own hand in real space, so the user's brain cannot accurately correct the movement of the hand as in normal cases. Therefore, coordination disorders become apparent, and abnormal movements of joints other than those that perform body movements become easier to detect. In this case, for example, a projection mapping system that displays a background on all or part of the user's body may be adopted as the blocking unit. Also, a contact lens type display that displays a virtual space may be adopted as the blocking unit.

[0017] In this embodiment, the controllers 234, 235 held by the user 220 and the base stations 231, 232 are shown as examples of sensors for detecting the position or movement of the user's hands or head, but the present invention is not limited to this. A camera (including a depth sensor) for detecting the "position of the hand itself" or movement of the user by image recognition processing, a sensor for detecting the position of the user's hand by temperature, a wristwatch-type wearable terminal worn on the user's arm, motion capture, etc. can also be applied to the present invention by linking with the movement detection unit 211. In other words, one embodiment is to use a three-dimensional tracking device or movement analysis device such as Kinect (registered trademark), or to wear a marker or the like on the body.

[0018] The rehabilitation support device 210 includes a movement detection unit 211 , display control units 212 and 213 , a feedback unit 214 , an evaluation update unit 215 , a task set database 216 , a setting unit 217 , and a movement guidance unit 218 .

[0019] The motion detection unit 211 acquires the positions of the controllers 234, 235 held by the user 220 via the base stations 231, 232, and detects the rehabilitation motion of the user 220 from a change in the position of the user's 220's hands.

[0020] The display control unit 212 generates and displays target objects 242a, 242b in the virtual space 240 to prompt the user 220 to perform a three-dimensional body movement. In particular, the display control unit 212 generates a target object 242a for performing a three-dimensional rehabilitation movement of a left body part of the user 220, and a target object 242b for performing a rehabilitation movement of a right body part of the user 220, in the virtual space 240. In other words, the display control unit 212 functions as a request unit that requests the user to reach for the target object.

[0021] In this embodiment, the user is requested to reach a virtual target displayed on the head-mounted display, but the present invention is not limited to this, and an object in the real world may be used as the target. The target may be an object suspended by two or more wires, or an object suspended in the air by ultrasonic vibration. The target may also be an object displayed three-dimensionally by a hologram. The shape of the target object as the target is not limited to a sphere, but may be a triangle, a square, or a dish. It may also be the shape of some character. The concept of the present invention also includes a system that partially blocks the user's vision by using an optical see-through or smart glass type head-mounted display to make the user's own movements invisible, while allowing the user to see only the target.

[0022] The display control unit 212 generates target objects 242a, 242b in the virtual space and moves them from above the user 220 downward. As a result, the target objects are displayed on the head mounted display 233 so that their display position and size change gradually (for example, they gradually become larger and then smaller). The movement direction of the target objects may be, for example, an upward direction from the floor surface to the overhead direction. The target objects may move in any three-dimensional manner, including movement in the depth direction or left and right direction in addition to movement in the up and down direction, or may be fixed at a certain coordinate position without moving depending on the user's stage.

[0023] Furthermore, the display control unit 212 generates avatar objects 241a and 241b in the virtual space that move according to the detected user motion. The avatar object 241a here is an object that moves in response to the controller 234 operated with the left hand, and represents the position of a part of the user's body on the left side. The avatar object 241b is an object that moves in response to the controller 235 operated with the right hand, and represents the position of a part of the user's body on the right side. The avatar object is not limited to an object that moves in response to the position of the controller. It may be a virtual object that moves with the change in the position of a part of the user's body using the three-dimensional body tracking technology described above. The object operated by the user is also not limited to the controller as shown in the figure, and may be, for example, a bat or a golf club. The display control unit 212 requests the user to perform a body motion that causes the avatar objects 241a and 241b to overlap the target objects 242a and 242b.

[0024] Even if the avatar objects 241a and 241b are displayed, the burden on the user's brain is greater when a part or all of the body is blocked and cannot be seen, compared to when the actual body movement is directly visible. This is because the brain needs to estimate the position (point) of the controller in the real space by looking at the position (point) of the controller in the virtual space when the user's body (including the controller in the real space) cannot be seen (so-called point estimation). In this case, the brain needs to imagine the state of the body quite strongly without relying on vision, which is medically expressed as requiring strong feedforward. A brain that requires strong feedforward finds it more difficult to coordinate the body, which makes coordination disorders more apparent and makes it easier for abnormal movements of joints other than those that perform body movements to occur. In addition, strong feedforward that requires point estimation can induce contraction of deep muscles of the body by stimulation via the pyramidal tract.

[0025] For example, when moving the right arm to move the avatar image 241b on the right controller 235 to the target object 242b on the right side, the left knee joint may be unconsciously forced, causing the lower leg of the left leg to move forward or backward. Here, such a phenomenon in which a joint not necessary for a desired movement moves unintentionally is called articular linkage, and is used as an index for improving the user's movement. Note that this articular linkage is manifested even without an anatomical connection called anatomical train. For example, when the joints of the toes are moved voluntarily, abnormalities may occur in the jaw joint, hip joint, or shoulder joint.

[0026] As also mentioned in Non-Patent Document 1, this phenomenon can be considered to be supported by the fact that the motor areas that control coordinated movements exist across all joints (FIG. 2B). Looking at FIG. 2B, it can be seen that in the latest cranial neurology, the cranial nerves are grouped into groups for symmetrical parts (such as the right hand and the left hand, the right foot and the left foot, and the right eye and the left eye), and that these motor areas are arranged symmetrically in the brain. Furthermore, it has been newly discovered that there seems to be neural areas that control the coordination of each part of the body shown in the puppets 264-266 between the foot motor area 261 and the hand motor area 262, and between the hand motor area 262 and the face motor area 263. Due to the existence of nerves that coordinate each part of the body like the puppets 264-266, it is considered that the joint coordination during movement becomes apparent in a specific environment in which point estimation is required in a situation where all or part of the body according to the present invention is not visible.

[0027] The inventors had in fact created the present invention by observing the rehabilitation movements of actual users before the publication of Non-Patent Document 2, but it can be said that the usefulness and logic of the present invention were theorized by the paper in Non-Patent Document 1.

[0028] In Fig. 2A, the movement of the knee joint is illustrated as an example of the generation of a joint connection, but the present invention is not limited to this, and all involuntary movements of joints other than the body part performing the requested movement are included in the concept of the present invention. For example, as shown in Fig. 2C, by detecting joint planes 271 and 272 and calculating their inclinations, the simultaneous occurrence of twisting of the shoulder joint or trunk (cervical joint, thoracic joint) and twisting of the waist (lumbar joint, hip joint) may be detected.

[0029] Specifically, joint interactions include twisting of the cervical intervertebral joint (appears in the direction of the head), twisting of the thoracic intervertebral joint (appears in the angle of the shoulder), elevation and abduction of the shoulder joint (opening of the armpits), extension and contraction of the knee joint, hip joint, and ankle joint (the knee and toes move up and down, inverted, and abducted), extension and contraction of the elbow joint (the arm that is not required to move is fixed in a flexed position), extension of the jaw joint (mouth opening), flexion and extension of the internal joints of the hand and finger joints, flexion and extension of the toe joints, etc. Such joint movements are not limited to flexion and extension, but also include adduction, abduction, internal rotation, external rotation, circumduction, or combinations of these, and movements not described here.

[0030] The system may have a joint connection evaluation unit 219 that detects and evaluates the position, twist, inclination, speed of movement, acceleration, etc. of each joint of the user during rehabilitation movement by installing a camera 236. The joint connection evaluation unit 219 detects the movement of each joint of the user 220 and evaluates whether or not joint connection is occurring based on the magnitude, speed, timing, etc. of the movement. The system requests the user to perform rehabilitation movements that reduce such joint connection, and has the user achieve them. If the brain is likened to a CPU, if the information processing process that is allocated 80% of the calculation capacity of the brain to joint connection is reorganized to 20, more complex movements can be performed and motor ability is improved. Alternatively, the idea is that cognitive ability will also improve if the CPU capacity allocated to cognitive processing is improved. By photographing the user with the camera 235 and comparing it with an image of a normal time (before exercise) or with a normal movement model, abnormal twisting or movement of the joint surface can be detected. In addition, the joint connection may be detected by attaching sensors (or by motion capture) to the user's joints (for example, both shoulders, both elbows, wrists, both knees, both ankle joints, etc.) and detecting the positions of the joints with the base stations 231 and 232. Of course, the operator may visually recognize the occurrence of the joint connection and input it to the system. The inclination of the head mounted display 233 may be detected with the base stations 231 and 232 to detect the joint connection of the neck. For example, if the head mounted display 233 is inclined at 30 degrees, it can be determined that the trunk is inclined at 15 degrees (determined by a camera image) and the neck is inclined at 15 degrees. The inclination of the wrist joints, etc. can be determined by detecting the positions and inclinations of the controllers 234 and 235 with the base stations 231 and 232. Furthermore, the joint connection evaluation unit 219 may display the movement as the joint connection as a distance, acceleration, etc. in a bull's eye heat map, or may display the change over time of the joint connection to provide information useful for controlling the difficulty of the movement. Specifically, it is possible to visualize how the joint connections change from the start to the end of a required reaching movement, and to compare the magnitude of the joint connections of each joint. This makes it easier to determine which joints to focus on.

[0031] It is possible to focus on detecting six joints: hip joint, knee joint, ankle joint, neck joint, trunk joint, and elbow joint. It is also possible to divide into more categories and look at each joint unit (68 in humans), or conversely, to focus on only one or a few joints where abnormalities are most easily detected. Furthermore, multiple evaluation indices may be combined into a single integrated index or parameter as a total score, etc. In other words, it is possible to weight and integrate each joint connection of the whole body or an important part, and perform a total joint connection evaluation (calculation of a joint connection score).

[0032] If the required action is to place a 2cm sensor on a 10cm target, the accuracy of point estimation is low, but if the required action is to place a 1cm sensor on a 1cm target, high accuracy of point estimation is required. In this case, if the user uses an obstruction such as a head-mounted display, the visual information of the real world is blocked, which puts the user in a restricted state, and the brain nerves for point estimation need to be sharpened, resulting in impaired coordination, i.e., joint coupling becomes more pronounced. This is one of the mechanisms of joint coupling.

[0033] As the rehabilitation level (user's motor ability) increases, the required accuracy of point estimation is increased by reducing both the size of the target and the sensor, and compensatory joint coupling is made more apparent. In this state, the joint coupling is relieved by repeatedly requesting rehabilitation movements, or by guiding the user to perform the correct movements more easily through verbal encouragement or physical contact. This results in the whole body being realigned, and coordination disorders are alleviated and improved.

[0034] The joint connection evaluation unit 219 may display the position and size of the detected joint connection on an operator screen (FIG. 3) or the like. The apparatus may further include a comparison unit that performs a comparison with past joint connections or a comparison with a normal motion model, and notify the operator of the comparison results. This allows the operator to recognize changes in the joint connection with higher accuracy.

[0035] If it is determined that the joint connection has become smaller than a certain standard, the load of the rehabilitation movement is increased, and if it is determined that the joint connection has become larger than a certain standard or that the increased state continues (the joint connection has not improved), the load of the rehabilitation movement is decreased.

[0036] Whether or not joint connection occurs is influenced by the size, angle, height, distance, direction of movement, presence or absence of movement, speed of movement, color, shape, visibility, the size, color, shape, visibility, background information, presence or absence of background sound, etc. of the device or indicator for superimposing by point estimation such as the controller. The goal of the rehabilitation operation in this system is to cause joint connection and calm it down. In other words, joint connection is the guideline for treatment. Joint connection may be loosened by pressing the user's knee or shoulder during rehabilitation operation. In this way, it is possible to manually and forcibly correct frayed neural circuits in the brain.

[0037] Returning to FIG. 2A, images of the avatar objects 241a, 241b and the target objects 242a, 242b are displayed on the display screen 240 according to the orientation and position of the head mounted display 233 detected by the motion detection unit 211. The images of the avatar objects 241a, 241b and the target objects 242a, 242b are displayed superimposed on the background image 243. Here, the avatar objects 241a, 241b have the same shape as the controllers 234, 235, but are not limited to this, and may be in the shape of a hand. Furthermore, the size, shape, and color may be different between the left and right. The controllers 234, 235 are provided with one or more buttons, and are configured so that various settings including initial settings such as origin setting can be performed by operating the buttons, but the button function may be disabled or the button itself may not be provided, and all settings may be performed using a separate external operation unit. The background image 243 is cut out from a virtual space including a horizon 244 and a ground surface object 245. Depending on the movement of the line of sight (the position and direction of the head mounted display), the appearance of the background image 243, the avatar objects 241a, 241b, and the target objects 242a, 242b changes.

[0038] The display control unit 212 generates an avatar object indicating the position of a part of the body on the right side and a target object indicating the target position of the avatar object on the right side in similar colors or shapes. The display control unit 212 also generates an avatar object indicating the position of a part of the body on the left side and a target object indicating the target position of the avatar object on the left side in similar colors or shapes. It is preferable to set the color or shape of the left avatar object and the left target object to be different from the color or shape of the right avatar object and the right target object. However, even if they are on the same side, the avatar object and the target object do not necessarily need to be set to similar colors or shapes, and there is no need to separate the colors for the right side and the left side. This may be set in any way depending on the user's sense of use.

[0039] Here, for example, the avatar objects 241a and 241b are color-coded, for example, blue and red, and the target objects 242a and 242b are also color-coded, blue and red. The task is accomplished by touching the blue target object 242a with the blue avatar object 241a. Similarly, the task is accomplished by touching the red target object 242b with the red avatar object 241b. That is, the task is not accomplished by touching avatar objects of different colors. Here, color coding of blue and red is shown as an example, but other color coding (yellow and green, etc.) may be used for color-blind users, or shapes, linguistic notations such as "left", "right", "L (Left)", and "R (Right)", symbols such as stars, triangles, and circles, or different patterns (stripes) such as 242a and 242b in the figure may be used to distinguish between the objects.

[0040] The movement inducing unit 218 induces the user to alternate between the rehabilitation movement of the part of the user's right body and the rehabilitation movement of the part of the user's left body. Specifically, the movement inducing unit 218 induces the user's rehabilitation movement by displaying characters, colors, or figures representing right and left, outputting voices to distinguish between "right" and "left," or by tactile stimulation using vibrations of controllers held by the user in the left and right hands, touching or tapping the shoulders or arms, etc.

[0041] When the avatar objects 241a and 241b collide with the target objects 242a and 242b, the display control unit 212 makes the target objects 242a and 242b disappear, and the feedback unit 214 performs visual feedback by displaying a message for the purpose of notifying or informing that the target motion has been achieved. If the sensor centers of the avatar objects 241a and 241b reach the centers of the target objects 242a and 242b, "Bravo!" is displayed for each task, and if the sensor centers reach the peripheral areas, "Well done!" is displayed. In this way, the achievement of each requested physical motion is fed back to the user. The feedback unit 214 may provide feedback by stimulating one or more of the user's five senses. If feedback is provided by stimulating two or more of the user's five senses, the user's cognitive ability or motor ability, or both, can be improved more effectively.

[0042] That is, the feedback unit 214 notifies the user 220 of the achievement of each rehabilitation motion for the target object 242a and the target object 242b. Various methods of notification are possible here. Words such as "Bravo" and "Well done" may be temporarily displayed on the display screen to inform the user of the degree of achievement of the target motion, or a similar voice or sound effect may be used to inform the user of the degree of achievement of the target motion. Furthermore, only one of the controllers 234 and 235 that has made a motion to contact the target object may be vibrated at the same time to notify the user of the degree of achievement of the target motion. Alternatively, the achievement may be notified in stages, such as complete achievement, incomplete achievement, and failure to achieve, to notify the degree and degree of achievement.

[0043] For example, when the shortest distance between the sensor object included in the avatar object 241 and the target objects 242a and 242b falls within a predetermined range, the goal is achieved, and the target objects 242a and 242b disappear. At this time, if the shortest distance between the sensor object included in the avatar object 241a and 241b (for example, a spherical object including the center point of the tip of the avatar object 241a and 241b) and the target objects 242a and 242b falls below a threshold, the goal is completely achieved, and for example, "Bravo!" is displayed and a corresponding voice is output as feedback. Such feedback that stimulates a plurality of different senses is called multi-channel biofeedback. At the same time, the controllers 234 and 235 may be vibrated, or a stimulation may be given to the sense of smell or taste. That is, the feedback unit 214 can stimulate any one of the five senses of sight, hearing, touch, taste, and smell as feedback. This sensory stimulation may be a combination of any two, a combination of three or more, or all of them. In a situation where point estimation is required, feedback is given each time a required movement is achieved, which triggers the brain to correct and improve the coordination disorder. In other words, joint connection and coordination disorder improve. The more stimulations of the five senses are given during feedback, the more efficiently the improvement proceeds, or the improvement is maintained for a long time. For example, compared to single sensory feedback, two sensory feedback improves symptoms over a period of several days, but the symptoms improve over a period of hours. Or, compared to two sensory feedback, three sensory feedback improves symptoms over a period of several minutes, but the symptoms improve over a period of several hours. Furthermore, even if a symptom improves with one sensory feedback but returns to normal immediately, the improvement effect is maintained for several days with two sensory feedback. Or, even if a symptom improves with two sensory feedback but returns to normal within a few days, the improvement effect is maintained for several weeks with three sensory feedback.To describe the characteristics of this treatment, it is also referred to in the medical field as "brain reprogramming therapy (BRT)," "brain re-wiring therapy (BRT)," or "motor coordination therapy (MCT)."

[0044] The rehabilitation movement may be evaluated in three or more stages depending on how far the distance between the sensor object and the target objects 242a, 242b has been reduced. Also, two or more target objects may be simultaneously generated and displayed in the virtual space.

[0045] The display control unit 213 displays the radar screen image 250 on the display screen 240 of the head mounted display 233. The radar screen image 250 is an announcement image for announcing the occurrence of the target object 152. The radar screen image 250 announces which direction the positions of the target objects 242a and 242b that will occur are in relative to a reference direction in the virtual space (here, initially set to be the front direction of a user sitting upright in a chair). The radar screen image 250 further announces how far the positions of the target objects 242a and 242b that will occur are from the user 220. Note that the announcement image is not limited to the radar screen image, and may be announced by characters, arrows, symbols, illustrations, types of light or color, strength, blinking, etc. Also, the announcement method is not limited to the image, and may be announced by voice, vibration, or any combination of voice, vibration, and image. The display control unit 213 displays the radar screen image 250 in the center (for example, within a range of -50 degrees to 50 degrees) of the display screen 240 of the head mounted display 233 regardless of the direction of the head of the user 220. However, the display part is not limited to the center, and may be any of the four corners, top end, bottom end, left end, and right end of the screen. The patient can estimate the difficulty of the next exercise action to be performed from the information on the position, angle, and number of target objects displayed on the radar screen, and when a more difficult action is predicted by the patient, the joint connection is more evident. In other words, it can be said that the display control unit also functions as a control unit for controlling the difficulty of the action required of the patient.

[0046] The radar screen image 250 includes a head image 251 representing the user's head as seen from above, a block image 252 obtained by dividing the periphery of the head image 251 into a plurality of blocks, and a fan-shaped image 253 as a visual field area image showing the user's visual field area. A target position image showing the position of a target object is shown by which block of the block image 252 is colored, blinks, or lights up. This allows the user 220 to know whether the target object is on the left side or the right side of the direction in which the user is facing. Note that in this embodiment, the block image 252 is fixed and the fan-shaped image 253 moves, but the present invention is not limited to this. The fan-shaped image 253 and the head image 251 may be fixed, and the block image 252 may be moved according to the direction of the head. Specifically, if the head faces left, the block image 252 may rotate to the right.

[0047] It is preferable that the feedback unit 214 changes the message type according to the evaluation of the rehabilitation movement via the display control unit 212. For example, if the sensor object touches the center of the target objects 242a, 242b, "Bravo!" is displayed, and if the sensor object touches only the peripheral part of the center of the target objects 242a, 242b, "Well done!" is displayed. The size of the target objects 242a, 242b and the size of the peripheral part can be set by the setting unit 217. The size of the sensor object can also be set by the setting unit 217. If the shortest distance between the sensor object included in the avatar object 241a, 241b and the target object 242a, 242b is equal to or less than the first threshold, the message "Bravo!" is displayed as the complete achievement of the goal, and the corresponding voice "Bravo!" is output as feedback. If the shortest distance between the sensor object included in the avatar object 241a, 241b and the target object 242a, 242b is equal to or more than the first threshold and equal to or less than the second threshold, the message "Well done!" is displayed as the achievement of the goal, and the corresponding voice "Well done!" is output as feedback. The voice message output does not have to be the same message, and a non-verbal sound effect such as "pillow" may be used. The shorter the distance required for the contact between the sensor object and the target object, the more accurate the movement required, so the brain needs to request the body to make a more accurate and sharpened movement as a motor command. As mentioned above, this brain image is called feedforward. That is, the smaller the distance between the sensor object and the target object required to complete the action, the stronger the feedforward required, thereby allowing the user's motor, cognitive and sensory load levels to be continuously adjusted.

[0048] The feedback unit 214 performs feedback to stimulate two or more of the five senses (sight, hearing, touch, taste, smell) for the user who virtually touches the target objects 242a, 242b, at the timing when the sensor object enters within a predetermined distance from the center of the target objects 242a, 242b, or at the timing when the sensor object comes into contact with the target objects 242a, 242b (called real-time multi-channel biofeedback or instantaneous multi-signal biofeedback). The effect is high if the delay from the timing to the feedback is, for example, within one second, and the closer the interval between the user's action timing and the feedback timing (the smaller the delay), the greater the effect. The feedback unit 214 performs feedback to stimulate the user's vision by an image saying "Bravo!", and at the same time, performs feedback to stimulate the user's hearing by a voice or sound effect output from a speaker. Notification of task completion for the five senses stimulation may be performed in any combination, for example, according to the type of action.

[0049] Furthermore, the feedback unit 214 may simultaneously output feedback stimulating the vision of the user 220 with an image of "Bravo!", feedback stimulating the hearing of the user 220 with a sound output from a speaker, and feedback stimulating the tactile sense of the user 220 by vibrating the controller 234. Also, the feedback unit 214 may simultaneously output only two types of feedback: feedback stimulating the vision of the user 220 with an image of "Bravo!", and feedback stimulating the tactile sense of the user 220 by vibrating the controller 234. Alternatively, the feedback unit 214 may simultaneously output only two types of feedback: feedback stimulating the hearing of the user 220 with a sound of "Bravo!", and feedback stimulating the tactile sense of the user 220 by vibrating the controller 234.

[0050] The motion of the user 220 moving the controllers 234 and 235 in this way is a rehabilitation motion, and the display of a target object that prompts the user 220 to perform one rehabilitation motion is called a task or a task. Information (task data) representing one task includes the direction of appearance of the target object (90 degrees right, 45 degrees right, front, 45 degrees left, and 90 degrees left relative to the front direction of the chair), the distance to the target object, the shape (size), the appearance position (distance and angle from the user), the appearance interval (time interval), the moving speed of falling or rising, the size, the color, which controller should be used to acquire the data, the number of target objects appearing simultaneously, the size of the sensor object, and the like. In other words, the task data includes at least one of the moving speed, the number of displayed objects, the size, the display position, and the display interval as attributes of the target objects 242a and 242b in the virtual space 240 as information. The distance in the depth direction from the user 220 to the drop position of the target objects 242a and 242b may be set continuously, or may be set to any of three stages, for example. For example, it is possible to change the location of the object to be dropped right next to the user 220 or to a location that the user 220 cannot reach unless the user 220 leans forward significantly. This makes it possible to control the exercise load applied to the user and the load on the spatial cognition ability or spatial grasping ability.

[0051] The evaluation update unit 215 evaluates the rehabilitation motion of the user 220 according to the quantity and quality of the tasks accomplished by the user 220, and adds points. The quality of the accomplished tasks includes whether the avatar object was “good” or “great”, that is, how close the avatar object was to the target object. The evaluation update unit 215 gives different points (high points for distant objects and low points for close objects) to each accomplished task. The evaluation update unit 215 can update the tasks according to the accumulated points. For example, the task achievement rate (number of goals achieved / number of tasks) may be used to update the task (attributes of the target object). The evaluation update unit 215 compares the rehabilitation motion detected by the motion detection unit 211 with the target position represented by the target object displayed by the display control unit 212, and evaluates the rehabilitation ability of the user 220. Specifically, whether or not avatar objects 241a, 241b, which move in response to the rehabilitation movement detected by the movement detection unit 211, overlap with target objects 242a, 242b is determined by comparing their positions in the three-dimensional virtual space. If they overlap, it is evaluated that one rehabilitation movement has been completed, and points are added. The display control unit 212 can make the target objects 242a, 242b appear at different positions in the depth direction (for example, three stages of positions). The evaluation update unit 215 assigns different points to each of the objects (higher points to distant objects, and lower points to closer objects).

[0052] The evaluation update unit 215 updates the target tasks according to the accumulated points. For example, the target tasks may be updated using a task achievement rate (number of goals achieved / number of tasks).

[0053] The task set database 216 stores a set of a plurality of tasks. A task indicates one rehabilitation movement to be performed by a user. Specifically, as information representing one task, the information stored includes information on the position, speed, and size of a target object that was made to appear, and the size of the avatar object at that time. The task set database 216 stores a task set that determines the order in which such a plurality of tasks are provided to a user. For example, a task set may be stored as a template for each hospital, or a history of executed task sets may be stored for each user. The rehabilitation support device 210 may be configured to be capable of communicating with other rehabilitation support devices via the Internet. In this case, one task set can be executed by the same user at multiple locations, and various templates can be shared among multiple users at different locations.

[0054] The setting unit 217 can set so that the generation of the left target object 242a and the generation of the right target object 242b are performed alternately, and the setting unit 217 sends an instruction to the display control unit 213 according to the setting.

[0055] The setting unit 217 can set the generation position in the depth direction in the virtual space 240 of the left target object 242a and the generation position in the depth direction in the virtual space 240 of the right target object 242b.

[0056] By controlling both the timing of generation of the target object and the vertical moving speed of the target object, the rehabilitation motion of the user is controlled to be always alternately left and right. That is, in the simplest method, the left and right target objects are not generated at the same time, and the left target object is generated after the reaching to the right target object is detected. However, this is not limited to this, and various control methods are possible. That is, the left and right alternating motion may be promoted by generating multiple target objects at the same time and changing the moving speed so that it decreases in the order of right → left → right → left. For example, after the reaching to the right object is detected, the right object and the left object may be generated at the same time, or the right object may be generated first, and the speed of the right object may be increased to cause the right object to overtake the left object. In this way, a high load can be applied to the brain. The request unit or the control unit may control the rehabilitation motion of the user to be alternately left and right.

[0057] On the other hand, when there is paralysis only on the right side or when the left upper body is to be actively rehabilitated, the right target object may be generated at a slow speed, and then the left target object may be generated at a fast speed after an interval. In other words, at least one of the timing and speed of the target object generation is controlled according to the vertical position of the body to be rehabilitated (according to the position of the target object to be reached). In the case of knee or lower body rehabilitation, the timing and speed of the target object generation are controlled so that the reaching is performed at a low position. Also, when it is desired to increase the cognitive load, the target objects are generated in the order of left → right → right, but only the first target object is moved slowly, and the second and third are moved at a fast speed. As a result, it is possible to reach in the order of right, left, right.

[0058] By rewiring the brain in this way, it is possible to improve the symptoms of diseases that are difficult to treat with modern medicine, such as chronic pain and restless legs syndrome (abnormality of the basal ganglia).

[0059] The setting unit 217 also sets at least one of the moving speed, the number of objects to be displayed, the shape, the color, the size, the display position (coordinate information, etc.), and the display interval as attributes of the target objects 242a and 242b in the virtual space 240. The setting unit 217 may set a delay time from the timing of notifying the generation of the target objects 242a and 242b to the timing of the generation of the target objects 242a and 242b, thereby controlling the cognitive load imposed on the user 220. In other words, the user must continuously remember the action to be performed from the time when the user knows the position in the virtual space where the target object is to be generated by the radar screen image 250 or the like (the position indicating the direction in which the head mounted display should be oriented to display the target object) until the target object is actually generated, and this "memory time" becomes the cognitive load for the user. The setting unit 217 may also control the cognitive load by changing the time "until the target object 152 approaches within the reach of the user 220" instead of "until the timing of the generation of the target object 152". The setting unit 217 may impose a cognitive load on the user 220 by displaying a background image 243 other than the target objects 242a and 242b on the head mounted display 233. When changing the cognitive load, it is desirable to notify the user in advance that the cognitive load will be increased or decreased. The notification method may be visually using characters or symbols, audio, or touching a part of the body, such as tapping the shoulder, elbow, arm, or leg.

[0060] The setting unit 217 controls the degree of difficulty of the physical movement according to the movement of a joint different from the body part where the user performs the physical movement, that is, the degree of joint connection. The setting unit 217 as a control unit controls the size of the target object, the distance from the user, the angle, the speed, the presence or absence of a background, the presence or absence of music, and the size of the visual aid image as point estimation parameters. For example, the presence or absence of music may greatly change the joint connection. The degree of the joint connection that occurs also changes depending on the type of music. For example, compared to the chirping of birds (which has a large effect on people with poor cognition), repetition of rhythmic music increases the workload for the brain to process information in the background, that is, the cognitive load, and therefore the joint connection is greater. Similarly, the complexity of the background information is also greatly related to the workload for the brain to process information in the background, that is, the cognitive load. For example, the cognitive load is greater in a three-dimensional space with no background or scenery, and the more moving animals or objects there are in it and the wider the range of the screen where the moving animals or objects appear, the greater the cognitive load and the more prominent the joint connection appears. In other words, it has been found that, basically, the more one tries to improve the accuracy of point estimation, or the greater the burden on the brain's background processing of sounds and background information, the less brain capacity is available for point estimation, and therefore the stronger the joint correlation becomes.

[0061] FIG. 3A is a diagram showing a screen (operation panel) 300 for an operator to operate. The setting unit 217 displays such an operation panel 300. In this embodiment, a task for rehabilitation support is created by setting point estimation parameters (distance to a target object, its height, direction, size of the target object, size of a visual recognition assistance image, moving speed, sensitivity = size of a sensor object, occurrence interval, type and presence or absence of background information, type and presence or absence of sound effects and BGM) using the intuitive operation panel 300. The operation can be performed in either a manual mode (a method in which point estimation parameters are set for each task and operated for each task), a template mode (a method in which point estimation parameters are set in advance for a plurality of task sets), or an automatic mode in which the device automatically generates a task, or a combination of these. In addition, the operation panel can be used to check various cognitive and motor function evaluation indexes and test results from basic information of the user, create templates, set and instruct the automatic mode, and so on.

[0062] The display that displays the operation panel 300 may be an external display connected to the rehabilitation support device 210, or may be a display built into the rehabilitation support device 210, and any device may be used. The operation panel 300 includes a user's visual field display area 301, various point estimation parameter setting areas 321 to 324, a score display area 303, a re-center button 305, and a BGM control button 307. For the sake of explanation, FIG. 3A includes an area 306 that shows the appearance of the actual user 220, but the operation panel 300 does not need to include the area 306.

[0063] The user visual field 301 displays an image that is actually displayed on the head mounted display 233. The user visual field 301 displays a reference direction in the virtual space. As described in FIG. 2, the radar screen image 250 is displayed in the center of the user visual field 301 (for example, in a visual angle range of −50 degrees to 50 degrees). The radar screen image 250 indicates which direction the target objects 242a and 242b that will appear next are located relative to the reference direction in the virtual space. In this example, the colored position of the block image 252 indicates that the target object 242 will appear at the farthest position to the left of the reference direction in the virtual space. The position of the sector image 253 and the direction of the head image 251 indicate that the user is already facing leftward.

[0064] Moreover, the various parameter setting areas 321 to 324 are screens for setting a plurality of parameters that define a task. The setting unit 217 can also receive input to the various parameter setting areas 321 to 324 from an input device (not shown). The input device may be a mouse, a numeric keypad, a keyboard, or may be any of various controllers, a joystick for games, a touch panel, or the like, and the technical configuration thereof is not important.

[0065] The various parameter setting areas 321 to 324 include an input area 320 for determining the sizes of the left and right target objects, an input area 321 for determining the size of the visual recognition assist image, an input area 322 for determining the size of the sensor areas of the avatar objects 241a and 241b, an input area 323 for determining the moving speed of the target object, and an input area 324 for determining the position of the next target object to appear. Furthermore, there is a check box 325 for setting whether or not to accept operation of the appearance position of the target object by various external parameter input devices such as hot keys and a numeric keypad.

[0066] The input areas 320 and 321 can set the radius of a visual object (visual size) for making it easier for the user to see the position of the target object, and the radius of a target object that reacts with the avatar objects 241a and 241b (evaluation size) on the right and left, respectively. That is, in the example of FIG. 3A, the user sees a circle with a radius of 20 cm, but the task is not completed unless the user actually touches a ball with a radius of 10 cm located at the center of the circle. If the visual size is small, it is difficult for the user to find the target object. If the visual size is large, the user can easily find the target object. If the evaluation size is large, the allowable amount of deviation of the avatar objects 241a and 241b is large. If the evaluation size is small, the allowable amount of deviation of the avatar objects 241a and 241b is small, and the rehabilitation movement can be evaluated more severely. The visual size and the evaluation size can also be made to match. These settings quantitatively change the accuracy of the feedforward, that is, the accuracy of information processing performed by the user's brain. It will be possible to control the treatment approach for physical function (upper limb function, walking function, trunk function, balance function), cognitive function (including spatial cognition and attention), sensory function (including the inner ear, vestibular system, touch, temperature and pain sensation, position sense, and deep sensation), and the degree of manifestation of joint involvement.

[0067] In the input area 322, the sensor sizes (sensor object sizes) of the left and right avatar objects 241a and 241b can be set. If the sensor size is large, the task is accomplished even if the hand position is significantly deviated from the target object, so the difficulty of the rehabilitation movement is reduced. Conversely, if the sensor size is small, the hand must be moved accurately to the central area (evaluation size) of the target object, so the difficulty of the rehabilitation movement is increased. In the example of FIG. 3A, the sensor size is 2 cm on each side. This element also quantitatively changes the feedforward accuracy, that is, the information processing accuracy for the brain, and makes it possible to control the treatment approach for physical functions (upper limb function, walking function, trunk function, balance function), cognitive functions (including spatial cognition and attention function), and sensory functions (including the inner ear, vestibular system, touch, thermal pain, position sense, and deep sensation), and the degree of manifestation of joint connections.

[0068] In the input area 323, the speed at which the target objects 242a, 242b move in the virtual space can be specified for the left and right, respectively. In this example, the speed is set to 45 cm / s.

[0069] Input area 324 is an image for inputting the position of the next task (distance and angle to the task), and has a shape that is an enlarged version of radar screen image 250. Check box 325 is checked, so an operation can be performed using an input device. When an operation of clicking or tapping on any of the multiple blocks in input area 324 is performed, or when an operation corresponding to the above is performed using an input device, target object 242a or target object 242b is generated at a position in the virtual space that corresponds to the position of the specified block.

[0070] That is, in the example of FIG. 3A, the task of the user 220 is to timely bring an avatar object (controller) including a 2 cm sensor part into contact with a target object (ball) of a radius of 10 cm falling at a speed of 45 cm / s at a distant location on the left side in the virtual space. FIG. 3A shows a state in which a target object 242a appears in the user visual field 301. In this state, when the user 220 extends his left arm as shown in the area 306, the avatar object 241a appears in the user visual field 301. A visual object 312 for improving the visibility of the target object 242a is displayed around the target object 242a. The visual size set in the input area 321 indicates the radius of this doughnut-shaped visual object 312. The target object and the avatar object may correspond to either the left or right, and the user may be prompted to move the left side of the body to the same side in the virtual space, that is, to the left, or to the right.

[0071] When the avatar object 241a touches the visual object 312, the target object 242a does not disappear and the task is not completely accomplished (it is an incomplete accomplishment, and a certain number of points are awarded and the task is evaluated as "good"). The task is not completely accomplished (evaluated as "perfect") until the avatar objects 241a and 241b touch the target objects 242a and 242b.

[0072] Meanwhile, the score display area 303 shows the total number of times the task was performed at each position and the number of times the task was performed. The score may be expressed as a fraction, a percentage, or a combination of both. The evaluation unit 214 uses the value of this score list 332 to derive a rehabilitation evaluation point after a series of rehabilitation movements determined by one task set.

[0073] The re-center button 305 is a button that accepts a reconstruction instruction from an operator to reconstruct the virtual space according to the position of the user 220. When the re-center button 305 is operated, the display control unit 212 reconstructs the virtual space with the position of the head mounted display 233 at that moment as the origin and the orientation of the head mounted display 233 at that moment as the reference direction. By operating the BGM control button 307, the background sound during task execution can be turned on or off. By eliminating the background sound, the task load can be reduced and the user can concentrate on the movement of the body. In general, for a user who has a joint connection, if the background sound is eliminated, the joint connection during the task tends to subside. Conversely, for a user who does not have a joint connection, the background sound can be turned on to intentionally make the joint connection appear. The display control unit 213 may display an information bar 310 on the display screen 240 of the head mounted display 233 in addition to the radar screen image 250. The information bar 310 displays a player's name 311, a task achievement level 312, and an elapsed time 313 since the start of play. Furthermore, the information bar 310 is parallel to the horizon 244 in the virtual space. If the patient's head position or body axis is distorted, the parallel relationship between the radar screen image 250, which follows the tilt of the head mounted display 233, and the information bar 310 is lost. For this reason, the patient can correct the inner ear information as a visual aid, and the therapist can easily notice the distortion of the patient's body axis. FIG. 3A shows an example of an image when the head position is tilted 20 degrees to the right. In this way, the user interface of the head mounted display 233 is designed to have medical practical benefits in addition to elements that allow the patient to immerse themselves in rehabilitation. 3B is a diagram showing another example of the operation panel 300. As shown in the joint connection message 308, the presence or absence and the size of the joint connection may be detected by a camera or the like and notified to the operator as text, an image, or a change in color. As described above, a heat map may be superimposed on the user image 306 to show in red image areas where joint connection is highly likely to occur.

[0074] The flow of the point estimation accuracy control for joint connections is as follows: (1) Perform rehabilitation movements with default settings. (2) The joint connections are too large (angle and distance of the legs walking wide apart, twisting of the shoulders, twisting of the waist, twisting of the head, etc. exceed the threshold). (3) Lower the required difficulty of point estimation (change the point estimation parameters). (4) The joint connections become appropriately small. (5) Repeat the rehabilitation movements alternately on the left and right. (6) The joint connections disappear. (7) Increase the required accuracy of point estimation (change the point estimation parameters). (8) The joint connections become appropriately large. (9) Return to (5). In (5), the joint connection may be suppressed by touching the user's body or speaking to the user. In (7), a load may be applied to the user by speaking to the user, which has the same effect as increasing the required accuracy of point estimation. It has been found that touching the user's body can release the user's joint connection more quickly. As described above, the user's direct view is blocked to perform point estimation, and strong motor commands via the target object cause deep muscles to contract and align the entire body. During this process, we find any fraying in the cranial nerves known as joint connections, and while controlling the required accuracy (various parameters) of point estimation, we repeat the rehabilitation movement until the joint connections are resolved. When multiple joint connections are present, it is necessary to determine which joint connection should be the main one to be corrected. For example, the knee (hip joint) that has moved significantly should be corrected first, and then the next joint (for example, the involuntary movement of the ankle joint causing the sole of the foot to turn inward) should be corrected. By performing rehabilitation movements with the goal of recovering joint connections, it is possible to very effectively improve the user's cognitive and motor abilities (unravel the knots in the brain). Joint connections serve as a guide for treatment. Stimuli (point estimation requests) are sent to the brain, and the reaction of the joint connections is observed to determine the treatment plan. Effective treatment is possible by actively contacting the parts of the body where joint connections are present.

[0075] FIG. 4A is a diagram showing a task table 400 stored in the task set database 216. In the task table 400, time (task occurrence timing) 461, task interval 462 from the end of the previous task, task type 463, task angle 464, and task distance (strength) 465 are stored, each associated with a task ID. Furthermore, in the task table 260, speed 466 of the target object, reference size 467 of perfect judgment (praise evaluation), reference size 468 of good judgment (splendid evaluation), size 469 of the sensor object, and task achievement result are stored, each associated with a task ID. In addition to these, a delay time (predetermined time) from task occurrence notification to task occurrence may be set for each task. FIG. 4B is a diagram showing another example of the task table 400 stored in the task set database 216. As shown in FIG. 4B, data 471 regarding joint connections may be accumulated.

[0076] 5 to 7 are diagrams showing an example of display on the head mounted display 233 according to this embodiment. Addition of such background information or addition of background sound as one of the implementation methods both places a burden on the brain's information processing, making the joint connection more prominent. In FIG. 5, an image showing an inro 511 as a target object is displayed in a background image 501 showing a townscape in the Edo period. Furthermore, a 1000-ryo box 513 is displayed below the inro 511 as an item that the user must protect, and a ninja 515 gradually approaches from the back. The speed of the ninja 511 is the speed set in the input area 323 of the operation panel 300 (the speed here is synonymous with the time limit). A circle 512 is displayed on the inro 511 as a visual aid image. The task is accomplished if the inro 511 is touched by a sensor object (the tip center of the avatar objects 241a and 241b) before the ninja 515 reaches the 1000-ryo box 513. Two types of circles 512 are provided, red and blue, and the task is to operate the red avatar object 241b on the right side, which corresponds to the controller 235 held in the right hand, to make contact with the inro 511 surrounded by the red circle 512. On the other hand, the task is to operate the red avatar object 241a on the left side, which corresponds to the controller 234 held in the left hand, to make contact with the inro 511 surrounded by the blue circle 512.

[0077] The inro 511 is displayed at a position (depth and angle) set in the input area 324 of the operation panel 300. The inro 511 does not change its position until the user touches the avatar objects 241a and 241b in the virtual space. In other words, it is a target object fixed in space (also called a horizontal task because it requires the user to stretch the body horizontally). Such a fixed target object is very effective as rehabilitation for diseases such as cerebellar ataxia, diplopia, and cochlear dysfunction. In other words, for patients who have forgotten how to move their bodies, it is possible to imprint an image of a limited body movement on their brains by feedforward. By increasing the distance in the depth direction of the inro 511, the exercise intensity can be changed. Furthermore, by combining multi-channel biofeedback, motor ability, physical function, cognitive function, and sensory function can be greatly improved. In addition, such a horizontal task can improve chronic pain by promoting reorganization of the cerebral cortex. Alternatively, it can be used to recover from cognitive dysfunction called chemobrain, sensory dysfunction in which the sense of position of cancer patients who have taken anticancer drugs is reduced due to nerve damage, and aftereffects of COVID-19 infection. The cognitive load can be reduced by informing the patient in advance of the location where the target object will appear and giving hints. Tactile inform by touching the body, repeated verbal inform multiple times, or a combination of these are more effective in reducing the cognitive load than verbal inform. The method of verbal inform may also be performed by lowering the cognitive load by giving simple instructions that are more concise and closer to an imperative form, or by taking more complex instructions in the form of a question, such as "Because it is blue? (Take it with your right hand)", or by providing verbal inform in the form of a cognitive task such as calculation, such as "When I say a number that is divisible by 2, take it with your right hand". It is also possible to configure the height of the inro 511 to be configurable, not just the horizontal position and depth at which it is generated.

[0078] FIG. 6 is a diagram showing an example of a screen for performing a task in which a target object moves vertically (vertical task or falling task). In FIG. 6, an image of a person representing a farmer is displayed as a trigger object 602, which triggers the appearance of a target object, in a background image 601 representing a field. That is, the display control unit 213 displays the trigger object 602 as an announcement image for announcing the appearance of the target object 603. A predetermined time after the trigger object 602 throws the potato-shaped target object 603 upward, a large potato-shaped target object 703 appears from the screen as shown in FIG. 7. The task is accomplished by moving the sieve-shaped avatar object 702 to catch the falling target object 703. The left and right avatar objects 702 move on the screen in conjunction with the movements of the controllers 234 and 235.

[0079] The setting unit 217 can adjust the cognitive impact on the user by setting a delay time from the timing when the trigger object 602 throws the target object 603 upward and notifies the generation of the target object 603 to the generation of the target object 703. The longer the delay time, the longer the period during which memory is retained, and the greater the load on the brain's information processing. In addition, the generation of the target object may be notified at the same timing by a radar chart-type notification image 250 in conjunction with the movement of the trigger object 602, or a voice notification may be combined.

[0080] In this way, the setting unit 217 can impose a cognitive load on the user not only in a task with only a horizon background as in Fig. 2, but also in a task with a background with a large amount of information as in Fig. 5 and Fig. 6. In other words, by making it difficult for the user to remember that the target object 603 has appeared and the position where the target object 703 is expected to fall, a cognitive load closer to that required in real life is imparted to the rehabilitation user.

[0081] In particular, the setting unit 217 changes the task mode and changes at least a part of the background image 301 over time, thereby applying a cognitive load to the user 220 to process the background image in the brain. In the example of FIG. 6, for example, clouds 604 may be moved, plants 605 may be swayed, or animals (not shown) unrelated to the target object may appear in the background image 601. This can prevent the user 220 from concentrating on the target object 603, making it more difficult for the user 220 to remember the position where the target object 603 is likely to fall. More technically, it can be said that the cognitive load is controlled by providing an environment that makes it difficult to concentrate on the target object by displaying information unrelated to the task in the background image, and intentionally inducing attention disorders (more specifically, selective attention disorder, distributed attention disorder, shifting attention disorder, and sustained attention disorder) to make memory difficult.

[0082] FIG. 8 is a diagram showing another example (vertical task or drop task) of the display on the display screen 240 according to the present embodiment. In FIG. 8, a trigger object 802 representing a monkey and a target object 803 representing an apple are displayed in a forest-like background image 801. The trigger object 802 representing a monkey drops the target object 803 representing an apple from a tree, and the target object 803 approaching the user is caught by an avatar object 804 representing a colander, thereby accomplishing the task. Here, too, the setting unit 217 starts the drop of the target object 803 after a predetermined time has elapsed since the trigger object 802 shook the tree to notify the user of the occurrence of the target object 803, thereby causing attention disorder and imposing a cognitive load on the user 220.

[0083] Furthermore, the setting unit 217 can impose a much stronger cognitive load on the user 220 by simultaneously causing at least two to five target objects 803 to exist in the three-dimensional virtual space and displaying them on the display screen 240. In other words, the setting unit 217 generates at least two target objects 803 at different positions in the left-right direction in the three-dimensional virtual space.

[0084] In particular, if at least two target objects 803 are generated at multiple positions in different directions (left and right directions in FIG. 8) with respect to the moving direction of the target object 803 (falling direction in FIG. 8), it is possible to impose an even greater cognitive load. In other words, the user 220 must move the controllers 234, 235 while taking into consideration the vertical movement, the difference in the horizontal generation position, and the difference in the falling position in the depth direction, and this also tests the spatial cognitive ability. In this way, by adjusting the type, number, size, spatial spread, position, amount, etc. of information included in the notification image and notification sound including the trigger object in addition to changing the predetermined time of the task, it is possible to quantitatively adjust and control the complexity of the information to be stored, that is, the cognitive load that the brain must process information for the user.

[0085] The evaluation update unit 215 evaluates the user's cognitive ability using information such as whether the avatar object reached the three-dimensional target position represented by the target object in a timely and accurate manner, the time interval and number of target objects from the notification of their occurrence to their occurrence, and the degree of load that causes attention disorders from the background image. The required accuracy of point estimation varies depending on which of the various modes (display screens) shown in Figures 2A, 5 to 8 is executed by the user. For example, compared to the case where there is no background as in Figure 2A, the required accuracy of point estimation increases in the order of Figures 5, 8, and 6. In addition, as described above, the required accuracy of point estimation changes depending on the background sound, which affects the joint connection.

[0086] 9 is a flowchart showing a process flow in the rehabilitation support device 210. In step S901, as a calibration process, a goal of a rehabilitation movement is initialized according to a user. Specifically, each user is first asked to perform an operation of acquiring a possible movement range as a calibration, and the range is set as an initial value, and then the goal is initialized according to the user. This initial value may be set according to the possible movement range of the user, or may be determined by a therapist providing treatment, taking into account the user's physical cognitive function and a desired therapeutic effect.

[0087] Next, in step S903, the right-side task (that is, display of the goal object by the display control unit 212 and evaluation of the achievement by the avatar object) is started.

[0088] In step S905, the left side task (that is, display of the goal object by the display control unit 212 and evaluation of achievement by the avatar object) is started.

[0089] In step S909, the degree of task achievement is obtained and points are added. After a task on one side occurs, the next task on the opposite side may be generated without waiting for the achievement evaluation of the task, and these processes may be processed in parallel or with a time lag. The first task may start on either the left or right side, and the last task may also start on either the left or right side. In other words, the flow described here is merely an example and is not limited to the present invention.

[0090] <Data> FIG. 10 is a diagram showing the results of using the rehabilitation support system 200 according to this embodiment.

[0091] The number of cases was 13, with an average age of 60±21 years, seven men (53.9%), five patients (38.4%) with cerebrovascular disease, four patients (30.8%) with orthopedic disease, and four patients (30.8%) with neurodegenerative disease. All patients had undergone regular rehabilitation for more than six months, and were told that further improvement was difficult. For this group of patients, even when rehabilitation was performed by intensively training the affected side for the first 20 minutes, STEF, TUG, and TMT-A did not improve statistically significantly (graph of 20 minutes of regular intervention in Figure 10). Here, STEF (Simple Test for Evaluating Hand Function) is an upper limb function test. TUG (Timed Up and Go test) is a comprehensive test index of walking function, trunk function, and balance function. TMT-A (Trail-Making Test-A) is a type of cognitive function test. The higher the STEF number, the better the performance, and the lower the TUG and TMT-A numbers. All data are shown as mean ± standard deviation. Statistical analysis was performed using paired t-test, with p<0.05 as the criterion for statistical significance.

[0092] On the other hand, by using the system 200 according to this embodiment to perform rehabilitation that focuses on depth alternating between the left and right sides, regardless of whether they are affected or healthy, STEF, TUG, and TMT-A all improved.

[0093] That is, as in the present embodiment, it has been found that alternating left and right training with an awareness of depth improves not only upper limb function, but also walking function (physical function) through improvements in trunk function and balance function, and further improves spatial cognition and attention function. It has been presumed that the improvement of the inner ear, vestibular system, tactile sense, thermo-pain sense, position sense, and deep sense also influences the improvement of STEF, TUG, and TMT-A. In this way, it has been proven that performing rehabilitation with an awareness of depth alternating between the left and right is effective in maximally stimulating the plasticity of the brain and maximizing the efficiency of motor learning, cognitive learning, and sensory learning. It has been found from other experimental results that the treatment of the present application improves more quickly when it is performed alternating between the left and right than when it is not performed. The symmetrical distribution of the motor cortex shown in Non-Patent Document 1 and FIG. 2B can be considered as knowledge that supports this fact. [Other embodiments] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the technical scope of the present invention. In addition, systems or devices that combine the separate features included in each embodiment in any way are also included in the technical scope of the present invention. The present invention may be applied to a system consisting of multiple devices, or to a single device. Furthermore, the present invention is also applicable to a case where an information processing program for implementing the functions of the embodiments is supplied to a system or device and executed by a built-in processor. In order to implement the functions of the present invention with a computer, a program installed on a computer, a medium storing the program, a server for downloading the program, and a processor for executing the program are also included in the technical scope of the present invention. In particular, at least a non-transitory computer readable medium storing a program for causing a computer to execute the processing steps included in the above-mentioned embodiments is included in the technical scope of the present invention.

Claims

1. A detection unit that detects the user's movements using a controller operated by the user or a position sensor that follows the user's body, A blocking unit for obstructing the user's view so that the controller or the position sensor is not visible in the real world, A display unit that displays the avatar object of the controller or the position sensor in a virtual space, A request unit that, by requesting the user to perform a bodily movement that overlaps the avatar object of the controller or the position sensor with the target object, causes the user's brain to visualize the state of the body without relying on vision, An information processing system equipped with [the following features].

2. The information processing system according to claim 1, further comprising a feedback unit that provides feedback to the user in the virtual space regarding the achievement of the physical movement requested by the request unit.

3. The information processing system according to claim 2, wherein the feedback unit provides feedback that stimulates two or more of the user's five senses each time the physical movement requested by the request unit is achieved.

4. The information processing system according to claim 1, further comprising a joint coupling detection unit for detecting a phenomenon in which a joint moves unintentionally, independently of the movement requested by the requesting unit.

5. The blocking unit is a head-mounted display that displays the virtual space, The information processing system according to claim 1, wherein the request unit generates the target object and the avatar object in the virtual space and displays them on the head-mounted display.

6. A control method for an information processing system having a detection unit, a display unit, and a request unit, The detection unit performs a detection step of detecting the user's movement using a controller operated by the user or a position sensor that follows the user's body, The display unit performs a display step of displaying an avatar object of the controller or the position sensor in a virtual space while obstructing the user's view so that the controller or the position sensor is not visible to the user in the real space, The requesting unit requests the user to perform a bodily movement that overlaps the avatar object of the controller or the position sensor with the target object, thereby causing the user's brain to visualize the state of their body without relying on vision; A control method including

7. A detection step of detecting the user's movement using a controller operated by the user or a position sensor that follows the user's body, A display step in which an avatar object of the controller or position sensor is displayed in a virtual space while the controller or position sensor is obscured from the user's view in the real space, A request step that causes the user's brain to visualize the state of their body without relying on vision by requesting the user to perform a bodily movement that overlaps the avatar object of the controller or the position sensor with the target object, An information processing program that causes a computer to execute something.

Images