Information processing system, information processing method, and information processing program

Abstract

The present invention addresses the problem of effectively improving the cognitive ability or exercise ability of a user. Provided is an information processing system comprising: a blocking unit for making the whole or a part of the body of a user invisible; a request unit for requesting the user to perform a body action so as to superimpose a body part of the user or an object operated by the user over a target object; and a control unit for controlling the difficulty of the action according to the movement of a joint different from the body part for performing the body action under an environment that actualizes the abnormal action of the joint and that is generated by the blocking unit and the request unit.

Description

Information processing system, information processing method, and information processing program

[0001] The present invention relates to an information processing system, an information processing method, and an information processing program.

[0002] In the above technical field, Patent Document 1 discloses a system for use on hemiplegic patients due to stroke or the like.

[0003] Japanese Patent Application Laid-Open No. 2015-228957

[0004] Nature "A somato-cognitive action network alternates with effector regions in motor cortex" by Evan M. Gordon et al. Published online:19 04 2023

[0005] However, the technology described in the above document does not provide an indication of what kind of movement or what kind of movement the user makes to move on to the next step. On the other hand, as disclosed in Non-Patent Document 1, it has been discovered that there is a neural region in the neural circuitry of the brain that controls coordinated movements that link multiple joints.

[0006] An object of the present invention is to provide a technique for solving the above-mentioned problems.

[0007] To achieve the above object, the system of the present invention is an information processing system comprising: a blocking unit that makes all or part of a user's body invisible; a request unit that requests the user to perform a physical movement such that a part of the user's body or an object operated by the user overlaps a target object; and a control unit that controls the difficulty of the movement in accordance with the movement of a joint other than the body part performing the physical movement, in an environment that makes abnormal joint movement apparent, generated by the blocking unit and the request unit.To achieve the above object, the method of the present invention is an information processing method including: a blocking step that makes all or part of the user's body invisible by the blocking unit; a request step that requests the user to perform a physical movement in accordance with the movement of a joint other than the body part performing the physical movement, in an environment that makes abnormal joint movement apparent, generated by the blocking step and the request step. In order to achieve the above object, the program of the present invention is an information processing program that causes a computer to execute the following steps: a blocking step of making all or part of a user's body invisible; a requesting step of requesting the user to perform a physical movement that causes part of the user's body or an object operated by the user to overlap with a target object; and a control step of controlling the difficulty of the movement in accordance with the movement of a joint different from the body part performing the physical movement in an environment that exposes abnormal joint movements, generated by the blocking step and the requesting step.

[0008] According to the present invention, it is possible to effectively improve the cognitive ability or motor ability of a user.

[0009] 1 is a block diagram showing the configuration of an information processing system according to a first embodiment. FIG. 2 is a block diagram showing the configuration of a rehabilitation support system according to a second embodiment. FIG. 3 is a diagram for explaining the configuration of the rehabilitation support system according to the second embodiment. FIG. 4 is a diagram for explaining the configuration of the rehabilitation support system according to the second embodiment. FIG. 5 is a diagram showing an example of an operation panel screen of the rehabilitation support system according to the second embodiment. FIG. 6 is a diagram showing another example of an operation panel screen of the rehabilitation support system according to the second embodiment. FIG. 7 is a diagram showing an example of a task data table of the rehabilitation support system according to the second embodiment. FIG. 8 is a diagram showing another example of a task data table of the rehabilitation support system according to the second embodiment. FIG. 9 is a diagram showing an example of a display screen on a head-mounted display of the rehabilitation support system according to the second embodiment. FIG. 10 is a diagram showing an example of a display screen on a head-mounted display of the rehabilitation support system according to the second embodiment. FIG. 11 is a diagram showing an example of a display screen on a head-mounted display of the rehabilitation support system according to the second embodiment. FIG. 12 is a flowchart showing the processing flow of the rehabilitation support system according to the second embodiment. FIG. 13 is data showing the results of rehabilitation performed by the rehabilitation support system according to the second embodiment.

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the components described in the following embodiments are merely examples and are not intended to limit the technical scope of the present invention.

[0011] First Embodiment An information processing system 100 according to a first embodiment of the present invention will be described with reference to Fig. 1. The information processing system 100 is a system that provides an environment that makes abnormal joint movements apparent.

[0012] 1 , the information processing system 100 includes a blocking unit 101, a requesting unit 102, and a control unit 103. The blocking unit 101 makes all or part of the body of a user 110 invisible. The requesting unit 102 requests the user 110 to perform a physical movement that causes a part of the user's 110 body or an object 121 operated by the user to overlap with a target object 111. The control unit 103 controls the difficulty of the physical movement in accordance with a movement 131 of a joint different from the body part that the user 110 performs the physical movement.

[0013] With the above configuration, abnormal movements of joints other than the body part where the user performs physical movements can be made apparent, and ultimately, the abnormal movements can be used as indicators to effectively improve the user's cognitive ability or motor ability.

[0014] Second Embodiment A rehabilitation support system 200 according to a second embodiment of the present invention will be described with reference to FIG. 2A . The rehabilitation support system 200 is an example of a system that provides an environment that exposes abnormal joint movements, but the present invention is not limited thereto. That is, the present invention is not limited to the concept of "rehabilitation." It also encompasses treatment systems that actively treat physical disorders, such as paralysis and ataxia, dementia, and schizophrenia, as well as cognitive and mental disorders. Furthermore, the present invention also encompasses motion verification systems and ability improvement / enhancement systems for improving the motor and cognitive abilities of healthy individuals who do not require any treatment or rehabilitation. FIG. 2A is a diagram illustrating the configuration of the rehabilitation support system 200 according to this embodiment. The rehabilitation support system 200 according to this embodiment improves at least one of physical functions, such as upper limb function, walking function, trunk function, and balance function; cognitive functions (including spatial cognition, attention, and higher brain functions); and sensory functions (including the inner ear, vestibular system, tactile sense, thermal / pain sensation, position sense, and deep sensation).

[0015] As shown in FIG. 2A , the rehabilitation support system 200 includes a rehabilitation support device 210 as a request unit and a control unit, two base stations 231 and 232, a head-mounted display 233 as an isolating unit, and two controllers 234 and 235. A user 220 wears the head-mounted display 233 while sitting in a chair 221 and moves his or her body in accordance with the display on the head-mounted display 233. While this embodiment is described assuming verification of a movement performed while sitting in a chair, the present invention is not limited thereto. The movement may be performed while standing, walking, on a bed, in a supine or prone position, running, or while performing other specific movements. While the user 220 is shown holding the controllers 234 and 235 with both hands, the present invention is not limited thereto. The controllers may be held or worn on parts of the body other than the hands, such as the feet or trunk.

[0016] The two base stations 231, 232 detect the movement of the head-mounted display 233 and the movements of the controllers 234, 235 and send the detected movement to the rehabilitation support device 210. The rehabilitation support device 210 controls the display on the head-mounted display 233 based on the movement of the head-mounted display 233. Specifically, the rehabilitation support device 210 changes the position and direction of the viewpoint in the virtual space according to the position and direction of the head-mounted display 233 and determines the image to be displayed on the head-mounted display 233. The rehabilitation movement of the user 220 is also evaluated based on the movements of the controllers 234, 235. The head-mounted display 233 may be a non-transparent type, a video see-through type, an optical see-through type, or a glasses type. In this embodiment, a virtual reality (VR) space is presented to the user. However, it is also possible to superimpose real and virtual spaces, as in AR (Augmented Reality), or to reflect real information in the virtual space, as in MR (Mixed Reality), or to use hologram technology as an alternative. The key point here is that the head-mounted display 233 obscures all or part of the user's body from direct view. Typically, in real space, humans visually recognize their body movements and unconsciously fine-tune those movements in their brains to move to a target. In this system, the user 220 cannot directly see their hands in real space, so the user's brain cannot accurately correct their hand movements as it normally would. This can expose coordination disorders and make it easier to detect abnormal movements of joints other than those performing physical movements. In this case, for example, a projection mapping system that displays a background on all or part of the user's body may be used as the blocking unit. Alternatively, a contact lens-type display that displays the virtual space may be used as the blocking unit.

[0017] In this embodiment, the controllers 234 and 235 held by the user 220 and the base stations 231 and 232 are shown as examples of sensors for detecting the position or movement of the user's hands or head, but the present invention is not limited to these. Cameras (including depth sensors) for detecting the "position of the user's hands" or movement by image recognition processing, sensors for detecting the position of the user's hands by temperature, wristwatch-type wearable devices worn on the user's arms, motion capture, and the like can also be applied to the present invention by linking with the movement detection unit 211. In other words, one embodiment is to use a three-dimensional tracking device or movement analysis device such as Kinect (registered trademark), or to wear markers or the like on the body.

[0018] The rehabilitation support device 210 includes a movement detection unit 211 , display control units 212 and 213 , a feedback unit 214 , an evaluation update unit 215 , a task set database 216 , a setting unit 217 , and a movement guidance unit 218 .

[0019] The motion detection unit 211 acquires the positions of the controllers 234 and 235 held by the user 220 via the base stations 231 and 232, and detects the rehabilitation motion of the user 220 from changes in the position of the user's 220 hands.

[0020] The display control unit 212 generates and displays target objects 242a and 242b in the virtual space 240 to prompt the user 220 to perform three-dimensional body movements. In particular, the display control unit 212 generates a target object 242a in the virtual space 240 for performing a three-dimensional rehabilitation movement on a part of the body on the left side of the user 220, and a target object 242b in the virtual space 240 for performing a rehabilitation movement on a part of the body on the right side of the user 220. In other words, the display control unit 212 functions as a request unit that requests the user to reach for the target objects.

[0021] Note that, while the present invention requires reaching a virtual target displayed on a head-mounted display, the present invention is not limited to this, and the target may be an object in the real world. The target may be an object suspended by two or more wires or an object suspended in the air by ultrasonic vibrations. The target may also be an object displayed three-dimensionally as a hologram. The shape of the target object is not limited to a sphere, but may be a triangle, square, or dish. It may also be the shape of a character. The concept of the present invention also includes a system that uses an optical see-through or smart glasses-type head-mounted display to partially block the user's vision, preventing the user from seeing their own movements, while allowing the user to see only the target.

[0022] The display control unit 212 generates target objects 242a and 242b in the virtual space and moves them from above to below the user 220. As a result, the target objects are displayed on the head-mounted display 233 so that their display position and size gradually change (for example, gradually increasing in size and then decreasing in size). The movement direction of the target objects may be, for example, an upward direction from the floor to above the head. The target objects may move in any three-dimensional manner, including movement in the depth direction and left and right directions in addition to movement in the up and down direction, or may be fixed at a specific coordinate position without moving depending on the user's stage.

[0023] Furthermore, the display control unit 212 generates avatar objects 241a and 241b in the virtual space that move in response to the detected user movements. The avatar object 241a is an object that moves in response to the controller 234 operated with the left hand and represents the position of a part of the user's body on the left side. The avatar object 241b is an object that moves in response to the controller 235 operated with the right hand and represents the position of a part of the user's body on the right side. The avatar objects are not limited to objects that move in response to the position of the controller. They may also be virtual objects that move in response to changes in the position of a part of the user's body using the three-dimensional body tracking technology described above. The object operated by the user is also not limited to the controller shown in the figure and may be, for example, a bat or a golf club. The display control unit 212 requests the user to perform a body movement that causes the avatar objects 241a and 241b to be superimposed on the target objects 242a and 242b.

[0024] Even when avatar objects 241a and 241b are displayed, the burden on the user's brain increases when part or all of the body is blocked from view compared to when the user's actual body movements are directly visible. This is because, without being able to see the user's own body (including the controller in real space), the brain must estimate the controller's position (point) in real space by looking at the controller's position (point) in virtual space (so-called point estimation). In this case, the brain must strongly imagine the body's state without relying on vision, which is medically referred to as requiring strong feedforward. A brain that requires strong feedforward feedback finds it more difficult to coordinate the body, resulting in coordination disorders and the likelihood of abnormal movements of joints other than those responsible for the body movements. Furthermore, strong feedforward feedback requiring point estimation can induce contractions of the body's deep muscles through stimulation via the pyramidal tract.

[0025] For example, when moving the right arm to move the avatar image 241b on the right controller 235 toward the target object 242b on the right side, the left knee joint may unconsciously exert force, causing the lower leg of the left leg to move forward or backward. Here, this phenomenon of unintentional movement of joints not necessary for the desired movement is referred to as articular linkage, and is used as an indicator for improving the user's movements. Note that this articular linkage can manifest even without the anatomical connection known as anatomical train. For example, voluntary movement of the toe joints can cause abnormalities in the temporomandibular joint, hip joint, or shoulder joint.

[0026] This phenomenon, as discussed in Non-Patent Document 1, can be considered to be supported by the fact that motor areas responsible for coordinated movement exist across all joints ( Figure 2B ). Looking at Figure 2B , it can be seen that in the latest neuroscience, cranial nerves are grouped into bilaterally symmetrical regions (such as the right and left hands, right and left feet, and right and left eyes), and that these motor areas are arranged symmetrically within the brain. Furthermore, new evidence suggests that neural regions responsible for linking the various body parts represented by puppets 264-266 likely exist between foot motor area 261 and hand motor area 262, and between hand motor area 262 and face motor area 263. The existence of nerves linking the various body parts, such as those represented by puppets 264-266, is thought to reveal joint connections during movement in the specific environment in which point estimation is required when all or part of the body is not visible, as described in the present invention.

[0027] The inventors had actually created the present invention by observing the rehabilitation movements of actual users before the publication of Non-Patent Document 2, but it can be said that the usefulness and logic of the present invention were theorized by the paper in Non-Patent Document 1.

[0028] 2A illustrates the movement of the knee joint as an example of the issuance of a joint connection, but the present invention is not limited to this, and all involuntary movements of joints other than the body part performing the requested movement are included in the concept of the present invention. For example, as shown in FIG. 2C, by detecting joint planes 271 and 272 and calculating their inclinations, simultaneous occurrence of twisting of the shoulder joint or trunk (cervical joint, thoracic joint) and twisting of the waist (lumbar joint, hip joint) may be detected.

[0029] Specifically, joint interactions include twisting of the cervical facet joints (manifested in head position), twisting of the thoracic facet joints (manifested in shoulder angle), elevation and abduction of the shoulder joints (opening of the armpits), extension and contraction of the knee, hip, and ankle joints (the knees and toes point up and down, or invert and abduct), extension and contraction of the elbow joint (an arm that is not required to move remains fixed in a flexed position), extension of the temporomandibular joint (mouth opening), flexion and extension of the internal joints of the hand and finger joints, and flexion and extension of the toe joints. These joint movements are not limited to flexion and extension, but also include adduction, abduction, internal rotation, external rotation, and rotation, or combinations of these, as well as movements not listed here.

[0030] The system may also include a joint link evaluation unit 219 that detects and evaluates the position, twist, tilt, movement speed, acceleration, etc. of each user's joints during rehabilitation movements using a camera 236. The joint link evaluation unit 219 detects the movement of each user's joints and evaluates whether joint linkage is occurring based on the magnitude, speed, and timing of the movement. The system requests rehabilitation movements that minimize such joint linkage and allows the user to achieve them. If we compare the brain to a CPU, streamlining the information processing process that previously accounted for 80% of the 100% of its computing power for joint linkages to 20% would enable more complex movements and improve motor skills. Alternatively, the idea is that improving the CPU capacity allocated to cognitive processing would also improve cognitive skills. By capturing images of the user with the camera 235 and comparing them with images taken during normal times (before exercise) or with a normal movement model, abnormal twisting or movement of the joint surfaces can be detected. Alternatively, joint connections may be detected by attaching sensors (or using motion capture) to the user's joints (e.g., both shoulders, both elbows, wrists, both knees, both ankles, etc.) and detecting their positions with the base stations 231, 232. Of course, the operator may visually recognize the occurrence of joint connections and input the information into the system. The tilt of the head-mounted display 233 may be detected with the base stations 231, 232 to detect neck joint connections. For example, if the head-mounted display 233 is tilted 30 degrees, it can be determined that the trunk is tilted 15 degrees (determined by a camera image) and the neck is tilted 15 degrees. The tilt of the wrist joints, etc., can be determined by detecting the positions and tilts of the controllers 234, 235 with the base stations 231, 232. Furthermore, the joint connection evaluation unit 219 may display the movements of the joint connections as distances, accelerations, etc., using a bull's-eye heat map, or may display changes in the joint connections over time to provide information useful for controlling the difficulty of the movement. Specifically, it is possible to visualize how the joint connections change from the start to the end of a required reaching movement, or to compare the magnitude of the joint connections of each joint, making it easier to determine which joints to focus on.

[0031] It is also possible to focus on detecting six joints: hip joints, knee joints, ankle joints, neck joints, trunk joints, and elbow joints. It is also possible to divide into more categories and examine each joint unit (68 in humans), or conversely, to focus on only one or a few joints where abnormalities are most easily detected. Furthermore, multiple evaluation indices may be combined into a single integrated index or parameter, such as a total score. In other words, it is also possible to weight and integrate each joint connection of the entire body or an important part, and perform a total joint connection evaluation (calculation of a joint connection score).

[0032] A motion requiring a 2-cm sensor over a 10-cm target requires low-precision point estimation, whereas a motion requiring a 1-cm sensor over a 1-cm target requires high-precision point estimation. In this case, if the user uses an obstruction device such as a head-mounted display, real-world visual information is blocked, resulting in a state of inconvenience. This requires the user to sharpen their cranial nerves for point estimation, resulting in a manifestation of impaired coordination, i.e., a more pronounced joint coupling. This is one mechanism by which joint coupling occurs.

[0033] As the rehabilitation level (user's motor ability) improves, the required accuracy of point estimation is increased by reducing both the size of the target and the sensor, making compensatory joint coupling more apparent. By repeatedly requesting rehabilitation movements in this state, or by guiding the user to perform the correct movements more easily through verbal prompts or physical contact, the joint coupling will subside. This results in the entire body being realigned, and coordination disorders are alleviated and improved.

[0034] The joint connection evaluation unit 219 may display the detected position and size of the joint connection on an operator screen (FIG. 3) or the like. A comparison unit may further be provided to compare the detected joint connection with past joint connections or with a normal movement model, and the comparison results may be notified to the operator. This allows the operator to recognize changes in the joint connection with greater accuracy.

[0035] If it is determined that the joint connection has become smaller than a certain standard, the load of the rehabilitation movement is increased, and if it is determined that the joint connection has become larger than a certain standard or that the increased state continues (the joint connection has not improved), the load of the rehabilitation movement is decreased.

[0036] Whether or not joint coupling occurs is influenced by factors such as the size, angle, height, and distance of the target object, the direction and presence or absence of movement, the speed of movement, color, shape, and visibility, the size, color, shape, and visibility of the device or indicator used for overlaying point estimation such as the controller, the presence or absence of background information, and the presence or absence of background sound. The goal of the rehabilitation movement in this system is to induce and then calm joint coupling. In other words, joint coupling serves as a guideline for treatment. Joint coupling can also be disengaged by pushing the user's knees or shoulders during rehabilitation movements. In this way, it is possible to manually and forcibly correct frayed neural circuits in the brain.

[0037] Returning to FIG. 2A , images of avatar objects 241a, 241b and target objects 242a, 242b are displayed on the display screen 240 according to the orientation and position of the head-mounted display 233 detected by the motion detection unit 211. The images of the avatar objects 241a, 241b and target objects 242a, 242b are superimposed on a background image 243. Here, the avatar objects 241a, 241b have the same shape as the controllers 234, 235, but this is not limiting and they may be hand-shaped. Furthermore, the left and right avatar objects may have different sizes, shapes, or colors. The controllers 234, 235 each have one or more buttons that can be operated to perform various settings, including initial settings such as setting the origin. However, the button functions may be disabled or buttons may not be provided at all, and all settings may be performed using a separate external operation unit. The background image 243 is cut out from a virtual space including a horizon 244 and a ground surface object 245. The appearance of the background image 243, avatar objects 241a, 241b, and target objects 242a, 242b changes depending on the movement of the line of sight (the position and direction of the head-mounted display).

[0038] The display control unit 212 generates an avatar object indicating the position of a body part on the right side and a target object indicating the target position of the right avatar object in similar colors or shapes. The display control unit 212 also generates an avatar object indicating the position of a body part on the left side and a target object indicating the target position of the left avatar object in similar colors or shapes. It is desirable to set the color or shape of the left avatar object and the left target object to be different from the color or shape of the right avatar object and the right target object. However, even if they are on the same side, the avatar object and the target object do not necessarily need to be set to similar colors or shapes, and there is no need to use different colors for the right and left sides. This may be set in any way that suits the user's needs.

[0039] Here, for example, avatar objects 241a and 241b are color-coded, for example, blue and red, and target objects 242a and 242b are also color-coded, for example, blue and red. The task is accomplished by touching blue target object 242a with blue avatar object 241a. Similarly, the task is accomplished by touching red target object 242b with red avatar object 241b. In other words, touching avatar objects of different colors does not result in task completion. While the color coding of blue and red is shown as an example here, other color coding (e.g., yellow and green) may be used for color-blind users. Alternatively, distinctions may be made without color by using shapes, linguistic notations such as "left," "right," "L (Left)," and "R (Right)," symbols such as stars, triangles, or circles, or by using different patterns (stripes) such as 242a and 242b in the figure.

[0040] The movement inducing unit 218 induces the user to alternate between rehabilitation movements of the right body part and rehabilitation movements of the left body part. Specifically, the movement inducing unit 218 induces the user's rehabilitation movements by displaying letters, colors, or figures indicating right and left, outputting audio to distinguish between "right" and "left," or by tactile stimulation using vibrations of the controllers held by the user in each hand, or by touching or tapping the shoulders or arms.

[0041] When the avatar objects 241a and 241b collide with the target objects 242a and 242b, the display control unit 212 causes the target objects 242a and 242b to disappear, and the feedback unit 214 determines that the target motion has been achieved and displays a message to notify or inform the user of the achievement of the target motion, thereby providing visual feedback. If the sensor centers of the avatar objects 241a and 241b reach the centers of the target objects 242a and 242b, a message such as "Bravo!" is displayed for each task. Even if the sensor centers reach the peripheral areas, a message such as "Well done!" is displayed for each task. This provides feedback to the user regarding the achievement of each requested physical motion. The feedback unit 214 may provide feedback by stimulating one or more of the user's five senses. Providing feedback by stimulating two or more of the user's five senses can more effectively improve the user's cognitive ability, motor ability, or both.

[0042] That is, the feedback unit 214 notifies the user 220 of the completion of each rehabilitation movement for the target object 242 a and the target object 242 b. Various notification methods are possible. The user may be notified of the degree of completion of the target movement by temporarily displaying text such as "Bravo!" or "Well done!" on the display screen, or by using similar voice or sound effects. Furthermore, the user may be notified of the degree of completion of the target movement by simultaneously vibrating only the controller 234 or 235 that made the movement to contact the target object. Alternatively, the notification of completion may be graded, such as complete completion, partial completion, or failure to complete, to indicate the degree or degree of completion.

[0043] For example, when the shortest distance between the sensor object included in the avatar object 241 and the target objects 242a and 242b falls within a predetermined range, the goal is achieved, and the target objects 242a and 242b disappear. At this time, if the shortest distance between the sensor object included in the avatar object 241a and 241b (e.g., a spherical object including the center of the tip of the avatar object 241a and 241b) and the target objects 242a and 242b falls below a threshold, the goal is completely achieved, and feedback is provided, such as by displaying a message like "Bravo!" and outputting a corresponding audio signal. Feedback that stimulates multiple different senses is called multi-channel biofeedback. The controllers 234 and 235 may be vibrated simultaneously, or olfactory or gustatory stimulation may be provided. That is, the feedback unit 214 can stimulate any of the five senses: sight, hearing, touch, taste, and smell. These sensory stimulations may be a combination of two, three, or more, or all of the five. In situations where point estimation is required, providing feedback each time a required movement is completed triggers the brain to correct and improve coordination disorders. In other words, joint coordination and coordination disorders improve. This improvement progresses more efficiently, or the improvement is maintained over a longer period, the more stimulation of the five senses is provided during feedback. For example, compared to single-sensory feedback, two-sensory feedback improves symptoms over hours, whereas symptoms that would have improved over several days with single-sensory feedback improve over hours. Similarly, compared to two-sensory feedback, three-sensory feedback improves symptoms over minutes, whereas symptoms that would have improved over hours with single-sensory feedback improve over minutes. Furthermore, even if symptoms improved with single-sensory feedback but quickly returned to normal, two-sensory feedback maintains the improvement for several days. Similarly, even if symptoms improved with two-sensory feedback but returned to normal within days, three-sensory feedback maintains the improvement for several weeks.To describe the characteristics of this treatment, it is also referred to in medical settings as "brain reprogramming therapy (BRT)," "brain rewiring therapy (BRT)," or "motor coordination therapy (MCT)."

[0044] The rehabilitation movement may be evaluated in three or more stages depending on how far the distance between the sensor object and the target objects 242a and 242b has been reduced. Also, two or more target objects may be simultaneously generated and displayed in the virtual space.

[0045] The display control unit 213 displays a radar screen image 250 on the display screen 240 of the head-mounted display 233. The radar screen image 250 is an alert image for alerting the user of the occurrence of a target object 152. The radar screen image 250 alerts the user of the occurrence of target objects 242a and 242b relative to a reference direction in the virtual space (which is initially set to be the front direction of a user sitting upright in a chair). The radar screen image 250 also alerts the user of the occurrence of target objects 242a and 242b, as well as how far the positions of the target objects 242a and 242b are from the user 220. The alert image is not limited to a radar screen image, and may be text, arrows, symbols, illustrations, types of light or color, intensity, blinking, or the like. The alert method is not limited to an image, and may be audio, vibration, or a combination of audio, vibration, and image. The display control unit 213 displays the radar screen image 250 in the center (for example, within a range of -50 degrees to 50 degrees) of the display screen 240 of the head-mounted display 233, regardless of the orientation of the user's 220's head. However, the display area is not limited to the center and may be any of the four corners, top, bottom, left, or right edges of the screen. The patient can estimate the difficulty of the next exercise movement to be performed based on the position, angle, and number of target objects displayed on the radar screen. If a more difficult movement is predicted for the patient, the joint linkage is more apparent. In other words, the display control unit can also function as a control unit for controlling the difficulty of the movement required of the patient.

[0046] The radar screen image 250 includes a head image 251 representing the user's head as viewed from above, block images 252 obtained by dividing the area around the head image 251 into multiple blocks, and a fan-shaped image 253 as a visual field image indicating the user's visual field. A target position image indicating the location of a target object is indicated by which blocks of the block image 252 are colored, flashing, or lit. This allows the user 220 to know whether the target object is on the left or right side of the direction in which the user is facing. Note that in this embodiment, the block image 252 is fixed and the fan-shaped image 253 moves, but the present invention is not limited to this. The fan-shaped image 253 and the head image 251 may be fixed, while the block image 252 moves depending on the direction of the head. Specifically, if the head turns left, the block image 252 may rotate to the right.

[0047] It is preferable that the feedback unit 214 changes the message type according to the evaluation of the rehabilitation movement via the display control unit 212. For example, if the sensor object contacts the center of the target objects 242a and 242b, "Bravo!" is displayed, and if the sensor object contacts only the peripheral area around the center of the target objects 242a and 242b, "Well done!" is displayed. The size of the target objects 242a and 242b and the size of the peripheral area can be set by the setting unit 217. The size of the sensor object can also be set by the setting unit 217. If the shortest distance between the sensor object included in the avatar object 241a, 241b and the target object 242a, 242b is less than or equal to a first threshold, the message "Bravo!" is displayed to indicate complete achievement of the goal, and a corresponding voice message "Bravo!" is output as feedback. If the shortest distance between the sensor object included in the avatar object 241a, 241b and the target object 242a, 242b is equal to or greater than the first threshold and less than a second threshold, the message "Well done!" is displayed to indicate achievement of the goal, and a corresponding voice message "Well done!" is output as feedback. Note that the output voice message does not have to be the same; for example, a non-verbal sound effect such as "Pirro." Note that the shorter the distance required for contact between the sensor object and the target object, the more precise the movement required, and therefore the brain must request more precise and refined movements from the body as motor commands. As mentioned above, this brain imagery is called feedforward. That is, the smaller the distance between the sensor object and the target object required to complete the action, the stronger the feedforward required, thereby enabling continuous adjustment of the user's motor, cognitive, and sensory load levels.

[0048] The feedback unit 214 provides feedback to the user who virtually touches the target objects 242a and 242b, stimulating, for example, two or more of the five senses (sight, hearing, touch, taste, and smell). This feedback stimulates two or more of the five senses when the sensor object enters within a predetermined distance from the center of the target objects 242a and 242b or when the sensor object comes into contact with the target objects 242a and 242b (this is called real-time multi-channel biofeedback or instantaneous multi-signal biofeedback). A delay of, for example, one second or less from the timing of the feedback to the user is highly effective. The shorter the interval between the user's action and the feedback (the shorter the delay), the greater the effect. The feedback unit 214 provides feedback to stimulate the user's vision by displaying an image that says "Bravo!", while simultaneously providing feedback to stimulate the user's hearing by outputting audio or sound effects from a speaker. Notification of task completion via the stimulation of the five senses may be provided in any combination, depending on the type of action, etc.

[0049] Furthermore, the feedback unit 214 may simultaneously output feedback that stimulates the user 220's vision with an image saying "Bravo!", feedback that stimulates the user 220's hearing with sound output from a speaker, and feedback that stimulates the user 220's tactile sense by vibrating the controller 234. Furthermore, the feedback unit 214 may simultaneously output only two types of feedback: feedback that stimulates the user 220's vision with an image saying "Bravo!" and feedback that stimulates the user 220's tactile sense by vibrating the controller 234. Alternatively, the feedback unit 214 may simultaneously output only two types of feedback: feedback that stimulates the user 220's hearing with sound saying "Bravo!" and feedback that stimulates the user 220's tactile sense by vibrating the controller 234.

[0050] The user 220's actions of moving the controllers 234 and 235 in this way are rehabilitation movements, and the display of a target object that prompts the user 220 to perform one rehabilitation movement is called a task or assignment. Information representing a task (task data) includes the target object's appearance direction (90 degrees right, 45 degrees right, front, 45 degrees left, and 90 degrees left relative to the chair's front direction), the distance to the target object, its shape (size), its appearance position (distance and angle from the user), its appearance interval (time interval), its movement speed (falling or rising, etc.), its size, color, whether the left or right controller should be used to acquire the information, the number of target objects appearing simultaneously, and the size of the sensor object. In other words, the task data includes at least one of the following attributes of the target objects 242a and 242b in the virtual space 240: their movement speed, number of objects displayed, size, display position, and display interval. The depth distance from the user 220 to the drop positions of the target objects 242a and 242b may be set continuously or, for example, in one of three stages. For example, the location can be changed so that the object is dropped right next to the user 220 or in a position that the user 220 cannot reach unless the user 220 leans forward significantly. This makes it possible to control the exercise load on the user and the load on spatial cognition or spatial grasping ability.

[0051] The evaluation update unit 215 evaluates the rehabilitation actions of the user 220 according to the quantity and quality of the tasks completed by the user 220 and adds points. The quality of the completed tasks includes whether the user 220 is “good” or “great,” i.e., how close the avatar object was to the target object. The evaluation update unit 215 assigns different points to each completed task (higher points for distant objects and lower points for closer objects). The evaluation update unit 215 can update the tasks according to the accumulated points. For example, the task completion rate (number of goals completed / number of tasks) may be used to update the task (attributes of the target object). The evaluation update unit 215 compares the rehabilitation actions detected by the action detection unit 211 with the target positions represented by the target objects displayed by the display control unit 212 to evaluate the rehabilitation ability of the user 220. Specifically, the system determines whether avatar objects 241a, 241b, which move in response to the rehabilitation movements detected by the movement detection unit 211, overlap with target objects 242a, 242b by comparing their positions in the three-dimensional virtual space. If they overlap, it is evaluated that one rehabilitation movement has been completed, and points are added. The display control unit 212 can make the target objects 242a, 242b appear at different positions in the depth direction (for example, three levels of positions). The evaluation update unit 215 assigns different points to each of the objects (higher points to more distant objects and lower points to closer objects).

[0052] The evaluation update unit 215 updates the target tasks in accordance with the accumulated points. For example, the target tasks may be updated using the task achievement rate (number of goals achieved / number of tasks).

[0053] The task set database 216 stores a set of tasks. A task represents a single rehabilitation movement to be performed by a user. Specifically, information representing a task includes the location, speed, and size of a target object that appeared, as well as the size of the avatar object at that time. The task set database 216 stores task sets that determine the order in which such tasks are provided to a user. For example, task sets may be stored as templates for each hospital, or a history of executed task sets may be stored for each user. The rehabilitation support device 210 may be configured to communicate with other rehabilitation support devices via the Internet. In this case, a single task set can be executed by the same user at multiple locations, or various templates can be shared among multiple users located far from each other.

[0054] The setting unit 217 can set the generation of the left target object 242a and the generation of the right target object 242b to be performed alternately, and the setting unit 217 sends an instruction to the display control unit 213 according to the setting.

[0055] The setting unit 217 can set the generation position in the depth direction of the left target object 242 a in the virtual space 240 and the generation position in the depth direction of the right target object 242 b in the virtual space 240 .

[0056] By controlling both the timing of target object generation and the vertical movement speed of the target object, the user's rehabilitation movements are controlled to ensure alternating left and right movements. In other words, the simplest method is to not generate left and right target objects simultaneously, but to generate the left target object after detecting a reaching for the right target object. However, various control methods are possible. Alternating left and right movements may be encouraged by simultaneously generating multiple target objects and varying their movement speeds so that they decrease in the order right → left → right → left. For example, after detecting a reaching for the right object, the right and left objects may be generated simultaneously, or the right object may be generated first, causing the right object to speed up and overtake the left object. This can place a high load on the brain. The request unit or control unit may control the user's rehabilitation movements to alternate left and right movements.

[0057] On the other hand, if the right target object is reached downward and the left target object is reached relatively upward, such as when there is paralysis only on the right side or when actively rehabilitating the left upper body, the right target object may be generated slowly, followed by a faster generation of the left target object at a later time. In other words, at least one of the timing and speed of the target object generation is controlled according to the vertical position of the body to be rehabilitated (according to the position of the target object to be reached). In the case of knee or lower body rehabilitation, the timing and speed of the target object generation are controlled to ensure reaching at a lower position. Furthermore, to increase cognitive load, the targets may be generated in the order left → right → right, but only the first target object may be moved slowly, and the second and third may be moved quickly. As a result, reaching can be performed in the order right, left, right.

[0058] By rewiring the brain in this way, it is possible to improve the symptoms of diseases that are difficult to treat with modern medicine, such as chronic pain and restless legs syndrome (abnormalities in the basal ganglia).

[0059] The setting unit 217 also sets at least one of the following attributes of the target objects 242a and 242b in the virtual space 240: movement speed, number of objects displayed, shape, color, size, display position (coordinate information, etc.), and display interval. The setting unit 217 may also set a delay time from the timing at which the target objects 242a and 242b are notified to the timing at which the target objects 242a and 242b are actually generated, thereby controlling the cognitive load imposed on the user 220. In other words, the user must continuously remember the actions they should perform from the time they learn the position in the virtual space where the target object will be generated (the position indicating the direction in which the head-mounted display should be pointed to display the target object) from the radar screen image 250 or the like until the target object actually appears. This "memory time" creates a cognitive load for the user. The setting unit 217 may also control the cognitive load by changing the time "until the target object 152 approaches within reach of the user 220" rather than "until the time at which the target object 152 is generated." The setting unit 217 may impose a cognitive load on the user 220 by displaying a background image 243 other than the target objects 242 a and 242 b on the head-mounted display 233. When changing the cognitive load, it is desirable to notify the user in advance that the cognitive load will be increased or decreased. The notification method may be visually using letters or symbols, audio, or touching a part of the body, such as tapping the shoulder, elbow, arm, or leg.

[0060] The setting unit 217 controls the difficulty of the physical movement based on the degree of joint interaction, i.e., the degree of joint interaction, of the joints different from the body parts performing the physical movement. The setting unit 217, which functions as a control unit, controls the size of the target object, its distance from the user, its angle, its speed, the presence or absence of a background, the presence or absence of music, and the size of the visual aid image as point estimation parameters. For example, the presence or absence of music can significantly affect the joint interaction. The type of music also changes the magnitude of the joint interaction. For example, compared to birdsong (which can have a significant impact on people with cognitive impairments), repetitive rhythmic music increases the workload for the brain to process information in the background, i.e., the cognitive load, resulting in greater joint interaction. Similarly, the complexity of background information is significantly related to the workload for the brain to process information in the background, i.e., the cognitive load. For example, the presence of some kind of background or scenery in a three-dimensional space without anything present increases the cognitive load. Furthermore, the presence of moving animals or objects within that background, and the wider the area of ​​the screen where those moving animals or objects appear, the greater the cognitive load, and the more pronounced the joint interaction becomes. In other words, it has been found that the more one tries to improve the accuracy of point estimation, or the greater the load on the brain's background processing of sounds and background information, the less brain capacity is available for point estimation, resulting in stronger joint correlation.

[0061] FIG. 3A illustrates a screen (operation panel) 300 for an operator to operate. The setting unit 217 displays this operation panel 300. In this embodiment, rehabilitation support tasks are created by intuitively setting point estimation parameters (distance to the target object, its height, direction, size of the target object, size of the visual aid image, movement speed, sensitivity (i.e., size of the sensor object), occurrence interval, type and presence of background information, and type and presence of sound effects and background music) using the operation panel 300. Tasks can be created in manual mode (a method in which point estimation parameters are set for each task and operated individually), template mode (a method in which point estimation parameters are pre-set for multiple task sets), or auto mode, in which the device automatically generates tasks, or a combination of these. The operation panel also allows the user's basic information, various cognitive and motor function assessment indices, test results, template creation, and auto mode settings and instructions.

[0062] The display that displays the operation panel 300 may be an external display connected to the rehabilitation support device 210 or a display built into the rehabilitation support device 210, and any device may be used. The operation panel 300 includes a user's visual field display area 301, various point estimation parameter setting areas 321 to 324, a score display area 303, a re-center button 305, and a background music control button 307. For the sake of explanation, FIG. 3A includes an area 306 that shows the appearance of the actual user 220, but the operation panel 300 does not need to include the area 306.

[0063] The user field of view 301 displays the image actually displayed on the head-mounted display 233. The user field of view 301 displays the reference direction in the virtual space. As described with reference to FIG. 2, the radar screen image 250 is displayed in the center of the user field of view 301 (for example, in a field of view angle range of −50 degrees to 50 degrees). The radar screen image 250 indicates the relative direction of the target objects 242a and 242b that will appear next relative to the reference direction in the virtual space. In this example, the colored position of the block image 252 indicates that the target object 242 will appear at the farthest position to the left of the reference direction in the virtual space. The position of the fan-shaped image 253 and the orientation of the head image 251 indicate that the user is already facing leftward.

[0064] Furthermore, various parameter setting areas 321 to 324 are screens for setting a plurality of parameters that define a task. The setting unit 217 can also accept input to the various parameter setting areas 321 to 324 from an input device (not shown). The input device may be a mouse, numeric keypad, keyboard, or may be any of various controllers, game joysticks, touch panels, etc., and the technical configuration thereof is not important.

[0065] The various parameter setting areas 321 to 324 include an input area 320 for determining the sizes of the left and right target objects, an input area 321 for determining the size of the visual recognition assist image, an input area 322 for determining the size of the sensor areas of the avatar objects 241 a and 241 b, an input area 323 for determining the movement speed of the target object, and an input area 324 for determining the position of the next target object to appear. Furthermore, there is a check box 325 for setting whether or not to accept operation of the appearance position of the target object by various external parameter input devices such as hot keys and a numeric keypad.

[0066] The input areas 320 and 321 allow the user to set the radius of the visual object (visual size) to make it easier for the user to see the location of the target object, and the radius of the target object (evaluation size) that interacts with the avatar objects 241a and 241b. In other words, in the example of FIG. 3A , the user sees a circle with a radius of 20 cm, but the task is not completed unless the user actually touches a ball with a radius of 10 cm located in the center of the circle. If the visual size is small, the user has difficulty finding the target object. If the visual size is large, the user can easily find the target object. If the evaluation size is large, the user can easily find the target object. If the evaluation size is large, the tolerance for deviation between the avatar objects 241a and 241b increases. If the evaluation size is small, the tolerance for deviation between the avatar objects 241a and 241b decreases, allowing for more rigorous evaluation of rehabilitation movements. These visual size and evaluation size can also be matched. These settings quantitatively change the feedforward accuracy, i.e., the accuracy of information processing performed by the user's brain. It is possible to control the degree of manifestation of joint dysfunction through treatment approaches to physical functions (upper limb function, walking function, trunk function, balance function), cognitive functions (including spatial cognition and attention), and sensory functions (including the inner ear, vestibular system, touch, temperature and pain sensation, position sense, and deep sensation).

[0067] In the input area 322, the left and right sensor sizes (sensor object sizes) of the avatar objects 241a and 241b can be set. A larger sensor size reduces the difficulty of the rehabilitation task, since the task can be completed even if the hand position is significantly off the target object. Conversely, a smaller sensor size increases the difficulty of the rehabilitation task, since the hand must be accurately moved to the center of the target object (evaluation size). In the example of Figure 3A, the sensor sizes are 2 cm on each side. This element also quantitatively changes the feedforward accuracy, i.e., the accuracy of information processing for the brain, and can control the therapeutic approach for physical functions (upper limb function, walking function, trunk function, and balance function), cognitive functions (including spatial cognition and attention), and sensory functions (including the inner ear, vestibular system, tactile sense, thermal and pain sensation, position sense, and deep sensation), as well as the degree of manifestation of joint connections.

[0068] The speed at which the target objects 242a and 242b move in the virtual space can be specified for each of the left and right sides in the input area 323. In this example, the speed is set to 45 cm / s.

[0069] Input area 324 is an image for inputting the position of the next task (distance and angle to the task), and is an enlarged version of radar screen image 250. Check box 325 is checked, so operation with an input device is possible. When one of the multiple blocks in input area 324 is clicked or tapped, or when a corresponding operation is performed with the input device, target object 242a or 242b is generated at a position in virtual space corresponding to the position of the specified block.

[0070] In other words, in the example of FIG. 3A , the task of the user 220 is to timely bring an avatar object (controller) including a 2-cm sensor portion into contact with a target object (ball) with a radius of 10 cm that is falling at a speed of 45 cm / s from a distant location on the left side of the virtual space. FIG. 3A shows a state in which a target object 242a appears in the user's field of view 301. In this state, when the user 220 extends his / her left arm as shown in area 306, an avatar object 241a appears in the user's field of view 301. A visualizing object 312 is displayed around the target object 242a to improve the visibility of the target object 242a. The visualizing size set in the input area 321 indicates the radius of this doughnut-shaped visualizing object 312. The target object and the avatar object may correspond to either the left or right side, and the user may be prompted to move the left side of the body to the same side in the virtual space, either to the left or to the right.

[0071] When the avatar object 241a touches the visual object 312, the target object 242a does not disappear, and the task is not completely accomplished (it is an incomplete accomplishment, and a certain number of points are awarded and a "good" evaluation is given). The task is completely accomplished (a "perfect" evaluation is given) only when the avatar objects 241a and 241b touch the target objects 242a and 242b.

[0072] Meanwhile, the score display area 303 shows the total number of times the task was performed at each position and the number of times the task was performed. The score may be expressed as a fraction, a percentage, or a combination thereof. After a series of rehabilitation movements determined by one task set, the evaluation unit 214 uses the values ​​in this score list 332 to derive rehabilitation evaluation points.

[0073] The recenter button 305 is a button that accepts a reconstruction instruction from the operator to reconstruct the virtual space according to the position of the user 220. When the recenter button 305 is operated, the display control unit 212 reconstructs the virtual space with the current position of the head-mounted display 233 as the origin and the current orientation of the head-mounted display 233 as the reference direction. Operating the background music control button 307 can turn background music on or off during task execution. Eliminating background music reduces the task load and allows the user to focus on body movements. Generally, for users who have demonstrated articulation, eliminating background music tends to reduce the articulation during a task. Conversely, for users who do not demonstrate articulation, turning on background music can intentionally cause articulation to appear. The display control unit 213 may display an information bar 310 on the display screen 240 of the head-mounted display 233 in addition to the radar screen image 250. The information bar 310 displays the player's name 311, task completion level 312, and elapsed time since the start of play 313. Furthermore, the information bar 310 is parallel to the horizon 244 in the virtual space. If the patient's head position or body axis is distorted, the parallel relationship between the radar screen 250, which tracks the tilt of the head-mounted display 233, and the information bar 310 is lost. This allows the patient to visually correct inner ear information and makes it easier for therapists to notice distortions in the patient's body axis. Figure 3A shows an example of an image when the patient's head is tilted 20 degrees to the right. In this way, the user interface of the head-mounted display 233 is designed to not only allow the patient to immerse themselves in rehabilitation but also to have practical medical benefits. Figure 3B shows another example of the operation panel 300. As shown in the joint articulation message 308, the presence and size of joint articulation may be detected using a camera or other device and notified to the operator via text, images, or color changes. As described above, a heat map may be superimposed on the user image 306 to show in red image areas where joint articulation is likely to occur.

[0074] The flow of controlling the accuracy of point estimation for joint connections is as follows: (1) Perform rehabilitation movements with default settings. (2) The joint connections are too large (the angle and distance of the user's wide-legged stance, shoulder twist, waist twist, head twist, etc. exceed the threshold). (3) Reduce the required difficulty of point estimation (change the point estimation parameters). (4) The joint connections become appropriately small. (5) Repeat rehabilitation movements alternately on the left and right sides. (6) The joint connections disappear. (7) Increase the required accuracy of point estimation (change the point estimation parameters). (8) The joint connections become appropriately large. (9) Return to (5). Note that in (5), the joint connections may be reduced by touching the user's body or speaking to the user. In (7), the same effect as increasing the required accuracy of point estimation may be achieved by applying a load to the user by speaking to the user. It has been found that touching the user's body can more quickly release the user's joint connections. As described above, the system blocks the user's direct view and allows point estimation. Strong motor commands via the target object cause deep muscles to contract, aligning the entire body. During this process, the system detects joint linkages, or frayed neural connections, and repeats rehabilitation movements until the joint linkages are resolved while controlling the required accuracy of point estimation (various parameters). When multiple joint linkages are present, the system determines which joint linkages to focus on. For example, first correcting a large movement in the knee (hip joint), then moving on to the next joint (e.g., involuntary movement of the ankle joint causing the sole of the foot to turn inward). Performing rehabilitation movements with the goal of restoring joint linkages can effectively improve the user's cognitive and motor abilities (loosening the knots in the brain). Joint linkages serve as a guide for treatment. Stimulation (point estimation requests) is sent to the brain, and the joint linkage responses are observed to determine the treatment plan. Effective treatment is possible by actively contacting the body parts where joint linkages are present.

[0075] FIG. 4A is a diagram showing a task table 400 stored in the task set database 216. The task table 400 stores, linked to a task ID, a time (task occurrence timing) 461, a task interval 462 from the end of the previous task, a task type 463, a task angle 464, and a task distance (strength) 465. The task table 260 also stores, linked to a task ID, a target object speed 466, a reference size for a perfect judgment (excellent evaluation) 467, a reference size for a good judgment (great evaluation) 468, a sensor object size 469, and a task completion result. In addition, a delay time (predetermined time) from the notification of task occurrence to the actual occurrence of the task may be set for each task. FIG. 4B is a diagram showing another example of the task table 400 stored in the task set database 216. As shown in FIG. 4B, data 471 related to joint connections may also be stored.

[0076] 5 to 7 are diagrams illustrating examples of displays on the head-mounted display 233 according to this embodiment. Adding background information, or adding background sound as one implementation method, places a burden on the brain's information processing and accentuates joint coordination. In FIG. 5 , an image of an inro 511 is displayed as a target object against a background image 501 depicting a townscape from the Edo period. A treasure chest 513 is displayed below the inro 511 as an item the user must protect, and a ninja 515 gradually approaches from the back. The speed of the ninja 511 is set in the input area 323 of the operation panel 300 (the speed here is synonymous with the time limit). A circle 512 is displayed on the inro 511 as a visual aid image. The task is completed if the ninja 515 touches the inro 511 with the sensor object (the center of the tip of the avatar object 241a or 241b) before reaching the treasure chest 513. Two types of circles 512 are provided, red and blue, and the task is to operate the red avatar object 241b on the right side, which corresponds to the controller 235 held in the right hand, to make contact with the inro 511 surrounded by the red circle 512. On the other hand, the task is to operate the red avatar object 241a on the left side, which corresponds to the controller 234 held in the left hand, to make contact with the inro 511 surrounded by the blue circle 512.

[0077] The inro 511 is displayed at the position (depth and angle) set in the input area 324 of the operation panel 300. The inro 511 remains in its position until the user touches the avatar objects 241a and 241b in the virtual space. In other words, it is a target object fixed in space (also called a horizontal task because it requires the user to stretch the body horizontally). Such a fixed target object is highly effective for rehabilitation of diseases such as cerebellar ataxia, diplopia, and cochlear dysfunction. In other words, for patients who have forgotten how to move their bodies, feedforward can imprint the image of a specific body movement in their brain. By increasing the depth distance of the inro 511, the exercise intensity can be changed. Furthermore, by combining it with multi-channel biofeedback, motor ability, physical function, cognitive function, and sensory function can be significantly improved. Furthermore, such horizontal tasks can also improve chronic pain by promoting cortical reorganization. It can also be used to improve cognitive dysfunction known as chemobrain, sensory dysfunction resulting from neurological damage that reduces position sense in cancer patients who have taken anticancer drugs, and after-effects of COVID-19 infection. Cognitive load can be reduced by providing hints in advance about the location of the target object. Tactile information via physical contact, repeated verbal information, or a combination of these are more effective than verbal information in reducing cognitive load. Verbal information can also be provided by providing simple, imperative instructions to reduce cognitive load. Or, more complex instructions in the form of a question, such as "Because it's blue? (Pick it up with your right hand)," or verbal information can include cognitive tasks such as calculations, such as "When I say a number divisible by two, pick it up with your right hand." The height of the inro 511 can be configured, in addition to its horizontal position and depth.

[0078] FIG. 6 shows an example of a screen for performing a task in which a target object moves vertically (a vertical task or a falling task). In FIG. 6 , an image of a person representing a farmer is displayed as a trigger object 602, which triggers the appearance of the target object, on a background image 601 representing a field. That is, the display control unit 213 displays the trigger object 602 as an announcement image for announcing the appearance of the target object 603. A predetermined time after the trigger object 602 throws the potato-shaped target object 603 upward, a large potato-shaped target object 703 appears on the screen as shown in FIG. 7 . The task is accomplished by moving the colander-shaped avatar object 702 to catch the falling target object 703. The left and right avatar objects 702 move on the screen in conjunction with the movements of the controllers 234 and 235.

[0079] The setting unit 217 can adjust the cognitive impact provided to the user by setting a delay time from the timing when the trigger object 602 throws the target object 603 upward and notifies the user of the generation of the target object 603 until the target object 703 is generated. The longer the delay time, the longer the period for memory retention and the greater the burden on the brain for information processing. Note that the generation of the target object may be notified at the same timing using a radar chart-type notification image 250 in conjunction with the movement of the trigger object 602, or a sound notification may be combined.

[0080] In this way, the setting unit 217 can impose a cognitive load on the user not only in a task with only a horizon background as in Fig. 2 but also in a task with a background with a large amount of information as in Fig. 5 and Fig. 6. In other words, by making it difficult for the user to remember that the target object 603 has appeared and the position where the target object 703 is expected to fall, a cognitive load closer to that required in real life is imparted to the rehabilitation user.

[0081] In particular, the setting unit 217 changes the task mode and changes at least a portion of the background image 301 over time, thereby imposing a cognitive load on the user 220 to process the background image in their brain. In the example of FIG. 6 , for example, clouds 604 may be moved, plants 605 may be swayed, or an animal (not shown) unrelated to the target object may appear in the background image 601. This may prevent the user 220 from concentrating on the target object 603, making it more difficult for the user 220 to remember the position where the target object 603 is likely to fall. More technically, by displaying information unrelated to the task in the background image, an environment that makes it difficult for the user 220 to concentrate on the target object is provided, and attention disorders (more specifically, selective attention disorder, allocated attention disorder, shifting attention disorder, and sustained attention disorder) may be intentionally induced to make memory more difficult, thereby controlling the cognitive load.

[0082] FIG. 8 is a diagram showing another example (vertical task or drop task) of a display on the display screen 240 according to the present embodiment. In FIG. 8 , a trigger object 802 representing a monkey and a target object 803 representing an apple are displayed in a forest-like background image 801. The trigger object 802 representing the monkey drops the target object 803 representing the apple from a tree, and the target object 803 approaches the user, and the user then receives the target object 803 with an avatar object 804 representing a colander, thereby completing the task. Again, the setting unit 217 starts the target object 803 from falling after a predetermined time has elapsed since the trigger object 802 shook the tree to notify the user of the arrival of the target object 803, thereby causing attention disorders and placing a cognitive load on the user 220.

[0083] Furthermore, the setting unit 217 can impose an extremely strong cognitive load on the user 220 by simultaneously causing at least two to five target objects 803 to exist in the three-dimensional virtual space and displaying them on the display screen 240. In other words, the setting unit 217 generates at least two target objects 803 in different positions in the left-right direction in the three-dimensional virtual space.

[0084] In particular, if at least two target objects 803 are generated at multiple positions in different directions (left and right in FIG. 8 ) relative to the movement direction of the target object 803 (the falling direction in FIG. 8 ), an even greater cognitive load can be imposed. In other words, the user 220 must move the controllers 234, 235 while taking into account the vertical movement, the difference in the horizontal generation position, and the difference in the falling position in the depth direction, which also tests the user's spatial cognitive ability. In this way, by adjusting the type, number, size, spatial extent, position, amount, etc. of information included in the notification image and notification sound, including the trigger object, in addition to changing the predetermined time of the task, it is possible to quantitatively adjust and control the complexity of the information to be memorized, i.e., the cognitive load that the user's brain must process information.

[0085] The evaluation update unit 215 evaluates the user's cognitive ability based on information such as whether the avatar object arrived at the three-dimensional target position represented by the target object in a timely and accurate manner, the time interval and number of target objects from the notification of their appearance until their appearance, and the level of attention-disturbing load of the background image. The required accuracy of point estimation also varies depending on which of the various modes (display screens) shown in Figures 2A and 5 to 8 the user is to execute. For example, compared to the case where there is no background as in Figure 2A, the required accuracy of point estimation increases in the order of Figures 5, 8, and 6. Furthermore, as mentioned above, the required accuracy of point estimation also changes depending on the background sound, which affects joint coordination.

[0086] 9 is a flowchart showing the processing flow in the rehabilitation support device 210. In step S901, as a calibration process, a rehabilitation movement goal is initialized to suit the user. Specifically, each user is first asked to perform a calibration task of acquiring a movement range, and the range is set as an initial value, and then the goal is initialized to suit the user. This initial value may be set to suit the movement range of the user, or may be determined by a therapist providing treatment taking into account the user's physical cognitive function and the desired therapeutic effect.

[0087] Next, in step S903, the right task (that is, display of the goal object by the display control unit 212 and evaluation of achievement by the avatar object) is started.

[0088] In step S905, the left-side task (that is, display of the goal object by the display control unit 212 and evaluation of achievement by the avatar object) is started.

[0089] In step S909, the degree of task achievement is obtained and points are added. After a task on one side is generated, the next task on the other side is generated without waiting for the evaluation of the achievement of the previous task, and these processes may be processed in parallel or with a time lag. Furthermore, the first task may start on either the left or right side, and the last task may also start on either the left or right side. In other words, the flow described here is merely an example and is not limited to the present invention.

[0090] <Data> FIG. 10 is a diagram showing the results of using the rehabilitation support system 200 according to this embodiment.

[0091] The study included 13 cases, with a mean age of 60±21 years. Seven (53.9%) were male, five (38.4%) had cerebrovascular disease, four (30.8%) had orthopedic disease, and four (30.8%) had neurodegenerative disease. All patients had undergone standard rehabilitation for more than six months and had been told that further improvement was unlikely. Even after the first 20 minutes of intensive rehabilitation focused on the affected side, STEF, TUG, and TMT-A scores did not improve significantly (see Figure 10 for the graph of the 20-minute standard intervention). STEF (Simple Test for Evaluating Hand Function) is a test of upper limb function. TUG (Timed Up and Go test) is a comprehensive test of walking function, trunk function, and balance function. TMT-A (Trail-Making Test-A) is a cognitive function test. Higher STEF scores indicate better performance, while lower TUG and TMT-A scores indicate better performance. All data are shown as mean ± standard deviation. Statistical analysis was performed using a paired t-test, with p < 0.05 as the criterion for statistical significance.

[0092] On the other hand, by using the system 200 according to this embodiment to perform rehabilitation alternating between the affected and healthy sides while paying attention to depth, STEF, TUG, and TMT-A all improved.

[0093] That is, as in this embodiment, alternating left and right training with an awareness of depth not only improves upper limb function but also trunk function and balance function, thereby improving walking function (physical function), and further improving spatial cognition and attention. It has been inferred that improvements in the inner ear, vestibular system, tactile sense, thermal pain sense, position sense, and deep sensation also contribute to improvements in STEF, TUG, and TMT-A. This demonstrates that alternating left and right depth-aware rehabilitation is effective in maximizing brain plasticity and maximizing the efficiency of the brain's motor, cognitive, and sensory learning. Other experimental results have shown that treatments described in this application improve more quickly when performed alternating left and right compared to when not performed. The bilaterally symmetrical distribution of the motor cortex shown in Non-Patent Document 1 and Figure 2B can be considered findings supporting this finding. [Other Embodiments] While the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. The configuration and details of the present invention may be modified in various ways that are understandable to those skilled in the art within the technical scope of the present invention. Furthermore, systems or devices that combine the individual features of each embodiment in any way are also within the technical scope of the present invention. The present invention may be applied to a system consisting of multiple devices or to a single device. Furthermore, the present invention may also be applied to a system or device in which an information processing program that implements the functions of the embodiments is supplied to the system or device and executed by an embedded processor. The technical scope of the present invention also includes a program installed on a computer to implement the functions of the present invention, a medium storing the program, a server from which the program is downloaded, and a processor that executes the program. In particular, the technical scope of the present invention includes at least a non-transitory computer-readable medium storing a program that causes a computer to execute the processing steps included in the above-described embodiments.

[0094] This application claims priority based on Japanese Patent Application No. 2023-97169, filed on June 13, 2023, the disclosure of which is incorporated herein in its entirety.

Claims

1. An information processing system comprising: a blocking unit that blocks all or part of a user's body from view; a request unit that requests the user to perform a physical movement such as superimposing a part of the user's body or an object operated by the user on a target object; and a control unit that controls the difficulty of the physical movement in accordance with a phenomenon known as joint correlation, in which joints that are not necessary for the movement requested by the request unit move unintentionally in an environment that makes abnormal joint movements apparent, generated by the blocking unit and the request unit.

2. The information processing system according to claim 1, further comprising a feedback section for feeding back to the user the achievement of the physical movement requested by the request section.

3. An information processing system according to claim 2, wherein the feedback unit provides feedback stimulating two or more of the user's five senses each time the physical movement requested by the request unit is accomplished.

4. The information processing system according to claim 1, further comprising a joint connection evaluation unit for evaluating the joint connection.

5. The information processing system according to claim 1, wherein the request unit or the control unit controls the user's body movements so that they alternate between left and right.

6. The information processing system of claim 1, wherein the control unit controls the difficulty of the physical movement by changing at least one of the size of the target object, the distance from the user, the angle, the speed, the presence or absence of a background, the presence or absence of music, and the size of a visual assistance image.

7. The information processing system of claim 1, wherein the blocking unit is a head-mounted display that displays a virtual space, and the request unit generates the target object and an avatar object representing a part of the user's body or an object operated by the user in the virtual space, and displays them on the head-mounted display.

8. The information processing system of claim 1, wherein the blocking unit is a projection mapping system that displays a part of the background on all or part of the user's body, or a contact lens type display that displays a virtual space, and the request unit uses the projection mapping system to display the target object and an avatar object representing a part of the user's body or an object operated by the user on the background, or generates the target object and an avatar object representing a part of the user's body or an object operated by the user in the virtual space and displays them on the contact lens type display.

9. An information processing method using an information processing system equipped with a blocking unit, a request unit, and a control unit, comprising: a blocking step of blocking all or part of a user's body from view by the blocking unit; a request step of requesting the user to perform a physical movement such as superimposing a part of the user's body or an object operated by the user on a target object by the request unit; and a control step of controlling the difficulty of the physical movement in accordance with a phenomenon known as joint coupling, in which joints not necessary for the movement requested by the request unit move unintentionally, in an environment that makes abnormal joint movements apparent, generated by the blocking step and the request step.

10. An information processing program that causes a computer to execute the following steps: a blocking step of making all or part of a user's body invisible; a requesting step of requesting the user to perform a physical movement such as overlapping a part of the user's body or an object operated by the user with a target object; and a control step of controlling the difficulty of the physical movement in accordance with a phenomenon called joint correlation, in which joints not necessary for the movement requested in the requesting step move unintentionally, in an environment that makes abnormal joint movements apparent, generated by the blocking step and the requesting step.

Images