Information processing system, information processing method, and information processing program

Abstract

According to the invention, the cognitive ability or exercise ability of the user is effectively improved. The information processing system includes: a blocking unit that makes all or parts of a user's body invisible; a request unit that requests the user to perform a body action in which a part of the body of the user or an object operated by the user overlaps with a target object; and a control unit that controls the difficulty of the body movement in accordance with the movement of a joint part different from the body part on which the body movement is performed, in an environment in which the abnormal movement of the joint generated by the shielding unit and the requesting unit appears.

Description

TECHNICAL FIELD

[0001] The present application relates to an information processing system, an information processing method, and an information processing program. BACKGROUND

[0002] In the above technical field, Patent Literature 1 discloses a system for treating a hemiplegic patient after a stroke.

[0003] LIST OF CITATIONS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: Japanese Patent Publication JP 2015-228957

[0006] NON-PATENT LITERATURE

[0007] Non-Patent Literature 1: Evan M. Gordon et al., “A somato-cognitive action network alternates with effector regions in motor cortex,” published in Nature, online publication date: April 19, 2023. SUMMARY

[0008] TECHNICAL PROBLEM

[0009] However, in the technology described in the above documents, when the user makes or does not make a certain action, there is no indicator indicating whether the next step should be entered. On the other hand, as disclosed in Non-Patent Literature 1, it has been known that there is a brain region that controls coordinated movements of multiple joint linkages with respect to neural circuits in the brain.

[0010] The present application can provide a technology that solves the above problems.

[0011] SOLUTION TO PROBLEM

[0012] One example aspect of the present application provides an information processing system including: an occlusion section that makes all or a part of a user's body invisible; a request section that requests the user to perform a body action that overlaps a part of the user's body or an object operated by the user with a target object; and a control section that controls a difficulty of the body action in accordance with a movement of a joint section different from a body section that performs the body action, in an environment in which an abnormal action of the joint section is manifested by the occlusion section and the request section.

[0013] Another example aspect of the present application provides an information processing method including: an occlusion step of making all or a part of a user's body invisible by an occlusion portion; a request step of requesting the user to perform a body action that overlaps a part of the user's body or an object operated by the user with a target object by a request portion; and a control step of controlling a degree of difficulty of the body action in accordance with a phenomenon called joint linkage in which a joint portion different from a body portion that performs the body action moves in an environment in which an abnormal action of a joint is exhibited by the occlusion step and the request step, by a control portion.

[0014] Still another example aspect of the present application provides an information processing program that causes a computer to execute a method including: an occlusion step of making all or a part of a user's body invisible; a request step of requesting the user to perform a body action that overlaps a part of the user's body or an object operated by the user with a target object; and a control step of controlling a degree of difficulty of the body action in accordance with a movement of a joint portion different from a body portion that performs the body action in an environment in which an abnormal action of a joint is exhibited by the occlusion step and the request step.

[0015] Advantages of the present application

[0016] According to the present application, it is possible to effectively improve a cognitive ability or a motor ability of a user. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A block diagram showing a configuration of an information processing system according to a first example embodiment;

[0018] Figure 2A A block diagram showing a configuration of a rehabilitation support system according to a second example embodiment;

[0019] Figure 2B A view for explaining a configuration of the rehabilitation support system according to the second example embodiment;

[0020] Figure 2C A view for explaining a configuration of the rehabilitation support system according to the second example embodiment;

[0021] Figure 3A A diagram showing an example of an operation panel screen of the rehabilitation support system according to the second example embodiment;

[0022] Figure 3B A diagram showing another example of the operation panel screen of the rehabilitation support system according to the second example embodiment;

[0023] Figure 4A A diagram showing an example of a task data table of the rehabilitation support system according to the second example embodiment;

[0024] Figure 4B A diagram illustrating another example of a task data table for a rehabilitation support system according to a second example embodiment;

[0025] Figure 5 A diagram illustrating an example of a display screen on a head-mounted display of a rehabilitation support system according to a second exemplary embodiment;

[0026] Figure 6 A diagram illustrating an example of a display screen on a head-mounted display of a rehabilitation support system according to a second exemplary embodiment;

[0027] Figure 7 A diagram illustrating an example of a display screen on a head-mounted display of a rehabilitation support system according to a second exemplary embodiment;

[0028] Figure 8 A diagram illustrating an example of a display screen on a head-mounted display of a rehabilitation support system according to a second exemplary embodiment;

[0029] Figure 9 A flowchart illustrating the processing procedure of the rehabilitation support system according to the second example embodiment; and

[0030] Figure 10 Data demonstrating the rehabilitation results of a rehabilitation support system according to a second example embodiment are shown. Detailed Implementation

[0031] Exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement of components, numerical expressions, and values ​​set forth in these exemplary embodiments do not limit the scope of the invention.

[0032] [First Example Implementation]

[0033] Reference Figure 1 An information processing system 100 according to a first exemplary embodiment of the present invention is described. The information processing system 100 is a system that provides an environment that makes abnormal joint movements apparent.

[0034] like Figure 1 As shown, the information processing system 100 includes an occlusion unit 101, a request unit 102, and a control unit 103. The occlusion unit 101 makes all or part of the user 110's body invisible. The request unit 102 requests the user 110 to perform a body movement that overlaps a part of the user 110's body or an object 121 operated by the user with a target object 111. The control unit 103 controls the difficulty of the body movement based on the movement 131 of joints that are different from the body parts that the user 110 moves to perform the body movement.

[0035] With the above configuration, it is possible to cause an abnormal movement of a joint different from a body part that the user moves for a physical movement to appear, and further, it is possible to effectively improve the cognitive ability or the motor ability of the user with the abnormal movement as an index.

[0036] [Second Example Embodiment]

[0037] A rehabilitation support system 200 according to a second example embodiment of the present application will be described with reference to Figure 2A A rehabilitation support system 200 according to a second example embodiment of the present application will be described with reference to

[0038] Figure 2A A block diagram for explaining a configuration of the rehabilitation support system 200 according to the present example embodiment is shown. The rehabilitation support system 200 according to the present example embodiment improves at least one function selected from the group of: physical functions such as upper limb function, walking function, trunk function, and balance function; cognitive functions including spatial cognitive function, attention function, and higher brain function; and sensory functions including inner ear, vestibular system, tactile sensation, temperature sensation and pain sensation, position sensation, and depth sensation.

[0039] As shown in Figure 2A The rehabilitation support system 200 includes a rehabilitation support device 210 as a request unit and a control unit, two base stations 231 and 232, a head-mounted display 233 as an occlusion unit, and two control units 234 and 235. The user 220 sits on a chair 221, wears the head-mounted display 233, and moves the body according to the display on the head-mounted display 233. In the present example embodiment, a movement performed while sitting on a chair will be described as an example. However, the present application is not limited thereto, and the movement can be performed while standing or walking, on a bed, in a supine position or prone position, or while running or performing other specific movements. Further, the user 220 holds the control units 234 and 235 with both hands here, but the present application is not limited thereto, and the control units can be held or attached to a body part other than the hand, such as a foot or a trunk.

[0040] The two base stations 231 and 232 sense the motion of the head-mounted display 233 and the motion of the control sections 234 and 235, and transmit them to the rehabilitation support apparatus 210. The rehabilitation support apparatus 210 controls the display on the head-mounted display 233 based on the motion of the head-mounted display 233. More specifically, the viewpoint position and direction in the virtual space are changed in accordance with the position and direction of the head-mounted display 233, and the image to be displayed on the head-mounted display 233 is decided. The rehabilitation support apparatus 210 also evaluates the rehabilitation movement of the user 220 based on the motion of the control sections 234 and 235.

[0041] Note that the head-mounted display 233 can be a non-transmissive type, a video see-through type, an optical see-through type, or a glasses type. In the present example embodiment, a virtual space of VR (Virtual Reality) is presented to the user. However, it is also possible to superimpose and display a real space and a virtual space like AR (Augmented Reality), or to reflect physical information on a virtual space like MR (Mixed Reality), or it is possible to use a hologram technology as an alternative means.

[0042] What is important here is that the whole or part of the user's body is made directly invisible through the head-mounted display 233. In a real space, generally, a person unconsciously fine-tunes in the brain while visually recognizing the movement of his or her own body, thereby reaching a target. In the present system, since the user 220 cannot directly see his or her own hands in a real space, the user's brain cannot accurately correct the movement of the hands as usual. Therefore, a coordination movement disorder is manifested, and an abnormal movement of a joint different from the body part that performs a body action can be easily detected. At this time, for example, a projection mapping system that displays a background on the whole or part of the user's body can be employed as the occlusion section. A contact lens type display that displays a virtual space can also be employed as the occlusion section.

[0043] In the present example embodiment, the control sections 234 and 235 held in the hands of the user 220, and the base stations 231 and 232 are described as examples of sensors for detecting the position or movement of the user's hands or head. However, the present application is not limited to this. A video camera (including a depth sensor) that detects the user's movement or "the position of the hands themselves" through image recognition processing, a sensor that detects the position of the user's hands through temperature detection, a watch type wearable terminal worn on the user's arm, a motion capture device, and the like can also be applied to the present application in conjunction with the movement detection section 211. That is, the use of a three-dimensional tracking device or a motion analysis device such as Kinect®, or the attachment of a marker to the body is also one of the example embodiments.

[0044] The rehabilitation support apparatus 210 includes the movement detection section 211, the display control sections 212 and 213, the feedback section 214, the evaluation update section 215, the task set database 216, the setting section 217, and the movement guidance section 218.

[0045] The action detection section 211 acquires the positions of the control sections 234 and 235 held by the user 220 via the base stations 231 and 232, and detects the rehabilitation action of the user 220 based on the position change of the hands of the user 220.

[0046] The display control section 212 generates and displays the target objects 242a and 242b for prompting the user 220 to perform the three-dimensional body action in the virtual space 240. In particular, the display control section 212 generates the target object 242a for prompting the part of the left side body of the user 220 to perform the rehabilitation action and the target object 242b for prompting the part of the right side body of the user 220 to perform the three-dimensional rehabilitation action in the virtual space 240. That is, the display control section 212 also functions as a request section that requests the user to reach the target object.

[0047] Note that here, the display control section 212 requests to reach the virtual target displayed on the head-mounted display, but the present application is not limited to this, and the target can also be an object set in the real space. The target can be an object suspended by two or more lines, or an object floating in the air by ultrasonic vibration. Alternatively, the target can be an object three-dimensionally displayed by holography. The shape of the target object as the target is not limited to a spherical shape, and can be a triangular shape, a rectangular shape, or a disc shape. It can also be the shape of a certain character. The concept of the present application also includes a system that partially blocks the user's vision by using an optical see-through type or a smart glasses type head-mounted display, allows the user to visually recognize only the target, while making the user's own movement invisible.

[0048] The display control section 212 generates the target objects 242a and 242b in the virtual space, and moves them from the upper direction to the lower direction of the user 220. Therefore, on the head-mounted display 233, each target object is displayed so that the display position and size gradually change (for example, gradually become large and then small). Note that for the moving direction of the target object, for example, the target object can be raised from the floor surface to above the head. In addition, the target object can be moved in any three-dimensional manner, including movement in the depth direction or the left-right direction, and movement in the vertical direction; or according to the stage of the user, the target object can be fixed at a specific coordinate position without moving.

[0049] Further, the display control section 212 generates avatar objects 241a and 241b that move in the virtual space in accordance with the detected user motion. Here, the avatar object 241a is an object that moves in correspondence with the control section 234 operated by the left hand and indicates the position of a part of the user's left side body. The avatar object 241b is an object that moves in correspondence with the control section 235 operated by the right hand and indicates the position of a part of the user's right side body. The avatar object is not limited to an object that moves in correspondence with the control section position. As described above, it can be a virtual object that moves with the change in the position of a part of the user's body using a three-dimensional body tracking technique. The object operated by the user is not limited to Figure 2A The control section shown, for example, can be an object like a racket or a golf club. The display control section 212 requests the user to perform a body motion that makes the avatar objects 241a and 241b overlap the target objects 242a and 242b.

[0050] Even if the avatar objects 241a and 241b are displayed, in the case where a part or the whole of the body is invisibly occluded, the burden on the user's brain is greater than in the case where the actual body motion is directly visible. This is because in the state where the user's body (including the control section in the real space) is not visible, the brain needs to work (so-called point estimation) to estimate the position (point) of the control section in the real space by observing the position (point) of the control section in the virtual space. In this case, the brain needs to very strongly imagine the state of the body without relying on vision, which is medically expressed as "strong feedforward is required". For the brain that is required to perform strong feedforward, it is more difficult to adjust the body coordinately. Therefore, a coordination disorder is manifested, and abnormal motion of a joint different from the part where the body motion is performed is likely to occur. Further, when strong feedforward that requires point estimation is performed, a contraction of a deep muscle of the body can be caused by stimulation via the pyramidal tract.

[0051] Therefore, for example, when the user moves the right arm so that the avatar object 241b of the right control section 235 reaches the target object 242b on the right side, he / she can unconsciously force the knee joint of the left leg, causing the lower leg part of the left leg to move forward or backward. Here, the phenomenon where an unnecessary joint moves unintentionally for the requested motion is called articular linkage and serves as an index for improving the user's motion. Note that the articular linkage is manifested even without an anatomical connection called an anatomic chain. For example, when the toe joint is actively moved, the temporomandibular joint, the hip joint, or the shoulder joint appears abnormal.

[0052] It can be considered that this phenomenon is supported by the fact that the motor area that controls coordination motion mentioned in Non-Patent Literature 1 is spread over all joints Figure 2B ). Referring to Figure 2BIn the latest neurology, it is found that cranial nerves are grouped by symmetrical parts (e.g., left and right hands, left and right legs, left and right eyes), and their motor areas are arranged symmetrically in the brain. It has also recently been learned that there seem to be nerve areas that control coordination of body parts, represented by puppets 264 to 266, between the motor area 261 of the leg and the motor area 262 of the hand, and between the motor area 262 of the hand and the motor area 263 of the face. It can be considered that, due to the presence of nerves that coordinate body parts like puppets 264 to 266, joint linkage during movement appears in a specific environment where point estimation is required when all or part of the body according to the present application is not visible.

[0053] In fact, the present inventors completed the present application by observing actual rehabilitation movements of users before the publication of Non-Patent Literature 2, but it can be said that the usefulness and logic thereof are provided with a theoretical basis by the paper of Non-Patent Literature 1.

[0054] Figure 2A Knee joint movement is shown as an example of joint linkage manifestation. However, the present application is not limited thereto, and all non-autonomous movements of joints different from the body part that performs the requested movement are included in the concept of the present application. For example, as shown in Figure 2C In some cases, joint planes 271 and 272 are sensed and the inclination thereof is calculated, thereby detecting the simultaneous occurrence of shoulder joint or trunk (cervical spine joint or thoracic spine joint) twist and hip (lumbar spine joint or hip joint) twist.

[0055] More specifically, twist of the cervical intervertebral joint (which is expressed in the direction of the head), twist of the thoracic intervertebral joint (which is expressed in the angle of the shoulder), elevation or abduction of the shoulder joint (opening of the axillary fossa), extension / flexion of the knee joint, hip joint, or ankle joint (knee or toe moving up and down or turning in, turning out), extension / flexion of the elbow joint (arm not requested to move is fixed in the flexion position), extension / flexion of the temporomandibular joint (mouth opening), flexion / extension of the hand joint or finger joint, and flexion (claw toe) or extension of the toe joint all occur as joint linkage. Such joint movements include not only flexion and extension, but also adduction, abduction, internal rotation, external rotation, circumduction, and combinations thereof, as well as movements not mentioned here.

[0056] The camera 236 can be installed, and the joint linkage evaluation section 219 can be provided, which detects and evaluates the position, twist, tilt, movement speed, and acceleration of each joint of the user during the rehabilitation movement. The joint linkage evaluation section 219 detects the movement of each joint of the user 220, and evaluates whether joint linkage has occurred based on the amplitude, speed, and timing of the movement. The rehabilitation movement that makes joint linkage small is requested to be performed, and the user is caused to achieve it. The brain is compared to a CPU with a calculation capacity of 100, and if the information processing process of joint linkage, which requires a capacity of 80, is reorganized to require 20, a more complicated movement than the request can be performed, and the movement ability is improved. Alternatively, if the CPU capacity available for cognitive processing is improved, the cognitive ability is also improved. When the user is photographed by the camera 236, and the photographed image is compared with the image of the normal state (before movement), or compared with the normal movement model, abnormal twist or movement of the joint surface can be detected. Alternatively, a sensor is attached to the joints of the user (for example, both shoulders, both elbows, wrists, both knees, and both ankle joints) (or motion capture is used), and the position of the joint is detected by the base stations 231 and 232, thereby detecting joint linkage. The operator can of course visually recognize the occurrence of joint linkage and input it to the system. The inclination of the head-mounted display 233 can be detected by the base stations 231 and 232, and the joint linkage of the neck can be sensed. For example, if the head-mounted display 233 is inclined by 30°, it can be judged that the trunk angle is 15° (judgment based on the camera image), and the neck angle is 15°. The inclination of the wrist joint or the like can be determined by detecting the position and inclination of the control sections 234 and 235 by the base stations 231 and 232. Furthermore, the joint linkage evaluation section 219 can express the joint linkage movement as distance or acceleration using a heat map on a bull's eye chart, or can express the change in joint linkage over time, and provide information useful for movement difficulty control. More specifically, how the joint linkage changes from the start to the end of the requested reaching movement can be visualized, or the amplitude of the joint linkage of the joint can be compared. This facilitates the judgment of which joint should be focused on.

[0057] Six joints including the hip joint, the knee joint, the ankle joint, the neck joint, the trunk, and the elbow joint can be collectively sensed. The joints can be classified into more categories, and all joint units (68 in the human body) can be observed, or, on the contrary, can be designed to focus on only one or a small number of joints that are easy to sense abnormalities. Furthermore, a plurality of evaluation indexes can be integrated into one comprehensive index or parameter as a total score. That is, the joint linkage of the whole body or important parts can be weighted and integrated, and the overall joint linkage evaluation (joint linkage score calculation) can be performed.

[0058] In actions requiring a 2cm sensor to cover a 10cm target, point estimation accuracy is low. In actions requiring a 1cm sensor to cover a 1cm target, precise point estimation is needed. In these situations, using obstructive devices such as head-mounted displays can be inconvenient for users because visual information from the real world is blocked. Cranial nerves need to be activated for point estimation, and coordination disorders become apparent, with joint synkinesis becoming more pronounced. This is one of the mechanisms underlying joint synkinesis.

[0059] As rehabilitation progresses (the user's motor skills improve), the required accuracy of point estimation increases by reducing the size of both the target and the sensor, leading to a stronger manifestation of compensatory joint synergy. Joint synergy decreases when the user is repeatedly instructed to perform rehabilitation movements or guided through verbal communication or physical contact in this state, making it easier for them to execute the correct movements. This allows for a reshaping of overall coordination and improves coordination disorders.

[0060] The joint linkage assessment unit 219 can be viewed on the operator's screen ( Figure 3A and 3B The system displays the position and magnitude of the sensed joint movements. It can also be further equipped with a comparison unit that compares joint movements with past movements or with a normal motion model, and notifies the operator of the comparison results. This allows the operator to more accurately identify changes in joint movements.

[0061] If it is determined that joint mobility has decreased below a certain threshold, the load of rehabilitation exercises should be increased. If it is determined that joint mobility has increased above that threshold or remains at a high level (joint mobility has not improved), the load of rehabilitation exercises should be decreased.

[0062] Note that the presence or absence of joint synkinesis is influenced by the following factors: the size, angle, height, distance, and direction of movement of the target object; the presence or absence of movement; the speed, color, shape, and visibility of movement; the size, color, shape, and visibility of overlapping indicators or devices (such as controls) used in point estimation; the presence or absence of background information; and the presence or absence of background sound. One goal of the rehabilitation movements in this system is to induce and eliminate joint synkinesis. In other words, joint synkinesis serves as a guide for treatment. Sometimes, joint synkinesis can be eliminated by pushing the user's knee or shoulder during rehabilitation movements. That is, disordered neural circuits in the brain can be manually and forcibly corrected.

[0063] Return to reference Figure 2AAccording to the direction and position of the head-mounted display 233 detected by the action detection section 211, the display control section 212 displays images of the avatar objects 241a and 241b and the target objects 242a and 242b on the display screen 240. The images of the avatar objects 241a and 241b and the target objects 242a and 242b are superimposed on a background image 243. Here, the avatar objects 241a and 241b have the same shape as the control sections 234 and 235. However, the present application is not limited to this, and they can also be in the shape of hands. Furthermore, the sizes, shapes, and colors of the left and right sides can be different. One or more buttons are provided for each of the control sections 234 and 235, and the control sections are configured to make various settings including initial settings (e.g., origin settings) by operating the buttons. However, the button functions can be disabled, or the buttons themselves can not be provided, and all settings can be performed using an external operation section. The background image 243 is taken from a virtual space containing a horizon 244 and a ground object 245. The appearance of the background image 243, the avatar objects 241a and 241b, and the target objects 242a and 242b changes with the movement of the line of sight (the position and direction of the head-mounted display).

[0064] The display control section 212 generates an avatar object indicating the position of a part of the right side of the body and a target object indicating the target position of the avatar object of the right side in similar colors or the same shape. Furthermore, the display control section 212 generates an avatar object indicating the position of a part of the left side of the body and a target object indicating the target position of the avatar object of the left side in similar colors or the same shape. The colors or shapes of the left side avatar object and the left side target object are preferably set to be different from the colors or shapes of the right side avatar object and the right side target object. However, the avatar object and the target object of the same side are not necessarily always set to similar colors or the same shape, and the colors of the right side and the left side are not necessarily different. These settings can be made in any manner according to the user's sense of use.

[0065] Here, as an example, the avatar objects 241a and 241b are colored, for example, blue and red, and the target objects 242a and 242b are also colored blue and red. The task is achieved by bringing the blue avatar object 241a into contact with the blue target object 242a. Similarly, the task is achieved by bringing the red avatar object 241b into contact with the red target object 242b. That is, even if avatar objects of different colors come into contact, the task cannot be achieved. An example is shown here in which the objects are colored blue and red. However, for colorblind users, other colors (such as yellow and green, etc.) can be displayed. Alternatively, the objects can be distinguished by not using colors but by using shapes; using linguistic labels such as "left," "right," "L (left)," or "R (right)"; using symbols such as stars, triangles, or circles; or like Figure 2ADifferent patterns (stripes) are used for the target objects 242a and 242b as with the avatar objects 241a and 241b.

[0066] The action guiding section 218 provides guidance so that the rehabilitation action of the part of the user's right side body and the rehabilitation action of the part of the user's left side body are performed alternately. More specifically, the action guiding section 218 guides the user's rehabilitation action by displaying characters, colors, or figures indicating left and right, outputting a voice for identifying "right" and "left", or providing a tactile stimulus by vibrating the control section held by the user's left and right hands, or by using a method of touching or patting the shoulder or arm.

[0067] When the avatar objects 241a and 241b touch the target objects 242a and 242b, the display control section 212 causes the target objects 242a and 242b to disappear. The feedback section 214 determines that the target action has been achieved and performs visual feedback by displaying a message for notifying or informing that the target action has been achieved. If the sensor center of the avatar objects 241a and 241b reaches the center of the target objects 242a and 242b, "excellent" is displayed. Even if the peripheral area is reached, "well done" is displayed for each task. In this way, the completion of each requested body action is fed back to the user. The feedback section 214 only needs to perform feedback by stimulating one or more of the five senses of the user. If feedback is performed by stimulating two or more of the five senses of the user, the cognitive ability or motor ability of the user can be more effectively improved, or both.

[0068] That is, the feedback section 214 notifies the user 220 of the completion of each rehabilitation action for the target objects 242a and 242b. The notification method can take various forms. The user can be made aware of the degree of achievement of the target action by temporarily displaying the character "excellent" or "well done" in the display screen, or a similar voice or sound effect can be used to inform the user of the degree of achievement of the target action. Furthermore, in the control sections 234 and 235, only the control section that has performed the contact target object action can be vibrated to notify the user of the degree of achievement of the target action. Alternatively, with respect to achievement notification, for example, a hierarchical classification such as "perfect achievement", "imperfect achievement", and "non-achievement" can be used to notify the degree of achievement.

[0069] For example, if the shortest distance between the sensor object included in the avatar object 241a or 241b and the target object 242a or 242b falls within a predetermined range, the target is achieved, and the target object 242a or 242b disappears. At this time, if the shortest distance between the target object 242a or 242b and the sensor object (for example, a spherical object including the center point of the top end of the avatar object 241a or 241b) included in the avatar object 241a or 241b is equal to or less than a threshold value, the target is completely achieved. For example, "Excellent" is displayed, and a corresponding voice is output for feedback. Such feedback that stimulates a plurality of different senses is called multi-channel biofeedback. At the same time, the control sections 234 and 235 can be vibrated, or olfactory or gustatory stimulation can be given. That is, as feedback, the feedback section 214 can stimulate one of the five senses (sight, hearing, touch, taste, and smell). For sensory stimulation, two senses can be combined, three or more senses can be combined, or all senses can be stimulated. In the case of performing point estimation, if feedback is given every time the requested action is achieved, an opportunity for the brain to correct and improve dyskinesia is obtained. That is, joint linkage and dyskinesia are improved. As the number of senses stimulated in feedback increases, the improvement efficiency improves or the improvement effect lasts longer. For example, if two senses are stimulated, a symptom that originally takes several days to improve is improved in several hours, compared to one sense. Alternatively, if three senses are stimulated, a symptom that originally takes several hours to improve is improved in several minutes, compared to two senses. Furthermore, even for a symptom that is improved by one sense feedback but immediately recurs, when two senses are stimulated, the improvement effect can be maintained for several days. Alternatively, even for a symptom that is improved by two senses feedback but immediately recurs, when three senses are stimulated, the improvement effect can be maintained for several weeks. In order to express such a therapeutic property, in a medical scene, the therapy is also called "brain reprogramming therapy (BRT)", "brain re-wiring therapy (BRT)", or "motor coordination therapy (MCT)".

[0070] Depending on the degree of reduction in the distance between the sensor object and the target objects 242a and 242b, the rehabilitation action can be evaluated in three or more levels. Furthermore, two or more target objects can also be generated and displayed in the virtual space at the same time.

[0071] The display control section 213 displays a radar screen image 250 on the display screen 240 of the head-mounted display 233. The radar screen image 250 is a notification image for notifying the user of the generated target object 152. The radar screen image 250 notifies the user of which relative direction the positions of the generated target objects 242a and 242b are located relative to the reference direction (initially set as the front of the user sitting straight on the chair) in the virtual space. The radar screen image 250 also notifies the user of how far the positions of the generated target objects 242a and 242b are from the user 220. Note that the notification image is not limited to the radar screen image, and can be notified by characters, arrows, symbols, illustrations, or the type, intensity, flicker, and the like of light or color. The notification method is also not limited to the image, and can use voice, vibration, or a combination of some of voice, vibration, and image. The display control section 213 displays the radar screen image 250 at the center (e.g., in the range of -50° to 50°) of the display screen 240 of the head-mounted display 233 regardless of the direction of the head of the user 220. However, the display portion is not limited to the center, and can be any position such as the corners, upper end, lower end, left end, and right end of the screen. The patient can estimate the difficulty of the exercise action to be performed next based on the position, angle, and number of information of the target objects displayed on the radar screen. If a more difficult action is predicted by the patient, the joint linkage is more apparent. That is, it can be said that the display control section also functions as a control section that controls the requested action difficulty of the patient.

[0072] The radar screen image 250 includes a head image 251 representing the head of the user viewed from above, a block image 252 obtained by dividing the periphery of the head image 251 into a plurality of blocks, and a sector image 253 as a field of view image representing the field of view of the user. A target position image representing the position of the target object is displayed by coloring, flickering, or lighting the blocks in the block image 252. This allows the user 220 to know whether the target object is on the left or right side relative to the direction he or she faces. Note that in the present example embodiment, the block image 252 is fixed, and the sector image 253 is moved. However, the present application is not limited to this, and the sector image 253 and the head image 251 can be fixed, and the block image 252 can be moved according to the direction of the head. More specifically, if the head is turned to the left, the block image 252 can be rotated to the right.

[0073] The feedback section 214 preferably changes the message type according to the evaluation of the rehabilitation movement by the display control section 212. For example, if the sensor object contacts the center of the target object 242a or 242b, "Excellent" is displayed. If the sensor object contacts only the peripheral portion of the center of the target object 242a or 242b, "Well done" is displayed. The size of the target objects 242a and 242b and the size of the peripheral portion can be set by the setting section 217. The size of the sensor object can also be set by the setting section 217. If the shortest distance between the target object 242a or 242b and the sensor object included in the avatar object 241a or 241b is not more than a first threshold value, "Excellent" is displayed due to the target being completely achieved, and the corresponding voice "Excellent" is output for feedback. If the shortest distance between the target object 242a or 242b and the sensor object included in the avatar object 241a or 241b is not less than the first threshold value and not more than a second threshold value, "Well done" is displayed due to the target being achieved, and the corresponding voice "Well done" is output for feedback. Note that the voice output does not need to be the same as the message, and a non-verbal sound effect such as "ding" can also be used. Note that as the distance required for the sensor object to contact the target object becomes shorter, more precise movement is required, and thus the brain needs to issue more precise and higher-level movement instructions to the body. As described above, this transfer of images to the brain is called feedforward. That is, the shorter the distance between the sensor object and the target object required to complete the movement, the stronger the feedforward required. This can continuously adjust the load level of the user's movement, cognition, and senses.

[0074] The feedback section 214 almost simultaneously provides stimulation such as feedback of two or more senses (vision, hearing, touch, taste, and smell) of the five senses to the user who virtually contacted the target objects 242a and 242b at the time when the sensor object enters a predetermined distance from the center of the target object 242a or 242b, or at the time when the sensor object contacts the target object 242a or 242b (referred to as real-time multi-channel biofeedback or instant multi-channel biofeedback). If the delay from this time to the feedback is, for example, within 1 second, the effect is greater. The shorter the interval between the user action time and the feedback time (the smaller the delay), the greater the effect. The feedback section 214 provides stimulation feedback to the user's vision through the image "Excellent!" and simultaneously provides stimulation feedback to the user's hearing through the voice or sound effect output by the speaker. The achievement notification of the task of stimulating the five senses can be combined in any combination, for example, according to the type of movement.

[0075] Further, the feedback section 214 can also output the following feedbacks at the same time: stimulus feedback to the vision of the user 220 through the image "Good!", stimulus feedback to the hearing of the user 220 through the voice output from the speaker, and stimulus feedback to the touch of the user 220 through vibration of the control section 234. Further, the feedback section 214 can also output only two types of feedbacks at the same time, including stimulus feedback to the vision of the user 220 through the image "Good!" and stimulus feedback to the touch of the user 220 through vibration of the control section 234. Alternatively, the feedback section 214 can also output only two types of feedbacks at the same time, including stimulus feedback to the hearing of the user 220 through the voice "Good!" and stimulus feedback to the touch of the user 220 through vibration of the control section 234.

[0076] The movement of the control sections 234 and 235 by the user 220 is a rehabilitation movement, and the display of the target object to urge the user 220 to perform one rehabilitation movement which the user 220 should perform is called a task. Information representing one task (task data) includes: an appearance direction of the target object (right 90°, right 45°, front, left 45°, left 90° with respect to the front of the chair), a distance to the target object, a shape (size), an appearance position (distance and angle from the user), an appearance interval (time interval), a moving speed of falling or rising, size, color, which control section should be used to acquire, a number of target objects appearing at the same time, a size of a sensor object, and the like. That is, as information, the task data contains at least one selected from the following group as attributes of the target objects 242a and 242b in the virtual space 240: a moving speed, a number of display objects, a size, a display position, and a display interval. A distance in a depth direction from the user 220 to a falling position of each of the target objects 242a and 242b can be continuously set, or can be set to one of, for example, three levels. For example, a change can be made so that the target object falls in a place very close to the user 220, or falls in a position which the user 220 cannot reach without greatly leaning forward the body. This can control an exercise load applied to the user, and a load on a spatial cognition ability or a spatial understanding ability.

[0077] The evaluation update section 215 evaluates the rehabilitation action of the user 220 from the number and quality of tasks achieved by the user 220 and accumulates the score. Here, the quality of the achieved task includes "well done" or "excellent", that is, how close the avatar objects can approach the target objects. The evaluation update section 215 accumulates different scores for the achieved tasks (high score for far target, low score for near target). The evaluation update section 215 can update the task according to the accumulated score. For example, the task achievement rate (number of achieved targets / task number) can be used to update the task (attribute of the target object). The evaluation update section 215 compares the rehabilitation action detected by the action detection section 211 with the target position indicated by the target object displayed by the display control section 212, and evaluates the rehabilitation ability of the user 220. More specifically, by comparing the positions in the three-dimensional virtual space, it is determined whether the avatar objects 241a and 241b corresponding to the rehabilitation action detected by the action detection section 211 and the target objects 242a and 242b overlap. If they overlap, it is evaluated that one rehabilitation action is completed, and the score is accumulated. The display control section 212 can cause the target objects 242a and 242b to appear at different positions in the depth direction (for example, three levels of positions). The evaluation update section 215 gives different scores (high score for far target, low score for near target).

[0078] The evaluation update section 215 updates the target task according to the accumulated score. For example, the task achievement rate (number of achieved targets / task number) can be used to update the target task.

[0079] The task set database 216 stores a set of a plurality of tasks. The task indicates one rehabilitation action that the user should perform. More specifically, as information indicating one task, the position of the appearing target object, its speed and size, and the size of the avatar object at that time are stored. The task set database 216 stores a task set that determines the order in which a plurality of tasks are provided to the user. For example, the task set can be stored as a template for each hospital, or the history of the executed task set can be stored for each user. The rehabilitation support device 210 can be configured to be able to communicate with other rehabilitation support devices through the Internet. In this case, the same user can perform one task set at multiple locations, or multiple users who are far apart can share various templates.

[0080] The setting section 217 can be set to alternately generate the left target object 242a and the right target object 242b. The setting section 217 sends an instruction to the display control section 213 according to the setting.

[0081] The setting section 217 can set the generation position of the left target object 242a in the depth direction in the virtual space 240 and the generation position of the right target object 242b in the depth direction in the virtual space 240.

[0082] By controlling the generation timing of the target object and the moving speed of the target object in the vertical direction, it is possible to control the rehabilitation movement of the user to be alternately performed in the left and right sides. That is, the simplest method is to generate the left target object after sensing the arrival of the right target object, instead of generating the left and right target objects at the same time. However, the present application is not limited to this, and various control methods can be employed. That is, while generating a plurality of target objects at the same time, the moving speed is changed so that the moving speed decreases in the order of right→left→right→left, thereby encouraging the user to move alternately in the left and right sides. For example, after sensing the arrival of the right object, the left and right objects are generated at the same time, or the right object is generated first. Then, the speed of the right object can be increased so as to overtake the left object. In this way, a high load can be applied to the brain. The request unit or the control unit controls the rehabilitation movement of the user to be alternately performed in the left and right sides.

[0083] On the other hand, for example, in the case where the user is only right-side paralyzed or needs to actively rehabilitate the upper left body, when it is necessary to arrive at the right target object at a lower position and the left target object at a relatively upper position, the left target object can be generated at a high speed after a lapse of a certain period of time after the right target object is generated at a low speed. That is, at least one of the generation timing and the speed of the target object is controlled in accordance with the vertical position of the body part to be rehabilitated (in accordance with the position of the target object to be arrived at). In the rehabilitation of the knee or lower body, the generation timing or the speed of the target object is controlled so as to arrive at a lower position. In order to increase the cognitive load, the target objects are generated in the order of left→right→right. When only the first target object moves slowly, and the second and third target objects move at a high speed, the user can eventually arrive at the target objects in the order of right→left→right.

[0084] In this way, when rewiring is performed in the brain, it is possible to improve the symptoms of diseases that are difficult to treat in modern medicine, such as chronic pain or restless leg syndrome (abnormality of the basal ganglia).

[0085] The setting section 217 sets at least one selected from the group of moving speed, number of displayed objects, shape, color, size, display position (coordinate information, etc.), and display interval as an attribute of the target objects 242a and 242b in the virtual space 240. The setting section 217 can also set a delay time from the time when the generation of the target objects 242a and 242b is notified to the time when the target objects 242a and 242b are actually generated, thereby controlling the cognitive load imposed on the user 220. That is, the user needs to continuously remember and hold the action that he or she should perform until the time when the target objects are actually generated after he or she knows the position at which the target objects will be generated in the virtual space through the radar screen image 250 or the like. This "memory time" is the cognitive load for the user. Further, the setting section 217 can also control the cognitive load by changing the time, that is, not the "time until the target objects 152 are generated" but the "time until the target objects 152 approach the range reachable by the user 220". The setting section 217 can also impose the cognitive load on the user 220 by displaying a background image 243 other than the target objects 242a and 242b on the head-mounted display 233. Note that when the cognitive load is changed, it is preferable to notify the user in advance that the cognitive load will be increased or decreased. As for the notification method, the notification can be made visually using characters or symbols, by voice, or by touching a part of the body (for example, tapping the shoulder, elbow, arm, or foot).

[0086] The setting section 217 controls the difficulty of the body action according to the degree of joint linkage, that is, the movement of the joint different from the body part moved by the user for the body action. The setting section 217 as the control section controls the size of the target object, the distance from the user, the angle, the speed, the presence or absence of the background, the presence or absence of the music, and the size of the visual recognition auxiliary image as the point estimation parameter. For example, the joint linkage sometimes greatly changes depending on the presence or absence of the music. The degree of the joint linkage that has occurred even differs depending on the type of music. For example, repetitive rhythm music increases the workload of the brain for processing the background information, that is, the cognitive load, compared with the sound of a bird chirping (which has a greater influence on people with poor cognitive ability), and thus greater joint linkage occurs. The complexity of the background information also greatly relates to the workload of the brain for processing the background information, that is, the cognitive load. For example, the cognitive load becomes greater if there are some backgrounds or scenery in a three-dimensional space compared with a space empty of everything. If there are also moving animals or objects there, and these moving animals or objects appear in a wider range of the screen, the cognitive load becomes greater and the joint linkage more obviously occurs. That is, basically, the higher the required precision of the point estimation or the greater the background processing load of the brain for the sound or the background information, the smaller the remaining capacity of the brain available for the point estimation. Thus it is known that the joint linkage strongly occurs.

[0087] Figure 3A A view of the screen (operation panel) 300 operated by the operator is shown. The setting section 217 displays this operation panel 300. In the present example embodiment, the point estimation parameters (distance to the target object, height of the target object, direction, size of the target object, size of the visual recognition auxiliary image, moving speed, sensitivity = size of the sensor object, generation interval, type or presence / absence of the background information, type or presence / absence of the sound effect or BGM) are set by the intuitive operation panel 300, thereby creating a task for supporting rehabilitation. This can be done by one or a combination of a manual mode (a method of setting the point estimation parameters for each task and operating based on the task), a template mode (a method of setting the point estimation parameters for a plurality of task sets in advance), and an automatic mode of letting the device automatically generate a task. Note that it is also possible to confirm the user's basic information, various cognitive and motor function assessment indexes and test results, create a template, and set and instruct the automatic mode on the operation panel.

[0088] The display that displays the operation panel 300 can be any device, and can be an external display connected to the rehabilitation support device 210 or a display built into the rehabilitation support device 210. The operation panel 300 includes a user visual field display area 301, parameter setting areas 321 to 324, a score display area 303, a re-centering button 305, and a BGM control button 307. For ease of explanation, Figure 3A The area 306 that displays the actual state of the user 220 is included, but the operation panel 300 does not necessarily include the area 306.

[0089] The user visual field area 301 displays the image actually displayed on the head-mounted display 233. The reference direction in the virtual space is displayed in the user visual field area 301. As described with reference to Figure 2A The radar screen image 250 is displayed in the center portion of the user visual field area 301 (e.g., in the range of -50° to 50° of the viewing angle). The radar screen image 250 displays the relative direction of the positions of the target objects 242a and 242b that will appear next, with respect to the reference direction in the virtual space. In this example, the colored position in the block image 252 indicates that the target object 242 will appear at the farthest position to the left with respect to the reference direction in the virtual space. Based on the position of the sector image 253 and the direction of the head image 251, it can be seen that the user is already facing to the left.

[0090] Each parameter setting area 321 to 324 is a screen configured to set multiple parameters for defining a task. The setting unit 217 can receive input for each parameter setting area 321 to 324 from an input device (not shown). The input device can be a mouse, numeric keypad, or keyboard, or various control units, game joysticks, or touch panels; any technical component can be used.

[0091] The parameter setting areas 321 to 324 include: an input area 320 for determining the size of the left and right target objects; an input area 321 for determining the size of the visual recognition auxiliary image; an input area 322 for determining the sensor area size of each of the avatar objects 241a and 241b; an input area 323 for determining the movement speed of the target object; and an input area 324 for determining the position of the next target quasi-object to appear. The operation panel 300 also includes a checkbox 325, which sets whether to accept operations on the appearance position of the target object through various external parameter input devices such as hotkeys and numeric keypads.

[0092] Input areas 320 and 321 can be set on each of the right and left sides to define the radius (visual recognition size) of the visually recognized object that makes it easy for the user to see the location of the target object, and the radius (evaluation size) of the target object that interacts with avatar objects 241a and 241b. That is, in Figure 3A In the example shown, the user can see a circle with a radius of 20 centimeters. In reality, the task is only completed when he / she touches a sphere with a radius of 10 centimeters located at the center of that circle. If the visual recognition size is small, the user will have difficulty finding the target object. If the visual recognition size is large, the user can easily find the target object. If the assessment size is large, the allowable deviation of avatars 241a and 241b is large. If the assessment size is small, the allowable deviation of avatars 241a and 241b is small, allowing for more rigorous assessment of rehabilitation movements. The visual recognition size and assessment size can be matched. Through these settings, the precision of the feedforward—that is, the precision with which the user's brain processes information—can be quantitatively altered. Treatment methods targeting physical functions (upper limb function, walking function, trunk function, and balance function), cognitive functions (including spatial cognition and attention function), and sensory functions (including inner ear, vestibular system, touch, temperature and pain sensation, position sense, and depth sense), as well as the degree of manifestation of joint linkages, can be controlled.

[0093] In input area 322, the left and right sensor sizes (sensor object sizes) of avatars 241a and 241b can be set. If the sensor sizes are large, the task can be accomplished even if the hand is significantly deviated from the target object. Therefore, the rehabilitation movement is less difficult. Conversely, if the sensor sizes are small, the hand needs to be accurately moved to the center area of ​​the target object (assessment size). Therefore, the rehabilitation movement is more difficult. Figure 3AIn the example shown, the sensor size on both the left and right sides is 2 cm. These elements also quantitatively change the accuracy of the feedforward, i.e., the accuracy of the brain information processing, so that it is possible to control the treatment method for the body functions (upper limb function, walking function, trunk function, and balance function), cognitive functions (including spatial cognition and attention function), and sensory functions (including the inner ear, vestibular system, tactile sensation, temperature sensation and pain sensation, position sensation, and depth sensation), as well as the degree of joint linkage.

[0094] In the input area 323, the speed at which the target objects 242a and 242b move in the virtual space can be defined on both the left and right sides. In this example, the speed is set to 45 cm / sec.

[0095] The input area 324 is an image for inputting the position (distance and angle to the task) of the next task, and is shaped like an enlarged radar screen image 250. Since the check box 325 is checked, it is possible to operate by the input device. If an operation of clicking or tapping one of the blocks in the input area 324 is performed, or an operation corresponding thereto is performed by the input device, the target objects 242a and 242b are generated in the virtual space position corresponding to the specified block position.

[0096] That is, in the example shown in FIG. 27, the user 220 is instructed to make the avatar object (control part) containing the sensor part with a size of 2 cm contact the target object (ball) with a radius of 10 cm falling at a speed of 45 cm / sec at a distant position on the left at an appropriate timing in the virtual space. Figure 3A In the example shown, the task of the user 220 is to make the avatar object (control part) containing the sensor part with a size of 2 cm contact the target object (ball) with a radius of 10 cm falling at a speed of 45 cm / sec at a distant position on the left in the virtual space at an appropriate timing. Figure 3A The state in which the target object 242a appears in the user's field of view area 301 is shown. In this state, when the user 220 stretches the left arm, the avatar object 241a appears in the user's field of view area 301 as shown in the area 306. The visual recognition object 312 for improving the visibility of the target object 242a is displayed around the target object 242a. The visual recognition size set in the input area 321 is the radius of the donut-shaped visual recognition object 312. Note that the target object and the avatar object can correspond to the left side or the right side, and the user can be prompted to perform an operation of moving the left side of the body to the same side (i.e., the left side in the virtual space), or the user can be prompted to perform an operation of moving it to the right side.

[0097] At the timing at which the avatar object 241a contacts the visual recognition object 312, the target object 242a does not disappear, and the task is not perfectly completed (imperfect completion, addition of a predetermined score, and giving of a good evaluation). Only when the avatar objects 241a and 241b contact the target objects 242a and 242b, the task is perfectly completed (perfect evaluation).

[0098] On the other hand, the score display area 303 displays the total number of tasks for each position, and the point count indicating the number of times the task has been achieved. The points can be displayed in the form of scores, percentages, or a combination thereof. After a series of rehabilitation movements determined by a task set is completed, the evaluation unit 214 calculates a rehabilitation evaluation score using the values in the score list 332.

[0099] The re-centering button 305 is a button that receives a re-centering instruction from the operator to reconstruct the virtual space according to the position of the user 220. If the re-centering button 305 is operated, the display control unit 212 sets the position of the head-mounted display 233 at that moment as the origin and reconstructs the virtual space with the direction of the head-mounted display 233 at that moment as the reference direction. By operating the BGM control button 307, the background music during the task execution can be turned on or off. Eliminating the background music can reduce the task load and allow the user to focus on the body movement. In general, for users with joint linkage, if the background music is turned off, the joint linkage during the task tends to decrease. Conversely, for users without joint linkage, the joint linkage can be induced by turning on the background music.

[0100] In addition to displaying the radar screen image 250, the display control unit 213 can display an information bar 310 on the display screen 240 of the head-mounted display 233. The information bar 310 displays the player's name 311, the task achievement level 312, and the elapsed time 313 from the start of the game. The information bar 310 is parallel to the horizon 244 in the virtual space. If the patient's head position or body axis is twisted, the parallel relationship between the radar screen image 250 that follows the tilt of the head-mounted display 233 and the information bar 310 is broken. Therefore, the patient can correct the inner ear information with visual assistance, and the therapist can easily notice the twist of the patient's body axis. Figure 3A An image when the head position is tilted 20 degrees to the right is illustrated. The user interface design of the head-mounted display 233 not only provides an element for the patient to deeply participate in rehabilitation, but also has medical utility.

[0101] Figure 3B A view to show another example of the operation panel 300 is illustrated. As shown in the joint linkage message 308, the presence or absence of joint linkage and its magnitude can be detected by a camera or the like and notified to the operator as characters, images, or color changes. As described above, a heat map can also be superimposed on the user image 306, and the image area with a high joint linkage possibility can be displayed in red.

[0102] The procedure for joint linkage control point estimation accuracy is as follows. (1) Perform the rehabilitation movement using the default settings. (2) Joint linkage is too large (the angle or distance of the bow leg, shoulder twist, hip twist, head twist, etc. exceeds the threshold value). (3) Reduce the required difficulty of point estimation (change the point estimation parameters). (4) Joint linkage is moderately reduced. (5) Repeat the rehabilitation movement alternately on the left and right sides. (6) Joint linkage is eliminated. (7) Increase the required accuracy of point estimation (change the point estimation parameters). (8) Joint linkage reaches a moderate size. (9) Return to step (5).

[0103] Note that in (5), joint linkage can also be suppressed by verbal communication or physical contact with the user's body. In addition, in (7), the user can be given a load by verbal communication to achieve the same effect as increasing the required accuracy of point estimation. It is known that joint linkage of the user can be quickly eliminated by contacting the user's body.

[0104] As described above, the direct view of the user is blocked, point estimation is performed, a strong movement instruction is issued by the target object, deep muscle contraction occurs, and adjustment of the entire body is obtained. At this time, the cranial nerves are found to be damaged, which is called joint linkage, and the rehabilitation movement is repeated while controlling the required accuracy of point estimation (various parameters) until the joint linkage is reduced.

[0105] If multiple joint linkages occur, it is necessary to determine which joint linkage should be the main correction object. For example, the knee joint (hip joint) with a larger movement amplitude is corrected first, and then the next joint (for example, the foot is turned inward due to involuntary movement of the ankle joint) is processed.

[0106] If the rehabilitation movement is performed with the goal of restoring joint linkage, the user's cognitive and motor abilities can be very effectively improved (the disorder of the brain can be corrected). Joint linkage is a guide to treatment. Stimulate the brain (point estimation requirements) and determine the treatment plan based on the response of the joint linkage. By actively contacting the body part where joint linkage occurs, effective treatment can be performed.

[0107] Figure 4ATo show a view of the task table 400 stored in the task set database 216. In the task table 400, the following are stored in association with the task ID: time (task generation timing) 461, task interval from the end of the previous task 462, task type 463, task angle 464, task distance (intensity) 465. In addition, in the task table 400, the following are also stored in association with the task ID: target object speed 466, perfect determination (excellent evaluation) reference size 467, good determination (good evaluation) reference size 468, sensor object size 469, task achievement result, and the like. In addition to these, a delay time (reservation time) from the task generation notification to the task generation can also be set for each task. Figure 4B is a view showing another example of the task table 400 stored in the task set database 216. As shown in Figure 4B , data 471 related to joint linkage can be accumulated.

[0108] Figures 5 to 7 To show an example of the display on the head-mounted display 233 according to the present exemplary embodiment. Adding such background information and adding background sound as a kind of execution method both increase the load of brain information processing and cause noticeable joint linkage. In Figure 5 , on a background image 501 showing a street of the Edo period, an image showing an ink cage 511 as a target object is displayed. In addition, a thousand two box 513 as an object that the user needs to protect is displayed below the ink cage 511, and a ninja 515 gradually approaches from a distance. The speed of the ninja 515 is a speed set in the input area 323 of the operation panel 300 (the meaning of speed here is the same as the time limit). A circle 512 as a visual recognition auxiliary image is displayed on the ink cage 511. If the user touches the ink cage 511 with a sensor object (the top center of the avatar object 241a or 241b) before the ninja 515 reaches the thousand two box 513, the task is achieved. Two types of circles are prepared as the circle 512: a red circle and a blue circle. The task of the ink cage 511 surrounded by the red circle 512 is to operate the right red avatar object 241b corresponding to the right hand held control section 235 so as to come into contact with the ink cage 511. On the other hand, the task of the ink cage 511 surrounded by the blue circle 512 is to operate the left red avatar object 241a corresponding to the left hand held control section 234 so as to come into contact with the ink cage 511.

[0109] The print cage 511 displays the position (depth and angle) set in the input area 324 of the operation panel 300. The position of the print cage 511 does not change until the user makes the avatar object 241a or 241b touch it in the virtual space. That is, the print cage 511 is a target object fixed in space (also called a horizontal task because the user is required to stretch the body horizontally). Such a fixed target object is very effective for the rehabilitation of diseases such as cerebellar ataxia, diplopia, or inner ear dysfunction. That is, a restricted body movement image can be imprinted in the brain of a patient who has forgotten how to move his or her body by feedforward. If the distance of the print cage 511 in the depth direction is increased, the movement intensity can be changed. Furthermore, if combined with multi-channel biofeedback, the motor ability, body function, cognitive function, and sensory function are significantly improved. In addition, according to this horizontal task, chronic pain can be improved by promoting the reconstruction of the cerebral cortex. Alternatively, cognitive impairment called chemo brain, sensory dysfunction of a cancer patient who has decreased position sense due to neuropathy caused by anticancer drugs, or sequelae symptoms caused by COVID-19 infection can be recovered. The position where the target object appears can be notified in advance to provide a cue and reduce cognitive load. Not only verbal notification, but also tactile notification by touching the body, repeated verbal notification, and a combination thereof can also effectively reduce cognitive load. For the method of verbal notification, cognitive load can be reduced by giving a short instruction close to an imperative. Alternatively, a more complex instruction can be given in the form of a question, such as "blue? (pick up with the right hand)". Verbal notification can also be given in the form of a cognitive task such as calculation, for example, "if you hear a number that can be divided by 2, pick up with the right hand". Note that not only the horizontal position and depth at which the print cage 511 is generated, but also the height can be set.

[0110] Figure 6 A diagram showing an example of a screen for performing a target object vertical movement task (vertical task or falling task). In Figure 6 In the background image 601 representing a field, a person representing a farmer is displayed as a trigger object 602, which is a trigger for the appearance of a target object 603. That is, the display control section 213 displays the trigger object 602 as a notification image for notifying the user of the generation of the target object 603. When a predetermined time elapses after the trigger object 602 throws the potato-shaped target object 603 upward, as shown in Figure 7 the target object 703 having a large potato shape appears on the upper side of the screen. When the falling target object 703 is caught by moving the avatar object 702 having a basket shape, the task is completed. The left and right avatar objects 702 move on the screen in synchronization with the movement of the control sections 234 and 235.

[0111] The setting section 217 sets a delay time from the time when the trigger object 602 throws the target object 603 and the target object 603 is notified of generation until the target object 703 is generated, and thus it is possible to adjust the cognitive load imposed on the user. The longer the delay time, the longer the period for which the memory needs to be maintained, and the more the load of information processing in the brain increases. Note that the generation of the target object can be notified of at the same time as the movement of the trigger object 602 using a radar pattern notification image 250, or can be notified of in conjunction with a voice notification.

[0112] In this way, the setting section 217 can impose a cognitive load on the user not only with a task including only a horizontal line background as shown in FIG. 2, but also with a task including a large amount of information as shown in FIGS. Figure 5 and Figure 6 That is, it makes it difficult to remember that the target object 603 has appeared and the position at which the target object 703 is expected to fall, and thus the load imposed on the rehabilitation user is closer to the cognitive load required in real life.

[0113] In particular, the setting section 217 changes the pattern of the task and changes at least a part of the background image 301 over time, and thus imposes a cognitive load on the user 220 in processing the background image in the brain. In the example shown in FIG. Figure 6 , for example, the cloud 604 can be moved in the background image 601, the plant 605 can be shaken, or an animal (not shown) that is not related to the target object can be made to appear. This hinders the user 220 from focusing on the target object 603, and makes it more difficult to remember the position at which the target object 603 is expected to fall. More precisely, it can be said that information unrelated to the task is displayed on the background image to create an environment in which it is difficult to focus on the target object and intentionally causes an attention disorder (more specifically, a selective attention disorder, a divided attention disorder, a shifting attention disorder, or a sustained attention disorder), and thus makes the memory difficult and controls the cognitive load.

[0114] Figure 8 A diagram showing another example (vertical task or falling task) of the display shown on the display screen 240 according to the present example embodiment is shown. In Figure 8 , in a background image 801 like a forest, a trigger object 802 representing a monkey and a target object 803 representing an apple are displayed. When the trigger object 802 representing a monkey throws the target object 803 representing an apple from a tree, and the target object 803 approaching the user is caught by moving an avatar object 804 representing a basket, the task is achieved. Also in this case, the setting section 217 starts the falling of the target object 803 after a predetermined time elapses from the time when the trigger object 802 shakes the tree and the target object 803 is notified of generation, and thus imposes a cognitive load on the user 220 while causing an attention disorder.

[0115] Further, the setting section 217 causes at least two to five target objects 803 to exist in the three-dimensional virtual space at the same time and displays them on the display screen 240, whereby a cognitive load extremely strong can be imposed on the user 220. In other words, the setting section 217 generates at least two target objects 803 at different positions in the left-right direction in the three-dimensional virtual space.

[0116] In particular, if the at least two target objects 803 are generated at a plurality of positions in a direction different from the moving direction (falling direction in Figure 5 ) of the target object 803, Figure 8 the cognitive load can be imposed more strongly. That is, since the user 220 needs to consider the movement in the vertical direction, the difference between the positions in the left-right direction, and the difference in the falling positions in the depth direction to move the control sections 234 and 235, the spatial cognitive ability is also tested. As described above, in addition to changing the task predetermined time, the type, number, size, spatial distribution, position, number, and the like of the information contained in the notification image or notification sound including the trigger object are adjusted, whereby the complexity of the information to be memorized and maintained, that is, the cognitive load on the user's brain to process the information, is quantitatively adjusted and controlled.

[0117] The evaluation update section 215 evaluates the cognitive ability of the user using information such as whether the avatar object has arrived at the three-dimensional target position indicated by the target object at a good timing, the time interval from the generation of the notification of the target object to the generation, the number of target objects, and the degree of load that causes attention disorders in the background image.

[0118] The required accuracy of the point estimation changes depending on which of the various modes (display screens) shown in Figure 2A , Figures 5 to 8 the user should perform. For example, the required accuracy of the point estimation is improved in the order of Figure 2A compared to the case without a background shown in Figure 5 → Figure 8 → Figure 6 . Further, as described above, the required accuracy of the point estimation also changes with the background sound, which affects the joint linkage.

[0119] Figure 9 A flowchart of a process for displaying the rehabilitation support apparatus 210 is shown. In step S901, as a calibration process, the target of the rehabilitation movement is initialized according to the user. More specifically, as a calibration, each user is first caused to perform a work to acquire a movement reachable range. This range is set as an initial value, and the target is initialized according to the user. The initial value can be set according to the movement reachable range of the user, or the therapist who provides the treatment can decide the initial value according to the body cognitive function of the user or the target treatment effect.

[0120] Next, in step S903, the right-side task is started (i.e., the target object is displayed by the display control section 212 and the completion is evaluated by the avatar object).

[0121] In step S905, the left-side task is started (i.e., the target object is displayed by the display control section 212 and the completion is evaluated by the avatar object).

[0122] In step S909, the task achievement level is acquired and the score is added. Note that after the task on one side occurs, the task on the other side can be generated without waiting for the evaluation of the completion of the task, and these processes can be performed in parallel or sometimes with a difference. The first task can be started from the left side or the right side, and the last task can also be executed on the left side or the right side. That is, the flow described here is only an example, and the present application is not limited thereto.

[0123] <DATA>

[0124] Figure 10 A graph showing the results obtained using the rehabilitation support system 200 according to the present example embodiment is shown.

[0125] The number of cases n = 13. The composition was as follows: average age 60 ± 21 years, 7 male patients (53.9%), 5 patients with cerebrovascular disease (38.4%), 4 patients with orthopedic disease (30.8%), and 4 patients with neurodegenerative disease (30.8%). All patients had received conventional rehabilitation therapy for more than six months, and it was said that these people were difficult to further improve. Even if the patients were concentrated on the rehabilitation of the affected side for the first 20 minutes, there were no statistically significant improvements in STEF, TUG, and TMT-A (p = 0.06, 0.06, and 0.06, respectively). Figure 10 The graph in the middle "conventional intervention for 20 minutes". Here, STEF (Simple Test for Evaluating Hand Function) is a test of upper limb function. TUG (Timed Up and Go test) is an index for comprehensive examination of walking function, trunk function, and balance function. TMT-A (Trail-Making Test-A) is a cognitive function examination. STEF is an index in which the larger the value, the better the result, and both TUG and TMT-A are indexes in which the smaller the value, the better the result. All data are expressed as mean ± standard deviation. Statistical analysis was performed using paired t-test (i.e., mean comparison test considering pairing), and p < 0.05 was used as the standard for statistically significant difference.

[0126] On the other hand, when the patient alternately performs rehabilitation with awareness of depth using the system 200 according to the present example embodiment, STEF, TUG, and TMT-A are improved regardless of the affected side or the healthy side.

[0127] That is, as shown in the present example embodiment, it was found that, when the patient performed the left-right alternating training with awareness of depth, not only the upper limb function, but also the trunk function and the balance function were improved, and further the walking function (body function) was improved, and the spatial cognitive ability and the attention function were also improved. Note that it is presumed that, for example, when the STEF, the TUG, and the TMT-A are improved, the improvement of the inner ear, the vestibular system, the tactile sensation, the temperature sensation and the pain sensation, the position sensation, and the depth sensation also has an influence. It was thereby demonstrated that it is effective to let the patient perform the rehabilitation with left-right alternation with awareness of depth in order to maximize the stimulation of the brain plasticity, and maximize the efficiency of the motor learning, the cognitive learning, and the sensory learning of the brain. Note that it was found from other experimental results, etc. that, when the treatment according to the contents of the present application is implemented with left-right alternation, the improvement can be obtained faster compared to other treatment cases. It can also be considered that the symmetrical distribution of the motor areas shown in Non-Patent Literature 1 or Figure 2B is evidence supporting this fact.

[0128] [Other Example Embodiments]

[0129] While the present application has been particularly shown and described with reference to example embodiments thereof, the present application is not limited to these example embodiments. It will be understood by those of ordinary skill in the art that various changes in form and details can be made therein without departing from the spirit and scope of the present application as defined by the following claims. A system or device including any combination of the features included in the respective example embodiments is also included in the scope of the present application.

[0130] The present application is applicable to a system including a plurality of devices or a single apparatus. That is, the present application is also applicable even if an information processing program for implementing the functions of the example embodiments is provided to the system or the apparatus directly or from a remote site. Therefore, the present application also includes a program installed in a computer so as to realize the functions of the present application by the computer, a medium storing the program, and a WWW (World Wide Web) server which lets a user download the program. In particular, the present application at least includes a non-transitory computer-readable medium storing a program which causes a computer to execute the processing steps included in the above-described example embodiments.

[0131] This application is based on Japanese Patent Application No. 2023-97169 filed on June 13, 2023, and claims priority thereto, the entire disclosure of which is incorporated herein by reference.

Claims

1. An information processing system comprising: an occlusion section that makes all or a part of a user's body invisible; a request section that requests the user to perform a body action that overlaps a part of the user's body or an object operated by the user with a target object; and a control section that controls a degree of difficulty of the body action in accordance with a phenomenon called joint linkage in which a joint that is not needed for a movement requested by the request section is moved unconsciously in an environment in which an abnormal movement of a joint is exhibited by the occlusion section and the request section.

2. The information processing system according to claim 1, wherein The information processing system further comprises a feedback section that feeds back a completion of the body action requested by the request section to the user.

3. The information processing system according to claim 2, wherein The feedback section performs stimulation feedback to not less than two of five senses of the user each time the body action requested by the request section is completed.

4. The information processing system according to claim 1, wherein The information processing system further comprises a joint linkage evaluation section that evaluates the joint linkage.

5. The information processing system according to claim 1, wherein The request section or the control section controls so that the body action of the user is alternately performed on a left side and a right side.

6. The information processing system according to claim 1, wherein The control section controls the degree of difficulty of the body action by changing at least one selected from a group consisting of a size of the target object, a distance from the user, an angle, a speed, a presence or absence of a background, a presence or absence of music, and a size of a visual recognition auxiliary image.

7. The information processing system according to claim 1, wherein the occlusion section is a head-mounted display that displays a virtual space, and the request section generates the target object and an avatar object that represents a part of the user's body or an object operated by the user in the virtual space, and displays the target object and the avatar object on the head-mounted display.

8. The information processing system according to claim 1, wherein the occlusion section is a projection mapping system that displays a part of a background on all or a part of the user's body, or a contact lens display that displays a virtual space, and the request section displays the target object and the avatar object that represents a part of the user's body or an object operated by the user on the background using the projection mapping system, or generates the target object and the avatar object that represents a part of the user's body or an object operated by the user in the virtual space, and displays the target object and the avatar object on the contact lens display.

9. An information processing method performed by an information processing system including an occlusion section, a request section, and a control section, the method comprising: an occlusion step of making all or a part of a user's body invisible by the occlusion section; a request step of requesting the user to perform a body action that overlaps a part of the user's body or an object operated by the user with a target object by the request section; and a control step of controlling a degree of difficulty of the body action in accordance with a phenomenon called joint linkage in which a joint that is not needed for a movement requested by the request section is moved unconsciously in an environment in which an abnormal movement of a joint is exhibited by the occlusion section and the request section. The control step, that is, controlling the difficulty of the body motion according to a phenomenon called joint linkage in which a joint that is not needed for the movement requested by the request step moves involuntarily in an environment in which abnormal motion of a joint is exhibited by the blocking step and the request step.

10. An information processing program that causes a computer to execute a method comprising the steps of: the blocking step, that is, making all or a part of the user's body invisible; the request step, that is, requesting the user to perform a body motion that overlaps a part of the user's body or an object operated by the user with a target object; and the control step, that is, controlling the difficulty of the body motion according to a phenomenon called joint linkage in which a joint that is not needed for the movement requested by the request step moves involuntarily in an environment in which abnormal motion of a joint is exhibited by the blocking step and the request step.

Images