Information processing system, information processing method, and information processing program

Abstract

To set parameters of a user's reaching target quickly while viewing the user's movement.SOLUTION: An information processing device for receiving operation contents when an operator operates a keypad while watching a user's state from the keypad, and requesting a reaching operation to the user includes: a control unit for causing a target object which is a target when the user performs the reaching operation to appear in a virtual space; and a setting unit for setting an appearance position of the target object in response to the input from the keypad. Using an information processing device like this, an operator can quickly set parameters of the user's reaching target while viewing the user's movement.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, and an information processing program.

Background Art

[0002] In the above technical field, Patent Document 1 discloses a system for hemiplegic patients due to stroke or the like.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, with the technology described in the above document, it was not possible to quickly set the target position of the user while observing the user's movement.

[0005] An object of the present invention is to provide a technology for solving the above problems.

Means for Solving the Problems

[0006] To achieve the above object, an apparatus according to the present invention includes: a control unit that causes a target object, which is a target to be reached by a user, to appear in a virtual space; a setting unit that sets an appearance position of the target object in response to an input from a numeric keypad; and is an information processing apparatus provided with the above.

[0007] To achieve the above object, a method according to the present invention includes: a control step in which a control unit causes a target object, which is a target to be reached by a user, to appear in a virtual space; A setting step in which a setting unit sets the appearance position of the target object according to an input from a numeric keypad, It is an information processing method including

[0008] To achieve the above object, the program according to the present invention A control step of causing a target object that is a target reached by a user to appear in a virtual space, A setting step of setting the appearance position of the target object according to an input from a numeric keypad, It is an information processing program that causes a computer to execute.

Effect of the Invention

[0009] According to the present invention, parameters of a user's reaching target can be quickly set while observing the user's movement.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2A

Figure 2B

Figure 2C

Figure 3A

Figure 3B

Figure 4

Figure 5

Figure 6A

Figure 6B

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

[0011] Hereinafter, with reference to the drawings, embodiments of the present invention will be exemplarily described in detail. However, the components described in the following embodiments are merely examples, and are not intended to limit the technical scope of the present invention thereto.

[0012] [First Embodiment] The information processing apparatus 100 according to the first embodiment of the present invention will be described with reference to FIG. 1. The information processing apparatus 100 is an information processing apparatus that requests a reaching operation from the user 110. Note that the reaching operation generally means an action of stretching out the hand, but here, for the sake of convenience, any action of moving a specific part of the body toward a specific coordinate in the three-dimensional space will be expressed as a reaching operation. That is, the reaching operation includes not only the so-called upper limbs including the hand and the forearm, but also actions of moving parts such as the shoulder, elbow, lower limbs (knee, foot), and head toward coordinates in the three-dimensional space.

[0013] As shown in FIG. 1, the information processing apparatus 100 includes a control unit 101 that causes target objects 151 and 152, which are targets for the user 110 to reach, to appear in the virtual space 150, and a setting unit 102 that sets the appearance positions of the target objects 151 and 152 in response to an input from the numeric keypad 170.

[0014] According to the above configuration, the target position of the user can be quickly set while observing the user's movement.

[0015] [Second Embodiment] Next, the information processing system 200 according to the second embodiment of the present invention will be described with reference to FIG. 2A. FIG. 2A is a diagram for explaining the configuration of the information processing system 200 according to the present embodiment. The information processing system 200 is a medical device including an information processing apparatus 210 as a computer equipped with rehabilitation support software, a base station 231 as a high-performance VR device, and a head-mounted display 233. The rehabilitation support system 200 according to the present embodiment can improve at least one functional abnormality of physical functions such as upper limb function, walking function, trunk function, and balance function, cognitive functions (including spatial cognition, attention function, and higher brain functions), and sensory functions (including inner ear, vestibular system, tactile sensation, temperature and pain sensation, position sense, and deep sensation).

[0016] The information processing system 200 shown here is an example of a medical system that provides an environment for manifesting abnormal joint movements or cooperative movement disorders, and the present invention is not limited to this embodiment. That is, the present invention should not be limited to the concept of "rehabilitation", and treatment systems that actively treat physical diseases such as paralysis and ataxia, as well as cognitive and mental diseases such as dementia and schizophrenia, are also included in the present invention. Furthermore, the present invention also includes an operation verification system, an ability improvement and enhancement system, and a sports athlete-oriented movement ability improvement system for improving the movement ability and cognitive ability of healthy people who do not require any treatment or rehabilitation more than the current situation. That is, this software may be more specifically expressed as software that provides a treatment method called somato-cognitive cooperative therapy, and cooperative movement disorders may be more specifically expressed as abnormalities or entanglements of the Somato-Cognitive Action Network (SCAN). Many users with cooperative movement disorders are completely unaware of them.

[0017] As shown in FIG. 2A, the rehabilitation support system 200 includes an information processing device 210, a base station 231, a head-mounted display 233 as a blocking unit, and two controllers 234 and 235.

[0018] The user 220 wears the head-mounted display 233 and sits on the chair 225, and moves the body according to the display of the head-mounted display 233. The user 220 holds or fixes the controllers 243 and 234 in both hands and performs a reaching motion in the seated position according to the display on the head-mounted display 233 or the voice of the operator 280.

[0019] The base station 231, the head-mounted display 233, and the controllers 234 and 235 as high-performance VR devices can accurately measure in millimeters in a three-dimensional space. That is, the postures and positions of the head-mounted display 233 or the controllers 234 and 235 are accurately measured.

[0020] In this embodiment, the description is based on the premise of verifying the operations performed while sitting on the chair 225. However, the present invention is not limited to this, and it may be performed while standing, on a bed, in a supine position or prone position, or while performing other specific operations. Also, here the user 220 is holding the controllers 234 and 235 with both hands, but the present invention is not limited to this either, and it may be held or worn on body parts other than the hands, such as the feet or the torso.

[0021] In the case of an athlete or the like, while walking or running on a treadmill, the reaching operation in this system may be performed. In this case, the speed of the treadmill may be changed according to the degree of achievement of the required operation.

[0022] Note that the head-mounted display 233 may be a non-transmissive type, a video see-through type, an optical see-through type, or a glasses type. In this embodiment, a virtual space of VR (Virtual Reality) is presented to the user, but like AR (Augmented Reality), the real space and the virtual space may be superimposed and displayed, or like MR (Mixed Reality), real information may be reflected in the virtual space, or hologram technology may be used as an alternative means.

[0023] What is important here is that by wearing the head-mounted display 233, all or part of the user's own body becomes directly invisible to the user 220. Normally, in real life (real space), humans visually directly recognize the movement of their body and unconsciously fine-tune the movement to reach the target (visual recognition correction). In this system, since the user 220 cannot directly see their own hand in the real space, the user's brain cannot accurately perform visual recognition correction on the movement of the hand as usual.

[0024] Therefore, the cooperative movement disorder (entanglement in SCAN) becomes apparent, and abnormal movements (involuntary movements) of joints different from the parts that perform body movements are likely to be detected. In this embodiment, the head-mounted display 233 is used, but for example, a contact lens type display or a glass type display that displays a virtual space may be adopted. Furthermore, a projection mapping system that displays a background on all or part of the body of the user 220 and makes it look as if all or part of the body is transparent may be adopted. As a device that can easily and completely block the vision of the user 220, a non-transmissive type head-mounted display is the most convenient to use.

[0025] The information processing apparatus 210 includes an operation detection unit 211, a control unit 212, a feedback unit 213, a setting unit 214, and a reaching evaluation unit 215.

[0026] The head-mounted display 233 and the controllers 234 and 235 receive an infrared signal from the base station 231 and send it to the operation detection unit 211. The operation detection unit 211 calculates the position and orientation of the head-mounted display 233 and the controllers 234 and 235 from the temporal pattern of the infrared signal.

[0027] The control unit 212 controls the display of the head-mounted display 233 based on the position and orientation of the head-mounted display 233. Specifically, the position and orientation of the viewpoint in the virtual space are changed according to the position and orientation of the head-mounted display 233, and the image to be displayed on the head-mounted display 233 is determined. The control unit 212 also causes a target object that the user aims to reach to appear in the virtual space.

[0028] The feedback unit 213 detects and evaluates the rehabilitation movement of the user 220 based on the position and / or movement of the controllers 234 and 235 held by the user 220.

[0029] In this embodiment, as an example of a sensor for detecting the position or movement of a user's hand or head, the controller 234, 235 of the type held by the user 220 and the base station 231 are shown, but the present invention is not limited thereto. A camera (including a depth sensor) for detecting the "position or movement of the hand itself" of the user by image recognition processing, a sensor for detecting the position of the user's hand by temperature, a wristwatch-type wearable terminal worn on the user's arm, and motion capture can also be applied to the present invention by being interlocked with the motion detection unit 211. That is, one embodiment is to use a three-dimensional tracking device such as Kinect (registered trademark) or a motion analysis device, or to wear a marker or the like on the body.

[0030] The control unit 212 causes the target objects 241 and 242, which are the targets that the user 220 reaches for, to appear in the virtual space 240, and prompts the user 220 to perform a three-dimensional body movement through the display of the head-mounted display 233. In particular, the control unit 212 generates a target object 241 for three-dimensionally rehabilitating a part of the user 220's left body and a target object 242 for rehabilitating a part of the user 220's right body in the virtual space 240. That is, the control unit 212 functions as a request unit that requests the user 220 to reach for the target objects 241 and 242.

[0031] Here, reaching a virtual target displayed on the head-mounted display 233 is required, but the present invention is not limited to this, and an object provided in the real world may be used as the target. An object suspended by two or more wires, an object floating in the air by ultrasonic vibration, etc. may be used as the target. Also, an object three-dimensionally displayed as a hologram may be used as the target. The shape of the target object is not limited to a spherical shape, and may be triangular, square, dish-shaped, or any character shape. A system that uses an optical see-through or smart glass type head-mounted display to partially block the user's vision and make the user's own movements invisible while allowing only the target to be visible is also included in the concept of the present invention.

[0032] The control unit 212 generates target objects 241 and 242 in the virtual space 240 and moves them, for example, from above the user 220 downward. Here, the number of target objects that the control unit 212 can simultaneously display in the virtual space 240 is not limited to two, and three or more target objects may be simultaneously generated and displayed in the virtual space.

[0033] In the head-mounted display 233, the display position and size of the target object 241 gradually change according to the distance between the target object in the virtual space and the viewpoint position (for example, in the case of falling, it gradually becomes larger and then smaller). Note that the moving direction of the target object is not limited to the downward direction, and may be, for example, the upward direction from the floor surface to the overhead direction. The target objects 241 and 242 may be moved so as to come from the back to the front toward the user 220. The target objects 241 and 242 may be moved so as to pass in front of the user 220 from right to left or from left to right. Furthermore, the movements in the vertical direction, depth direction, and horizontal direction may be appropriately combined. For example, three-dimensional movements such as coming while falling or sliding to the right while falling are also conceivable. It is important to cause an operation according to the user's cognitive ability and motor ability. When at least one of the cognitive function and motor function of the user 220 is extremely low, the target object may be fixed at a specific coordinate position without moving the target object to prompt the user's reaching. For users 220 who are almost immobile or immobile, such as those with hemiplegia or cerebral palsy who cannot move their bodies voluntarily, or those who are bedridden due to higher brain dysfunction, severe intellectual disability, severe dementia, etc., a controller may be held or fixed to a body part such as the user's hand, elbow, or shoulder, and the target object may be dropped at that position so that the user can touch the target object (this is called intracerebral reaching in the sense of not involving physical movement). In this case, although it may seem that there is no reaction from the user at first until the information transmission neural pathways between the brain and the body related to body movement and the information transmission neural pathways related to cognition are connected, by repeatedly stimulating the brain, these body-cognition information transmission neural pathways will be connected and the body will start to move or a cognitive reaction will be obtained.

[0034] The control unit 212 generates avatar objects 243 and 244 that move according to the actions of the user 220 detected by the base station 231 within the virtual space 240. Here, the avatar object 243 is an object that moves corresponding to the controller 234 operated by the left hand, representing the position of a part of the user's left body. The avatar object 244 is an object that moves corresponding to the controller 235 operated by the right hand, representing the position of a part of the user's right body. The control unit 212 requests the user to perform a body movement that superimposes the avatar objects 243 and 244 on the target objects 241 and 242.

[0035] The avatar objects 243 and 244 are not limited to objects that move corresponding to the movements of the controllers 234 and 235. They may be virtual objects that move along with changes in the positions of parts of the user's body using three-dimensional body tracking technology as described above. The objects operated by the user are also not limited to the controllers as shown in the figure, and may be, for example, a racket, bat, or golf club equipped with a position sensor.

[0036] The control unit 212 causes the avatar objects 243 and 244 and the target objects 241 and 242 to be displayed on the display screen 245 of the head-mounted display 233 according to the orientation and position of the head-mounted display 233 detected by the motion detection unit 211. The images of the avatar objects 243 and 244 and the target objects 241 and 242 are superimposed and displayed on the background image. Here, the avatar objects 243 and 244 have the same shape as the controllers 234 and 235, but are not limited thereto, and may be in the shape of a hand. Furthermore, the size, shape, and color may be changed between the left and right. The background image is cut out from a virtual space including the horizon 246 and the ground surface object 247. According to the movement of the line of sight (the position and orientation of the head-mounted display), the appearance of the background image, the avatar objects 243 and 244, and the target objects 241 and 242 changes.

[0037] The control unit 212 generates an avatar object 244 indicating the position of a part of the right body and a target object 242 indicating the target position of the avatar object 244 on the right side in the same color or the same shape. Further, the control unit 212 generates an avatar object 243 indicating the position of a part of the left body and a target object 241 indicating the target position of the avatar object 243 on the left side in the same color or the same shape. Also, it is desirable to set the color or shape of the left avatar object 243 and the left target object 241 to be different from the color or shape of the right avatar object 244 and the right target object 242.

[0038] The operations of the users 220 moving the controllers 234, 235 are rehabilitation operations, and the display of the target object that prompts one rehabilitation operation that the user 220 should perform is called a task or an assignment. Here, as an example, the avatar objects 243 and 244 are color-coded, for example, in blue and red, and the target objects 241 and 242 are also color-coded in blue and red. Touching the blue avatar object 243 to the blue target object 241 achieves the task. Similarly, touching the red avatar object 244 to the red target object 242 achieves the task. That is, touching avatar objects of different colors does not achieve the task. Although color-coding in blue and red is shown here as an example, for color-blind users, it may be displayed in other color-codings (such as yellow and green), or without using colors, by shapes, or by linguistic notations such as "left", "right", "L (Left)", "R (Right)", or by symbols such as asterisks, triangles, or circles, or by different patterns (stripes) like 243 and 244 in the figure so that they can be distinguished.

[0039] As information representing one task (task data), it includes the appearance direction of the target object (90 degrees to the right, 45 degrees to the right, front, 45 degrees to the left, 90 degrees to the left with respect to the front direction of the chair) (※ or it may be expressed as 0 degrees for directly to the left like a protractor, 45 degrees for diagonally in front to the left, 90 degrees for the front, 135 degrees for diagonally in front to the right, 180 degrees for directly to the right), the distance to the target object (such as short distance, middle distance, long distance), the size of the target object, the appearance interval (time) of the target object, the moving speed of the target object, the number of target objects appearing simultaneously, the size of the sensor object, etc. The distance in the depth direction from the user 220 to the falling positions of the target objects 241 and 242 can also be set in, for example, three levels. For example, it can be changed to drop right next to the user 220, drop at a position where the user 220 cannot reach without a large forward lunge, or drop at an intermediate position. Thereby, the exercise load imposed on the user, and the load on the spatial awareness ability or the spatial grasping ability can be controlled.

[0040] [Feedback] When the avatar objects 243 and 244 collide with the target objects 241 and 242, the control unit 212 eliminates the target objects 241 and 242, and the feedback unit 213 performs visual feedback using visual effects such as displaying a message for the purpose of notifying or informing that the target action has been achieved, or bursting the target object, assuming that the target action has been achieved. If the avatar objects 243 and 244 reach the centers of the target objects 241 and 242, "Well done" is displayed, and if they reach the peripheral area, "Great" is displayed for each task. Thereby, the achievement of each required physical action is fed back to the user while distinguishing the achievement accuracy. The feedback unit 213 may perform feedback by stimulating one or more of the five senses of the user. If feedback is performed by stimulating two or more of the five senses of the user, the cognitive ability or the motor ability of the user, or both, can be effectively improved.

[0041] That is, the feedback unit 213 notifies the user 220 of the achievement of each rehabilitation operation for the target objects 241 and 242. Various methods can be considered for the notification here. As described above, characters such as "Amazing (Perfect)" and "Wonderful (Good)" can be temporarily displayed within the display screen 245 to inform the user of the achievement status of the target operation. Similarly, auditory stimuli can be used to inform the user of the achievement status of the target operation through the use of the same voice or sound effects. Furthermore, among the controllers 234 and 235, only the controller that has moved to contact the target object can be vibrated to notify the user of the achievement status of the target operation through tactile stimuli. Alternatively, the achievement notification can be graded, for example, as complete achievement, incomplete achievement, or unachievable, to notify the degree of achievement. Taste or olfactory stimuli can also be used.

[0042] More specifically, when the shortest distance between the sensor objects (spherical objects that define the positions and areas of the avatar objects) included in the avatar objects 243 and 244 and the target objects 241 and 242 is within a predetermined range, the target is achieved. Then, the target objects 241 and 242 disappear. The sensor object is, for example, a spherical object that includes the center point of the tip of the avatar objects 243 and 244. The sizes of the target objects 241 and 242 and the sizes of their surrounding parts can be set by the setting unit 214. Also, the size of the sensor object can be set by the setting unit 214.

[0043] The feedback unit 213 preferably changes the message type via the control unit 212 according to the evaluation of the rehabilitation operation. For example, if the sensor object touches the center of the target objects 241 and 242, "Amazing" is displayed, and if the sensor object touches only the surrounding part of the center of the target objects 241 and 242, "Wonderful" is displayed.

[0044] For example, on the screen of FIG. 3B, the radius of the target object is adjusted to 10 cm, the radius of the visual assistance object is adjusted to 20 cm, and the radius of the sensor object is adjusted to 2 cm. If a part of the sensor object (a sphere with a radius of 2 cm) overlaps with the target object even slightly, it is a Perfect determination. On the other hand, if the target object is not touched but the visual assistance object is touched, it is a Good determination. However, at this time, even if it enters the Good determination range, as long as the controller is directed towards the direction of the Perfect radius, the hit determination of the Good determination is not finalized. It becomes a Good determination under the condition that it enters the Good determination range and the sensor object moves away from the center of the Perfect radius. In this embodiment, the size of the visual assistance object is made to match the size of the Good determination area, but the present invention is not limited to this, and the visual assistance object may be larger or smaller than the Good determination area.

[0045] If the shortest distance between the outer edges of the target objects 241 and 242 and the outer edge of the sensor object becomes equal to or less than the first threshold value, it is regarded as complete achievement of the target, and for example, an image of "amazing" is displayed and the corresponding sound is output for feedback. If it is larger than the first threshold value but equal to or less than the second threshold value, it is regarded as achievement of the target, and for example, an image of "wonderful" is displayed and the corresponding sound is output for feedback. However, as described above, even if it is equal to or less than the second threshold value, if the sensor object is moving in a direction such that it will become equal to or less than the first threshold value, the determination that it is equal to or less than the second threshold value is not made.

[0046] Such feedback that stimulates a plurality of different senses (here, vision and hearing) is called multi-channel biofeedback, or multi-signal biofeedback, or multi-sensory biofeedback. At the same time, the controllers 234 and 235 may be vibrated to stimulate the sense of touch, or stimulation may be given to the sense of smell or taste.

[0047] That is, the feedback unit 213 can stimulate any one of the five senses of vision, hearing, touch, taste, and smell as feedback. These sensory stimulations may be combined in any two, three or more, or all of them may be stimulated.

[0048] Note that the output voice does not have to be the same as the message, and for example, a non-verbal sound effect such as "peeron" may be used. Note that as the distance required for the contact between the sensor object and the target object is reduced in this way, a more accurate operation is required. Therefore, the brain needs to require the body to perform more accurate and refined operations as movement commands. As described above, this imaging of the brain is called feedforward. That is, the smaller the distance between the sensor object and the target object required for the completion of the operation, the more powerful the feedforward is required, and this can continuously adjust the load levels of the user's movement, cognition, and sensation.

[0049] Here, the feedback unit 213 evaluated and notified in two stages of excellent and wonderful, but the rehabilitation operation may be evaluated in three or more stages according to how much the distance between the sensor object and the target objects 241 and 242 has been reduced.

[0050] The feedback unit 213 provides feedback that stimulates two or more of the five senses (vision, hearing, touch, taste, smell) to the user who virtually touches the target objects 241 and 242. This feedback is provided almost simultaneously with the timing when the sensor object enters within a predetermined distance from the centers of the target objects 241 and 242, or the timing when the sensor object comes into contact with the target objects 241 and 242 (referred to as real-time multi-channel biofeedback, or immediate multi-signal biofeedback, or immediate multi-sensory biofeedback). If the delay from those timings to the feedback is, for example, within 1 second, the effect is high, and the closer the interval between the user's operation timing and the feedback timing (the smaller the delay), the greater the effect. The feedback unit 213 provides feedback that stimulates the user's vision with an image saying "Well done!" and at the same time, or with a delay of a few tenths of a second, provides feedback that stimulates the user's hearing with voice or sound effects output from a speaker. The notification of task achievement for the five-sense stimulation may be combined in any way according to the operation type.

[0051] Furthermore, the feedback unit 213 may simultaneously output visual feedback by an image saying "Well done!", auditory feedback by voice, and tactile feedback by vibration. Also, the feedback unit 213 may simultaneously output only two types of feedback, visual feedback by an image saying "Well done!" and tactile feedback by vibration. Alternatively, the feedback unit 213 may simultaneously output only two types of feedback, auditory feedback by a voice saying "Well done!" and tactile feedback by vibration.

[0052] [Joint connection] Here, although the avatar objects 243 and 244 are visible to the user 220, since the actual body movements are not directly visible, part or all of the body is blocked from view, increasing the burden on the user's brain. In a state where part or all of one's own body (including the controller in the real space) is not visible, the brain is made to estimate (overlay / touch) a certain position (point) of the sensor object at the tip of the controller in the virtual space, more specifically in FIG. 2A, with a specific coordinate in the three-dimensional space, more specifically the position of the target object in FIG. 2A. This is called point estimation.

[0053] When point estimation is performed, the brain needs to vividly imagine (construct a body image) the state of the body in the brain without relying on vision, which is quite powerful. In medicine, this is expressed as "requiring a strong feedforward". A brain that is required to have a strong feedforward makes it more difficult to coordinate the body, resulting in the manifestation of coordination disorders and making it easier for abnormal movements of joints different from the parts where body movements are performed to occur. Also, when a strong feedforward that requires point estimation is performed, it becomes possible to induce the contraction of the deep muscles of the body by stimulation through the pyramidal tract or the like.

[0054] When a large load is placed on the brain, for example, when the avatar image 244 of the right controller 235 reaches the right target object 242, there is a user 220 who unconsciously applies force to the knee joint of the left foot and the lower leg of the left foot moves forward or backward (221). In this specification, the phenomenon in which joints that are not necessary for such required movements move involuntarily and unintentionally is called articular linkage, or an abnormality of the Somato-Cognitive Action Network (SCAN) or an entanglement of the SCAN. Articular linkage can be used as an index for improving the movement of the user 220. Note that this articular linkage is manifested even without an anatomical connection called an anatomical train. For example, when the joints of the toes of the foot are moved voluntarily, involuntary movements occur in the temporomandibular joint, hip joint, shoulder joint, etc. This is a phenomenon that is completely different from the phenomenon in which the left hand moves to balance when the body is tilted to the right, and is a phenomenon that cannot be seen in healthy people.

[0055] Articular linkage is a phenomenon in which joints that should not originally move move, and can also be expressed as involuntary movements that appear in joints other than the intended movement site. Intervention is performed by paying attention to this articular linkage. So-called bug fixing of the brain's information processing network is performed. According to the applicant, by minimizing the visual information of the user and requiring left-right alternating reaching movements in a sitting position or the like while performing point estimation, and by notifying the degree of movement achievement by multisensory biofeedback when the movement is achieved, it has been empirically found that articular linkage can be improved immediately. As a result, the cooperative movement disorder can be treated and improved. In other words, the information processing process between the brain and the body can be organized (untangled). Articular linkage can also be referred to as an entangled somato-cognitive action network as described above. Based on the same principle, it is an effective treatment method for patients called developmental disorders and hyperactivity. It has also been found that even patients who have had their brains greatly removed by brain tumor surgery can be compensated for by the system according to this embodiment.

[0056] As shown in Figure 2C, which was also mentioned in the non-patent literature (Nature "A somato-cognitive action network alternates with effector regions in motor cortex" by Evan M. Gordon et al. Published online: 19 04 2023), the motor areas responsible for coordinated movement exist across all joints. Looking at this Figure 2C, it can be seen that in modern neurology, the coronal cross-sectional views of the primary motor areas of the brain are grouped by bilaterally symmetric parts (such as the right and left hands, right and left feet, right and left eyes, etc.) (※ actually, they are distributed concentrically in three dimensions), and these motor areas are arranged symmetrically in the brain. Furthermore, it has newly been found that there are nerve regions responsible for the coordination of each body part shown by the puppet between the foot motor area and the hand motor area, and between the hand motor area and the face motor area. Due to the existence of nerves that coordinate each body part like a puppet, it is considered that joint linkages during movement become apparent in a specific environment where point estimation is required in a situation where all or part of the body is not visible.

[0057] The inventor actually arrived at the creation of the present invention by observing the rehabilitation movements of actual users even before the publication of the above non-patent literature, and it can be said that its usefulness and logic have been theoretically supported by this paper.

[0058] In Figure 2A, the movement of the knee joint is illustrated as an example of the occurrence of joint linkage 221, but the present invention is not limited to this, and all involuntary movements of joints different from the body part performing the required movement are included in the concept of the present invention. For example, as shown in Figure 2B, by detecting joint planes 371 and 372 and calculating their inclinations, it may also be possible to detect the simultaneous occurrence of torsion of the shoulder joint and the trunk (cervical vertebra joints, thoracic vertebra joints) and torsion of the waist (lumbar vertebra joints, hip joints).

[0059] Specifically, joint-related issues occur, such as torsion of the cervical facet joints (manifested in the direction of the head), torsion of the thoracic facet joints (manifested in the angle of the shoulders), elevation or abduction of the shoulder joints (the sides open), extension and contraction of the knee joints, hip joints, and ankle joints (the knees and toes move up and down, or invert and abduct), extension and contraction of the elbow joints (the arm not required to move is fixed in a flexed position), extension of the temporomandibular joints (the mouth opens), flexion progression of the metacarpophalangeal joints and interphalangeal joints, flexion (claw toe) and progression of the interphalangeal joints of the toes, etc. Such joint movements are not limited to flexion and extension, but also include adduction, abduction, internal rotation, external rotation, circumduction, or combinations thereof, as well as movements not described herein.

[0060] A joint-related evaluation unit 219 may be provided to install the camera 236 and detect and evaluate the position, torsion, inclination, movement speed, acceleration, etc. of each joint of the user 220 during the rehabilitation movement. The joint-related evaluation unit 219 detects the movement of each joint of the user 220 and evaluates whether joint-related issues are occurring based on the magnitude, speed, timing, etc. of the movement.

[0061] Specifically, movements that occur even in healthy individuals, such as the swinging of the hands observed during walking, are not judged as joint-related, while movements that are not expected to occur in healthy individuals, such as the movement of both lower limbs, the trunk, and the inclination of the head and neck when reaching out the hands while sitting, are judged as joint-related. Alternatively, the system may define joint-related issues according to the direction and magnitude of the movement of specific joints.

[0062] The camera 236 takes pictures of the user, and the joint correlation evaluation unit 219 can detect abnormal twists and movements of the joint surface by comparing with the image in the normal state (before exercise) or with a normal motion model. The joint correlation evaluation unit 219 may be provided with a database of movements to be judged as joint correlations and / or a database of movements not to be judged as joint correlations. Sensors may be attached to the user's joints (e.g., both shoulders, both elbows, wrists, both knees, both ankle joints, etc.) (or by motion capture), and the base station 231 may detect the position thereof to detect joint correlations. Of course, the operator may visually recognize the occurrence of joint correlations and input them into the system.

[0063] The joint correlation evaluation unit 219 may detect the inclination of the head-mounted display 233 at the base station 231 to detect the joint correlation of the neck. For example, if the head-mounted display 233 is inclined by 30 degrees, it can be determined that the trunk is 15 degrees (determined by the camera image) and the neck is 15 degrees. The inclination of the wrist joint, etc., can be determined from the position and inclination of the controllers 234 and 235.

[0064] Six joints of the hip joint, knee joint, ankle joint, neck joint, trunk, and elbow joint may be intensively detected. Further, it may be divided into more classifications, or all joint units (a person has 68) may be viewed, or conversely, devices may be made to focus on one or a small number of joints where abnormalities are most easily detected. Furthermore, a plurality of evaluation indexes may be reduced to one integrated index or parameter such as a total score. That is, weights may be assigned to and integrated with the joint correlations of the whole body or important parts, and a total joint correlation evaluation (calculation of joint correlation score) may be performed.

[0065] If the required operation is to overlap a 2-cm sensor on a 10-cm target, the accuracy of point estimation is low. However, for the required operation of overlapping a 1-cm sensor on a 1-cm target, a high-accuracy point estimation is required. In this case, if the user uses a blocking part such as a head-mounted display, the visual information of the real world is blocked, resulting in an inconvenient state. Therefore, the cranial nerves for point estimation need to be sharpened, and the associated movement disorder becomes prominent, that is, the joint association appears more significantly. This is one of the mechanisms of joint association occurrence.

[0066] If the rehabilitation level (the user's motor ability or cognitive ability) improves, by reducing the sizes of both the target and the sensor, the required accuracy of point estimation can be increased, making the compensatory joint association more prominent and induced. By repeatedly requiring rehabilitation operations in that state, or by guiding the user to perform correct operations more easily through vocalization or physical contact, the joint association will be cured. As a result, the overall body alignment is adjusted, and the SCAN becomes rewired, and the associated movement disorder is alleviated and improved.

[0067] The joint association evaluation unit 219 may display the detected position and size of the joint association on the operator screen 290 or the like. It may further have a comparison unit that compares with past joint associations or a normal operation model, and notify the operator of the comparison result. Thereby, the operator can recognize the change of the joint association more accurately.

[0068] The joint association evaluation unit 219 may generate and display a heat map in a bull's-eye image or the like with the movement as a joint association in terms of distance, acceleration, etc. It may also describe the change over time of the joint association and provide information useful for controlling the difficulty of the operation. Specifically, it may visualize how the joint association changes from the start to the end of the required reaching operation, or compare the sizes of the joint associations of each joint. This makes it easier to determine which joint to pay attention to.

[0069] The operator 280 or the information processing device 210 requests a rehabilitation operation so that the joint connection becomes smaller and causes the user 220 to achieve it.

[0070] For example, if the brain is compared to a CPU, and the information processing process that was previously allocated 80 for body movement processing due to the existence of joint connection out of its computing power of 100 is reorganized to 20, more complex operations than those required become possible, and the motor ability is improved. Alternatively, if the CPU itself, which is used for cognitive processing, is improved, the cognitive ability and attention function will also be improved.

[0071] When it is determined that the joint connection has become smaller than a certain reference, the load of the rehabilitation operation is increased. When it is determined that the joint connection has become larger than a certain reference value or that a state where it has become larger than a predetermined value has continued for a predetermined time or more (indicating that the joint connection has not improved), the load of the rehabilitation operation is decreased.

[0072] Note that whether or not joint connection occurs is affected by factors such as the size, angle, height, distance, direction of movement or presence / absence of movement, speed of movement, color, shape, visibility of the target object, the size, color, shape, visibility of the device or indicator for overlay by point estimation such as a controller, the presence / absence of background information, and the presence / absence of background sound. Causing joint connection and suppressing it is the goal of the rehabilitation operation in this system. That is, joint connection serves as a treatment guideline. During the rehabilitation operation, pressing the user's knee or shoulder may cause the joint connection to disappear. In this way, it is also possible to manually and forcibly correct the fraying of the neural circuits in the brain.

[0073] In a situation where reaching (point estimation) is performed while blocking vision, feedback is provided each time the required operation is achieved in this way, which triggers the brain to correct and improve the movement disorder. That is, the joint connection and the movement disorder are improved. This improvement progresses more efficiently as the number of sensory stimulations during feedback increases, or the improvement is retained as memory in the long term. For example, compared with one-sensory feedback, symptoms that took several days to improve by performing two-sensory feedback will improve in units of time. Or compared with two-sensory feedback, symptoms that took several hours to improve by performing three-sensory feedback will improve in units of minutes. Furthermore, even for symptoms that returned to their original state immediately after improvement during one-sensory feedback, the improvement effect will be maintained for several days by performing two-sensory feedback. Or even for symptoms that returned to their original state within several days after improvement during two-sensory feedback, the improvement effect will be maintained for several weeks by performing three-sensory feedback. To express such characteristics of this treatment, in the medical field, this treatment is also expressed as "brain reprogramming therapy (BRT)", "brain re-wiring therapy (BRT)", or "motor coordination therapy (MCT)", "somat-cognitive coordination therapy (SCCT)", etc.

[0074] The flow of accuracy control for point estimation regarding joint connection is as follows. (1) Perform rehabilitation movements with default settings. (2) The joint connection is too large (the angles or distances of the knee joints, shoulder twists, waist twists, head twists, etc. exceed the threshold). (3) Lower the required difficulty of point estimation (change the point estimation parameters). (4) The joint connection becomes moderately smaller. (5) Repeat the rehabilitation movements alternately on the left and right. (6) The joint connection disappears. (7) Raise the required accuracy of point estimation (change the point estimation parameters). (8) The joint connection becomes of moderate size. (9) Return to (5).

[0075] In addition, in (5), it may be possible to suppress joint entanglement by touching the user's body or speaking to the user. Also, in (7), it may be possible to apply a load to the user by speaking, achieving the same effect as increasing the required accuracy of point estimation. It has been found that by touching the user's body, the user's joint entanglement can be resolved more quickly.

[0076] As described above, point estimation is performed by blocking the user's direct line of sight, and the deep muscles contract due to a strong motion command via the target object to align the whole body. At this time, joint entanglement, or a disruption of the cranial nerve called entangled SCAN, is detected, and the rehabilitation operation is repeated until the joint entanglement subsides while controlling the required accuracy of point estimation (various parameters).

[0077] When multiple joint entanglements appear, it is necessary to determine which joint entanglement to correct first. For example, first correct the largely moving knee (hip joint), and then move on to the next joint (for example, the involuntary movement of the ankle joint causes the sole of the foot to turn inward).

[0078] If the rehabilitation operation is performed with the goal of recovering joint entanglement, it is possible to very effectively improve the user's cognitive motor ability (relax the brain's fraying). Joint entanglement serves as a treatment guideline. Stimulate the brain (point estimation requirement) and judge the treatment plan by observing the reaction of joint entanglement. Effective treatment becomes possible by actively contacting the body part where joint entanglement appears.

[0079] [Radar Screen] The control unit 212 causes the radar screen image 250 to be displayed on the display screen 240 of the head-mounted display 233. The radar screen image 250 is a notification image for notifying the occurrence of the target object 152. The radar screen image 250 notifies in which direction the positions of the next-occurring target objects 241 and 242 are relative to the reference direction in the virtual space (usually set to the front direction of the chair 225 by calibration). The radar screen image 250 further notifies how far the positions of the occurring target objects 241 and 242 are from the user 220. Note that the notification image is not limited to the radar screen image, and may be notified by characters, arrows, symbols, illustrations, types, intensities, blinking, etc. of light and color. Also, the notification method is not limited to images, and may be performed by sound, vibration, or any combination of sound, vibration, and image.

[0080] Regardless of the orientation of the user 220's head, the control unit 212 causes the radar screen image 250 to be displayed in the central portion (for example, within the range of -50 degrees to 50 degrees) of the display screen 240 of the head-mounted display 233. However, the display portion is not limited to the center, and may be any location such as the four corners, the upper end, the lower end, the left end, or the right end of the screen. The patient can estimate the difficulty of the next movement operation from the information on the position, angle, and number of the target objects displayed on the radar screen. When a more difficult operation is predicted by the patient, the joint correlation becomes more prominent.

[0081] The radar screen image 250 includes a head image 251 representing the user's head as viewed from above, a block image 252 obtained by dividing the periphery of the head image 251 into a plurality of blocks, and a fan-shaped image 253 as a visual field area image indicating the user's visual field area. The target position image indicating the position of the target object is indicated by which block of the block image 252 is colored or flashing or lit. Thereby, the user 220 can know whether the target object is on the left side or the right side with respect to the direction in which he is facing. In the present embodiment, the block image 252 is fixed and the fan-shaped image 253 moves. However, the present invention is not limited to this, and the block image 252 may be moved according to the direction of the head while the fan-shaped image 253 and the head image 251 are fixed. Specifically, when the head is turned to the left, the block image 252 may be configured to rotate to the right.

[0082] In this example, the coloring position and color of the block image 252 indicate that the next target object 242 for the right hand appears at the farthest position to the left with respect to the reference direction in the virtual space. And from the position of the fan-shaped image 253 and the orientation of the head image 251, it can be seen that the user is already facing the left direction.

[0083] [Various setting items, task data] The setting unit 214 can set the user's basic information (ID, name, gender, age, disease, past medical history, various cognitive function and motor function evaluation indexes, various examination results, etc.) and stores it in the user database so that it can be searched. The user ID is also associated with the records of past rehabilitation (facility, date and time, task data) in the reaching evaluation unit 215.

[0084] The setting unit 214 first sets one of a manual mode (a mode in which point estimation parameters are set as task data for each task), a template mode (a mode in which two or more consecutive task data set in advance are used as a task data set), or a delegation mode (auto mode) in which the device automatically generates tasks. The setting unit 214 can create a task data set to be used in the template mode based on an instruction from the operator.

[0085] The setting unit 214 can further select a task type from among a horizontal (stationary) task (without background image), a horizontal task (with background image), a dropping task (without background image), a dropping task (with simple background image), and a dropping task (with complex background image) by operating the operator. In the present embodiment, the above five types of tasks will be described, but the present invention is not limited thereto.

[0086] The horizontal (stationary) task is a task that requires reaching the target object that is stationary or fixed in the three-dimensional space, as will be described later with reference to FIG. 10, with a sensor within a predetermined time (for example, until the interfering object approaches).

[0087] For the horizontal task, patterns with and without a background image are prepared. Also, the BGM can be turned on or off for each.

[0088] The dropping task (without background image) is a task that requires reaching the target object that has fallen from above with a sensor in a state where the absolute position of the background (horizontal line) does not change within the virtual space, as shown in FIG. 2A.

[0089] The dropping task (simple background image) is a task that requires reaching a target object that has fallen from above with a sensor in a state where there is a background image and a distracting stimulus (here, an image such as a monkey jumping from tree to tree) is mainly placed only in the upper half of the screen, as will be described later with reference to FIG. 11. Note that the placement of the image as a distracting stimulus does not have to be in the upper half of the screen. For example, it may be in the left half, the right half, or the lower half. The position of the stimulus placement does not necessarily occupy half of the screen. It can be one-third, one-fourth, or even if the intensity of the distracting stimulus is weak, it may be placed in a large part of the screen, for example, three-fifths. Also, the distracting video stimulus does not have to be limited to animals such as monkeys. For example, it may be something like leaves falling from a tree, and is not limited to a video moving around within the screen.

[0090] The dropping task (complex background image) is a task that requires reaching a target object that has fallen from above with a sensor in a state where the background image is changing complexly or a distracting video stimulus is placed throughout the screen, as will be described later with reference to FIGS. 12 and 13. Also, in the dropping task (complex background image), prediction and memory of the appearance of the target object are required, such as a specification in which the period from the occurrence of an event to when actual movement is required is extended.

[0091] FIG. 3 is a diagram showing a screen (operation panel) 300 for an operator to operate. In the present embodiment, as an example, the operation panel 300 for the dropping task in the manual mode will be described. In the manual mode, the setting unit 214 causes such an operation panel 300 to be displayed on a display 290 or the like connected to the information processing apparatus 210. In the present embodiment, task creation for rehabilitation support is performed by setting point estimation parameters using an intuitive operation panel 300.

[0092] The display that shows the operation panel 300 may be a projector or an external display 290 connected to the information processing apparatus 210, or may be a display built into the information processing apparatus 210. The operation panel 300 includes a user visual field area 301, various parameter setting areas 302, a score display area 303, a task history display area 304, check boxes 305, a stop button 306, a re-center button 307, and a BGM control button 308. In the example of FIG. 3, the operation panel 300 includes a user image area 309 representing the state of the actual user 220, but the present invention is not limited to this.

[0093] The user visual field area 301 on the right side of the screen displays the image that is shown on the head-mounted display 233 and that the user 220 is viewing. Thereby, the visual field of the patient can be confirmed. Although omitted in FIG. 2A, around the target objects 241, 242, visual recognition assist objects 341, 342 for improving the visibility of the target objects are displayed. In the case of a user 220 with a low cognitive function, the visual recognition assist objects 341, 342 are set to be large to make it easier to find the target objects. In the rehabilitation support system according to the present embodiment, unlike a general game screen or the like, various contrivances are made everywhere to make it easier to achieve such tasks.

[0094] Conversely, although the user can see a circle with a radius of 20 cm (visual assistance object), actually touching a ball with a radius of 10 cm (target object) located at the center of the circle is required to completely complete the task. If the size of the visual assistance object is small, it becomes difficult for the user to find the target object. If the visual assistance object is enlarged, it becomes easier for the user to find the target object. If the target object is enlarged, the allowable tolerance for sensor reaching deviation increases. If the target object is reduced, the allowable tolerance for sensor reaching deviation decreases, and the rehabilitation operation can be evaluated more severely. The radius of the visual assistance object can also be made to match the radius of the target object. These settings quantitatively change the feed-forward accuracy, that is, the accuracy of the information processing performed by the user's brain. It becomes possible to control the treatment approach for physical functions (upper limb function, walking function, trunk function, balance function), cognitive functions (including spatial cognition and attention function), and sensory functions (including inner ear, vestibular system, tactile sense, thermoalgesia, position sense, and deep sensation), as well as the degree of manifestation of joint-related disorders.

[0095] The various parameter setting area 302 is an area for setting a plurality of parameters that define the task. The various parameter setting area 302 includes a speed setting area 321 for the left hand target object, a radius setting area 322 for the left hand target object, a radius setting area 323 for the visual assistance object of the left hand target object, and a sensor radius setting area 324 for the left hand avatar object. The various parameter setting area 302 includes a speed setting area 325 for the right hand target object, a radius setting area 326 for the right hand target object, a radius setting area 327 for the visual assistance object of the right hand target object, and a sensor radius setting area 328 for the right hand avatar object. The various parameter setting area 302 further includes a target object position input area 329.

[0096] If the sensor size set in the sensor radius setting areas 324 and 328 is large, even if the hand position deviates significantly from the target object, the task is considered to be achieved, so the difficulty level of the rehabilitation operation decreases. Conversely, if the sensor size is small, the hand must be accurately moved to the central area (evaluation size) of the target object, so the difficulty level of the rehabilitation operation increases. In the example of FIG. 3A, the sensor size is 2 cm on each of the left and right sides. This element also quantitatively changes the feedforward accuracy, that is, the information processing accuracy for the brain, and enables control of the treatment approach for physical functions (upper limb function, walking function, trunk function, balance function), cognitive functions (including spatial cognition and attention function), and sensory functions (including inner ear, vestibular system, tactile sense, thermoalgesia, position sense, deep sensation), as well as the degree of manifestation of joint-relatedness.

[0097] The target object position input area 329 is an image for setting the position of the target object to be generated (distance from the user and angle from the reference direction), and has a shape obtained by enlarging the radar screen image 250. The input area 329 includes a plurality of blocks with different distances from the user and angles from the reference direction of the target object. When an operation of selecting any one of the plurality of blocks is performed by the PC or the numeric keypad 290, the target object 241 or the target object 242 is generated at a position in the virtual space corresponding to the position of the specified block. The input area 329 is here divided into 18 blocks, but the present invention is not limited to this.

[0098] In the example of FIG. 3A, in the virtual space, for a target object 241 with a radius of 10 cm falling at a speed of 45 cm / s at a far left location, the task is to timely contact an avatar object 243 including a 2-cm sensor part. When the avatar object 243 touches the visual object 341, the target object 241 does not disappear, and the task is not completely achieved (a certain score is given and it is evaluated as good). The task is completely achieved (perfect evaluation) only when the sensors of the avatar objects 243 and 244 touch the target objects 241 and 242.

[0099] In the score display area 303, the total number of times of the task for each appearance position of the target object and the number of times indicating how many times the task has been achieved are shown. Here, the score is expressed in fractional form and percentage notation, but it is not limited to this. After the rehabilitation operation, the reaching evaluation unit 215 uses the value of this score display area 303 to derive a rehabilitation evaluation point. Depending on which task among the static task, horizontal task, falling task (without background image), falling task (simple background image), and falling task (complex background image) is performed, the above score may be weighted. For example, the weight coefficients such as 1 times for the static task, 1.2 times for the horizontal task, 1.5 times for the falling task (without background image), 2 times for the falling task (simple background image), and 3 times for the falling task (complex background image) may be determined in advance. Then, the score may be multiplied by the weight coefficient to calculate the point.

[0100] In the task history display area 304, the total rehabilitation time, the total number of tasks, the total number of left-hand tasks at the 0-degree, 45-degree, and 90-degree positions, and the total number of right-hand tasks at the 0-degree, 45-degree, and 90-degree positions are displayed. By checking this task history display area 304, the operator can grasp the degree of load imposed on the user.

[0101] In addition, the setting unit 214 can also receive inputs for various parameter setting areas 321 to 329 from the numeric keypad 270, and the operation panel 300 has a check box 305 for setting whether to accept the operation of the appearance position of the target object by the numeric keypad 270. Here, since the check box 305 is checked, the operation by the input device can be performed.

[0102] The stop button 306 is a button for instructing the temporary stop or termination of a task. These stop instructions are designed in such a way that they cannot be performed from the numeric keypad 270. In the manual mode, clicking on the target object position input area 329 serves as an instruction to start the task, and without such a click, the target object will not appear, i.e., it will be in a temporary stop state. On the other hand, in the template mode, selecting the stop button 306 stops the appearance of the target object. When the end button is selected, the reaching evaluation unit 215 calculates the rehabilitation evaluation points from the scores of the tasks performed up to that point.

[0103] The re-center button 307 is a button for redefining the position of the user 220's head-mounted display to the central position within the virtual space. The BGM control button 308 is a button for turning the background music on and off.

[0104] When the re-center button 307 is operated, the control unit 212 reconstructs the virtual space with the position of the head-mounted display 233 at that moment as the origin and the orientation of the head-mounted display 233 at that moment as the reference direction. By operating the BGM control button 307 to turn off the background music during task execution, the cognitive load of the task can be reduced and the user can concentrate on the body movements. Generally, in users with joint pain, the joint pain during the task tends to subside when the background music is turned off. Conversely, for users without joint pain, the background music can be turned on to deliberately cause joint pain.

[0105] In the template settings for the template mode, the generation interval of the target object can be set as a task setting parameter.

[0106] In addition to the radar screen image 250, the control unit 212 may also display the information bar 310 on the display screen 240 of the head-mounted display 233. The information bar 310 displays the player name 311, the task achievement level 312, the elapsed time since the start of the game 313, and so on. Furthermore, the information bar 310 is always parallel to the horizon 246 in the virtual space. When the patient's head position and body axis are distorted, the parallel relationship between the radar screen image 250 that follows the inclination of the head-mounted display 233 and the information bar 310 is disrupted. Therefore, the patient can visually assist in correcting the inner ear information, and the therapist can also easily notice the distortion of the patient's body axis. In FIG. 3, an image when the head position is tilted 20 degrees to the right is illustrated. In this way, the user interface of the head-mounted display 233 is designed to have both elements for the patient to concentrate on rehabilitation and medical benefits.

[0107] Also, FIG. 3B is a diagram showing the operation panel 350 as another example. In the operation panel 350, compared with the operation panel 300, it is significantly different in that the joint connection generation button 351 is provided. The operator checks the movement of the user 220 and clicks the joint connection generation button 351 if a joint connection appears. Also, a bull's-eye-shaped heat map 352 may be superimposed and displayed on the user image 309 to show the image area where a joint connection is likely to occur in red.

[0108] The setting unit 214 can set the generation of the left target object 241 and the right target object 242 to be alternated, and the setting unit 214 sends an instruction to the control unit 212 according to the setting. The setting unit 214 can set the generation position of the left target object 241 in the depth direction in the virtual space 240 and the generation position of the right target object 242 in the depth direction in the virtual space 240, respectively.

[0109] By controlling both the generation timing of the target object and the moving speed of the target object in the vertical direction, the rehabilitation movement of the user is controlled so that it is always alternating left and right. That is, in the simplest method, the left and right target objects are not generated simultaneously. After reaching the right target object is detected, the left target object is generated. However, it is not limited to this, and various control methods are possible. That is, while generating a plurality of target objects simultaneously, the moving speed may be changed so that it decreases in the order of right → left → right → left to encourage alternating left and right movement. For example, after reaching the right object is detected, the right object and the left object may be generated simultaneously, or the right object may be generated first, the speed of the right object may be increased, and the left object may be made to perform an overtaking operation. By doing so, a high load can be applied to the brain. The control unit 212 may control the rehabilitation movement of the user so that it alternates left and right.

[0110] On the other hand, when there is paralysis only on the right side or when it is desired to actively rehabilitate the left upper body, etc., when it is desired to reach the right target object below and the left target object relatively above, after generating the right target object at a slow speed, the left target object may be generated at a high speed with an interval. That is, at least one of the generation timing and speed of the target object is controlled according to the vertical position of the body for which rehabilitation is desired (according to the position of the target object to be reached). In the case of rehabilitation of the knees or lower body, the generation timing and speed of the target object are controlled so that reaching occurs at a low position. Also, when it is desired to increase the cognitive load, it is generated in the order of left → right → right, but only the first target object is moved slowly, and the second and third are moved at high speed. As a result, reaching can be performed in the order of right, left, right.

[0111] By performing traffic control in the brain (rewired in the sense of untangling the intertwined SCAN) in this way, it becomes possible to improve the symptoms of diseases that are difficult to treat in modern medicine, such as chronic pain including pain modulation pain and restless leg syndrome (abnormality of the basal ganglia).

[0112] The setting unit 214 may set the delay time from the timing when the generation of the target objects 241 and 242 is notified to the timing when the target objects 241 and 242 are generated, and thereby control the cognitive load imposed on the user 220. That is, the user has to continuously remember the position in the virtual space where the target object is generated (the position indicating in which direction the head-mounted display should be directed to display it), such as on the radar screen image 250, until the actual generation of the target object, and this "memory time" becomes the cognitive load for the user.

[0113] Also, the setting unit 214 may control the cognitive load by changing the time "until the target object 152 approaches the reachable range of the user 220" instead of "until the timing when the target object 152 is generated". The setting unit 214 may impose a cognitive load on the user 220 by displaying a background image 245 other than the target objects 241 and 242 on the head-mounted display 233. When changing the cognitive load, it is desirable to notify the user in advance whether the cognitive load is increased or decreased. The notification method may be visual using characters or symbols, by voice, or in a form of touching a part of the body, such as tapping the shoulder, elbow, arm, or leg.

[0114] The setting unit 214 controls the difficulty of the body movement according to the movement of joints different from the body part where the user performs the body movement, that is, the degree of joint connection. The setting unit 214 as a control unit controls the size of the target object, the distance from the user, the angle, the speed, the presence or absence of background, the presence or absence of music, and the size of the visual assistance object as point estimation parameters. For example, the joint connection may change significantly depending on the presence or absence of music. The size of the joint connection generated also varies depending on the type of music. For example, compared with the chirping of birds (which also has a great impact on people with poor cognition), the repetition of rhythm music increases the amount of work, that is, the cognitive load, for the brain to process information in the background, so the joint connection becomes larger. Similarly, the complexity of background information is also greatly related to the amount of work, that is, the cognitive load, for the brain to process information in the background. For example, the cognitive load is greater when there is some background or scenery than in a three-dimensional space where there is nothing, and furthermore, the more animals or objects moving in it, and the more widely the moving animals or objects appear on the screen, the greater the cognitive load and the more prominent the joint connection appears. That is, basically, the more we try to improve the accuracy of point estimation, or the higher the load of the brain's background processing such as sound and background information, the less reserve amount of the brain available for point estimation, so it is known that the joint connection becomes stronger.

[0115] [Reaching Evaluation] The reaching evaluation unit 215 evaluates the rehabilitation actions of the user 220 according to the amount and quality of the tasks achieved by the user 220, and adds points. Here, the quality of the tasks achieved refers to "wonderful" or "amazing", that is, it includes how close the avatar object can approach the target object. The reaching evaluation unit 215 assigns different points (higher points for distant objects and lower points for close objects) to the achieved tasks respectively. The reaching evaluation unit 215 can update the tasks according to the accumulated points. For example, the task (the attributes of the target object) may be updated using the task achievement rate (the number of targets achieved / the number of tasks). The reaching evaluation unit 215 compares the rehabilitation action detected by the action detection unit 211 with the target position represented by the target object displayed by the control unit 212 to evaluate the rehabilitation ability of the user 220. Specifically, it is determined by comparing the positions in the three-dimensional virtual space whether the avatar objects 243, 244 that move corresponding to the rehabilitation action detected by the action detection unit 211 overlap with the target objects 241, 242. If these overlap, it is evaluated that one rehabilitation action has been cleared, and points are added.

[0116] The reaching evaluation unit 215 may update the target task (task data) according to the accumulated points. For example, the target task may be updated using the task achievement rate (the number of targets achieved / the number of tasks).

[0117] FIG. 4 is a diagram showing the history database 400 stored in the reaching evaluation unit 215. The history database 400 includes the task ID, time (the timing of the target object generation), target type (left and right), angle (0 to 180), distance (short, middle, long), the speed of the target object, the radius of the target object, the radius of the visual assistance object, the radius of the sensor, and the result (wonderful: 2, amazing: 1, failure: 0).

[0118] That is, by looking at the history database 400, it is possible to know, after how many seconds from the start of rehabilitation, what type, radius, and visual recognition radius of the target object fell where at what speed, and whether reaching was achieved as a result of reaching with a sensor of what size. And the occurrence of joint-relatedness at that time may also be accumulated as history. Furthermore, of course, a patient ID is associated with this history data. Video data of the patient may be associated with each history data. Such history data can be used to analyze changes in the user's cognitive function and motor function.

[0119] [Numeric keypad] Returning to FIG. 2A, the control (parameter setting) of the task given to the user can be executed using the wireless numeric keypad 270. The operator 280 operates the numeric keypad 270 with one hand while checking the state of the user 220, the magnitude of the joint-relatedness 221, and the user's field of view. If assistance with both hands is required, it is also possible to operate the numeric keypad 270 with the feet on the floor. The setting unit 214 sets the appearance positions of the target objects 241, 242 according to the input from the numeric keypad 270.

[0120] The appearance of the numeric keypad 270 is shown in FIG. 5. In the present embodiment, a commercially available numeric keypad with a sticker for this system attached to its buttons 1 to 9 is used. A direction sticker representing the roles of the buttons 1, 4, 7, 3, 6, 9 of the numeric keypad and a distance sticker representing the roles of the buttons 2, 5, 8 of the numeric keypad are prepared and pasted on the buttons 1 to 9 of the commercially available numeric keypad, thereby completing the numeric keypad 270 as shown in FIG. 5. That is, the content indicated by those stickers is also part of the system according to the present embodiment. The sticker includes the original numbers of the numeric keypad and arrows and characters representing the roles of each button as display content. The sticker displays the buttons 1, 4, 7 and the buttons 3, 6, 9 of the numeric keypad in different colors, which are colors corresponding to the colors of the target objects. Here, for convenience, stickers are used, but the content described on the above stickers may be directly printed on the numeric keypad, or may be scribed, engraved, or carved using laser printing technology or the like.

[0121] This numeric keypad 270 corresponds to a system that can display a target object in six directions: directly to the left of the user (0 degrees in the protractor notation with the user as the origin), diagonally forward to the left (45 degrees), directly in front (90 degrees to the left), directly in front (90 degrees to the right), diagonally forward to the right (135 degrees), and directly to the right of the user (180 degrees). The appearance directions of the target object in this embodiment are merely examples, and the present invention is not limited to these angles. Each of the buttons 1, 4, 7, 3, 6, and 9 on the numeric keypad 270 is assigned to each of the above six directions. For example, pressing the button 1 may cause the target object to appear to the left (not limited to 0 degrees, for example, in the direction of -10 degrees to 10 degrees), and pressing the button 3 may cause the target object to appear in the right horizontal direction (not limited to 180 degrees, for example, in the direction of 170 degrees to 190 degrees). Similarly, pressing the button 7 may cause the target object for the left hand task to appear in the front direction (not limited to 90 degrees, for example, in the direction of 80 to 100 degrees), and pressing the button 9 may cause the target object for the right hand task to appear in the front direction (not limited to 90 degrees, for example, in the direction of 80 to 100 degrees). Pressing the button 4 may cause the target object for the left hand task to appear in any diagonal direction between the front and the left (for example, in the direction of 30 degrees to 60 degrees), and pressing the button 6 may cause the target object for the right hand task to appear in any diagonal direction between the front and the right (for example, in the direction of 120 degrees to 150 degrees).

[0122] Furthermore, as a modification of this embodiment, as shown in FIG. 6B, for the case where the operator stands in front of the user and operates the numeric keypad while looking at the user, a key assignment may be used in which up, down, left, and right are reversed. Alternatively, a configuration in which the key assignment in FIG. 6A and the key assignment in FIG. 6B can be selected may also be used.

[0123] When each of the buttons 1, 4, and 7 on the numeric keypad 270 is pressed, the control unit 212 and the setting unit 214 cause a target object for the left half of the user's body (here, the left hand, but it may also be the left foot, etc.) to appear. When each of the buttons 3, 6, and 9 on the numeric keypad 270 is pressed, the control unit 212 and the setting unit 214 cause a target object for the right half of the user's body (here, the right hand, but it may also be the right foot, etc.) to appear.

[0124] Each of the buttons 2, 5, and 8 on the numeric keypad 270 is used to set the appearance distance from a reference position preset in the virtual space as the appearance position of the target object. Specifically, when the button 2 is pressed, the setting unit 214 sets to make the target object appear at a position close to the user. When the button 8 is pressed, the setting unit 214 sets to make the target object appear at a position far from the user. When the button 5 is pressed, the setting unit 214 sets to make the target object appear at an intermediate position between them. At this time, when the distance buttons 2, 5, and 8 are pressed, the distance information is saved, and it is no longer necessary to specify the distance with the distance buttons 2, 5, and 8 every time after the next time. That is, if 1, 4, 7, 3, 6, and 9 are pressed after pressing 5, tasks at a medium distance will appear for all angles, and the distance information will not be changed unless the button 2 or 8 is pressed.

[0125] The numeric keypad 270 has functions assigned so as to be in a symmetrical arrangement, and the buttons 7 and 9 of the numeric keypad are used to make the target object appear in the front direction, the buttons 4 and 6 are used to make the target object appear in the diagonal direction, and the buttons 1 and 3 are used to make the target object appear in the left and right directions.

[0126] The setting unit 214 may provide a parameter cycling selection function such as the speed of the target object to the " / " button 501 of the numeric keypad 270. By repeatedly pressing the " / " button 501, the numerically changeable parameters are switched in the order of the speed setting area 321 of the left-handed target object → the radius setting area 322 of the left-handed target object → the radius setting area 323 of the visual assistance object of the left-handed target object → the sensor radius setting area 324 of the left-handed avatar object → the speed setting area 325 of the right-handed target object → the radius setting area 326 of the right-handed target object → the radius setting area 327 of the visual assistance object of the right-handed target object → the sensor radius setting area 328 of the right-handed avatar object, and then return to the speed setting area 321 of the left-handed target object.

[0127] Furthermore, the "+" button 503 and "-" button 502 of the numeric keypad 270 may be provided with functions for increasing and decreasing parameter values. In this case, it is also preferable that the amount of increase or decrease when the "+" or "-" button is pressed once varies depending on the type of parameter to be changed. For example, the speed setting increases or decreases by 5 cm per second, while the information related to the radius of the object increases or decreases by 1 cm each time. Furthermore, when the " / " button 501 and "*" button 504 of the numeric keypad 270 are pressed simultaneously, the on / off state of the check box 305 is switched. That is, based on the input from the numeric keypad, it is switched whether or not to set the appearance position of the target object.

[0128] In addition, two numeric keypads may be connected to the information processing device and operated by two operators. Similar to a training vehicle, it becomes possible for an instructor to operate the numeric keypad while giving instructions.

[0129] FIG. 6 is a diagram that clearly explains the operation method of the numeric keypad 270. As is clear here, the numeric keypad 270 does not implement a function to end the task. This is to avoid task interruption or task termination due to incorrect operation.

[0130] [Processing flow] FIG. 7 is a flowchart showing the processing flow in the information processing device 210. In step S701, as calibration processing, the target position of the rehabilitation operation is initialized according to the user 220. Specifically, first, the user 220 is asked to perform an operation to obtain the operable range as calibration, and after setting that range as the initial value, the target is initialized according to the user. This initial value may be set according to the user's operable range, or may be determined by a therapist providing treatment taking into account the user's body awareness function and the target treatment effect.

[0131] FIG. 8 is a diagram showing an example of a calibration screen 800. On the calibration screen 800, similar to the operation panel 300, in the user's visual field area 801 on the right side of the screen, an image that is displayed on the head-mounted display 233 and that the user 220 is viewing is shown. On the head-mounted display 233, an avatar object (in this example, the avatar object 244 on the right hand side) on either the left or right side for which calibration is to be performed is displayed.

[0132] Then, for each of the six directions of directly to the user's left, diagonally forward to the left, directly in front of the left side, directly in front of the right side, diagonally forward to the right, and directly to the user's right, the shortest distance (or in the case of the stationary mode or horizontal mode, also including height information) and the longest distance at which a target object appears are determined. Therefore, for each of those directions, the nearest position coordinates (distance and height) when the user 220 simply extends an arm and the position coordinates (distance and height) when the user bends the torso forward and extends the arm as much as possible are obtained. A position that is 90% of the position coordinates when the user bends the torso forward and extends the arm as much as possible is set as the farthest position coordinates (long coordinates) at which the target object appears. For a person with a bent waist, it is preferable to set the nearest position coordinates (reference position) higher. Also, for a patient with a problem in the feet such as the knees, setting the nearest position coordinates (reference position) lower is preferable in terms of enhancing the therapeutic effect on the physical disability.

[0133] The progress display area 801 is an area indicating the progress of calibration. For positions for which coordinates have already been acquired, a circle is drawn in the progress display area 801. S surrounded by a circle represents the shortest distance (Short coordinates), and L surrounded by a circle represents the longest distance (long coordinates). The acquisition timing of each coordinate may be performed by the user in response to a button operation provided on the controller, but when a check box 802 for disabling the user operation is checked, only the operator can input coordinates. On the numeric keypad, coordinate acquisition is operated by the "Enter" button 505.

[0134] When the nearest position coordinates are input, a wall 811 appears on the head-mounted display 233. By pushing the wall 811 forward, a wireframe 812 appears, and the wall 811 moves away (Fig. 9), and the position coordinates are obtained with the trunk tilted forward and the arms fully extended. At this time, a sound of "gugugugu" is emitted and the controller vibrates. That is, for all of vision, hearing, and touch, the feeling of pushing the wall is conveyed to the user. In this way, high affordance is realized so that the user can intuitively perform calibration. In addition, on the calibration screen 800, a measurement value display area 803, a decision button 804, a return button 805, a manual input area 806, an angle skip button 807, etc. are provided. On the numeric keypad, the decision is operated by the "Enter" button 505, the return is by the "-" button 502, and the skip is by the "+" button 503.

[0135] Returning to Fig. 7, next, in step S703, mode selection is performed. That is, for the user, either a manual mode (a mode in which point estimation parameters are set as task data for each task), a template mode (a mode in which two or more consecutive task data set in advance are used as a task data set), or a delegate mode (auto mode) in which the device automatically generates tasks is set. Further, here, the operator selects the type of task from among a horizontal (stationary) task (with or without a background image), a falling task (without a background image), a falling task (simple background image), and a falling task (complex background image).

[0136] Next, in step S704, task data is set. In the case of the template mode, reading is performed from the template (task data set).

[0137] In step S705, in manual mode, when each of the buttons 1, 4, and 7 on the numeric keypad 270 is pressed, the control unit 212 and the setting unit 214 cause a target object for the user's left hand (or left half of the body) to appear. When each of the buttons 3, 6, and 9 on the numeric keypad 270 is pressed, the control unit 212 and the setting unit 214 cause a target object for the user's right hand (or right half of the body) to appear. Then, a task (display of the target object by the control unit 212 and evaluation of achievement by the avatar object) is started.

[0138] Until it is determined that the task has ended in step S707, the processes of steps S704 and S705 are repeated.

[0139] When it is determined that the task has ended in step S707, the process proceeds to step S709, where the achievement score of the task is obtained and points are calculated.

[0140] According to the present embodiment, while assisting the user with one hand, the user can perform a numeric keypad operation with the other hand to generate a task. In rehabilitation, it is most important to match the pace of the patient. By using the numeric keypad, while operating the system, the user can concentrate on the movement of the patient, and furthermore, it is also possible to assist.

[0141] [Examples of other task screens] FIGS. 10 to 13 are diagrams showing other examples of the display on the head-mounted display 233 according to the present embodiment.

[0142] Figure 10 is an example of a screen that requests a user to perform a horizontal task. In a background image 1001 representing the streetscape of the Edo period, an image showing a medicine box 1011 as a target object is displayed. Further, below the medicine box 1011, a thousand-tael box 1013 is displayed as an item to be protected by the user, and a ninja 1015 is configured to gradually approach from the back. The speed of the ninja 1015 is the speed set in the input area 323 of the operation panel 300 (the speed here is synonymous with the time limit). Note that when the speed is set to 0, the ninja does not appear in the first place. A circle 1012 is displayed on the medicine box 1011 as a visual assistance object. When the medicine box 1011 is touched with a sensor object (the tip center of the avatar objects 243 and 244) before the ninja 1015 reaches the thousand-tael box 1013, the task is completed. Two types of circles 1012 are prepared, red and blue. For the medicine box 1011 surrounded by the red circle 1012, the task is to operate and touch the red avatar object 244 on the right corresponding to the controller 2310 held in the right hand. On the other hand, for the medicine box 1011 surrounded by the blue circle 1012, the task is to operate and touch the red avatar object 243 on the left corresponding to the controller 234 held in the left hand.

[0143] The seal box 1011 is displayed at the position (depth and angle) set in the input area 324 of the operation panel 300. The seal box 1011 does not change its position until the user touches the avatar objects 243, 244 in the virtual space. That is, it is a target object fixed in space (referred to as a horizontal task because it requires stretching the body horizontally). Such a fixed target object is very effective as a rehabilitation or treatment method for diseases such as cerebellar ataxia, diplopia, and inner ear dysfunction, and for rehabilitation and treatment methods for orthopedic diseases of the lower extremities such as the hip joint, knee joint, and ankle joint. That is, for patients who have forgotten how to move their bodies, it is possible to imprint a limited image of body movement on the brain by feedforward and align the alignment around the joints. By increasing the distance in the depth direction of the seal box 1011, the exercise intensity can be changed. Furthermore, by combining multi-channel biofeedback, motor ability, physical function, cognitive function, and sensory function are greatly improved. Also, according to such a horizontal task, chronic pain can be improved by promoting the reorganization of the cerebral cortex. Alternatively, it is also possible to recover peripheral nerve disorders including cognitive dysfunction called chemo brain, sensory dysfunction in which the position sense of cancer patients who have taken anticancer drugs decreases due to nerve disorders, and sequelae symptoms associated with novel coronavirus (COVID-19) infection. It may be possible to give hints in advance about where the target object will appear to reduce the cognitive load. Tactile information obtained by touching the body, multiple repetitions of verbal information, or a combination thereof is also effective in reducing the cognitive load, rather than verbal information. The method of verbal information may also reduce the cognitive load by giving a simpler instruction that is more straightforward and closer to the imperative form, or may take a more complex form of instruction in the form of a question, such as "Because it's blue? (Take it with your right hand)", or verbal information may be given in a form that includes a cognitive task such as calculation, such as "When you say a number divisible by 2, take it with your right hand". Note that not only the horizontal position and depth generated by the seal box 1011 but also the height may be configured to be settable.

[0144] FIG. 11 is an example of a screen that requests a user to perform a dropping task (simple background video). In FIG. 11, in a background image 1101 like a forest, a trigger object 1102 representing a monkey and a target object 1103 representing an apple are displayed. The task is achieved when the trigger object 1102 representing the monkey drops the target object 1103 representing the apple from the tree and the avatar object 1104 representing the monkey catches the target object 1103 that has approached the user. Here too, from the timing when the control unit 212 and the trigger object 1102 shake the tree and notify the occurrence of the target object 1103, the dropping of the target object 1103 is started after a predetermined time has elapsed, imposing a cognitive load on the user 220 while causing an attention disorder.

[0145] Furthermore, the control unit 212 can impose a much stronger cognitive load on the user 220 by simultaneously causing at least two to five target objects 1103 to exist in the three-dimensional virtual space. In other words, the control unit 212 causes at least two target objects 1103 to occur at different positions in the left-right direction within the three-dimensional virtual space.

[0146] In particular, if at least two target objects 1103 are generated at a plurality of positions in a direction different from the moving direction (the dropping direction in FIG. 11) of the target object 1103 (the left-right direction in FIG. 11), an even greater cognitive load can be imposed. That is, the user 220 has to move the controllers 234 and 235 considering the movement in the vertical direction, the difference in the generation positions in the left-right direction, and further the difference in the dropping positions in the depth direction, which means that the spatial awareness ability is also being tested. In this way, in addition to changing the predetermined time of the task, by adjusting the type, number, size, spatial spread, position, quantity, etc. of the information included in the notification image and notification sound including the trigger object, it becomes possible to quantitatively adjust and control the complexity of the information to be retained in memory, that is, the cognitive load that the brain should process for the user.

[0147] FIG. 12 and FIG. 13 are diagrams showing examples of screens for requesting a user to perform a dropping task (complex background video). In a background image 1201 representing a field, an image of a person representing a farmer is displayed as a trigger object 1202 that triggers the appearance of a target object. That is, the control unit 212 displays the trigger object 1202 as a notification image for notifying the occurrence of the target object 1203. After the trigger object 1202 throws the target object 1203 in the shape of a taro upward and after a predetermined time, a target object 1203 in the shape of a large taro appears from above the screen as shown in FIG. 13. The task is completed by moving the avatar object 1202 in the shape of a monkey to catch the falling target object 1203. The left and right avatar objects 1202 move on the screen in conjunction with the movements of the controllers 234 and 235.

[0148] The setting unit 214 can adjust the cognitive addition given to the user by setting the delay time from the timing when the trigger object 1202 throws the target object 1203 upward and notifies the occurrence of the target object 1203 until the target object 1303 is generated. The longer the delay time, the longer the period for retaining memory and the greater the load on the brain's information processing. Note that in conjunction with the movement of the trigger object 1202, the occurrence of the target object may be notified at the same timing with a radar chart type notification image 250, or voice notification may be combined.

[0149] In this way, the setting unit 214 can impose a cognitive load on the user not only in tasks with a background of only the horizon 246 as in FIG. 2A but also in tasks with a background having a large amount of information as in FIGS. 12 and 13. That is, it makes it difficult to remember the appearance of the target object 1203 and the position where the target object 1303 will fall, and gives a load close to the cognitive load required in real life to the rehabilitation user.

[0150] In particular, the control unit 212 imposes a cognitive load on the user 220 to process the background image in the brain by changing at least a part of the background image 245 over time. In the example of FIG. 12, for example, among the background images 1201, the clouds 1204 may be moved, the vegetation 1205 may be swayed, or an animal (not shown) not related to the target object may appear. Thereby, it is possible to prevent the user 220 from concentrating on the target object 1203 and make it difficult to remember the position where the target object 1203 will fall. More specifically, by displaying information unrelated to the task in the background image, an environment in which it is difficult to concentrate on the target object is prepared, and it can be said that the memory is made difficult by intentionally causing an attention disorder (more specifically, selective attention disorder, divided attention disorder, switching attention disorder, sustained attention disorder), and the cognitive load is controlled.

[0151] The reaching evaluation unit 215 evaluates the user's cognitive ability using information such as whether the avatar object accurately reaches the three-dimensional target position represented by the target object in a timely manner, the time interval and number from the generation notification of the target object to its generation, and the degree of the load that causes the attention disorder of the background image.

[0152] Among the various modes (display screens) as shown in FIGS. 2A, 10 to 13, the required accuracy of point estimation also differs depending on which mode is to be executed for the user. For example, the required accuracy of point estimation increases in the order of FIGS. 10 → 11 → 12 as compared with the case where there is no background as in FIG. 2A. Also, as described above, the required accuracy of point estimation also changes depending on the background sound, and it affects the joint connection.

[0153] [Other Embodiments] Although the invention of the present application has been described with reference to the embodiments above, the invention of the present application is not limited to the above embodiments. Various changes that can be understood by those skilled in the art within the technical scope of the invention of the present application can be made to the configuration and details of the invention of the present application. Also, a system or device in which the separate features included in each embodiment are combined in any manner is also included in the technical scope of the present invention.

[0154] In addition, the present invention may be applied to a system composed of a plurality of devices or to a single device. Further, the present invention is also applicable when an information processing program for realizing the functions of the embodiments is supplied to a system or a device and executed by a built-in processor. In order to realize the functions of the present invention by a computer, a program installed in the computer, a medium storing the program, a server for downloading the program, and a processor for executing the program are all included in the technical scope of the present invention. In particular, at least a non-transitory computer readable medium storing a program for causing a computer to execute the processing steps included in the above-described embodiments is included in the technical scope of the present invention.

Claims

1. An information processing device that receives operation contents from a numeric keypad by an operator operating the numeric keypad while watching a user and requests the user to perform a reaching operation, a control unit that causes a target object to appear in a virtual space when the user performs a reaching motion; a setting unit that sets an appearance position of the target object in response to an input from the numeric keypad; An information processing device comprising:

2. the setting unit includes, as parameters for the appearance position of the target object, an appearance distance from a reference position previously set in the virtual space and an appearance direction with respect to a reference direction previously set in the virtual space; The "2", "5", and "8" buttons on the numeric keypad are used to switch the appearance distance, 2. The information processing apparatus according to claim 1, wherein the buttons "1", "4", "7", "3", "6", and "9" on the numeric keypad are used to switch the appearance direction.

3. 3. The information processing device according to claim 2, wherein when the "2" button on the numeric keypad is pressed, the setting unit sets the target object to appear at a position close to the user, when the "8" button is pressed, the setting unit sets the target object to appear at a position far from the user, and when the "5" button is pressed, the setting unit sets the target object to appear at an intermediate position between the above.

4. the setting unit sets that the target object for the left half of the user's body appears when the buttons "1", "4", and "7" of the numeric keypad are pressed, and the control unit causes the target object for the left half of the user's body to appear when the buttons "1", "4", and "7" of the numeric keypad are pressed as a trigger; 2. The information processing device of claim 1, wherein the setting unit sets the target object for the right half of the user's body to appear when the ``3'', ``6'', or ``9'' buttons on the numeric keypad are pressed, and the control unit causes the target object for the right half of the user's body to appear in response to the pressing of the ``3'', ``6'', or ``9'' buttons on the numeric keypad.

5. The control unit is When the "1" or "3" button on the numeric keypad is pressed, the target object appears to the side of the user; When the "7" or "9" button on the numeric keypad is pressed, the target object appears in front of the user; The information processing apparatus according to claim 1 , wherein when the "4" or "6" button of the numeric keypad is pressed, the target object appears in a diagonal direction between a front direction and a lateral direction of the user.

6. a first notation or sticker indicating the role of each of the "1", "4", "7", "3", "6", and "9" buttons of the numeric keypad; A second notation or sticker indicating the role of each of the "2", "5", and "8" buttons on the numeric keypad; The information processing device according to claim 2 , further comprising:

7. The information processing device according to claim 6 , wherein the first and second notations or stickers include, as display contents, numbers that are originally used for the numeric keypad, and arrows and characters that indicate the functions of each button.

8. the first and second notations or stickers display the numeric keys "1", "4", and "7" in different colors from the numeric keys "3", "6", and "9", The information processing apparatus according to claim 6 , wherein the color corresponds to the color of the target object.

9. 9. The information processing apparatus according to claim 1, wherein the " / " button of the numeric keypad has a function of selecting a task parameter, and the "+" and "-" buttons of the numeric keypad have a function of increasing or decreasing a parameter value.

10. A method for controlling an information processing device, which receives operation content from a numeric keypad by an operator operating the numeric keypad while watching a user and requests the user to perform a reaching operation, comprising: a control step of causing a target object to appear in a virtual space as a target when the user performs a reaching motion; a setting step of setting an appearance position of the target object in response to an input from the numeric keypad; A method for controlling an information processing device comprising:

11. An information processing program for receiving operation contents from a numeric keypad by an operator operating the numeric keypad while watching a user and requesting the user to perform a reaching operation, a control step of causing a target object to appear in a virtual space as a target when the user performs a reaching motion; a setting step of setting an appearance position of the target object in response to an input from the numeric keypad; An information processing program that causes a computer to execute the above.

Images