Rehabilitation support system

Abstract

It can be introduced relatively easily and at a low cost, making it easier to carry out rehabilitation during the maintenance phase. [Solution] This disclosure proposes a rehabilitation support system that uses object images displayed in a virtual space to support rehabilitation performed by a rehabilitation practitioner, and which performs the following processes: detecting the movement of the rehabilitation practitioner's body parts; determining whether the body parts are designated body parts for rehabilitation; and determining whether to enable interaction with the object image to be manipulated by the rehabilitation practitioner based on the determination of whether or not it is a designated body part.

Description

Technical Field

[0005] , ,

[0001] The present disclosure relates to a rehabilitation support system.

Background Art

[0002] Currently, as rehabilitation for stroke patients (hereinafter referred to as "rehabilitation"), those using XR technology (a general term for cross-reality such as VR (virtual reality), AR (augmented reality), MR (mixed reality), etc.) implemented in hospitals and rehabilitation facilities are known. Rehabilitation in hospitals is performed in the acute phase, and rehabilitation in rehabilitation facilities is performed in the recovery phase. For example, Patent Document 1 discloses a device for supporting reaching movements in rehabilitation using VR in the recovery phase. It becomes like this.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] <00000(26]]However, the technology disclosed in Patent Document 1 relates to a rehabilitation support technology specialized for reaching movements, and is not a technology for supporting rehabilitation by more complex operations and movements such as pinching movements by fingers. Further, the technology according to Patent Document 1 requires a large-scale device despite supporting a relatively simple rehabilitation movement such as a reaching movement, and there is a problem that it cannot support rehabilitation in situations where it is difficult to introduce large-scale equipment, such as when performing rehabilitation at home.

[0005] Furthermore, rehabilitation includes not only the rehabilitation during the acute and recovery phases mentioned above, but also the rehabilitation during the subsequent maintenance phase (home care phase). How well this maintenance phase rehabilitation is conducted is crucial because it has a significant impact on the patient's life afterward. Neglecting maintenance phase rehabilitation can lead to severe after-effects. However, maintenance phase rehabilitation generally has the problem of patients neglecting or giving up on continuing rehabilitation because it is boring or they do not feel it is effective. Also, traditionally, rehabilitation focused on training compensatory movements (using healthy body parts (upper limbs) to substitute for the function of paralyzed body parts (upper limbs)) has been the mainstream. However, in recent years, the effectiveness of rehabilitation while restraining (fixing) a healthy body part (for example, the left upper limb in the case of right hemiplegia) to prevent movement (CI therapy: a method of rehabilitation while restraining a healthy body part (for example, the upper limbs)) has attracted attention. While CI therapy has been shown to be effective in maintenance phase rehabilitation, patients tend to feel a great deal of stress due to the restraint. This stress is a major factor in patients giving up on rehabilitation midway through the maintenance phase, and the same problems exist with CI therapy as described above.

[0006] In light of these circumstances, this disclosure proposes a technology that can be easily implemented at a relatively low cost and facilitates the performance of maintenance-phase rehabilitation. [Means for solving the problem]

[0007] To address the above issues, this disclosure provides, as an example, A rehabilitation support system that uses object images displayed in a virtual space to support rehabilitation performed by rehabilitation practitioners, A storage device that holds data of the object and a program to support the rehabilitation, A display device that displays the aforementioned object image on a screen, The system includes a control device that reads the program from the storage device and controls the display of the object image on the screen to support the rehabilitation, The control device is The process of detecting the movement of the body parts of the person performing the rehabilitation, A process to determine whether the body part is a designated body part specified for rehabilitation, Based on the determination of whether or not it is the designated body part, a process is performed to determine whether or not to enable interaction with the object image to be manipulated by the rehabilitation practitioner, We propose a rehabilitation support system that implements the following.

[0008] Further features relating to this disclosure will become apparent from the description herein and the accompanying drawings. Furthermore, aspects of this disclosure are achieved and realized by elements and various combinations of elements and the modes of the claims described herein in detail thereafter. The descriptions herein are typical examples only and do not limit in any way the claims or applications of this disclosure. [Effects of the Invention]

[0009] The technology disclosed herein makes it possible to realize a rehabilitation support system that can be easily implemented at a relatively low cost and facilitates the performance of maintenance-phase rehabilitation. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an overview (process) of the rehabilitation implementation in this embodiment. [Figure 2] This is a block diagram showing an example of the internal configuration of the HMD10, which corresponds to the rehabilitation support device according to this embodiment. [Figure 3] This figure shows an example of the configuration of object data management information 300 according to this embodiment. [Figure 4] This figure shows an example of the configuration of the rehabilitation setting screen 400 according to this embodiment. [Figure 5] This flowchart provides a detailed explanation of the overall rehabilitation support process according to this embodiment. [Figure 6]This is a diagram for explaining the setting of the thumb width threshold. [Figure 7] This is a flowchart for explaining the thumb width threshold setting process by the rehabilitation practitioner 1 or their supporter (e.g., family member). [Figure 8] This is a diagram showing the concept of thumb width setting according to the patient. [Figure 9] This is a flowchart for explaining the evaluation process according to the rehabilitation progress. [Figure 10A] This is a diagram for explaining the concept of the function of sharing an object (3DCG) used in rehabilitation among multiple people, showing the transmission of operation information from the HMD1_10A of the rehabilitation practitioner 1 to the HMD2_10B of the rehabilitation instructor. [Figure 10B] This is a diagram for explaining the concept of the function of sharing an object (3DCG) used in rehabilitation among multiple people, showing the transmission of display information from the HMD2_10B of the rehabilitation instructor to the HMD2_10B of the rehabilitation practitioner 1. [Figure 11A] This is a flowchart (first half) for explaining the details of the object sharing process when sharing a rehabilitation object (3DCG) among multiple people. [Figure 11B] This is a flowchart (second half) for explaining the details of the object sharing process when sharing a rehabilitation object (3DCG) among multiple people.

Mode for Carrying Out the Invention

[0011] This embodiment mainly discloses a function for disabling interaction with an object image (3DCG) to be operated on by a specific (designated) body part of a rehabilitation implementer in rehabilitation using technology (xR) that fuses the real world and the virtual world, a function that enables variable setting of a threshold value for recognizing that rehabilitation has been performed (the rehabilitation operation has been successful) according to the state of the rehabilitation implementer, and a function for sharing rehabilitation objects (3DCG) among multiple people. With these functions, it becomes possible to realize a rehabilitation support system that can be easily introduced at a relatively low cost and facilitates the performance of rehabilitation during the maintenance period.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may sometimes be denoted by the same number. Note that the accompanying drawings show specific embodiments and implementation examples in accordance with the principles of the present disclosure, but these are for the purpose of understanding the present disclosure and are not used to limit the interpretation of the present disclosure in any way.

[0013] In this embodiment, although the description is provided in sufficient detail for those skilled in the art to implement the present disclosure, other implementations and forms are possible, and it is necessary to understand that configuration and structural changes and replacement of various elements can be made without departing from the scope and spirit of the technical idea of the present disclosure. Therefore, the following description should not be construed as being limited thereto.

[0014] <Overview of Rehabilitation Implementation> Figure 1 is a diagram illustrating the overview (process) of rehabilitation implementation in this embodiment. Rehabilitation practitioner 1 starts rehabilitation in a room (space S) at home. When rehabilitation practitioner 1 puts on a head-mounted display with a camera (hereinafter referred to as HMD) 10 and inputs the instruction to start rehabilitation (for example, rehabilitation starts by activating the HMD 10), the HMD 10 superimposes and displays each object (3DCG) at predetermined three-dimensional coordinate positions on a display screen that shows the appearance of space S. Each object includes, for example, an image representing a virtual hand (hand of rehabilitation practitioner 1) (hand image 21), an image representing a virtual instructor (instructor image 22), an image representing a virtual table (table image 23), an image representing the object to be operated on during rehabilitation (bird image 24), an image representing a virtual cage (cage image 25), a virtual bulletin board image showing the operation rules (rule bulletin board image 26), and a virtual bulletin board image showing the operation results (result bulletin board image 27). The data for each object is stored in a memory device (see Figure 2: memory device 102) included in the HMD10. Furthermore, the data for each object is associated with three-dimensional coordinate data in a spatial coordinate system with the HMD10's position as the origin, and this data is also stored in the aforementioned memory device.

[0015] When the HMD10 is activated, the initial position of the rehabilitation practitioner's hand is set to the origin, for example, the same as the coordinate position of the HMD10, and the hand image 21 is displayed at that origin. When the rehabilitation practitioner moves their hand, the camera (see Figure 2: camera device 104) equipped with the HMD1 tracks the rehabilitation practitioner's hand and moves the superimposed hand image 21 in accordance with the change in the three-dimensional coordinates of the hand (the display position of the hand image 21 is changed by converting the space S into coordinates with the HMD10 as the origin). Furthermore, by using this camera tracking technology, it is possible to recognize the fingertips and each joint of the rehabilitation practitioner's hand, and it is possible to detect that a predetermined operation has been performed from the position of the fingertips and each joint. To recognize the rehabilitation practitioner's hand, for example, the technology disclosed at https: / / developers.meta.com / horizon / documentation / unity / unity-isdk-input-processing?locale=ja_JP can be used.

[0016] In the rehabilitation support device according to this embodiment, the rehabilitation practitioner 1 performs a predetermined action (for example, a pinching or grasping motion) on the bird image 24, which is the target of the operation, in the space S where each object is superimposed, according to the operation rules shown on the rule bulletin board image 26. Then, the HMD 10 uses camera tracking technology to determine whether the rehabilitation practitioner 1's hand was able to reach and pinch the bird image 24, and whether, after pinching, was able to move the bird image 24 to the location of the cage image 25 and store (drop) it in the cage. When the operation on one target (bird image 24) is completed, the HMD 10 displays the next target on the display screen, and the rehabilitation operation by the rehabilitation practitioner 1 is repeated. To detect the pinching motion by the rehabilitation practitioner 1, for example, the technology disclosed at https: / / developers.meta.com / horizon / documentation / unity / unity-isdk-hand-pose-detection / can be used.

[0017] The result display board image 27 is an image (3DCG) showing the rehabilitation results by the rehabilitation practitioner 1. In FIG. 1, an example of the actual value of the distance between the pinched fingers (between the thumb and other fingers) is shown.

[0018] Furthermore, the basket image 25 shows the stored bird image 24 and the animal image 28 that is converted when a plurality of bird images 24 collide. When more images collide, it is converted into an image of a larger animal. For example, when two bird images collide, it is converted into a cat image, when a bird image 24 collides with the cat image, it is converted into a dog image, and when a bird image 24 collides with the dog image, it is converted into a horse image, and so on.

[0019] The instructor image 22 may simply be displayed at a predetermined position, or for example, it may be configured to issue a predetermined message at a predetermined timing by voice synthesis. For example, when performing a pinching operation on the first bird image 24 according to the operation rules, "Please pinch the yellow chick with your thumb and index finger and put it in the basket", when it comes to the second bird image 24, "Please pinch the light brown little bird with your thumb and middle finger and put it in the basket",... and so on, the instructor image 22 may issue such voice messages (synchronized with the mouth movement).

[0020] <Internal configuration example of HMD> FIG. 2 is a block diagram showing an internal configuration example of the HMD 10 corresponding to the rehabilitation support device (rehabilitation support system) according to the present embodiment.

[0021] (i) The HMD 10 is composed of a processor such as a CPU or MPU, a control device 101 including an internal memory not shown, a storage device 102 composed of a memory, RAM (Random Access Memory), ROM (Read Only Memory), HDD (Hard Disk Drive), etc., an input device 103 composed of a switch button, keyboard, liquid crystal touch panel, remote controller, microphone, etc., a camera device 104, a display device 105, and a communication device 106.

[0022] (ii) The control device 101 reads the rehabilitation support program from the memory device 102, loads it into its internal memory, and constructs the body part detection unit 1011, the object drawing unit 1012, the object position calculation unit 1013, the interaction disabling unit 1014, and the evaluation unit 1015.

[0023] The memory device 102 stores the rehabilitation support program, various object data, various parameters (including the thumb width threshold described later), and past rehabilitation result information.

[0024] The input device 103 is used to start and stop the HMD 10, select rehabilitation modes (such as rehabilitation using pinching movements, rehabilitation using grasping movements, and rehabilitation using leg movements), and input (change) the pinch width threshold described later.

[0025] The camera device 104 is a sensor configured to track (detect) the position of the rehabilitation practitioner 1's hand and the position of the 3DCG object being manipulated (bird image 24), and to transmit the position information to the control device 101. In addition to the camera device 104, the HMD 10 may also be equipped with an IR sensor, a temperature sensor, an acceleration sensor, and the like.

[0026] The display device 105 is configured to display a real image of the space S (such as a room in the home) where the rehabilitation practitioner 1 is located, and various objects (such as the table image 23 and the bird image 24 mentioned above) superimposed on it.

[0027] The communication device 106 is configured to communicate with other HMDs via a network (local network or internet). Alternatively, the communication device 106 may be configured to communicate with a dedicated rehabilitation support terminal (not shown) installed in a hospital or rehabilitation facility.

[0028] (iii) The body part detection unit 1011, built into the internal memory of the control device 101, identifies (detects) the body part (hand) based on the captured image and position information (e.g., position information of the fingertips and each joint) of the body part (hand) acquired from the camera device 104, and provides the identification / detection result to the object drawing unit 1012 and the object position calculation unit 1013. For example, the body part detection unit 1011 can detect and identify whether the rehabilitation practitioner 1 moved their right hand or their left hand from the captured image of the rehabilitation practitioner 1's hand and the position information of the fingertips and each finger joint.

[0029] The object drawing unit 1012 acquires data for each object (see Figure 3) from the memory device 102 and places (draws) images of each object at predetermined placement coordinates (virtual space coordinates in space S). The object drawing unit 1012 also changes the position of the hand image 21 to match the position information (tracked by the camera device 104) of the body parts (e.g., the right hand) that change as the rehabilitation practitioner 1 performs rehabilitation movements, and draws (overlays) this image.

[0030] The object position calculation unit 1013 primarily calculates the position of the image to be manipulated (bird image 24). In reality, the image to be manipulated cannot be tracked by the camera device 104. Therefore, the object position calculation unit 1013 uses the x and y coordinates of the position information of the body part of the rehabilitation practitioner 1 (fingertip: for example, in the case of a grasping motion with the thumb and middle finger, the midpoint of the tip of the thumb and the tip of the middle finger) as the x and y coordinates of the image to be manipulated (bird image 24). For the z coordinate, the z coordinate of the position of the fingertip may be used as the position of the image to be manipulated (bird image 24), or a position that is a predetermined distance away from the position of the fingertip (a different distance is predetermined depending on the type of bird image 24) may be used as the center position of the image to be manipulated (bird image 24). The object position calculation unit 1013 provides the calculated position information (x, y, z) of the image to be manipulated (bird image 24) to the object drawing unit 1012.

[0031] The interaction disabling unit 1014 obtains information from the body part detection unit 1011 about the body part (actual body part) that the rehabilitation practitioner 1 is using to manipulate the target (bird image 24), and information from the memory device 102 about the body part (designated body part) that is the target of rehabilitation (see Figure 4), and compares these. If the actual body part is different from the designated body part, the interaction disabling unit 1014 disables the manipulation (interaction) with the target (bird image 24). For example, if the right thumb and right index finger (pinching motion) are designated as the body parts to be manipulated, the manipulation (interaction) with the target (bird image 24) by the rehabilitation practitioner 1 using their left hand will be disabled. Here, interaction disabling means that even if the rehabilitation practitioner 1 tries to pinch or move the bird image 24, no response will be generated to those actions.

[0032] The judgment and evaluation unit 1015 compares the information of the distance between the fingers of the rehabilitation practitioner 1 (for example, the distance between the tip of the thumb and the tip of the index finger) detected by the body part detection unit 1011 with the pinch width threshold (see Figure 4) to determine whether the operation (pinching motion) on the target object (bird image 24) by the rehabilitation practitioner 1 was successful. The judgment and evaluation unit 1015 also compares the information of the distance between the fingers, which the body part detection unit 1011 continues to detect, with the pinch width threshold (see Figure 4) to determine whether the operation (pinching motion) on the target object (bird image 24) by the rehabilitation practitioner 1 is being maintained. Furthermore, the judgment and evaluation unit 1015 calculates the degree of rehabilitation achievement based on the rehabilitation results (for example, by assigning points to the rehabilitation results) and displays the calculated degree of achievement on the display device 105. For example, if the player can perform a pinching motion on each target object (bird image 24) and complete the action from the initial display position of bird image 2 to placing it in the cage, they will be awarded 10 points for that action. If the pinching motion is insufficient along the way to the cage and bird image 24 is dropped, the player may be awarded 10 points multiplied by the percentage of distance (percentage) that was moved as the score for that action. Further details regarding the rehabilitation support process in HMD10 will be described later, following the flowchart in Figure 5.

[0033] <Object Data Management Information> Figure 3 shows an example of the configuration of object data management information 300 according to this embodiment. In Figure 3, object data is managed using a table format, but any format is acceptable.

[0034] The object data management information 300 includes, as constituent items: identification information 301 for uniquely identifying and specifying each object; object type 302 indicating the type of each object; object data 303 indicating the 3DCG data of each object; and placement position (x,y,z) 304 to 306 indicating the coordinate position (initial coordinate position) where each object is first displayed. However, it may also contain other information.

[0035] Object type 302 includes, for example, a table, a storage basket, an instructor, a right hand, a left hand, and objects to be manipulated (bird 1, bird 2, ...) when performing a pinching motion as a rehabilitation exercise, as shown in Figure 1.

[0036] Object data 303 is a pre-prepared 3D image (3DCG) that is actually displayed on the display surface. Alternatively, only pointer information may be stored in object data 303, while the actual 3DCG is stored in a separate memory area of ​​memory device 104 or in another memory device. This allows for easy modification of the object data.

[0037] The initial positions of each object are shown in placement positions 304 to 306, but the table, storage basket, and instructor object may remain displayed in a fixed position (initial position) in space S. On the other hand, the right hand, left hand, and manipulation object (bird 1, bird 2, ...) objects are made to change their display position from the initial position according to the actions of rehabilitation practitioner 1.

[0038] <Rehabilitation settings screen (GUI: Graphical User Interface)> Figure 4 shows an example of the configuration of the rehabilitation setting screen 400 according to this embodiment. The rehabilitation settings screen 400 includes the following components: date and time of rehabilitation 401, person performing the rehabilitation 402, rehabilitation instructor 403, number of rehabilitation sessions 404, target body part 405, rehabilitation movement 406, number of operations 407, body part being operated on 408, pinch width threshold (operation success / failure judgment threshold) 409, and previous rehabilitation information 410.

[0039] The date and time of rehabilitation session 401 can be entered directly by the rehabilitation provider 1 or their assistant (for example, a family member of the rehabilitation provider 1), or it can be automatically entered from calendar information.

[0040] The information from rehabilitation practitioner 402 to rehabilitation instructor 403, and from the target body part 405 to the pinch width threshold 409, is information set and entered by rehabilitation practitioner 1, etc. However, for example, rehabilitation practitioner 402, rehabilitation instructor 403, rehabilitation movement 406, number of operations 407, and body part 408 may be automatically entered with the same information once it is initially set until it is changed.

[0041] For example, if rehabilitation provider 1 (and their family) performs rehabilitation alone (without seeking instructions from a doctor or instructor), rehabilitation instructor 403 will enter "NA (Not Applicable)". If rehabilitation provider 1 and the rehabilitation instructor are in the same space or separate spaces and sharing objects, the name of the rehabilitation instructor will be entered.

[0042] The pinch width threshold 409 is a threshold used to determine whether the rehabilitation exercise, when it involves pinching, was successful in pinching (interacting) with the target object (bird image 24). In other words, if the rehabilitation practitioner 1 performs a pinch motion on the bird image 24 with a width smaller than the threshold (for example, if the distance between the thumb and index finger is less than 1.5 cm), the pinch motion is judged to be successful. This pinch width threshold 409 can be changed as rehabilitation continues day by day. Therefore, the threshold can be set (customized) according to the degree of disability of the rehabilitation practitioner 1 (patient), allowing them to feel the effects of rehabilitation during the maintenance phase (home phase). This helps to maintain and improve the rehabilitation practitioner 1's motivation for rehabilitation.

[0043] The previous rehabilitation information (410) is a section that displays the rehabilitation movements performed and the results (movements achieved) from the previous rehabilitation session, and it is displayed automatically.

[0044] <Overall picture of rehabilitation support processing: detailed content> Figure 5 is a flowchart illustrating in detail the overall structure of the rehabilitation support process according to this embodiment. Hereinafter, the content of the rehabilitation support process will be explained assuming that the processing entity for each step is one of the body part detection unit 1011 to the evaluation unit 1015 or the control device 101. However, since the body part detection unit 1011 to the evaluation unit 1015 are processes built within the processor, it may be understood that the processing entity for each step is entirely the control device (processor).

[0045] (i) Step S501 When rehabilitation practitioner 1 or their assistant turns on the HMD 10, the control device 101 performs an initialization process. During the initialization process, the control device 101 displays, for example, a reset rehabilitation settings screen 400, fixed items and continuing items (the same information as last time), and reflects (stores) the information of the fixed and continuing items, as well as the information of newly set (input) items, in the memory device 104 as information for rehabilitation execution.

[0046] (ii) Step S502 The object drawing unit 1012 retrieves each object data 303 from the object data management information 300 stored in the storage device 104 and displays it superimposed on the initial position (coordinates of placement positions 304 to 306) on the three-dimensional coordinate system of the display screen space S.

[0047] (iii) Step S503 The body part detection unit 1011 acquires imaging information from the camera device 104 and tracks the movement of the body parts of the rehabilitation practitioner 1. The body part detection unit 1011 also acquires position information of the extremities and joints of the body parts through tracking and determines whether the body part being moved by the rehabilitation practitioner 1 is the same as the target body part (designated body part) 405 set via the rehabilitation setting screen 400.

[0048] (iv) Step S504 The body part detection unit 1011 continues to trunk the movement of the rehabilitation practitioner's body parts (e.g., hands) based on the information from the camera device 104, and determines whether the hands have reached the position of the object to be manipulated (bird image 24). If the hands have reached the object to be manipulated (Yes in step S504), the process proceeds to step S505. If the hands have not reached the object to be manipulated (No in step S504), the process proceeds to step S503, and the tracking process of the rehabilitation practitioner's hands continues.

[0049] (v) Step S505 The interaction disabling unit 1014 obtains the detection result from the body part detection unit 1011 and determines whether the operation (pinching motion) performed by the rehabilitation practitioner 1 on the target of the body part detection unit 1011 (bird image 24) is performed by the target body part (designated body part). If the body part being operated is the designated body part (for example, the right hand) (if Yes in step S505), the process proceeds to step S507. If the body part being operated is not the designated body part (for example, the right hand) (if No in step S505: if it is the left hand), the process proceeds to step S506.

[0050] (vi) Step S506 The interaction disabling unit 1014 disables operations performed by the rehabilitation practitioner 1. For example, if the rehabilitation is set to perform a pinching motion with the right hand (thumb and index finger), the rehabilitation practitioner 1 is controlled so that they cannot pinch the target object (bird image 24) with their left hand or with fingers other than the designated ones on their right hand (interaction disabling). At this time, the interaction disabling unit 1014 may also display a message on the display screen of the display device 105 prompting the rehabilitation practitioner 1 to change the body part used for the operation to the designated body part. Then, the process moves to step S503, and tracking processing for the rehabilitation practitioner 1's body part (designated body part) continues.

[0051] (vii) Step S507 The judgment and evaluation unit 1015 obtains information on the distance between designated body parts of the rehabilitation practitioner 1 (for example, the tip of the right thumb and the tip of the middle finger) from the body part detection unit 1011, compares this with the pinch width threshold 409 (for example, 2.5 cm) obtained from the memory device 10, and determines whether the operation (pinching motion) by the rehabilitation practitioner 1 was successful or not by determining whether the distance between the designated body parts is less than or equal to the pinch width threshold. If the operation is determined to be successful (Yes in step S507), the process proceeds to step S508. If the operation is determined to be unsuccessful (No in step S507), the process proceeds to step S511.

[0052] (viii) Step S508 The body part detection unit 1011 uses imaging information from the camera device 104 to track a designated body part (for example, the tip of the right thumb and the tip of the middle finger), detects the position (three-dimensional coordinates) of the designated body part, and calculates the distance between the designated body parts (for example, the distance between the tip of the right thumb and the tip of the middle finger).

[0053] Furthermore, the object position calculation unit 1013 calculates the position (three-dimensional coordinates) of the object being manipulated (pinched) by the designated body part (bird image 24). Since the object being manipulated (bird image 24) is an object (3DCG), its position cannot be detected using camera tracking technology. Therefore, by setting the position of the object being manipulated (bird image 24) to the point where it is pinched by the designated body part, the coordinate position of the object being manipulated can be made the same as the position of the designated body part. Alternatively, the x and y coordinates of the object being manipulated may be made the same as the x and y coordinates of the designated body part, and the z coordinate may be set to the value obtained by subtracting a predetermined distance (a predetermined numerical value) from the z coordinate of the designated body part.

[0054] (ix) Step S509 The judgment / evaluation unit 1015 obtains the distance between specified body parts (for example, the distance between the tip of the right thumb and the tip of the middle finger) from the body part detection unit 1011, and compares this with the pinch width threshold to determine whether the operation (pinching motion) on the target of the operation (bird image 24) is being maintained. If the operation is being maintained (Yes in S509), the process proceeds to step S510. If the operation is no longer being maintained (S509: the bird image 24 is no longer being pinched and is dropped), the process proceeds to step S511.

[0055] (x) Step S510 The judgment and evaluation unit 1015 determines, based on the position information of the designated body part obtained from the body part detection unit 1011, whether it was possible to reach the target location (for example, the cage) (for example, the x and y coordinates of the right hand being within a predetermined range from the x and y coordinates of the cage's installation position) and drop the object to be manipulated (bird image 24) into the cage. When the designated body part is located on the target location (cage) (within the predetermined range), the object to be manipulated (bird image 24) is also located on the target location, so the object drawing unit 1012 draws the object to be manipulated (bird image 24) on the display screen so that it falls in a predetermined motion.

[0056] If the object to be operated on (bird image 24) reaches the target location (cage) and can be stored there (if the answer is Yes in step S510), the process proceeds to step S512. On the other hand, if the object to be operated on (bird image 24) does not reach the target location (cage) (if the answer is No in step S510), the process proceeds to step S511.

[0057] (xi) Step S511 The judgment and evaluation unit 1015 determines that the operation performed by the rehabilitation practitioner 1 in that instance was unsuccessful. Note that the degree of success may be evaluated differently depending on whether (a) the operation (pinching the bird image 24) itself was unsuccessful, or (b) the operation (pinching the bird image 24) was successful but dropped on the way to the target location. For example, in the former case (a), the degree of success may be set to "0 (0 points)", and in the latter case (b), the degree of success may be set to "(actual distance traveled / distance to target location) × 100 (points)".

[0058] (xii) Step S512 The control device 101 determines whether operations have been completed for all targets (i.e., whether a preset number of operations has been reached). If operations have been completed for all targets (i.e., Yes in step S512), the process proceeds to step S513. If operations have not been completed for all targets (i.e., No in step S512), the process proceeds to step S514. Note that the determination of the end of rehabilitation may be made not only based on whether a preset number of operations has been reached, but also by adding conditions such as whether a predetermined time has elapsed or whether an action at the target threshold has been performed.

[0059] (xiii) Step S513 The judgment and evaluation unit 1015, for example, sums up the scores for operations on all target subjects (all bird images 24) and displays them on the display screen. The judgment and evaluation unit 1015 may also output evaluation comments for the rehabilitation performance. For example, if the total score is above a predetermined score, the rehabilitation action may be judged as a success and a comment recommending that the pinch width threshold be narrowed in the next rehabilitation may be output on the display screen. If the total score is below the predetermined score, the rehabilitation action may be judged as a failure and a comment recommending that the same pinch width threshold be used in the next rehabilitation may be output.

[0060] (xiv) Step S514 The control device 101 selects the next target for operation (bird image 24) and instructs the object drawing unit 1012 to display the selected bird image 24 at a predetermined initial position on the display screen.

[0061] <Changing the thumb width threshold> Figure 6 is a diagram illustrating the setting of the pinch width threshold. Figure 6A is Table 600, which shows examples of pinch width threshold settings in the previous and current rehabilitation support. Figure 6B is a diagram showing the positions of the fingertips and joints of each finger on the right hand of rehabilitation provider 1.

[0062] (i) Table 600 holds information (values) of pinch width thresholds 601 to 604 for determining whether a pinch operation has been performed by the thumb and fingers of the right hand (example), which is separately specified in "Specified Body Part 405" on the rehabilitation setting screen 400. In Figure 6A, the pinch width threshold used in this rehabilitation support is set to be narrower than the pinch width threshold used in the previous rehabilitation support. However, if the pinch operation was unsuccessful in the previous rehabilitation, the pinch width threshold for this rehabilitation can be the same value as the previous one.

[0063] Furthermore, even if the rehabilitation practitioner 1 or their supporter (family member) has successfully completed the rehabilitation exercise using the previous pinch width threshold, they may consider the progress of the rehabilitation and / or the feelings of the rehabilitation practitioner 1, and either adopt the same pinch width threshold again or deliberately set a wider pinch width threshold. This makes it possible to continue rehabilitation without forcing the rehabilitation practitioner 1 to perform unnatural rehabilitation exercises and causing excessive stress.

[0064] (ii) As shown in Figure 6B, the pinch width 610 is defined by the distance between the tip of the thumb 611_1 and any of the tips of the index finger 611_2 to the tip of the little finger 611_5. The pinch width 610 can be determined by measuring the position (coordinates) of each fingertip 611_1 to 611_5 using tracking technology based on images captured by the camera device 104, and calculating the distance between the coordinates of the thumb tip and the coordinates of each fingertip. Furthermore, using the above tracking technology, not only the position of each fingertip but also the position of each finger joint 612 can be identified. Therefore, when defining the pinch width threshold 610, it is also possible to set the pinch width threshold 409 as the pinch width 610 by setting, for example, the first joint of the thumb and the second joint of the index finger as "specified body parts 405" and the width between them as the pinch width 610, in addition to the distance between each fingertip.

[0065] <Thumb width threshold setting process> Figure 7 is a flowchart illustrating the process of setting the pinch width threshold by the rehabilitation provider 1 or their supporter (e.g., a family member).

[0066] (i) Step S701 When rehabilitation practitioner 1 or their supporter (hereinafter referred to as rehabilitation practitioner 1, etc.) inputs instructions for setting the thumb width threshold via the input device 103, the control device 101 displays the rehabilitation setting screen 400 (GUI) shown in Figure 4 on the display screen of the display device 105.

[0067] (ii) Step S702 When rehabilitation practitioner 1 or the like inputs the pinch width threshold for each finger into the pinch width threshold 409 on the rehabilitation setting screen 400 displayed on the display screen, the control device 101 accepts that value.

[0068] (iii) Step S703 The control device 101 refers to the previous pinch width threshold (or each pinch width threshold used in previous rehabilitation sessions) and determines whether the input value is appropriate. For example, if the input value is less than half of the previous pinch width threshold, the control device 101 may determine that it is inappropriate because it represents a sudden change that could cause stress. Alternatively, the control device 101 may calculate the average rate of change from the transitions (changes) of the pinch width thresholds up to now and determine that the input value is appropriate if the current pinch width threshold is within the average rate of change. Here, the control device 101 determines that the input value is inappropriate when the pinch width threshold decreases too rapidly (when the load on rehabilitation practitioner 1 is thought to be excessive), but it may also determine that the input value is inappropriate when the change in the pinch width threshold is too gradual (when the rehabilitation effect cannot be expected).

[0069] If the input value is valid (Yes in step S703), the process proceeds to step S704. If the input value is invalid (No in step S703), the process proceeds to step S705.

[0070] (iv) Step S704 The control device 101 decides to adopt the input pinch width threshold in the next rehabilitation session.

[0071] (v) Step S705 The control device 101 displays a message on the screen recommending that, in the next rehabilitation session, the input thumb width threshold should not be used, and that a thumb width threshold with a more gradual change should be used instead.

[0072] (vi) Step S706 The control device 101 displays a message on the display screen asking whether to forcibly adopt or change the entered thumb width threshold. If the rehabilitation practitioner 1 or the like chooses to forcibly apply the entered thumb width threshold (Yes in step S706), the process proceeds to step S704. On the other hand, if the rehabilitation practitioner 1 or the like chooses to change the entered thumb width threshold (No in step S706), the process proceeds to step S702.

[0073] In this example, a threshold of 409 is used to set the pinching width threshold for pinching movements, but thresholds for other rehabilitation movements may also be used. For example, thresholds for grasping movements using the entire hand, sliding movements in arm and leg movements, and flexion / extension movements in finger flexion / extension movements may be set for various rehabilitation movements.

[0074] <The concept of adjusting the knob width to suit the patient> Figure 8 shows the concept of setting a pinch width threshold tailored to the patient. Figure 8A shows the case where the pinch width threshold is set relatively large (setting scenario 801), and Figure 8B shows the case where the pinch width threshold is set relatively small (setting scenario 802).

[0075] In setting scenario 801, the knob width is relatively wide, but if it is narrower than the knob width threshold, it is determined that a knob operation has occurred. On the other hand, in setting scenario 802, the knob width in setting scenario 801 is not considered to be a knob operation, and it is only determined that a knob operation has occurred when the knob width becomes narrower.

[0076] The pinch width threshold in setting scenario 801 can be set to, for example, the threshold for the initial stage of the maintenance phase (home care phase) described above, while the pinch width threshold in setting scenario 802 can be set to the threshold for the middle stage and beyond of the maintenance phase. In this way, the pinch width threshold can be changed according to the patient's rehabilitation stage and condition, making it possible to support the implementation of rehabilitation without causing excessive stress to the patient, while allowing them to feel the effects of the rehabilitation.

[0077] <Evaluation based on rehabilitation progress> Figure 9 is a flowchart illustrating the evaluation process based on the progress of rehabilitation.

[0078] (i) Step S901 The judgment and evaluation unit 1015 acquires information from the memory device 102 regarding the results of rehabilitation exercises performed to date (rehabilitation achievement level: total score for each) and the pinch width threshold used for each rehabilitation exercise.

[0079] (ii) Step S902 The judgment and evaluation unit 1015 determines that the progress is "good" if there has been no significant drop in the rehabilitation achievement level to date (a decrease of more than a predetermined percentage from the previous achievement level) and the progress is smooth (the achievement level is above a certain value), or if there has been a drop but recovery has been observed in the achievement level of the next rehabilitation (for example, if it is above the achievement level of the time before last). Furthermore, if the progress of the rehabilitation achievement level is on a downward trend, it may be judged as "poor progress" or "stagnation." However, the value of the rehabilitation achievement level will change depending on the set thumb width threshold. In other words, if only a loose thumb width threshold is used, the achievement level is likely to be higher, and if only a strict thumb width threshold is used, the achievement level is likely to be lower. Therefore, when the rate of change of the thumb width threshold is set to be greater than the predetermined rate of change (rate of change from the previous thumb width threshold), the rehabilitation achievement level corresponding to the thumb width threshold may be multiplied by a coefficient (for example, 1.5) corresponding to the rate of change of the thumb width threshold to determine the progress of the rehabilitation achievement level. For example, the coefficient multiplied by the achievement level for each rehabilitation session may be determined by calculating the average percentage change of the pinch width threshold from the previous pinch width threshold for each rehabilitation session, and then dividing the percentage change for each rehabilitation session by the average percentage change. In other words, for example, if the average percentage change of the pinch width threshold is 10%, and the percentage change of the pinch width threshold for the current rehabilitation session is 20%, the coefficient would be 2.

[0080] (iii) Step S903 The judgment / evaluation unit 1015 outputs the progress evaluation obtained in step S902 onto the display screen of the display device 105.

[0081] Figures 10A and 10B illustrate the concept of a function that allows multiple people to share objects (3DCG) used in rehabilitation. Figure 10A shows the transmission of operation information from HMD1_10A of rehabilitation practitioner 1 to HMD2_10B of rehabilitation instructor. Figure 10B shows the transmission of display information from HMD2_10B of rehabilitation instructor to HMD1_10A of rehabilitation practitioner 1. In this embodiment, HMD2_10B of rehabilitation instructor operates as the host computer (hereinafter referred to as the host), and HMD1_10A of rehabilitation practitioner 1 operates as the client computer (hereinafter referred to as the client). The host primarily (primarily) owns the ownership of each object shared by the host and client. When interacting with an object (performing a predetermined operation), the client must temporarily acquire ownership of the corresponding object from the host (secondary (subsidiary) acquisition of ownership). Here, ownership is a concept that encompasses the reference of position coordinates (three-dimensional coordinates), the judgment of executing processing, etc. Additionally, ownership can be set for each object.

[0082] In this example, HMD1_10A is set as the client and HMD2_10B as the host, but either can be configured to act as the host. Furthermore, this embodiment primarily describes the sharing of objects between two individuals, a rehabilitation practitioner 1 and a rehabilitation instructor, but objects can also be shared by multiple rehabilitation instructors and multiple rehabilitation practitioners (assuming group rehabilitation in an MR space). Here, we assume that rehabilitation practitioner 1 and the rehabilitation instructor are physically present in space S while sharing objects. Additionally, it is assumed that multiple HMDs each hold common object data (each 3DCG data) in their respective memory devices.

[0083] When multiple HMDs share the same object, the three-dimensional coordinates used as a reference differ for each HMD, requiring adjustment of the three-dimensional coordinates. For example, by aligning the position of the object (the object being manipulated) displayed on HMD1_10A with the position of the object displayed on HMD2_10B, rehabilitation practitioner 1 and the rehabilitation instructor can view (observe) the same object from different positions without positional discrepancies. Here, "no positional discrepancies" means that when the object's position information in the three-dimensional coordinates of HMD1_10A is converted to the three-dimensional coordinates of HMD2_10B, the converted position information matches the position information of the object as seen from HMD2_10B.

[0084] Specifically, when each HMD is activated, the origin in the three-dimensional coordinate system is set to a predetermined position on each HMD (for example, the midpoint of the display screen of the display device 105). HMD2_10B (host) then acquires the location information of other HMD1_10A (clients) based on the image captured by the camera device 104 (using tracking information). HMD2_10B then transmits the location information of HMD1_10A as seen from HMD2_10B (relative location information) to HMD1_10A via the network 1000 (for example, LAN: Local Area Network) (see Figure 10B). HMD1_10A can then recognize the degree of deviation (difference) from HMD2_10B's three-dimensional coordinate system based on the relative location information acquired from HMD2_10B.

[0085] Furthermore, each HMD has position information for each object in its own three-dimensional coordinate system. For each object, HMD2_10B transmits the position information (coordinate position in the three-dimensional coordinate system of HMD2) to HMD1_10A. HMD1_10A adjusts (determines) the display position of each object in HMD1_10A based on the position information of each object obtained from HMD2_10B and information on the degree of the shift in the three-dimensional coordinate system of HMD2_10B. For example, the display positions (coordinates) of fixed objects (table image 23, cage image 25, rule bulletin board image 26, and result bulletin board image 27, etc.) and the initial positions of manipulated objects (bird image 24) and moving objects (hand image 21) are predetermined in the three-dimensional coordinate system of each HMD. Therefore, HMD1_10A adjusts the position of each object placed in the three-dimensional coordinate system of HMD1_10A by considering the difference in the origin position of the three-dimensional coordinate system with HMD2_10B. This allows rehabilitation practitioner 1 and rehabilitation instructor to view the same object from different positions in a consistent and natural manner (without causing positional discrepancies).

[0086] When the object to be manipulated (bird image 24) is manipulated (pinched) by rehabilitation practitioner 1, HMD1_10A detects the pinching motion by rehabilitation practitioner 1 via camera tracking and requests a transfer of ownership to HMD2_10B. Once ownership is transferred from HMD2_10B, HMD1_10A uses the camera device 104 to acquire positional information of rehabilitation practitioner 1's body part (e.g., right hand) and transmits that body part's positional information to HMD2_10B. Then, each HMD, or either HMD, calculates the difference between the position coordinate system of HMD1 (first coordinate system) and the position coordinate system of HMD2 (second coordinate system) and renders the object to be manipulated (rehabilitation object: bird image 24). When the rehabilitation operation is completed, HMD1_10A returns the temporarily acquired ownership to HMD2_10B.

[0087] A different method may be used for rendering the rehabilitation object. For example, when the object to be manipulated (bird image 24) is manipulated (pinched) by the rehabilitation practitioner 1, HMD1_10A determines the success or failure of the pinching motion (as described above, by comparing the distance between the fingers with the pinching width threshold) using camera tracking, and transmits information on the success or failure of the operation (operation information) to HMD2_10B via network 1000 (Figure 10A). If the operation (pinching motion) is successful, HMD2_10B obtains the position information of the rehabilitation practitioner 1's hand using camera tracking, reflects this in the position of the object (manipulated object: bird image 24) in its own three-dimensional space, and displays the object on the display screen. Then, HMD2_10B transmits the changing position information of the object to HMD1_10A via network 1000 (Figure 10B). HMD1_10A calculates the position of an object based on the acquired position information of the changing object and the information of the three-dimensional coordinate shift, and displays the object on the HMD1_10A's display screen.

[0088] Furthermore, when developing the position sharing function with rehabilitation objects, the inventors referred to the techniques described at https: / / mirror-networking.gitbook.io / docs and https: / / mirror-networking.gitbook.io / docs / manual / components / network-transform, and added or improved upon them.

[0089] <Details of object sharing process> Figures 11A and 11B are flowcharts illustrating the details of the object sharing process when multiple people share a rehabilitation object (3DCG).

[0090] (i) Steps S1101A and S1101B When rehabilitation practitioner 1 and rehabilitation instructor (supporter) each put on HMD1_10A and HMD2_10B respectively and activate them (+ select object sharing function), the object sharing process begins. At this time, HMD2_10B on the rehabilitation instructor's side is set as the host computer, and HMD1_10A on the rehabilitation practitioner 1's side is set as the client computer. Alternatively, the HMD that starts the object sharing function application first may be set as the host, and the HMD that starts later may be set as the client.

[0091] (ii) Steps S1102A and S1102B HMD1_10A and HMD2_10B each perform initialization processing. This initialization processing includes setting parameters to the information entered on the rehabilitation settings screen 400 (see Figure 4), setting the origin of each HMD's three-dimensional coordinate system to a predetermined position on each HMD (for example, the midpoint on the display screen of the display device 105), and sharing the amount of misalignment between the three-dimensional coordinates of each HMD. The amount of misalignment between the three-dimensional coordinates (the degree of misalignment (difference) between the three-dimensional coordinates of the HMDs) can be determined, as described above, for example, by HMD2_10B (host) acquiring the position information of HMD1_10A (client) based on the image captured by the camera device 104 (using tracking information) and transmitting it to HMD1_10A, and by HMD1_10A acquiring the relative position information from HMD2_10B.

[0092] (iii) Steps S1103A and S1103B The host, HMD2_10B, retrieves object data from the memory device 102 and renders (places) each object (3DCG) at a predetermined position in the three-dimensional space of HMD2. HMD2_10B also transmits information about the placement position (three-dimensional coordinates) of each object in the three-dimensional space of HMD2 to HMD1_10A.

[0093] HMD1_10A calculates the position of each object in the three-dimensional coordinates of HMD1 by reflecting the amount of displacement between the three-dimensional coordinates obtained in step S1102A in the three-dimensional coordinate information of each object obtained from HMD2_10B, and then draws (places) them. Alternatively, HMD2_10B may calculate the placement position of each object in HMD1_10A and provide it to HMD1_10A.

[0094] (iv) Step S1104A HMD1_10A detects the movement of a body part (a designated body part: for example, the right hand) of the rehabilitation practitioner 1 using camera tracking. Similarly, HMD2_10B also detects the movement of the above body part (acquires position information) using camera tracking, and draws (places) object data corresponding to that body part (for example, 3DCG data of a hand) at the detected position, and transmits the position information of the above body part (position information in the three-dimensional coordinates of HMD2) to HMD1_10A. Then, based on the amount of discrepancy between the position information of the above body part (hand) acquired from HMD2_10B and the above three-dimensional coordinates, HMD1_10A calculates the position of the above body part (hand) in the three-dimensional coordinates of HMD1, and draws (places) the corresponding object data (3DCG of a hand) at the calculated position (coordinates).

[0095] (v) Step S1105A HMD1_10A determines whether there has been an interaction (pinching motion) with the object to be manipulated (bird image 24). Here, whether or not an interaction has occurred is determined by, for example, if the operation to be performed is a pinching motion, by HMD1_10A using camera tracking to determine whether the body part that performed the operation is a specified body part (e.g., the right hand) and whether the distance between the specified fingers is smaller than the set pinching width threshold. If there has been an interaction with the object to be manipulated (Yes in step S1105A), the process proceeds to step S1106A. If there has been no interaction with the object to be manipulated (No in step S1105A: for example, if the operation is not performed with the right hand or if the pinching width due to the pinching motion is larger than the set pinching width threshold), HMD1_10A waits until there is an interaction with the object to be manipulated.

[0096] Although not shown in Figure 11A, as described above (see Figure 5), if the operation is not performed by a designated body part, the operation on the target (bird image 24) will be invalidated. Also, if the pinch width performed by rehabilitation practitioner 1 is greater than the set pinch width threshold, after waiting for a predetermined time, the operation (rehabilitation action) on the target (bird image 24) will be judged to have failed.

[0097] (vi) Step S1106A HMD1_10A determines whether it owns the object to be manipulated (bird image 24). If it owns the object to be manipulated (Yes in step S1106), the process proceeds to step S1108A. If it does not own the object to be manipulated (No in step S1106), the process proceeds to step S1107A.

[0098] (vii) Step S1107A HMD1_10A (client) sends an ownership reference request to HMD2_10B (host) so that HMD2_10B can refer to the ownership of the object being operated on (whether ownership belongs to the host or another client).

[0099] (viii) Step S1104B HMD2_10B waits for ownership reference requests from HMD1_10A and receives them when they are sent.

[0100] (ix) Step S1105B HMD2_10B checks whether it owns the object being operated on. If it owns the object (Yes in step S1105B), the process proceeds to step S1106B. On the other hand, if it does not own the object (No in step S1105B), the process proceeds to step S1107B.

[0101] (x) Step S1106B HMD2_10B transfers ownership of the object being manipulated to HMD1_10A (the client). As a result, HMD1_10A, as the client, can reflect the interaction (pinching motion) of the rehabilitation practitioner 1 with the object being manipulated (bird image 24) (pinching bird image 24).

[0102] (xi) Step S1107B HMD2_10B identifies an HMD other than HMD1_10A that owns the object being operated on and acquires ownership from it. Then, HMD2_10B transfers the acquired ownership to HMD1_10A (step S1106B).

[0103] (xii) Steps S1108A and S1109A HMD1_10A acquires operation information (pinching motion) from camera tracking and transmits the acquired operation information to HMD2_10B (step S1108A). The acquisition and transmission of said operation information continues until the interaction (pinching motion and movement) with the object being operated (bird image 24) is completed. Interaction with the object being operated is determined to be completed, for example, when rehabilitation practitioner 1 places the bird image 24 into the cage 25, or when the bird image 24 is dropped because the pinching motion cannot be continued while carrying it to the cage 25 (for example, if the distance between the fingers used for the pinching motion becomes greater than the pinching width threshold).

[0104] (xiii) Steps S1108B and S1109B HMD2_10B (host) receives operation information sent from HMD1_10A (client) (step S1108B). HMD2_10B may also receive an operation completion notification from HMD1_10A when the interaction with the target of the operation is finished.

[0105] Then, HMD2_10B reflects the acquired operation information onto the target object (bird image 24) (step S1109B). That is, when HMD2_10B receives operation information from HMD1_10A that "a pinching action was performed", it moves the target fingers in the hand object (hand image 21). Also, when HMD2_10B receives operation information from HMD1_10A that "the pinching action was successful and the target object (bird image 24) was lifted", it moves the target fingers in the hand image 21 and also moves the position coordinates of the hand image 21 and the bird image 24 by the amount of displacement caused by the rehabilitation practitioner 1. At this time, HMD2_10B also acquires the position information of the hand image 21 and the position information of the target bird image 24 in its own three-dimensional coordinate system. The position information of the object to be manipulated (bird image 24) may be set to be the same as the position information of the hand image 21, as described above, or it may be set to be shifted by a predetermined distance from the position information of the hand image 21.

[0106] (xiv) Step S1110B HMD2_10B (host) sends the position information of the objects (the hand image 21 and the bird image 24 above) to HMD1_10A (client). HMD1_10A converts the object's position information obtained from HMD2_10B into position information in HMD1_10A's three-dimensional coordinate system and draws the object (step 1103A). Then, the subsequent processing (processing from step S1104A to step S1109A) continues.

[0107] (xv) Step S1110A Once the interaction with the target object is complete, HMD1_10A (client) returns ownership of the target object (the bird image 24 for this session) to HMD2_10B (host).

[0108] (xvi) Step S1111B HMD2_10B acquires ownership of the object that was being manipulated.

[0109] (xvii) Step S1111A HMD1_10A determines whether all operations and processing for all targets in the rehabilitation session have been completed. For example, it determines whether the pinching action → transporting to the cage 25 (including failures) has been completed for all 10 objects (bird image 24). If processing for all targets has been completed (Yes in step S111A), the process proceeds to step S1113A. If there are still unprocessed targets remaining (No in step S111A), the process proceeds to step S1112A.

[0110] (xviii) Step S1113A HMD1_10A sends a rehabilitation completion notification to HMD2_10B indicating that the rehabilitation session for that time has ended.

[0111] (xix) Step S1112B HMD2_10B determines whether it has received a rehabilitation completion notification from HMD1_10A within a predetermined time after acquiring ownership in step S1111B. If it receives the rehabilitation completion notification within the predetermined time (if it answers Yes in step S1112B), the object sharing process ends. If HMD2_10B does not receive the rehabilitation completion notification within the predetermined time and determines that the number of operations 407 set on the rehabilitation settings screen 400 has not reached the set value (if it answers Yes in step S1112B), the process proceeds to step S1113B. Note that each setting information on the rehabilitation settings screen 400 is sent from HMD1_10A to HMD2_10B at the start of rehabilitation, and each setting information is shared between both HMDs.

[0112] (xx) Steps S1112A and S1113B HMD1_10A and HMD2_10B change the target object to the next object (the next bird image 24). Then, in HMD1_10A, the process moves to step 1103A. Also, in HMD2_10B, the process moves to step S1103B. <When rehabilitation practitioner 1 and rehabilitation instructor are in different locations, the object

[0113] Processing > The object sharing process (basic form) described in Figures 11A and 11B describes the case where the rehabilitation practitioner 1 and the rehabilitation instructor are physically in the same space S. However, the rehabilitation practitioner 1 and the rehabilitation instructor do not necessarily need to be in the same physical space. The technical concept of the object sharing process according to this embodiment is also applicable when the two are in different locations (object sharing process (remote form)).

[0114] In this case, HMD1_10A and HMD2_10B each have common object data in their respective memory devices 102 (for example, a hand image 21, a table image 23, a bird image 24, a cage image 25, a rule board image 26, a results board image 27, and a hand image of the rehabilitation instructor (not shown in Figure 1)), and each may also have its own unique object data. For example, HMD1_10A of rehabilitation practitioner 1 may further have object data of the rehabilitation instructor's body (excluding hands) (instructor image 22), and HMD2_10B of rehabilitation instructor may further have object data showing the body of rehabilitation practitioner 1 (excluding hands) (practitioner image).

[0115] Furthermore, in the object sharing process (basic form) described above, a host HMD and a client HMD were configured. Even if HMD1_10A and HMD2_10B exist remotely, one can be configured as the host and the other as the client, and the object sharing process (remote form) can be executed using the same process as described in Figures 11A and 11B. However, if the rehabilitation practitioner 1 and the rehabilitation instructor are physically separated, their HMDs will communicate using the internet, which may cause inconveniences such as processing delays.

[0116] Therefore, at the start of rehabilitation, HMD1_10A and HMD2_10B communicate with each other and calculate the difference (amount of deviation) between their three-dimensional coordinates. At this time, HMD1_10A transmits information about the position of rehabilitation practitioner 1 (standing position or sitting position) (i.e., the origin position information of HMD1's three-dimensional coordinates) to HMD2_10B. On the other hand, HMD2_10B transmits information about the position of the rehabilitation instructor (standing position) (i.e., the origin position information of HMD2's three-dimensional coordinates) to HMD1_10A. This makes it possible for each HMD to calculate the amount of deviation (distance and vector) between its three-dimensional coordinates and those of the other HMD.

[0117] Furthermore, the display position of each object data is determined by the other HMD, which uses the three-dimensional coordinate position of one of the HMDs as a reference to determine the display position of each object data in its own three-dimensional coordinate system.

[0118] Furthermore, if the rehabilitation practitioner 1 moves their hand or interacts with the object being manipulated (bird image 24) through rehabilitation exercises (pinching motion), HMD1_10A transmits its own three-dimensional coordinate position information (changed position information) to HMD2_10B. Also, if the rehabilitation practitioner 1 is unable to continue the pinching motion or is able to carry the object being manipulated (bird image 24) to the cage 25, the object will fall. In this case, HMD1_10A transmits operation information to HMD2_10B indicating that the pinching motion has ended (information indicating that the distance between the target fingers has become greater than the pinching width threshold). Then, based on the position information and operation information obtained from HMD1_10A, HMD2_10B calculates the position information of each object (hand image 21 and bird image 24) in its own three-dimensional coordinates, determines the display position of each object, and displays them on the display screen.

[0119] Furthermore, when the rehabilitation instructor moves their hand, HMD2_10B detects the position of that hand using camera tracking, displays a hand object (an image of the instructor's hand) at that position, and transmits the instructor's hand position information (the three-dimensional coordinates of HMD2) to HMD1_10A. HMD1_10A converts the instructor's hand position information into position information in the three-dimensional coordinates of HMD1 and displays the instructor's hand image on the display screen.

[0120] The above operations in HMD1_10A and HMD2_10B are repeated until the number of operations 407 set in the rehabilitation settings screen 400 is achieved.

[0121] <Other features> (i) Conversation with a virtual leader In the scenario shown in Figure 1 (Figure 5), when rehabilitation is performed, the conversation between the virtual instructor image (instructor object) 22 and the rehabilitation practitioner 1 may be realized using AI technology.

[0122] For example, if rehabilitation practitioner 1 speaks to the instructor image 22 displayed on the screen ("I want to use my right hand to play games once my right hand gets better"), the instructor object may respond by generating a reply via AI ("Yes, I'm sure it will get better soon!") and speaking it aloud. Also, when a predetermined amount of time (for example, 30 minutes) has elapsed since the start of rehabilitation, the instructor object may make a verbal announcement ("Let's take a short break" or "Rotate your shoulders and relax," etc.). Furthermore, the AI's reaction to the rehabilitation practitioner may be shown not only by voice but also by avatar actions. For example, visual feedback may be presented to rehabilitation practitioner 1 by displaying motions that improve rehabilitation movements on the display device 105 via the avatar.

[0123] (ii) Object-specific processing in object sharing scenarios When rehabilitation practitioner 1 and rehabilitation instructor share an object in the same space (processing in Figures 11A and 11B), the execution of processing specific to the shared rehabilitation object is entirely determined by the host (HMD2_10B on the rehabilitation instructor's side). For example, if rehabilitation practitioner 1 pinches bird image 24, HMD2_10B detects this action or, based on operation information from HMD1_10A, controls the generation of animation for the pinching action (visualizing the object's movement) or the playback of sound effects at that time.

[0124] On the other hand, if the rehabilitation practitioner 1 and the rehabilitation instructor are not in the same space and are separated from each other, the concepts of host and client are not used, so the animation generation control and sound effect playback control described above can be performed on HMD1_10A. In this case, information on the movement (changing position information) of the target object (bird image 24), sound effects, and their output timing can be transmitted from HMD1_10A to HMD2_10B, and HMD2_10B can reflect this information on the target object.

[0125] <Summary> (i) In this embodiment, the HMD 10, which corresponds to a rehabilitation support system (rehabilitation support device), performs the following processes: detecting the movement of a body part (e.g., hand) of the rehabilitation practitioner 1 (using camera tracking); determining whether the body part is a designated body part (e.g., rehabilitation movement with the right thumb and index finger); and determining whether to enable interaction with the object image (bird image 24) to be operated by the rehabilitation practitioner 1 based on the determination of whether it is a designated body part. For example, if it is specified that the operation should be performed with the right hand, interaction with the object to be operated with the left hand (bird image 24) is disabled (interaction is not possible). In this way, rehabilitation can be performed without restraining any part of the rehabilitation practitioner 1's body, thus preventing excessive stress on the rehabilitation practitioner.

[0126] (ii) In this embodiment, as shown in Figure 4, it is possible to set a variable threshold (for example, a pinch width threshold in the case of pinch motion rehabilitation) according to the condition of the rehabilitation practitioner 1. This pinch width threshold is a threshold for determining whether or not the rehabilitation motion performed by interaction was successful. Specifically, the HMD 10 detects the movement distance (pinch width) of the designated body part of the rehabilitation practitioner 1 (right hand, thumb and index finger), and determines that the rehabilitation motion of the rehabilitation practitioner 1 was successful (that they were able to pinch the bird image 24 that was the target of the operation) when this movement distance (pinch width) is smaller than the above variable threshold (pinch width threshold). In this way, since the threshold for determining the success or failure of the rehabilitation motion is variable, rehabilitation can be effectively performed according to the level of disability of the rehabilitation practitioner 1.

[0127] Furthermore, the HMD10 evaluates the rehabilitation actions performed by the rehabilitation practitioner 1 based on the success or failure of the rehabilitation actions performed on all pre-set target object images (for example, 10 bird images 24), and outputs the evaluation results to the display device. This allows rehabilitation to be performed in a game-like manner without becoming boring, and the effectiveness of rehabilitation can be improved efficiently.

[0128] Furthermore, a single rehabilitation support session includes rehabilitation actions for multiple object images. At this time, the HMD10 performs a process to evaluate the overall effectiveness of the rehabilitation support based on the progress of evaluation results from multiple rehabilitation support sessions. By performing this evaluation based on progress, it becomes possible to verify the content of the rehabilitation for rehabilitation provider 1 (patient) and adjust or change it to the optimal rehabilitation program.

[0129] (iii) The HMD 10 displays an object image (right hand image 21) representing a designated body part (for example, the right hand) at the location of the rehabilitation practitioner 1. When the rehabilitation practitioner 1 moves the designated body part (right hand), the HMD 10 makes the display of the object image (right hand image 21) of the designated body part (right hand) follow this movement. This allows the rehabilitation practitioner 1 to recognize that rehabilitation support is in progress, thereby encouraging them to engage in rehabilitation movements more seriously.

[0130] Furthermore, the HMD 10 acquires images of the real world (images of the rehabilitation room) using the camera device 104, and overlays multiple object images (for example, a desk image 23 and a basket image 25) onto the real world image at predetermined, fixed display positions and displays them on the display device. This makes it possible to easily conduct rehabilitation without actually having to prepare a desk or basket.

[0131] (iv) This embodiment proposes a function in which the rehabilitation practitioner 1 and the rehabilitation instructor each wear an HMD and share rehabilitation object images (3DCG). There are two ways in which these object images are shared: the first is when they are shared in the same physical space, and the second is when they are shared while in separate locations.

[0132] In either embodiment, the HMD1_10A worn by the rehabilitation practitioner 1 and the HMD2_10B worn by the rehabilitation instructor are configured to communicate with each other via the network 1000. Furthermore, both HMD1_10A and HMD2_10B hold common object image data in the memory device 102. Because HMD1_10A and HMD2_10B exist in different locations (even if they are in the same physical space), the three-dimensional coordinate system (first coordinate system) of HMD1_10A and the three-dimensional coordinate system (second coordinate system) of HMD2_10B are different coordinate systems (their origins are different). In this situation, at least one of HMD1_10A or HMD2_10B calculates the position in which to display the common object image in the first or second coordinate system based on the amount of displacement between the first and second coordinate systems (they may calculate this together, or one may calculate it and provide it to the other HMD). Then, at least one of HMD1_10A or HMD2_10B performs a process to align the display positions of the common object image in the first coordinate system and the second coordinate system (they may align each other, or one may align based on the other's information). As a result, there is no discrepancy between the display position of the common object image displayed in the first coordinate system and the display position of the same common object image displayed in the second coordinate system, so that rehabilitation practitioner 1 and rehabilitation instructor can view the same common object image without positional discrepancies (there is no inconsistency in the display of the common object when viewed from different locations) via HMD1_10A or HMD2_10B.

[0133] For example, if the common object is the object image to be manipulated (bird image 24), HMD1_10A or HMD2_10B can determine whether the rehabilitation practitioner 1 has interacted with the object image to be manipulated based on the images captured by their respective camera devices, and reflect the content of the interaction in the display of the object image to be manipulated (bird image 24). For example, if the rehabilitation practitioner 1 and the rehabilitation instructor share a common object in the same physical space (such as the same room), or if the rehabilitation practitioner 1 and the rehabilitation instructor are in separate locations but share a common object, HMD1_10A detects the gripping motion of the rehabilitation practitioner 1 via camera tracking and requests a transfer of ownership from HMD2_10B. Once ownership is transferred from HMD2_10B, HMD1_10A uses the camera device 104 to obtain location information of the rehabilitation practitioner 1's body parts and transmits that location information to HMD2_10B. Then, each HMD, or either HMD, calculates the difference between the position coordinate system of HMD1 (first coordinate system) and the position coordinate system of HMD2 (second coordinate system) and draws the object to be manipulated (rehabilitation object: bird image 24). When the rehabilitation operation is completed, HMD1_10A returns the temporarily acquired ownership to HMD2_10B. Alternatively, in another form, for example, HMD2_10B may detect interaction with the bird image 24 using an image of the designated body part (right hand) of the rehabilitation practitioner 1 acquired by its own camera device 104, and reflect that position information in the display position of the image of the designated body part on HMD2_10B. Then, HMD2_10B provides the position information in the second coordinate system to HMD1_10A, and HMD1_10A converts (corrects) the position information in the second coordinate system to position information in the first coordinate system. This makes it possible to display the movement of the specified body part (right hand) on the display device 105 without any inconsistencies, even in the HMD1_10A.

[0134] (v) In this embodiment, HMD2_10B can be set as the host and HMD1_10A as the client (or vice versa). In this case, HMD2_10B has ownership of the object image to be manipulated (bird image 24), including the authority to determine the reference of the display position coordinates and whether or not to perform an interaction by the rehabilitation practitioner 1. This ownership can be temporarily transferred from HMD2_10B to HMD1_10B. If HMD1_10A does not have ownership, it requests HMD2_10B to transfer ownership, and after acquiring ownership, the rehabilitation practitioner 1 can interact with the object image to be manipulated (bird image 24).

[0135] In this scenario, HMD2_10B functions as the host computer (host) that temporarily owns the object image (bird image 24) to be manipulated, and HMD1_10A functions as the client computer (client) that secondarily acquires ownership. If HMD1_10A temporarily acquires ownership from HMD2_10, it returns ownership to HMD2_10B when the operation on the object to be manipulated is completed.

[0136] Let's assume that, in this case where the host and client are specified, the rehabilitation practitioner 1 and the rehabilitation instructor are physically located in different places within the same space (same room). In this case, when the object to be manipulated (bird image 24) is manipulated (pinched) by the rehabilitation practitioner 1, HMD1_10A detects the pinching motion of the rehabilitation practitioner 1 via camera tracking and requests a transfer of ownership from HMD2_10B. Once ownership is transferred from HMD2_10B, HMD1_10A uses the camera device 104 to acquire positional information of the rehabilitation practitioner 1's body parts and transmits that positional information to HMD2_10B. Then, each HMD, or either HMD, calculates the difference between the positional coordinate system of HMD1 (first coordinate system) and the positional coordinate system of HMD2 (second coordinate system) and draws the object to be manipulated (rehabilitation object: bird image 24). When the rehabilitation movement is completed, HMD1_10A returns the temporarily acquired ownership to HMD2_10B. Alternatively, HMD2_10B uses its camera device to capture the movement (interaction) of the rehabilitation practitioner's body part (e.g., right hand) to the object image (bird image 24), detects the position information of the body part in the three-dimensional coordinate system of HMD2_10B using camera tracking technology based on the captured image, and transmits this to HMD1_10A. Meanwhile, HMD1_10A receives the position information of the body part (right hand) in the three-dimensional coordinate system of HMD2_10B via network 1000, and converts the position information of HMD2_10B in the three-dimensional coordinate system to the position information of HMD1_10A in the three-dimensional coordinate system based on the difference in the two coordinate systems. Furthermore, HMD1_10A may display the object image (bird image 24) to be manipulated on the display device 105 based on the position information in its own three-dimensional coordinates obtained through the transformation.

[0137] (vi) The functions of this embodiment can also be realized by software program code. In this case, a storage medium on which the program code is recorded is provided to a system or device, and the computer (or CPU or MPU) of that system or device reads the program code stored on the storage medium. In this case, the program code read from the storage medium itself realizes the functions of the embodiment described above, and the program code itself and the storage medium on which it is stored constitute the present disclosure. Examples of storage media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0138] Furthermore, based on the instructions in the program code, the operating system (OS) running on the computer may perform some or all of the actual processing, thereby realizing the functions of the embodiment described above. In addition, after the program code read from the storage medium is written to the computer's memory, the computer's CPU may perform some or all of the actual processing based on the instructions in the program code, thereby realizing the functions of the embodiment described above.

[0139] Furthermore, the program code for the software that realizes the functions of this embodiment may be distributed via a network and stored in a storage means such as a hard disk or memory of the system or device, or in a storage medium such as a CD-RW or CD-R, so that when in use the system or device's computer (or CPU or MPU) reads and executes the program code stored in the storage means or storage medium.

[0140] The processes and technologies described herein are not inherently related to any specific device and can be implemented by combining the components. Various types of general-purpose devices can also be added. Dedicated devices may be constructed to perform the functions of this embodiment and each example. Furthermore, various functions can be formed by appropriately combining the multiple components disclosed in this embodiment. For example, some components may be removed from all the components shown in the embodiment and each example, or components from different embodiments may be appropriately combined.

[0141] This disclosure describes specific embodiments, which are for illustrative purposes (to understand the technology of this disclosure) and not for limitation in any respect. A person with ordinary skill in the art will understand that there are many combinations of hardware, software, and firmware suitable for implementing the technology of this disclosure. For example, the described software can be implemented in a wide range of programming or scripting languages, such as assembler, C / C++, Perl, Shell, PHP, and Java®.

[0142] Furthermore, in the embodiments described above, the control lines and information lines shown are those deemed necessary for illustrative purposes, and not all control lines and information lines are necessarily shown in the actual product. All components may be interconnected.

[0143] In addition, any person with ordinary skill in the art can see from the consideration of these embodiments and examples that other implementations of this disclosure may be apparent. The specification and examples are typical, and the scope and spirit of the art of this disclosure are shown in the subsequent claims. [Explanation of symbols]

[0144] 1. Rehabilitation provider 10 Head-mounted displays 21 Hand image (3DCG) 22. Leader image (3DCG) 23 Table Images (3DCG) 24 Bird Images (3DCG) 25 basket images (3DCG) 26. Rule board image (3DCG) 27 Results bulletin board image (3DCG) 101 Control device 1011 Body part detection unit 1012 Object Drawing Section 1013 Object position calculation unit 1014 Interaction Disabling Unit 1015 Judgment and Evaluation Department 102 Storage Devices 103 Input Devices 104 Camera Devices 105 Display Devices 106 Communication devices

Claims

1. A rehabilitation support system that uses object images displayed in a virtual space to support rehabilitation performed by rehabilitation practitioners, A storage device that holds data of the object and a program to support the rehabilitation, A display device that displays the aforementioned object image on a screen, The system includes a control device that reads the program from the storage device and controls the display of the object image on the screen to support the rehabilitation, The control device is The process of detecting the movement of the body parts of the person performing the rehabilitation, A process to determine whether the body part is a designated body part specified for rehabilitation, Based on the determination of whether or not it is the designated body part, a process is performed to determine whether or not to enable interaction with the object image to be manipulated by the rehabilitation practitioner, A rehabilitation support system that performs [the following actions].

2. In claim 1, The control device is A process for receiving a variable threshold set according to the condition of the rehabilitation practitioner, which is used to determine whether the rehabilitation action performed by the interaction was successful or not. A process for detecting the movement distance of the designated body part of the rehabilitation practitioner, A process that compares the movement distance of the designated body part with the variable threshold and determines whether the rehabilitation movement is successful or not, A rehabilitation support system that performs [the following actions].

3. In claim 1, Furthermore, equipped with a camera device, The control device is a rehabilitation support system that detects the movement of the body part using tracking technology with images captured by the camera device.

4. In claim 3, The control device is a rehabilitation support system that displays an object image representing the designated body part at the location of the designated body part and causes the object image of the designated body part to track the movement of the designated body part.

5. In claim 3, The control device is a rehabilitation support system that displays an image of the real space captured by the camera device on the display device, and also displays a plurality of object images on the display device, positioned at predetermined locations in the image of the real space.

6. In claim 2, The control device is a rehabilitation support system that evaluates the rehabilitation actions performed by the rehabilitation practitioner based on the success or failure results of the rehabilitation actions for all pre-set object images of the targets of the operation, and outputs the evaluation results to the display device.

7. In claim 6, A single rehabilitation support session includes the rehabilitation actions performed on multiple object images of the target of the operation. The control device is a rehabilitation support system that performs a process to evaluate the overall effect of the rehabilitation support based on the progress of the evaluation results of multiple rehabilitation support sessions.

8. In claim 2, The control device is A process of displaying a settings screen on the display device for setting multiple conditions for the rehabilitation, including the change threshold, A process to determine the success or failure of the rehabilitation action based on the change threshold entered via the settings screen, A rehabilitation support system that performs [the following actions].

9. In claim 2, The rehabilitation support system is a rehabilitation gaze system configured as a head-mounted display comprising the memory device, the display device, the control device, and the camera device.

10. The rehabilitation practitioner wears a first head-mounted display, which includes a first camera device, A rehabilitation instructor will wear a second head-mounted display, which includes a second camera device, The first head-mounted display and the second head-mounted display are configured to communicate with each other via a network. The first head-mounted display and the second head-mounted display each possess common object image data. The first coordinate system, which is a three-dimensional coordinate system used by the first head-mounted display, and the second coordinate system, which is a three-dimensional coordinate system used by the second head-mounted display, are coordinate systems with different origin positions. At least one of the first head-mounted display or the second head-mounted display calculates the position for displaying the common object image in the first coordinate system or the second coordinate system based on the amount of displacement between the first coordinate system and the second coordinate system, and performs a process to align the display positions of the common object image in the first coordinate system and the second coordinate system. A rehabilitation support system configured so that the rehabilitation provider and the rehabilitation instructor can view the same common object image without any positional misalignment through the first head-mounted display or the second head-mounted display.

11. In claim 10, The aforementioned common object image includes an image of the object that the rehabilitation practitioner interacts with and manipulates. A rehabilitation support system comprising: the first head-mounted display or the second head-mounted display, which determines, based on the image captured by the first camera device or the second camera device, whether the rehabilitation practitioner has interacted with the object image of the target of manipulation, and reflects the content of the interaction in the display of the object image of the target of manipulation.

12. In claim 10, Either the first head-mounted display or the second head-mounted display has ownership rights, including the authority to determine the reference of the display position coordinates of the object image to be operated on and whether or not the rehabilitation provider has performed an interaction with it. A rehabilitation support system comprising: a process for the first head-mounted display to request the second head-mounted display to transfer ownership if it does not possess the ownership rights, and enabling the rehabilitation practitioner to interact with the object image of the object to be manipulated after acquiring the ownership rights.

13. In claim 12, The second head-mounted display temporarily functions as the host computer that owns the object image to be manipulated, The first head-mounted display functions as a client computer that secondarily acquires ownership, A rehabilitation support system in which, when the first head-mounted display acquires ownership from the second head-mounted display, the first head-mounted display returns ownership to the second head-mounted display when the operation on the target object is completed.

14. In claim 13, The rehabilitation provider and the rehabilitation instructor are located in different places within the same physical space. The first head-mounted display whose ownership has been transferred is: The first camera device processes the movement of the body parts of the rehabilitation practitioner on the object image being manipulated, A process for detecting the positional information of the body part in the first coordinate system using camera tracking technology based on the captured image, The process of transmitting the position information of the body part to the second head-mounted display via the network is performed. The second head-mounted display is A process of receiving positional information of the body part in the first coordinate system from the first head-mounted display via the network, A process to convert the position information of the first coordinate system into position information of the second coordinate system based on the amount of displacement, Based on the position information of the second coordinate system obtained by the transformation, the process of displaying the object image to be operated on on the display device, A rehabilitation support system that performs [the following actions].

Images