Game-based system and methods for rehabilitation

Abstract

A novel system and method for rehabilitation, using a computer game, comprising a computer, a camera and a display screen. The rehabilitation method runs on the computer, on which the game is also played. The camera monitors movements and positions, while the subject uses the limb or fingers to perform monitored therapy-related movements or exert force. When the patient reaches a predefined goal, the relevant feature of the computerized game is triggered, just like the click or movement of a computer mouse or joystick. The method is useable at the clinic or at home and is suitable for any computer game preferred by the patient. It requires no wearables, may use common occupational therapy accessories, adaptive to patient capabilities, and adjustable for motion in all six degrees of freedom. These advantages make therapy sessions cost effective, simple to operate, flexible, engaging and suitable for multiple ages and disabilities.

Description

[0001] GAME-BASED SYSTEM AND METHODS FOR REHABILITATION

[0002] FIELD OF THE INVENTION

[0003] The present disclosure describes technology related to the field of limb rehabilitation, especially as used in a clinical or home setting, using a PC, tablet or smartphone to play computer games.

[0004] BACKGROUND

[0005] Gamification is considered to be an important aspect for encouraging patients to adhere to tedious rehabilitation exercise requirements. In many rehabilitation sessions, games are included as an integral part of the patient's effort and have proven effective in creating motivation and achieving better outcomes. Numerous rehabilitation systems currently require special cameras and / or wearables (such as gloves or head mounts) that track limb movement anEd process it in relation to specific rehabilitation-oriented games. Other solutions may use wearable virtual reality headsets. Examples for existing systems are Microsoft Kinect of Microsoft, Redmond, USA and the Ultraleap Motion Controller of Ultraleap Ltd., Bristol, UK. Examples for companies that use the above-mentioned imaging systems, at least in some of their products, are Mindmaze of Lausanne, Switzerland, NeuroTechR3 of Newark NJ, USA and Rehametrics of Valencia, Spain.

[0006] Existing systems require development of designated rehabilitation games and the use of special cameras and / or wearables. In the case of systems that utilize gloves or special software applications, these can be relatively expensive and may be limited in their ability to identify and measure rotational motions and / or patient hand's adductions and abductions movements. Some examples for wearable rehabilitation glove products are Vrehab-M3 of Vendra Medical (Arcueil, France) and C 12 robotic hand rehabilitation gloves of Syrebo (Shanghai, China). Moreover, some of them may not easily integrate many of the common household Occupational Therapy (OT) accessories for rehabilitation sessions, such as for instance rubber bands and cloth pins, which are professionally considered important tools for achieving therapy goals.

[0007] Examples of some previously described methods and devices for rehabilitation, can be found in the following patent applications or patents: US20170209737A1 to T. Tadi et al., for “Upper limb rehabilitation system”, US20090098519A1 to J. Byerly, for "Device and method for employment of video games to provide physical and occupational therapy and measuring and monitoring motor movements and cognitive stimulation and rehabilitation" , US20170151500A9 to A. Domanski et al., for " Personal digital trainer for physiotheraputic and rehabilitative video games", US20220005577A1 to R. Komatireddy for Systems, apparatus and methods for non-invasive motion tracking to augment patient administered physical rehabilitation, US11317975B2 to S. Mason et al. for "Method and system for treating patients via telemedicine using sensor data from rehabilitation or exercise equipment", and US 6,712,692 to Basson S.H. et al, for "Using existing videogames for physical training and rehabilitation".

[0008] The disclosures of each of the publications mentioned in this section and in other sections of the specification, are hereby incorporated by reference, each in its entirety.

[0009] SUMMARY

[0010] The present disclosure provides a novel method and system that overcome at least some of the disadvantages of prior art systems and methods. The disclosure describes an innovative rehabilitation method that does not require special cameras, design of suitable computer games or any wearables that may limit patient limb or finger movement or the ability to measure movements or forces. The method can integrate simple Occupational Therapy (OT) accessories such as, but not limited to, rubber bands, clothespins, sponge balls or a spring. As described in International Published Application WO2023135602A1 for “System and Method for Hand Rehabilitation”, commonly owned by the present applicant, the method also does not require any force sensors to measure the forces exerted by the patient. Hence the proposed method is easy to use, cost effective and can readily be used at the clinic or at a home setting for a wide range of disabilities and for population of all ages.

[0011] Previously related disclosures for using gesture recognition technology in rehabilitation based on existing videogames, such as described in US 6,712,692 to Basson S.H. et al, for "Using existing videogames for physical training and rehabilitation "do not include an option to flexibly and timely pre-plan required movements in all 6 degrees of freedom (DOF) according to therapy needs, nor provide detection of force or automatic adjustment of the difficulty level based on the patient's capabilities. Furthermore, the use of rehabilitation tools may require detection of these tools by optical coding, motion sensors or other form of communication. In some cases, such as US9457256B2 to T. Aragones for “Method and system for automated personal training that includes training programs”, wearable sensors are required to track motion.

[0012] Hence, some of these previously available systems require special equipment and / or wearables, including wearable fiducial markers or optical markers on the subject's limbs or on the rehabilitation tools. The previous equipment may be expensive and complicated to install and use at home, require continuous mentoring and guidance, as well as planning and programming designated computer games to which the patient should adjust. Such solutions may also be limited in their ability to provide training in all 6 DOF, to measure force without using any wearables, and to automatically adjust to patient needs.

[0013] According to one implementation of the methods and systems of the present disclosure, the camera of a personal computer, a tablet or even a smartphone optically detects the movements of the limb undergoing rehabilitation. One limb commonly needing recuperation, typically after an injury though not limited thereto, is the hand or more particularly the wrist and / or the fingers of the hand. These body parts are generally used in the present disclosure, as an example to illustrate the methods and systems therein, though it is to be understood that the application is not intended to be limited to these examples, but that the described methods and systems are applicable to any other limb of the body which requires rehabilitation. According to one feature of the present disclosure, the hand motions needed by the Occupational Therapist (OT) to perform the rehabilitation process being followed, are intended to simulate the motion or operation of a device for activating a feature of a computer game, which, in its normal use, would require a device such as a joystick or a mouse or a gaming controller to provide the actions necessary to play the game. The required movement of the limb can thus be termed a quasi-activation device. The feature can be taken from any computer game, such that, differently from most existing systems and methods, there is no need to develop special computer games designated for the rehabilitation process, and the patients can choose games available on the web or any other games according to their preferences. The ability to use a computer game familiar to the subject may contribute to the success of the rehabilitation procedure, since the subject may use a favorite game enjoyed by him / her, and will therefore persevere with the exercises required by the OT. In practice, the movements of the limb being imaged by the camera device are processed and converted by means of a motion-tracking application into motion and vector data. Several of such applications are available commercially, such as, but not limited to, MediaPipe of Google, Mountain View, CA, USA. The digital data is then compared in the system controller with the required levels of motion and the vector position values required to be regarded as successful achievement of the exercise goals, and if successful achievement is found, the system recognizes the imaged movement as if a real activation device of the computer game had been successfully triggered. The digital data may also be made available for display on the screen of the computer device, together with the display of the video game as it proceeds, so that the relationship can be displayed between the hand or finger movement of the subject under rehabilitation, and the on-screen result arising from the actuation of the game’s quasi-activator device.

[0014] The simulation of the activating device of the computer game can be performed using the hand or its finger parts alone, or together with a physiotherapeutic accessory, such as a sponge ball, or a rubber band or a clothes pin or the like, used to require of the subject to perform the action against a force, thereby assisting the rehabilitation process, and adding more information to the results of the procedure regarding the progress of the rehabilitation course. The use of the accessories does not require any markers / sensors (optical or other) on the subject's hand or on the accessories themselves.

[0015] The data from the training sessions performed by the patient may be stored in a database for the later use by OT personnel such as for reviewing the rehabilitation procedure, or for suggesting possible changes in the training program, or for statistical analysis needs, or for any other appropriate purpose related to the rehabilitation process. The method also enables telerehabilitation which may include a session attended by the subject and the remotely located therapist, or sessions for just the subject to enable performance of the required exercises by him / her self at the clinic or in a home setting. The system can also use the data to automatically adapt the level of difficulty (i.e. range of motion or force) in accordance with patient impairment and capabilities.

[0016] The system may also utilize machine learning and artificial intelligence capabilities to make the system adaptive to the success of the subject’s efforts. These capabilities inform the system of the level of the subject's capabilities, according to the procedures being exercised, and the system can then adjust the levels of difficulty of the exercise in order to advance in the rehabilitation program. Thus, if the system discerns that the subject is performing the exercise, for example, more quickly or against a higher load than required, and hence more easily than expected by the program at that stage of the recuperation process, the system can request an increase in the rate, or the range or the force required for performing the exercise, The reverse adaption can be applied if the subject is having difficulty in performing the required movements.

[0017] The system may count the number of exercises performed or the amount of time practiced for any given movement as determined in the pre-planned rehabilitation program. When the time or level of movements has reached its goal, the system automatically tells the patient to move to the next exercise, according to the pre-defined therapy program. The motion extent and / or force exertion may define the measure of success and trigger the game features accordingly.

[0018] According to the program pre-defined by the therapist, the system can demonstrate to the patient when an accessory, and which type of accessory, should be used. The system can also demonstrate, according to the pre-defined program, when a splint may be required.

[0019] In one embodiment, the method defines when an exercise using in OT accessory was successfully accomplished and therefore triggers the game features, using pre-defined geometric parameters. For instance, when exercising with separable magnets, success will be considered when the subject's hands are distanced from each other after overcoming the magnetic attraction or repulsion. If the exercise includes grasping and picking up a bottle from a table, success will be considered by the distance of the bottle and / or the hand / s from the table. In another example, if the exercise is defined as working against plasticine resistance, then accomplishment can be defined by the distance of the hand or finger movement against this resistance.

[0020] In yet another feature, if the exercise is defined as folding a paper or a tissue into the subject's hand, then success can be defined if the controller does not identify any of part the paper outside of the subject's hand.

[0021] Since this is an adaptive method the parameters for defining success can be adjustable. For example, an adjustable distance of separating the hands, moving the fingers against resistance, the horizontal distance of bottle from table, or the amount of tissue / paper that is left outside of the subject's grasp. Features of a pre-planned therapy program may include specific limb or finger movements, range of motion, forces, games to be played, OT accessories to be used or the performance of specific exercises with or without a splint. Prior to initiating the training session, a calibration phase which detects the patient's limb, hand or finger physical properties, should be performed in order to define the goal which the rehabilitation programme tries to achieve, and the ranges of motion / forces that will trigger the game features accordingly.

[0022] According to the pre-defined exercise plan, the controller shows the patient, on the computer screen or by audio, or both, the hand movements which the OT programme attempts to achieve. When a certain time frame for a specific task ends, or when the patient reaches a therapy goal, for instance the number of exercises to be performed for a specific posture, the controller may automatically switch the limb or finger movements needed for the patient to operate any of the game features, such as by changing from a flexion to an extension movement. In this way, different rehabilitation exercises can be preprogrammed into the system, and operated sequentially as required. The patient is notified by the controller what are the required movements in every phase of the training session. This notification may be displayed in various forms such as, but not limited to, an image of the hand, written text notification on a screen, audible information, or any combination of the above. In another embodiment, the patient or the therapist can choose from the options displayed on the screen what posture is required, without the need for automatic replacement during the therapy session.

[0023] Movement of the whole limb, such as the hand, can be used to move the game screen or the cursor right or left according to the game requirements, while the specific required posture which the patient is attempting to achieve, activates the game feature.

[0024] The different phases of the training session may be accomplished with no need to switch games and therefore can provide a complete therapy session according to therapy plan requirements. The very same sessions can be performed at the clinic or at home, which make therapy simple and enable independent patient work without continuous guidance, in cases where the latter is not required. Additionally, the proposed method can be practiced with either remote or face-to-face guidance, or with no guidance at all. The system records all parameters of the therapy session for further analysis by the occupational therapist. Specifically, the system may record the outcomes of the exercise session, such as the amount of time invested in the exercise session, either daily, monthly, weekly or any analysis according to therapy needs, the amount of time and the number of exercises per each of the postures, such as pinch, flexion, etc. To create further motivation, the system can also record the number of points earned per each of the exercised postures in the games, and the longest play for each of them. The results can be demonstrated as a graph enabling to track therapy parameters, for instance session length, number of exercises for each posture and the longest play, over time, thus creating motivation to continue therapy practice.

[0025] The system can also define and recognize which limb, which hand, whether right, left or both, or which fingers are required to perform the therapy and to activate game features, therefore minimizing the patient's motivation to perform compensatory movements, and encouraging them to concentrate on the movements required for therapy.

[0026] The proposed method does not require any special gloves or any other wearables on the patient hand or limb. It is simple to install and operate on a personal computer or a smartphone or any other such device, using the embedded camera or an external camera and requires no dedicated design of game applications. These advantages make therapy sessions at the clinic or at the patient's home, cost-effective, simple to operate, flexible, engaging and suitable for all ages and rehabilitation conditions.

[0027] The operational methods of the described system and methods can also be used to plan therapy sessions based on computerized activities other than gaming. Such activities would be based on assessing the results of the requested OT exercise, and not the way in which the exercise activates the playing of a computer game. Such activities could include activities such as computer-based drawing, web searches, graphic design, scrolling through a book or playing a computerized musical instrument. Thus, for example, an exercise could be to have the subject draw a circle, using a computer coupled writing pen. The programme could investigate how long it takes to draw the circle, and how accurate the geometric shape of the drawn circle is. Any combination of these results could be used in the programme to provide an output related to an improvement in the drawing action, as compared with previous attempts, i.e. the effectiveness of the rehabilitation exercises. The occupational therapist can amend the conditions according to the exercise desired. The way in which the subject is holding the pen is not a result-calculated phenomenon, but more

[0028] SUBSTITUTE SHEET (RULE 26) like the type of exercise used in a computer game, namely exercising the grip or posture of the fingers while drawing a desired shape.

[0029] Similarly, when playing an electronic musical instrument, the accuracy of the tune can provide an indication of cognitive damage and improvement, while a view of the hand movements while playing can be used to provide corrective exercise to improve the strength and grip of the hand.

[0030] Throughout this application, the term “limb” may have been used as an example of a moveable body part of the subject needing occupational therapy to correct a disability. It is to be understood that although the term “limb” is generally and popularly applied only to arms and legs, in this application the term “limb” is intended to cover any body part which the subject can move by muscular action, such as a hand or one or more fingers, and is not intended to be limited to the popularly understood meaning of an arm or a leg of the subject,

[0031] There is thus provided in accordance with an exemplary implementation of the methods described in this disclosure, a a system for performing a rehabilitation programme on a subject having a disability in at least one limb, the system comprising:

[0032] (i) a computer system adapted to play computer games, in which at least one command for playing a game is entered using at least one activating trigger;

[0033] (ii) a camera adapted to image the at least one limb of the subject, while the at least one limb is executing an exercise of the rehabilitation programme, and

[0034] (iii) a system controller, adapted to receive a set of conditions that define successful execution of the exercise of the rehabilitation programme, wherein the system controller is adapted to generate a digital equivalence between the successful execution of the exercise of the rehabilitation programme, and the activating trigger used to enter the command in playing the computer game, wherein the system controller is adapted to generate an initial triggering level according to information which the camera captures regarding the range of motion initially achieved by the at least one limb, and wherein when performing an exercise of the rehabilitation program, the imaging camera provides real time indication of the successful execution of the exercise, and the system controller is adapted to output the triggering level for playing the computer game. In such a system, for an exercise of the rehabilitation programme involving a predetermined level of difficulty for its execution, the set of conditions may be determined by at least one of (a) the speed of execution of the exercise, and (b) the number of attempts needed for successful execution of the exercise. In such a case, the predetermined level of difficulty for the execution of the exercise of the rehabilitation programme, is defined by at least one of a predetermined force exertion of the limb, and a predetermined motion extent of the limb.

[0035] Furthermore, the force exertion of the at least one limb may be determined by measurement of the change of shape generated by the at least one limb, of an occupational therapy accessory having a predetermined elasticity. Such an occupational therapy accessory may be any one of an elastically flexible band, an elastically deformable ball, or a spring loaded clip. The spring loaded clip may be a clothes pin.

[0036] According to further implementations of the systems of the present application, if the subject succeeds in successful execution of the exercise, the predetermined level of difficulty is raised by an increase in at least one of the predetermined force exertion on the at least one limb or the predetermined motion extent of the at least one limb, Such an increase may be input into the system controller by the subject or by an occupational therapist, on detecting the successful execution of the exercise. Alternatively, the increase may be executed automatically by the system controller, on detecting the successful execution of the exercise within a predetermined time or within a predetermined number of attempts. In such a case, the automatically executed increase may be determined by the system controller by use at least one of machine learning and artificial intelligence.

[0037] On the other hand, if the subject does not succeed in successful execution of the exercise, the predetermined level of difficulty is lowered by a decrease in at least one of the predetermined force exertion on the at least one limb or the predetermined motion extent of the at least one limb, Such a decrease may be input into the system controller by the subject or by an occupational therapist, on detecting the unsuccessful execution of the exercise. Alternatively, the decrease may be executed automatically by the system controller, on detecting the lack of a successful execution of the exercise within a predetermined time or within a predetermined number of attempts. In such a case, the automatically executed decrease may be determined by the system controller by use at least one of machine learning and artificial intelligence. Such a system using at least one of machine learning and artificial intelligence, may enable the system controller to learn the capabilities of the subject, and to accordingly adjust the level of difficulty of the exercise.

[0038] In any of the previously described systems, the system controller may be adapted to enable the subject to play the computer game by replacing the activating trigger used to enter commands in playing the computer game, with the successful execution of the exercise of the rehabilitation programme.

[0039] Furthermore, the system controller may be adapted to receive the input of a demonstration of a successful execution of the exercise, before the commencement of the rehabilitation programme. In such a case, the required posture of the at least one limb is demonstrated either by the subject or by the occupational therapist or is obtained from the exercise history, for display on a screen of the computer system.

[0040] According to yet more implementations of such systems, the system controller may update the triggering level in accordance with information which the camera captures regarding the range of motion of the at least one limb, for any specific posture of the limb. Alternatively, the system controller may update the triggering level in accordance with information which the camera captures regarding the range of force exertion of the at least one limb, as determined by the change of shape generated by the at least one limb, of an occupational therapy accessory having a predetermined elasticity.

[0041] According to further implementations of such systems, the imaging camera may be adapted to input a demonstration of a required rehabilitation posture, and the system controller then adjusts the triggering level accordingly. Additionally, the at least one limb may be either the hand of the subject, or at least one finger of the hand of the subject. Also, the exercise of the rehabilitation programme may comprises movements in up to all of the six directions of freedom of the at least one limb. Such exercises of the rehabilitation programme may comprises exercises in the correct use of a pen or a pencil. Furthermore, in any of such systems, the camera is adapted to follow movements of the at least one limb of the subject, without the need for a tracking attachment on the at least one limb. In such case, the control system may use an image processing routine to follow the movements of the at least one limb.

[0042] There is further provided according to exemplary implementations of the methods described in the present application, a method of performing a rehabilitation programme on a subject having a disability in at least one limb, the method comprising the steps of:

[0043] (i) determining a set of conditions that define successful execution of an exercise of the rehabilitation programme;

[0044] (ii) determining an activating trigger used to enter commands into a computer game;

[0045] (iii) using a system controller to generate an initial activating triggering level, according to information which an imaging camera captures regarding the range of motion initially achieved by the at least one limb;

[0046] (iv) using the system controller to generate a computer equivalence between the successful execution of the exercise of the rehabilitation programme, and the activation of the trigger used to enter commands into the computer game; and

[0047] (v) using an imaging camera to track whether the set of conditions that define the successful execution of the exercise of the rehabilitation programme has been achieved, and if so, outputting the triggering level for playing the computer game.

[0048] In such a method for an exercise of the rehabilitation programme involving a predetermined level of difficulty, the set of conditions that define successful execution of the action of the at least one limb, is determined by at least one of (a) the speed of execution of the exercise, and (b) the number of attempts needed for successful execution of the exercise. The predetermined level of difficulty for the successful execution of the exercise of the rehabilitation programme, may be defined by at least one of a predetermined force exertion of the at least one limb or a predetermined motion extent of the at least one limb.

[0049] Furthermore, the force exertion of the at least one limb may be determined by measurement of the change of shape generated by the at least one limb, of an occupational therapy accessory having a predetermined elasticity. According to further implementation of these methods, if the subject succeeds in successful execution of the exercise, the predetermined level of difficulty is then raised by increasing at least one of the predetermined force exertion on the at least one limb, and the predetermined motion extent of the at least one limb. Such an increase may be input into the system controller by the subject or by an occupational therapist. Alternatively , the increase may be executed automatically by the system controller, on detecting the successful execution of the exercise within a predetermined time or within a predetermined number of attempts. In such a case, the automatic execution of the increase may be performed by the system controller using at least one of machine learning or artificial intelligence.

[0050] On the other hand, if the subject does not succeed in successful execution of the exercise, the predetermined level of difficulty is then lowered by decreasing at least one of the predetermined force exertion on the at least one limb, or the predetermined motion extent of the at least one limb. Such a decrease may be input into the system controller by the subject or by an occupational therapist. Alternatively , the decrease may be executed automatically by the system controller, on detecting the lack of success in the execution of the exercise within a predetermined time or within a predetermined number of attempts. In such a case, the automatic execution of the decrease may be performed by the system controller using at least one of machine learning or artificial intelligence.

[0051] Any of these methods may further comprise the step of updating the triggering level in accordance with information which the camera captures, regarding the range of motion of the at least one limb, for any specific posture of the at least one limb. Additionally, any such methods may further comprise the step of updating the triggering level in accordance with information which the camera captures regarding the range of force exertion of the at least one limb, as determined by the change of shape generated by the at least one limb, of an occupational therapy accessory having a predetermined elasticity.

[0052] BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The present invention will be understood and appreciated more fully from the following detailed description, taken in conjunction with the drawings in which: Fig. 1 shows the components of the system and its operation, using a laptop computer camera which detects hand movements and activates the features of a computer game;

[0054] Fig. 2 shows schematically the therapy session process, including (i) a pre-planning stage, (ii) a calibration stage based on visual data of the patient's limb, as acquired by the camera, followed by (iii) exercise sessions where data from the camera is used by the system to activate any computer game at the patient's convenience, the requirements being based on therapist-guided or subject- guided pre-definition of a variety of range of motion and forces exerted by the limb, hand or fingers, and (iv) the continuous monitoring of the progress of the subject to match the exercises to that progress; and

[0055] Fig. 3 to Fig. 7 show examples of screenshots related to the proposed systems and methods of the present application, and its features.

[0056] DETAILED DESCRIPTION

[0057] The presently described exemplary systems present a cost-effective and easy to operate gamebased method for performing rehabilitation occupational therapy (OT) exercises at the clinic or at the subject's home. Prior to, or during a training session, the therapist is able to pre-define the limb movements required for the patient's rehabilitation, for instance flexion, extension, pinching, adduction, abduction, finger pointing, and the like, and the ranges of motion desired for limbs for each individual subject. These may be defined in all 6 degrees of freedom, for the leg, shoulder, entire hand or for each of the fingers. Accordingly, the proposed method allows focusing training sessions on specific parts of the limb, such as fingers, and different movement requirements for each finger or fingers, based on patient- specific rehabilitation needs. Thus, there is great flexibility for treating different disability situations.

[0058] Based on the goals of each pre-defined therapy, a controller of the system identifies, using information from an imaging camera viewing the limb or limbs being exercised, when pre-defined motions and / or force exertion requirements have been successfully accomplished by the patient. Successful accomplishment of a predefined movement or more accurately, range of movement, triggers features of the computerized game, in a manner similar to a computer mouse click or movement of the mouse or the use of a joystick, or a game controller, or even a keyboard entry. The patient or subject and the therapist are able to watch, in real time, the changes in movement or in forces exerted by the patient's limb or hand during the exercise sessions and amend the therapy accordingly.

[0059] Reference is first made to Figure 1 which shows an exemplary system for performing the methods of rehabilitation of a hand according to the present disclosure. The system is shown in Fig. 1 using a laptop computer 1 on which the subject is playing the computer game being used to perform the OT session. In the example shown in Fig. 1, a simple game is illustrated on the computer screen, in which the user has to target bottles to knock off a wall. The computer’s integral camera 2 is being used in order to surveil the progress of the subject’s exercising of his / her hand 3, though a separate dedicated video camera or a smartphone camera could also be used for that purpose. The video imaging information is transmitted to the system controller 4 which converts the video imaging data, using a motion tracking application 5, into information which identifies the spatial positions of the fingers of the hand 3 as the subject performs the required exercise, which, in the example shown in Fig. 1 is a pinching exercise. In Fig. 1, the pinching exercise is shown being performed using a clothes pin 6 to provide a predetermined finger force and range of motion required by the exercise, though any other object, such as a compressible ball, could be used. Alternatively, just the finger motion itself, without any external applied force, can also be used as a criterion for the exercise during the rehabilitation. The angular location of the hand 3 should also be determined, and transferred to the unit 16 for defining the aim and position of the hand, as it is this pose of the hand that the computer game uses to determine whether the user has correctly aimed in order to hit one of the bottles of the game. The system controller 4 detects when the subject can achieve one or more of a pre-defined movement, a pre-specified range of motion or a predetermined level of force, by comparing in a comparator module 7, the motion data from the motion tracking application 5, with the motion speed or range required by the OT exercise, taking into account the demanded force requirement. The occupational therapist will have selected a predetermined training program 8, which can be stored in the system memory for use in the motion comparator module 7, and the therapist also decides and inputs to the system, which level of achievement of the exercise is an indication of a successful outcome of the exercise. When that criterion has been met, as determined by the system controller in comparator module 7, using the motion information obtained from the camera and its subsequent motion tracking application 5, the trigger signal generator 15 outputs a signal 9 to the game controller 13, indicating when the subject is activating a triggering feature of the computer game. In the example shown, that action is the throwing of a ball to hit a bottle. The signal 9 is input into the feature activating unit 10 of the game controller application, in lieu of the usually input signals from a joystick 11, or a mouse 12, or a keyboard stroke, or a similar triggering input device. The successful achievement of the OT requirement thus enables activation of that feature of the computer game in a manner similar to the result of activating a real joystick 11 or a mouse 12 or a game controller of the video game itself, and the generation of a signal 14 inputting the activation command to the computer for showing its implementation on the screen image of the game. Additionally, the signal 16 relating to the aiming direction of the user has also been input from the system controller 4, into the aim confirmation unit 17 of the game control module 13 to enable the user to play the computer game in a realistic manner, merely substituting the physical activation trigger with a signal generated as a result of the successful performance of the OT exercise.

[0060] The system controller 4 and the game control module 13 are shown for clarity as separate units in Fig. 1, but in practice, all of those functions could be performed by applications incorporated into the game controlling routines within the user’ s computer 1.

[0061] With reference to the use of the cloth pin 6 for the OT exercise, common OT accessories, such as rubber bands, sponge balls, clothes pins and the like, can be supplied with various degrees of stiffness for use during the therapy sessions. The disclosed method enables tracking the hand manipulating the common accessories, and thus allows performing exercises which require forces together with range of motion therapy. Forces can be measured without the need for using force sensors, according to the methods describe in the commonly owned above mentioned International Published Application WO2023135602A1, and the computer game features can be activated based on the measurement of the forces achieved by the subject in each exercise. It should be noted that training can be performed with or without the OT accessories as well as with or without a splint.

[0062] The disclosed method is applicable to any computer game, while the game is operated by the patient's bare hand(s) or other limb and does not require any other wearables or use of a computer mouse or joystick. Furthermore, it does not necessitate the creation of designated computer game applications as is currently done in most of the previously used systems, and for the training exercise sessions, the subject can use games that he / she is familiar with and enjoys using, or can choose any other gaming application. These features make the proposed method readily tolerated by a wide range of patients of all ages, and encourages perseverance in the exercises because any discomfort may be countered by the user’s enjoyment of the game. Furthermore, the system and methods of the present disclosure can be used to play computerized simulations of board games, or even games of chess, for instance, where the OT exercise could include both a pinching component for picking up the virtual piece, and a motion control exercise in moving the piece to its intended position on the board.

[0063] Reference is now made to Fig. 2 which shows schematically an exemplary program showing a training process using a method and system of the present disclosure. In the planning stage 20, the therapist, or the subject him / herself, can define a therapy plan and a time-frame for each of the OT exercises. That can include one or more of the number of exercises for each of the required postures, the force exertion requirement for each of the required postures, or the number of exercises for each of the required postures. The motion extent of the limb for each OT exercise can also be used as an input for the therapy. This enables great flexibility for the rehabilitation process. During visits to the clinic or as part of a tele -rehabilitation session, the therapist can guide the patient in real-time to correctly perform the exercises or can dynamically change the preplanned program according to the patient's needs and achievements.

[0064] In the calibration stage 21, which is performed before the formal performance of the training session themselves, the physical properties of the limb(s) to be exercised, are obtained by a camera of a PC, a tablet, a smartphone, or an independent camera, which detects limb, hand and finger poses in 6 degrees of freedom, finds the maximum / minimum range of motion or force according to patient capabilities, and adjusts the triggering of the game feature accordingly. The adjustment of the triggering level can be preformed either by the occupational therapist’s assessment of the achievement of the subject during the calibration stage, or by an automatic controller decision, based on the controller’s analysis of the achievement of the subject. This decision can be based on any of the limits of the abilities of the subject in performing the exercise, such as the extent of motion of the limb, or the level of force exertion which the limb can achieve, or any other criterion related to the subject’s present abilities. The system controller can use either or both of machine learning or artificial intelligence routines, based on expected performance improvement scenarios, in order to use the subject’s calibration achievements to assess a realistic triggering level for this subject.

[0065] Once this has been determined, the training or exercise stage 22 can commence, in which the patient can first choose a favorite computer game, and then performs training according to the pre- defined training plan from step 20, and using the triggering level determined in step 21. In one embodiment, the controller may track the patient's movements and notify whether the exercise was correctly performed. The system can also adaptively change the levels of difficulty by learning the patient's capabilities versus the therapy goals, using artificial intelligence and machine learning. All parameters of the training session are recorded for further analysis.

[0066] During the further training stage 23, after the time objective for a specific exercise has been reached, the system may automatically switch to train another posture planned by the rehabilitation program, and a similar gaming command is triggered when the new posture is detected. The system notifies this to the user, and indicates when an OT accessory should be used. Likewise, the subject’s need for a splint to ease the achievement of the required performance of the movement may also be indicated by the system.

[0067] In step 24, the session's achievements are continuously and dynamically monitored and adaptively accommodated by the therapist or by the system to match the capabilities of the patient. Therapist guidance can be performed either physically in the clinic or the home, live on-line in the form of a remote tele-rehabilitation, or in an off-line setting after the therapy session. The controller may also use machine learning capabilities and / or artificial intelligence, to learn by itself the movements which the patient is able to perform and what movement and what extent of that movement, should be used to trigger the game features. The camera and controller record and learn specific limb or finger movements which the patient is capable to perform, as the triggers for the game. When the patient performs these movements during the therapy sessions, it triggers a feature of a game. Hence therapy can be tailored to a wide range of disabilities and conditions, including situations where movement or force exertion is much impaired.

[0068] In step 25, the session's achievements are continuously and dynamically monitored and adaptively accommodated by the therapist or by the system to match the capabilities of the patient.

[0069] Therapist guidance can be performed either physically in the clinic or the home, on-line remotely in the form of tele-rehabilitation or in an off-line setting after the therapy session. These postures are pre-defined as part of the desired therapy goals with their associated ranges of motions (possibly for the limb, hand or finger / s). Prior to the session, the patient or the therapist demonstrate the required motions to the controller and the latter identifies the required motions and / or forces and sets the threshold for the exercise session. The system detects when the range of motion has been reached to trigger the game features. In both settings, pre-session or intra-session automatic learning, the motions which trigger the system are not limited and may be applicable to multiple medical conditions as well as tailored to specific patients, ages, and impairment conditions.

[0070] In the embodiment described in Fig. 2, in place of the level of the exercise being defined by the patient’s present capabilities, the controller can adaptively change the levels of difficulty by learning the patient's capabilities versus therapy goals using artificial intelligence and machine learning. For example, a certain feature of a computer game can be triggered by flexing the hand to a certain degree during the initial training session. The learning process can detect whether the patient is struggling to achieve a certain movement, and consequently can lower the threshold to a smaller range of motion or less force. In contrast to this, if the system detects that a certain movement or force exertion is easily performed by the patient, it can raise the difficulty according to a threshold predefined by the therapist or the patient. These adjustments can be accomplished for any required movement, range of motion, force exertion or their combination. For example, if according to the rehabilitation plan, pinching action is being exercised, the game feature can be triggered by a wide or a narrow pinching movement, while continuously playing the same favorite game. Game achievements, such as moving to a higher level, or earning more points, or the like, can indicate the patient’s improvement and encourage the patient to continue playing the game and hence practicing the training movement. The features of the disclosed method allow great flexibility in determining ranges of motion, forces, and levels of difficulty, applying different thresholds for triggering game or application features. As described, the levels of difficulty may be determined by the therapist prior to, after, or during the training session or automatically by the controller. For example, the system controller can extend the range of motion, instruct the insertion of more force, or concentrating on specific exercises required to improve the rehabilitation process.

[0071] Reference is now made to Fig. 3, Fig. 4, Fig. 5, Fig. 6 and Fig. 7, which show examples in the form of exemplary screenshots demonstrating the steps of the proposed method.

[0072] Fig. 3 is a representation of the main operation screen 30 which includes a dropdown list of optional games 31, and a search game option 32 which enables searching for any desired game available online by typing the URL into a URL box 33. A start / stop activation button 34 on the screen 30 enables either (i) activating the system to allow the camara to image the limb being exercised, and thus for the limb, on achieving its required level of performance, to act in the place of a typical joystick or mouse, or (ii) returning to the regular mouse and keyboard mode. The game to be played appears on the game screen 36. A border zone 35 of the game screen 36, shows the range in which the camera is able to see the motions of the limb / hand and fingers. It should be noted that Fig. 3, as also with Fig. 4 to 7, is merely a graphic representation of what would be shown on the display of the gaming computer, and that since the drawing is not to scale, the game screen 36 itself, though small in Fig. 3, in fact captures most of the computer screen. The extent to which a motion triggers the game features is shown on the current trigger level bar 37, where a higher trigger level requires a greater range of motion and / or force, while a lower trigger level requires less. The exercises which trigger the applications or the game features, such as, but not limited to pinch, flexion, extension, adduction, abduction and the like, are shown in a separately list 38, and inform the patient which movements are required, and whether there are pauses during the therapy program process, according to a delay option. A virtual image of the user's limb or hand showing its movement in real time, appears in window 39.

[0073] Fig. 4 shows yet another option for the main operation screen 49. A listing 40 of applications or games is provided, from which the patient can choose the game to be played, according to his / her preference. A virtual image of the user's limb or hand showing its movement in real time appears in window 41. Images of required postures (for example pinch, flexion, extension) appear on the screen 42, 44, 50. The patient or the therapist can click on one of these images to choose which posture should be exercised. Alternatively, the system can switch between the postures and show automatically on the screen an image of the posture that triggers the game features. A required posture may be notified to the patient by for instance, enlarging the image of the posture, a voice notification or any such combination. Once a posture is chosen it is clearly displayed to the patient. The system tracks the number of exercises performed for each of the postures, out of the preplanned number, and displays it in a box 43 on the screen, thus enabling the patient and therapist to track performance versus the pre-defined therapy goals. The game is played on the game screen 36 using the specific required posture for triggering the game feature. In case calibration is needed for each of the postures, a link 45 is provided to the calibration screen.

[0074] The patient or therapist may simply choose to switch between the right and left hand by clicking on a specified link 46. An option to calibrate the speed of the computer mouse as replaced by the hand movements, is also provided in box 47, thus enabling flexibility for patients moving more slowly. Finally, the threshold for any of the postures triggering the game features can be set manually 48 or through a designated camera-based calibration screen.

[0075] Figure 5 shows one example of the user calibration screen 50. The controller displays to the patient the required limb or hand gesture, whether flexion, pinch, adduction, or any other motion, 51 which defines the type of trigger and its difficulty for every patient. A visualization of the limb, or hand and fingers appears on the screen 52 and the time frame for the calibration process is demonstrated 53. An active trigger level bar 54 shows the maximum 55 and minimum 57 required levels to trigger the relevant application feature or features, and also the current user trigger value 56, all which can be adjusted for any specific patient.

[0076] Figure 6 shows yet another option for the calibration screen 60. The controller displays to the patient the required limb or hand gesture, whether flexion, pinch, adduction, or any other motion, 61, which defines the type of trigger and its difficulty for every patient. The patient is requested to position the limb in front of the camera, displayed as a designated box on the screen. Then, a visualization of the limb, or hand and fingers appears on the screen 62. When the patients have reached their maximum range of motion or force exertion, they are asked to click the 'Done' button 64 to end the calibration process. In one option, the system may demonstrate the limb image to the patients and re-ask them if they are sure that they have reached their maximum range of motion, hence verifying the calibration process results.

[0077] Figure 7 shows the exercise designer screen 70. The screen displays an inventory of exercises 71 from which the therapist or the patient can choose. Some examples for such exercises are shown in this screen representation. The exercise timeline 75 and the duration bar 73 define the timeframe for each exercise, and the type of trigger 72, and their associated order, including pre-planned pauses. A calculation of the total exercise duration is also shown 74. The designer screen enables a simple graphic format for designing the exercises to be performed during a therapy session, in order to meet therapy goals. For instance, the therapist or the patient can drag the building blocks of the therapy session from the exercise inventory 71 to the exercise duration bar 73 and adjust their duration.

[0078] It will be appreciated by those skilled in the art, in view of these teachings, that alternative embodiments may be implemented without intentionally deviating from the claimed invention. For instance, different exercises can be defined or different screen design options can be applied, and the application is not intended to be limited by the examples shown in this disclosure.

[0079] Example embodiments are provided so that this disclosure will be thorough, and will fully convey the scope to those who are skilled in the art. Numerous specific details are set forth such as examples of specific components, devices, and methods, to provide a thorough understanding of embodiments of the present disclosure. It will be apparent to those skilled in the art that specific details need not be employed, that example embodiments may be embodied in many different forms and that neither should be construed to limit the scope of the disclosure. Furthermore, it is appreciated by persons skilled in the art that the present invention is not limited by what has been particularly shown and described hereinabove. Rather the scope of the present invention includes both combinations and subcombinations of various features described hereinabove as well as variations and modifications thereto which would occur to a person of skill in the art upon reading the above description and which are not in the prior art.

Claims

CLAIMS1. A system for performing a rehabilitation programme on a subject having a disability in at least one limb, the system comprising: a computer system adapted to play computer games, in which at least one command for playing a game is entered using at least one activating trigger; a camera adapted to image the at least one limb of the subject, while the at least one limb is executing an exercise of the rehabilitation programme, and a system controller, adapted to receive a set of conditions that define successful execution of the exercise of the rehabilitation programme, wherein the system controller is adapted to generate a digital equivalence between the successful execution of the exercise of the rehabilitation programme, and the activating trigger used to enter the command in playing the computer game, wherein the system controller is adapted to generate an initial triggering level according to information which the camera captures regarding the range of motion initially achieved by the at least one limb, and wherein when performing an exercise of the rehabilitation program, the imaging camera provides real time indication of the successful execution of the exercise, and the system controller is adapted to output the triggering level for playing the computer game.

2. A system according to claim 1, wherein, for an exercise of the rehabilitation programme involving a predetermined level of difficulty for its execution, the set of conditions is determined by at least one of (a) the speed of execution of the exercise, or (b) the number of attempts needed for successful execution of the exercise.

3. A system according to claim 2, wherein the predetermined level of difficulty for the execution of the exercise of the rehabilitation programme, is defined by at least one of a predetermined force exertion of the at least one limb, or a predetermined motion extent of the at least one limb.

4. A system according to claim 3, wherein the force exertion of the at least one limb is determined by measurement of the change of shape generated by the at least one limb, of an occupational therapy accessory having a predetermined elasticity.

5. A system according to claim 4 wherein the occupational therapy accessory may be any one of an elastically flexible band, an elastically deformable ball, or a spring loaded clip.

6. A system according to claim 5, wherein the spring loaded clip is a clothes pin.

7. A system according to claim 3, wherein if the subject succeeds in successful execution of the exercise, the predetermined level of difficulty is raised by an increase in at least one of the predetermined force exertion on the at least one limb or the predetermined motion extent of the at least one limb.

8. A system according to claim 7, wherein increase of at least one of the predetermined force exertion or the predetermined motion extent, is input into the system controller by the subject or by an occupational therapist, on detecting the successful execution of the exercise.

9. A system according to claim 7, wherein increase of at least one of the predetermined force exertion or the predetermined motion extent, is executed automatically by the system controller, on detecting the successful execution of the exercise within a predetermined time or within a predetermined number of attempts.

10. A system according to claim 9, wherein the system controller uses at least one of machine learning and artificial intelligence, to determine when an increase of at least one of the predetermined force exertion or the predetermined motion extent, is to be executed automatically by the system controller.

11. A system according to claim 3, wherein if the subject does not succeed in successful execution of the exercise, the predetermined level of difficulty is lowered by a decrease in at least one of the predetermined force exertion on the at least one limb or the predetermined motion extent of the at least one limb.

12. A system according to claim 11, wherein decrease of at least one of the predetermined force exertion on the at least one limb or the predetermined motion extent of the at least one limb, is input into the system controller by the subject or by an occupational therapist, on detecting of the lack of successful execution of the exercise.

13. A system according to claim 12, wherein decrease of at least one of the predetermined force exertion on the at least one limb or the predetermined motion extent of the at least one limb, is executed automatically by the system controller, on detecting the lack of successful execution of the exercise within a predetermined time or within a predetermined number of attempts.

14. A system according to claim 13, wherein the system controller is adapted to use at least one of machine learning and artificial intelligence, to determine when a decrease of at least one of the predetermined force exertion or the predetermined motion extent, is to be executed automatically by the system controller.

15. A system according to any of the previous claims, wherein the system controller is adapted to enable the subject to play the computer game by replacing the activating trigger used to enter commands in playing the computer game, with the successful execution of the exercise of the rehabilitation programme.

16. A system according to either of claims 10 or 14, wherein the at least one of machine learning and artificial intelligence enables the system controller to learn the capabilities of the subject, and to accordingly adjust the level of difficulty of the exercise.

17. A system according to any of the previous claims, wherein the system controller is adapted to receive the input of a demonstration of a successful execution of the exercise, before the commencement of the rehabilitation programme.

18. A system according to claim 17, wherein the required posture of the at least one limb is demonstrated either by the subject or by the occupational therapist or is obtained from the exercise history, for display on a screen of the computer system.

19. A system according to any of the previous claims wherein the system controller updates the triggering level in accordance with information which the camera captures regarding the range of motion of the at least one limb, for any specific posture of the limb.

20. A system according to any of the previous claims wherein the system controller updates the triggering level in accordance with information which the camera capturesregarding the range of force exertion of the at least one limb, as determined by the change of shape generated by the at least one limb, of an occupational therapy accessory having a predetermined elasticity.

21. A system according to any of the previous claims wherein the imaging camera is adapted to input a demonstration of a required rehabilitation posture, and the system controller adjusts the triggering level accordingly.

22. A system according to any of the previous claims, wherein the limb is either the hand of the subject, or at least one finger of the hand of the subject.

23. A system according to any of the previous claims, wherein the exercise of the rehabilitation programme comprises movements in up to all of the six directions of freedom of the limb.

24. A system according to any of the previous claims wherein the exercise of the rehabilitation programme comprises exercises in the correct use of a pen or a pencil.

25. A system according to any of the previous claims wherein the camera is adapted to follow movements of the at least one limb of the subject, without the need for a tracking attachment on the at least one limb.

26. A system according to claim 25, wherein the control system uses an image processing routine to follow the movements of the at least one limb.

27. A method of performing a rehabilitation programme on a subject having a disability in at least one limb, the method comprising the steps of: determining a set of conditions that define successful execution of an exercise of the rehabilitation programme; determining an activating trigger used to enter commands into a computer game; using a system controller to generate an initial activating triggering level, according to information which an imaging camera captures regarding the range of motion initially achieved by the at least one limb;using the system controller to generate a computer equivalence between the successful execution of the exercise of the rehabilitation programme, and the activation of the trigger used to enter commands into the computer game; and using an imaging camera to track whether the set of conditions that define the successful execution of the exercise of the rehabilitation programme has been achieved, and if so, outputting the triggering level for playing the computer game.

28. A method according to claim 27, wherein, for an exercise of the rehabilitation programme involving a predetermined level of difficulty, the set of conditions that define successful execution of the an action of the at least one limb, is determined by at least one of (a) the speed of execution of the exercise, or (b) the number of attempts needed for successful execution of the exercise.

29. A method according to claim 28, wherein the predetermined level of difficulty for the successful execution of the exercise of the rehabilitation programme, is defined by at least one of a predetermined force exertion of the limb or a predetermined motion extent of the limb.

30. A method according to claim 29, wherein the force exertion of the at least one limb is determined by measurement of the change of shape generated by the at least one limb, of an occupational therapy accessory having a predetermined elasticity.

31. A method according to claim 30, wherein if the subject succeeds in successful execution of the exercise, then the predetermined level of difficulty is raised by increasing at least one of the predetermined force exertion on the at least one limb, or the predetermined motion extent of the at least one limb.

32. A method according to claim 31, wherein the step of increasing at least one of the predetermined force exertion on the at least one limb or the predetermined motion extent of the at least one limb, is entered into the system controller by the subject or by an occupational therapist.

33. A method according to claim 31, wherein the step of increasing at least one of the predetermined force exertion on the at least one limb or the predetermined motion extent ofthe at least one limb, is executed automatically by the system controller, on detecting the successful execution of the exercise within a predetermined time or within a predetermined number of attempts.

34. A method according to claim 33, wherein the automatic execution of the step of increasing at least one of the predetermined force exertion on the at least one limb or the predetermined motion extent of the at least one limb, is performed by the system controller using at least one of machine learning or artificial intelligence.

35. A method according to claim 29 wherein if the subject does not succeed in successful execution of the exercise, then performing the step of decreasing at least one of the predetermined force exertion on the at least one limb or the predetermined motion extent of the at least one limb.

36. A method according to claim 35, wherein the step of decreasing at least one of the predetermined force exertion on the at least one limb or the predetermined motion extent of the at least one limb, is instructed by the subject or by an occupational therapist.

37. A method according to claim 35, wherein the step of decreasing at least one of the predetermined force exertion on the at least one limb, or the predetermined motion extent of the at least one limb, is executed automatically by the system controller, on detecting the successful execution of the exercise within a predetermined time or within a predetermined number of attempts.

38. A method according to claim 37, wherein the automatic execution of the step of increasing at least one of the predetermined force exertion on the at least one limb or the predetermined motion extent of the at least one limb, is performed by the system controller using at least one of machine learning or artificial intelligence.

39. A method according to any of claims 27 to 38, further comprising the step of updating the triggering level in accordance with information which the camera captures, regarding the range of motion of the at least one limb, for any specific posture of the at least one limb.

40. A method according to any of claims 27 to 38, further comprising the step of updating the triggering level in accordance with information which the camera captures regarding the range of force exertion of the at least one limb, as determined by the change of shape generated by the at least one limb, of an occupational therapy accessory having a predetermined elasticity.

Images