Systems and methods for determining user intent from limb or body movements or trajectories and using them to control neuromuscular stimulation or prosthetic movement

Abstract

A device for restoring movement to a person's paralyzed body part (e.g., the person's hand) is disclosed. The device senses movement of another part of the person's body that is not affected by the paralysis (e.g., the person's arm or shoulder). When the person moves the non-paretic body part, a motion sensor generates a motion signal. A processor stores information associating a predefined trajectory with a specific action (e.g., closing the hand to grasp an object). The processor monitors the motion signal, and if the movement corresponds to the predefined trajectory, the processor energizes a muscle stimulator connected to a muscle to control the paretic hand to perform the action (e.g., closing the hand around an object to grasp the object).

Description

[Technical field]

[0001] The present disclosure relates to systems, devices, applications and methods for assisting partially disabled people by providing voluntary movement of a paretic joint or prosthesis by determining the person's intention to move the joint or device from an analysis of the limb or body movement of the person's healthy joint. More specifically, the system, method or device to which the present disclosure relates determines that the overall movement (translation and / or rotation) or trajectory of a neurologically intact limb or other body part is sufficient to determine the user's intention (to perform an action with the paretic or missing limb) and, in response to the determined intention, stimulates the neurologically paretic part (via the nerves and / or muscles controlling the part) or neural targets (nerves, spinal cord or brain) to restore movement or function by promoting nerve growth / regeneration or strengthening connections or to control a prosthetic valve replacement to perform the action. A device according to one embodiment of the present disclosure detects the reaching trajectory of a person's arm, identifies the person's intent to grasp an object, and activates or adjusts a neuromuscular stimulation device (NMES) to enable the person's normally paralyzed hand (or activates the person's robotic / prosthetic hand) to open and close to grasp and hold the object. [Background technology]

[0002] Description of Related Art In the United States alone, there are approximately 5.4 million people living with paralysis. The two leading causes of paralysis are stroke and spinal cord injury. There are over 17,700 cases of spinal cord injury in the United States each year (NSCISC, 2019). Many of these injuries lead to incomplete quadriplegia (48%) and complete quadriplegia (20%), severely impacting survivors' arm and hand movement and resulting in a reduced quality of life.

[0003] A top priority for individuals living with quadriplegia is hand function. A variety of invasive and non-invasive neuromuscular electrical stimulation (NMES) devices have been proposed to rehabilitate or induce upper limb and hand movements. These known systems have deficiencies. Freehand systems use shoulder movements coupled with switches that trigger selected hand movements through electrical muscle stimulation via implanted electrodes. Switch activation can be cumbersome and the operation of the muscle stimulator may force the user to perform unnatural movements. Such movements may draw attention to the user's disability and affect others' perception of the user. The repertoire of hand movements available to the user may also be limited by the number of switches that can be activated by the user's shoulder muscles.

[0004] Other systems may require the performance of surgery. For example, some systems rely on implanted myoelectric sensors to detect the patient's intent to move a paretic or surgically amputated joint. Cortical brain / computer interfaces (BCIs) have been used to control NMES devices by recording and decoding motor activity in the brain, allowing voluntary control of a normally paretic hand. These approaches require electrodes or other structures to be implanted into the user's body, which may pose medical risks and significant costs to the user. Summary of the Invention

[0005] The present disclosure relates to devices and methods that address these challenges. Patients living with paralysis want to integrate into society while avoiding as much attention as possible to their disability. Although rehabilitation can restore at least partial mobility to some patients, restoring fine motor control (e.g., a user reaching out to grasp an object (e.g., a drinking glass or food)) can be difficult or impossible. The present disclosure allows patients with limited manual grasping movement to perform tasks (e.g., feeding themselves) (without relying on equipment (e.g., tools fixed to the hand)) and perform daily activities.

[0006] For patients living with paralysis due to stroke, spinal cord injury or other conditions, they may be unable to move their hands and / or legs, but are often able to move other body parts. For example, spinal cord injuries at the C5 level, the most common level of injury for quadriplegic patients, leave hand movement severely impaired, but shoulder movement and elbow flexion remain. Similarly, while full arm (shoulder and elbow) movement after a stroke is often recoverable through intensive rehabilitation, recovery of hand movement remains a mystery. Finally, paraplegics or stroke victims may be unable or incompletely unable to use their legs or may experience foot drop (inability to flex and extend the ankle), but may be able to move their arms or torso or hips.

[0007] Disclosed herein are methods and systems for restoring voluntary control of a user's paretic joint and / or external device (by sensing and recognizing the motion and trajectory of the user's still healthy joint). The system uses computerized algorithms (e.g., machine learning adapted to the user's specific body movements) to identify the user's intent to perform an action with the paretic joint or a prosthetically replaced joint. The detected body movements and trajectories can then be used to produce a variety of desired results. According to one embodiment, such a system determines a person's intent to reach out and grasp an object, and activates a NMES device to open and close the user's paretic hand to grasp and hold the object.

[0008] The devices included in the present disclosure sense and recognize limb trajectories (e.g., reaching movements controlled by residual shoulder and elbow movements) as well as other body movements, positions or orientations, and activate the muscles of the paretic body part through electrical stimulation via electrodes or electrode arrays to perform a specific activity (e.g., a pinching movement of the hand to "grab a key") or energize actuators on the prosthetic body part. A variety of predefined trajectories and limb or body movements can be stored, which may be combinations of translational and rotational movements. Each trajectory or movement is associated with a different behavior. Based on the recognized movements, the devices according to the embodiments of the present disclosure can also be used to control external devices (e.g., a computer or a powered wheelchair). Furthermore, a large number of distinct trajectories can be identified with different actions, thus expanding the repertoire of actions available to the user.

[0009] Further included in the present disclosure is a device that recognizes motion around healthy joints (e.g., hips, lumbar spine, and knees) to identify motions associated with a person's walking and apply stimulation signals to muscles as the person walks. Such devices can be used to restore more effective walking motion (where nerve damage has impaired the movement of the person's foot, ankle, or leg). Such devices can be used to strengthen muscles required for walking pre-operatively (e.g., before hip or knee replacement surgery) and / or post-operatively as part of rehabilitation therapy.

[0010] According to another embodiment, instead of or in addition to energizing the electrodes or prostheses to enable movement, the system according to the present disclosure delivers electrical stimulation to the site of nerve injury or to nerve pathways connected to nerve injury (e.g., spinal cord, brain, or peripheral nerves). The system according to the present disclosure can assist in repair of damaged motor fibers, nerves, or neurons by providing electrical stimulation with electrodes placed percutaneously or epidurally on or near the site of spinal cord, nerve, or brain injury while moving the affected limb. In the case of spinal cord injury, the system can also assist in healing of damaged sensory fibers, nerves, or neurons by providing electrical stimulation with electrodes placed percutaneously or epidurally on or above or near the site of injury.

[0011] Sensors on the arms, legs and / or body can recognize a variety of two-dimensional and three-dimensional (2D / 3D) movements (translational acceleration, rotational speed and orientation relative to the earth's magnetic field). According to some embodiments, such movements are detected by an inertial motion unit (IMU) with a total of 3-9 degrees of freedom. According to other embodiments, a visual image of the movement can also be recognized. Just as a child uses a sparkler to draw letters, numbers and patterns in the air, the device recognizes smooth natural curved arms reaching trajectories and pre-trained patterns (e.g., known script numbers and letters). The user can then perform the movement or natural hand reaching trajectory that they wish to perform, and once these movements are recognized, these movements are used to control a variety of neuromuscular stimulation and prosthetic / robotic devices that promote movement of the paretic joint. In the arm, the arm movement trajectory is driven by the residual shoulder movement and can be used to drive the stimulation or robotic control of multiple wrist, hand and finger movements (or external devices (e.g., computer, stereo)).

[0012] In addition to allowing patients to use remaining mobility to grasp objects, devices according to embodiments of the present disclosure may improve neurological function by providing feedback to the patient's central nervous system to correlate healthy joint and limb movements with activation of paretic body parts. Thus, using such devices to drive neuromuscular or robotically driven movements in paretic joints has assistive, rehabilitative and therapeutic applications for stroke, spinal cord injury and other neurodegenerative conditions. This approach also has applications in general physical therapy following injury or surgery to the hands, feet, legs or other body parts.

[0013] Additionally, the disclosed embodiments may be used to measure, track and recognize (e.g., through machine learning algorithms as disclosed) the quality of limb / body motion trajectories over time in rehabilitation applications. Joint motion is captured, recorded and recognized or graded, allowing physical therapists to monitor a patient's progress and tailor treatment to address specific areas of motion that are problematic. Machine learning or other forms of artificial intelligence (e.g., deep learning methods) allow for analysis of aggregate data (from many anonymous patients) to discover general patterns and metrics (indicative of progress or regression and issues that can be flagged for review or corrective action).

[0014] According to one embodiment, the disclosed device includes one or more motion sensors generating one or more respective motion signals indicative of a motion of a first body part of a human, a muscle stimulator generating one or more stimulation signals for displacing a second body part by one or more muscles to perform at least one action, and a processor connected to the one or more motion sensors and the muscle stimulator. The processor includes a data store including at least one expected trajectory associated with the human's intention to perform the at least one action. The processor receives the one or more signals from the one or more motion sensors, calculates an actual trajectory of the first body part, compares the actual trajectory to the expected trajectory, and activates the muscle stimulator based on the comparison to displace the second body part to perform the at least one action. The processor may calculate a difference between the actual trajectory and the expected trajectory, perform the comparison, and activate the muscle stimulator based on the difference.

[0015] According to one embodiment, the at least one behavior includes a plurality of behaviors, and the at least one expected trajectory includes a plurality of expected trajectories. Each of the plurality of expected trajectories is associated with at least one of the plurality of behaviors. The processor compares the actual trajectory with the plurality of expected trajectories to identify a first trajectory associated with a first behavior of the plurality of behaviors, and the processor activates the muscle stimulator to perform the first behavior. The device may include an input device connected to the processor, the input device adapted to receive a feedback signal. The feedback signal may indicate that the behavior is an intended behavior of the human. The processor may generate the expected trajectory based on a set of training movements. The one or more stimulation signals for performing the at least one behavior may include a pattern of a stimulation signal, and the pattern of the stimulation signal may be determined from muscle displacements sensed during the set of training movements. The sensing of the muscle displacements may be performed using one or more of an electromyographic sensor, a camera, an inertial unit, a flexion / joint angle sensor, and a force sensor. The comparison by the processor may be performed using one or more of a support vector machine (SVM) algorithm, a handwriting recognition algorithm, a dynamic time warping algorithm, a deep learning algorithm, a recurrent neural network, a shallow neural network, a convolutional neural network, a convergent neural network, or a deep neural network. The processor may perform the comparison using a long-short-term memory recurrent neural network. The set of training movements may be performed by a second human. The set of training movements may be performed by the human using a body part laterally opposite the first body part. The motion sensor may be disposed on the human's arm, and the muscle stimulator may be adapted to stimulate muscles to move one or more fingers of the human's hand to enable a prehensile movement. The expected trajectory may be in the form of an alphanumeric character.

[0016] According to one embodiment, the device includes an orientation sensor. The orientation sensor may be connected to the processor and adapted to monitor the orientation of the first body part. The force applied by the grasping movement may depend on the amplitude of the stimulation signal, and the processor may adjust the amplitude of the stimulation signal based at least in part on the output of the orientation sensor. In response to the output of the orientation sensor, the processor may adjust the grasping movement to be a key grasp, a cylindrical grasp, or a vertical pinch. A camera may be included in the device, connected to the processor and positioned adjacent the hand to capture an image of the object to be grasped. The processor may adjust the grasping movement based in part on the image. The processor may include a close delay timer, and at the end of the actual trajectory determined by the close delay timer, the processor may delay stimulating the grasping movement for a predetermined period of time. In response to a post-grasp signal from the motion sensor, the processor may stimulate the hand to perform a post-grasp activity. A post-grasp action can be opening the hand to release the grasp, and a post-grasp signal can be one or more taps on the surface of the grasped object.

[0017] According to another embodiment, a device is disclosed that includes one or more motion sensors that generate one or more respective motion signals indicative of a motion of a first body part of a human, a muscle stimulator that generates a stimulation signal adapted to generate or increase a contraction of a first muscle, the first muscle being a neurologically damaged muscle, a paralyzed muscle, a partially paralyzed muscle, or a normal muscle, and a processor connected to the sensor and the muscle stimulator. The processor includes a data store that includes at least one expected trajectory associated with the human's intent to contract the muscle. The processor receives the one or more motion signals from the one or more sensors, calculates an actual trajectory of the first body part, compares the actual trajectory to the expected trajectory, determines an intent to contract the muscle based on the comparison, and causes the stimulator to one or more of contract the first muscle, assist the contraction of the first muscle, and generate an antagonistic contraction of a second muscle, the contraction of the second muscle opposing the motion caused by the contraction of the first muscle. The device may include a neurostimulator connected to and operable by the processor, and in response to the processor determining an intent to contract the first muscle, the neurostimulator may apply a neurostimulation signal to the human nerve. The human nerve may be selected from one or more of a vagus nerve, a trigeminal nerve, a cranial nerve, a peripheral nerve leading to the first muscle, and the human spinal cord. The nerve may be the spinal cord, and the neurostimulator may include a percutaneous electrode positioned above, on or below the person's spinal cord injury.

[0018] In one embodiment, the disclosed device includes one or more motion sensors generating one or more respective motion signals indicative of a motion of a first body part of a human, a prosthetic limb including an actuator configured to change a configuration of the prosthetic limb to perform an action, and a processor coupled to the one or more motion sensors and the actuator. The processor includes a data store including at least one expected trajectory associated with the human's intent to perform the action. The processor receives the one or more motion signals from the one or more motion sensors, calculates an actual trajectory of the first body part, compares the actual trajectory to the expected trajectory, and actuates the actuator based on the comparison to change a configuration of the prosthetic limb to perform the action. The prosthetic limb may include a prosthetic hand, and the actuator may include one or more of a wrist actuator and a finger actuator. [Brief description of the drawings]

[0019] A better understanding of the present disclosure and many of its attendant advantages will be readily obtained by reference to the following detailed description, when taken in conjunction with the accompanying drawings, in which:

[0020] [Figure 1] 1 shows a person's arm and hand equipped with a device according to an embodiment of the present disclosure undergoing a test to measure finger dexterity.

[0021] [Diagram 2] FIG. 1 is a block diagram of a system according to one embodiment of the present disclosure.

[0022] [Diagram 3] 1 shows the position, velocity and acceleration of a person's arm as he or she moves the arm along a "C" shaped motion path, equipped with a device as shown in FIG.

[0023] [Figure 4]Illustrated is the position, velocity and acceleration of a person's wrist as they move their arm along a motion path in the shape of a "number 3" as shown in FIG. 1 .

[0024] [Diagram 5] 1 illustrates how a system according to an embodiment of the present disclosure is integrated with a cosmetic patch.

[0025] [Figure 6] 1 shows a person's arm and hand equipped with a system according to an embodiment of the present disclosure moving a pen from one location to another.

[0026] [Figure 7] 13 is a graph illustrating the performance of an apparatus according to an embodiment of the present disclosure in identifying the movement of a patient's limb using a predefined trajectory.

[0027] [Figure 8] 1 illustrates a comparison of confusion matrices using different machine learning algorithms for identifying predefined trajectories in accordance with an embodiment of the present disclosure.

[0028] [Figure 9] 1 illustrates a prosthetic limb including a device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Some patients who have suffered nerve damage (e.g., stroke or spinal cord injury) may lose the ability to control the movement of some parts of their body, but retain the ability to move other parts. In some cases, the movement of the remaining limbs may allow the patient to move their shoulders and upper arms and bend their elbows, but the ability to control the movement of the hands, for example to grasp objects, may be lost. In other cases, the patient may lose the ability to articulate their knees and ankles, but retain movement of the hips. In the case of amputees, the patient may retain full function of the remaining part of a limb that was surgically amputated.

[0030] A system according to an embodiment of the present disclosure senses and recognizes the trajectory and body movement of the remaining limb in space through machine learning methods to identify the user's intent to perform a particular action. Sensors on the arms, legs and / or body allow recognition of a variety of two-dimensional and three-dimensional (2D / 3D) movements (e.g., translation, rotation or combinations thereof). Circuits included in the system deliver NMES signals to muscles controlling movement of the paretic body part or operate a robotic / prosthetic limb to restore control of the hand / arm or foot / leg.

[0031] According to further embodiments, the system detects the natural, streamlined curvilinear path of movement of a functional body part that is normally associated with a desired action, and causes the paretic body part to perform the action. For example, in the case of a patient who has residual movement in his shoulder and upper arm, the device recognizes the trajectory of reaching and causes the paretic hand of the patient to open and close to grasp an object. As used herein, the term "trajectory" refers to the overall movement of a body part (e.g., the translational and / or rotational movement of a body part in space) as well as the angular displacement of a body part around a joint (e.g., flexion of an elbow, shoulder, hip, or knee).

[0032] Different trajectories of the reach can be detected, and in response, the system positions the patient's hand appropriately for that type of reach. For example, if the patient moves his / her arm and shoulder forward or in a curved path (with the wrist in a neutral "handshake" position), the system identifies that the patient intends to grasp a vertically oriented object (e.g., a glass bottle or water bottle placed on a tabletop ("cylindrical grip")) by comparing the actual trajectory of the arm or shoulder with the expected trajectory associated with the patient. In response to this identified intention, the system energizes the NMES electrodes on the patient's forearm to activate the appropriate muscles to open the hand to grasp the object, and after a delay, the system stimulates the muscles to ensure that the fingers grip and hold the object around it. Alternatively, if the patient uses their remaining shoulder and arm movements to reach in a "rainbow" arc along the vertical direction of the handle, the system identifies that the user intends to use a hand movement to grasp to pick up the object from above ("vertical grasp"). The patient may also attempt to reach for the object using a "bottle opener" motion to indicate an intention to perform a third type of grasp (e.g., a "claw grasp" to pick up the object). The device activates the NMES electrodes controlling the hand to cause the patient's thumb and fingers to open and then converge around the top of the object. The advantage of using the natural movements of the remaining body parts to control the paretic body parts is that the patient's behavior more closely matches that of an able-bodied person. This may reduce attention to the user and may promote neuroplasticity and rehabilitation in, for example, stroke or recent spinal cord injury patients.

[0033] This type of residual movement detection may also include pre-determined trajectories where the patient performs, for example, arm movements along a "C" shaped path. Just as a child draws letters, numbers and patterns in the air with a sparkler, the device recognizes the patterns. The patient moves his healthy joints along pre-determined expected trajectories, and the system identifies that a specific action is intended. In response, the system activates the NMES electrodes to generate muscle contractions for the execution of the desired action. For example, the patient may perform a "C" shaped movement with the shoulder and upper arm when opening and closing the hand around a cylindrical object, while closing the hand in a pinching motion. The advantage of using pre-programmed expected trajectories is the large number of specific movements that can be encoded. The device can be programmed to recognize both pre-trained patterns and natural reaching trajectories. Additionally, new trajectories for new actions can be added to the patient's behavioral repertoire.

[0034] According to one embodiment, the device energizes the NMES electrodes to stimulate appropriate muscle contractions for the execution of the intended action. According to another embodiment, the device recognizes the motion path of the patient's healthy body parts for the operation of prosthetic / robotic devices that facilitate the movement of the paretic joint. In the arm, the arm motion trajectory can be used to drive the stimulation or robotic control of a prosthetic hand that is driven by the residual shoulder motion and adapted to perform multiple movements of the wrist, hand, and fingers. Such a prosthetic hand includes a combination of wrist actuators and finger actuators. In addition, specific movements can be detected for the control of external devices (e.g., computers, stereos, powered wheelchairs, etc.). Due to the large number of distinct motion paths, the device can be used both to control the paretic body parts (e.g., using the natural trajectory of the shoulder in a reaching movement for the control of the paretic hand) and to control external devices such as computers using pre-programmed motion paths (e.g., a "C" shaped path).

[0035] Using the device according to the embodiments of the present disclosure for driving the robotically driven movements in neuromuscular or paralyzed joints can lead to further assistive, rehabilitative and therapeutic applications in stroke, spinal cord injury and neurodegenerative conditions. The patient uses the remaining movements in healthy body joints, strengthening the musculature and neural connections used to execute the remaining movements. In addition, with the use of the device, brain plasticity associates the remaining movements (both natural movements and pre-programmed movement paths) with the desired behavior, making the patient's movements more fluid, like those of a healthy person. This approach also provides applications in general physical therapy after injury or surgery to the hands, feet or other body parts. Furthermore, the disclosure herein can be used to measure and track the quality of limb / body movement trajectories over time in rehabilitation applications.

[0036] FIG. 1 shows a patient's hand and forearm equipped with a device according to an embodiment of the present disclosure performing the "9-hole peg test." This test is a standard measure of hand dexterity and is known in the art. A wearable sensor housing 10 is provided on top of the patient's wrist. The wearable sensor housing 10 includes motion sensors that detect the path of the patient's hand movement and the orientation of the patient's limbs. As described in more detail below, the sensors include an inertial motion unit (IMU) that detects 3-axis acceleration, a gyro sensor that detects rotational speed, and a magnetic sensor that detects orientation in the earth's magnetic field. According to other embodiments, the sensors may also include joint angle / flexion sensors that detect flexion of joints such as the elbow, knee, or hip. A computer (or microprocessor embedded in the device) (not shown in FIG. 1) communicates with the IMU. The computer includes a processor, memory, and / or output device. According to the embodiment shown in FIG. 1, the IMU communicates with the computer via a radio frequency Bluetooth link. In this test, the NMES electrodes 12 contact the patient's abductor pollicis brevis and flexor pollicis brevis muscles, which are responsible for the basic movements of the thumb.

[0037] FIG. 2 is a block diagram illustrating an embodiment of the system in FIG. 1. The sensor housing 10 includes sensors 16a, 16b, ... 16n. These may include IMUs, joint flexion / angle sensors, cameras, gyro sensors, force sensors, and other sensors that monitor movement and orientation. A microcontroller 18 is connected to the sensors and processes signals from the sensors to integrate outputs from the various sensors to provide trajectory data (e.g., body part orientation, three-axis acceleration corrected for gravity, or global movement (translation and / or rotation) information). Output from the microcontroller 18 is provided to a computer system 20, which provides signals indicative of the path of the patient's hand movement and analyzes the movement, as described below. According to one embodiment, the microcontroller 18 and computer 20 include radio frequency transceivers 19a and 19b (e.g., Bluetooth or ZigBee protocol devices for wirelessly communicating movement data). According to other embodiments, the functionality of the computer 20 may be integrated with the microcontroller 18. The microprocessor may be a neural processor or neural processing unit or tensor processor, optimized for low power consumption machine learning or deep learning, making it ideal for wearable devices (e.g., the M1 processor by Apple (Cupertino, CA) or the Cortex-M55 by Arm (Cambridge, UK)). The computer 20 may also include a network of locally connected computers and / or a computer system remote from the wearer (e.g., a cloud computing system).

[0038] According to the embodiment shown in FIG. 1, the sensor housing 10 is worn like a wristwatch. Other types of housings can be used. For example, the sensor housing 10 can be incorporated into a cuff, sleeve, or wearable adhesive patch (with electrodes, microprocessor or artificial neural network or AI processor, visual indicator (e.g., with LED), wireless communication, and disposable conductive adhesive material) on a glove worn on the patient's forearm or on the patient's hand. Such a sleeve or wearable adhesive patch can employ a joint flexion / angle sensor to detect flexion of the patient's elbow. In applications where the actuation of a paretic limb or an external device is controlled by the remaining movement of other body parts, the device can be worn as a belt (to detect waist movement), as part of a hat or headband (to detect the movement and orientation of the patient's head), or incorporated into clothing worn anywhere on the patient's body.

[0039] The computer 20 is connected to a NMES driver 14, which generates and applies electrical current to a number of NMES electrodes 12a, 12b, 12c, ... 12n, which are placed on the patient's forearm or in a cuff, sleeve, or adhesive patch. According to one embodiment, the NMES electrodes 12a, 12b, 12c, ... 12n are placed in a sleeve that fits securely onto the patient's forearm as shown in Figure 5, which is described in more detail below. The electrode configuration is selected to correspond to the muscular anatomy of the forearm. Once in place, the NMES electrodes may be mapped to the patient's musculature.

[0040] Stimulation waveforms generated by the NMES driver 14 are applied to selected sets of electrodes. The parameters of the waveforms (e.g., wave shape (square, sine, triangular, etc.), pulse width, pulse frequency, voltage and duty cycle) are selected and the NMES driver is configured to apply these signals in response to control signals from the computer 20. According to one embodiment, the stimuli are applied as a series of short bursts separated by inter-burst periods. The NMES parameters may be selected to improve skin penetration for more precise separation of finger and thumb movements and reduced fatigue. These electrodes are mapped to specific muscles in the patient's forearm such that selected muscles are activated (by stimulation signals from the NMES driver) to activate flexion and extension of the fingers and thumb.

[0041] In the example shown in FIG. 1, the NMES electrodes are applied to the patient's forearm using adhesive tape or conductive adhesive material or hydrogel. Alternatively, the electrodes may be in a patch-like configuration (with a disposable adhesive hydrogel) or a cuff with integrated sensors and a microprocessor or AI processing unit may be worn on the patient's forearm as shown in FIG. 5, as described below. Other methods known to those skilled in the art for connecting and orienting the electrodes to the patient's musculature may be used. In the embodiment shown in FIG. 1, the electrodes 12a, 12b, 12c, ... 12n are positioned to apply a stimulation current to one or more of the thumb muscles that control the thumb (abductor pollicis brevis, flexor pollicis brevis, and opponens pollicis), which triggers a variety of useful thumb movements (e.g., "pick up" (with the index finger tip) and a "hook" type grasp).

[0042] Hardware and software components included in the computer 20 receive signals from the sensors 16a, 16b, ... 16n to determine the trajectory and orientation of the housing 10 and thus the movement path and orientation of the patient's limbs. Based on this, the computer 20 sends signals to the NMES driver 14 to energize the electrodes 12a, 12b, 12c, ... 12n (according to a sequence that causes the patient's hand to assume the intended configuration). According to one embodiment, the computer 20 also provides output to an output device 22 (e.g., a display monitor or screen) and receives input from one or more input devices 24 (e.g., a keyboard, a computer mouse or other pointing device, and / or a microphone). The output from the computer may also be recorded and used by a medical professional in the evaluation of the patient's progress during physical therapy. In addition, as described in more detail below, this output may be anonymized and collected along with similar data from a patient population and used to train a machine learning system aimed at improving the recognition of body movements and trajectories indicative of the user's intent to perform the intended action.

[0043] According to another embodiment, the computer 20, the NMES driver 14, the microcontroller 18, and the array of NMES electrodes 12a, 12b, 12c, ... 12n are integrated with the sensor housing 10 to form a portable, wearable system. Such a wearable system may include a touch screen or other input / output device (similar to a "smart watch" that allows the patient to interact with the system (e.g., to train the system to improve identification of the patient's intent)). The connection between the computer 20 and the other components of the system may be a physical connection (e.g., cable). Alternatively, the computer 20 may communicate signals wirelessly by a radio frequency link (e.g., Bluetooth, ZigBee) or via infrared. The computer 20 includes memory storage and is programmed to perform various algorithms as described in more detail below. According to another embodiment, the computer 20 is also integrated with the sensor housing 10. Such an embodiment results in a self-contained system that allows the wearable system to be used in a manner independent of any wired or wireless interface.

[0044] FIG. 5 shows an embodiment of the present disclosure in which an array of NMES electrodes 12a, 12b, ... 12n is integrated onto a wearable patch 15. An electrical coupling layer 13 (e.g., a hydrogel layer) is provided between the electrode array and the wearer's skin. In the embodiment shown in FIG. 5, the electrodes 12a, 12b, ... 12n are arranged in a pattern adjacent to the musculature controlling the wearer's hand. According to one embodiment, other components (e.g., sensor housing 10 including IMU sensors 16a, 16b, ... 16n, microcontroller 18, NMES driver 14, computer 20 and power source) are also arranged on the wearable patch 15. This embodiment eliminates the need for cabling, allowing the user to freely move healthy joints (in this example, shoulder, torso, and upper arm or lower back) to activate the system to stimulate the intended action in the paretic joint of the hand, lower limb, or foot. Eliminating the need for cabling allows the device to be worn continuously to assist the user in daily activities. The electrode array 12 can be programmed to map specific NMES drivers 12a, 12b,...12n to the wearer's musculature such that energizing a particular electrode or set of electrodes results in a particular movement (e.g., a grasping movement of the leg or foot as described above). Such mapping can use machine learning techniques to fine-tune muscle activation to the wearer's intent.

[0045] In the example shown in Figure 1, inertial sensors (IMUs) 16a, 16b, ... 16n in housing 10 on the patient's wrist sense 2D and 3D arm trajectories and send signals to computer 20. These signals are analyzed and compared (by data fitting and / or machine learning algorithms running on computer 20) to one or more expected trajectories associated with a desired action. If a trajectory or movement is recognized that indicates the patient is about to perform a particular action, computer 20 sends a signal to NMES driver 14 to activate selected electrodes 12a, 12b, 12c, ... 12n to control neuromuscular stimulation patterns in the forearm to control the "opening" and "closing" of the hand.

[0046] These IMUs monitor the patient's limbs and provide signals that are analyzed to indicate a desired movement or device action. These IMUs may detect 6-axis (acceleration and rotational speed) or 9-axis (addition of magnetic field information) movement. One or more housings containing IMUs may be placed at various limb, body or head locations and used to provide orientation and translation information for a patient's limb segments in the legs, hands, feet, hips, neck, head or any other body part.

[0047] As the patient in FIG. 1 moves his / her hand in a vertical "rainbow" arc, the output of the IMU attached to the patient's wrist (or forearm) is analyzed by computer 20 to detect this as the expected trajectory and identify the patient as attempting to grasp the peg from the pegboard using a "key grip" type of motion. As the patient's hand approaches the end of the vertical arc trajectory, computer 20 causes NMES driver 14 to stimulate the patient's muscles to curl the fingers and move the thumb away from the index finger, preparing the thumb to extend and assume the "key grip" configuration on the peg. Once the hand reaches the end of the vertical arc trajectory, computer 20 positions the thumb spaced to the side of the index finger for a time delay, allowing the patient (with their remaining shoulder and arm function) to position the hand relative to the peg. At the end of this delay, computer 20 activates a NEMS electrode on the abductor pollicis brevis muscle to close the grip on the peg. The NEMS signals remain active, so the peg remains securely gripped. Sensing other gross movements of the patient's wrist or forearm (i.e., translational and / or rotational movements) can determine the patient's intent to perform other types of prehensile movements. For example, the patient may indicate their intent to perform a different type of grasp (e.g., a "claw grasp" to pick up an object) by reaching for the object using a "bottle opener" movement.

[0048] The computer 20 keeps the muscle activated until the patient makes another movement or trajectory (indicating that the patient wants to release his / her grip). According to one embodiment, the patient moves his / her hand (with residual shoulder / elbow movement) in a small clockwise or counterclockwise motion (in a horizontal plane parallel to the table surface) which is detected by the accelerometer (e.g., one or more of the IMU sensors 16a, 16b, ... 16n). This movement is interpreted by the computer 20 as an indication that the patient wants to release the peg. The computer 20 applies a NMES current to move the thumb away from the index finger to open the grip and release the peg. Other movements that indicate the need to release the object can be used (e.g., a pronation or supination (rotation) type movement of the forearm). The user can select any pattern of movement or body movement that indicates an intention to release the grip, which is recognizable to pattern recognition and / or machine learning algorithms, thus evoking a "hand opening" neuromuscular stimulation pattern. According to another embodiment, instead of or in addition to a body movement or trajectory, the signal indicating the patient's intention to release the object is an abrupt signal (e.g., one or more taps on the surface of the object), which generates an accelerometer signature signal. A tapping signal is particularly advantageous when a cylindrical object (e.g., a water cup) is grasped, where the tapping is weak and does not draw attention to the person's disability. A simple clockwise or counterclockwise circular movement in a horizontal plane can also be used to indicate the user's desire to open his or her hand to release the object.

[0049] According to another embodiment, instead of sending signals to a NMES driver, the computer 20 is connected to a motorized actuator of a robotic / prosthetic hand that replaces the patient's surgically amputated hand. In this embodiment, the robotic hand is controlled to perform a grasping action in response to the detection of the arm trajectory.

[0050] According to one embodiment, the interpretation of the trajectory depends on the state of the system before the trajectory detection. In the previous example, when an object is grasped, a clockwise circular movement / trajectory in the horizontal plane is interpreted as a command to release the object. If the system is in a different initial state (e.g., the hand is in an "open" position), a clockwise circular movement may result in a different behavior (e.g., to perform a claw grasp).

[0051] The embodiments of the present disclosure are not limited to detecting hand or arm movements. The human body can move limbs and torso in a variety of patterns and achieve a myriad of movements in space. In particular, the rotation and trajectory of a human's arms and legs, and even hips and torso, in space contains a huge amount of information. Disclosed herein are methods and devices for sensing and recognizing a variety of movements for the achievement of a variety of desired results in a robust and highly accurate manner. A natural reaching movement can be described (with residual shoulder movement) by a specific linear or curved movement in space (sometimes involving limb (or body) rotation). For example, using this approach, a quadriplegic user can move his or her arm along a curved path to an object, and after the trajectory is automatically recognized, neuromuscular stimulation is triggered to open and close (after a short delay) the user's hand around the object.

[0052] The IMU can also provide orientation information, which can be very useful. For example, if the IMU is placed on the back of the wrist (the forearm side of the wrist where the watch face is placed), the hand is in a neutral (handshake) position, and this information, combined with a particular reaching trajectory, can indicate that the user wishes to grasp a cylindrical object (e.g., a water bottle or a glass). The 2D arm trajectory and / or orientation pattern can be used to drive a multitude of behaviors (e.g., device control and muscle stimulation patterns for various hand / leg movements). Furthermore, various trajectories can be used to control different types of grasps. As mentioned above, a rainbow-like arc trajectory as a user reaches over an object placed on a table allows for triggering a claw-type open / close grasp (to pick up the object from above). A clockwise bottle-open reaching trajectory can be used to control a cylindrical grasp, and a counterclockwise bottle-open reaching pattern can be used for a pick-up grasp.

[0053] In addition to the IMU, other sensors may be used for detection of healthy joint motion. According to one embodiment, a flexion sensor is provided at the elbow to provide additional input. This input may be used to further identify a specific trajectory. Elbow flexion may also be used to adjust current amplitude for neuromuscular stimulation to drive grip strength (or robot end effector closure force) during gripping behavior.

[0054] According to another embodiment of the present disclosure, instead of or in addition to detecting natural body movements, the device detects one or more predefined trajectories. Similar to moving a sparkler in the air, recognizable patterns and shapes can be generated (e.g., letters, numbers, bottle opener / spiral). The sensors 16a, 16b, ... 16n detect the movements associated with such patterns, and the computer 20 analyzes the signals from the sensors to determine if the patient has performed a pattern corresponding to a particular behavior. The user can select any pattern he or she likes and link the pattern to various movements or device actions (home electronics, computers, mobile devices, robotic arms, wheelchairs). These trajectories can be used to interact with, control, or drive these devices under direct user control.

[0055] A device was constructed according to an embodiment of the present disclosure. The sensors 16a, 16b, ... 16n consist of Bosch Sensors TecBNO0559-axis IMU. The sensors were connected to a microcontroller 18, here a 32-bit ARM microcontroller unit (MCU) from Adafruit (Feather Huzzah32). The IMU has a built-in processor and algorithms to estimate orientation and perform gravity compensation in real time to generate linear acceleration in three orthogonal directions. Linear acceleration along the X-axis, Y-axis and Z-axis was made available externally via an I2C interface. A flexible printed circuit board was designed to interconnect the IMU with the MCU 18. Data from the MCU was streamed to the computer 20 continuously at 50Hz via Bluetooth. For embodiments where processing was performed offline, the motion data was stored and processed using MATLAB® 2019a on the computer 20.

[0056] In another embodiment, the MCU 18 performs real-time data processing to activate a muscle stimulator positioned on the subject's forearm. Neuromuscular stimulation was performed by a battery-operated 8-channel voltage-controlled stimulator with a stimulation pulse frequency of 20 Hz. These stimulation channels are mapped to individual or multiple electrodes on a fabric sleeve to evoke various finger flexion and extension movements. Different types of grasps (e.g., cylindrical grasp and pick-up grasp) were programmed by grouping multiple stimulation channels and sequencing the activation profiles.

[0057] FIG. 3 shows a movement recorded by a device according to a further embodiment of the present disclosure. In this example, a healthy individual wearing a device according to an embodiment of the present disclosure moved his arm through a "C" shaped trajectory. In this example, the individual repeated this movement three times. Signals from the IMU provided six-axis data (acceleration and rotational velocity) of the individual's wrist. The IMU output was corrected for gravity to provide repeatable acceleration data. This acceleration data is integrated to determine the time-dependent position (i.e., trajectory) of the limb during the movement. Based on this trajectory, computer 20 determines that a "C" shaped trajectory has occurred. In each repetition, the "C" shaped shape appears at the X / Y position displayed in the right-most column.

[0058] The computer 20 may use pattern recognition algorithms to analyze and identify limb and body movements and trajectories to identify the patient's intended action. This analysis may include signal processing algorithms (e.g., Dynamic Time Warping (DTW) to compare the actual trajectory of the patient's limb movements with the trajectory expected to correspond to the intended action). DTW has the advantage of being able to accommodate different movement / trajectory speed or timing profiles that may vary from user to user.

[0059] According to other embodiments, the sensor output is analyzed by applying machine learning techniques to identify the user's intent to perform a particular action and to identify other movements that do not involve the user's intent to perform an action. According to one such embodiment, a Convolutional Neural Network (CNN) or Recurrent Neural Network (RNN) included in the computer 20 analyzes data from the IMU and other sensors to identify body movements and trajectories or camera data that signal the patient's intent to perform an action, providing further contextual information to further identify the user's intent or information about the object the hand is approaching (the shape and size of the object that needs to be accommodated and grasped by the hand). The RNN implements techniques such as long-short-term memory (LSTM) to identify the intent-signaling movements. Utilizing such techniques, the system iteratively and reliably identifies specific trajectories or body movements to activate the patient's muscles or motorized prostheses to perform the intended action. In addition, because the sensor data is recorded, the system according to embodiments of the present disclosure can be continuously trained to better identify the patient's intent. Once data from multiple patients is properly anonymized, the resulting data can be collected and utilized to train machine learning algorithms. A variety of other machine learning algorithms can be used to analyze and identify natural and pre-programmed trajectories. Non-limiting examples include support vector machine (SVM) algorithms, handwriting recognition algorithms, and deep learning algorithms. Such machine learning algorithms can be executed locally on the computer 20 worn by the patient (e.g., embedded in the prosthesis or connected to the sensor housing 10). As an alternative or in addition to local processing, machine learning algorithms can be executed on a computer system located remotely from the user (e.g., on a cloud computing network). This allows the systems and methods disclosed herein to be adapted with the collection of additional data over time. Such algorithms can recognize repetitive body movements by the patient, allowing the algorithm to recognize movements that are not highly accurate at first detection.

[0060] FIG. 4 shows another example of motion detection by a device according to an embodiment of the present disclosure, where a healthy individual performed three repetitions of a "3" shaped movement. Again, the IMU provided gravity-corrected acceleration data, and the computer calculated the time-dependent trajectories of the individual's limbs. Again, the "3" shaped shape was seen in each repetition, as shown by the position graphs in the right-most column. If the user associates the "3" shaped and "C" shaped movements with different actions (e.g., a "key grasp" and a "cylindrical grasp"), the computer 20 can apply different patterns of neuromuscular stimulation, resulting in hand movements performing one type of grasp or the other.

[0061] According to another embodiment, a training set of motion data was prepared for a variety of alphanumeric trajectories. First, the raw 3-axis gravity-compensated accelerations obtained from the IMU were band-pass filtered (Butterworth, 8th order, 0.2-6Hz) and processed offline to identify training samples. The absolute value of the acceleration along the 3 axes was used to identify the onset of motion (by setting a threshold of 0.95g). This onset of motion was then used to segment the acceleration data over time along the X, Y and Z axes into a window of -0.1s to 0.9s relative to the onset. Each trial was visually checked for the absence of any noise artifacts or excessive jerks (acceleration derivatives) or beyond the 1s window. Such trials were removed from further analysis. These training sets were used to train the Dynamic Time Warping (DTW) and Long Short-Term Memory (LSTM) network algorithms.

[0062] The DTW algorithm optimally aligns a sample trajectory with a previously determined template trajectory, thereby minimizing the Euclidean distance between these two samples. This is achieved by iteratively expanding or contracting the time axis until an optimal match is found. For multivariate data (e.g., acceleration), the algorithm uses dependent time warping to simultaneously minimize the distance along different dimensions. According to the present embodiment, the algorithm computed the optimal distance between the test sample and all templates associated with the 2D and 3D trajectories. The template with the least optimal distance to the test sample was selected as the output of the classifier. Since the output of the classifier depends on the quality of the template, an inner optimization loop was used to select the best template trajectory from a set of training trajectories. Within this loop, the DTW score of each training sample was computed along with the other training samples. The training sample with the lowest sacrifice DTW score was then selected as the template for that trajectory (i.e., the expected trajectory).

[0063] In some embodiments, an LSTM network is used for analyzing the motion data. According to one such embodiment, the LSTM network includes a single bidirectional layer with over 100 hidden units with the MATLAB® R2019b Deep Learning Toolbox. Default values ​​were selected for most parameters. The LSTM network converts 2D or 3D acceleration data into input to a fully connected layer whose outcome was binary (i.e., 0 or 1). A softmax layer was then used to determine the probability of multiple output classes. Finally, the network output mode was set to "last" so that a decision is only generated if the final time step has passed. This allows the LSTM classifier to behave similarly to the DTW and classify the trajectory window. When training the weights of the LSTM network, an adaptive moment estimation (ADAM) solver was used with a gradient threshold of 1 and a maximum number of epochs of 200. Zero padding was not used because all training and validation data were 1 second long. For the implementation of the LSTM network, the MATLAB® R2019b Deep Learning Toolbox was used with default values ​​for parameters except those mentioned above.

[0064] According to one embodiment, online classification of arm trajectories was performed by filtering and processing the raw acceleration signal in real time (using a MATLAB script looped at 50 Hz). Within this loop, acceleration data was divided into 1 second long segments with 98% overlap. A DTW-based classifier was implemented and designed to compare the incoming acceleration windows with the 2D trajectories. If the optimal distance between the trajectories falls below 10 units (empirically determined), a positive classification is issued, after which the NMES driver 14 is triggered to stimulate the muscles to perform the entire hand opening and closing movement sequence.

[0065] According to one embodiment, the sensor data is input into a machine learning algorithm, which is trained to identify certain movements as expected trajectories (for association with actions). Such training can be achieved by utilizing the non-affected side (mirror image of the movements) in a healthy person or stroke patient. Hemiparesis (paralysis of one side of the body) is not uncommon in stroke patients. According to one embodiment, the stroke user uses his / her non-affected side to train the device's algorithm or to further fine-tune its movements. In either case, the user performs natural reaching and diverse trajectories (with additional sensors to detect hand opening and different gripping actions) under real-world conditions when wearing the device. Such additional sensors include EMG (electromyography) sensors placed on relevant muscles to determine the gripping actions of the hand (opening, closing, key grip, cylindrical grip). The amplitude of this EMG signal indicates the strength (e.g., duration and change over time) of the muscle contraction, and this data can directly inform the applied electrical stimulation amplitude and timing from the NMES driver 14 for delivery of a stimulation signal pattern to perform a grasping action when a desired movement is recognized through the movement / trajectory recognition algorithm. The device for detecting hand opening and different grasping actions may also include a camera that records images of such actions, an IMU, or a joint angle / flexion sensor or force sensor attached to the healthy hand (used to train the system to determine the stimulation signal pattern).

[0066] Further sensors may include a camera as well as flexion / joint angle sensors and force sensors. The camera is linked to image analysis and positioned to capture the reaching trajectory and / or grasping movement. The captured trajectory and grasping data is used to build a database of pre-trained trajectory or movement patterns (for association with specific hand actions). This data is used to train machine learning algorithms (e.g., deep learning neural networks). After training (or partially training) the device, it is fitted to the person with paralysis. Such training may include the use of input to the computer via the input device 24. For example, if a person is training the system to recognize a particular trajectory as an "S" shaped path (indicating a cylindrical grasp), the person may audibly utter words (such as "cylindrical grasp", "open" and "closed") simultaneously with the movement. Using a microphone as input and known voice recognition technology, the computer 20 reads the voice input and tags the sensor data. This tagged data becomes part of the training set data for the machine learning algorithm. Alternatively, keystrokes on a keyboard may be used to tag the movements used to train the algorithm, or the computer 20 may be equipped with a camera that captures visual images of the user performing various tasks (e.g., grasping an object on a table, inserting a peg into a board) (while recording movement data from an IMU to correlate associated hand movements with ``natural'' prehensile movements).

[0067] In another embodiment, a camera placed on the wrist (as part of a band, sleeve / patch or garment) recognizes the object as it approaches, thereby influencing the stimulation pattern to change the type of hand opening (e.g., activating all fingers or only the thumb-index extensor muscles). If the object slows / stops relative to the hand, the flexor muscles are automatically activated to initiate the grasp. Real-time object recognition techniques using small, portable devices with battery-powered microprocessors (e.g., cell phone technology) are well known in the art. These techniques, when combined with AI and machine learning methods (e.g., support vector machines, convolutional neural networks, and long-short-term memory (LSTM) recurrent neural networks for static and dynamic image classification), allow visual cues (e.g., the type of object being grasped) to inform the system how to position the patient's hand to correctly and reliably grasp the object.

[0068] Given 2D and 3D motion (e.g., bottle-opening motion in air), a variety of trajectories can be computationally identified and associated with a variety of behaviors. These trajectories can be used to drive prosthetic / robotic or motility devices (e.g., wheelchairs) as well as to drive neuromuscular stimulation for movement restoration.

[0069] A study was conducted using the system according to the present disclosure to detect movement trajectories corresponding to selected predefined trajectories. In this study, two participants with tetraplegia participated. Participant 1 was a 32-year-old male with a C4 / C5 ASIA (American Spinal Injury Association) B injury that was injured 6 years ago. Participant 1 participated in 10 sessions, 7 of which were used to record 2D and 3D arm movement trajectories. In the remaining 3 sessions, the grasping intent was decoded online (in real time) and used to drive a custom neuromuscular stimulator with textile-based electrodes 12a, 12b, ... 12n housed in a sleeve. This enabled the participant to perform functional movements (e.g., eating a granola bar). Participant 2 was a 28-year-old male with a C4 / C5 ASIA A injury that was injured 10 years ago. Participant 2 participated in 3 sessions, including 2 training sessions and 1 online testing session.

[0070] Participants were initially seated with their hands on a table. A wireless sensor module was attached to the wrist of the participant's arm using Velcro. The sensor module included motion sensors 16a, 16b, ... 16n and MCU 18 as disclosed in the above embodiment. Although both participants' impairments were bilateral, residual movement was possible in each participant, allowing them to reach with at least one arm, which was ultimately utilized in the study.

[0071] During the study, verbal cues associated with different 2D and 3D movement trajectories were randomly invoked by participants. Participants were instructed to perform the reaching trajectories by starting the movement at the edge or corner of the table and then moving to the center, with smooth movements lasting up to 2 seconds. Three different 3D reaching trajectories were trained: a lateral arc, a vertical arc (e.g., reaching for a pen or marker placed on a table), and a bottle-opener type movement. In addition, four 2D trajectories (performed in the horizontal plane) corresponding to well-known English and Greek letters: S, ε (epsilon or E), γ (gamma), and M were trained. To minimize participant fatigue, the experiment was performed in blocks of 18–20 trials, with sufficient breaks between blocks. Participants were initially asked to perform only the S and ε trajectories, as these were easily memorized and did not cause fatigue. Then, after participants had become familiar with moving their arms, additional 2D and 3D trajectories were added to the study, such that the percentage of 2D trajectories (especially S and ε) was higher than the remaining trajectories in the final dataset.

[0072] For participant 1, over 250 training samples were recorded across seven motion trajectories, and for participant 2, 96 samples were recorded from five motion trajectories. Trials with noisy sensor data or incorrect labels were visually identified and removed from the training set. A five-fold stratified cross-validation scheme was chosen for the evaluation of the DTW-based and LSTM-based classifiers. Figure 7 shows the mean ± standard deviation (SD) classification accuracy for the two participants. The bar graphs provide a comparison of the classification accuracy (mean ± SD) using the two methods, DTW and LSTM. Both offline (2D&3D) and online (2D only) arm trajectories were used for performance evaluation. The statistical significance threshold was set at p<0.05.

[0073] In the offline scenario, both the DTW-based classifier and the LSTM-based classifier performed well for 2D trajectories, achieving accuracies of 94±5% and 98±3%, respectively. However, for offline 3D trajectories, LSTM outperformed DTW, with an accuracy of 99±3% for LSTM, while DTW was 83±16%. Using a two-tailed Wilcoxon rank-sum test, the LSTM-based classification accuracy was significantly higher than DTW in both cases (p<0.05). In Figure 7, we also show the online performance of the DTW-based classifier for 2D trajectories. During online classification, we achieved an accuracy of 79±5% when comparing between two trajectories (e.g., Sv / sε) or between a single trajectory and the remaining trajectories (e.g., Mv / time remaining (sec)). To further evaluate the performance of each classifier for type I and II errors, a cumulative confusion matrix was calculated by adding up the confusion matrices from each weight for each participant. The resulting confusion matrices are shown in Figure 8 for both classifiers and both types of trajectories.

[0074] For the DTW-based classifier, type I errors were more frequent in 3D trajectories than in 2D trajectories. The percentage of type I errors was highest for the corkscrew trajectory (37.8%), followed by the vertical arc trajectory (14%), the ε trajectory (10.2%) and the M trajectory (10%). For type II errors, the DTW-based classifier misclassified the vertical arc trajectory (14.5%), the lateral arc trajectory (13.8%) and the S-shaped trajectory (8.33%) compared to the remaining classes. For the LSTM-based classifier, type I and II errors were very few, ranging from 0 to 3% for almost all trajectories, except for the M trajectory, which had a type I error rate of 40%. Our guess is that since we only ran 10 trials to train on the M trajectory, the sample set was too small for the LSTM classifier to distinguish this trajectory from other classes with larger samples.

[0075] As a further example, a system according to an embodiment of the present disclosure was tested with a paralyzed individual who had residual shoulder and arm movement but no residual hand movement. As shown in Figure 6, the device recognized the natural reaching movement of the individual's arm and shoulder and stimulated the individual's thumb adductor and abductor muscles to grasp a pen propped up in one cup. The individual was able to use the residual arm and shoulder movement to pick up the pen and move it to the second cup while the device continually activated the patient's muscles to maintain the grip.

[0076] As another example, in testing a system according to an embodiment of the present disclosure, a healthy subject was used to predict muscle activation during a reach-to-grasp movement based on training an LSTM network with EMG signals. The subject was fitted with EMG sensors on the flexor and extensor muscles of the ring finger and an IMU 16a on the wrist. Signals from the EMG and IMU were pre-processed by a microcontroller 18 implemented on a circuit board (Arduino® Nano 33BLE). Data from the circuit board was wirelessly communicated to a computer 20 running the LSTM network as described in the above embodiment. Data from the IMU and EMG were provided to the LSTM network while the subject repeatedly performed reach and grasp movements. After training the LSTM, the LSTM was able to predict the timing and amplitude of flexor and extensor muscle activity based on the subject's wrist trajectory.

[0077] According to other embodiments, the device according to the present disclosure may be used to enable lower limb movement. In one such embodiment, an IMU is fixed to the patient's hips. 2D and 3D hip movement detection is performed by analyzing data from the IMU. Again, training of the algorithm can be achieved by wearing an IMU, a camera observing the limb position and movement, flexion / joint angle sensors at the leg joints, and / or EMG sensors on the muscles to be stimulated in paralyzed individuals as a garment on a healthy person. The use of hip movement allows activation of muscles using NMES (e.g., to modify the person's gait or to facilitate walking when the muscle is weak, paralyzed, or has foot drop). During normal walking, the left hip / upper body traverses a 2D "C" shaped curvilinear trajectory in space, followed by lifting of the right foot. According to one embodiment, this telltale signature trajectory is recognized and used to trigger muscle stimulation patterns in the right leg for stepping sequence assistance. NMES electrodes can be placed on any muscle that actuates the joint of interest. For example, if a device according to the present disclosure is used to facilitate rehabilitation after knee surgery, actuators can be placed on the quadriceps, hamstrings, calf and foot extensors to aid in muscle stimulation to improve the wearer's gait. Stimulation can be combined with the person's use of their arms to partially support their own weight on a walker or parallel bars to aid in hip / upper body movement. The trajectory of the right hip is then detected and used to stimulate the muscles of the left leg.

[0078] Systems according to embodiments of the present disclosure may be integrated with gloves, shoes, and other apparel that contain force sensors. Such sensors detect contact and pressure between the wearer's hands and a grasped object, or monitor foot placement during stride. Such apparel may also include flexion / angle sensors at elbows, wrists, knees, ankles, or other joints, providing trajectory, orientation, motion information, and / or intent-related data to the system (when training machine learning algorithms in able-bodied users).

[0079] According to another embodiment of the present disclosure, information about the trajectory, orientation and position of the patient's limbs is collected and recorded by the system. Such information is used to track the trajectory and / or joint movement (or range of motion) of body parts during physical therapy. The system according to the present disclosure provides medical professionals with a low-cost method for tracking progress and characterizing movement (e.g., the overall movement of the arm in space) in the rehabilitation of stroke or spinal cord injury patients. Such a system is less expensive and less bulky than current methods that rely on expensive robotic systems or table-sized devices to monitor limb position and movement. In addition, machine learning algorithms can compare the patient's movements with those of healthy volunteers and other patients at various stages or grades of recovery, or classify the patient's movements. This information can enable professionals to optimize treatment, provide better feedback to patients, and indicate the patient's progress in recovery.

[0080] During training of user-specific (custom) trajectories, voice recognition, brain / computer interface (BCI)-non-invasive or invasive (EEG), touchpad, and / or healthy hand / leg movements can be used to initiate training or select the desired action or hand or leg movement (with which the trained trajectory is associated). Furthermore, pre-trained trajectory profiles can be stored in the device / system, making training unnecessary. For example, letters, numbers, and patterns that are known to the user can be used and automatically recognized without user-specific training.

[0081] According to another embodiment, as an alternative or addition to stimulating muscles to perform actions in response to the recognized trajectory, the system may also apply therapeutic stimulation anywhere in the patient's nervous system. According to a further embodiment, one or more of the electrodes 12a, 12b, 12c...12n are adapted to apply stimulation currents to the patient's peripheral nerves or to the patient's central nervous system (CNS) for various applications (e.g., movement / sensation restoration and chronic pain). It is well known that neural stimulation is also effective for pain treatment through implanted transcutaneous stimulation devices. It is also well known that certain types of movements (raising the arms or bending forward from the waist) can cause pain. According to some embodiments of the present disclosure, translational and / or rotational movements of body parts that can cause pain trigger stimulation for the reduction of pain caused by the detected movements.

[0082] According to embodiments of the present disclosure, triggering various types of stimulators with movement / trajectory recognition may provide many benefits. For example, vagus nerve stimulation has been shown to improve the effectiveness of upper limb rehabilitation. According to one embodiment, the system triggers vagus nerve stimulation cervically (neck) or auricularly (ear) during movement rehabilitation for stroke, SCI, traumatic brain injury, MS, etc. Such therapeutic stimulation may be added to other nerves (e.g., trigeminal and other cranial nerves or peripheral nerves leading to target muscles). The system according to the present disclosure may also be used to trigger, control and / or regulate various forms of brain stimulation (e.g., TMS (transcranial magnetic stimulation) and tDCS (transcutaneous direct current stimulation), tACS (transcutaneous alternating current stimulation), TENS (transcutaneous electrical nerve stimulation), or spinal cord stimulation (sending cues down the spinal cord and up to the brain)), which promotes neuroplasticity, recovery and / or pain reduction after stroke or traumatic brain injury.

[0083] Due to spinal cord injury, signals from the brain in spinal cord injury patients may be blocked or attenuated before reaching the muscles. Stimulation occurring at or near the damaged spinal cord pathways can lead to excitability in those pathways, facilitating movement and rehabilitation in spinal cord injury patients. Known systems for application of spinal cord stimulation are often controlled manually through a control pad or device (rather than by patient movement). According to an embodiment of the present disclosure, one or more electrodes 12a, 12b, 12c, ... 12n are positioned epidurally or preferably percutaneously on the patient's spinal cord. The system senses the specific trajectory taken by the patient during physical therapy, and in addition to applying NMES stimulation to cause the muscles to perform the desired movement of the paretic limb, the system triggers percutaneous spinal cord stimulation to increase (by increasing the excitability of interneurons) nerve signals that are decreased due to spinal cord injury. Similarly, one or more electrodes 12a, 12b, 12c,...12n may be positioned above, on or below the site of spinal cord injury to provide stimulation to the spinal cord injury and / or pathways above and / or below the injury to help repair neurons damaged due to the injury and / or strengthen neuronal connections. Combining voluntary patient movement with such stimulation allows the patient to control their own stimulation patterns, further promoting neuroplasticity and movement and / or recovery of sensory function.

[0084] In the case of patients who have suffered a stroke, application of electrical or transcranial magnetic stimulation to or near the site of injury in the brain or brainstem can help repair neuronal damage. According to a further embodiment of the present disclosure, one or more electrodes are positioned on the scalp or on a magnetic coil on the scalp. In response to detection of a movement trajectory, a stimulation signal is applied to these electrodes or coils (as an alternative or preferably in addition to a NMES signal to move the patient's paretic or appendicular limb). Such brain stimulation, when combined with the patient's intention to move the paretic or appendicular limb, can help repair some of the motor neuron function that was damaged due to the stroke.

[0085] Because the systems disclosed herein are relatively inexpensive, portable, and can be controlled solely by the patient without the assistance of a therapist or other professional, patients can remain at home wearing the device (e.g., wearable sleeve(s), patch(es)), leading to an increase in the number of hours per week available for rehabilitation.

[0086] FIG. 9 illustrates another embodiment of the present disclosure. A prosthetic hand 100 is attached to the arm of an amputee. The prosthetic hand 100 includes a sensor housing 10. The sensor housing 10 may include sensors 16a, 16b, ... 16n as described above for detecting acceleration, velocity, position and rotation of the wearer's arm. A controller 21, which integrates the functions of the MCU 18 and computer 20 described in the above embodiment, is connected to the sensor array and receives signals indicative of the movement trajectory performed by the wearer using healthy joints (e.g., shoulder, torso and upper arm). As in the above embodiment, the controller 21 determines whether the wearer has performed a movement corresponding to the intended activation of the hand. The controller 21 is connected to actuators 112a, 112b, ... 112n. These actuators drive the movement of the fingers of the prosthesis 100. As in the above embodiment, one or more pre-determined trajectories are associated with specific movements of the hand. For example, if the wearer moves their upper arm, shoulder and torso to create a “rainbow arc” around the prosthesis, as described above, this may indicate an intent to perform a pick-up grasp, and the actuators 112a, 112b, ... 112n are energized to move the fingers to perform this intended grasping movement.

[0087] The embodiment shown in Fig. 9 is of a prosthetic hand 100. The present disclosure is not limited to hand prostheses. Other types of prostheses can be controlled by a device according to the present disclosure. For example, a foot prosthesis can be provided that senses the walking movement of the wearer's legs and activates actuators to orient the foot in a manner synchronous with the wearer's walking.

[0088] According to another embodiment, the system according to the present disclosure may assist in the training or physical therapy of normally able-bodied individuals for the provision of active resistance in athletics. For sports training or physical therapy purposes, the motion / trajectory recognition of various limbs is used to stimulate non-paretic muscles. For example, the rotational velocity and linear acceleration of the individual's forearm is detected using IMU and / or gyro data from a sensor mounted on the forearm as part of a sleeve, patch or other attachment. This motion is primarily performed by the biceps. In response to the detected motion, the system triggers antagonistic muscles, including the triceps, to provide active resistance to the biceps (proportional to the measured rotational velocity of the forearm). According to one embodiment, the proportionality constant is a configurable parameter, thereby allowing the user to modify the resistance.

[0089] Although exemplary embodiments of the present disclosure have been described and illustrated above, it should be understood that these are illustrative of the present disclosure and should not be considered as limiting. Additions, omissions, substitutions and other modifications are possible without departing from the spirit or scope of the present disclosure. Thus, the present disclosure should not be considered as limited by the above description.

Claims

1. A device, comprising: one or more motion sensors that generate one or more respective motion signals indicative of a motion of a first body part of the human; a muscle stimulator that generates one or more stimulation signals for displacing a second body part by one or more muscles to perform at least one action; a processor coupled to the one or more motion sensors and the muscle stimulator, the processor including a data store including at least one expected trajectory associated with the human's intent to perform the at least one action, the processor comprising: receiving the one or more signals from the one or more motion sensors; Calculating an actual trajectory of the first body part; comparing the actual trajectory to the expected trajectory, and based on the comparison, activating the muscle stimulator to displace the second body part to perform the at least one action; A processor; Including, the device.

2. The device of claim 1 , wherein the processor calculates a difference between the actual trajectory and the expected trajectory and activates the muscle stimulator based on the difference.

3. 2. The device of claim 1, wherein the at least one behavior includes a plurality of behaviors, the at least one expected trajectory includes a plurality of expected trajectories, each of the plurality of expected trajectories being associated with at least one of the plurality of behaviors, the processor compares the actual trajectory with the plurality of expected trajectories to identify a first trajectory associated with a first behavior of the plurality of behaviors, and the processor activates the muscle stimulator for performance of the first behavior.

4. 13. The device of claim 1, further comprising an input device coupled to the processor, the input device adapted to receive a feedback signal, the feedback signal indicating that the action was an intended action of the human.

5. The device of claim 1 , wherein the processor generates the expected trajectory based on a set of training movements.

6. 6. The device of claim 5, wherein the one or more stimulation signals for performance of the at least one action include a pattern of stimulation signals, the pattern of stimulation signals being determined from sensed muscle displacements during the set of training movements.

7. The device of claim 6 , wherein the sensing of muscle displacement is performed using one or more of an electromyographic sensor, a camera, an inertial unit, a flexion / joint angle sensor, and a force sensor.

8. 10. The device of claim 1, wherein the comparison performed by the processor employs one or more of a support vector machine (SVM) algorithm, a handwriting recognition algorithm, a dynamic time warping algorithm, a deep learning algorithm, a recurrent neural network, a shallow neural network, a convolutional neural network, a convergent neural network, or a deep neural network.

9. The device of claim 7 , wherein the processor performs the comparison using a long-short-term memory type recurrent neural network.

10. The device of claim 5 , wherein the set of training movements is performed by a second person.

11. The device of claim 5 , wherein the set of training movements is performed by the human using a body part laterally opposite the first body part.

12. 2. The device of claim 1, wherein the motion sensor is positioned on the human's arm and the muscle stimulator is adapted to stimulate muscles for moving one or more fingers of the human's hand for performance of a grasping movement.

13. The device of claim 1 , wherein the expected trajectory takes an alphanumeric shape.

14. 13. The device of claim 12, further comprising an orientation sensor connected to the processor and adapted to monitor an orientation of the first body part, wherein a force applied by the grasping movement is dependent on an amplitude of the stimulation signal, and wherein the processor adjusts the amplitude of the stimulation signal based at least in part on an output of the orientation sensor.

15. The device of claim 14 , wherein the processor adjusts the grasping movement to be a key grasp, a cylindrical grasp, or a vertical grasp in response to an output of the orientation sensor.

16. 13. The device of claim 12, further comprising a camera coupled to the processor and positioned proximate the hand to capture an image of an object to be grasped, the processor adjusting the grasping movement based in part on the image.

17. 13. The device of claim 12, wherein the processor further comprises a close delay timer, the processor delaying stimulation of the grasping movement for a predetermined period of time at the end of the actual trajectory determined by the close delay timer.

18. The device of claim 12 , wherein the processor is responsive to a post-grasp signal from the motion sensor to generate stimulation of the hand for performance of a post-grasp activity.

19. 20. The device of claim 18, wherein the post-grasp action is opening the hand to release the grasp.

20. 20. The device of claim 18, wherein the post-grasping signal is one or more taps on the surface of the grasped object.

21. A device, comprising: one or more motion sensors that generate one or more respective motion signals indicative of a movement of a first body part of the human; a muscle stimulator generating a stimulation signal adapted to generate or increase contraction of a first muscle, the first muscle being a neurologically damaged muscle, a paralyzed muscle, a partially paralyzed muscle, or a normal muscle; a processor coupled to the sensor and the muscle stimulator, the processor including a data store including at least one expected trajectory associated with the human's intent to contract the first muscle, the processor further comprising: receiving the one or more motion signals from the one or more sensors; Calculating an actual trajectory of the first body part; Comparing the actual trajectory to the expected trajectory; determining an intent to contract the first muscle based on the comparison; causing the stimulator to do one or more of the following: generating a contraction of the first muscle; assisting the contraction of the first muscle; and generating an antagonistic contraction of a second muscle, the contraction of the second muscle opposing the movement caused by the contraction of the first muscle; A processor; Including, the device.

22. 22. The device of claim 21, further comprising a nerve stimulator coupled to and operable by the processor, wherein in response to the processor determining an intent to contract the first muscle, the nerve stimulator applies a nerve stimulation signal to the human nerve.

23. 23. The device of claim 22, wherein the human nerve is selected from one or more of the vagus nerve, the trigeminal nerve, a cranial nerve, a peripheral nerve leading to the first muscle, and the human spinal cord.

24. 24. The device of claim 23, wherein the nerve is the spinal cord and the neurostimulator includes a percutaneous electrode positioned above, on or below the human's spinal cord injury.

Images