A system and method for determining user intent from limb, body movement, or trajectory and using it to control neuromuscular stimulation or prosthetic device movement.

Abstract

To provide a system and a method for determining user intention from limb or body motion or trajectory to use them to control neuromuscular stimulation or prosthetic device operation.SOLUTION: Disclosed is a device for restoring motion to a person's paralyzed body part. The device senses movement of another part (e.g., the person's arm or shoulder), of the person's body not affected by the paralysis. Motion sensors generate motion signals as the person moves the non-paralyzed body part. A processor stores information associating a predefined trajectory with a particular action (e.g., closing the hand to grasp an object). The processor monitors the motion signals and, when the motion corresponds with the predefined trajectory, the processor energizes muscle stimulators connected with muscles to control the paralyzed hand to perform the action (e.g., to place the hand around an object, close the hand, and grasp the object).SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to systems, apparatuses, applications and methods for assisting persons with partial disabilities by providing voluntary movement of paralyzed joints or prosthetic organs. Said provision of voluntary movement is performed by determining the person's intention to move a joint or device from analysis of a limb or body movement of the person's healthy joint. More specifically, the system, method or device to which the present disclosure relates determines that the overall movement (translational and / or rotational movement) or trajectory of a neurologically intact limb or other body part is sufficient to determine a user's intention to perform an action using a paralyzed or missing external limb, and in response to the determined intention, by stimulating a neurologically paralyzed part (via the nerves and / or muscles that control said part) or a neural target (a nerve, the spinal cord or the brain), the device achieves recovery of movement or function by promoting nerve growth / regeneration or connection strengthening, or control of prosthetic valve replacement to perform said action. A device according to an embodiment of the present disclosure detects a reaching trajectory of a hand of a person's arm, identifies the person's intention to grasp an object, and activates or adjusts a neuromuscular stimulation device (NMES) to allow opening and closing of the person's hand that is paralyzed in a normal state (or to activate the person's robot / prosthetic hand), so as to enable grasping and holding of the object.

Background Art

[0002] (Description of Related Art) In the United States alone, approximately 5.4 million people live with paralysis. The two main causes of paralysis are stroke and spinal cord injury. In the United States, the annual number of new spinal cord injury cases exceeds 17,700 (NSCISC, 2019). Many of these injuries lead to incomplete quadriplegia (48%) and complete quadriplegia (20%), which severely affect the movement of the arm and hand of survivors and lead to a decline in quality of life.

[0003] For individuals living with quadriplegia, hand function is a top priority. A variety of invasive and non-invasive neuromuscular electrical stimulation (NMES) devices have been proposed for the rehabilitation or induction of upper limb and hand movements. However, these known systems have drawbacks. In freehand systems, shoulder movements connected to switches are used; these switches trigger selected hand movements through electrical muscle stimulation via implanted electrodes. Switch operation can be cumbersome, and the operation of the muscle stimulator may force the user to make unnatural movements. Such movements can draw attention to the user's disability and affect how others perceive the user. Furthermore, the repertoire of hand movements the user can perform may be limited by the number of switches that can be activated by the user's shoulder muscles.

[0004] Other systems may require surgical intervention. For example, some systems rely on implanted electromyography sensors to detect a patient's intention to move a paralyzed or surgically amputated joint. Corticobrain / computer interfaces (BCIs) are used to control NMES devices by recording and decoding motor activity in the brain, typically enabling voluntary control of a paralyzed hand. These approaches require the implantation of electrodes or other structures into the user's body, which can lead to increased medical risks and significant costs for the user. [Overview of the project]

[0005] This disclosure relates to devices and methods for addressing these difficulties. What patients living with paralysis desire is to integrate into society while avoiding attention to their disability as much as possible. Although rehabilitation can restore at least partial mobility in some patients, restoring fine motor control (e.g., a user reaching out to grasp an object, e.g., a drinking glass or food) can be difficult or impossible. This disclosure enables patients with impaired grasping movements using their hands to perform tasks (e.g., eating independently) without relying on devices (e.g., hand-mounted tools), and to carry out daily activities.

[0006] In patients living with paralysis resulting from stroke, spinal cord injury, or other conditions, movement of the hands and / or legs may be lost, but movement of other body parts is often possible. For example, in the case of C5 level spinal cord injury, the most common level of injury in patients with quadriplegia, hand movement is significantly impaired, but shoulder movement and elbow flexion / extension remain. Similarly, while overall arm (shoulder and elbow) movement after stroke is often recoverable through intensive rehabilitation, recovery of hand movement remains unresolved. Finally, in the case of paraplegia or stroke victims, arm movement or torso / hip movement may be possible, although they may be unable to use their legs or may not be able to use them at all, or may develop foot drop (inability to flex or extend the ankle).

[0007] Disclosed herein are methods and systems for restoring voluntary control of a paralyzed joint and / or external device of a user (by sensing and recognizing the movement and trajectory of a healthy joint still possessed by the user). The system uses a computerized algorithm (e.g., machine learning adapted to the user's specific body movements) to identify the user's intention to perform an action using the paralyzed joint or a joint replaced with a prosthesis. The detected body movements and trajectories can then be used to produce a variety of desired results. According to one embodiment, such a system determines a person's intention to reach out and grasp an object, and activates an NMES device to open and close the user's paralyzed hand to grasp and hold the object.

[0008] The devices included in this disclosure sense and recognize limb trajectories (e.g., extending a hand controlled by remaining shoulder and elbow movements) as well as other body movements, positions, or orientations, and activate muscles in the paralyzed body part through electrical stimulation via electrodes or electrode arrays to perform specific activities (e.g., pinching movements of the hand to "grasp a key") or energize actuators on prosthetic body parts. A variety of predefined trajectories and limb or body movements can be combinations of translational and rotational movements and can be stored. Each trajectory or movement is associated with a different action. Based on the recognized movements, the devices according to embodiments of this disclosure can also be used to control external devices (e.g., computers or electric wheelchairs). Furthermore, since a number of distinct trajectories can be identified by different actions, this leads to an expansion of the repertoire of actions available to the user.

[0009] Devices further included in this disclosure recognize movement around healthy joints (e.g., glutes, lumbar spine, and knees), identify movements associated with a person's walking, and apply stimulating signals to muscles simultaneously with the person's walking. Such devices may be used to restore more effective walking movements (in cases where movement of the person's foot, ankle, or leg is impaired due to nerve damage). Such devices may be used to strengthen the muscles necessary for walking as part of rehabilitation therapy before surgery (e.g., before hip or knee replacement surgery) and / or after surgery.

[0010] In another embodiment, instead of or in addition to the energization of electrodes or prosthetic tubes to enable movement, the system according to the Disclosure delivers electrical stimulation to a nerve injury site or nerve pathway connected to a nerve injury (e.g., spinal cord, brain, or peripheral nerve). According to the system according to the Disclosure, by providing electrical stimulation with electrodes positioned percutaneously or epidurally on or near the site of spinal cord, nerve, or brain injury, while simultaneously moving the affected limb, the repair of damaged motor fibers, nerves, or neurons may be supported. In the case of spinal cord injury, the system may also support the healing of damage to sensory fibers, nerves, or neurons by providing electrical stimulation percutaneously or epidurally on, above, or near the site of injury.

[0011] Sensors on the arms, legs, and / or body can recognize a variety of two-dimensional and three-dimensional (2D / 3D) movements (translational acceleration relative to the Earth's magnetic field, rotational speed, and orientation). According to some embodiments, such movements are detected by an inertial unit (IMU) with a total of 3 to 9 degrees of freedom. According to other embodiments, visual images of the movements can also be recognized. Just as a child uses a sparkler to draw letters, numbers, and patterns in the air, the device recognizes smooth, natural curved arms reaching trajectories and pre-trained patterns (e.g., well-known script numbers and letters). The user can then perform movements of their choosing or natural outstretched hand trajectories, and once these movements are recognized, they are used to control a variety of neuromuscular stimuli and prosthetic / robot devices that facilitate movement of paralyzed joints. In the arm, the arm's movement trajectory is driven by residual shoulder movement and can be used to stimulate multiple wrist, hand, and finger movements (or external devices (e.g., computer, stereo)) or to drive robotic control.

[0012] In addition to enabling patients to grasp objects using their remaining motor skills, devices according to embodiments of this disclosure can improve neurological function by providing feedback to the patient's central nervous system to associate healthy joint and limb movements with activation of paralyzed body parts. Thus, driving neuromuscular or robot-driven movements in paralyzed joints using such devices can provide supportive, rehabilitative, and therapeutic applications for stroke, spinal cord injury, and other neurodegenerative conditions. This approach also has applications in general physiotherapy after injury or surgery of the hands, feet, legs, or other body parts.

[0013] Furthermore, embodiments of the disclosure may be used for measuring, tracking, and recognizing the temporal quality of limb / movement trajectories in rehabilitation applications (e.g., through machine learning algorithms as disclosed). Joint movements are captured, recorded, and recognized or graded, allowing physical therapists to monitor patient progress and individually tailor treatment to address specific parts of movement that are problematic. Machine learning or other forms of artificial intelligence (e.g., deep learning) can be used to analyze aggregate data (from a large number of anonymous patients) to discover general patterns and quantitative information (indicating progress or regression and issues that can be flagged as targets for re-examination or corrective action).

[0014] According to one embodiment, the disclosed device includes one or more motion sensors, each generating one or more motion signals indicating the movement of a first body part of a human; a muscle stimulator, each generating one or more stimulation signals for displacing a second body part by one or more muscles to perform at least one action; and a processor connected to the one or more motion sensors and the muscle stimulator. The processor includes a data storage unit, which includes at least one expected trajectory associated with the human intention to perform the at least one action. The processor receives the one or more signals from the one or more motion sensors, calculates the actual trajectory of the first body part, compares the actual trajectory with the expected trajectory, and activates the muscle stimulator based on the comparison to displace the second body part to perform the at least one action. The processor may calculate the difference between the actual trajectory and the expected trajectory, perform the comparison, and activate the muscle stimulator based on the difference.

[0015] According to one embodiment, the at least one action includes a plurality of actions, and the at least one expected trajectory includes a plurality of expected trajectories. Each of the plurality of expected trajectories is associated with at least one of the plurality of actions. The processor compares the actual trajectory with the plurality of expected trajectories to identify a first trajectory associated with a first action among the plurality of actions, and the processor activates the muscle stimulator to perform the first action. The device may include an input device connected to the processor, the input device being adapted to receive a feedback signal. The feedback signal may indicate that the action is an action intended by the person. The processor may generate the expected trajectory based on a set of training movements. One or more stimulation signals for performing the at least one action may include a pattern of stimulation signals, the pattern of stimulation signals may be determined from muscle displacements sensed during the set of training movements. Sensing of muscle displacements may be performed using one or more of the following: an electromyograph sensor, a camera, an inertia unit, a flexion / joint angle sensor, and a force sensor. The above comparison by the above processor may be performed using one or more of the following: support vector machine (SVM) algorithms, handwriting recognition algorithms, dynamic time warping algorithms, deep learning algorithms, recurrent neural networks, shallow neural networks, superimposed neural networks, convergent neural networks, or deep neural networks. The above processor may perform the above comparison using long- and short-term memory type recurrent neural networks. The above set of training movements may be performed by a second person. The above set of training movements may be performed by the above person using the opposite body part in a lateral direction to the above first body part. The above motion sensor may be placed on the arm of the above person, and the above muscle stimulator may be adapted to stimulate muscles to enable grasping movements by moving one or more fingers of the above person's hand. The above expected trajectory may take the shape of alphanumeric characters.

[0016] According to one embodiment, the device includes an orientation sensor, which is connected to the processor and may be adapted to monitor the orientation of the first body part. The force applied by the grasping movement may depend on the amplitude of the stimulus signal, and the processor may adjust the amplitude of the stimulus signal at least in part on the output of the orientation sensor. In response to the output of the orientation sensor, the processor may adjust the grasping movement to be a key grasp, cylindrical grasp, or vertical grasp. A camera, which may be included in the device, is connected to the processor and positioned near the hand to capture an image of the object to be grasped. The processor may adjust the grasping movement at least in part on the image. The processor may include a close delay timer, which may delay stimulating the grasping movement for a predetermined period of time at the end of the actual trajectory determined by the close delay timer. In response to a post-grasp signal from the motion sensor, the processor may stimulate the hand to perform post-grasp activity. Post-grasp activity may involve opening the hand and releasing the grip. Post-grasp signal may involve tapping the surface of the grasped object one or more times.

[0017] According to another embodiment, a device is disclosed. This device includes one or more motion sensors, each of which generates one or more motion signals indicating the movement of a first body part of a human; a muscle stimulator, each of which generates a stimulation signal adapted to cause or increase the contraction of a first muscle, the first muscle being a neurologically damaged muscle, a paralyzed muscle, a partially paralyzed muscle or a normal muscle; and a processor connected to the sensors and the muscle stimulator. The processor includes a data storage unit, the data storage unit includes at least one expected trajectory associated with the human intention to cause the muscle to contract. The processor receives one or more motion signals from one or more sensors, calculates the actual trajectory of the first body part, compares the actual trajectory with the expected trajectory, determines an intention to contract the muscle based on the comparison, and causes the stimulator to perform one or more of the following: contract the first muscle, assist the contraction of the first muscle, and generate an antagonistic contraction of a second muscle, the contraction of the second muscle counteracting the movement caused by the contraction of the first muscle. The device may include a nerve stimulator, which is connected to and operable by the processor, and in response to the processor determining an intention to contract the first muscle, the nerve stimulator may add a nerve stimulation signal to the human nerve. The human nerve may be selected from one or more of the vagus nerve, trigeminal nerve, cranial nerve, peripheral nerves connected to the first muscle, and the human spinal cord. The nerve may be the spinal cord, and the nerve stimulator may include a transcutaneous electrode positioned above, on, or below the human spinal cord injury.

[0018] A device disclosed in one embodiment includes one or more motion sensors, each of which generates one or more motion signals indicating the movement of a first body part of a human; a prosthetic limb including actuators configured to modify the configuration of the prosthetic limb to perform an action; and a processor connected to the one or more motion sensors and the actuators. The processor is a data storage unit, which includes at least one expected trajectory associated with the human intention to perform the action. The processor receives the one or more motion signals from the one or more motion sensors, calculates the actual trajectory of the first body part, compares the actual trajectory with the expected trajectory, and, based on the comparison, activates the actuators to modify the configuration of the prosthetic limb to perform the action. The prosthetic limb may include a prosthetic hand, and the actuators may include one or more wrist actuators and finger actuators. [Brief explanation of the drawing]

[0019] A deeper understanding of the accompanying benefits can be obtained by referring to the following detailed description along with the attached drawings, and a deeper understanding of this disclosure, as well as many of those benefits, can be easily obtained.

[0020] [Figure 1] This image shows a test being conducted to measure finger dexterity using the arm and hand of a person equipped with a device according to an embodiment of the present disclosure.

[0021] [Figure 2] This is a block diagram of a system according to one embodiment of the present disclosure.

[0022] [Figure 3] Figure 1 shows the position, speed, and acceleration of the arm of a person equipped with a device as shown in the figure, as the person moves their arm along a "C" shaped motion path.

[0023] [Figure 4]It shows the position, velocity, and acceleration of the wrist of a subject provided with a device as shown in FIG. 1 when the subject moves the arm along a movement path shaped like "the numeral 3".

[0024] [Figure 5] It shows a system according to an embodiment of the present disclosure integrated with a wearable patch.

[0025] [Figure 6] It shows a person's arm and hand equipped with a system according to an embodiment of the present disclosure moving a pen from one location to another location.

[0026] [Figure 7] It is a graph showing the performance of a device according to an embodiment of the present disclosure for identifying movement of a patient's limb using a predefined trajectory.

[0027] [Figure 8] It shows a comparison of confusion matrices using different machine learning algorithms for identifying predefined trajectories in an embodiment of the present disclosure.

[0028] [Figure 9] It shows a prosthetic limb including a device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF EMBODIMENTS

[0029] Some patients who have suffered nerve damage (e.g., stroke or spinal cord injury) have lost the ability to control movement of part of their body, but retain the ability to move other body parts. In some cases, a patient may be able to bend the elbow by moving their shoulder and upper arm using remaining limb movement, while having lost the ability to control hand movement, for example, for grasping an object. In other cases, the patient may have lost the ability to coordinate their knee and ankle movements, while retaining movement of the waist. In the case of an amputee patient, the patient may retain full function of the residual site of the surgically amputated limb.

[0030] The system according to embodiments of this disclosure senses and recognizes the trajectory and movement of remaining limbs in space through machine learning methods and identifies the user's intention to perform a specific action. Sensors on the arms, legs, and / or body enable the recognition of a variety of two-dimensional and three-dimensional (2D / 3D) movements (e.g., translation, rotation, or combinations thereof). Circuits included in the system deliver NMES signals to the muscles that control the movement of the paralyzed body part or operate a robot / prosthetic limb to restore control of the hand / arm or foot / leg.

[0031] In a further embodiment, the system detects a streamlined, natural curved path of movement of a functional body part, typically associated with a desired action, and causes the paralyzed body part to perform that action. For example, in the case of a patient with residual movement in their shoulder and upper arm, the device recognizes an outstretched hand trajectory and causes the patient's paralyzed hand to open and close to grasp an object. As used herein, the term “trajectory” means the overall movement of a body part (e.g., translation and / or rotation of a body part in space) as well as angular displacement of a body part around a joint (e.g., flexion of the elbow, shoulder, hip, or knee).

[0032] Different trajectories of outstretched hands may be detected, and in response, the system appropriately positions the patient's hand to match that type of outstretched hand. For example, if a patient moves their arm and shoulder forward or in a curved path (with the wrist in a neutral "handshake" position), the system compares the actual trajectory of the arm or shoulder with the expected trajectory associated with the patient to identify that the patient intends to grasp a vertically oriented object (e.g., a glass bottle or water jug ​​placed on a tabletop ("cylindrical grip")). In response to this identified intention, the system energizes NMES electrodes on the patient's forearm to activate the appropriate muscles and open the hand to grasp the object. After a certain delay, the system stimulates the muscles to cause the fingers to grip and securely hold the object. Alternatively, if the patient extends their hand in a “rainbow” arc along the vertical direction of the grip using the remaining movement of their shoulder and arm, the system recognizes that the user intends to pinch the object from above with a grasping hand movement (“vertical grasping”). The device can also indicate an intention to perform a third type of grasping (e.g., a “nail-pinching” grasp to pick up an object) by attempting to reach for the object using a “bottle opener” motion. This activates NMES electrodes that control the hand, causing the patient’s thumb and fingers to spread and then converge around the top of the object. An advantage of using the natural movement of the remaining body part to control the paralyzed body part is that the patient’s behavior more closely matches that of a healthy person. This can reduce user focus and potentially facilitate neuroplasticity and rehabilitation, for example, in stroke patients or those with recent spinal cord injury.

[0033] In detecting residual movements of this type, pre-determined trajectories may also be included, for example, when a patient performs arm movements along a "C"-shaped path. Similar to how a child draws letters, numbers, and patterns in the air with a sparkler, the device recognizes patterns. The patient moves their healthy joints along a pre-determined expected trajectory, and the system identifies that a specific action is intended. In response, the system activates NMES electrodes to generate muscle contractions in order to perform the desired action. For example, a patient might perform a "C"-shaped movement using their shoulder and upper arm when opening and closing their hand around a cylindrical object, while also performing a "C"-shaped movement when closing their hand in a pinching motion. An advantage of using pre-programmed expected trajectories is the vast number of specific movements that can be encoded. The device can be programmed to recognize both pre-trained patterns and trajectories of natural hand-extension ranges. Furthermore, it is possible to add new trajectories for novel actions to the patient's behavioral repertoire.

[0034] According to one embodiment, the device energizes NMES electrodes to stimulate appropriate muscle contractions in order to perform an intended action. According to another embodiment, the device recognizes the movement pathways of the patient's healthy body parts for the operation of a prosthetic / robot device that facilitates movement of a paralyzed joint. In the arm, the arm movement trajectory can be used to drive the stimulation or robotic control of a prosthetic hand adapted to perform multiple movements of wrist, hand, and finger movements, driven by residual shoulder movement. Such a prosthetic hand includes a combination of wrist and finger actuators. In addition, specific movements can be detected for the control of external devices (e.g., computers, stereos, electric wheelchairs, etc.). Because the number of distinct movement pathways is extremely large, the device can be used for both controlling a paralyzed body part (e.g., using the natural shoulder trajectory in an outstretching movement for controlling a paralyzed hand) and controlling external devices such as computers using pre-programmed movement pathways (e.g., a "C" shaped pathway).

[0035] The use of devices according to embodiments of this disclosure for driving robot-driven movements in neuromuscular or paralyzed joints could lead to further support, rehabilitation, and therapeutic applications in stroke, spinal cord injury, and neurodegenerative conditions. Because patients utilize residual movements in healthy body joints, the muscular and neural connections used to perform those residual movements are strengthened. In addition, with the use of the device, brain plasticity allows residual movements (both natural movements and pre-programmed movement pathways) to be associated with desired behaviors, resulting in smoother movements in the patient, similar to those of a healthy person. This approach also offers applications in general physical therapy after injury or surgery to the hands, feet, or other body parts. Furthermore, the disclosures herein could be used to measure and track the quality of limb / body movement trajectories over time in rehabilitation applications.

[0036] Figure 1 shows a patient's hand and forearm performing the “9-hole peg test” with a device according to an embodiment of the present disclosure. This test is a standard measure of hand dexterity and is well known in the art. A wearable sensor housing 10 is provided above the patient's wrist. The wearable sensor housing 10 includes motion sensors that detect the path of the patient's hand movements and the orientation of the patient's limbs. As will be described in more detail below, the sensors include an inertial unit (IMU) that detects three-axis acceleration, a gyroscope that detects rotational speed, and a magnetic sensor that detects orientation in the Earth's magnetic field. According to other embodiments, the sensors may also include joint angle sensors / flexion sensors that detect flexion of joints such as the elbow, knee, or hip. A computer (or a microprocessor embedded in the device) (not shown in Figure 1) communicates with the IMU. The computer includes a processor, memory, and / or output devices. According to the embodiment shown in Figure 1, the IMU communicates with the computer via a radio frequency Bluetooth link. In this test, the NMES electrode 12 makes contact with the patient's abductor pollicis brevis and flexor pollicis brevis muscles, which control the basic movement of the thumb.

[0037] Figure 2 is a block diagram showing an embodiment of the system in Figure 1. The sensor housing 10 includes sensors 16a, 16b, ... 16n. These may include an IMU, joint flexion / angle sensors, a camera, a gyroscope, a force sensor, and other sensors that monitor motion and orientation. A microcontroller 18 is connected to the sensors and processes the signals from the sensors to integrate the outputs from various sensors and provide trajectory data (e.g., body part orientation, gravity-corrected triaxial acceleration, or overall motion (translational and / or rotational) information). Once the output from the microcontroller 18 is provided to the computer system 20, signals indicating the path of the patient's hand movement are provided and the movement is analyzed, as described below. According to one embodiment, the microcontroller 18 and the computer 20 include radio frequency transceivers 19a and 19b (e.g., a Bluetooth device or a ZigBee protocol device for wirelessly transmitting motion data). According to another embodiment, the functions of the computer 20 may be integrated with the microcontroller 18. This microprocessor may be a neural processor or neural processing unit or tensor processor, in which case it would be optimized for low-power machine learning or deep learning, making it ideal for wearable devices (for example, the M1 processor by Apple (Cupertino, CA) or the Cortex-M55 by Arm (Cambridge, UK). Computer 20 may also include a network of locally connected computers and / or a computer system remote from the wearer (e.g., a cloud computing system).

[0038] According to the embodiment shown in Figure 1, the sensor housing 10 is worn like a wristwatch. Other types of housings may also be used. For example, the sensor housing 10 may be embedded in a cuff, sleeve, or wearable adhesive patch on a glove worn on the patient's forearm or hand (electrodes, a microprocessor or artificial neural network or AI processor, a visual indicator (e.g., one with LEDs), wireless communication, and a disposable conductive adhesive material). Such a sleeve or wearable adhesive patch may employ a joint flexion / angle sensor to detect flexion of the patient's elbow. In applications where the operation of a paralyzed limb or external device is controlled by the remaining movement of other body parts, the device may be worn as a belt (for detecting lumbar movement), as part of a hat or headband (for detecting movement and orientation of the patient's head), or embedded in clothing worn at any location on the patient's body.

[0039] The computer 20 is connected to the NMES driver 14. The NMES driver 14 generates an electric current and applies it to a plurality of NMES electrodes 12a, 12b, 12c, ... 12n. These NMES electrodes are placed on the patient's forearm or are provided within a cuff, sleeve, or adhesive patch. According to one embodiment, the NMES electrodes 12a, 12b, 12c, ... 12n are placed within a sleeve securely fitted onto the patient's forearm, as shown in Figure 5. This sleeve is described in detail below. The electrode placement configuration is selected to correspond to the anatomical structure of the forearm muscles. After being placed in position, these NMES electrodes can be mapped to the patient's muscular system.

[0040] The stimulation waveform generated by the NMES driver 14 is applied to a selected set of electrodes. The waveform parameters (e.g., waveform type (rectangular, sine, triangular, etc.), pulse width, pulse frequency, voltage, and duty cycle) are selected, and the NMES driver is configured to apply these signals in response to control signals from the computer 20. According to one embodiment, the signals are applied as a series of short bursts, separated by intervals between bursts. The NMES parameters may be selected to improve skin penetration for more precise separation of finger and thumb movements and to reduce fatigue. Mapping of these electrodes to specific muscles in the patient's forearm is performed so that the selected muscles are activated (by the stimulation signals from the NMES driver) to initiate flexion and extension of the fingers and thumb.

[0041] In the example shown in Figure 1, the NMES electrodes are attached to the patient's forearm using adhesive tape, conductive adhesive material, or hydrogel. Alternatively, the electrodes may be in a patch-like structure (with disposable adhesive hydrogel), or a cuff containing an integrated sensor and microprocessor or AI processing unit may be worn on the patient's forearm as shown in Figure 5, as described below. Other methods known to those skilled in the art may be used to connect and orient the electrodes to the patient's muscular system. In the embodiment shown in Figure 1, electrodes 12a, 12b, 12c, ... 12n are positioned to apply a stimulating current to one or more of the thumb muscles that control the thumb (abductor pollicis brevis, flexor pollicis brevis, and opponens pollicis), which triggers a variety of useful thumb movements (e.g., "pinching" (with the tip of the index finger)) and "key" gripping.

[0042] Hardware and software components included in the computer 20 receive signals from sensors 16a, 16b, ... 16n to determine the trajectory and orientation of the housing 10 and, by extension, the movement path and orientation of the patient's limbs. Based on this, the computer 20 transmits signals to the NMES driver 14 to energize electrodes 12a, 12b, 12c, ... 12n (according to a sequence that brings the intended configuration into the patient's hands). In one embodiment, the computer 20 also provides output to an output device 22 (e.g., a display monitor or screen) and receives input from one or more input devices 24 (e.g., a keyboard, computer mouse or other pointing device and / or microphone). The output from the computer may also be recorded and used by medical professionals in evaluating patient progress during physical therapy. In addition, as will be described in more detail below, this output may be anonymized and collected along with similar data from a patient population and used to train a machine learning system aimed at improving the recognition of body movements and trajectories that indicate the user's intention to perform intended actions.

[0043] In another embodiment, a computer 20, an NMES driver 14, a microcontroller 18, and an array of NMES electrodes 12a, 12b, 12c, ... 12n are integrated with a sensor housing 10 to form a portable wearable system. Such a wearable system may include a touchscreen or other input / output devices (similar to a “smartwatch” that allows a patient to interact with the system, for example, for training the system to improve the identification of patient intent). The connection between the computer 20 and the other components of the system may be a physical connection (e.g., a cable). Alternatively, the computer 20 may communicate wirelessly via a radio frequency link (e.g., Bluetooth, ZigBee) or via infrared. The computer 20 includes a memory storage device and is programmed to perform a variety of algorithms, as described in more detail below. In another embodiment, the computer 20 is also integrated with the sensor housing 10. The self-contained system obtained by such an embodiment allows the wearable system to be used independently of any wired or wireless interface.

[0044] Figure 5 shows an embodiment of the present disclosure in which an array of NMES electrodes 12a, 12b, ... 12n is integrated onto a personal patch 15. An electrical coupling layer 13 (e.g., a hydrogel layer) is provided between the electrode array and the wearer's skin. In the embodiment shown in Figure 5, the electrodes 12a, 12b, ... 12n are arranged in a pattern adjacent to the muscular system that controls the wearer's hand. According to one embodiment, other components (e.g., a sensor housing 10 including IMU sensors 16a, 16b, ... 16n, a microcontroller 18, an NMES driver 14, a computer 20, and a power supply) are also placed on the personal patch 15. This embodiment eliminates the need for cabling, allowing the user to freely move their healthy joints (in this example, the shoulder, torso, and upper arm or waist) to activate the system and stimulate intended actions in the paralyzed joints of the hand, lower limb, or foot. Eliminating cabling allows for continuous wear of the device, supporting the user's daily activities. The electrode array 12 can be programmed to map specific NMES drivers 12a, 12b, ... 12n to the wearer's muscular system so that the application of current to a specific electrode or set of electrodes leads to a specific movement (e.g., the aforementioned grasping movement of the lower limb or foot). In such mapping, machine learning techniques may be used to fine-tune muscle activation in response to the wearer's intentions.

[0045] In the example shown in Figure 1, inertial sensors (IMUs) 16a, 16b, ... 16n located within a housing 10 on the patient's wrist sense 2D and 3D arm trajectories and transmit signals to a computer 20. These signals are analyzed and compared to one or more expected trajectories associated with a desired action (by data fitting and / or machine learning algorithms performed on the computer 20). If a trajectory or movement indicating that the patient is about to perform a particular action is recognized, the computer 20 transmits to the NMES driver 14 to activate selected electrodes 12a, 12b, 12c, ... 12n to control neuromuscular stimulation patterns in the forearm to control the "opening and closing" of the hand.

[0046] These IMUs monitor and provide signals to the patient's limbs. These signals are analyzed to indicate desired movement or device action. These IMUs may detect 6-axis (acceleration and rotational velocity) or 9-axis (additional magnetic field information) movement. One or more housings containing IMUs may be positioned in various limb, body, or head locations and may be used to provide orientation and translation information for patient limb segments within the legs, hands, feet, hips, neck, head, or any other body part.

[0047] As shown in Figure 1, when the patient moves their hand in a vertical "rainbow" arc, the output of an IMU attached to the patient's wrist (or forearm) is analyzed by the computer 20 to detect this as the expected trajectory and identify that the patient is attempting to grasp a peg from the pegboard using a "key-grasping" motion. As the patient's hand approaches the end of the vertical arc trajectory, the computer 20 causes the NMES driver 14 to stimulate the patient's muscles to curl the fingers, move the thumb away from the index finger, and extend the thumb to prepare for a "key-grasping" position on the peg. When the hand reaches the end of the vertical arc trajectory, the computer 20 positions the thumb away from the side of the index finger for a time delay, allowing the patient to position their hand relative to the peg (using their remaining shoulder and arm function). At the end of this delay, the computer 20 activates the NEMS electrode on the abductor pollicis brevis muscle, thereby tightening the grip on the peg. Since the NEMS signal remains active, the peg also remains securely gripped. Sensing other overall movements of the patient's wrist or forearm (i.e., translational and / or rotational movements) may determine the patient's intention to perform other types of grasping movements. For example, the patient may indicate an intention to perform another type of grasping (e.g., a "nail grip" to pick up an object) by reaching for an object using a "bottle opener" motion.

[0048] Computer 20 maintains the muscles activated until the patient performs another movement or trajectory (indicating the patient's desire to release their grip). In one embodiment, when the patient moves their hand (in a horizontal plane parallel to the table surface) in a small clockwise or counterclockwise motion (due to residual shoulder / elbow movement), this movement is detected by an accelerometer (e.g., one or more of IMU sensors 16a, 16b, ..., 16n). This movement is interpreted by Computer 20 as indicating the patient's desire to release the peg. Computer 20 applies an NMES current to move the thumb away from the index finger, opening the grip and releasing the peg. Other movements may be used to indicate the need to release an object (e.g., pronation or supination (rotational) movements of the forearm). The user may select any pattern of movement or body motion indicating an intention to release the grip, which is recognizable to pattern recognition algorithms and / or machine learning algorithms, thereby eliciting a “hand open” neuromuscular stimulation pattern. In another embodiment, instead of, or in addition to, body movement or trajectory, a signal indicating the patient's intention to release an object is a sudden signal (e.g., one or more taps on the object's surface) which generates an accelerometer signature signal. A tapping signal is particularly advantageous when tapping occurs when a cylindrical object (e.g., a drinking cup) is grasped, as the tapping is weak and may not draw attention to the person's injury. Simple clockwise or counterclockwise circular movements in the horizontal plane may also be used to indicate that the user wishes to open their hand and release the object.

[0049] In another embodiment, instead of transmitting signals to an NMES driver, the computer 20 is connected to an electric actuator of a robotic / prosthetic hand that replaces the patient's surgically amputated hand. In this embodiment, the robotic hand is controlled to perform grasping actions in response to detection of arm trajectories.

[0050] According to one embodiment, the interpretation of the trajectory depends on the state of the system before trajectory detection. In the previous example, when an object is being grasped, a clockwise circular motion / trajectory in the horizontal plane is interpreted as a command to release the object. If the system is in a different initial state (for example, if the hand is in an "open" position), a different action may occur as a result of the clockwise circular motion (for example, to perform a grip with the fingernails).

[0051] Embodiments of this disclosure are not limited to the detection of hand or arm movements. The human body can achieve countless movements in space, along with the movement of the limbs and torso in a variety of patterns. In particular, the rotations and trajectories of human arms and legs, and even the hips and torso, in space contain a vast amount of information. Disclosed herein are methods and devices for sensing and recognizing a variety of movements in a robust and highly accurate manner for achieving a variety of desired results. Natural outstretching movements can be described (using residual shoulder movement) by specific linear or curved movements in space (sometimes involving rotation of the limbs (or body)). For example, using this approach, a quadriplegic user can move their arm along a curved path toward an object, and after this trajectory is automatically recognized, neuromuscular stimulation is triggered to cause the user's hand to open and (after a short delay) close around the object.

[0052] The IMU can also provide orientation information, which can be extremely useful. For example, if the IMU is positioned on the back of the wrist (the forearm side of the wrist, the side where the watch face is placed), the hand is in a neutral (handshake) position, and this information, combined with a specific hand-extending trajectory, can indicate that the user wants to grasp a cylindrical object (e.g., a water jug ​​or glass). 2D arm trajectories and / or orientation patterns can be used to drive a number of actions (e.g., device control and muscle stimulation patterns for various hand / leg movements). Furthermore, various trajectories can be used to control different types of grasping. As mentioned above, a rainbow-like arc trajectory drawn when a user reaches over an object placed on a table can trigger a claw-type opening and closing grasp (for picking up an object from above). A clockwise bottle-opener-type hand-extending trajectory can be used to control cylindrical grasping, and a counterclockwise bottle-opener-type hand-extending pattern can be used for picking up grasps.

[0053] In addition to the IMU, other sensors may be used to detect movement of healthy joints. According to one embodiment, a flexion sensor is provided at the elbow to provide further input. This input may be used to further identify a specific trajectory. Elbow flexion may also be used to adjust the current amplitude for neuromuscular stimulation aimed at driving gripping strength (or closing force of the robot end effector) during gripping behavior.

[0054] According to another embodiment of the present disclosure, in lieu of or in addition to the detection of natural body movement, the device detects one or more predefined trajectories. Recognizable patterns and shapes can be generated (e.g., letters, numbers, bottle opener / spiral), similar to moving a sparkler in the air. Sensors 16a, 16b, ... 16n detect movement associated with such patterns, and computer 20 analyzes the signals from the sensors to determine whether the patient performed a pattern corresponding to a particular action. The user may select any pattern of their choice and link such pattern to a variety of movements or device actions (home electronics, computers, mobile devices, robotic arms, wheelchairs). These trajectories can be used to interact with, control, or drive these devices under direct user control.

[0055] A device was constructed according to embodiments of this disclosure. Sensors 16a, 16b, ... 16n consist of Bosch TecBNO0559 axial IMUs. The sensors were connected to a microcontroller 18 (here, a 32-bit ARM microcontroller unit (MCU) from Adafruit (Feather Huzzah32)). The IMU has an internal processor and algorithms, so orientation estimation and gravity compensation are performed in real time, and linear acceleration is generated in three orthogonal directions. Linear acceleration along the X, Y, and Z axes was made available externally via an I2C interface. A flexible printed circuit board was designed to interconnect the IMU with the MCU 18. Data from the MCU was streamed continuously at 50Hz via Bluetooth to a computer 20. In embodiments where processing was performed offline, motion data was stored and processed using MATLAB® 2019a on the computer 20.

[0056] In another embodiment, the MCU18 performs data processing in real time to activate a muscle stimulator positioned on the subject's forearm. Neuromuscular stimulation is performed by a battery-operated 8-channel voltage-controlled stimulator with a stimulation pulse frequency of 20 Hz. These stimulation channels are mapped to individual electrodes or multiple electrodes on a cloth sleeve for invoking various finger flexion and extension movements. Different types of gripping (e.g., cylindrical gripping and pinching gripping) are programmed by grouping multiple stimulation channels and ordering activation profiles.

[0057] Figure 3 shows motion recorded by a device according to a further embodiment of the present disclosure. In this example, a healthy person wearing a device according to an embodiment of the present disclosure moved their arm in a "C" shaped trajectory. In this example, the person repeated this movement three times. Signals from the IMU provided 6-axis data (acceleration and rotational velocity) of the person's wrist. The IMU output was corrected for gravity to obtain repeatable acceleration data. This acceleration data was integrated to determine the time-dependent position (i.e., trajectory) of the limb during the movement. Based on this trajectory, the computer 20 determined that a "C" shaped trajectory had occurred. In each iteration, the "C" shape appeared at the X / Y position shown in the rightmost column.

[0058] Computer 20 may use pattern recognition algorithms to analyze and identify limbs, body movements, and trajectories to identify the patient's intended actions. This analysis may include signal processing algorithms (e.g., dynamic time warping (DTW) which compares the actual trajectory of the patient's limb movements with the trajectory expected to correspond to the intended action). DTW has the advantage of being able to accommodate different movement / trajectory velocity or timing profiles that may vary among users.

[0059] In other embodiments, machine learning techniques are applied to analyze sensor outputs to identify a user's intention to perform a specific action and to identify other movements in which the user does not intend to perform an action. In one such embodiment, a superimposed neural network (CNN) or recurrent neural network (RNN) included in computer 20 analyzes data from the IMU and other sensors to identify body movements and trajectories or camera data that signal a patient's intention to perform an action, providing further contextual information to further identify the user's intention or information about an object the hand is approaching (the shape and size of the object that needs to be received and grasped by the hand). The RNN performs techniques such as long-shorter-short-term memory (LSTM) to identify intention-signaling movements. By utilizing such techniques, the system can iteratively and reliably identify specific trajectories or body movements and activate the patient's muscles or motorized prostheses to perform the intended action. In addition, since sensor data is recorded, the system according to embodiments of the disclosure can be continuously trained to better identify patient intentions. After appropriately anonymizing data from multiple patients, the resulting data can be collected and used to train machine learning algorithms. A variety of other machine learning algorithms can be used for the analysis and identification of natural and pre-programmed trajectories. Non-limiting examples include support vector machine (SVM) algorithms, handwriting recognition algorithms, and deep learning algorithms. Such machine learning algorithms may be executed locally on a computer 20 worn by the patient (e.g., embedded in a prosthesis or connected to a sensor housing 10). Alternatively, or in addition to local processing, machine learning algorithms may be executed on a computer system located remotely from the user (e.g., on a cloud computing network). This allows for the adaptation of the systems and methods disclosed herein, along with the collection of further data over time. Such algorithms may enable algorithmic motion recognition when initial detection is not highly accurate by repetitive body movements by the patient.

[0060] Figure 4 shows another example of motion detection by the device according to an embodiment of the present disclosure. Here, a healthy person performed a figure-three movement in three repetitions. Again, the IMU provided gravity-corrected acceleration data, and the computer calculated the time-dependent trajectory of the person's limbs. Again, the figure-three shape was observed in each repetition, as shown by the position graph in the rightmost column. If the user associates the figure-three shape and the figure-three movement with different actions (e.g., "key grasping" and "cylindrical grasping"), the computer 20 can add different patterns of neuromuscular stimulation, resulting in the hand movement performing one type of grasping or another type of grasping.

[0061] In another embodiment, a training set of motion data was prepared to match various alphanumeric trajectories. First, the raw, three-axis gravity-compensated acceleration obtained from the IMU was subjected to a bandpass filter (Butterworth, 8th order, 0.2–6 Hz) and processed offline for the identification of training samples. The absolute values ​​of acceleration along the three axes were used to identify the start of motion (by setting the threshold to 0.95 g). Next, using this start of motion, the time-series acceleration data along the X, Y, and Z axes was segmented into windows of -0.1 s to 0.9 s relative to the start. In each trial, it was visually confirmed that there were no arbitrary noise artifacts or excessive jerks (derivative values ​​of acceleration) or that the 1 s window had been exceeded. Such trials were excluded from further analysis. These training sets were used to train a dynamic time warping (DTW) algorithm and a long-short-term memory (LSTM) network algorithm.

[0062] The DTW algorithm minimizes the Euclidean distance between two samples by optimally aligning the sample trajectories with the previously determined template trajectory. This is achieved by repeatedly expanding or contracting the time axis until an optimal match is obtained. For multivariate data (e.g., acceleration), the algorithm simultaneously minimizes distances along different dimensions using dependent time warping. In this embodiment, the algorithm was used to calculate the optimal distance between the test sample and all templates associated with the 2D and 3D trajectories. The template with the lowest degree of optimality in distance to the test sample was selected as the classifier output. Since the classifier output depends on the quality of the templates, an internal optimization loop was used to select the best template trajectory from a set of training trajectories. Within this loop, the DTW score for each training sample was calculated along with the other training samples. Next, the training sample with the least-sacrificing DTW score was selected as the template (i.e., the expected trajectory) for that trajectory.

[0063] In some embodiments, LSTM networks are used for the analysis of motion data. According to one such embodiment, the LSTM network includes a single bidirectional layer with more than 100 hidden units using the MATLAB® R2019b deep learning toolbox. Default values ​​were selected for most parameters. The LSTM network transforms 2D or 3D accelerated data into inputs for a fully connected layer where the result was binary (i.e., 0 or 1). Next, a softmax layer was used to determine the probabilities of multiple output classes. Finally, the network output mode was set to "last" so that a decision is generated only when the final time step has elapsed. This allows the LSTM classifier to behave similarly to DTW and classify trajectory windows. During training of the LSTM network weights, the Adaptive Moment Estimation (ADAM) solver was used with a gradient threshold of 1 and a maximum number of epochs of 200. Zero padding was not used because the length of all training and validation data was 1 second. For running the LSTM network, the MATLAB® R2019b Deep Learning Toolbox was used, along with default values ​​for all parameters except those mentioned above.

[0064] According to one embodiment, online classification of arm trajectories was performed by filtering and processing the unprocessed acceleration signal in real time (using a MATLAB® script looped at 50 Hz). Within this loop, the acceleration data was divided into segments of 1 second in length, with 98% overlap. A DTW-based classifier was executed and designed to compare the incoming acceleration window with the 2D trajectory. If the optimal distance between trajectories was less than 10 units (empirically determined), a positive classification was issued, and then the NMES driver 14 was triggered to stimulate the muscles and perform the entire hand opening and closing motion sequence.

[0065] In one embodiment, when sensor data is input to a machine learning algorithm, this data is trained to identify specific movements as expected trajectories (for association with behavior). Such training can be achieved by using the unaffected side (a mirror image of the movement) in a healthy person or a stroke patient. In stroke patients, hemiplegia (paralysis of one side of the body) is not uncommon. In one embodiment, the stroke user uses their unaffected side to train the device's algorithm or to further fine-tune its movements. In either case, the user, while wearing the device, performs natural hand-reaching trajectories and a variety of trajectories in real-world conditions (along with additional sensors to detect hand-opening and different grasping behaviors). Examples of such additional sensors include EMG (electromyography) sensors placed on the relevant muscles to determine grasping behaviors (opening, closing, key-grabbing, cylindrical gripping). The amplitude of this EMG signal indicates the intensity of muscle contraction (e.g., duration and change over time), and this data allows for the direct notification of the electrical stimulation amplitude and timing added by the NMES driver 14 for the delivery of a pattern of stimulation signals to perform a grasping action, when the desired movement is recognized through the movement / trajectory recognition algorithm. Devices for detecting hand-opening and different grasping actions may also include a camera that records images of such actions, an IMU, or joint angle / flexion sensors or force sensors attached to a healthy hand (used to train the system to determine patterns of stimulation signals).

[0066] Further sensors may include a camera as well as flexion / joint angle sensors and force sensors. This camera is coupled to image analysis and positioned to capture the trajectory of the outstretched hand and / or grasping movements. The captured trajectory and grasping data is used to build a database of pre-trained trajectory or movement patterns (for association with specific hand actions). This data is used to train machine learning algorithms (e.g., deep learning neural networks). After the device is trained (or partially trained), it is fitted to a person with paralysis. One example of such training is the use of input to a computer via input device 24. For example, if a person is training the system to recognize a particular trajectory as an "S" shaped path (representing cylindrical grasping), that person may simultaneously utter audible words (such as "cylindrical grasping," "open," and "closed") at the same time as the movement. The computer 20 takes the microphone as input and uses known speech recognition techniques to read the speech input and tag the sensor data. This tagged data becomes part of the training set data for the machine learning algorithm. Alternatively, the tagging of movements used to train the algorithm may be performed using keystrokes on a keyboard, or the computer 20 may be equipped with a camera that captures the user's visual image while performing a variety of tasks (e.g., grasping an object on a table, inserting a peg into a board) (while recording movement data from the IMU to associate associated hand movements with "natural" grasping movements).

[0067] In another embodiment, a camera positioned on the wrist (as part of a band, sleeve / patch, or clothing) recognizes an object as it approaches, influencing the stimulus pattern to change the type of hand opening (e.g., activating all fingers or only the thumb-index finger extensor muscles). If the object's relative position to the hand slows down / stops, the flexor muscles are automatically activated to initiate grasping. Real-time object recognition technologies using small, portable devices with battery-powered microprocessors (e.g., cellphone technology) are well known in the field. Combining these technologies with AI and machine learning methods (e.g., support vector machines, superimposed neural networks, and long-short-term memory (LSTM) regression neural networks for static and dynamic image classification) enables visual cues (e.g., the type of object being grasped) to inform the system how to position the patient's hand so that grasping the object can be performed correctly and reliably.

[0068] Considering both 2D and 3D motion (e.g., the motion of a bottle opener in the air), a variety of trajectories can be identified by computers and associated with a variety of actions. These trajectories can be used not only to drive neuromuscular stimulation for motion recovery, but also to drive prosthetic / robotic devices or mobility devices (e.g., wheelchairs).

[0069] The system described in this disclosure was used to investigate the detection of movement trajectories corresponding to selected, predefined trajectories. Two participants with quadriplegia participated in this investigation. Participant 1 was a 32-year-old male who sustained an injury 6 years prior and had a C4 / C5 ASIA (American Spinal Injury Association) B injury. Participant 1 participated in 10 sessions, during which 2D and 3D arm movement trajectories were recorded in 7 of those sessions. During the remaining 3 sessions, grasping intentions were decoded online (in real time) and used to drive a custom-made neuromuscular stimulator containing textile-based electrodes 12a, 12b, ... 12n housed in a sleeve. As a result, the participant was able to perform functional movements (e.g., eating a granola bar). Participant 2 was a 28-year-old male who sustained an injury 10 years prior and had a C4 / C5 ASIA A injury. Participant 2 participated in 3 sessions, consisting of 2 training sessions and 1 online test session.

[0070] Participants were initially asked to sit with their hands on the table. A wireless sensor module was attached to the participant's wrist using Velcro®. This sensor module included motion sensors 16a, 16b, ... 16n and an MCU 18, as disclosed in the above embodiment. Although both participants had bilateral impairments, each participant was able to extend their hand with at least one arm, as they had some remaining movement ability, and this was ultimately used in this study.

[0071] During this study, participants were randomly given verbal cues associated with different 2D and 3D motion trajectories. Participants were instructed to perform reaching trajectories by starting their movement from the edge or corner of a table, then moving towards the center, and limiting the length of the smooth movement to 2 seconds. They were trained on three different 3D reaching trajectories: a horizontal arc, a vertical arc (e.g., reaching for a pen or marker placed on a table), and a bottle opener-type motion. In addition, they were trained on four 2D trajectories corresponding to the well-known English and Greek letters: S, ε (epsilon or E), γ (gamma), and M (performed on a horizontal plane). To minimize participant fatigue, the experiment was conducted in 18-20 trial blocks with sufficient rest between blocks. Since the S and ε trajectories are easy to remember and do not cause fatigue, participants were initially asked to perform only the S and ε trajectories. Next, after participants became accustomed to moving their arms, additional 2D and 3D trajectories were added to the study. As a result, the percentage of 2D trajectories (particularly S and ε) in the final dataset was higher than that of the remaining trajectories.

[0072] For participant 1, over 250 training samples were recorded across seven motion trajectories, while for participant 2, 96 samples were recorded from five motion trajectories. Trials containing noisy sensor data or inaccurate labels were visually identified and removed from the training set. A five-layered cross-validation scheme was selected to evaluate the DTW-based and LSTM-based classifiers. Figure 7 shows the mean ± standard deviation (SD) classification accuracy for the two participants. A bar graph compares the classification accuracy (mean ± SD) using the two methods, DTW and LSTM. Performance evaluation was performed using both offline (2D & 3D) and online (2D only) arm trajectories. The statistical significance threshold was set at p < 0.05.

[0073] In offline scenarios, both DTW-based and LSTM-based classifiers performed well for 2D trajectories, achieving accuracies of 94±5% and 98±3%, respectively. However, for offline 3D trajectories, LSTM performed better than DTW, achieving an accuracy of 99±3% for LSTM compared to 83±16% for DTW. Using a two-tailed Wilcoxon rank-sum test, the LSTM-based classification accuracy was significantly higher than DTW in both cases (p<0.05). Figure 7 also shows the online performance of the DTW-based classifier for 2D trajectories. During online classification, an accuracy of 79±5% was achieved when comparing two trajectories (e.g., Sv / sε) or a single trajectory with the remaining trajectories (e.g., Mv / remaining time (seconds)). To further evaluate the performance of each classifier for type I and II errors, cumulative confusion matrices were calculated by adding the confusion matrices from each participant's weight. The resulting confusion matrices are shown in Figure 8 for both classifiers and the trajectories of both types.

[0074] For DTW-based classifiers, Type I errors occurred more frequently in 3D trajectories than in 2D trajectories. The percentage of Type I errors was highest for the bottle opener trajectory (37.8%), followed by the vertical arc trajectory (14%), the ε trajectory (10.2%), and the M trajectory (10%). For Type II errors, the DTW-based classifier misclassified the vertical arc trajectory (14.5%), the lateral arc trajectory (13.8%), and the S-shaped trajectory (8.33%) compared to the remaining classes. For LSTM-based classifiers, Type I and II errors were extremely rare, ranging from 0-3% for almost all trajectories, except for the M trajectory, which had a Type I error rate of 40%. It is speculated that because only 10 trials were performed for training on the M-trajectory, the sample set was too small for the LSTM classifier to distinguish this trajectory from other classes with a larger sample size.

[0075] As a further example, the system according to embodiments of the present disclosure was tested on a paralyzed person who retained shoulder and arm movement but no hand movement. As shown in Figure 6, the device recognized the person's natural hand-extending movement of the arm and shoulder and stimulated the person's adductor and abductor muscles of the thumb to grasp a pen propped up in one cup. The person was able to lift the pen using their remaining arm and shoulder movement and move the pen to a second cup, while the device continuously activated the patient's muscles to maintain the grip.

[0076] As another example, in testing of the system according to the embodiment of this disclosure, muscle activation during reaching and grasping movements was predicted using a healthy person based on training an LSTM network with EMG signals. An EMG sensor was attached to the flexor and extensor muscles of the ring finger of the subject, and an IMU 16a was attached to the wrist. Signals from the EMG and IMU were pre-processed by a microcontroller 18 mounted on a circuit board (Arduino® Nano33BLE). Data from the circuit board was wirelessly transmitted to a computer 20 running the LSTM network as described in the above embodiment. While the subject repeatedly performed reaching and grasping movements, data from the IMU and EMG was provided to the LSTM network. After training the LSTM, the LSTM was able to predict the timing and amplitude of flexor and extensor muscle activity based on the trajectory of the subject's wrist.

[0077] In other embodiments, the device according to this disclosure may be used to enable lower limb movement. In one such embodiment, the IMU is fixed to the patient's buttocks. Detection of 2D and 3D buttock movement is performed by analyzing data from the IMU. Again, the achievement of algorithm training is made possible by attaching the IMU, a camera to observe limb position and movement, flexion / joint angle sensors at the leg joints and / or EMG sensors on the muscles to be stimulated in a paralyzed person as wearables on a healthy person. The use of buttock movement enables muscle activation using NMES (e.g., to modify the person's gait or to facilitate gait if the muscles are weak, paralyzed, or have foot drop). During normal walking, the left buttock / upper body traverses a 2D "C"-shaped curved trajectory in space, after which the right foot lifts. According to one embodiment, this evidentiary signature trajectory is recognized and used to trigger a muscle stimulation pattern in the right leg for gait sequence assistance. NMES electrodes can be placed on any muscle that acts on the target joint. For example, if the device according to this disclosure is used to facilitate rehabilitation after knee surgery, placing actuators on the quadriceps, popliteal tendon, calf, and ankle extensors can help improve the wearer's walking ability through muscle stimulation. Combining stimulation with the wearer's use of their own arms can partially support their own weight on a walker or parallel bars, thereby assisting hip / upper body movement. Next, the trajectory of the right hip is detected and used to stimulate the muscles of the left leg.

[0078] Systems according to embodiments of the present disclosure may be integrated with gloves, shoes, and other clothing including force sensors. Such sensors detect contact and pressure between the wearer's hand and a grasped object, or monitor the position of the foot during walking. Such clothing may also include flexion / angle sensors at the elbow, wrist, knee, ankle, or other joints, thereby providing trajectory, orientation, motion information and / or data related to intent (when training machine learning algorithms in a healthy body user) to the system.

[0079] According to another embodiment of this disclosure, information about the trajectory, orientation, and position of a patient's limbs is collected and recorded by the system. Such information is used to track the trajectory and / or joint movement (or range of motion) of body parts during physical therapy. The system according to this disclosure provides healthcare professionals with a low-cost method for tracking progress and characterizing movement (e.g., the overall movement of an arm in space) in the rehabilitation of stroke or spinal cord injury patients. Such a system is less expensive and less bulky than current methods that rely on expensive robotic systems or table-sized devices to monitor limb position and movement. In addition, machine learning algorithms can compare the patient's movements to those of healthy volunteers and other patients at various stages or grades of recovery, or classify the patient's movements. This information may enable professionals to optimize treatment, provide better feedback to patients, and demonstrate patient progress during recovery.

[0080] During the training of user-specific (custom) trajectories, voice recognition, brain / computer interface (BCI) – non-invasive or invasive (EEG), touchpad, and / or healthy hand / leg movements can be used to initiate training or select a desired action or hand / foot movement (associated with the trained trajectory). Furthermore, pre-trained trajectory profiles can be stored within the device / system, eliminating the need for training. For example, letters, numbers, and patterns known to the user may be available and can be automatically recognized without user-specific training.

[0081] In another embodiment, as an alternative to or addition to muscle stimulation to induce action in response to a recognized trajectory, the system may also add therapeutic stimulation to any location in the patient's nervous system. In a further embodiment, for a variety of applications (e.g., motion / sensory recovery and chronic pain), one or more of the electrodes 12a, 12b, 12c…12n are adapted to add stimulating current to the patient's peripheral nerves or the patient's central nervous system (CNS). Nerve stimulation is also well known to be effective in pain treatment through implanted transcutaneous stimulation devices. It is also well known that certain types of movement (raising an arm or bending forward from the waist) can cause pain. In some embodiments of the present disclosure, translational and / or rotational movements of a body part that may cause pain trigger stimulation to reduce pain caused by the detected movement.

[0082] According to embodiments of this disclosure, triggering various types of stimulators by motion / trajectory recognition can yield many advantages. For example, vagus nerve stimulation has been shown to improve the effectiveness of upper limb rehabilitation. In one embodiment, the system triggers vagus nerve stimulation cervically (cervical) or aurally (aurally) during motion rehabilitation for stroke, SCI, traumatic brain injury, MS, etc. Such therapeutic stimulation can be added to other nerves (e.g., the trigeminal nerve and other cranial nerves or peripheral nerves connected to the target muscle). The system according to this disclosure can also be used to trigger, control and / or modulate various forms of brain stimulation (e.g., TMS (transcranial magnetic stimulation) and tDCS (transcutaneous direct current stimulation), tACS (transcutaneous alternating current stimulation), TENS (transcutaneous electrical nerve stimulation), or spinal cord stimulation (sending signals downward to the spinal cord and upward to the brain)), thereby promoting neuroplasticity, recovery after stroke or traumatic brain injury, and / or pain reduction.

[0083] Due to spinal cord injury, signals from the brain within a spinal cord injury patient may be blocked or attenuated before reaching the muscles. Stimulation of the damaged spinal cord pathway or its vicinity can lead to excitability in that pathway, potentially facilitating movement and rehabilitation in the spinal cord injury patient. Known systems for adding spinal cord stimulation are often controlled manually through control pads or devices (rather than by the patient's body movements). According to embodiments of this disclosure, one or more electrodes 12a, 12b, 12c, ... 12n are positioned extradurally or, preferably, percutaneously on the patient's spinal cord. The system senses specific trajectories drawn by the patient during physiotherapy and, in addition to causing muscles to perform desired movements of the paralyzed limb through the addition of NMES stimulation, the system triggers percutaneous spinal cord stimulation to increase nerve signals that are reduced due to spinal cord injury (by increasing the excitability of interneurons). Similarly, one or more electrodes 12a, 12b, 12c, ... 12n may be positioned above, on, or below the spinal cord injury site, thereby stimulating the spinal cord injury and / or pathways above and / or below the injury, which may support the repair of neurons damaged as a result of the injury and / or strengthening of neuronal connections. By combining the patient's voluntary movements with such stimulation, the patient can control their own stimulation patterns, which may further facilitate neuroplasticity and movement and / or the recovery of sensory function.

[0084] In patients who have suffered a stroke, electrical stimulation or transcranial magnetic stimulation applied to the site of injury or its vicinity in the brain or brainstem may help repair neuronal damage. According to further embodiments of the present disclosure, one or more electrodes are positioned on the scalp or on a magnetic coil on the scalp. In response to the detection of a movement trajectory, a stimulation signal is applied to these electrodes or coil (as an alternative to, or preferably in addition to, NMES signals for moving the patient's paralyzed limb or extrinsic limb). When such brain stimulation is combined with the patient's intention to move the paralyzed limb or extrinsic limb, it may help repair some of the motor neuron function damaged as a result of the stroke.

[0085] The system described herein is relatively inexpensive, portable, and can be controlled independently by the patient without the assistance of a therapist or other professional. This allows patients to stay at home while wearing the device (e.g., one or more wearable sleeves, one or more patches), leading to an increase in the number of hours per week that can be used for rehabilitation.

[0086] Figure 9 shows another embodiment of the present disclosure. A prosthetic hand 100 is attached to the arm of a person who has had an amputation. The prosthetic hand 100 includes a sensor housing 10. The sensor housing 10 may include sensors 16a, 16b, ... 16n as described above for detecting the acceleration, velocity, position and rotation of the wearer's arm. A controller 21, integrating the functions of the MCU 18 and computer 20 as described in the above embodiments, is connected to the sensor array and receives signals indicating the movement trajectory performed by the wearer using healthy joints (e.g., shoulder, torso and upper arm). As in the above embodiments, the controller 21 determines whether the movement corresponding to the intended activation of the hand has been performed. The controller 21 is connected to actuators 112a, 112b, ... 112n. These actuators drive the finger movements of the prosthesis 100. As in the above embodiments, one or more predetermined trajectories are associated with specific hand movements. For example, if the wearer moves their upper arm, shoulder, and torso to make the prosthesis trace a “rainbow arc,” as described above, this may indicate an intention to grasp it, and actuators 112a, 112b, ..., 112n are energized to move the fingers in order to perform this intended grasping movement.

[0087] The embodiment shown in Figure 9 relates to a prosthetic hand 100. This disclosure is not limited to hand prostheses. Other types of prostheses can be controlled by the devices of this disclosure. For example, a foot prosthesis may be provided that senses the walking motion of the wearer's leg and activates an actuator to orient the foot in a manner synchronized with the wearer's gait.

[0088] In another embodiment, the system according to this disclosure can assist in training or physiotherapy of a normally healthy person to provide active resistance during exercise. For the purpose of sports training or physiotherapy, it stimulates non-paralyzed muscles using diverse limb movement / trajectory recognition. For example, the rotational velocity and linear acceleration of a person's forearm are detected using IMU and / or gyro data from sensors mounted on the forearm as part of a sleeve, patch, or other attachment. This movement is primarily performed by the biceps. In response to the detected movement, the system triggers antagonist muscles, including the triceps, to provide active resistance to the biceps (proportional to the measured forearm rotational velocity). In one embodiment, the proportionality constant is a configurable parameter, so that the resistance can be changed by the user.

[0089] While exemplary embodiments of this disclosure have been described and illustrated above, it should be understood that these are illustrative and should not be considered limiting. Additions, omissions, substitutions, and other modifications are possible without departing from the intent or scope of this disclosure. Therefore, this disclosure should not be considered limited by the foregoing descriptions.

Claims

1. It is a device, One or more motion sensors, each of which generates one or more motion signals indicating two-dimensional or three-dimensional movement of a first body part of a human, The prosthetic limb includes an actuator configured to modify the configuration of the prosthetic limb in order to perform an intended action selected by the human, A processor connected to one or more motion sensors and actuators, wherein the processor includes a data storage unit, the data storage unit includes a plurality of pre-trained expected two-dimensional or three-dimensional trajectories associated with the human intention to perform each of a plurality of intended actions, and the selected intended action is associated with a pre-trained trajectory selected from the plurality of expected two-dimensional or three-dimensional trajectories, The input device connected to the processor is configured to receive a feedback signal input by the human and to transmit the feedback signal to the processor, wherein the feedback signal indicates that the modification of the configuration of the prosthetic limb by the actuator has been identified by the human as the selected intended action, The aforementioned processor, The system receives one or more motion signals from the one or more motion sensors. The actual two-dimensional or three-dimensional trajectory of the first body part is calculated, The actual two-dimensional or three-dimensional trajectory is compared with the plurality of pre-trained expected two-dimensional or three-dimensional trajectories, and based on the comparison, the actual trajectory and the selected pre-trained trajectory are identified. The actuator is activated to change the configuration of the prosthetic limb so that the selected intended action is performed, and The system receives the feedback signal indicating that the change in the configuration was the selected intended action intended by the human being. device.

2. The device according to claim 1, wherein the prosthetic limb includes a prosthetic hand, and the actuator includes one or more of a wrist actuator and a finger actuator.

3. The device according to claim 1, wherein the processor calculates the difference between the actual two-dimensional or three-dimensional trajectory and the expected two-dimensional or three-dimensional trajectory, and operates the actuator based on the difference.

4. The device according to claim 1, wherein the processor generates muscle displacements based on the plurality of expected two-dimensional or three-dimensional trajectories detected during a set of training movements.

5. The device according to claim 1, wherein the processor generates one or more control signals that modify the configuration of the prosthetic limb in order to perform the selected intended action.

6. The device according to claim 5, wherein the one or more control signals include a pattern of control signals, the pattern of control signals is determined from the movement of a first body part sensed during a set of training movements.

7. The device according to claim 6, wherein the motion sensor includes one or more of the following: an electromyograph sensor, a camera, an inertia unit, a flexion / joint angle sensor, and a force sensor.

8. The device according to claim 1, wherein in the comparison performed by the processor, one or more of the following are used: a support vector machine (SVM) algorithm, a handwriting recognition algorithm, a dynamic time warping algorithm, a deep learning algorithm, a recurrent neural network, a shallow neural network, a superimposed neural network, a convergent neural network, or a deep neural network.

9. The device according to claim 8, wherein the processor performs the comparison using a long- and short-term memory type regression neural network.

10. The device according to claim 4, wherein the set of training movements is performed by a second person.

11. The device according to claim 4, wherein the set of training movements is performed by the person using the body part opposite to the first body part in the lateral direction.

12. The device according to claim 1, wherein the plurality of pre-trained expected two-dimensional or three-dimensional trajectories include at least one expected two-dimensional trajectory in an alphanumeric shape.

13. The device according to claim 5, further comprising an orientation sensor, the orientation sensor being connected to the processor and adapted to monitor the orientation of the first body part, the force applied by the selected intended action depending on the amplitude of the control signal, and the processor adjusting the amplitude of the control signal at least in part on the output of the orientation sensor.

14. The device according to claim 13, wherein the plurality of intended actions include one or more grips, the one or more grips include key grips, cylindrical grips, and vertical grips, and the processor selects from among the grips in response to the output of the orientation sensor.

15. The device according to claim 5, further comprising a camera, the camera being connected to the processor and positioned near the prosthetic limb to capture an image of an object to be grasped, the processor adjusting the one or more control signals partially based on the image.

16. The device according to claim 15, wherein the processor further includes a close delay timer, and the processor delays the generation of the one or more control signals for a predetermined period of time at the end of the actual two-dimensional or three-dimensional trajectory determined by the close delay timer.

17. The device according to claim 16, wherein the processor modifies the configuration of the prosthetic limb for the execution of post-grasp activity in response to a post-grasp signal from the motion sensor.

18. The device according to claim 17, wherein the activity after grasping is to open the prosthetic outer limb in order to release the grasp.

19. The device according to claim 18, wherein the post-grasp signal is a tap on the surface of the grasped object one or more times.

Images