Wearable electromyographic device and system and methods for motion decoding of body parts

Abstract

A computer-implemented method of training a neural network for generating upper or lower limb motion control signals to operate a virtual, robotic or prosthetic upper or lower limb, comprising providing or receiving, at a neural network, electromyographic data and associated kinematic upper or lower limb target data provided as a plurality of data chunks or blocks, and processing the plurality of data chunks or blocks to carrying out training of the neural network based on the plurality of data chunks or blocks; wherein each data chunk or block comprises a plurality of channel window data extracts and kinematic upper or lower limb target data associated with each channel window data extract; and optimizing the neural network architecture and hyperparameters using an optimization algorithm to provide a pretrained neural network for generating upper limb distal extremity motion control signals.

Description

[0001] Wearable electromyographic device and system and methods for motion decoding of body parts

[0002] Cross-reference to related applications

[0003] The present application claims priority to the European patent application EP23159952.3 that was filed on March 3rd, 2023, the entire contents thereof being herewith incorporated by reference.

[0004] Technical Field

[0005] The present invention generally belongs to the field of devices, systems and methods for tracking and analysing body parts movements, like fingers or hand movements, for rehabilitation, gaming or teleoperation tasks.

[0006] Background Art

[0007] The human hand is an extremely dexterous “tool” that allows people to continuously control their fingers, to grasp and manipulate objects with high precision. However, to interact with devices such as phones, computers, drones, or gaming consoles, interfaces have been developed such as mice or keyboards, controllers, or tactile screens that may have a considerable learning curve and offer limited functionality. In many cases, controllers are not intuitive and require learning the mapping between the different buttons and the intended action.

[0008] In many situations, such as in the case of teleoperation of systems like drones, medical apparatuses, robotic arms and the like, nothing could compare to the dexterity of hand manipulation, which has brought research to explore new avenues; for instance, to tele-control real objects (human-machine interaction) or manipulate virtual objects in a Virtual or Augmented Reality (VR / AR) setting, Electromyography (EMG)-based armbands such as the Myo Gesture Control Armband have been implemented.

[0009] In more complex contexts such as in the case of a limb loss or a stroke episode, these cause severe physical debilitation and often distress, besides an evident permanent or semipermanent functional limb damage that hinders the use of the above-mentioned interfaces. An ideal prosthesis or orthosis should reproduce or help the bidirectional link between the user's nervous system and the peri-personal environment by exploiting the post-amputation or post-damage persistence of the central and peripheral neural networks and pathways devoted to limb motor control and sensing.

[0010] In the case of the upper limb, skilful object grasping and manipulation is compromised, thus depriving the person of the most immediate and important source of tactile sensing in the body. For these reasons, replacing a e.g. lost hand and its precise functionalities is a major unmet clinical need that is receiving attention from engineers, neurophysiologists, and clinicians among others.

[0011] With regards to EMG-based devices, typically sleeves, such as textile-based, compliant sleeves, are used that include arrays of electrodes for tracking, recording and analysing myoelectric activity, thus allowing a more standardized and precise evaluation of the neurophysiological aspects of movement, in particular in rehabilitative and assistive contexts. Current standard practice for measuring EMG consists of carefully positioning for each muscle of interest, a pair of electrodes on top of the muscle plus often a reference on a neutral region (e.g. elbow) to form bipolar derivations. Correct positioning and labelling of the electrodes as well as the selection of the muscles of interest for a particular application requires a certain degree of specialized knowledge, often available only to clinicians. This is a further barrier to the widespread use of myoelectrical imaging in everyday contexts. On top of that, especially for myoelectric control of prostheses for upper limb amputees, low residual muscle innervation might render correct positioning very difficult, even to a specialized clinician.

[0012] Non-invasive devices and systems (i.e. systems that do not require surgeries or implants) based on EMG are generally difficult to calibrate and not sufficiently precise to meet the user’s expectations and daily activities outside a clinical setting, mainly due to computationally heavy calibration algorithms, lack of reliable calibration modes, need for very precise positioning of electrodes, need for wet electrodes and additional sensors and the like.

[0013] Actual solutions commercially available, such as the cited Myo Gesture Control Armband, are not suitable as a solution for addressing medical needs; onset myoelectrical detection is not enough to have natural decoding and can provide only discrete commands to control devices. Additionally, using only a wrist or arm band shows some limitations in decoding performance due to the lack of spatio-temporal information from the forearm muscle activity, which further engenders a reduction of long-term decoding robustness.

[0014] Accordingly, despite the remarkable advancements of the research in the field and a plethora of available products, it is still desirable to have an EMG-based device, system and methods for addressing the drawbacks of current solutions, particularly in terms of rapidity and robustness of calibration modes, ease of adoption by a user, good wear-ability, and possibility to suitably move or manipulate virtual, prosthetic and / or robotic arm / hand or leg in a precise fashion, even and overall in case of limb amputation. The present invention addresses these issues. Summary of invention

[0015] In order to address and overcome at least some of the above-mentioned drawbacks of the prior art solutions, the present inventors developed new methods and wearable devices having improved features and capabilities.

[0016] In particular, a first purpose of the present disclosure is that of providing a wearable device and system which is extremely easy and intuitive to calibrate, from one user to another and among various use sessions.

[0017] Another purpose of the present disclosure is to provide a wearable device that can be simply worn and autonomously set up outside clinical settings, by any user and without timeconsuming procedures.

[0018] Still another purpose of the present disclosure is that of providing a wearable device and system for manipulation of robotic, prosthetic or virtual limbs by any kind of user, including bilateral hand / arm amputees or leg amputees.

[0019] All these aims have been accomplished, as described herein and in the appended claims.

[0020] Aspects of the disclosure relate, in an implemented and non-limiting exemplary embodiment, to a medium density EMG device that can be used as an interface to read finger motions precisely. It consists of an elastic sleeve with medium-density derivations, that is placed around the forearm (or part of the forearm) that is modular in the number of electrodes and forearm sizes. This interface allows for intuitive and continuous control of six or more degrees of freedom that can be useful for many purposes, including for instance object manipulation in immersive VR / AR settings, control of robotic arms / hands / limbs, prosthesis / orthosis control, drone control, or everyday life controllers interacting with a smartphone (switching songs for instance).

[0021] Using deep learning algorithms, the inventors have demonstrated that it is possible to decode finger motions from EMG signals with precision higher than standard EMG devices and decoding approaches. The calibration of the decoding algorithm is short (~5 minutes of data) thanks to transversal (along several users) and longitudinal (along several trainings of the same user) transfer learning and requires only wearing the EMG sleeve on the forearm without precise placement and a webcam to track finger movements. Results show the possibility to control a robotic hand with up to 6 degrees of freedom in healthy subjects as well as in patients with an amputation.

[0022] This EMG sleeve and decoding framework allow interaction with any device naturally and intuitively without requiring a controller. Moreover, the device can easily be enhanced with inertial measurement units (IMUs) for arm and wrist orientation tracking and vibrators for haptic feedback.

[0023] In terms of wearable device support and decoding, a particularly advantageous aspect of the disclosure relies on the presence of a plurality of different bands of monopolar electrodes configured in / on the sleeve support to be located around the forearm. As for classical approaches used for instance in embedded EMG systems controlling prosthetic hands, an envelope of the signal can be extracted and subsampled at low frequency, such as for instance 100Hz, to allow for faster training of models and for reducing greatly the computational power needed.

[0024] However, and in a counter-intuitive manner, the inventors configured the system to sum the signal envelopes together in bands (also referred to herein as “rings”) of electrodes so that the rotation of the sleeve around the forearm does not impact the input of the decoding model as much. Instead of input of shape (time*64 channels), the system uses only (time*10 rings) to reduce even more the computational burden, while keeping the performance on a single recording similar, an intersession performance greatly improved and wearability of the device universal, as user do not need to worry about precise electrode positioning. Up to the inventors’ knowledge, this is the first report of this kind of decoding approach.

[0025] In view of the above-summarized drawbacks and / or problems affecting devices of the prior art, according to the present invention there is provided a computer-implemented method of training a neural network to generate upper or lower limb motion control signals or distal extremity motion control signals to operate a virtual, robotic or prosthetic upper or lower limb according to claim 1.

[0026] Further advantageous features are defined by the appended dependent claims.

[0027] The above and other objects, features and advantages of the herein presented subject-matter will become more apparent from a study of the following description with reference to the attached figures showing some preferred aspects of said subject-matter.

[0028] Brief description of drawings

[0029] Figure 1 shows a user wearable electromyographic support or sleeve and in particular the sleeve shape before attachment to a forearm and further details concerning exemplary fastening means for fastening the sleeve to a forearm. Figures 2A show forearm and sleeve sizes, and Figure 2B shows a table of forearm and sleeve sizes.

[0030] Figure 3 shows an exemplary electrode arrangement of the sleeve that contacts a user forearm to capture myoelectric signals.

[0031] Figure 4 shows an exemplary electrode holder, a plurality of which are included in the sleeve to hold the electrodes in a desired arrangement.

[0032] Figure 5 shows an exemplary medium density electromyography approach of the present disclosure in which a large surface area is covered without increasing the number of sensors / electrodes to an unmanageable level and permits to create maps of decoding performance and signal amplitude during each phase of a movement. Such data constitutes a far superior input to a decoding algorithm than the bipolar derivations of known superficial EMG approaches. A large surface can be covered using, for example, 64 to 128 channels. Setup can be achieved in 1 minutes with dry electrodes.

[0033] Figure 6 shows measurements results obtained using this medium density electromyography approach during arm flexion and extension.

[0034] Figure 7 shows exemplary hand landmarks (courtesy of the company MediaPipe Hands) that can be located on a hand from camera images or video, and that can then be used to extract a finger angle for each finger and the thumb.

[0035] Figure 8 shows an exemplary system for recording data acquired from a wearable electromyographic support or sleeve, a camera and optionally an inertial measurement unit, this data can then be used, following preparation or processing, to pre-train a neural network. Figure 9 shows an exemplary system for generating upper limb distal extremity motion control signals to operate a virtual, robotic or prosthetic upper limb.

[0036] Figure 10 shows exemplary processing of the data provided by the exemplary system of Figure 8 to prepare training data for training a neural network of the exemplary system of Figure 9.

[0037] Figure 11 shows exemplary processing of the data obtained by the exemplary system of Figure 8 where the data measured by a wearable electromyographic support or sleeve, a camera and an inertial measurement unit is synchronized and interpolated.

[0038] Figure 12 shows further exemplary processing of the obtained data where extracts of the EMG data and of the kinematic data are prepared and placed in a data input format suitable to train a neural network.

[0039] Figure 13 shows the extracted data as processed in Figure 12 for multiple different recordings from N different human sources, and the creation of a data chunk or data block comprising a plurality of shuffled and randomly ordered EMG channel window data extracts and associated kinematic upper limb distal extremity target data extracted from multiple different recordings. Figures 14 and 15 show exemplary upper limb distal extremity motion control signal generation systems configured to carry out transfer learning to calibrate a pre-trained neural network of the system.

[0040] Figure 16 shows an overview of transfer learning pipeline of the present disclosure in which transversal transfer learning (along several users) is performed, and then longitudinal transfer learning (along several trainings of the same user) is carried out to calibrate the system for the end user.

[0041] Figure 17 shows another exemplary system for generating upper limb distal extremity motion control signals to operate a virtual, robotic or prosthetic upper limb.

[0042] Figure 18 shows exemplary positions of the user wearable electromyographic support or sleeve on an upper limb and a lower limb for measuring or obtaining EMG data or signals.

[0043] Figure 19 shows exemplary bodily or pose landmarks (courtesy of the company MediaPipe) and lower limb landmarks that can be determined and located on a body from camera images or video, and that can then be used to extract lower limb joint angles for the knee and ankle. Figure 20 shows an exemplary lower limb motion control or distal extremity motion control signal generation system configured to carry out transfer learning to calibrate a pre-trained neural network of the system.

[0044] Figure 21 shows an exemplary system for generating lower limb motion control or distal extremity motion control signals to operate a virtual, robotic or prosthetic lower limb.

[0045] Detailed description of the invention

[0046] The subject-matter described in the following will be clarified by means of a description of those aspects which are depicted in the drawings. It is however to be understood that the scope of protection of the invention is not limited to those aspects described in the following and depicted in the drawings; to the contrary, the scope of protection of the invention is defined by the claims. Moreover, it is to be understood that the specific conditions or parameters described and / or shown in the following are not limiting of the scope of protection of the invention, and that the terminology used herein is for the purpose of describing particular aspects by way of example only and is not intended to be limiting.

[0047] Unless otherwise defined, technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Further, unless otherwise required by the context, singular terms shall include pluralities and plural terms shall include the singular. The methods and techniques of the present disclosure are generally performed according to conventional methods well known in the art and as described in various general and more specific references that are cited and discussed throughout the present specification unless otherwise indicated. Further, for the sake of clarity, the use of the term “about” is herein intended to encompass a variation of + / - 10% of a given value.

[0048] Non-limiting aspects of the subject-matter of the present disclosure will be described by way of example with reference to the accompanying figures, which are schematic and are not intended to be drawn to scale. For purposes of clarity, not every component is labelled in every figure, nor is every component of each aspect of the invention shown where illustration is not necessary to allow those of ordinary skill in the art to understand the invention.

[0049] The following description will be better understood by means of the following definitions.

[0050] As used in the following and in the appended claims, the singular forms "a", "an" and "the" include plural referents unless the context clearly dictates otherwise. Also, the use of "or" means "and / or" unless stated otherwise. Similarly, "comprise", "comprises", "comprising", "include", "includes" and "including" are interchangeable and not intended to be limiting. It is to be further understood that where for the description of various embodiments use is made of the term "comprising", those skilled in the art will understand that in some specific instances, an embodiment can be alternatively described using language "consisting essentially of" or "consisting of."

[0051] In the frame of the present disclosure, the expression “operatively connected” and similar reflects a functional relationship between the several components of the device or a system among them, that is, the term means that the components are correlated in a way to perform a designated function. The “designated function” can change depending on the different components involved in the connection. Likewise, any two components capable of being associated can also be viewed as being “operably couplable,” to each other to achieve the desired functionality. A person skilled in the art would easily understand and figure out what are the designated functions of each and every component of the device or the system of the invention, as well as their correlations, on the basis of the present disclosure.

[0052] The present disclosure pertains to a wearable electromyographic device, system and methods for motion decoding of body parts, particularly for motion decoding of hands and fingers of human users. The disclosure is founded around several main concepts,:

[0053] - a method of calibration of the system of the present disclosure exploiting a novel approach based on longitudinal and transversal transfer learning (TL);

[0054] - a method of decoding EMG signals based on analysis of rings or bands of electrodes located in or on an elastic, wearable support such as a sleeve instead that on decoding of single electrodes;

[0055] - a method of moving virtual or prosthetic or robotic arm / hand or leg; - a method for training a model by acquisition of several recordings by loading data in chunks in a pseudo-random manner, as will be detailed later on in more details.

[0056] Additional matter forming the present disclosure comprises:

[0057] - a system comprising a wearable support, such as a sleeve, implementing the methods of the present disclosure. The system can comprise, accordingly, a wearable support of the present disclosure in operative connection with a data processor adapted to perform one or more methods of the present disclosure. In addition, the system can comprise further operatively connected elements including

[0058] - a camera or a virtual hand for performing calibration steps

[0059] - a prosthetic or robotic limb such as an arm or a hand or a leg.

[0060] Advantageously, systems and methods of the invention are adaptable to accident amputees, congenital amputees and / or people with limited residual muscle force (e.g. stroke survivals), and are amenable as well to single or both hands amputees.

[0061] EXAMPLES

[0062] In a non-limiting example, the inventors implemented an EMG sleeve 3 allowing to decode finger motions so to map movements and control any device intuitively. The calibration is very simple as it requires only wearing the device 3 and filming the hand with a webcam 7. Moreover, it leverages deep learning and transfer learning to improve decoding performance and reduce calibration time.

[0063] EMG systems are, for now, mainly used in the fields of physiotherapy, biomechanics, or for patients with an amputation to control a robotic prosthetic hand. The traditional muscle-specific approach requires considerable expertise to correctly position bipolar derivations over each muscle of interest. To tackle this issue, Thalmic Labs released years ago an armband with eight electrodes (as described for instance in US patent 10,684,692 the entire contents of which are enclosed herein by reference) that can perform gesture recognition. Onset detection is however not sufficient to have natural decoding and can provide only discrete commands to control devices. On the other hand, using only a wrist or arm band shows some limitations in decoding performance due to the lack of spatio-temporal information from the forearm muscle activity. This also induces a reduction of long-term decoding robustness.

[0064] The device 3 of the present disclosure comprises a grid of monopolar electrodes EL that can cover the whole forearm as a sleeve and can record spatio-temporal information from muscle activity. The material used avoids the sleeve shifting around the forearm and therefore increases the long-term reliability of the signal acquired. The methods of the present disclosure allow for proportional decoding of finger angles and simple calibration using only a webcam 7. For each new user, a calibration is required and can be performed in 5 minutes as transfer-learning is used to leverage information gathered from other users.

[0065] Results show better decoding capabilities than gold-standard clinical EMG systems. The combination of the sleeve 3 and the decoding algorithm also works with upper limb amputees. Specifically, a person with a congenital amputation could directly control a robotic hand 9 after a 5-minute long calibration, as experimentally shown by the inventors. Additionally, the calibration was not necessary when wearing the sleeve the next day.

[0066] Surprisingly advantageous aspects are:

[0067] • The incredible ease of use with respect to traditional EMG, which requires very specific knowledge on where to place electrodes;

[0068] • The versatility in allowing transfer learning to work so well. Without transfer learning, the calibration is 15 minutes, and training the model takes ~50min on a GPU-accelerated laptop. Transfer learning has the clear advantage of reducing by 10 times the time required to train the decoder with only a 5 minutes calibration to obtain comparable performance;

[0069] • The ability of the system to work in minutes with amputees and one patient with congenital amputation, still working on the next day. The sleeve 3 was placed by the family without caring to place the electrodes EL in the exact same place as the day before;

[0070] • The calibration with only one webcam 7 based on an external library works better than more complex calibration setups which simplifies a lot the acquisition of EMG data;

[0071] • The huge difference in decoding performance using similar algorithms on the sleeve 3 that has dry EMG electrodes and standard EMG systems that use gel electrodes.

[0072] THE SLEEVE

[0073] The sleeve 3 was designed so as to avoid both the traditional superficial EMG (sEMG) approach and the high-density sEMG approach.

[0074] The traditional muscle-specific sEMG approach requires considerable expertise to correctly position bipolar derivations over each muscle of interest. This reduces the availability of myoelectric imaging tools for non-professional users in everyday life conditions. Plus, to successfully identify each muscle and correctly position electrodes, it is often necessary to have the subject or patient performing several movements, which might be a challenge in case of impairment (e.g., acute post-stroke subjects) or low / non-existent residual muscle activation capability.

[0075] High-density EMG (HDsEMG) consists of recording the sEMG via a dense array of electrodes that restricts the imaging surface to a narrow area, thus limiting the usability of the device. For example, covering the forearm with HDsEMG standard electrode density would require around 250 channels.

[0076] The medium density sEMG approach, on the other hand, has greater usability while maintaining comparable performance to the HDsEMG. Thanks to this approach it is possible to identify similar grand-average dynamic-time warped amplitudes and extract maps reflecting the activation of muscles at different movement phases (see for instance Figure 5 which shows the map of the decoding accuracy at each electrode site of a simple elbow flexion / extension movement).

[0077] Concerning the material of the sleeve, different functions needed to be fulfilled.

[0078] • Stretchability: the material of the sleeve 3 had to be stretchable to first, fit different arm sizes and second, compensate for arm movements while ensuring proper fixation of the electrodes.

[0079] • Skin Comfort: Long-term usages are intended. Issues occurring due to discomfort could include rejection, unnatural movements, and a decrease in signal quality because of sweat.

[0080] • Sturdiness: While being stretchable, the material needs, at the same time, to be sturdy to a certain extent. Otherwise, proper fixation of the electrodes to the sleeve could be problematic.

[0081] • Anti-slip: To avoid shifts in electrodes’ positions, relative movements between the sleeve and the skin should be prevented. This could be ensured by a material with an anti-slip surface.

[0082] These demands require a high functionality of the textile. Clothing for sports, medical, and other professional applications are examples that could fulfil those. The material purchased is a reusable elastic compression tape adapted for sport (such as those provided by Neostrap or Ezyplast). The maximum elongation Emax was empirically determined at 53% with respect to its length at rest and it fulfilled the requirement for stretchability. At the same time with a thickness of 2.5 mm, the material was sturdy enough to provide support for the placement of the electrodes. Additionally, a foam layer offered high comfort while possessing an anti-slip surface. To examine long-term effects, a strap was worn for several hours by a test subject. No remarkable discomfort or sweat was reported. Then a second fabric was used for an outer layer to create pressure on the electrodes. For this, a thin, stretchable textile in polyester, commonly available in textile stores can be used. The exemplary shape of the sleeve 3 is shown in Fig. 1 , the sleeve 3 is shown in an unwrapped arrangement permitting to receive, for example, a forearm. It is made to approximately fit the standard shape of a forearm. The curving at the proximal end PE is to ensure freedom of movement when flexing the elbow joint. The cuts CT enable relative movement between the pieces to improve adaption to the skin, especially during pronation and supination.

[0083] The sleeve 3 should preferably be sized to fit as many people as possible. When simplified, the forearm can, for example, be described by three parameters, length, minimal and maximal circumference (Figure 3). It was found that demographic data does not provide sufficient information on those measures. Therefore, a simple study was conducted by the Inventors.

[0084] The forearm sizes of a total of 9 persons (4 male and 5 female) have been measured. The average, maximum and minimum values are presented in Fig 2B. The sizes that are assumed for a forearm when designing the sleeve layout are considered 220 mm in length, 155 mm minimal and 235 mm maximal circumference. If the forearm has a larger circumference, the difference can be overcome to a certain extent by the flexibility of the textile. In the other direction, the edges of the sleeve 3 can be overlapped. In special cases, the sleeve 3 can additionally be folded to compensate for very small forearms. Alongside the arm, excess length can be compensated in the same way, but stretching is not possible due to missing fixation points. Therefore the length was chosen closer to the maximum value. These considerations allowed the sleeve to fit on most forearms. However, the resulting inter electrode distance IED and possibly also electrodes EL without contact need to be taken into account in those cases.

[0085] Sleeve positioning and fixation

[0086] The sleeve 3 is donned by lying it on the outside of the forearm and then wrapping it around. To keep it in this position, hook-and-loop fasteners HLF are used (see, for example, Figure 1 ). Additional fixation is included to assure improved or proper skin contact of the electrodes without position shifts. For this reason, longer straps out of a thinner and more flexible fabric are attached to the sleeve.

[0087] After closing the basic part, the longer straps can be wrapped around the sleeve 3 to apply a small force. To compensate for the cone-shape of the forearm, additional rigid parts can be integrated. To achieve a reproducible setup, an easy and reliable way of positioning the sleeve 3 on the limb is needed. This can be done, for example, by visual markers provided on the sleeve 3, signalling the position of underlying bones, which are easily locatable. A line, for example, follows the ulna and a second line the distal end of the radius. All markers can refer, for example, to the posture of 90 degrees flexion of the elbow joint, palm flat in the extension of the forearm, facing the head. During the setup, the sleeve 3 can be aligned with those markers, before fixing it. See Figures 2A and 3.

[0088] Electrode placement

[0089] With the MD sEMG approach, a large number of electrodes EL are placed over a skin area without respect to the positions of underlying muscles.

[0090] To ensure maximal coverage of the muscles, even in different arm postures, and to gain spatial data, electrodes EL should be placed around the whole limb.

[0091] To simplify processing and obtain an equal spatial resolution, the inter electrode distance IED is to be kept constant, as far as the shape of the limb and the limits of the design allow. For this, electrodes EL are, for example, placed in rows equalling IED along the circumference of the forearm.

[0092] The rows are shifted with respect to each other so that neighbouring electrodes form triangles, for example, equilateral triangles. This principle is schematically shown and explained in Fig.

[0093] 3.

[0094] To mechanically keep the electrodes EL in place, holders HD are provided. These holders are sewed to the basic layer of the sleeve 3. Thereby, the mechanical connection between the sleeve 3 and the electrodes EL is created in a way that allows later replacement of the electrodes EL and thereby maintenance and a larger number of possible experimental setups. For convenience, the holders HD are, for example, 3D-printed. A schematic is shown in Fig.

[0095] 4.

[0096] The electrodes can be purchased pre-attached to cables. They are inserted through the channel on the side shown in cut B (Figure 4(i)). Then, the cable is clipped into the electrode holder HD visible in cut A (Figure 4(ii)), which is possible due to the elasticity of the cable isolation. The convexity at the bottom is made to fit the electrode size to provide good support. This ensures a stable position of the electrode EL on the sleeve 3. The holes for sewing are presented in cut C (Figure 4(iv)).

[0097] THE DECODING

[0098] The “perfect” motor decoding strategy has to be as close as possible to the natural control of the hand, hence, the inventors designed a decoding algorithm capable of decoding fingers angle continuously and simultaneously.

[0099] For grasp classification, deep learning showed higher decoding performance compared to the standard approach that relies on the extraction of discriminative features of the signal that are used as input of the machine learning model.

[0100] Therefore, the Inventors designed and compared a single finger proportional control with a deep learning approach.

[0101] Data Recording & Synchronization

[0102] For the calibration phase, finger angles are extracted and synchronized with EMG signals.

[0103] Motion capture systems or kinematic gloves are bulky and expensive and cannot be installed at home for daily calibration. A solution is to use a virtual hand that the user has to follow to infer finger angles, however, as delay and asynchronization between the user and the virtual hand can appear, this approach is not suitable for precise finger angle decoding.

[0104] Therefore, the inventors used MediaPipe, an open-source library that can infer 21 3D landmarks of the hand from a single camera 7 that is synchronized precisely with EMG signals.

[0105] Figure 7 shows the 21 exemplary hand landmarks (image courtesy of the company MediaPipe Hands). The hand landmarks are determined from camera 7 images of a user’s hand and the located landmarks allow a finger angle for each finger and the thumb to be determined based on determined joint angles. A joint angle is, for example, the angle defined between finger bones or thumb bones that articulate with respect to each other via a directly interconnecting joint. The joint angles are determined from the hand landmarks and permit to determine, for example, finger flexion / extension as well as the carpometacarpal joint angle for thumb opposition. This approach can be used with unilateral amputees who are asked to perform mirror motions between their healthy and phantom hands.

[0106] The angle extraction phase is parallelized to use the full extent of a computer’s computing capabilities. The time necessary to extract the angles is roughly equal to the length of the video divided by the number of CPU cores available (8 in general).

[0107] To pre-train the deep network and quantify performance, a substantial amount of time was spent recording data from different healthy and amputee subjects with different forearm sizes, in different arm positions performing single and multi-finger motions.

[0108] Model Training & Optimization

[0109] Training with several recordings is not trivial as all the data cannot fit at once in the system memory and the order of the inputs is preferably randomized for increased performance.

[0110] The inventors developed a framework that loads small chunks or blocks of data from each recording available in a pseudo-randomized manner to make sure the whole dataset can be used.

[0111] This framework allowed to create models trained on several GB of data.

[0112] Figure 8 shows an exemplary system 11 for recording data acquired from a wearable electromyographic support or sleeve 3, at least one or a single camera 7 and optionally inertial measurement units IMU. The system 11 further includes, for example, a computer or computer system 17 including electronic calculation or computing means 19 such as a microprocessor 19, and storage means 21 , for example, semiconductor memory, HDD, or flash memory. Further details relating to the electronic calculation or computing means 19 and the storage means 21 of the systems of the present disclosure at provided below.

[0113] The inertial measurement unit or device IMU is, for example, an electronic device configured to measure and provide data or data signals representing, for example, one or all of acceleration, orientation, angular rates. It permits to provide information concerning movement and orientation of an object to which it is attached. The inertial measurement unit may include a plurality of accelerometers (for example, 3) that provide a measure of specific force / acceleration, a plurality of gyroscopes (for example, 3) that provide a measure of angular rate, and optionally magnetometers (for example, 3). This permits, for example, to determine parameters such as roll, pitch, and yaw of an object to which the IMU is attached.

[0114] The computer 17 is configured to receive and record to the storage means 21 electromyographic (EMG) data provided by the sleeve 3, IMU data representing wrist angles provided by the IMUs and image data provided by the camera 7. The computer 17 can, for example, also communicate via communications network 23 with a server 25 to provide received hand images or video from the camera 7 to an online service or application configured to provide (for example, in real-time) hand landmarks extracted from the hand images or video, (for example, via the service or application MediaPipe Hands). The received hand landmarks or hand landmarks data are stored, for example, in the storage means 21 and the computer 17 is configured to determine the finger angles of each finger and thumb from the receive hand landmarks hand landmarks data. The storage means 21 includes, for example, one or more computer programs comprising instructions permitting to manage and operate the above-mentioned actions of the system 11 .

[0115] Figure 10 schematically shows (topmost acquisition step) some exemplary details of this data acquisition and also shows exemplary further processing of the acquired data to prepare the training data for pretraining a neural network and providing a pre-trained model or pre-trained neural network, as will now be further detailed. The system 11 is configured to carry out the data processing and preparation shown in Figure 10 and described in more detail below.

[0116] Data Recording:

[0117] Subjects are instructed to follow a sequence of finger and wrist movements repeated, for example, six times. The recording time of one session is 15 minutes. Multiple recordings from different subjects are recorded.

[0118] Setup, signal acquisition and pre-processing description:

[0119] The system 11 setup comprises, a sleeve 3 of or including, for example, 64 EMG electrodes (or less) and at least one (or more) camera 7 facing the hand 27 of a subject. In some cases, inertial measurement units (IMUs) may be added or included to also track wrist angles.

[0120] Each EMG electrode of the sleeve 3 provides a measured channel signal or measured channel data that is acquired and recorded following, for example, processing of the acquired measured channel signal or measured channel data. In the exemplary embodiment, the 64 EMG electrodes provide 64 measured channels signal or 64 channels of measured EMG data. EMG signals are recorded from the subject’s forearm, wrist angles from IMUs placed, for example, on the elbow and the back of the hand 27, and finally, hand kinematics are extracted from the images or video provided by the camera(s) 7 facing the hand 27 of the subject. Both or all signals are sent to a computer and timestamped using, for example, an application such as Lab Streaming Layer.

[0121] EMG signals are, for example, acquired at, for example, 2400Hz and are bandpass filtered between 5 and 500Hz as well as notch filtered at, for example, 50Hz. The envelope of the signal is then extracted using, for example, standard techniques (such as signal rectification, and / or low pass filtered with cutoff frequency at, for example, 2Hz) and subsampled at, for example, 100Hz (see, for example, Figure 10).

[0122] Frames of the video or images are acquired at, for example, 60Hz and processed to extract landmarks (hand landmarks), for example, using an application where the video frames are used as the input of a deep network to extract 21 landmarks of the hand in 3D (for example, via the service or application MediaPipe Hands). These landmarks are then used to determine or extract finger angles. Noise present in the deep network landmark prediction (MediaPipe’s prediction) can be removed, for example, using a Savitzky-Golay filter.

[0123] Using, for example, the Madgwick Orientation Filter, four quaternions describing the IMU’s orientation are outputted from each IMU and are acquired at, for example, 140Hz with a microcontroller connected via USB to the PC 17. The orientation difference between a reference IMU placed, for example, on the elbow and the IMU placed, for example, on the back of the hand 27 is used to extract wrist pronation / supination and wrist flexion / extension. A Savitzky-Golay filter can, for example, also be applied to remove noise in the obtained wrist angles.

[0124] Finally, wrist and finger angles are merged in a single table for prediction and all angles are rescaled between 0 and 1 .

[0125] Signal Processing:

[0126] For each recording, kinematic data (finger and wrist angles) is upsampled and interpolated to match the EMG sampling frequency (100Hz), as shown for example in Figure 11 . Each dataset is divided non-randomly into Train / Val / Test sets with a ratio of, for example, 4 / 1 / 1.

[0127] As shown in Figure 12, a for example 200ms sliding window is extracted on each set separately with overlap, for example, 75% overlap and the finger angles (and wrist angles) of the end of each window are kept. The data windows are the input of the deep model (shape: time*channels). The extracted electromyographic data and the associated kinematic target data are used to pre-train a neural network using the extracted and prepared electromyographic data and the associated kinematic target data as inputted training data.

[0128] Model pre-training:

[0129] However, as the Inventors are using deep models, it is necessary to tune a high number of parameters that cannot be optimized with a simple grid search. To address this, the Inventors developed a genetic algorithm that allows to find a good set of parameters without the need to go through all combinations possible. A subset of 4000 models was trained using the high- performance computing clusters of the EPFL to find an optimal set of parameters to train the model.

[0130] This model was then used as a starting point with further training performed with user-specific calibration data. The advantage of using a pre-trained model is the possibility to block convolutional layers of the network so that it increases the training speed. This way the Inventors reduced the training time of 50 minutes on a GPU accelerated laptop to 5 minutes on a PC without GPU.

[0131] To reduce calibration time for new subjects, a neural network NN that is, for example, a convolutional neural network (CNN) is, for example, pre-trained on data recorded from one or several subjects.

[0132] The entire dataset might not fit in the random-access memory RAM during pre-training, therefore, the data is loaded in a pseudo-random manner. More precisely, extracted windows from each available recording are loaded in chunks or blocks BK (the number of windows loaded each time depends on the available RAM or memory capacity of the training system). Each chunk BK contains random windows from the training set of each recording available. For every chunk BK, all the recordings are, for example, equally represented. Figure 13 shows the preparation of such a data chunk or data block BK. The data chunk or data block BK includes, for example, random windows of the extracted electromyographic data of the processed EMG recording as well as the associated kinematic target data for these windows. Each data chunk or block BK comprises a plurality of channel window data extracts and kinematic limb distal extremity target data associated with each channel window data extract.

[0133] Each channel window data extract is, for example, extracted from electromyographic signals of recorded electromyographic EMG channels that may for example have been subsequently processed, as previously described.

[0134] The associated kinematic upper limb distal extremity target data associated with a channel window data extract is extracted from synchronously recorded kinematic upper limb distal extremity data. The associated kinematic upper limb distal extremity target data includes kinematic upper limb distal extremity data obtained from images or video frames captured or acquired by the camera 7, or obtained from images or video frames and from data measured by the one or more inertial measurement units IMU.

[0135] Loading the data this way, the CNN architecture and hyperparameters are optimized using, for example, a genetic algorithm. During optimization, each chromosome contains a set of genes that encode hyperparameters for model architecture and training: number of convolutional layers, number of maximum pooling layers, number of filters for each layer, kernel shape, pooling shape, number of fully connected layers, number of nodes for each fully connected layer, dropout applied on all layers, L1 and L2 normalization (or least absolute deviations normalization or least squares normalization), learning rate, and optimizer type. The batch size is fixed depending on the GPU memory available, the number of epochs is set to, for example, 50 and the learning rate is, for example, divided by two every 10 epochs. The genetic algorithm can be set to run for, for example, 200 generations with a population size of, for example, 20 chromosomes. The mutation rate can be set to 0.1 and the selection rate to 0.6 keeping only the model with the best validation loss. To reduce overfitting, if the model does not get a lower validation rate for more than, for example, 13 epochs, the training is aborted. Finally, the model that outputs the lowest validation loss is selected for the next steps.

[0136] The article entitled ‘Deep learning with convolutional neural network for proportional control of finger movements from surface EMG recordings’ by Mendez V, Pollina L, Artoni F, Micera S and published in Int. IEEE / EMBS Conf. Neural Eng. NER, May 4-6 2021 , pages 1074-8, fully incorporated herein by reference, provides further exemplary details in relation to such hyperparameter optimization when training a model and CNN network.

[0137] One aspect of the present disclosure thus concerns a training method, that is, for example a computer-implemented method. The method concerns a method of training a neural network for generating upper limb distal extremity motion control signals.

[0138] The neural network is trained to configure the neural network to generate a prediction signal or predication data when user electromyographic data is inputted to the neural network from the sleeve 3 during deployment or use of the trained neural network.

[0139] The prediction signal or data is used to generate upper limb distal extremity motion control signals of distal extremities of the upper limb, for example, by providing the prediction signal or data to a controller 33 (see, for example, Figure 9) which subsequently generates the motion control signals. The prediction signal or data comprises kinematic upper limb distal extremity data. The controller 33 is configured to receive this data and to generate upper limb distal extremity motion control signals permitting to move a virtual, robotic or prosthetic upper limb to which control signals are provided by the controller 33. The kinematic upper limb distal extremity data includes, for example, data or information representing a finger angle for each finger and a thumb angle, and possibly also a wrist angle, permitting to reproduce the movement of the user of the sleeve 3.

[0140] The storage means 21 includes, for example, one or more computer programs comprising instructions permitting to manage and operate the above-mentioned actions of the system 35.

[0141] The neural network is, for example, a CNN. The upper limb distal extremities whose movement are controlled are, for example, a finger angle for each finger and the thumb, and optionally, a wrist angle.

[0142] The motion control signals are configured to operate a virtual, robotic or prosthetic upper limb 31 that is for example a virtual, or robotic or prosthetic hand 31 .

[0143] The method includes receiving EMG data and associated kinematic upper limb distal extremity target data at the neural network, or providing EMG data and associated kinematic upper limb distal extremity target data to the neural network. The EMG data and associated kinematic upper limb distal extremity target data is provided as a plurality of data chunks or blocks BK, and training of the neural network is carried out based on the plurality of data chunks or blocks BK. This permits the trained or pretrained neural network PNN to be obtained.

[0144] The EMG data and associated kinematic upper limb distal extremity target data is prepared prior to training as described previously. That is, each data chunk or block BK comprises a plurality of channel window data extracts and kinematic upper limb distal extremity target data associated with each channel window data extract. The channel window data extract is extracted from electromyographic signals of recorded electromyographic EMG channels and the associated kinematic upper limb distal extremity target data is extracted from the synchronously recorded kinematic upper limb distal extremity data. The upper limb distal extremities include, for example, fingers, thumbs and for example the wrist, and the kinematic upper limb distal extremity target data comprises, for example, kinematic data associated or produced form recorded movements of the fingers, thumb and the wrist, as previously described.

[0145] The EMG data and associated kinematic upper limb distal extremity target data is processed by the neural network.

[0146] The neural network, for example, processes the data chunks or blocks BK and, more particularly, processes the EMG data and associated kinematic upper limb distal extremity target data through the layers of the neural network in accordance with parameters associated with the layers. The parameters of the layers are updated as the multiple data chunks BK and the EMG data and associated kinematic upper limb distal extremity target data are processed by the neural network.

[0147] The method further includes optimizing the neural network architecture and hyperparameters using the genetic algorithm to provide a trained or pretrained neural network configured to generate upper limb distal extremity motion control signals to operate a virtual, robotic or prosthetic upper limb.

[0148] Each data chunk or block BK may, for example, comprise a plurality of shuffled and randomly ordered channel window data extracts and associated kinematic upper limb distal extremity target data extracted from multiple different recordings, as for example shown in Figure 13. The neural network is, for example, trained end-to-end.

[0149] The electromyographic EMG data is, for example, defined by electromyographic signals measured from the plurality of electrodes EL of the electromyographic sleeve 3, from which a signal envelope is extracted for each of the electromyographic signals. The envelope of the signal can be extracted and, for example, subsampled at low frequency, such as for instance 100Hz, to allow for faster training of models and for reducing greatly the computational power needed.

[0150] A plurality of the extracted signal envelopes can then be summed to create a plurality of signal bands, each band comprising a plurality of summed signal envelopes. The system is, for example, additionally configured to sum the signal envelopes together in bands (or “rings”) of electrodes so that the rotation of the sleeve around the forearm impacts less the decoding model. Instead of the input of shape (time*64 channels) previously mentioned in relation to Figure 12, the system uses, for example, only (time*10 rings) to reduce even more the computational burden, while keeping the performance on a single recording similar, an intersession performance greatly improved and universal wearability of the device 3 as a user does not need to worry about precise electrode positioning.

[0151] The present disclosure also concerns a computer program comprising program instructions, which when executed by the computing means or a computer cause the computing means or computer to carry out the above-mentioned training method. The present disclosure also concerns a computer-readable data carrier having the computer program stored thereon, or a data carrier signal carrying the computer program.

[0152] Online Validation

[0153] To validate the approach of the present disclosure, the Inventors developed the necessary software to stream the predicted angles in real-time. After some performance optimization steps, an average PC (intel core i5, no GPU) could record the EMG signals from the 64 channels of the sleeve at 2400Hz, make the angle prediction and stream it to a robotic or a virtual hand.

[0154] The predicted angles online can be post-processed with a moving-median filter to avoid noise that may make the fingers shake. Fine-Tuning:

[0155] The pre-trained model is then fine-trained or calibrated.

[0156] As mentioned, the present disclosure concerns the method of training the neural network NN to provide the pre-trained neural network PNN for generating upper limb distal extremity motion control signals to operate the virtual, robotic or prosthetic upper limb 31. The present disclosure also concerns a method of generating user calibrated upper limb distal extremity motion control signals for operating of the virtual, robotic or prosthetic upper limb (31 ), which may, for example, be included as part of a method of training a neural network NN in addition to the training of the neural network NN to provide the pre-trained neural network PNN. The present disclosure also concerns a computer program comprising program instructions, which when executed by the computing means or a computer cause the computing means or computer to carry out the training method, as well as a computer-readable data carrier having the computer program stored thereon, or a data carrier signal carrying the computer program.

[0157] Figures 14 and 15 show exemplary systems 29 in which the pre-trained model PNN is calibrated or fine-tuned by a user. The calibration of the neural network is short (for example, only about 5 minutes of data provided by the user is required) thanks, for example, to transfer learning applied to the pre-trained neural network PNN.

[0158] The system 29 is, for example, an upper limb distal extremity motion control signal generation system 29 that is fine-tuned to output user calibrated upper limb distal extremity motion control signals to operate a virtual, robotic or prosthetic upper limb 31. As previously mentioned, the upper limb distal extremities whose movement are controlled or operated are, for example, fingers, thumb and a wrist of a virtual, robotic or prosthetic hand 31 , for example, a finger angle for each finger and the thumb, and optionally, a wrist angle.

[0159] The neural network (or pre-trained neural network) is included as a subsystem of the system 29. The neural network subsystem NNS also includes the controller 33. As mentioned above, the controller 33 is configured to receive the prediction signals or data generated by the neural network and subsequently generate the motion control signals. The controller 33 is configured to receive this data and to generate upper limb distal extremity motion control signals permitting to move a virtual, robotic or prosthetic upper limb 31 to which control signals are provided by the controller 33. The kinematic upper limb distal extremity data includes, for example, data or information representing a finger angle for each finger and a thumb angle, and possibly also a wrist angle, permitting to reproduce the movement of the user of the sleeve 3.

[0160] The neural network subsystem NNS is configured to operate in a training mode or a predictor mode. In the training mode, new user provided data is received by the neural network and processed thereby to update or fine-tune the neural network. In a predictor mode, the latest model or fine-tuned neural network is deployed and used by the subsystem to make predictions and to generate user calibrated upper limb distal extremity motion control signals to operate a virtual, robotic or prosthetic upper limb 31 .

[0161] The system 29 includes similar components to the data recording system 11 (Figure 8) and includes the wearable electromyographic support or sleeve 3, the camera 7 and optionally inertial measurement units IMU. The sleeve 3, the camera 7 and optionally inertial measurement units IMU may locally in connection with the computer 17 and may locally and / or directly communicate with the computer 17, or may alternatively communicate with the computer 17 via an external communications network (not shown). The system 29 further includes, for example, the computer 17 including the electronic calculation or computing means 19 and storage means 21.

[0162] The electronic calculation or computing means 19 may, for example, comprise one or more processors or microprocessors or processing devices, or microcontrollers, microcomputers, programmable logic controllers (PLC), or application specific integrated circuits, and other programmable circuits.

[0163] The storage means 21 may, for example, comprise a computer-readable medium or computer- readable memory for example a semiconductor memory, or a non-volatile medium or memory such as flash memory; or comprise for example a hard disk drive HDD. The storage means 21 may be removable or non-removable. The computer-readable medium or memory is, for example, a non-transitory computer-readable medium or memory. The storage means 21 may comprise, for example, random-access memory, semiconductor memory, HDD, or flash memory. The computer 17 can, for example, also communicate via communications network 23 with a server 25 to provide received hand images or video from the camera 7 to an online service or application configured to provide (for example, in real-time) hand landmarks extracted from the hand images or video, (for example, via the service or application MediaPipe Hands). Alternatively, the hand landmarks may be extracted from the hand images or video that is processed locally, for example, in or by the computer 17. The neural network subsystem NNS may include, for example, the microprocessor 19 and storage means 21 which includes the neural network NN (pre-trained neural network PNN) and the constituent features or elements of the neural network, as well as updated neural networks or models. The storage means 21 includes, for example, one or more computer programs comprising instructions permitting to manage and operate the actions of the system 29 described herein.

[0164] For each new recording (different subject or same subject on a different day), the subject is, for example, asked to follow the same sequence of movements as those performed during pre-training. In the fine-tuning phase, the computer 17 or computer operation is, for example, compartmentalized into 3 processes or 3 parallel processes and configured to carry out these processes. One process is or comprises, for example, acquisition / recording, synchronizing, and saving the data acquired from the camera(s) 7, IMUs, and the EMG sleeve 3. The second process is or comprises, for example, extracting the hand landmarks and finger angles from the video frames, for example, as mentioned previously using an external application accessed via the network 23, for example MediaPipe hand in real-time. The third parallel process loads all the data available (for example, up to 2 minutes in the past) to fine-tune the CNN and saves a version as soon as the training has done, for example, 20 epochs. Every ~40 seconds for example, a new model is ready and is loaded from the first process that will use the received data to feed the trained model to output predictions. The output predictions are, for example, provided by the neural network subsystem NNS and the controller 33 operates the virtual, robotic or prosthetic upper limb 31 so that the user is presented with the output prediction results during this further training or fine-tuning phase.

[0165] As mentioned, the neural network NN is provided, loaded or fed in real-time with this new or further fine-tuning training data comprising the acquired and processed EMG signals and kinematic data. The neural network NN is provided, loaded or fed in real-time simultaneously to the measurements and acquisitions being obtained from the subject carrying out the sequence of movements that allows (further) fine-tuning training data to be obtained.

[0166] When the user is satisfied with the outputs of the trained model (usually after 5 minutes), the user can remove their hand from the camera 7 and the last (updated or fined-tuned) model will be used to predict angles from EMG signals only, and training will stop. The model can be fine-tuned by changing fewer hyperparameters. The learning rate is reduced, for example, an Adam optimizer can be used even if the genetic algorithm found another optimizer to be best.

[0167] The present disclosure also thus concerns such a fine-tuning method, that is, for example a computer-implemented method. This method concerns a method of generating user calibrated upper limb distal extremity motion control signals for operating of the virtual, robotic or prosthetic upper limb 31 .

[0168] The method includes providing user EMG data and associated user kinematic upper limb distal extremity target data to a pre-trained neural network PNN of the neural network subsystem NNS of the system 29. The pre-trained neural network subsystem is configured to output control signals for operating the virtual, robotic or prosthetic upper limb 31 based on inputted EMG data.

[0169] The pre-trained neural network is now trained using user EMG data and associated user kinematic upper limb distal extremity target data to update, for example, a subset of hyperparameters of the pre-trained neural network PNN to provide a fine-tuned pre-trained neural network.

[0170] The pre-trained neural network PNN is a trained neural network trained using EMG data and associated kinematic upper limb distal extremity target data recorded from an upper limb and limb extremities of a plurality of different humans.

[0171] The EMG data and associated kinematic upper limb distal extremity target data is processed by the trained neural network. The EMG data and associated kinematic upper limb distal extremity target data is processed through the layers of the neural network in accordance with parameters associated with the layers. A limited number of parameters of the layers are updated.

[0172] The neural network is, for example, a convolutional neural network CNN and the hyperparameters of the convolutional layers are, for example, blocked or unchanged. The hyperparameters of the fully connected layers are modified. The hyperparameters of the convolutional layers are, for example, blocked or unchanged and the hyperparameters of the fully connected layers are, for example, modified when providing user EMG data and associated user kinematic upper limb distal extremity target data to the data to the pre-trained neural network PNN to train the pre-trained neural network PNN. This is done, for example, prior to starting fine-tuning of the pre-trained neural network PNN. As mentioned, an optimization algorithm, for example, an Adam optimization algorithm can be used, for example, to update the subset of hyperparameters of the neural network. The Adam optimizer (Adam optimization algorithm), or Adaptive Moment Estimation optimization algorithm permits to minimize a loss function during the training of the neural network NN.

[0173] The method also includes registering or storing constituent features or elements of the finetuned pre-trained neural network. The fine-tuned or updated neural network or model can be loaded for use in the prediction mode.

[0174] The registered fine-tuned pre-trained neural network TNN can then be used to provide updated prediction signals or data and used by the controller 33 to output user calibrated upper limb distal extremity motion control signals to operate the virtual, robotic or prosthetic upper limb 31.

[0175] Further user data and associated user kinematic upper limb distal extremity target data can be provided at least once more or a plurality of times, and training is again carried out on the fine-tuned pre-trained neural network TNN to update a subset of hyperparameters to provide an updated fine-tuned neural network. The updated fine-tuned neural network is registered and used in the generation of user calibrated upper limb distal extremity motion control signals to operate the virtual, robotic or prosthetic upper limb 31 .

[0176] The method also includes stopping training of the neural network and providing user EMG data to the registered neural network to output the user calibrated upper limb distal extremity motion control signals to operate the virtual, robotic or prosthetic upper limb 31.

[0177] Data acquired by a user wearable electromyographic sleeve 3 is recorded as well as data from the camera 7 and the inertial measurement unit IMU. The user wearable electromyographic support data is synchronized with the camera and the IMU data. User kinematic upper limb distal extremity target data is determined from video frames provided by camera and the IMU data.

[0178] The provision of the user EMG data includes receiving electromyographic signals from a plurality of electrodes EL of the user wearable electromyographic sleeve 3, extracting a signal envelope for each of the received electromyographic signals, summing a plurality of the extracted signal envelopes to create a plurality of signal bands, where each band comprising a plurality of summed signal envelopes.

[0179] The present disclosure also concerns a computer program comprising program instructions, which when executed by the computing means 19 or a computer 17 cause the computing means 19 or computer 17 to carry out the above-mentioned method. The present disclosure also concerns a computer-readable data carrier having the computer program stored thereon, or a data carrier signal carrying the computer program.

[0180] A further aspect of the present disclosure concerns the upper limb distal extremity motion control signal generation system 29 including the computing means 19, and storage means 21 including the at least one computer program comprising program instructions, which when executed by the computing means 19 cause the computing means 19 to carry out the abovedescribed method.

[0181] The storage means 21 includes, for example, one or more computer programs comprising instructions permitting to manage and operate the above-mentioned actions of the system 29.

[0182] Another aspect of the present disclosure concerns a method or computer-implemented method of generating upper limb distal extremity motion control signals to operate the virtual, robotic or prosthetic upper limb 31 . The method uses the trained neural network that has been trained as described above based on a transfer learning approach and a transfer learning trained neural network pre-trained and the fine-tuned as described previously. The method can for example be carried out using the exemplary system 35 shown in Figure 9 and illustrated by the exemplary system 35B in Figure 17.

[0183] This method of generating user calibrated upper limb distal extremity motion control signals for operating of the virtual, robotic or prosthetic upper limb 31 of the present disclosure may, for example, be included in the method of training the neural network NN in addition to the training of the neural network NN to provide the pre-trained neural network PNN and the finetuned neural network. The present disclosure also concerns a computer program comprising program instructions, which when executed by the computing means or a computer cause the computing means or computer to carry out this training method, as well as a computer- readable data carrier having the computer program stored thereon, or a data carrier signal carrying the computer program.

[0184] T1 The method, for example, comprises providing user EMG data to the trained neural network TNN of the neural network subsystem NNS of the upper limb distal extremity motion control signal generation system 35B to generate a prediction signal or data after processing of the user EMG data by the trained neural network. As mentioned, the trained neural network is a transfer learning trained neural network. The trained neural network subsystem NNS is configured to output control signals for operating the virtual, robotic or prosthetic upper limb 31 based on the inputted EMG data to the transfer learning trained neural network TNN. The neural network subsystem NNS is used to output upper limb distal extremity motion control signals based on the prediction signal or data to operate the virtual, robotic or prosthetic upper limb 31.

[0185] The storage means 21 includes, for example, one or more computer programs comprising instructions permitting to manage and operate the above-mentioned actions of the system 35B.

[0186] Figure 16 shows an overview of a transfer learning pipeline of the present disclosure in which transversal transfer learning (along several users) is performed (to provide the pretrained model or neural network PNN), and then longitudinal transfer learning (along, for example, several trainings of the same user) is carried out to calibrate the system and / or neural network NN for the end user.

[0187] The approach described above in relation to an upper limb is equally applicable to a lower limb 27A that is, for example, a leg.

[0188] A further aspect of the present disclosure concerns a method or a computer-implemented method of training a neural network to generate lower limb motion control signals or distal extremity motion control signals to operate a virtual, robotic or prosthetic lower limb 31 B.

[0189] The present disclosure further concerns a method or a computer-implemented method of generating user calibrated lower limb motion control signals or distal extremity motion control signals for operating of a virtual, robotic or prosthetic lower limb 31 B. The present disclosure also further concerns a lower limb motion control signal or distal extremity motion control signal generation system. The lower limb 27A is, for example, a leg. The kinematic lower limb target data or distal extremity target data or joint target data includes kinematic lower limb distal extremity data similarly obtained from images or video frames captured by the camera 7.

[0190] Lower Limb Kinematic Analysis for Knee Joint or Ankle Joint Decoding:

[0191] The previously described medium-density electromyographic (MD-EMG) system 11 , 29, 35 as for example shown in Figures 8, 9, 14 and 17 can be placed on several portions of the human body where muscles are present below the skin and used in other parts of the human body in addition to the upper limb. The generalizability of the MD EMG device from upper limb to lower limb applications is supported by the underlying principle of muscle activity interpretation and joint angle decoding. Similar to how the system decodes finger movements from forearm muscle activity, (Figure 8), or elbow angle (elbow flexion / extension movement) from arm muscles (Figure 5), the same approach is applied to decode knee and ankle movements from the muscular activities of the thigh and calf.

[0192] This continuity in application is further confirmed by existing scientific literature, such as the studies referenced below, which demonstrate the feasibility of using EMG and IMU data for lower limb motion analysis and control.

[0193] This further embodiment relates to the medium-density electromyographic (MD EMG) device as previously described for upper limb applications, and now used for precise knee and / or ankle joint motion analysis in the lower limb. The system employs the elastic sleeve 3 embedded with medium-density EMG sensors, with the same design but resized to encircle the thigh and / or calf, and configured to provide a flexible fit to accommodate various leg sizes. Figure 18 shows exemplary positions of the user wearable electromyographic support or sleeve 3 on the lower limb (leg) for measuring or obtaining EMG data or signals.

[0194] System Configuration for Knee joint and ankle Joint Decoding:

[0195] MD EMG Sleeve Placement: The sleeve 3 is positioned, for example, in the middle of the thigh or calf (see, for example, Figure 18) to capture EMG signals from the muscles influencing knee joint movements or the ankle joint movements.

[0196] Knee joint angle and ankle joint angle Recording: The knee joint angles and the ankle joint angles are to be recorded. Any motion tracking system would be sufficient however, the inventors recorded leg kinematics using the full body model of MediaPipe (see references) that tracks 33 body landmark locations including landmarks that allow to capture and determine lower limb movements and to capture and determine the dynamic movements of the knee and ankle accurately.

[0197] Figure 19 shows exemplary bodily or pose landmarks (courtesy of the company MediaPipe) and relevant lower limb landmarks that can be determined and located on a body from camera images or video, and that can then be used to extract lower limb joint angles for the knee and ankle.

[0198] The leg landmarks are determined from camera 7 images of a user’s body and the located landmarks allow a knee joint angle and / or ankle joint angle for each orientation to be determined based on determined joint angles. A joint angle is, similarly to the upper limb case, for example, the angle defined between leg bones that articulate with respect to each other via a directly interconnecting joint.

[0199] The joint angles are determined from the leg landmarks and permit to determine, for example, knee flexion / extension, internal / external rotations as well as varus / valgus angles. For the ankle, the possible angles that can be recorded are Plantarflexion / dorsiflexion and Inversion / eversion. It is also possible to use IMU data similarly to the upper limbs with wrist angle extraction.

[0200] For the calibration when the MD EMG system is placed on the lower limb, the process mirrors the upper limb approach but now with leg movements. The participant performs a series of leg movements, to articulate the knee and / or ankle joint in a controlled manner. The person is, for example, seated on a high chair where their legs are not touching the ground and is asked to flex completely the knee, come back to a resting position, and extend completely before coming back to a resting position. The same applies to all the other degrees of freedom DoF of the joint (maximum in one direction, rest, maximum in the other direction, rest). Some angle combinations of the same joint can be part of the sequence (e.g. flexion + rotation).

[0201] In the case of unilateral amputation, the patient is asked to perform the task with both legs simultaneously. In the case of healthy individuals, the subject can walk in free space in the field of view of the camera 7, and their joint angles are mapped with EMG signals such that muscle activity is closer to the real use case. This data can be used to generalize to patients with an amputation from seated position to walking for instance. The following is an exemplary description as to how the above-mentioned joint angles are determined by or using the Mediapipe application. The frame should be rotated from the camera frame (x horizontal, y vertical, z through the camera) to the specific "thigh" reference frame where x_thigh is along the tight, z_thigh is defined by the cross product of the tight, and the calf segments (vectors 26-24 and 26-28 in mediapipe landmarks) and y_thigh is then defined by the cross product of x_thigh and z_thigh. Now that one has the frame, one expresses the vectors in this new reference frame using a 3D rotation matrix. Then, one extracts the angles of interest. One defines an angle by 3 points A-B-C and a plane to project the angle on. Then the angle is extracted on this plane.

[0202] The follwing are, for example, the angles + plane projected with the Mediapipe landmarks if one supposes a person sitting on a high chair: knee flexion / extension is the angle of 26-24 and 26-28 on the plane defined by the vectors x_thigh , y_thigh. varus / valgus is the angle of 26-24 and 26-28 on the plane defined by the vectors y_thigh and z_thigh.

[0203] Knee internal / external rotation angle is defined by the angles of 24-26 and 28-32 on the place defined by x_thigh and z_thigh.

[0204] For the ankle, a similar approach is used to get the specific "calf reference frame. x_calf is along the calf, z_calf is defined by the cross product of the calf and the foot segments (28-26 and 28-32) and y_calf is defined by the cross-product of x_calf and z_calf. Vectors are now expressed in this new frame.

[0205] Ankle Plantarflexion / dorsiflexion is the angle between 26-28 and 28-32 on the plane defined by the vectors x_calf and y_calf.

[0206] Inversion / eversion is the angle between 28-26 and 28-30 on the place defined by x_calf and z_calf. Data Synchronization: As for the previously described upper limb embodiment, the EMG signals and the knee and / or ankle joint angle data are synchronized to ensure a precise correlation between muscle activities and the corresponding knee and / or ankle movements. This synchronization is important for the data processing and model training phases that follow.

[0207] Data Processing and Model Training for Knee Joint or ankle joint Control:

[0208] Signal Processing: The EMG signals from the thigh are processed, involving filtering and envelope extraction, paralleling the upper limb process. The knee joint and / or the ankle joint angle data, processed in the same way, is synchronized with the EMG signal frequency to ensure compatibility for model training.

[0209] Model Architecture Optimization for Knee or ankle Control: Utilizing a genetic algorithm, the neural network model architecture is optimized for knee or ankle joint motion decoding from thigh EMG signals or calf EMG signals. This optimization process is exactly the same as for the upper limb, the optimization objective is to reduce regression errors of the predicted joint angle based on EMG signals and true join angle.

[0210] Real-Time Training and Calibration for Knee Joint or ankle joint Decoding: The real-time training and calibration process, adopted from the upper limb embodiment, is applied to personalize the model to the user's specific knee or ankle movement patterns and muscle activations. This personalized calibration ensures accurate knee or ankle joint control, whether for prosthetic, exoskeletons applications or rehabilitation exercises.

[0211] The application Scenarios of this embodiment are:

[0212] • Lower Limb Prosthetics: The system and method can be utilized to provide intuitive control over prosthetic legs, enhancing the user's mobility and comfort.

[0213] • Rehabilitation: In rehabilitation settings, the system and method aids in the monitoring and analysis of lower limb movements, offering valuable insights for therapy adjustments and progress tracking.

[0214] • Exoskeletons: For lower limb exoskeletons, the system and method ensures that the device's movements are in harmony with the user's intended motions, providing support and strength augmentation. The methods, algorithms and or procedures described herein may be encoded as executable instructions embodied in a tangible, non-transitory, computer readable medium, including, without limitation, a storage device and / or a memory device. Such instructions, when executed by a processor, cause the processor to perform at least a portion of the methods, algorithms or procedures described herein.

[0215] While the invention has been disclosed with reference to certain preferred embodiments, numerous modifications, alterations, and changes to the described embodiments, and equivalents thereof, are possible without departing from the sphere and scope of the invention. Accordingly, it is intended that the invention not be limited to the described embodiments, and be given the broadest reasonable interpretation in accordance with the language of the appended claims.

[0216] REFERENCES

[0217] 1. High-density EMG, IMU, kinetic, and kinematic open-source data for comprehensive locomotion activities - DOI: https: / / doi.org / 10.1038 / s41597-023-02679-x

[0218] 2. Personalized Human Activity Recognition Based on Integrated Wearable Sensor and Transfer Learning - DOI : https: / / doi.org / 10.3390 / s21030885

[0219] 3. Accurate recognition of lower limb ambulation mode based on surface electromyography and motion data using machine learning - DOI: https: / / doi.Org / 10.1016 / j.cmpb.2020.105486

[0220] 4. sEMG-signal and IMU sensor-based gait sub-phase detection and prediction using a user- adaptive classifier- DOI: https: / / doi.org / 10.3390 / s21030885

[0221] 5. Estimation of the Continuous Walking Angle of Knee and Ankle (Talocrural Joint, Subtalar Joint) of a Lower-Limb Exoskeleton Robot Using a Neural Network - DOI: https: / / doi.org / 10.3390 / s21082807

[0222] 6. G. Hajian et aL, "Generalizing Upper Limb Force Modeling With Transfer Learning: A Multimodal Approach Using EMG and IMU for New Users and Conditions," in IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 32, pp. 391-400, 2024, doi: 10.1109 / TNSRE.2024.3351829.

[0223] 7. EMG and IMU Data Fusion for Locomotion Mode Classification in Transtibial Amputees - DOI: https: / / doi.Org / 10.3390 / prosthesis5040085

[0224] 8. Mediapipe - https: / / developers.google.com / mediapipe / solutions / vision / pose_landmarker

Claims

CLAIMS1. A computer-implemented method of training a neural network (NN) for generating upper limb distal extremity motion control signals or lower limb motion control signals to operate a virtual, robotic or prosthetic upper or lower limb (31 , 31 B), the method comprising: providing or receiving, at a neural network (NN), electromyographic (EMG) data and associated kinematic upper limb distal extremity target data or associated kinematic lower limb target data provided as a plurality of data chunks or blocks (BK), and processing the plurality of data chunks or blocks (BK) through the neural network (NN) to carry out training of the neural network (NN) based on the plurality of data chunks or blocks (BK); wherein each data chunk or block (BK) comprises a plurality of channel window data extracts and kinematic upper limb distal extremity target data or kinematic lower limb target data associated with each channel window data extract, and wherein each channel window data extract is extracted from electromyographic signals of recorded electromyographic (EMG) channels and the associated kinematic upper limb distal extremity target data is extracted from synchronously recorded kinematic upper limb distal extremity data or the associated kinematic lower limb target data is extracted from synchronously recorded kinematic lower limb data, wherein the associated kinematic upper limb distal extremity target data includes kinematic upper limb distal extremity data obtained from images or video frames captured by at least one camera (7), or obtained from images or video frames captured by at least one camera (7) and from data measured by at least one inertial measurement unit (IMU); or the associated kinematic lower limb target data includes kinematic lower limb data obtained from images or video frames captured by at least one camera (7), or obtained from images or video frames captured by at least one camera (7) and from data measured by at least one inertial measurement unit (IMU); optimizing the neural network architecture and hyperparameters using an optimization algorithm to provide a pretrained neural network (PNN) for generating upper limb distal extremity motion control signals or lower limb motion control signals to operate a virtual, robotic or prosthetic upper or lower limb (31 , 31 B).

2. The computer-implemented method according to claim 1 , wherein each data chunk or block (BK) comprises a plurality of shuffled and randomly ordered channel window data extracts and associated kinematic upper limb distal extremity target data or kinematic lower limb target data extracted from multiple different recordings.

3. The computer-implemented method according to any one of the previous claims, wherein the electromyographic (EMG) data is defined by electromyographic signals measuredfrom a plurality of electrodes (EL) of a user wearable electromyographic support (3), from which a signal envelope is extracted for each of the electromyographic signals, and a plurality of the extracted signal envelopes is summed to create a plurality of signal bands, each band comprising a plurality of summed signal envelopes.

4. The computer-implemented method according to any one of the previous claims, wherein each of the data chunks or blocks (BK) of the plurality of data chunks or blocks (BK) is individually processed through the neural network (NN) to carrying out the training of the neural network (NN).

5. The computer-implemented method according to any one of the previous claims, wherein the electromyographic (EMG) channel is a signal or data channel that provides or communicates measured electromyographic signals from at least one electromyographic (EMG) measurement electrode (EL) attached to an upper or lower limb of at least one human.

6. The computer-implemented method according to any one of the previous claims, wherein the optimization algorithm includes is used to update the hyperparameters or a subset of the hyperparameters of the neural network (NN).

7. The computer-implemented method according to any one of the previous claims, wherein the optimization algorithm permits to minimize a loss function during the training of the neural network (NN).

8. The computer-implemented method according to any one of the previous claims, wherein the neural network (NN) is a convolutional neural network (CNN).

9. The computer-implemented method according to the previous claims, wherein elements of the neural network architecture and the hyperparameters optimized using the optimization algorithm include at least one or a plurality of items of the following group: number of convolutional layers, number of maximum pooling layers, number of filters for each layer, kernel shape, pooling shape, number of fully connected layers, number of nodes for each fully connected layer, dropout applied on all layers, L1 normalization, L2 normalization, learning rate, optimizer type.

10. The computer-implemented method according to the previous claims, further including generating user calibrated upper limb distal extremity motion control signals or lower limb motion control signals for operating the virtual, robotic or prosthetic upper or lower limb (31 , 31 B), by: providing user electromyographic (EMG) data and associated user kinematic upper limb distal extremity target data or associated user kinematic lower limb target data to the pre-trained neural network (PNN) of a neural network subsystem (NNS) of an upper or lower limb motion control signal generation system (29), wherein the pre-trained neural network subsystem (NNS) is configured to output control signals for operatingthe virtual, robotic or prosthetic upper or lower limb (31 , 31 B) based on inputted electromyographic (EMG) data, wherein the associated kinematic upper limb distal extremity target data includes kinematic upper limb distal extremity data obtained from images or video frames captured by at least one camera (7), or obtained from images or video frames captured by at least one camera (7) and from data measured by at least one inertial measurement unit (IMU), or the associated kinematic lower limb target data includes kinematic lower limb data obtained from images or video frames captured by at least one camera (7), or obtained from images or video frames captured by at least one camera (7) and from data measured by at least one inertial measurement unit (IMU); training the pre-trained neural network (PNN) using the user electromyographic (EMG) data and the associated user kinematic upper or lower limb distal extremity target data to update a subset of hyperparameters of the pre-trained neural network (PNN) to provide a fine-tuned pre-trained neural network (TNN); registering the fine-tuned pre-trained neural network; and using the registered fine-tuned pre-trained neural network of the neural network subsystem (NNS) to output user calibrated upper or lower limb distal extremity motion control signals to operate a virtual, robotic or prosthetic upper or lower limb (31 , 31 B).11 . The computer-implemented method according to claim 10, further comprising, at least once or a plurality of times, providing further user electromyographic (EMG) data and associated user kinematic upper limb distal extremity target data or associated user kinematic lower limb target data, training the fine-tuned pre-trained neural network to update a subset of hyperparameters to provide an updated fine-tuned neural network, registering the updated fine-tuned neural network and using the updated fine-tuned neural network to output user calibrated upper limb distal extremity motion control signals or user calibrated lower limb motion control signals to operate a virtual, robotic or prosthetic upper or lower limb (31 , 31 B).

12. The computer-implemented method according to claim 10 or 11 , wherein the pretrained neural network (PNN) is a trained neural network trained using electromyographic (EMG) data and associated kinematic upper limb distal extremity target data recorded from upper limb distal extremities of a plurality of different humans or associated kinematic lower limb target data recorded from the lower limb of a plurality of different humans.

13. The computer-implemented method according to claim 11 or 12, further comprising stopping training of the neural network (NN) and providing user electromyographic (EMG) data to the registered neural network to output user calibrated upper limb distal extremity motion control signals or user calibrated lower limb motion control signals to operate the virtual, robotic or prosthetic upper or lower limb (31 , 31 B).

14. The computer-implemented method according to any one of the previous claims 10 to13, wherein the neural network (NN) is a convolutional neural network (CNN) and the hyperparameters of the convolutional layers are blocked or unchanged and the hyperparameters of the fully connected layers are modified when providing the user electromyographic (EMG) data and associated user kinematic upper limb distal extremity target data or associated user kinematic lower limb target data to the pre-trained neural network (PNN) to train the pre-trained neural network (PNN).

15. The computer-implemented method according to any one of the previous claims 10 to14, wherein an optimization algorithm is used to update the subset of hyperparameters of the neural network.

16. The computer-implemented method according to any one of the previous claims 10 to15, further comprising recording data acquired from a user wearable electromyographic support (3), a camera (7) and at least one inertial measurement unit (IMU), synchronizing the user wearable electromyographic support data with the camera (7) and the at least one inertial measurement unit (IMU) data, and determining user kinematic upper limb distal extremity target data or user kinematic lower limb target data from video frames provided by the camera (7) and the at least one inertial measurement unit (IMU) data.

17. The computer-implemented method according to any one of the previous claims 10 to16, further comprising, to provide the user electromyographic (EMG) data, receiving electromyographic signals from a plurality of electrodes (EL) of a user wearable electromyographic support (3), extracting a signal envelope for each of the received electromyographic signals, summing a plurality of the extracted signal envelopes to create a plurality of signal bands, each band comprising a plurality of summed signal envelopes.

18. A computer program comprising instructions which, when the program is executed by a computer (17), cause the computer to carry out the method of any one of claims 1 to 17.

19. Upper limb distal extremity motion control signal or lower limb motion control signal generation system (29) including:- at least one user wearable electromyographic support (3),- at least one camera (7),- at least one inertial measurement unit (IMU),- computing means (19), wherein the at least one user wearable electromyographic support (3), the at least one camera (7), and the at least one inertial measurement unit (IMU) are operatively connected to the computing means (19) to provide measured or captured signals or data thereto,- a virtual, robotic or prosthetic upper or lower limb (31 , 31 B);- at least one controller (33) operatively connected to the to computing means (19) and the virtual, robotic or prosthetic upper or lower limb (31 , 31 B), the virtual, robotic or prosthetic upper or lower limb (31 , 31 B) being configured to respond to upper limb distal extremity motion control signals or lower limb motion control signals provided by the controller (33), and- storage means (21 ) including at least one computer program comprising program instructions, which when executed by the computing means (19) cause the computing means (19) to carry out the method of any one of claims 10 to 17.

20. The computer-implemented method according to any one of the previous claims, further including generating upper limb distal extremity motion control signals or lower limb motion control signals to operate the virtual, robotic or prosthetic upper or lower limb (31 , 31 B), by: providing user electromyographic (EMG) data to the pre-trained neural network (PNN) or the fine-tuned pre-trained neural network (TNN) of a neural network subsystem (NNS) of an upper limb distal extremity motion control signal or lower limb motion control signal generation system (29, 35B) to generate a prediction signal or data after processing of the user electromyographic (EMG) data by the pre-trained neural network (PNN) or the fine-tuned pre-trained neural network (TNN), wherein the trained neural network subsystem (NNS) is configured to output control signals for operating the virtual, robotic or prosthetic upper or lower limb (31 , 31 B) based on the inputted electromyographic (EMG) data to the pre-trained neural network (PNN) or the finetuned pre-trained neural network (TNN), and wherein the pre-trained neural network or the fine-tuned pre-trained neural network (TNN) is a transfer learning trained neural network; and using the neural network subsystem (NNS) to output upper limb distal extremity motion control signals or lower limb motion control signals based on the prediction signal or data to operate the virtual, robotic or prosthetic upper or lower limb (31 , 31 B).21 . A computer program comprising instructions which, when the program is executed by a computer (17), cause the computer to carry out the method of any one of claims 19 to 20.

Images