Method and apparatus for providing submuscular control
Through multiple neuromuscular sensors and computer processors, the neuromuscular signal is analyzed, the activation mode of the submuscular structure is identified and the control signal is generated, which solves the problems of information loss and insufficient control accuracy in the prior art, and achieves a more efficient submuscular structure Activate recognition and device control.
Patent Information
- Application Number
- CN201980049646.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-05-25
- Filing Date
- 2019-04-19
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2039-04-19
AI Technical Summary
The prior art is difficult to effectively identify and control the activation mode of the human body's sub-muscle structure, resulting in the loss of information and insufficient control accuracy in motion control and equipment operation.
Using multiple neuromuscular sensors and computer processors, the activation mode of submuscular structures is identified by analyzing and processing neuromuscular signals, and a control signal for different submuscular control channels is generated to control the operation of the device.
The recognition accuracy and amount of control information for submuscular structural activation are improved, the control accuracy and reliability of device operation is enhanced, user fatigue is reduced, and recognition accuracy is maintained in different sessions.
Smart Images

Figure CN112469469B_ABST
Abstract
Description
[0001] Related Applications
[0002] This application claims priority under 35 U.S.C. §119(e) to U.S. Provisional Application Serial No. 62 / 676,567, filed on May 25, 2018, entitled “METHODS AND APPARATUS FOR PROVIDING SUB-MUSCULAR CONTROL,” the entire contents of which are incorporated herein by reference.
[0003] background
[0004] Neuromuscular signals generated by the human central nervous system can reflect neural activation that causes one or more muscles in the human body to contract. An example of a neuromuscular sensor includes an electromyography (EMG) sensor, which is placed on the surface of the human body and records the neuromuscular activity generated when skeletal muscle cells are activated. The neuromuscular activity measured by the neuromuscular sensor can be caused by neural activation, muscle excitation, muscle contraction, or a combination of neural activation and muscle contraction. The electrical signals recorded by the neuromuscular sensor can be used to evaluate neuromuscular dysfunction in patients with motor control disorders and can be used as control signals for devices (such as prostheses) in some applications.
[0005] Overview
[0006] The coordinated movement of skeletal muscles that together result in the performance of motor tasks in the human body is caused by nerve signals generated in the central nervous system. Nerve signals travel from the central nervous system to the muscles via spinal motor neurons, each of which has a cell body in the spinal cord and an axon terminal on one or more muscle fibers. In response to receiving the nerve signals, the muscle fibers contract, resulting in muscle movement. The spinal motor neurons and the muscle fibers they innervate are collectively referred to as "motor units". Muscles typically include muscle fibers from hundreds of motor units, and muscle contraction that results in muscle movement typically requires simultaneous contraction of muscle fibers in multiple motor units.
[0007] Neuromuscular sensors (such as EMG sensors) record biological signals that cause motor activity (such as muscle contraction). In the case where the EMG sensors are arranged on the surface of the human body, the recorded biological signals are related to the generation of action potentials in muscle fibers. Some embodiments are intended to analyze neuromuscular signals to identify patterns of activation associated with biological structures beneath the muscle (e.g., a single motor unit or a group of motor units). Control signals determined based on the activation of the structures beneath the muscle can be used to control the operation of the device.
[0008] Some embodiments relate to a computerized system. The computerized system includes a plurality of neuromuscular sensors configured to record a plurality of neuromuscular signals from a user, and at least one computer processor, wherein the plurality of neuromuscular sensors are arranged on one or more wearable devices. The at least one computer processor is programmed to derive information for a first submuscular control channel of a plurality of submuscular control channels from the plurality of neuromuscular signals, wherein each of the plurality of submuscular control channels is configured to process information associated with activation of one or more submuscular structures; generate a control signal based on the derived information for the first submuscular control channel; and provide the control signal to a control interface to control the operation of the device.
[0009] In at least one aspect, the at least one computer processor is further programmed to provide the derived information for the first submuscular control channel as an input to the trained inference model, and wherein generating the control signal comprises generating the control signal based on an output of the trained inference model.
[0010] In at least one aspect, the at least one computer processor is further programmed to provide a plurality of neuromuscular signals as input to the trained inference model, and wherein deriving information for the first submuscular control channel comprises deriving the information based on processing of the plurality of neuromuscular signals by the trained inference model.
[0011] In at least one aspect, deriving information for a first submuscular control channel includes decomposing a plurality of neuromuscular signals into signal components representative of a plurality of submuscular control channels.
[0012] In at least one aspect, the first submuscular control channel is configured to process information generated by activation of a single motor unit.
[0013] In at least one aspect, a second submuscular control channel of the plurality of submuscular control channels is configured to process information generated by activation of the plurality of motor units.
[0014] In at least one aspect, the first submuscular control channel is configured to process information resulting from combined activation of at least one first submuscular structure associated with the first muscle and at least one second submuscular structure associated with the second muscle.
[0015] In at least one aspect, the computerized system further comprises at least one auxiliary sensor configured to record an auxiliary signal while recording the plurality of neuromuscular signals from the user, wherein the at least one auxiliary sensor is disposed on the one or more wearable devices.
[0016] In at least one aspect, the at least one auxiliary sensor includes at least one inertial measurement unit.
[0017] In at least one aspect, at least one computer processor is further programmed to, store a representation of derived information for a first submuscular control channel determined during a first session; and calibrate neuromuscular signal data recorded from the user during a second session following the first session, wherein the calibration is performed at least in part based on the stored representation of derived information for the first submuscular control channel determined during the first session.
[0018] Some embodiments are directed to a computerized system for training a user to activate submuscular structures. The system includes a plurality of neuromuscular sensors configured to record a plurality of neuromuscular signals from a user when the user activates one or more submuscular structures, wherein the plurality of neuromuscular sensors are arranged on one or more wearable devices; and at least one computer processor. The at least one computer processor is programmed to provide feedback to the user based on the plurality of neuromuscular signals, wherein the feedback includes information about a pattern of activation identified in the plurality of neuromuscular signals; adjust the feedback provided to the user based on the recorded plurality of neuromuscular signals; and store information mapping the pattern of activation identified in the plurality of neuromuscular signals to a control signal.
[0019] In at least one aspect, the submuscular structure is a single motor unit.
[0020] In at least one aspect, providing feedback to the user based on the plurality of neuromuscular signals includes generating the feedback from an unprocessed version of the plurality of neuromuscular signals.
[0021] In at least one aspect, at least one computer processor is also programmed to determine, based on the plurality of neuromuscular signals, which of the plurality of submuscular structures the user has activated, from the plurality of submuscular structures, and wherein providing feedback to the user based on the plurality of neuromuscular signals includes providing first feedback when it is determined that a first submuscular structure of the plurality of submuscular structures has been activated, and providing second feedback when it is determined that a second submuscular structure of the plurality of submuscular structures has been activated.
[0022] In at least one aspect, the first feedback and the second feedback have different characteristics to enable a user to distinguish between the first feedback and the second feedback.
[0023] In at least one aspect, the first feedback includes auditory feedback having a first pitch, and wherein the second feedback includes auditory feedback having a second pitch different from the first pitch.
[0024] In at least one aspect, the feedback includes feedback selected from the group consisting of auditory feedback, visual feedback, tactile feedback, and feedback provided by electrical stimulation.
[0025] In at least one aspect, at least one computer processor is further programmed to derive a control signal based on the plurality of neuromuscular signals; and provide the control signal to a device having an operation that a user is attempting to control, wherein providing feedback to the user based on the plurality of neuromuscular signals comprises changing the behavior of the device based on the control signal.
[0026] In at least one aspect, the device includes a display.
[0027] In at least one aspect, deriving the control signal includes providing the plurality of neuromuscular signals or information derived from the plurality of neuromuscular signals as input to an inference model, and deriving the control signal based on an output of the inference model.
[0028] In at least one aspect, at least one computer processor is further programmed to train an inference model to map the pattern of activation determined from the plurality of neuromuscular signals to one or more sub-muscular activation patterns, and wherein storing the information mapping the pattern of activation identified in the plurality of neuromuscular signals to control signals comprises storing the trained inference model.
[0029] In at least one aspect, the at least one computer processor is further programmed to computationally map the one or more sub-muscle activation patterns to control signals.
[0030] In at least one aspect, storing information mapping a pattern of activation identified in the plurality of neuromuscular signals to a control signal includes storing information describing the pattern of activation determined from the plurality of neuromuscular signals.
[0031] In at least one aspect, the at least one computer processor is further programmed to identify a plurality of submuscular structures activated by the user based on the plurality of neuromuscular signals.
[0032] In at least one aspect, identifying multiple submuscular structures activated by a user includes decomposing multiple neuromuscular signals into signal components representing activation of specific submuscular structures, and mapping information derived from the multiple neuromuscular signals to one or more control signals includes mapping the signal components representing activation of specific submuscular structures to one or more control signals.
[0033] In at least one aspect, at least one computer processor is also programmed to identify multiple submuscular structures activated by the user based on multiple neuromuscular signals; select a subset of submuscular structures for training based on activation characteristics associated with each of the multiple submuscular structures activated by the user; and wherein the feedback includes information about the pattern of activation of the subset of submuscular structures.
[0034] In at least one aspect, the characteristics of activation associated with the submuscular structure are selected from the group consisting of: the type of motor unit associated with the submuscular structure, the amplitude of motor unit action potentials associated with the submuscular structure, the similarity of the waveform used for activation of the submuscular structure to the waveform used for activation of other submuscular structures, and the activation rate and timing statistics associated with the activation of the submuscular structure.
[0035] In at least one aspect, the computerized system further comprises at least one storage device configured to store spatiotemporal information regarding at least one activated submuscular structure.
[0036] In at least one aspect, the computerized system further comprises a user interface configured to instruct a user to activate a specific submuscular structure, wherein the plurality of neuromuscular sensors are configured to record a plurality of neuromuscular signals in response to the user attempting to activate the specific submuscular structure indicated by the user interface.
[0037] In at least one aspect, instructing the user to activate a particular submuscular structure includes instructing the user to activate a plurality of submuscular structures in sequence, and wherein the at least one computer processor is further programmed to perform calibration based at least in part on the plurality of neuromuscular signals.
[0038] Some embodiments are directed to a computer-implemented method of controlling a device. The method includes receiving a plurality of neuromuscular signals recorded by a plurality of neuromuscular sensors disposed on one or more wearable devices worn by a user; deriving information for a first submuscular control channel of a plurality of submuscular control channels from the plurality of neuromuscular signals, wherein each of the plurality of submuscular control channels is configured to process information associated with activation of one or more submuscular structures; generating a control signal based on the derived information for the first submuscular control channel; and providing the control signal to a control interface to control operation of the device.
[0039] Some embodiments are directed to a computer-implemented method of training a user to activate a submuscular structure. The method includes: recording a plurality of neuromuscular signals from a plurality of neuromuscular sensors disposed on one or more wearable devices worn by the user; providing feedback to the user based on the plurality of neuromuscular signals, wherein the feedback includes information about a pattern of activation identified in the plurality of neuromuscular signals; adjusting the feedback provided to the user based on the recorded plurality of neuromuscular signals; and storing information mapping the pattern of activation identified in the plurality of neuromuscular signals to a control signal.
[0040] It should be appreciated that all combinations of the preceding concepts and the additional concepts discussed in more detail below (assuming such concepts are not mutually inconsistent) are contemplated as part of the inventive subject matter disclosed herein. In particular, all combinations of the claimed subject matter appearing at the end of this disclosure are contemplated as part of the inventive subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Various non-limiting embodiments of the technology will be described with reference to the following drawings. It should be appreciated that the drawings are not necessarily drawn to scale.
[0043] Figure 1 is a flow chart of a biological process for performing a motor task according to some embodiments of the technology described herein;
[0044] Figure 2 is a schematic diagram of a computer-based system for generating submuscular control information according to some embodiments of the technology described herein;
[0045] Figure 3 is a flow chart of a process for training a user to activate submuscular structures according to some embodiments of the technology described herein;
[0046] Figure 4 is a flow chart of a process for selecting a set of submuscular structures for training according to some embodiments of the technology described herein;
[0047] Figure 5 is a flow chart of a process for calibrating a control system according to some embodiments of the techniques described herein;
[0048] Figure 6 is a flow chart of a process for providing a control signal using a calibrated control system based on muscle activation according to some embodiments of the technology described herein;
[0049] Figure 7 shows a wristband / armband having EMG sensors circumferentially arranged thereon according to some embodiments of the technology described herein;
[0050] Figure 8 A user is shown wearing a keyboard while typing on it according to some embodiments of the technology described herein. Figure 7 Wristband / armband,
[0051] Fig.9A shows a wearable system according to some embodiments of the technology described herein, the wearable system having 16 EMG sensors arranged circumferentially around an elastic band configured to be worn around a user's lower arm or wrist;
[0052] Fig. 9B yes Fig.9A a cross-sectional view of one of the 16 EMG sensors shown in FIG. 1 ; and
[0053] Fig. 10A and Fig. 10B Components of a computer-based system upon which some embodiments are implemented are schematically shown. Fig. 10A The wearable portion of the computer-based system is shown, and Fig. 10B A dongle portion connected to a computer is shown, wherein the dongle portion is configured to communicate with the wearable portion.
[0054] Detailed Description
[0055] Figure 1 A flow chart of a biological process 100 for initiating a motor task by the coordinated movement of one or more muscles is shown. In action 102, an action potential is generated in one or more efferent spinal motor neurons. Motor neurons carry neuronal signals to leave the central nervous system and toward the skeletal muscles in the periphery. For each motor neuron where an action potential is generated, the action potential travels along the axon of the motor neuron from the body in the spinal cord where the action potential is generated to the axon terminal of the motor neuron, which dominates the muscle fibers included in the skeletal muscle. Motor neurons and the muscle fibers they dominate are referred to as motor units in this article. The muscle fibers in the motor unit are activated together in response to the action potential generated in the corresponding motor neurons of the motor unit. Each muscle typically includes muscle fibers from hundreds of motor units, and the muscle fibers in many motor units contract simultaneously, resulting in muscle contraction that is manifested as perceptible muscle movement.
[0056] The chemical synapse formed at the interface between the axon terminal of the spinal motor neuron and the muscle fiber is called the neuromuscular junction. When the action potential transmitted along the axon of the motor neuron reaches the neuromuscular junction, the process 100 proceeds to action 104, wherein the action potential is generated in the muscle fiber as a result of the chemical activity at the neuromuscular junction. In particular, the acetylcholine released by the motor neuron diffuses across the neuromuscular junction and binds to the receptors on the surface of the muscle fiber, triggering the depolarization of the muscle fiber. Although the neuromuscular signal sensed on the body surface generated by the depolarization of each muscle fiber is very small (e.g., less than 100 μV), the collective action of multiple muscle fibers simultaneously conducting leads to a detectable voltage potential that can be recorded by a neuromuscular sensor (e.g., EMG sensor) located on the surface of the body. As described above, the collective conduction of muscle fibers from many motor units leads to muscle contraction and perceptible movement. Therefore, when the user performs a move or posture, the corresponding recorded neuromuscular signal includes the contribution from multiple activated motor units.
[0057] After the action potential is generated in the muscle fiber, process 100 proceeds to act 106, in which the propagation of the action potential in the muscle fiber causes a series of chemically mediated processes in the muscle fiber. For example, depolarization of the muscle fiber causes calcium ions to flow into the muscle fiber. The calcium ions in the muscle fiber bind to the troponin complex, causing the troponin complex to separate from the myosin binding site on the actin microfilament in the muscle fiber, thereby exposing the myosin binding site.
[0058] Following these chemically mediated processes, process 100 proceeds to act 108, in which the muscle fibers contract. Contraction of the muscle fibers is achieved due to the binding of the exposed myosin heads to the actin filaments in the muscle fibers to create cross-bridge structures. Process 100 then proceeds to act 110, in which the collective contraction of the muscle fibers in one or more muscles results in the performance of a motor task.
[0059] During the performance of a motor task, as muscle tension increases, the firing rate of active neurons increases, and additional neurons may become active, a process known as motor unit recruitment. The pattern in which neurons become active and increase their firing rate is stereotyped, so that the expected motor unit recruitment pattern defines an activity manifold associated with a standard or normal movement. Some embodiments are intended to teach users to activate a single motor unit or a group of motor units that are "out of the manifold" because the pattern of motor unit activation is different from the expected or typical motor unit recruitment pattern. This activation out of the manifold is referred to herein as "submuscular activation" or "activation of submuscular structures," where the submuscular structure refers to a single motor unit or a group of motor units associated with the activation out of the manifold. Examples of recruitment patterns for motor units out of the manifold include, but are not limited to, selectively activating high-threshold motor units without activating low-threshold motor units that would normally be activated earlier in the recruitment order, and regulating the firing rate of motor units over a considerable range without regulating the activity of other neurons that would normally be co-regulated in a typical motor recruitment pattern. According to some embodiments of the technology described herein, submuscular activation is used to generate control information, as described in more detail below.
[0060] When a user performs a motor task (such as moving their arms), a group of muscles necessary to perform the motor task is activated. When a user performs a motor task while wearing a wearable device including a neuromuscular sensor, the neuromuscular signals recorded by the sensors on the body surface correspond to the superimposed activity of all motor units in the muscles in the group activated during the performance of the motor task. The neuromuscular signals can be analyzed and mapped to control signals to control the device based on the type of movement or gesture performed by the user. For example, if a user performs a thumbs-up gesture with their hands, a corresponding control signal for selecting an object in a user interface can be generated. The mapping between the sensor signal and the control signal can be implemented, for example, using an inference model trained to associate a specific sensor signal input with a control signal output. In some implementations, the inference model can include one or more statistical models, one or more machine learning models, and / or a combination of one or more statistical models and / or one or more machine learning models. Further discussion of the implementation of the inference model is provided below. In some embodiments, the output of the trained inference model can be musculoskeletal position information, which describes the position and / or force of elements in a musculoskeletal model implemented by a computer, for example. Since the neuromuscular signals are continuously recorded, the musculoskeletal model can be updated using the musculoskeletal position information predicted to be output from the inference model. A control signal can then be generated based on the updated musculoskeletal position information. In other embodiments, the output of the trained inference model can be the control information itself, thereby not using a separate musculoskeletal model.
[0061] As described above, each muscle of the human body typically includes muscle fibers from hundreds of motor units. During normal motor control, in a system that generates control signals based on the activation of one or more muscles (e.g., when a user activates a muscle or uses a group of muscles to perform a movement), the joint activity of the motor unit in each muscle is projected onto a single dimension corresponding to the activation or tension of the muscle. By projecting the multidimensional sensor signal onto a single dimension, information about the activation of a single submuscular structure (e.g., one or more motor units) is lost because only the collective activation of all motor units in each muscle used to perform a movement, posture, or gesture is considered when determining the corresponding control signal to be generated. Some embodiments of the technology described herein are intended to use neuromuscular sensors to identify the activation of submuscular structures, and to generate control signals based at least in part on the identified activated submuscular structures. The inventors have recognized and appreciated that by identifying the activation of submuscular structures, a control system comprising multiple submuscular control "channels" can be designed, each submuscular control channel corresponding to a pattern of activation identified in one or more motor units. Thus, in some embodiments, information about the activation of underlying muscle structures that is typically lost in traditional neuromuscular sensor-based control systems that are projected down to a single dimension is utilized to increase the amount of control information available for controlling a device. Additionally, by training the user to activate individual motor units or groups of motor units, some embodiments are configured to generate control information based on recorded neuromuscular signals without requiring perceptible movement of a muscle or muscle group.
[0062] Throughout this disclosure, EMG sensors are used as examples of types of neuromuscular sensors configured to detect neuromuscular activity. However, it should be understood that, according to some embodiments, other types of neuromuscular sensors (including but not limited to myograph (MMG) sensors, electrical impedance tomography (EIT) sensors, and sonomyogram (SMG) sensors) may be used in addition or alternatively in conjunction with EMG sensors to detect neuromuscular activity. According to the techniques described herein, neuromuscular signals recorded by neuromuscular sensors may be used to identify activation of underlying muscle structures.
[0063] Figure 2A system 200 according to some embodiments is shown. The system includes a plurality of sensors 210 configured to record signals generated by activation of motor units using a human body part. The sensor 210 may include a plurality of neuromuscular sensors configured to record signals generated by neuromuscular activity in human skeletal muscle. The term "neuromuscular activity" as used herein refers to neural activation, muscle activation, muscle contraction, or any combination of neural activation, muscle activation, and muscle contraction of spinal motor neurons that innervate muscles. In some embodiments, a plurality of neuromuscular sensors may be used to sense submuscular activity associated with submuscular structures. In some embodiments, spatial information (e.g., position and / or orientation information) and force information describing submuscular activation may be predicted based on the sensed neuromuscular signals when a user activates the submuscular structure over time.
[0064] Sensor 210 may include one or more inertial measurement units (IMUs) that measure a combination of physical aspects of motion using, for example, accelerometers, gyroscopes, magnetometers, or any combination of one or more accelerometers, gyroscopes, and magnetometers. In some embodiments, the IMU may be used to sense information about the movement of a body part to which the IMU is attached, and information derived from the sensed data (e.g., position and / or orientation information) may be tracked as the user moves over time. For example, one or more IMUs may be used to track the movement of a user's body part (e.g., arm, leg) proximal to the user's torso relative to the sensor as the user moves over time.
[0065] In an embodiment including at least one IMU and multiple neuromuscular sensors, the IMU and neuromuscular sensors may be arranged to detect movement of different parts of the human body. For example, the IMU may be arranged to detect movement of one or more body segments (e.g., upper arms) at the proximal end of the trunk, and the neuromuscular sensors may be arranged to detect the activity of motor units within one or more body segments (e.g., forearms or wrists) at the distal end of the trunk. However, it should be understood that the sensors may be arranged in any suitable manner, and embodiments of the technology described herein are not limited to specific sensor arrangements. For example, in some embodiments, at least one IMU and multiple neuromuscular sensors may be co-located on a body segment to track the activity of motor units and / or movement of body segments using different types of measurements. In one implementation described in more detail below, an IMU sensor and multiple EMG sensors are arranged on a wearable device that is configured to be worn around the user's lower arm or wrist. In this arrangement, the IMU sensors may be configured to track movement information associated with one or more arm segments (e.g., position and / or orientation over time) to determine, for example, whether a user has raised or lowered their arm, while the EMG sensors may be configured to determine submuscular information associated with activation of submuscular structures in muscles of the wrist or hand.
[0066] Each sensor 210 includes one or more sensing components configured to sense information about the user. In the case of an IMU, the sensing components may include one or more accelerometers, gyroscopes, magnetometers, or any combination thereof to measure characteristics of body movement, examples of which include, but are not limited to, acceleration, angular velocity, and magnetic fields sensed around the body. In the case of a neuromuscular sensor, the sensing components may include, but are not limited to, electrodes configured to detect electrical potentials on the surface of the body (e.g., for an EMG sensor), vibration sensors configured to measure vibrations of the skin surface (e.g., for an MMG sensor), and acoustic sensing components configured to measure ultrasonic signals generated by muscle activity (e.g., for an SMG sensor).
[0067] In some embodiments, at least some of the multiple sensors are arranged as part of a wearable device that is configured to be worn on or around a user's body part. For example, in one non-limiting example, an IMU sensor and multiple neuromuscular sensors are arranged circumferentially around an adjustable and / or elastic band (such as a wristband or armband configured to be worn around a user's wrist or arm). Alternatively, at least some of the sensors may be arranged on a wearable patch that is configured to be attached to a portion of the user's body. In some embodiments, multiple wearable devices may be used to generate control information based on activation from submuscular structures and / or movement involving multiple parts of the body, each wearable device including one or more IMUs and / or neuromuscular sensors thereon.
[0068] In one implementation, 16 EMG sensors are arranged circumferentially around an elastic band configured to be worn around a user's lower arm. Figure 7 EMG sensors 504 are shown arranged circumferentially around the elastic band 502. It should be understood that any suitable number of neuromuscular sensors may be used, and the number and arrangement of the neuromuscular sensors used may depend on the specific application for which the wearable device is used. For example, a wearable armband or wristband may be used to generate control information to control a robot, control a vehicle, scroll through text, control an avatar, or any other suitable control task. For example, Figure 8 As shown, user 506 can wear elastic band 502 on hand 508. In this way, EMG sensor 504 can be configured to record EMG signals when the user uses finger 540 to control keyboard 530. In some embodiments, elastic band 502 can also include one or more IMUs (not shown) configured to record movement information as discussed above.
[0069] In some embodiments, multiple wearable devices may be used to generate control information based on activation associated with submuscular structures and / or movement involving multiple parts of the body, each wearable device including one or more IMUs and / or neuromuscular sensors thereon.
[0070] In some embodiments, sensor 210 includes only a plurality of neuromuscular sensors (e.g., EMG sensors). In other embodiments, sensor 210 includes a plurality of neuromuscular sensors and at least one "auxiliary" sensor configured to continuously record a plurality of auxiliary signals. Examples of auxiliary sensors include, but are not limited to, IMU sensors, imaging devices (e.g., cameras), radiation-based sensors used with radiation-generating devices (e.g., laser scanning devices), or other types of sensors (such as heart rate monitors).
[0071] In some embodiments, hardware signal processing circuitry may be used to optionally process (e.g., perform amplification, filtering, and / or rectification) the output of one or more sensing components. In other embodiments, at least a portion of the signal processing of the output of the sensing component may be performed in software. Thus, signal processing of the signals recorded by the sensor may be performed in hardware, software, or by any suitable combination of hardware and software, as aspects of the technology described herein are not limited in this respect.
[0072] In some embodiments, as described in more detail below, the recorded sensor data may optionally be processed to compute additional derived measurements, which are then provided as input to the inference model. For example, the recorded signals from an IMU sensor may be processed to derive an orientation signal that specifies the orientation of a body segment over time. The sensor may implement the signal processing using components that integrate the sensing components, or at least a portion of the signal processing may be performed by one or more components that communicate with but are not directly integrated with the sensing components of the sensor 210.
[0073] The system 200 also includes one or more computer processors 212 programmed to communicate with the sensors 210. For example, signals recorded by the one or more sensors may be provided to the processor 212, which may be programmed to perform one or more machine learning techniques to process the signals output by the sensors 210 to train one or more inference models 214, and the trained (or retrained) inference models 214 may be stored for subsequent use in generating control signals (as described in more detail below).
[0074] In some embodiments, the reasoning model 214 can be a neural network, for example, a recurrent neural network. In some embodiments, the recurrent neural network can be a long short-term memory (LSTM) neural network. However, it should be understood that the recurrent neural network is not limited to the LSTM neural network, and can have any other suitable architecture. For example, in some embodiments, the recurrent neural network can be a fully recurrent neural network, a gated recurrent neural network, a recursive neural network, a Hopfield neural network, an associative memory neural network, an Elman neural network, a Jordan neural network, an echo state neural network, a second order recurrent neural network, and / or any other suitable type of recurrent neural network. In other embodiments, a neural network that is not a recurrent neural network can be used. For example, a deep neural network, a convolutional neural network, and / or a feedforward neural network can be used.
[0075] In some embodiments, the output of the inference model provides a discrete output. For example, when the desired output is to know whether the user is currently performing a specific pattern of activation (including individual neural spike events), a discrete output (e.g., a classification label) can be used. For example, a model can be trained to estimate whether a user is activating a specific motor unit, activating a specific motor unit with a specific timing, activating a specific motor unit with a specific discharge pattern, or activating a specific combination of motor units. On a shorter time scale, discrete classification is used in some embodiments to estimate whether a specific motor unit has discharged an action potential within a given amount of time. In this scenario, these estimates can then be accumulated to obtain an estimated discharge rate for the motor unit.
[0076] In an embodiment where the inference model is implemented as a neural network that is configured to output discrete signals, the neural network may include a softmax transfer function (softmax) layer such that the outputs sum to 1 and can be interpreted as probabilities. The output of the softmax transfer function layer may be a set of values corresponding to a corresponding set of control signals, where each value indicates a probability that a user wants to perform a particular control action. As a non-limiting example, the output of the softmax transfer function layer may be a set of three probabilities (e.g., 0.92, 0.05, and 0.03) that indicate a corresponding probability that a detected activity pattern is one of three known patterns.
[0077] It should be understood that when the inference model is a neural network configured to output discrete signals, the neural network need not produce outputs that add up to 1. For example, instead of a softmax layer, the output layer of the neural network can be a sigmoid layer (which does not have the restriction that probabilities add up to 1). In such an embodiment, the neural network can be trained with a sigmoid cross-entropy cost. This implementation can be advantageous in this case when multiple different control actions may occur within a threshold amount of time and it is not important to distinguish the order in which these actions occur (for example, a user may activate two neural activity patterns within a threshold amount of time). In some embodiments, any other suitable non-probabilistic multi-class classifier may be used, as aspects of the technology described herein are not limited to this aspect.
[0078] In some embodiments, the output of the inference model can be a continuous signal rather than a discrete signal. For example, the model can output an estimate of the firing rate of each motor unit, or the model can output a time series of electrical signals corresponding to each motor unit or underlying muscle structure.
[0079] It should be understood that aspects of the technology described herein are not limited to the use of neural networks, as other types of inference models may be employed in some embodiments. For example, in some embodiments, the inference model may include a hidden Markov model (HMM), a switching HMM (where switching allows jumping between different dynamic systems), a dynamic Bayesian network, and / or any other suitable graphical model with a time component. Any such inference model may be trained using recorded sensor signals.
[0080] As another example, in some embodiments, the inference model is a classifier that uses features derived from recorded sensor signals as input. In such embodiments, the classifier can be trained using features extracted from sensor data. The classifier can be a support vector machine, a Gaussian mixture model, a regression-based classifier, a decision tree classifier, a Bayesian classifier, and / or any other suitable classifier, because the various aspects of the technology described herein are not limited to this aspect. The input features to be provided to the classifier can be derived from the sensor data in any suitable manner. For example, sensor data can be analyzed as time series data using wavelet analysis techniques (e.g., continuous wavelet transform, discrete time wavelet transform, etc.), Fourier analysis techniques (e.g., short-time Fourier transform, Fourier transform, etc.), and / or any other suitable type of time-frequency analysis techniques. As a non-limiting example, sensor data can be transformed using wavelet transform, and the obtained wavelet coefficients can be provided as input to the classifier.
[0081] In some embodiments, the values of the parameters of the inference model can be estimated based on the training data. For example, when the inference model is a neural network, the parameters (e.g., weights) of the neural network can be estimated based on the training data. In some embodiments, gradient descent, stochastic gradient descent, and / or any other suitable iterative optimization technique can be used to estimate the parameters of the inference model. In an embodiment where the inference model is a recurrent neural network (e.g., LSTM), the inference model can be trained using stochastic gradient descent and time back propagation. The training can use a cross-entropy loss function and / or any other suitable loss function, as the aspects of the technology described herein are not limited to this aspect.
[0082] Optionally, the system 200 also includes one or more controllers 216. For example, the controller 216 can be a display controller configured to display a visual representation (e.g., a visual representation of a hand) on a display. As discussed in more detail below, the one or more computer processors can implement one or more trained inference models that receive sensor signals as input and provide information as output that is used to generate control signals.
[0083] In some embodiments, a computer application configured to simulate a virtual reality environment may be instructed to display a visual character, such as an avatar (e.g., an avatar via controller 216). Positioning, movement, and / or forces exerted by portions of the visual character within the virtual reality environment may be displayed based on the output of the trained reasoning model. The visual representation may be dynamically updated as continuous signals are recorded by sensors 210 and processed by trained reasoning model 104 to provide a computer-generated representation of the character's movement that is updated in real time.
[0084] As described above, some embodiments are intended to use, at least in part, an inference model to map sensor signals to control signals. The inference model may receive as input an IMU signal, a neuromuscular signal (e.g., an EMG, MMG, and SMG signal), an external device signal (e.g., a camera or laser scanning signal), or a combination of an IMU signal, a neuromuscular signal, and an external device signal detected when a user performs one or more muscle activations. The inference model may be used to predict control information without the user having to make perceptible movements.
[0085] In some embodiments, the system 200 is trained to generate control information when a user performs a submuscular activation. In some embodiments, the system 200 is trained by recording signals from the neuromuscular sensor 210 and ground truth data representing corresponding control signals for specific submuscular activations. For example, a user may be instructed to activate a submuscular structure to perform a specific control action. The control signal may be generated based on the precise timing of the sensed neural activation (time encoding of detected spikes), or the control signal may be generated based on the rate of motor unit action potentials (rate encoding).
[0086] The inventors have recognized that one of the challenges of using submuscular activation as a basis for generating control signals in a control system is training a user how to selectively activate a single motor unit or a group of motor units. For example, an instruction to contract a single muscle or muscle group to perform a movement (such as making a fist) may be easy for a user to understand. However, instructing a user to selectively activate individual submuscular structures may not be so simple, and the user may need to be trained to reliably perform such activation. Therefore, some embodiments are directed to techniques for providing feedback to a user during recording of neuromuscular signals to facilitate the process of training a user to activate submuscular structures (e.g., one or more motor units) when needed.
[0087] Optionally, the system 200 also includes a user interface 218. Feedback based on the signals recorded by the sensor 210 and processed by the processor 212 can be provided through the user interface 218 to facilitate the user's understanding of how the system interprets the user's intentional sub-muscle activation. The user interface 218 can be implemented in any suitable manner, including but not limited to an audio interface, a video interface, a tactile interface, and an electrical stimulation interface, or any combination of the foregoing.
[0088] Figure 3 A process 300 for training a user to activate a submuscular structure using neuromuscular signals and feedback generated by processing the recorded signals is shown according to some embodiments. In action 310, multiple neuromuscular signals are recorded by multiple neuromuscular sensors worn by the user when the user activates one or more submuscular structures. Process 300 then proceeds to action 320, in which feedback generated based on the recorded neuromuscular signals is provided to the user based on the recorded neuromuscular signals. The feedback can be provided through a user interface (e.g., user interface 218 in system 200). The feedback generated in action 320 provides the user with information about the recorded neuromuscular signals, which in turn enables the user to learn how to selectively activate the submuscular structure. Although feedback is described in conjunction with the process 300 for training a user to activate a submuscular structure, in some embodiments, feedback is provided to the user even after the system has been trained. Providing feedback when the user is using the system after training can help the user understand how the trained system interprets the user's intentional submuscular activation, and can remind the user whether further training of the system is needed.
[0089] In some embodiments, the feedback provided to the user in action 320 is generated using the original (e.g., unprocessed) neuromuscular signal recorded by the neuromuscular sensor. For example, the original neuromuscular signal can be converted into an audio signal played through an audio interface (e.g., a speaker). Alternatively, the original neuromuscular signal can be displayed on a visual interface (such as a display) to provide feedback. In other embodiments, the original neuromuscular signal can be analyzed to identify the activation of a specific submuscular structure. The inventor has recognized and realized that the activation of the submuscular structure is manifested in the neuromuscular signal in a unique manner (e.g., timing, waveform), so that the signal generated by one submuscular structure can be separated from the signal generated by another submuscular structure. The activation of a specific submuscular structure can be identified according to the original neuromuscular signal in any suitable manner. For example, the original neuromuscular signal can be decomposed into signal components corresponding to the activation generated by each motor unit (e.g., each spike event in the motor unit) or a motor unit group (e.g., using independent component analysis, convolution blind source separation, a spike classification protocol including event detection with subsequent clustering or classification, or another suitable technology).
[0090] The features or "signatures" of the submuscular activation identified in the recorded neuromuscular signals can be used to generate feedback provided to the user when the user activates the submuscular structure. For example, audio feedback can be provided to the user, and the audio feedback uses audio with different characteristics to encode the activation of different submuscular structures. In a simplified example, the system can analyze the original neuromuscular signal and identify the activation of two motor units. Audio feedback can be generated, and the audio feedback includes a first audio tone with a first pitch and a second audio tone with a second pitch, wherein the first audio tone corresponds to the activation of one of the two motor units, and the second audio tone corresponds to the activation of the other of the two motor units. In some embodiments, the timing of the presentation of the tone in the audio feedback can correspond to the timing of the activation (e.g., neuromuscular spike activity) of the corresponding motor unit or other submuscular structure. In some embodiments, the amplitude of the tone can correspond to the rate or intensity of activating the submuscular structure.
[0091] The feedback received by the user provides the user with information about whether and when the system is able to detect a pattern of neuromuscular activity associated with a particular submuscular structure, and allows the user to adjust their neuromuscular activity to learn how to activate the submuscular structure, such as a single motor unit. For example, if the user receives audio feedback including a tone with multiple pitches, the user is made aware of a pattern of activity of multiple submuscular structures that the system has identified as being activated. Based on this feedback, the user can consciously modify their neuromuscular activity in an attempt to invoke a particular activation pattern associated with the submuscular structure. As the user modifies their motor activity, the feedback provided to the user also changes based on the recorded neuromuscular signals so that the user can understand in real time how the system interprets the modifications in their activity. The user can continuously modify their activity based on the feedback provided to learn how to activate a particular pattern of submuscular activity.
[0092] When an audio signal is used as feedback, audio features other than the pitch of a tone can be used to represent differences between activated submuscular structures identified in the recorded neuromuscular signal. For example, the loudness, duration, timbre, or other perceived audio features of the audio feedback can be adjusted to represent the identity of a specific submuscular structure in the neuromuscular signal. Additionally, audio signals other than tones can also be used. For example, different activated submuscular structures can be represented by different instruments in the audio feedback.
[0093] According to some embodiments, alternatively, feedback other than audio feedback may be used. Non-limiting examples of feedback that may be provided to a user to facilitate training the user to activate submuscular structures include visual feedback, tactile / haptic feedback, and feedback provided by electrical stimulation. For any selected form of feedback, the perceptual characteristics of the components of the feedback provided may be updated in real time based on submuscular activations identified in the recorded neuromuscular signals to enable the user to learn how to modify their neuromuscular activity using specific activation patterns to activate one or more submuscular structures. For example, a user may be trained to alternately discharge two motor units, create a rhythmic discharge pattern for a single motor unit, or adjust the rate of one or more motor units in a time-dependent manner, and feedback may be provided to facilitate training.
[0094] As described above, feedback provided to the user can be generated based on raw (e.g., unprocessed) sensor data. In other embodiments, feedback can be generated based on information derived from recorded sensor data. For example, recorded sensor data can be filtered or otherwise processed before being used to generate feedback provided to the user.
[0095] As described above, some embodiments use a reasoning model trained to output information, which is used to generate a control signal based on sensor data provided to the model as input. In some embodiments, feedback provided to the user is generated based on the output of the trained reasoning model. For example, a control signal generated based on the output of the trained reasoning model can be provided to a display controller (or other suitable controller), which updates the display with information notifying the user about the specific submuscular structure activated. When the user modifies their neuromuscular activation based on the feedback provided on the display, the control signal generated by the system is also updated, thereby generating an updated visual representation on the display. In this way, the feedback provided to the user can be presented in the form of a game that imitates and encourages the user to complete a specific task, wherein the completion of the task is associated with the successful activation of the submuscular activity of a specific mode. This feedback may be easier to understand and useful for some users, enabling these users to learn how to submuscular structure in a specific way.
[0096] Recognizing that different users may learn how to activate submuscular structures in different ways, some embodiments provide feedback to users in a variety of different ways. Users may be able to select the type of feedback that will best facilitate their training. In some embodiments, the type of feedback provided to a user may be determined and / or recommended by the system based, at least in part, on a measure of the effectiveness of the feedback in helping the user learn. For example, a measure of the time required for a user to successfully complete a task associated with activating a submuscular structure may be used to determine which type of feedback the system recommends for a particular user.
[0097] As shown in process 300, feedback can be provided continuously when the user learns to call a specific activation mode, and neuromuscular signals are continuously recorded when the user modifies their behavior in response to the feedback. Training can continue until it is determined that the user has successfully learned how to activate a specific submuscular structure as needed (e.g., in a specific mode). This determination can be made by the user, or can be made at least in part using an automated process (e.g., implemented using processor 212). For example, the signal recorded by the sensor can be compared with a template describing a known or desired activation mode, and when it matches the template enough, it can be determined that the training of the activation mode under a specific muscle is terminated. Even if other submuscular structures or activity patterns are observed in the recorded sensor data, it can be determined that the user has successfully trained the submuscular pattern. For example, if the system is programmed to determine whether the user has activated motor unit A, it can be determined that the user has activated motor unit A when the system determines that motor units A, B, and C (or other unidentified sources) have been activated, rather than when the system determines that motor units C and D (but not A) have been activated. In some embodiments, the system is programmed to determine that the user has activated motion unit A when only motion unit A has been activated.
[0098] After it has been determined to stop training, process 300 proceeds to act 330, in which information about the activation of the muscle under which the user is training is stored. In some embodiments, a reasoning model trained to map recorded sensor data to one or more control signals associated with structures under the muscle can be stored and / or updated. For example, when the reasoning model is implemented as a neural network, weights associated with connections in the neural network can be updated to reflect the mapping between the recorded sensor data and the control signals output or derived from the output of the reasoning model.
[0099] The inventors have realised that storing information about submuscular activation by the user enables the user to activate the same one or more submuscular structures between sessions. Figure 2 A second session of the system of multiple neuromuscular sensors 210 and other components shown in can occur minutes, hours, days, weeks, months, or even years after the first session. The stored representation of the user's muscle activation can be accessed and used as a control signal.
[0100] In general, an inference model may be implemented that calibrates neuromuscular sensor data recorded from a user so as to reliably derive information about one or more underlying muscle structures activated by the user. Calibration may be necessary for several reasons. For example, calibration may be necessary because when the user wears the neuromuscular array in a second session, one or more of the neuromuscular sensors may be in a different location on the user's body than in the first session. When the neuromuscular sensors are arranged such as Figure 7 and Fig.9A When a radial array on a wearable device is shown, the location of the neuromuscular sensors on the body can be rotated between sessions, and the inference model can be used to calibrate the neuromuscular sensor data so as to reliably identify the same submuscular structure (or the same multiple submuscular structures) that were identified in the first session. In another example of why calibration may be necessary, a radial neuromuscular array (e.g., Figure 7 and Fig.9A The system shown in the figure) can be located at a location farther or closer on the user's arm, and the inference model can be used to calibrate the neuromuscular sensor data to reliably identify the same submuscular structure (or multiple submuscular structures) that were identified in the first session. In yet another example of why calibration may be necessary, a radial neuromuscular array (e.g., Figure 7 and Fig.9A The system shown) can be placed on the user's arm in an opposite orientation, i.e., a side of the neuromuscular array that faces proximally in a first session can face distally in a second session, and an inference model can be used to identify whether the neuromuscular array is facing distally or proximally; and the anatomical locations of the neuromuscular sensors can be reordered in a processing step prior to identifying one or more previously identified submuscular structures.
[0101] The inventors have recognized that systems and methods for storing information about a user's submuscular activation are advantageous for identifying changes in submuscular activation over time in a user, and the information about a user's submuscular activation enables the user to activate the same submuscular structure or the same multiple submuscular structures between sessions. Tracking changes in submuscular activation is advantageous for monitoring changes in the user's motor nervous system. Changes in submuscular activation may change for any of several reasons, including, but not limited to: muscle fatigue, training that strengthens or otherwise affects patterns of submuscular activation (e.g., motor learning or strength training), diet, time of day, the user's sleep amount and quality, the presence of compounds that affect the motor nervous system (e.g., pharmaceutical agents, drugs of abuse, caffeine, alcohol, and nicotine), peripheral neuropathy, neurodegeneration, peripheral nerve damage, brain damage, or other diseases that affect the motor system.
[0102] The inventors have recognized that while some users may be able to selectively activate the submuscular structure while inhibiting other neural activity, other users may have difficulty learning to selectively activate the submuscular structure. In some embodiments, the control signal output from the system is mapped to a subset of motor units activated during a motor task (e.g., moving the hand up and down), wherein the subset of motor units is associated with the submuscular structure. During training, initially the user may move their hand up and down, thereby activating the motor units mapped to the control signal in the subset and other motor units. Over time, the user may learn that as long as the user's movement still activates the motor units in the subset, generating smaller and smaller movements will still result in the output of the control signal from the system. Ultimately, through additional training, as long as the motor units corresponding to the submuscular structure in the subset are activated, the user may be able to generate a control signal without making a perceptible movement. According to some embodiments, training the user to make small movements to activate the submuscular structure enables the creation of a control system in which user fatigue is reduced and in which the user can control the device discretely (e.g., when the user is in a place where making a larger movement to generate a control signal is not an appropriate behavior) or independently of a larger movement.
[0103] As briefly discussed above, activation of a particular submuscular structure can be observed in the recorded sensor data as a timing and / or signal waveform shape that characterizes activation from that structure. The spatiotemporal activation pattern that characterizes activation of a submuscular structure is also referred to herein as the "signature" of that structure. The ability of the system to effectively separate activations from different submuscular structures and thereby create separate control channels for activations associated with individual submuscular structures or multiple submuscular structures (e.g., combined codes) may depend on the extent to which the signatures of the submuscular structures differ. In addition, the set of submuscular structures whose signatures achieve optimal separation may vary from user to user. Prior to training a user how to activate a submuscular activation pattern, a set of target submuscular structures that the user can be trained to activate can be identified.
[0104] Figure 4A process 400 for identifying a set of submuscular structures to be used as training targets according to some embodiments is shown. In action 410, multiple neuromuscular signals are recorded from multiple sensors arranged on a wearable device while the user performs one or more movements or postures. Process 400 then proceeds to action 412, in which multiple activated submuscular structures (e.g., motor units) are identified based on the recorded neuromuscular signals. Multiple activated submuscular structures can be identified in any suitable manner. For example, the recorded neuromuscular signals can be decomposed into signal components that characterize the activation of multiple activated submuscular structures. In some embodiments, the signal components can represent individual spike events, and multiple activated submuscular structures can be identified by determining which of the individual spike events are associated with the activation of a particular submuscular structure. Alternatively, the recorded neuromuscular signals can be analyzed in any other suitable manner to identify multiple submuscular structures.
[0105] Process 400 then proceeds to act 414, in which a subset of submuscular structures is selected for use in training. The selection of submuscular structures included in the training set can be based at least in part on features of activation associated with each submuscular structure identified in act 412. As described above, in order to enable the system to distinguish activation of different submuscular structures, the inventors have recognized that it is advantageous to train users to activate submuscular structures that have different activation features or signatures as displayed in the recorded sensor data. Examples of features of activation that can be used to select submuscular structures for inclusion in the training set in some embodiments include, but are not limited to, the type of motor unit associated with the submuscular structure, the amplitude of motor unit action potentials associated with the submuscular structure, the similarity of waveforms for activation of the submuscular structure to waveforms for activation of other submuscular structures, and activation rates and timing statistics associated with activation of the submuscular structure.
[0106] In some embodiments, the number of submuscle structures included in the training set is determined based on the expected number of submuscle control channels of the control system. For example, in order to build a neuromuscular-based control system with eight submuscle control channels, the multiple activated submuscle structures identified in action 412 can be analyzed to determine eight activated submuscle structures that have reliable, separable activation features to be included in the training set. It should be understood that not all submuscle structures selected for inclusion in the training set need to be separated from other submuscle structures in the training set using the same activation features. For example, the first submuscle structure in the training set can be separated from other structures in the training set based on the shape of the signal waveform, and the second submuscle structure in the training set can be separated from other structures in the training set based on the amplitude of the motor unit action potential. In addition, it should be understood that the submuscle structures selected for the training set can be separated based on a combination of activation features, and the embodiments are not limited in this respect.
[0107] The selection of the submuscular structure in action 414 can be performed automatically by one or more processors of the system 100 using a criterion of maximizing or increasing the distance between the activation features or signatures of the activated submuscular structures. Alternatively, the selection of the submuscular structure can be at least partially driven by the user. For example, feedback generated based on the recorded neuromuscular signals can be provided to the user, and the user can be able to select the submuscular structures to be included in the training set. Allowing the user to participate in the process of selecting the submuscular structure allows the control system to implement a less robust source separation technique than would be required if the selection of the submuscular structure for the training set was done entirely automatically by the system.
[0108] After selecting the set of submuscular structures for the training set, one can use e.g. Figure 3 The training process 300 shown and described above is used to train the user to activate each submuscular structure in the set. Once the user and the system have been trained, at least one storage device associated with the system can store spatiotemporal information about the submuscular structures activated in the training set and / or information mapping neuromuscular sensor signals to submuscular activations (e.g., a user-specified or trained inference model) for use in controlling a device using the system.
[0109] The inventors have recognized that each time a user begins a control session wearing a wearable device that includes a neuromuscular sensor, it may be difficult for the system, at least initially, to recover activity from the same underlying muscle structures that the system was trained for. For example, the placement of the wearable device including the sensor on the body may be different each time the user uses the device. Therefore, before using a system including a trained inference model to control a device, the system may be calibrated. Calibration may be performed automatically by the system when the user begins using the device, or calibration may be at least partially driven by the user to relax the requirement that the system perform automatic calibration without user input. Additionally, it should be understood that calibration may be performed at any suitable time, including during a control session.
[0110] Figure 5 A process 500 for calibrating a system according to some embodiments is shown. In act 510, a sensor disposed on a wearable device worn by a user records a neuromuscular signal. The process 500 then proceeds to act 512, in which the recorded neuromuscular signal is combined with the above Figure 3 The system can be configured to automatically identify the activation of the submuscle structures in the recorded sensor data by, for example, performing source separation on the recorded neuromuscular signals.
[0111] In embodiments where one or more processors of the system are not configured to perform automatic calibration, or where automatic calibration fails, the user can provide input to facilitate the calibration process. For example, the user can be prompted to activate multiple submuscular structures in sequence (e.g., structure A, structure B, structure C). As the user activates each submuscular structure, feedback can be provided to the user to let the user know whether the system correctly interpreted the intended activation.
[0112] Process 500 then proceeds to act 514, in which the system is calibrated based on analysis of the recorded neuromuscular signals and the stored information. Calibration can be performed in any suitable manner to enable the system to recover activation of the underlying muscle structures in the training set from the recorded neuromuscular signals. In some embodiments, calibration can include converting some of the recorded neuromuscular signals or information derived from the recorded neuromuscular signals to a representation that more closely matches the stored information. In other embodiments, feedback can be provided to the user instructing the user to adjust the positioning of the wearable device on the body, and additional neuromuscular signals can be recorded until the system can reliably recover activation of the underlying muscle structures from the recorded sensor data.
[0113] After the system has been calibrated, the system can be used to derive information of a plurality of submuscular control channels from the recorded signal data, control signals can be generated based on the derived information, and the control signals are provided to a control interface to control the operation of a physical or virtual device. In this manner, a control system capable of controlling a device using submuscular activation is provided.
[0114] Figure 6 A process 600 for providing control signals to a device based on submuscular control information recorded from sensor data is shown in accordance with some embodiments. In act 610, a plurality of neuromuscular signals from neuromuscular sensors disposed near or on a surface of a user's body are recorded. One or more auxiliary sensors (e.g., IMU sensors) may also be used to record sensor data for providing control information, and the embodiments are not limited in this respect. Process 600 then proceeds to act 612, in which a source separation process is performed on the recorded neuromuscular signals to identify submuscular control channels associated with activations identified in the neuromuscular signals. Any suitable source separation technique (examples of which are discussed above) may be used to identify submuscular control channels.
[0115] Process 600 then proceeds to action 614, in which one or more control signals are generated based on the identified submuscular control channels and the activated patterns represented in the recorded neuromuscular signals. For example, the neuromuscular signals can be provided as input to a trained reasoning model, and the output of the trained reasoning model can be used to generate one or more control signals. In one implementation, the output of the trained reasoning model can be a set of one or more control signals. In another implementation, the control signals can be generated based on the activated patterns in the neuromuscular signals without using the trained reasoning model. Process 600 then proceeds to action 614, in which the control signals are provided to a control interface of the device to control the operation of the device. For example, the device can be a display, and the control signals can be provided to a display controller of the display. The control signals can include instructions to update the information displayed on the display. Alternatively, the device can be a computer or other computing device (e.g., a smart phone), and the control signals can be provided to a controller of the computing device to change the operation of the device. In yet another example, the control signals can be used to control a device (e.g., a musical instrument) to provide artistic expression. It should be understood that any device having a control interface can be controlled using a control system designed according to the techniques described herein.
[0116] Fig.9AA human-machine interface (also referred to herein as an EMG control interface) is shown as a wearable system having 16 neuromuscular sensors 910 (e.g., EMG sensors) arranged circumferentially around an elastic band 920 that is configured to be worn around a user's lower arm or wrist. As shown, the EMG sensors 910 are arranged circumferentially around the elastic band 920. It should be understood that any suitable number of neuromuscular sensors may be used. The number and arrangement of the neuromuscular sensors may depend on the specific application in which the wearable system is used. For example, a wearable armband or wristband may be used to generate control information to control an augmented reality system, a robot, control a vehicle, scroll through text, control an avatar, or for any other suitable control task. As shown, the sensors may be coupled together using flexible electronics incorporated into a wireless device, Fig. 9B Shown by Fig.9A A cross-sectional view of one of the sensors of the wearable system is shown.
[0117] In some embodiments, the output of one or more sensing components may optionally be processed using hardware signal processing circuitry (e.g., to perform amplification, filtering, and / or rectification). In other embodiments, at least some of the signal processing of the output of the sensing components may be performed in software. Thus, the signal processing of the signals sampled by the sensors may be performed in hardware, software, or by any suitable combination of hardware and software, as the aspects of the technology described herein are not limited in this respect. Fig. 10A and Fig. 10B A non-limiting example of a signal processing chain for processing recorded data from sensor 910 is discussed in more detail.
[0118] Fig. 10A and Fig. 10B 10 is a schematic diagram of the internal components of a wearable system having sixteen EMG sensors. As shown in the figure, the wearable system includes a wearable portion 1010 ( Fig. 10A ) and a dongle portion 1020 that communicates with the wearable portion 1010 (e.g., via Bluetooth or another suitable short-range wireless communication technology) Fig. 10B ).like Fig. 10A As shown, the wearable part 1010 includes a sensor 910, combined with Fig.9A and Fig. 9BAn example of this is described. The output of the sensor 910 is provided to an analog front end 1030, which is configured to perform analog processing (e.g., noise reduction, filtering, etc.) on the recorded signal. The processed analog signal is then provided to an analog-to-digital converter 1032, which converts the analog signal into a digital signal that can be processed by one or more computer processors. An example of a computer processor that can be used according to some embodiments is a microcontroller (MCU) 1034 shown in FIG. 10A. As shown, the MCU 1034 may also include inputs from other sensors (e.g., an IMU sensor 1040) and a power and battery module 1042. The output of the processing performed by the MCU can be provided to an antenna 1050 for transmission to the dongle portion 1020 shown in FIG. 10B.
[0119] The dongle portion 1020 includes an antenna 1052 configured to communicate with an antenna 1050 included as part of the wearable portion 1010. Communication between the antennas 1050 and 1052 may be performed using any suitable wireless technology and protocol, non-limiting examples of which include radio frequency signaling and Bluetooth. As shown, signals received by the antenna 1052 of the dongle portion 1020 may be provided to a host computer for further processing, display, and / or enabling control of a particular physical or virtual object.
[0120] Although the reference Fig.9A , Fig. 9B and Fig. 10A , Fig. 10B The examples provided are discussed in the context of an interface having an EMG sensor, but it should be understood that the techniques described herein for reducing electromagnetic interference may also be implemented in wearable interfaces having other types of sensors, including but not limited to myofasciography (MMG) sensors, sonomyography (SMG) sensors, and electrical impedance tomography (EIT) sensors.
[0121] The above embodiments may be implemented in any of a variety of ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code may be executed on any suitable processor or processor cluster, whether provided in a single computer or distributed among multiple computers. It should be appreciated that any component or component cluster that performs the above functions may be generally considered to be one or more controllers that control the functions discussed above. One or more controllers may be implemented in a variety of ways, such as using dedicated hardware or using one or more processors programmed with microcode or software to perform the above functions.
[0122] In this regard, it should be understood that one implementation of an embodiment of the present invention includes at least one non-transitory computer-readable storage medium (e.g., computer memory, portable memory, optical disk, etc.) encoded with a computer program (i.e., multiple instructions), which, when executed on a processor, performs the functions discussed above of an embodiment of the present invention. The computer-readable storage medium may be transportable so that the program stored thereon can be loaded onto any computer resource to implement the various aspects of the present invention discussed herein. In addition, it should be recognized that reference to a computer program that performs the functions discussed above when executed is not limited to an application program running on a host computer. Instead, the term "computer program" is used in a general sense herein to refer to any type of computer code (e.g., software or microcode) that can be used to program a processor to implement the various aspects of the present invention discussed above.
[0123] The various aspects of the present invention may be used alone, in combination, or in various arrangements not specifically discussed in the previously described embodiments, and therefore their application is not limited to the details and arrangements of the components set forth in the foregoing description or shown in the accompanying drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.
[0124] In addition, embodiments of the present invention may be implemented as one or more methods, examples of which have been provided. The actions performed as part of the method may be ordered in any suitable manner. Accordingly, embodiments may be constructed in which the actions are performed in a different order than shown, which may include performing some actions simultaneously, even though shown as sequential actions in the illustrative embodiments.
[0125] The use of ordinal terms (such as "first," "second," "third," etc.) in the claims to modify claim elements does not, by itself, imply any priority, precedence, or order of one claim element relative to another claim element or the temporal order in which the actions of a method are performed. Such terms are merely used as labels to distinguish one claim element having a particular name from another element having the same name (but for the use of ordinal terms).
[0126] The phraseology and terminology used herein are for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "involving," and variations thereof is meant to include the items listed thereafter and additional items.
[0127] After describing several embodiments of the present invention in detail, it will be easy for those skilled in the art to think of various changes and improvements. These modifications and improvements are intended to be included in the spirit and scope of the present invention. Therefore, the foregoing description is merely by way of example and is not intended to be restrictive. The present invention is limited only by the appended claims and their equivalents.
Claims
1. A computerized system comprising: a plurality of neuromuscular sensors configured to record a plurality of neuromuscular signals from a user, wherein the plurality of neuromuscular sensors are disposed on one or more wearable devices; and at least one computer processor programmed to: deriving information for a first submuscular control channel of a plurality of submuscular control channels and information for a second submuscular control channel of a plurality of submuscular control channels from the plurality of neuromuscular signals, wherein each submuscular control channel of the plurality of submuscular control channels is configured to process information associated with activation of one or more submuscular structures; each submuscular structure comprising a motor unit, the motor unit comprising a spinal motor neuron and associated muscle fibers innervated by the spinal motor neuron; generating a first control signal based on the derived information for the control channel under the first muscle; generating a second control signal based on the derived information for the second submuscle control channel; and providing the first control signal and the second control signal to a control interface to control the operation of the device; wherein the first submuscular control channel is configured to process information generated by activation of the first submuscular structure; The second submuscular control channel is configured to process information resulting from activation of a second submuscular structure; and Wherein, the first submuscular structure and the second submuscular structure are automatically selected by the at least one computer processor of the system using a criterion of maximizing or increasing a distance between activation features or signatures of the activated first submuscular structure and the second submuscular structure.
2. The computerized system of claim 1, wherein: The at least one computer processor is further programmed to: The derived information for the first sub-muscle control channel is provided as an input to a trained inference model, and wherein generating the control signal comprises generating the control signal based on an output of the trained inference model.
3. The computerized system of claim 1, wherein: The at least one computer processor is further programmed to: The plurality of neuromuscular signals are provided as input to a trained inference model, and wherein deriving information for the first submuscular control channel comprises deriving the information based on processing of the plurality of neuromuscular signals by the trained inference model.
4. The computerized system of claim 1, wherein: Deriving information for the first submuscular control channel includes decomposing the plurality of neuromuscular signals into signal components representative of the plurality of submuscular control channels.
5. The computerized system of claim 1, wherein: The first submuscular control channel is configured to process information generated by activation of a single motor unit.
6. The computerized system of claim 5, wherein: A second submuscular control channel of the plurality of submuscular control channels is configured to process information generated by activation of the plurality of motor units.
7. The computerized system of claim 1, wherein: The first submuscular control channel is configured to process information resulting from activation of a combination of at least one first submuscular structure associated with a first muscle and at least one second submuscular structure associated with a second muscle.
8. The computerized system of claim 1 , further comprising at least one auxiliary sensor configured to record an auxiliary signal while recording the plurality of neuromuscular signals from the user, wherein The at least one auxiliary sensor is disposed on the one or more wearable devices.
9. The computerized system of claim 8, wherein: The at least one auxiliary sensor includes at least one inertial measurement unit.
10. The computerized system of claim 1, wherein: The at least one computer processor is further programmed to: storing a representation of the derived information for the first submuscular control channel determined during a first session; as well as Calibrate neuromuscular signal data recorded from the user during a second session following the first session, wherein calibration is performed based at least in part on a stored representation of the derived information for the first submuscular control channel determined during the first session.
11. A computerized system for training a user to activate submuscular structures, the system comprising: a plurality of neuromuscular sensors configured to record a plurality of neuromuscular signals from the user when the user activates one or more submuscular structures, wherein: The plurality of neuromuscular sensors are disposed on one or more wearable devices; as well as at least one computer processor programmed to: providing feedback to the user based on the plurality of neuromuscular signals, wherein the feedback includes information regarding patterns of activation identified in the plurality of neuromuscular signals; deriving information for a first submuscular control channel of a plurality of submuscular control channels and information for a second submuscular control channel of a plurality of submuscular control channels from the plurality of neuromuscular signals, wherein each submuscular control channel of the plurality of submuscular control channels is configured to process information associated with activation of one or more submuscular structures; each submuscular structure comprising a motor unit, the motor unit comprising a spinal motor neuron and associated muscle fibers innervated by the spinal motor neuron; wherein the first submuscular control channel is configured to process information generated by activation of the first submuscular structure; and The second submuscular control channel is configured to process information resulting from activation of a second submuscular structure; adjusting the feedback provided to the user based on the recorded plurality of neuromuscular signals; wherein the adjusted feedback includes instructions to activate the first submuscular structure and the second submuscular structure; wherein the first submuscular structure and the second submuscular structure are automatically selected by the at least one computer processor of the system using a criterion of maximizing or increasing a distance between activation features or signatures of the activated first submuscular structure and the second submuscular structure; and storing information mapping the pattern of activation of the first submuscular structure to a first control signal; Information mapping the pattern of activation of the second submuscular structure to a second control signal is stored.
12. The computerized system of claim 11, wherein: The submuscular structure is a single motor unit.
13. The computerized system of claim 11, wherein: Providing feedback to the user based on the plurality of neuromuscular signals includes generating the feedback from an unprocessed version of the plurality of neuromuscular signals.
14. The computerized system of claim 11, wherein: The at least one computer processor is further programmed to: Based on the multiple neuromuscular signals, it is determined from a plurality of submuscular structures which one of the multiple submuscular structures the user has activated, and wherein providing feedback to the user based on the multiple neuromuscular signals includes providing first feedback when it is determined that a first submuscular structure of the multiple submuscular structures has been activated, and providing second feedback when it is determined that a second submuscular structure of the multiple submuscular structures has been activated.
15. The computerized system of claim 14, wherein: The first feedback and the second feedback have different characteristics so that the user can distinguish the first feedback from the second feedback.
16. The computerized system of claim 15, wherein: The first feedback comprises auditory feedback having a first pitch, and wherein the second feedback comprises auditory feedback having a second pitch different from the first pitch.
17. The computerized system of claim 11, wherein: The feedback includes feedback selected from the group consisting of auditory feedback, visual feedback, tactile feedback, and feedback provided by electrical stimulation.
18. The computerized system of claim 11, wherein: The at least one computer processor is further programmed to: deriving a control signal based on the plurality of neuromuscular signals; as well as providing the control signal to a device having an operation that the user is attempting to control, Wherein providing feedback to the user based on the plurality of neuromuscular signals comprises changing a behavior of the device based on the control signal.
19. The computerized system of claim 18, wherein: The device includes a display.
20. The computerized system of claim 18, wherein: Deriving the control signal includes providing the plurality of neuromuscular signals or information derived from the plurality of neuromuscular signals as input to an inference model, and deriving the control signal based on an output of the inference model.
21. The computerized system of claim 11, wherein: The at least one computer processor is further programmed to: An inference model is trained to map the pattern of activation determined from the plurality of neuromuscular signals to one or more submuscular activation patterns, and wherein storing the information mapping the pattern of activation identified in the plurality of neuromuscular signals to control signals includes storing the trained inference model.
22. The computerized system of claim 21, wherein: The at least one computer processor is further programmed to: The one or more muscle activation patterns are computationally mapped to the control signal.
23. The computerized system of claim 11, wherein: Storing the information mapping the pattern of activation identified in the plurality of neuromuscular signals to a control signal includes storing information describing the pattern of activation determined from the plurality of neuromuscular signals.
24. The computerized system of claim 21, wherein: The at least one computer processor is further programmed to: Based on the plurality of neuromuscular signals, a plurality of submuscular structures activated by the user are identified.
25. The computerized system of claim 24, wherein: identifying the plurality of submuscular structures activated by the user comprises decomposing the plurality of neuromuscular signals into signal components indicative of activation of specific submuscular structures, and Wherein mapping the information derived from the plurality of neuromuscular signals to the one or more control signals comprises mapping the signal components characterizing activation of the particular submuscular structure to the one or more control signals.
26. The computerized system of claim 11, wherein: The at least one computer processor is further programmed to: identifying a plurality of submuscular structures activated by the user based on the plurality of neuromuscular signals; selecting a subset of submuscular structures for training based on activated features associated with each of the plurality of submuscular structures activated by the user; and Wherein the feedback comprises information about the pattern of activation of the subset of submuscular structures.
27. The computerized system of claim 26, wherein: The characteristics of activation associated with the submuscular structure are selected from the group consisting of: the type of motor unit associated with the submuscular structure, the amplitude of motor unit action potentials associated with the submuscular structure, the similarity of the waveform used for activation of the submuscular structure to the waveform used for activation of other submuscular structures, and activation rate and timing statistics associated with activation of the submuscular structure.
28. The computerized system of claim 11, further comprising: At least one storage device configured to store spatiotemporal information about at least one activated submuscular structure.
29. The computerized system of claim 11, further comprising: A user interface configured to instruct a user to activate a specific submuscular structure, wherein the plurality of neuromuscular sensors are configured to record the plurality of neuromuscular signals in response to the user attempting to activate the specific submuscular structure indicated by the user interface.
30. The computerized system of claim 29, wherein: Instructing the user to activate the particular submuscular structure includes instructing the user to activate a plurality of submuscular structures in sequence, and wherein the at least one computer processor is further programmed to perform calibration based at least in part on the plurality of neuromuscular signals.
31. A computer-implemented method of controlling a device, the method comprising: receiving a plurality of neuromuscular signals recorded by a plurality of neuromuscular sensors disposed on one or more wearable devices worn by a user; deriving information for a first submuscular control channel of a plurality of submuscular control channels and information for a second submuscular control channel of a plurality of submuscular control channels from the plurality of neuromuscular signals, wherein each submuscular control channel of the plurality of submuscular control channels is configured to process information associated with activation of one or more submuscular structures; each submuscular structure comprising a motor unit, the motor unit comprising a spinal motor neuron and associated muscle fibers innervated by the spinal motor neuron; processing, by the first submuscular control channel, information generated by activation of the first submuscular structure; processing, by the second submuscular control channel, information generated by activation of the second submuscular structure; wherein the first submuscular structure and the second submuscular structure are automatically selected by at least one computer processor of the system using a criterion of maximizing or increasing a distance between activation features or signatures of the activated first submuscular structure and the second submuscular structure; generating a first control signal based on the derived information for the control channel under the first muscle; generating a second control signal based on the derived information for the second submuscle control channel; and The first control signal and the second control signal are provided to a control interface to control the operation of a device.
32. A computer-implemented method for training a user to activate submuscular structures, the method comprising: recording a plurality of neuromuscular signals from a plurality of neuromuscular sensors disposed on one or more wearable devices worn by a user; providing feedback to the user based on the plurality of neuromuscular signals, wherein the feedback includes information regarding patterns of activation identified in the plurality of neuromuscular signals; deriving information for a first submuscular control channel of a plurality of submuscular control channels and information for a second submuscular control channel of a plurality of submuscular control channels from the plurality of neuromuscular signals, wherein each submuscular control channel of the plurality of submuscular control channels is configured to process information associated with activation of one or more submuscular structures; each submuscular structure comprising a motor unit, the motor unit comprising a spinal motor neuron and associated muscle fibers innervated by the spinal motor neuron; processing, by the first submuscular control channel, information generated by activation of the first submuscular structure; processing, by the second submuscular control channel, information generated by activation of the second submuscular structure; adjusting the feedback provided to the user based on the recorded plurality of neuromuscular signals; wherein the adjusted feedback includes instructions to activate the first submuscular structure and the second submuscular structure; wherein the first submuscular structure and the second submuscular structure are automatically selected by at least one computer processor of the system using a criterion of maximizing or increasing a distance between activation features or signatures of the activated first submuscular structure and the second submuscular structure; and storing information mapping the pattern of activation of the first submuscular structure to a first control signal; Information mapping the pattern of activation of the second submuscular structure to a second control signal is stored.
