Method and apparatus for unsupervised machine learning for gesture classification and applied force estimation

Abstract

Method and apparatus for training a classification model and using the trained classification model to recognize a gesture performed by a user. An apparatus includes a processor programmed to: receive a first plurality of neuromuscular signals from a user via a plurality of neuromuscular sensors when the user performs a first single action of a gesture; train a classification model based on the first plurality of neuromuscular signals, the training comprising: deriving values ​​from the first plurality of neuromuscular signals, the values ​​indicating distinctive features of the gesture, the distinctive features comprising at least one feature that varies linearly with force applied during performance of the gesture; and generating a first classification representation of the gesture in the classification model based on the values; and determining that the user performed a second single action of the gesture based on the trained classification model and the second plurality of neuromuscular signals.

Description

[0001] background

[0002] Systems that utilize machine learning techniques to recognize and model gestures performed by users typically require large sets of labeled data or training samples that are labeled by humans and are susceptible to human bias or labeling errors. For example, in the context of image recognition, supervised learning models are trained to recognize gestures based on labeled data, such as multiple images capturing different angles of a particular gesture, where the images are labeled by humans to indicate the angle, type, and other aspects of the gesture.

[0003] Overview

[0004] In systems for recognizing and modeling human gestures, the inventors have recognized that it is desirable for the system to quickly learn gestures from very few training samples. It may also be desirable for the system to capture and interpret meaningful features from gestures in an unsupervised manner, e.g., using unlabeled training data. Such meaningful features may include features indicating the force or magnitude of force applied during the performance of a gesture, which may convey different meanings. For example, different magnitudes of force applied to the same gesture may allow the system to generate different command signals for controlling objects in a virtual or augmented reality environment, controlling devices in a user's environment, or other suitable systems and / or devices.

[0005] Some embodiments relate to a device that includes a processor, a plurality of neuromuscular sensors coupled to the processor, and a memory storing instructions. When executed by the processor, the instructions cause the processor to: receive a first plurality of neuromuscular signals from the user via the plurality of neuromuscular sensors when the user performs a first single action of a gesture, and train a classification model based on the first plurality of neuromuscular signals. Training the classification model includes: deriving one or more values ​​from the first plurality of neuromuscular signals, the one or more values ​​indicating distinctive features of the gesture, the distinctive features including at least one feature that varies linearly with the force applied during the performance of the gesture; and generating a first classification representation of the gesture in the classification model based on the one or more values ​​derived from the first plurality of neuromuscular signals. When executed by the processor, the instructions cause the processor to: receive a second plurality of neuromuscular signals from the user via the plurality of neuromuscular sensors when the user performs a second single action of the gesture; and determine that the user performed the second single action of the gesture based on the classification model and the second plurality of neuromuscular signals.

[0006] Other embodiments relate to an apparatus comprising a processor, a plurality of neuromuscular sensors coupled to the processor, and a memory storing instructions. When executed by the processor, the instructions cause the processor to: train a classification model based on a first plurality of neuromuscular signals and a second plurality of neuromuscular signals, the first plurality of neuromuscular signals and the second plurality of neuromuscular signals being received via the plurality of neuromuscular sensors. The training includes deriving a first set of values ​​indicating distinguishing features of a first gesture and a second set of values ​​indicating distinguishing features of a second gesture based on a clustering technique, the distinguishing features of the first gesture including at least one feature that varies linearly with the force applied during the execution of the first gesture, and the distinguishing features of the second gesture including at least one feature that varies linearly with the force applied during the execution of the second gesture; and generating a classification representation of the first gesture and a classification representation of the second gesture in the classification model. When executed by the processor, the instructions cause the processor to determine whether the user has performed a subsequent action of the first gesture or a subsequent action of the second gesture based at least in part on a third plurality of neuromuscular signals and the classification model.

[0007] Other embodiments are directed to a method comprising receiving, at a processor of a wearable device, a plurality of neuromuscular signals from a plurality of neuromuscular sensors included in the wearable device, the plurality of neuromuscular signals corresponding to neuromuscular signals sampled from a user when the user performs a single action of a gesture; training a classification model based on the single action of the gesture via an unsupervised machine learning technique, the classification model comprising a classification representation of the gesture; determining, based on the classification representation of the gesture, whether the user has performed a subsequent single action of the gesture; determining at least one force value corresponding to a force applied by the user during performance of the subsequent single action of the gesture; and generating a command signal to be transmitted to the device in response to determining that the user has performed the subsequent single action of the gesture, wherein the command signal indicates the gesture and at least one force value applied during performance of the gesture.

[0008] Still other embodiments relate to a computerized system for training a classification model based on a single motion of a first gesture. The system includes a plurality of neuromuscular sensors configured to record a plurality of neuromuscular signals from a user when the user performs the single motion of the first gesture; and at least one computer processor programmed to: train the classification model using unsupervised machine learning techniques to create a unique representation of the gesture in the classification model based on at least some of the plurality of neuromuscular signals.

[0009] In one aspect, the at least one computer processor is further programmed to identify at least one activity period in the plurality of recorded neuromuscular signals, wherein at least some of the plurality of neuromuscular signals used to train the classification model are neuromuscular signals recorded during the at least one activity period.

[0010] In another aspect, at least some of the plurality of neuromuscular signals used to train the classification model do not include neuromuscular signals indicative of a resting or neutral position.

[0011] In another aspect, identifying at least one active period includes identifying the active period as a time period during which a power value associated with each of the plurality of neuromuscular signals is above a threshold.

[0012] In another aspect, creating a unique representation of the first gesture in the classification model includes processing at least some of the plurality of neuromuscular signals to generate a plurality of points in a component feature space; and clustering at least some of the generated points to create a unique representation of the first gesture.

[0013] In another aspect, processing at least some of the plurality of neuromuscular signals to generate a plurality of points in a component feature space includes performing a principal component analysis on at least some of the plurality of neuromuscular signals.

[0014] In another aspect, clustering at least some of the generated points comprises applying K-means cluster analysis to generate one or more clusters of points in the component feature space; and including the one or more clusters of points in the unique representation of the first gesture based on a similarity measure.

[0015] In another aspect, the similarity measure comprises cosine distance.

[0016] In another aspect, creating the unique representation of the first gesture includes generating a vector in the component feature space based on the generated points.

[0017] In another aspect, points along the vector represent different amounts of force used to perform the first gesture.

[0018] In another aspect, the at least one computer processor is further programmed to associate the first control signal with a unique representation of the first gesture.

[0019] In another aspect, at least one computer processor is further programmed to associate the first control signal and the second control signal with a unique representation of the first gesture, wherein the system is configured to generate the first control signal when the first gesture is performed while applying a force below a threshold, and to generate the second control signal when the first gesture is performed while applying a force equal to or above the threshold.

[0020] On the other hand, the plurality of neuromuscular sensors are also configured to record a second plurality of neuromuscular signals when the user performs a single action of a second gesture; and at least one computer processor is programmed to: train the classification model using unsupervised machine learning techniques to create a unique representation of the second gesture in the classification model based on at least some of the second plurality of neuromuscular signals.

[0021] On the other hand, creating a unique representation of the second gesture in the classification model includes: processing at least some of the second plurality of neuromuscular signals to generate a plurality of second points in the component feature space; and clustering at least some of the plurality of second points to create a unique representation of the second gesture.

[0022] On the other hand, creating a unique representation of a first gesture in a classification model includes: processing at least some of the multiple neuromuscular signals to generate a first plurality of points in a component feature space; and generating a first vector in the component feature space based on the generated first plurality of points, wherein creating a unique representation of a second gesture includes generating a second vector in the component feature space based on the second plurality of points, and wherein the second vector associated with the second gesture is different from the first vector associated with the first gesture.

[0023] In another aspect, a plurality of neuromuscular sensors are disposed on one or more wearable devices.

[0024] Other embodiments relate to a computerized system for classifying gestures performed by a user. The system includes a plurality of neuromuscular sensors configured to record a plurality of neuromuscular signals from a user when the user performs a first gesture and at least one computer processor. The at least one computer processor is programmed to: create a representation of the first gesture in a component feature space of a classification model, wherein the classification model is trained to include a unique representation of each of a plurality of gestures in the component feature space; determine whether the representation of the first gesture corresponds to any unique representation of the plurality of gestures included in the classification model; and when it is determined that the representation of the first gesture corresponds to one of the unique representations, generate a control signal associated with the unique representation.

[0025] In one aspect, creating the representation of the first gesture includes processing at least some of the plurality of neuromuscular signals to generate a plurality of points in a component feature space; and clustering at least some of the plurality of points to create the representation of the first gesture.

[0026] In another aspect, processing at least some of the plurality of neuromuscular signals to generate a plurality of points in a component feature space includes performing a principal component analysis on at least some of the plurality of neuromuscular signals.

[0027] In another aspect, clustering at least some of the plurality of points comprises: applying k-means cluster analysis to generate one or more clusters of points in the component feature space; and including the one or more clusters of points in the representation of the first gesture based on a similarity measure.

[0028] In another aspect, the similarity measure comprises cosine distance.

[0029] In another aspect, creating a representation of the first gesture includes generating a first vector in a component feature space based on the plurality of points.

[0030] In another aspect, determining whether the representation of the first gesture corresponds to one of the unique representations includes determining whether a first vector associated with the first gesture corresponds to one of a plurality of vectors associated with a plurality of gestures based on a similarity measure.

[0031] In another aspect, the similarity measure is a cosine distance between the first vector and each of the plurality of vectors in the component feature space.

[0032] In another aspect, a plurality of neuromuscular sensors are disposed on one or more wearable devices.

[0033] It should be appreciated that all combinations of the preceding concepts and the additional concepts discussed in more detail below (assuming such concepts are not mutually inconsistent) are contemplated as part of the inventive subject matter disclosed herein. In particular, all combinations of the claimed subject matter appearing at the end of this disclosure are contemplated as part of the inventive subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Various non-limiting embodiments of the technology will be described with reference to the following drawings. It should be appreciated that the drawings are not necessarily drawn to scale.

[0036] Figure 1 is a schematic diagram of a computer-based system for processing neuromuscular sensor data according to some embodiments of the technology described herein;

[0037] Figure 2A shows a wearable system according to some embodiments of the technology described herein, the wearable system having 16 EMG sensors arranged circumferentially around an elastic band configured to be worn around a user's lower arm or wrist;

[0038] Figure 2B yes Figure 2A A cross-sectional view of one of the 16 EMG sensors shown in FIG.

[0039] Figure 3A and Figure 3B Components of a computer-based system upon which some embodiments are implemented are schematically shown. Figure 3A shows the wearable portion of the computer-based system, Figure 3B showing a dongle portion connected to a computer, wherein the dongle portion is configured to communicate with the wearable portion;

[0040] Figure 4 is a flow chart of a process for generating a classification model based on neuromuscular signals recorded from sensors according to some embodiments of the techniques described herein;

[0041] Figure 5 shows examples of user-defined gestures according to some embodiments of the technology described herein;

[0042] Fig. 6A showing a first set of gestures associated with a first user according to some embodiments of the techniques described herein;

[0043] Figure 6B showing a second set of gestures associated with a second user according to some embodiments of the techniques described herein;

[0044] Figure 7 is a flowchart of a process for recognizing gestures based on a trained classification model according to some embodiments of the technology described herein;

[0045] Figure 8 Some embodiments of the technology described herein are shown by arranging Figure 2A Sixteen sensors on the wearable system captured signals representing three different gestures;

[0046] Fig. 9 shows an example of a covariance matrix calculated from neuromuscular signals associated with different gestures according to some embodiments of the techniques described herein;

[0047] Fig.10 shows a graph illustrating power values ​​associated with data points corresponding to stationary and active positions according to some embodiments of the techniques described herein;

[0048] Fig.11 shows tangent space and principal component analysis (PCA) space computed from a training data set according to some embodiments of the techniques described herein;

[0049] Fig.12 shows an example of learning gesture vectors according to some embodiments of the technology described herein;

[0050] Fig.13 shows a graph showing force calculated as a function of maximum voluntary contraction for each gesture performed during a training phase according to some embodiments of the technology described herein;

[0051] Fig.14 shows identifying three different gestures based on their calculated cosine distances according to some embodiments of the techniques described herein;

[0052] Fig.15 shows the force inferred for each gesture recognized during the classification phase, according to some embodiments of the techniques described herein;

[0053] Fig.16 Shown are three unseen vectors mapped to principal component analysis (PCA) space according to some embodiments of the techniques described herein.

[0054] Detailed Description

[0055] Some existing techniques for recognizing and modeling human gestures use labeled training data that is prone to human errors. Mislabeled training data leads to inaccurate training of supervised learning models, which in turn leads to gesture recognition errors. In addition, such techniques usually require a large amount of labeled training data, which is difficult and time-consuming to acquire and generate. In addition, processing such a large amount of labeled data consumes a lot of computing and memory resources.

[0056] Some existing techniques for recognizing and modeling human gestures also fail to capture meaningful features associated with human gestures, such as the force or magnitude of force applied during the execution of a gesture. For example, in the context of image recognition, supervised learning models can be trained to recognize gestures based on labeled image data. These labeled images reflect the positioning of a person's hand during the execution of a gesture, but the labeled images do not reflect information associated with the force applied by the user when performing the gesture.

[0057] The inventors have recognized that prior art techniques for recognizing and modeling human gestures can be improved by utilizing unsupervised machine learning or training methods to train a classification model to recognize and model one or more user-defined gestures and the forces applied by the user when performing those gestures. The classification model can be trained with unlabeled data obtained from a neuromuscular sensor disposed on a wearable device. The unlabeled data can include neuromuscular signals recorded from the neuromuscular sensor.

[0058] The inventors have recognized that by utilizing an unsupervised training method (an example of which is described in detail below), a classification model can be trained based on neuromuscular signals recorded when a user performs a single action of a gesture (also known as "one-shot" training or learning). The classification model is trained to recognize a gesture by generating a classification representation of the gesture in the model. The classification representation of the gesture can be generated based on one or more features derived from the recorded neuromuscular signals and / or one or more force values ​​associated with the force applied during the performance of the gesture. The classification representation can include a representation from which the type of gesture performed by the user and the amount of force applied by the user during the performance of the gesture can be inferred. Using the trained classification model, subsequent performances of the same gesture by the user can be identified based on its classification representation in the classification model.

[0059] According to some embodiments, a classification model can be trained to recognize multiple gestures performed by one or more users. For example, the classification model can be trained based on neuromuscular signals recorded when the user performs a single action of each gesture. Clustering techniques can be used to divide the recorded sensor data into multiple clusters, each cluster associated with a specific gesture. A classification representation for each gesture can be determined based on the associated clusters. For example, the classification representation can include information that identifies the type of gesture, direction, and force scale associated with the gesture (e.g., a range of force values / force magnitudes). After training the classification model, the system can determine whether the gesture performed by the user maps to any classification representation associated with the different gestures represented in the classification model.

[0060] Figure 1 A system 100 according to some embodiments is shown. The system includes a plurality of sensors 102, which are configured to record signals generated by neuromuscular activity in human skeletal muscles. The term "neuromuscular activity" used herein refers to any combination of neural activation, muscle activation, muscle contraction, or neural activation, muscle activation, and muscle contraction of the spinal motor neurons that control muscles. Neuromuscular sensors may include one or more electromyographic (EMG) sensors, one or more myomechanical (MMG) sensors, one or more sonomyographic (SMG) sensors, two or more types of EMG sensors, a combination of MMG sensors and SMG sensors, and / or one or more sensors of any suitable type configured to detect neuromuscular signals. In some embodiments, a plurality of neuromuscular sensors may be used to sense muscle activity associated with the movement of a body part controlled by a muscle, and the neuromuscular sensors are arranged to sense muscle activity from a muscle. When a user moves or performs one or more gestures over time, spatial information (e.g., position and / or orientation information) and force information describing movement may be predicted based on the sensed neuromuscular signals.

[0061] The sensor 102 may include one or more inertial measurement units (IMUs) that measure a combination of physical aspects of motion using, for example, accelerometers, gyroscopes, magnetometers, or any combination of one or more accelerometers, gyroscopes, and magnetometers. In some embodiments, the IMU may be used to sense information about the movement of a body part to which the IMU is attached, and information derived from the sensed data (e.g., position and / or orientation information) may be tracked as the user moves over time. For example, one or more IMUs may be used to track the movement of a portion of the user's body proximal to the user's torso (e.g., arms, legs) relative to the sensor as the user moves over time or performs one or more gestures.

[0062] In an embodiment including at least one IMU and multiple neuromuscular sensors, the IMU and neuromuscular sensors can be arranged to detect movement of different parts of the human body. For example, the IMU can be arranged to detect the movement of one or more body segments (body segments) (e.g., upper arms) at the proximal end of the torso, and the neuromuscular sensors can be arranged to detect the movement of one or more body segments (e.g., forearms or wrists) at the distal end of the torso. However, it should be understood that autonomous sensors can be arranged in any suitable manner, and embodiments of the technology described herein are not limited to specific sensor arrangements. For example, in some embodiments, at least one IMU and multiple neuromuscular sensors can be co-located on a body segment to track the movement of the body segment using different types of measurements. In one implementation described in more detail below, the IMU sensor and multiple EMG sensors are arranged on a wearable device that is configured to be worn around the user's lower arm or wrist. In such an arrangement, the IMU sensors can be configured to track motion information associated with one or more arm segments (e.g., position and / or orientation over time) to determine, for example, whether a user has raised or lowered their arm, while the EMG sensors can be configured to determine motion information associated with the wrist or hand segments to determine, for example, whether the user has an open or closed hand configuration.

[0063] Each sensor 102 includes one or more sensing components configured to sense information about the user. In the case of an IMU, the sensing components may include one or more accelerometers, gyroscopes, magnetometers, or any combination thereof to measure characteristics of body movement, examples of which include, but are not limited to, acceleration, angular velocity, and magnetic fields sensed around the body. In the case of a neuromuscular sensor, the sensing components may include, but are not limited to, electrodes configured to detect electrical potentials on the surface of the body (e.g., for an EMG sensor), vibration sensors configured to measure vibrations of the skin surface (e.g., for an MMG sensor), and acoustic sensing components configured to measure ultrasonic signals generated by muscle activity (e.g., for an SMG sensor).

[0064] In some embodiments, at least some of the plurality of sensors 102 are arranged as part of a wearable device that is configured to be worn on or around a body part of a user. For example, in one non-limiting example, an IMU sensor and a plurality of neuromuscular sensors are arranged circumferentially around an adjustable and / or elastic band (e.g., a wristband or armband configured to be worn around a user's wrist or arm). Alternatively, at least some of the autonomous sensors may be arranged on a wearable patch that is configured to be attached to a portion of a user's body. In some embodiments, multiple wearable devices may be used to predict musculoskeletal position information involving movements of multiple body parts, each wearable device including one or more IMUs and / or neuromuscular sensors.

[0065] In some embodiments, sensor 102 includes only a plurality of neuromuscular sensors (e.g., EMG sensors). In other embodiments, sensor 102 includes a plurality of neuromuscular sensors and at least one "auxiliary" sensor configured to continuously record a plurality of auxiliary signals. Examples of auxiliary sensors include, but are not limited to, other autonomous sensors (e.g., IMU sensors) and non-autonomous sensors (e.g., imaging devices (e.g., cameras)), radiation-based sensors used with radiation-generating devices (e.g., laser scanning devices), or other types of sensors (e.g., heart rate monitors).

[0066] In some embodiments, hardware signal processing circuitry may be used to process the output of one or more sensing components (e.g., to perform amplification, filtering, and / or rectification). In other embodiments, at least a portion of the signal processing of the output of the sensing components may be performed in software. Thus, signal processing of the signals recorded by the sensors may be performed in hardware, software, or by any suitable combination of hardware and software, as aspects of the techniques described herein are not limited in this regard.

[0067] In some embodiments, as described in more detail below, the recorded sensor data can be processed to calculate additional derived measurements or features, which are then provided as input to a classification model. For example, recorded signals from an IMU sensor can be processed to derive an orientation signal that specifies the orientation of a rigid body segment over time. The sensor 102 can implement signal processing using components that are integrated with the sensing components, or at least a portion of the signal processing can be performed by one or more components that communicate with but are not directly integrated with the sensing components of the sensor.

[0068] The system 100 also includes one or more computer processors 104 programmed to communicate with the sensor 102. For example, the signals recorded by the one or more sensors can be provided to the processor, which can be programmed to process the signals output by the sensor 102 to train one or more classification models 106, and the trained (or retrained) classification models 106 can be stored for later use in recognizing / classifying gestures and generating control / command signals, as described in more detail below. In some embodiments, the processor 104 can be programmed to derive one or more features associated with one or more gestures performed by the user, and the derived features can be used to train the one or more classification models 106. The processor 104 can be programmed to recognize subsequently performed gestures based on the trained one or more classification models 106. In some implementations, the processor 104 can be programmed to map the recognized gestures to one or more control / command signals, at least in part, using the classification models.

[0069] Figure 2A Shown is a wearable system with sixteen neuromuscular sensors 210 (e.g., EMG sensors), which are arranged circumferentially around an elastic band 220, which is configured to be worn around the user's lower arm or wrist. As shown, the EMG sensors 210 are arranged circumferentially around the elastic band 220. It should be understood that any suitable number of neuromuscular sensors can be used. The number and arrangement of the neuromuscular sensors can depend on the specific application used for the wearable device. For example, a wearable armband or wristband can be used to generate control information for controlling an augmented reality system, a virtual reality system, a robot, controlling a vehicle, scrolling text, controlling a virtual avatar, or any other suitable control task. As shown, the sensors 210 can be coupled together using a flexible electronic device 230 incorporated into the wearable device. Figure 2B Shows Figure 2A A cross-sectional view of one of the sensors 210 of the wearable device is shown.

[0070] In some embodiments, hardware signal processing circuitry may optionally be used to process (e.g., perform amplification, filtering, and / or rectification) the output of one or more sensors. In other embodiments, at least some of the signal processing of the sensor outputs may be performed in software. Thus, the processing of the signals sampled by the sensors may be performed in hardware, software, or any suitable combination of hardware and software, as aspects of the technology described herein are not limited in this regard. Figure 3A and Figure 3B A non-limiting example of a signal processing chain for processing recorded data from sensor 210 is discussed in more detail.

[0071] Figure 3A and Figure 3B 1 shows a schematic diagram of some embodiments of the technology described herein, showing some internal components of a wearable system with sixteen EMG sensors. As shown, the wearable system includes a wearable portion 310 ( Figure 3A ) and a dongle portion 320 (eg, via Bluetooth or another suitable short-range wireless communication technology) that communicates with the wearable portion 310 Figure 3B ).like Figure 3A As shown, the wearable portion 310 includes a sensor 210, which is an example of a combination of Figure 2A and Figure 2B The output of sensor 210 is provided to an analog front end 330, which is configured to perform analog processing (e.g., noise reduction, filtering, etc.) on the recorded signal. The processed analog signal is then provided to an analog-to-digital converter 332, which converts the analog signal into a digital signal that can be processed by one or more computer processors (e.g., processor 104). An example of a computer processor that can be used according to some embodiments is Figure 3A 334. As shown, the MCU 334 may also include inputs from other sensors (e.g., IMU sensor 340) and power and battery module 342. The output of the processing performed by the MCU may be provided to antenna 350 for transmission to Figure 3B A dongle portion 320 is shown.

[0072] The dongle portion 320 includes an antenna 352 configured to communicate with an antenna 350 included as part of the wearable portion 310. Communication between the antennas 350 and 352 may be performed using any suitable wireless technology and protocol, non-limiting examples of which include radio frequency signaling and Bluetooth. As shown, signals received by the antenna 352 of the dongle portion 320 may be provided to a host computer for further processing, display, and / or enabling control of a particular physical or virtual object or objects.

[0073] Figure 4 A process 400 for generating (also referred to herein as “training”) a classification model 106 using signals recorded from sensors 102 , including, for example, sensor 210 , is described. Figure 4 denoting a “training phase” for training a classification model. Process 400 may be performed by any suitable computing device, as aspects of the techniques described herein are not limited in this respect. For example, process 400 may be performed by reference to Figure 1 , Figure 3A and Figure 3BAs another example, one or more servers (e.g., servers included as part of a cloud computing environment) may be used to perform one or more actions of process 400. For example, at least a portion of action 408 associated with training of a classification model may be performed using a cloud computing environment. In some embodiments, the classification model may be trained in near real time. As used herein, the term "near real time" refers to instantaneous perception by a user, such as on the order of milliseconds.

[0074] Process 400 begins at act 402, where a user performs one or more gestures (e.g., Figure 5 In some embodiments, the plurality of sensor signals may be recorded as part of process 200. In other embodiments, the plurality of sensor signals may have been recorded prior to performing process 400 and are accessed (rather than recorded) at act 402.

[0075] As used herein, the term "gesture" refers to a static or dynamic configuration of one or more body parts and the forces associated with the configuration, including the position of one or more body parts. For example, gestures include discrete gestures (e.g., pressing the palm down on a solid surface or grabbing a ball), continuous gestures (e.g., waving fingers back and forth or throwing a ball), or a combination of discrete gestures and continuous gestures (e.g., grabbing and throwing a ball). Gestures can be arbitrarily defined by the user. In some cases, hand gestures and arm gestures may be symbolic and used for communication according to cultural standards.

[0076] In some embodiments, gestures may include non-limiting or custom gestures defined by the user (also referred to herein as "user-defined" gestures), which may represent the user's own and unique set of gestures. These gestures may not be pre-defined or pre-modeled in the system, and the classification model may be trained to recognize any type of gesture when any amount of force is applied. It will be understood that the present disclosure is not limited to utilizing or recognizing Figure 5 The gestures depicted in the figure can be utilized by any type of user-defined gesture.

[0077] In some embodiments, the plurality of sensor signals may include sensor signals recorded for a single user performing a single gesture or multiple gestures, and the classification model may be trained to recognize the gestures performed by the user. When the user performs a gesture, the sensor signal may be recorded. The sensor signal may be recorded by any suitable number of sensors located in any suitable position to detect the user's movement indicating the gesture. For example, when the user uses the fingers of his / her right hand to perform a gesture, the sensor signal may be recorded by a plurality of neuromuscular sensors arranged circumferentially (or otherwise) around the user's right lower arm to detect the muscle activity in the right lower arm that causes the user's right hand to move. One or more IMU sensors may be arranged to predict the joint angle of the user's arm relative to the user's torso. As another example, when the user uses his legs to perform a gesture, the sensor signal may be recorded by a plurality of neuromuscular sensors arranged circumferentially (or otherwise) around the user's legs to detect the muscle activity in the legs that causes the foot to move, and one or more IMU sensors may be arranged to predict the joint angle of the user's legs relative to the user's torso.

[0078] In some embodiments, the plurality of sensor signals may include sensor signals recorded for a plurality of users performing various gestures, and the classification model may be trained to recognize gestures performed by the plurality of users. For example, the classification model may be trained to recognize a first set of gestures associated with a first user of the system (e.g., Fig. 6A ) and a second set of gestures associated with a second user of the system (as shown Figure 6B ). In some cases, the first set of gestures can be different from the second set of gestures. In some other instances, the first set of gestures and the second set of gestures can be the same. In still other instances, some gestures in the first set and the second set can overlap (e.g., the groups can have one or more common gestures).

[0079] In some embodiments, the sensor signals obtained in action 402 correspond to signals from one type of sensor (e.g., one or more IMU sensors or one or more neuromuscular sensors), and a classification model can be trained based on sensor signals recorded using a particular type of sensor, resulting in a trained classification model specific to the sensor type. For example, the sensor signals obtained can include multiple EMG sensor signals arranged around a user's lower arm or wrist, and a classification model can be trained to recognize gestures performed by the user.

[0080] In embodiments where gestures are recognized based on multiple types of sensors (e.g., IMU sensors, EMG sensors, MMG sensors, SMG sensors), a separate classification model may be trained for each type of sensor, and the outputs of the sensor type-specific models may be combined to recognize gestures performed by a user. In other embodiments, the sensor signals obtained from two or more different types of sensors in action 402 may be provided to a single classification model that is trained based on signals recorded from different types of sensors. In one illustrative implementation, as discussed in more detail below, an IMU sensor and multiple EMG sensors are arranged on a wearable device configured to be worn around a user's forearm, and the signals recorded by the IMU sensor and the EMG sensor are provided jointly as input to the classification model.

[0081] In some embodiments, when the user performs one or more gestures, the sensor signals obtained in action 402 are recorded at multiple time points. As a result, the recorded signal of each sensor may include data obtained at each time point in the multiple time points. Assuming that n sensors are arranged to simultaneously measure the user's movement information during the performance of the gesture, the sensor signals recorded for the user may include time points t1, t2, ..., t K K n-dimensional vectors {x k |1≤k≤K} time series.

[0082] Next, process 400 proceeds to act 404, where the sensor signal obtained in act 202 is optionally processed. For example, the sensor signal may be processed using amplification, filtering, rectification, or other types of signal processing. In some embodiments, the sensor signal obtained in act 402 may be filtered using a bandpass or high-pass filter.

[0083] Next, process 400 proceeds to act 406, where a training data set is generated for the purpose of training the classification model. The training data set may be generated based on the sensor signals obtained in act 402 or the processed sensor signals from act 404. In some embodiments, generating the training data set may include removing certain samples corresponding to static or neutral positions from the training data set, as described in more detail below.

[0084] Next, process 400 proceeds to action 408, where a classification model is trained based on the training data set. In some embodiments, training the classification model may include determining one or more distinguishing features associated with a gesture based on values ​​sampled or otherwise derived from sensor signals in the training data set (e.g., data obtained at multiple time points). The one or more distinguishing features may include features that vary linearly with the force applied by the user or other attributes associated with the gesture (e.g., gesture type, direction, force scale, etc.) during performance of the gesture. According to some aspects, training the classification model may include generating a classification representation for each gesture performed by the user in the classification model. The classification representation may be generated based on values ​​indicating one or more distinguishing features derived from the sensor signals.

[0085] In some embodiments, training a classification model can be performed by providing the model with a sequence of data sets as input, each data set in the sequence comprising an n-dimensional vector of sensor data. The classification model can provide as output the likelihood that a particular gesture recognized by the classification model based on the input signals corresponds to a gesture performed by a user. For example, the classification model can use the data sets at time points t1, t2, ..., t K The vector sequence {x k |1≤k≤K} as input, where the vector x j The i-th component is generated by the i-th sensor at time t j The measured values ​​and / or the values ​​from the i-th sensor at time t j In another non-limiting example, the derived value provided as input to the classification model may include or otherwise indicate the measured value at time t j and / or time t j Features previously extracted from data from all sensors or a subset of sensors (e.g., a covariance matrix, a cospectral matrix, a power spectrum, a combination thereof, or any other suitable derived representation.) Based on such input, a classification model can be trained to recognize gestures performed by a user or multiple users.

[0086] In embodiments including different types of sensors (e.g., IMU sensors and neuromuscular sensors) configured to simultaneously record different types of motion information during task performance, sensor data for the different types of sensors may be recorded using the same or different sampling rates. When sensor data is recorded at different sampling rates, at least some of the sensor data may be resampled (e.g., upsampled or downsampled) so that all sensor data provided as input to the classification model corresponds to time series data of the same temporal resolution. At least some of the sensor data may be resampled in any suitable manner, including but not limited to upsampling using interpolation and downsampling using decimation.

[0087] In addition to or as an alternative to resampling at least some sensor data when recorded at different sampling rates, some embodiments employ a classification model configured to accept multiple inputs asynchronously. For example, the classification model can be configured to model the distribution of "missing" values ​​in input data with a lower sampling rate. Alternatively, training of the classification model occurs asynchronously as inputs from multiple sensor data measurements become available as training data.

[0088] Next, process 400 proceeds to act 410, where the trained classification model is stored (e.g., in a data storage device not shown). Any suitable format may be used to store the trained statistical model, as aspects of the technology described herein are not limited in this respect. Thus, the classification model generated during execution of process 400 may be used at a later time, for example, to recognize a gesture performed by a user or multiple users based on a given set of input sensor data, as described below.

[0089] Figure 7 A process 700 is shown for recognizing user-defined gestures based on recorded sensor data in accordance with some embodiments. Figure 7 denoting a "gesture recognition phase," wherein a trained classification model is used to recognize gestures performed by a user and generate corresponding command / control signals. In action 702, sensor data recorded by one or more sensors is provided as input to one or more trained classification models, the classification models including one or more classification representations of each gesture previously performed by the user or users, as briefly described above.

[0090] Process 700 then proceeds to act 704, where a gesture performed by the user is recognized or identified based on the trained classification model. In some embodiments, it can be determined whether the gesture performed by the user maps to a previously learned gesture in the trained classification model. In some embodiments, neuromuscular signals are continuously recorded during user movement (including during gesture performance) and continuously provided as input to the trained classification model, resulting in near real-time prediction / recognition of gestures performed by the user based on the trained classification model.

[0091] Process 700 then proceeds to action 706, where one or more command / control signals may be generated based on the recognized gesture. The one or more control signals may include signals to activate / deactivate a computing device, control an object in a virtual environment, control an object in an augmented reality environment, control an IoT (Internet of Things) device, and / or other suitable systems and / or devices. In some embodiments, a single gesture may be configured to generate different signals based on the force applied by the user during the execution of the gesture. In some embodiments, for example, in action 704, the level or magnitude of the force applied by the user during the execution of the gesture may be determined. The measured force may be calculated as a continuous variable, an ordered variable, an interval variable, a descriptive statistical measure, or other suitable form.

[0092] Example Techniques for Training Classification Models - Training Phase

[0093] In some embodiments, a wearable system (e.g., Figure 2A , Figure 3A A user of a system (such as that depicted in ) can perform a set of gestures (e.g., G1, G2, and G3) or any other number of gestures in succession, including only one gesture. The user can perform each gesture only once (i.e., perform a single action of each gesture). As the user performs the gestures, the system can record and store neuromuscular signals (e.g., EMG signals). The gestures in the set of gestures may include, for example, G1 = fist gesture, G2 = open hand gesture, and G3 = index finger pinch gesture. As described above, the user can alternatively perform other types of gestures different from G1, G2, and G3. In other words, the user is not limited to performing a set of predefined gestures. More specifically, the unsupervised machine learning techniques used in accordance with some embodiments can be configured to recognize any type of gesture (e.g., arm or hand gestures), regardless of their form.

[0094] like Figure 8As shown, when the user performs gestures G1, G2, and G3, the system can capture and record a set of neuromuscular signals sampled from the user. The set of neuromuscular signals can be used as a training data set for the classification model. In some embodiments, the set of neuromuscular signals may include signals recorded for each of the sixteen neuromuscular sensors of the wearable system. Each signal in the set can be filtered using, for example, bandpass, high pass, or other filtering techniques.

[0095] In some embodiments, one or more values ​​indicative of a distinctive feature of each gesture may be derived from the set of neuromuscular signals. The distinctive feature may include at least one feature that varies linearly with the force applied by the user during performance of the gesture. Examples of distinctive features include, but are not limited to, a covariance matrix calculated from the values ​​sampled from the neuromuscular signals, a cospectral matrix calculated from these values, a set of filter bank covariance matrices corresponding to different frequencies, and a power spectrum for each sensor (e.g., a log-PSD (power spectral density) detected by the sensor).

[0096] In some embodiments, the covariance between the filtered signals from the EMG sensors (corresponding to 16 EMG channels) is calculated, for example, over a 200 ms window with a step size of 25 ms. Given the signals x(t) of N channels at time t, a matrix X = [x(t), ., x(t+T)] can be defined, (e.g., a N _channel x T_time_sample matrix). Given a centered signal x(t) (e.g., with zero mean after applying a bandpass filter), the covariance matrix can be expressed as:

[0097] C=XX T

[0098] Fig. 9 Examples of covariance matrices calculated for different gestures are shown. The covariance matrix corresponding to the same gesture, such as a fist, can also provide force characteristics so that the system can distinguish between a loose fist gesture (e.g., a fist gesture performed with a smaller amount of force) and a tight fist gesture (e.g., a fist gesture performed with a larger amount of force).

[0099] In some embodiments, samples indicating a stationary or neutral position are removed from the training set because such signals are unlikely to include features that can be used to predict or identify gestures. For example, EMG samples indicating a stationary or neutral position can be identified by calculating a threshold (e.g., based on EMG log power), which distinguishes a stationary or neutral position from a gesture position (also referred to herein as an active position). Therefore, samples with an EMG log power below the calculated threshold can be excluded from the training set. Excluding samples from the training data set (e.g., based on their EMG log power) improves system prediction accuracy and, in some cases, can prevent the system from generating false positives. For example, in some cases, a gesture performed with a negligible amount of force (e.g., below a calculated threshold) can be considered to be performed unintentionally or accidentally by the user and should therefore not be recognized by the system as an intentional gesture. Fig.10 Shown is a graph depicting EMG log power for data points corresponding to resting and active positions.

[0100] In one embodiment, the threshold can be calculated, for example, by: (a) calculating the 10th percentile of the data and using the calculated value as the threshold to capture data containing active positions (approximately the 90th percentile) and containing static positions (approximately the 10th percentile), (b) rescaling the samples (e.g., EMG samples) to between [0, 1] and then setting the threshold to 0.1, or (c) determining the start of the gesture (e.g., by looking at the time difference of the power) and then setting the threshold to the value of the previous point before the start. Similarly, in some implementations, the threshold can be calculated by spectral clustering techniques or other suitable clustering techniques.

[0101] According to some aspects, the neuromuscular signal can be processed to generate a plurality of data points in a feature space. In some embodiments, the vectorized covariance can be projected or mapped to the tangent space of the manifold. For example, given the covariance matrix C i , the vectorized covariance can be calculated as follows:

[0102] S i =log(P -(1 / 2) C i P -(1 / 2) )

[0103] where P is a reference point and may correspond to, for example, an identity matrix or an average covariance matrix computed from samples taken during a user's static or neutral position. i The size of is the same as the covariance matrix. The vectorized covariance can be based on the symmetric matrix S i The upper diagonal part of is calculated, for example, by keeping the symmetric matrix S iThe upper triangular part, and it is vectorized by calculating the unit weights of the diagonal elements and the square root weights of the non - diagonal elements.

[0104] The inventors have recognized that covariance and tangent space mapping provide a richer and more sensitive feature set than other features such as the mean absolute value (MAV) of the i - th EMG channel calculated over a time window, because MAV does not take into account the correlations between different EMG channels.

[0105] In some embodiments, the average covariance of the stationary or neutral position data in the training dataset can be used as a reference point, such as the zero point of the tangent space. In some cases, dimensionality reduction techniques (e.g., principal component analysis (PCA)) can be performed on the neuromuscular signals to generate multiple data points in the feature space. The PCA technique can be performed to compress the training dataset while retaining the data attributes useful for distinguishing two or more gestures and generating a set of effective gesture features.

[0106] In some embodiments, the number of principal components (e.g., linearly uncorrelated variables) used in the PCA analysis can be arbitrarily set to the number of gestures to be recognized by the system. This dimensionality reduction of the training dataset S can be represented as S=(x1,..,x m ), the feature mapping and the data matrix defined as (Φ(x1),...,Φ(x m )) where the i - th data point can be represented by x i =Φ(x1) or the i - th column of X, which is an N - dimensional vector. Dimensionality reduction techniques such as PCA can be performed, e.g., to find a k - dimensional representation of the data related to the original representation of X for k << N

[0107] PCA can be calculated by fixing k∈[1,N] and letting X be a data matrix centered at the mean (that is ), defining P k as a set of N - dimensional orthogonal projection matrices of rank k. Thereafter, the N - dimensional input data can be projected onto a k - dimensional linear subspace that minimizes the reconstruction error (e.g., the sum of the squared L2 distances between the original data and the projected data). Thus, PCA can be defined as the solution P* of the orthogonal projection matrix that minimizes the error value, which is calculated as a function of the Frobenius norm F given by:

[0108]

[0109] Fig.11The computed tangent space (at the top of the figure) and the computed PCA feature space (at the bottom of the figure) of the preprocessed training dataset are shown. In the tangent space, positive values ​​represent an increase in signal energy relative to the resting / neutral position and negative values ​​represent a decrease in signal energy relative to the resting / neutral position. The application of the PCA technique produces a low-dimensional representation of the dataset, as shown at the bottom of the figure (the signs are arbitrary in the PCA space). The PCA space shows the sequence of linear combinations of variables that have the largest variance and are uncorrelated with each other.

[0110] According to some aspects, the plurality of data points generated in a feature space (e.g., a PCA feature space) may be clustered by a clustering technique to create a classification representation of each performed gesture G1, G2, and G3. In some implementations, a K-means clustering technique may be used to partition the data points into K clusters based on a similarity metric. It should be understood that in embodiments where the classification model is trained to recognize a single gesture, clustering of the data points may not be performed because it may be assumed that all data points correspond to a single cluster associated with a single gesture.

[0111] In some embodiments, similarity between data points can be calculated by identifying the interrelationships between different data points that are interrelated, how the data points are similar or dissimilar to each other, and determining a measure of similarity or regularity (also referred to herein as a similarity measure) between the compared data points. K-means defines a prototype in terms of a centroid, which is the average of a set of points in a continuous n-dimensional space. A pseudo-code representation of the K-means clustering technique is provided below:

[0112] 1. Select K points as the initial centroids.

[0113] 2. Repetition

[0114] 3. Form k clusters by assigning all points to the nearest centroid.

[0115] 4. Recalculate the centroid of each cluster.

[0116] 5. Until the center of mass remains unchanged

[0117] The technique starts by selecting K initial centroids, where K is the number of desired clusters. Each data point is assigned to the nearest centroid, and each group of data points assigned to a centroid is considered a cluster. The centroids of the clusters may be updated several times based on the data points assigned to the clusters. Steps 3 (assignment) and 4 (update) are repeated until no data point changes cluster, that is, until the centroids remain unchanged.

[0118] In some embodiments, the similarity metric used to divide the EMG samples (data points) is based on the cosine distance between the data points. Cosine distance is an example of a suitable metric to distinguish EMG samples belonging to the same gesture, because each gesture can be characterized by a unique direction. Therefore, EMG samples or data points belonging to the same gesture show similar cosine distances between each other compared to data points belonging to different gestures.

[0119] Using cosine distance as a similarity metric, a K-means clustering technique is performed on the preprocessed training dataset, excluding data points corresponding to static or neutral positions / gestures. After performing the K-means clustering technique, the centroid of each cluster is stored or recorded to determine the orientation of the feature space corresponding to each gesture.

[0120] Clusters that share the same orientation in feature space can be efficiently identified, in part because they are represented as a single cluster. Thereafter, K orientations or vectors v can be determined k (For example, Fig.12 As shown, each gesture has a vector v k ).

[0121] v k , where k∈[1,K]

[0122] where K is the number of different gestures recognized during the training phase.

[0123] Therefore, for a given vector u of class y, class It can be determined by the following formula:

[0124]

[0125] force f (i.e., the force applied by the user during a given gesture) is given by:

[0126]

[0127] Fig.12An example of a result curve after applying the K-means clustering technique to a training data set on three PCA principal components associated with gestures G1, G2, and G3 is shown. As shown, different non-overlapping directions associated with each of the three clusters can be identified. Each gesture spans a unique direction in the feature space. If a gesture involves a group of muscles, uniformly increasing or decreasing the applied force while maintaining the gesture will increase or decrease the contraction of each muscle in the group of muscles involved. Therefore, each arrow represents a gesture vector that conveys the attributes of the identified gesture, including direction and force scale (e.g., the range of force values ​​that a user can apply when repeating a gesture or subsequently performing a gesture in the classification stage). The level of force applied when performing a particular gesture can be defined by the magnitude of the gesture vector in the determined direction. In some embodiments, the gesture vector associated with the gesture represents a classification representation of the gesture in the classification model.

[0128] Fig.13 The calculated forces associated with each gesture performed by the user during the training phase are shown. In some embodiments, for each of the three different gestures, the calculated forces may be expressed as a function of the maximum voluntary contraction (MVC).

[0129] Example technology for recognizing gestures based on a trained classification model - Gesture recognition phase

[0130] As described above, after calculating the direction and average magnitude corresponding to each gesture performed by the user during the training phase, a gesture classification can be effectively determined for the unseen gesture vectors. An unseen gesture vector is a vector derived from an unclassified user-defined gesture. As described below, when unseen gestures are related to gestures learned during the training phase, they can be classified, and properties of such unseen gesture vectors can be inferred, including the force applied by the user when performing the unseen gesture. In some embodiments, gesture vectors for unseen gestures can be generated in a manner similar to the generation of gesture vectors during the training phase.

[0131] Given an unseen gesture vector, a gesture can be inferred or classified based on a similarity measure (e.g., cosine of an angle) between the unseen gesture vector and a set of gesture vectors generated during a training phase of the system. Each gesture vector in the set of gesture vectors corresponds to a gesture learned by the model during a training phase. For example, a match between an unseen gesture vector and a learned gesture vector can be inferred by selecting a learned gesture vector from the set of learned gesture vectors that has a minimum cosine distance relative to the unseen gesture vector. The force of the unseen gesture vector can be inferred by calculating the relative magnitude of the unseen gesture vector projected onto the selected learned gesture vector (e.g., the learned gesture vector that shows the minimum cosine distance relative to the unseen gesture vector). Thus, the relative magnitude of the unseen gesture vector corresponds to the value of the scalar product between the unseen gesture vector and the selected learned gesture vector normalized by the magnitude of the selected learned gesture vector.

[0132] Therefore, in some cases, unseen gesture vectors computed from gestures performed by subsequent users (after the training phase) can be classified based on their cosine distance. Fig.14 The cosine distance between the learned gesture vector and the unseen gesture vector is shown. When the unseen gesture vector is correlated with the learned gesture vector, the cosine distance between the unseen gesture vector and the learned gesture vector is closer to zero, and when the unseen gesture vector is uncorrelated with the learned gesture vector, the cosine distance deviates from zero. For example, Fig.14 The cosine distances of three different unseen gesture vectors are shown. In this case, each of the three unseen gesture vectors matches a different learned gesture G1, G2, and G3.

[0133] In some other cases, the unseen gesture vector may be associated with a stationary or neutral position and may therefore be classified as a stationary or neutral gesture. The unseen gesture vector may be classified as a stationary or neutral gesture by determining whether the unseen gesture vector is associated with an EMG signal acquired when the user was in a stationary or neutral position during a training phase. For example, if the distance between the unseen gesture vector and the learned gesture vector is not less than a predetermined threshold (e.g., 0.25 radians), it may be inferred that the unseen gesture vector corresponds to a stationary or neutral position. Otherwise, the unseen gesture vector may be classified as the learned gesture vector that exhibits the smallest cosine distance relative to the unseen gesture vector.

[0134] As described above, the force associated with an unseen gesture vector may be inferred by calculating the relative magnitude of the unseen gesture vector relative to the learned gesture vector that was predicted to be associated with the unseen gesture vector. Fig.15 The inferred forces for each of the three recognized gesture vectors are shown.

[0135] Fig.16 The data mapping of the unseen gesture vectors on the PCA space is shown. It should be understood that Fig.12 The directions of the learned gesture vectors shown in Fig. 2 remain strong predictive features even though these features were extracted from a single movement of the gesture during the training phase.

[0136] Example use cases for wearable systems

[0137] Users can wear a wearable system with 16 neuromuscular sensors (e.g. Figure 2A ). A user can perform K gestures, where K can be equal to one or any other number of gestures. The user can perform gestures over a period of time, varying the force applied when performing each gesture. A classification model can be trained by an unsupervised machine learning approach to identify the gestures performed and generate a classification representation for each identified gesture. The classification representation can capture attributes associated with the identified gestures (e.g., gesture type, amount of force applied during performance of the gesture, etc.). The learned gestures and / or forces can be mapped to command / control signals such that each time the user performs the learned gesture, the system can generate a control / command signal. It should be understood that training and use are not limited to wearable systems including armbands, and the techniques described herein can be implemented with EMG sensors and / or other sensor arrangements other than armbands.

[0138] In some embodiments, the wearable system can be configured to generate one or more control signals when the user performs a user-defined gesture. Such control signals can activate / deactivate computing devices, control objects in virtual environments, control objects in augmented reality environments, control IoT devices, and other suitable systems and / or devices. In some cases, a single gesture can be configured to generate different signals based on the force applied by the user during the execution of the gesture. For example, a fist gesture performed by a user using a force below a threshold corresponding to half of the user's maximum voluntary contraction (MVC) can cause the system to generate a signal to activate a motor connected to a curtain or blind system, thereby causing the motor to close the curtain or blind. Similarly, if the user performs a fist gesture that applies a force greater than or equal to the aforementioned threshold, the system can generate a second signal configured to turn off the light in the room where the user is located. As another example, a gesture can be configured as a control signal so that the force applied during the execution of the gesture is proportional to the attribute of the signal (e.g., a voltage or other signal generated in an augmented or virtual reality environment). As yet another example, a gesture can be configured to control a dimmer or potentiometer to adjust the light intensity in the room so that it is proportional to the force applied by the user during the execution of the gesture.

[0139] The above embodiments may be implemented in any of a variety of ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code may be executed on any suitable processor or processor cluster (whether provided in a single computer or distributed between multiple computers). It should be appreciated that any component or component cluster that performs the above functions may be generally considered to be one or more controllers that control the functions discussed above. One or more controllers may be implemented in a variety of ways, such as using dedicated hardware or using one or more processors that are programmed to perform the functions listed above using microcode or software.

[0140] In this regard, it should be understood that one implementation of an embodiment of the present invention includes at least one non-transitory computer-readable storage medium (e.g., computer memory, portable storage, optical disk, etc.) encoded with a computer program (i.e., multiple instructions), which, when executed on a processor, performs the functions discussed above of an embodiment of the present invention. The computer-readable storage medium may be transportable so that the program stored thereon can be loaded onto any computer resource to implement the various aspects of the present invention discussed herein. In addition, it should be recognized that reference to a computer program that, when executed, performs the functions discussed above is not limited to an application program running on a host computer. Instead, the term "computer program" is used in a general sense herein to refer to any type of computer code (e.g., software or microcode) that can be used to program a processor to implement the various aspects of the present invention discussed above.

[0141] The various aspects of the present invention may be used alone, in combination, or in various arrangements not specifically discussed in the previously described embodiments, and therefore their application is not limited to the details and arrangements of the components set forth in the foregoing description or shown in the accompanying drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

[0142] In addition, embodiments of the present invention may be implemented as one or more methods, examples of which have been provided. The actions performed as part of the method may be ordered in any suitable manner. Accordingly, embodiments may be constructed in which the actions are performed in a different order than shown, which may include performing some actions simultaneously, even though shown as sequential actions in the illustrative embodiments.

[0143] The use of ordinal terms such as "first," "second," "third," etc. in the claims to modify claim elements does not, by itself, imply any priority, precedence, or order of one claim element relative to another claim element or the temporal order in which the actions of a method are performed. Such terms are merely used as labels to distinguish one claim element having a particular name from another element having the same name (but for the use of ordinal terms).

[0144] The phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” “having,” “containing,” “involving,” and variations thereof, is meant to encompass the items listed thereafter and additional items.

[0145] Having described several embodiments of the present invention in detail, it will be readily apparent to those skilled in the art that various modifications and improvements may be made. Such modifications and improvements are intended to be within the spirit and scope of the present invention. Therefore, the foregoing description is intended to be exemplary only and is not intended to be limiting. The present invention is limited only by the appended claims and their equivalents.

Claims

1. A device comprising: processor; a plurality of neuromuscular sensors coupled to the processor; and a memory storing instructions that, when executed by the processor, cause the processor to: receiving a first plurality of neuromuscular signals from a user via the plurality of neuromuscular sensors when the user performs a first single motion of a hand gesture while varying an amount of force applied during performance of the hand gesture; Training a classification model using training data generated based on the first plurality of neuromuscular signals, the training data comprising data indicative of changes in magnitude of force applied during performance of the gesture, wherein training the classification model comprises: deriving one or more values ​​from the first plurality of neuromuscular signals, the one or more values ​​being indicative of distinctive characteristics of the gesture; and generating a first classification representation of the gesture in the classification model based on the one or more values ​​derived from the first plurality of neuromuscular signals, wherein the first classification representation characterizes gestures performed using different magnitudes of force, and wherein, Different magnitudes of force detected from the neuromuscular signals are associated with different command signals that can be sent to one or more connected devices; receiving a second plurality of neuromuscular signals from the user via the plurality of neuromuscular sensors when the user performs a second single action of the gesture; and determining, based on the classification model and the second plurality of neuromuscular signals, that the user has performed the second single action of the gesture; Wherein, when executed by the processor, the instructions further cause the processor to: generating a first command signal to be transmitted to a device remote from the apparatus in response to determining that at least one value indicative of a force applied by the user during performance of a second single action of the gesture exceeds a predetermined force threshold; and In response to determining that at least one value indicating a force applied by the user during performance of a second single action of the gesture does not exceed the predetermined force threshold, generating a second command signal to be transmitted to the device, wherein the first command signal is different from the second command signal.

2. The device according to claim 1, wherein: Unsupervised machine learning techniques are used to generate a classification representation of the gesture in the classification model.

3. The device according to claim 1, wherein: The categorized representation of the gesture includes a force scale indicative of an amount of force applied during performance of the gesture.

4. The device according to claim 1, wherein: The classification model is trained in near real time.

5. The device according to claim 1, wherein: The plurality of neuromuscular sensors are arranged on a wearable device.

6. The device according to claim 1, wherein: When executed by the processor, the instructions further cause the processor to: At least one value indicative of a force applied by a user during performance of a second single action of the gesture is determined.

7. The device according to claim 1, wherein: The second single action of determining that the user has performed the gesture includes: deriving one or more second values ​​from the second plurality of neuromuscular signals; generating a second categorized representation of the gesture based on the derived one or more second values; and Based on a similarity measure calculated between the first categorized representation of the gesture and the second categorized representation of the gesture, it is determined that the user performed a second single action of the gesture.

8. The device according to claim 1, wherein: When executed by the processor, the instructions further cause the processor to: Based on the classified representation of the gesture, an estimate of a force applied by a user during performance of a second single action of the gesture is determined.

9. The device according to claim 1, wherein: When executed by the processor, the instructions further cause the processor to: calculating, based on the categorized representation of the gesture, a sequence of values, each value from the sequence of values ​​indicating a force applied by a user during performance of a second single action of the gesture, the sequence of values ​​indicating a change in the force applied by the user during performance of the second single action of the gesture; and At least one command signal is generated to be transmitted to the device, wherein the at least one command signal is indicative of a change in force.

10. The device according to claim 1, wherein: When executed by the processor, the instructions further cause the processor to: In response to determining that the user has performed a second single action of the gesture, a command signal is generated to be transmitted to a device remote from the apparatus, the command signal being configured to change the appearance of a graphical object displayed by the device.

11. The device according to claim 1, wherein: When executed by the processor, the instructions further cause the processor to: In response to determining that the user has performed a second single action of the gesture, a command signal is generated to be transmitted to a device remote from the apparatus, the command signal being configured to change a position of a graphical object displayed by the device.

12. The device according to claim 1, wherein: When executed by the processor, the instructions further cause the processor to: A command signal is generated to be transmitted to a device remote from the apparatus, wherein the command signal indicates a number of times the user has performed the gesture.

13. An apparatus comprising: processor; a plurality of neuromuscular sensors coupled to the processor; and a memory storing instructions that, when executed by the processor, cause the processor to: training a classification model using training data based on a first plurality of neuromuscular signals and a second plurality of neuromuscular signals received via the plurality of neuromuscular sensors; Wherein, when executed by the processor, the instructions further cause the processor to: receiving the first plurality of neuromuscular signals from the user via the plurality of neuromuscular sensors when the user performs a first single action of a first gesture; and receiving, via the plurality of neuromuscular sensors, the second plurality of neuromuscular signals from the user when the user performs a first single action of a second gesture; The training data includes data indicative of changes in the amount of force applied during performance of the gesture, wherein the training includes: Based on a clustering technique, deriving a first set of values ​​indicating distinctive characteristics of the first gesture and a second set of values ​​indicating distinctive characteristics of the second gesture; and generating in the classification model a classification representation of the first gesture and a classification representation of the second gesture, wherein the classification representations characterize gestures performed using different magnitudes of force, and wherein different magnitudes of force detected from the neuromuscular signals are associated with different command signals capable of being sent to one or more connected devices; and determining, based at least in part on a third plurality of neuromuscular signals and the classification model, whether a user has performed the first gesture or a follow-up to the second gesture; Wherein, when executed by the processor, the instructions further cause the processor to: generating a first command signal to be transmitted to the one or more connected devices remote from the apparatus in response to determining that at least one value indicative of a force applied by the user during performance of a second single action of the gesture exceeds a predetermined force threshold; and In response to determining that at least one value indicating the force applied by the user during performance of a second single action of the gesture does not exceed the predetermined force threshold, generating a second command signal to be transmitted to the one or more connected devices, wherein the first command signal is different from the second command signal.

14. The device according to claim 13, wherein: Unsupervised machine learning techniques are used to generate a classification representation of the first gesture and a classification representation of the second gesture.

15. The device according to claim 13, wherein: The classification model is trained in near real time.

16. The device according to claim 13, wherein: The plurality of neuromuscular sensors are arranged in a wearable device.

17. The device according to claim 13, wherein: The categorized representation of the first gesture includes a force scale indicating an amount of force applied during performance of the first gesture, and the categorized representation of the second gesture includes a force scale indicating an amount of force applied during performance of the second gesture.

18. The device according to claim 13, wherein: When executed by the processor, the instructions further cause the processor to: A value indicative of a force applied by a user during a subsequent action to perform the first gesture or a value indicative of a force applied by a user during a subsequent action to perform the second gesture is determined.

19. The device according to claim 13, wherein: When executed by the processor, the instructions further cause the processor to: A plurality of signals are generated to be transmitted to a device remote from the apparatus, each of the plurality of signals indicating a subsequent action of the first gesture, and each of the plurality of signals indicating a different force value calculated based on the subsequent action of the first gesture.

20. The device according to claim 13, wherein: When executed by the processor, the instructions further cause the processor to: A plurality of signals are generated to be transmitted to a device remote from the apparatus, each of the plurality of signals indicating a subsequent action of the second gesture, and each of the plurality of signals indicating a different force value calculated based on the subsequent action of the second gesture.

21. The device according to claim 13, wherein: The subsequent actions of determining whether the user has performed the first gesture or the second gesture include: deriving a third set of values ​​from the third plurality of neuromuscular signals; determining that a user performed a subsequent action of the first gesture based on a first similarity metric value calculated between the third set of values ​​and the classification representation of the first gesture; and Based on a second similarity metric value calculated between the third set of values ​​and the classification representation of the second gesture, it is determined that the user performed a subsequent action of the second gesture.

22. A method comprising: receiving, at a processor of the wearable device, a plurality of neuromuscular signals from a plurality of neuromuscular sensors included in the wearable device, the plurality of neuromuscular signals corresponding to neuromuscular signals sampled from a user when the user performs a first single motion of a gesture while varying an amount of force applied during performance of the gesture; training, via unsupervised machine learning techniques, a classification model using training data generated based on the first plurality of neuromuscular signals, the training data comprising data indicative of changes in magnitude of force applied during performance of the gesture, the classification model comprising a classification representation of the gesture, wherein a first classification representation characterizes gestures performed using different magnitudes of force, and wherein different magnitudes of force detected from the neuromuscular signals are associated with different command signals capable of being sent to one or more connected devices; Based on the classification representation of the gesture, determining whether the user has performed a subsequent single action of the gesture; determining at least one force value corresponding to a force applied by the user during performance of a subsequent single action of the gesture; and generating a command signal to be transmitted to a device in response to determining that the user has performed a subsequent single action of the gesture, wherein the command signal indicates the gesture and the at least one force value applied during performance of the gesture; The method further comprises storing instructions in a memory, which, when executed by the processor, cause the processor to: generating a first command signal to be transmitted to a device remote from the apparatus in response to determining that at least one value indicative of a force applied by the user during performance of a second single action of the gesture exceeds a predetermined force threshold; and In response to determining that at least one value indicating a force applied by the user during performance of a second single action of the gesture does not exceed the predetermined force threshold, generating a second command signal to be transmitted to the device, wherein the first command signal is different from the second command signal.

23. The method according to claim 22, wherein: The classification model is trained in near real time, and training the classification model via the unsupervised machine learning technique includes: A classification representation of the gesture is generated in the classification model based at least on the plurality of neuromuscular signals, wherein the classification representation includes a modeled force scale to determine one or more force values ​​applied during performance of individual actions of the gesture.

24. The method according to claim 22, wherein: The plurality of neuromuscular signals includes a first plurality of neuromuscular signals, the gesture includes a first gesture, and the method further includes: retraining the classification model via the unsupervised machine learning technique to determine whether the user performs a single action of a second gesture, wherein the first gesture is different from the second gesture, and the retraining comprises: generating in the classification model a classification representation of the second gesture based at least on a second plurality of neuromuscular signals, the classification representation of the second gesture including a modeled force scale to determine one or more force values ​​applied during performance of individual movements of the second gesture; and Based on the classification model, it is determined whether a third plurality of neuromuscular signals corresponds to a single motion of the first gesture or a single motion of the second gesture.

25. The method according to claim 22, wherein: The command signal causes the device to perform the operation using the at least one force value as a parameter of the operation.

Images