A method for training a gesture recognition model based on incremental learning

Abstract

The application provides a gesture recognition model training method based on incremental learning, which comprises the following steps: obtaining a trained old gesture recognition model, which comprises a feature extractor for extracting gesture features from electromyographic signal data; obtaining an initial new gesture recognition model, which comprises a feature extractor for extracting gesture features and a classifier for gesture recognition according to the gesture features; obtaining training samples in a sample pool of each old user in a plurality of old users and training samples of a new user, each training sample comprising electromyographic signal data and a label for indicating a gesture category to which the electromyographic signal data belongs; extracting old gesture features of the electromyographic signal data in each training sample in the training set based on the trained old gesture recognition model; and performing multiple iterative training on the initial new gesture recognition model by using the training set, and obtaining an incremental learning new gesture recognition model after the last training.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of incremental learning, gesture recognition and ubiquitous computing, and particularly, to a method for training a gesture recognition model based on incremental learning. BACKGROUND

[0002] Hand is the most complex part of human body, which plays an important role in completing various basic or complex behaviors and actions in daily life. In addition, there is a significant correlation between hand function and brain activity, and the hand has rich nerve endings that can transmit signals to the brain, and then the brain adjusts other body functions according to the received signals. Therefore, hand function rehabilitation has been widely studied in the field of rehabilitation, and the flexibility of the hand is measured to assess the cognitive and motor abilities of patients according to several cognitive assessment standards, including upper extremity function test (UEFT), Fugl-Meyer assessment (FMA), etc. Among various hand activity assessment methods, gesture recognition based on surface electromyography signal (sEMG) is a key technology for measuring human behavior ability and further perceiving human intention, and the surface electromyography signal is collected by a non-invasive method to reflect the activity state of human muscle, and with the continuous development of wearable devices, the cost of surface electromyography acquisition device is continuously reduced, and the sensitivity of electromyography signal acquisition is continuously improved. Therefore, in recent years, gesture recognition based on sEMG has attracted widespread attention and is widely used in many fields such as health assistance, virtual reality, smart home, interactive entertainment, etc.

[0003] sEMG gesture recognition can be defined as a machine learning problem, which is usually implemented by traditional machine learning methods or deep learning methods. Traditional machine learning methods usually use feature extractors and classifiers for gesture recognition, the feature extractor converts the original sEMG into a meaningful feature vector, such as time domain features, frequency domain features, etc., and uses support vector machines (SVM), decision trees (DT) and other classifiers to perform gesture recognition based on the feature vector and obtain effective classification performance. As early as 2008, some scholars used traditional machine learning methods to conduct experiments on sEMG gesture recognition and achieved high accuracy.

[0004] In recent years, with the continuous development of deep learning theory, methods based on deep learning have achieved good performance in sEMG gesture recognition, which is attributed to their strong feature learning ability, among which convolutional neural network (CNN) and recurrent neural network (RNN) are commonly used deep learning tools. In addition, there are patent application documents that provide a user-independent electromyography gesture recognition system based on adaptive learning (application number: CN202111376022), which contains data preprocessing, feature extraction and normalization modules, and uses a KNN classifier for adaptive training. There are also patent application documents that provide an online gesture recognition method for electromyography individual difference problems (application number: CN201810165350), which includes the establishment of a gesture electromyography data set, the training of an initialization classification model, and the clustering and screening of new samples.

[0005] However, in actual situations, sEMG gesture recognition models based on machine learning may lead to inconsistencies in data distribution between users due to electrode offset in data collection, changes in user limb position, and physiological state differences between users (muscle fatigue, skin resistance, muscle strength, etc.), thereby affecting recognition accuracy. And even if it is the same gesture type, the signals between different users will have significant differences, which will lead to catastrophic forgetting. In real interaction scenarios, when the model adapts to new user data (generally fine-tuning the original model with new user data), the gesture recognition accuracy of old users drops sharply, which is unacceptable. SUMMARY

[0006] Therefore, the purpose of the present application is to overcome the defects of the prior art and provide a method for training a gesture recognition model based on incremental learning.

[0007] The purpose of the present application is achieved by the following technical solutions:

[0008] According to a first aspect of the present application, a method for training a gesture recognition model based on incremental learning is provided, the method comprising the following steps: obtaining a trained old gesture recognition model comprising a feature extractor for extracting gesture features from electromyography signal data; obtaining an initial new gesture recognition model comprising a feature extractor for extracting gesture features from electromyography signal data and a classifier for recognizing gestures based on gesture features, wherein the parameters of the feature extractor in the initial new gesture recognition model are initialized by the parameters of the feature extractor in the trained old gesture recognition model; obtaining training samples in a sample pool of each old user in a plurality of old users and training samples of a new user to construct a training set, wherein the sample pool of each old user is a set of representative training samples of each gesture class of the old user, and each training sample comprises electromyography signal data and a label indicating the gesture class to which the electromyography signal data in the training sample belongs; extracting old gesture features of the electromyography signal data in each training sample in the training set based on the feature extractor of the trained old gesture recognition model; iteratively training the initial new gesture recognition model using the training set, and obtaining an incrementally learned new gesture recognition model after the last training, wherein each training comprises: inputting the electromyography signal data of each training sample into the feature extractor of the current new gesture recognition model to extract new gesture features, and determining the confidence of each new gesture feature in each gesture class using the classifier of the new gesture recognition model; and updating the parameters of the feature extractor and the classifier of the new gesture recognition model according to the classification loss determined based on the confidence of each new gesture feature in each gesture class and the corresponding label, and the maximum mean discrepancy loss determined based on all old gesture features and all new gesture features.

[0009] In some embodiments of the present application, the method further comprises: taking the new gesture recognition model obtained by the current incremental learning as the old gesture recognition model for the next incremental learning, and taking each new user in the current incremental learning as an old user for the next incremental learning; and creating a sample pool of the old user for the next incremental learning using the old gesture recognition model for the next incremental learning.

[0010] In some embodiments of the present application, the sample pool of each old user for the next incremental learning is created in the following manner: obtaining all training samples of the old user, extracting gesture features of each electromyography signal data in the all training samples using the old gesture recognition model; determining a feature center in each gesture class based on the gesture features of the each electromyography signal data and the gesture class to which the each electromyography signal data belongs; and selecting representative training samples in the gesture class according to the distance between the gesture features in the gesture class and the feature center in the gesture class.

[0011] In some embodiments of the present application, during training, a total loss is determined according to a weighted sum of the classification loss and the maximum mean discrepancy loss, and the parameters of the feature extractor and the classifier of the new gesture recognition model are updated by gradient descent according to the total loss.

[0012] In some embodiments of the present application, the total loss is determined in the following manner:

[0013]

[0014] wherein, represents the classification loss, represents the maximum mean discrepancy loss, and a represents a hyperparameter.

[0015] In some embodiments of the present application, the maximum mean discrepancy loss is calculated in the following manner:

[0016]

[0017] wherein, represents the i-th old gesture feature extracted by the old gesture recognition model, represents the i'-th old gesture feature extracted by the old gesture recognition model, represents the j-th new gesture feature extracted by the new gesture recognition model, represents the j'-th new gesture feature extracted by the new gesture recognition model, represents a kernel function calculating the inner product of old gesture features and in the feature space, represents a kernel function calculating the inner product of old gesture features and new gesture features in the feature space, represents a kernel function calculating the inner product of new gesture features and in the feature space, and m represents the total number of old feature vectors and the total number of new feature vectors.

[0018] In some embodiments of the present application, the set of labels includes finger actions, hand actions, wrist actions, grips, and functional actions or combinations thereof; or, the set of labels includes finger extension, finger flexion, wrist rotation, finger grip, and functional actions thereof or combinations thereof; or, the set of labels includes index finger flexion, index finger extension, middle finger flexion, middle finger extension, ring finger flexion, ring finger extension, little finger flexion, little finger extension, thumb abduction, thumb adduction, thumb flexion, thumb extension, thumb up, index and middle finger extension, other finger flexion, ring and little finger flexion, other finger extension, thumb to little finger base, all fingers spread, closed fist, index finger pointing, finger retraction, wrist supination, wrist pronation, wrist supination, wrist pronation, wrist flexion, wrist extension, wrist radial deviation, wrist ulnar deviation, wrist extension, hand tight grip, large diameter grip, small diameter grip, fixed hook grip, index finger extension grip, middle wrap, ring grip, prism four-finger grip, rocker grip, writing three-finger grip, full force ball grip, three-finger ball grip, precision sphere grip, tripod grip, prism pinch grip, fingertip pinch grip, four-legged grip, lateral pinch grip, parallel extension grip, extended grip, full force disc grip, gripping a bottle cap in a tripod grip and opening the water bottle, gripping a screwdriver in a rocker grip and turning the screw, gripping a knife in an index finger extension grip and cutting something, or combinations thereof.

[0019] According to a second aspect of the present application, a gesture recognition method is provided, the method comprising: T1, obtaining electromyographic signal data of a user; T2, performing gesture recognition on the electromyographic signal data of the user using a new gesture recognition model obtained by the method of the first aspect of the present application.

[0020] Compared with the prior art, the present application has the following advantages:

[0021] 1) The feature center-based sampling method measures the importance of data by calculating the distance between data features and feature centers to construct a sample pool for each old user data. This can better extract abstract features in old user data, while saving more representative data, so that the new gesture recognition model can learn the knowledge in the previous task well in the incremental learning process, improving the recognition ability of the new gesture recognition model, and thus improving the accuracy of gesture recognition.

[0022] 2) The present application saves the parameters of the old gesture recognition model, and initializes the parameters of the feature extractor of the new gesture recognition model using the parameters of the feature extractor of the old gesture recognition model at the beginning of a new round of incremental learning. This can consolidate the knowledge learned from old user data and integrate new user data, so that the new gesture recognition model can overcome catastrophic forgetting.

[0023] 3) In the training of the new gesture recognition model, the maximum mean difference loss calculated by the old gesture features extracted by the old gesture recognition model and the new gesture features extracted by the new gesture recognition model is used as a regularization term to constrain the new gesture recognition model, which solves the problem of excessive parameter deviation of the new gesture recognition model when learning new data, and further causes the recognition accuracy of the model to be reduced and the problem of catastrophic forgetting. BRIEF DESCRIPTION OF DRAWINGS

[0024] The embodiments of the present application will be further described below with reference to the drawings, in which:

[0025] Figure 1 A method for training a gesture recognition model based on incremental learning according to an embodiment of the present application.

[0026] Figure 2 A structure diagram of a feature extractor of a gesture recognition model according to an embodiment of the present application.

[0027] Figure 3 A gesture category diagram according to an embodiment of the present application.

[0028] Figure 4 A method flow diagram for training a gesture recognition model based on incremental learning according to an embodiment of the present application.

[0029] Figure 5 A method flow diagram for training a gesture recognition model based on incremental learning according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0031] As described in the background, in the prior art, good performance can be achieved in sEMG gesture recognition based on deep learning methods, but in actual situations, due to the electrode offset in data collection, the change of the user's limb position, and the physiological state difference (muscle fatigue, skin resistance, muscle strength, etc.) between users, the data distribution between users may be inconsistent, so as to fail to achieve high recognition accuracy, and even for the same gesture type, the signals between different users will have significant differences, which will also cause catastrophic forgetting. Moreover, in real interaction scenarios, when the model adapts to new user data, it may cause the problem of excessive parameter shift, so that the gesture recognition accuracy of the model for old users decreases sharply. Therefore, there is a problem that the sEMG gesture recognition model of the prior art has low accuracy in gesture recognition and the gesture recognition accuracy for old users decreases to an unacceptable level.

[0032] In order to solve the above problems, the inventors found that, during the research on electromyography gesture recognition, the electromyography signal has user dependence, and at the same time in the real world, the data of different users does not arrive at the same time. In view of this, as Figure 1As shown, the present application proposes a gesture recognition model training method based on incremental learning, which comprises: obtaining a trained old gesture recognition model, which comprises a feature extractor for extracting gesture features from electromyographic signal data; obtaining an initial new gesture recognition model, which comprises a feature extractor for extracting gesture features from electromyographic signal data and a classifier for gesture recognition according to gesture features, wherein the parameters of the feature extractor in the initial new gesture recognition model are initialized with the parameters of the feature extractor in the trained old gesture recognition model; obtaining training samples in a sample pool of each old user in a plurality of old users and training samples of a new user to construct a training set, wherein the sample pool of each old user is a set of representative training samples of each gesture category of the old user, and each training sample comprises electromyographic signal data and a label indicating the gesture category to which the electromyographic signal data in the training sample belongs; based on the feature extractor of the trained old gesture recognition model, old gesture features of the electromyographic signal data in each training sample in the training set are extracted respectively; based on the initial new gesture recognition model, the training set is used for multiple iteration training, and the last time the training is completed to obtain a new gesture recognition model after incremental learning. The gesture recognition model training scheme based on incremental learning of the present application not only can retain part of the representative old user data, so as to reduce the consumption of training data to space in the training process, but also can determine the maximum mean difference loss according to all old gesture features extracted by the old gesture recognition model and all new gesture features extracted by the new gesture recognition model to constrain the new gesture recognition model, solve the problem that the new gesture recognition model produces too large parameter deviation when learning new data, and further cause the recognition accuracy of the model to be reduced and the problem of catastrophic forgetting, so that the trained new gesture recognition model not only overcomes the problem of catastrophic forgetting, but also has good recognition ability to old user data and new user data.

[0033] The method of training gesture recognition model based on incremental learning aims to explore the effectiveness of incremental learning on sEMG data stream. represents the training set of the tth user, where represents the i th input data of the t th user, represents the label corresponding to the i th input data of the t th user. represents the test data of the t th user. represents the training sample sampled from the training set of the t th user. In our scenario, user data arrives at the model in the form of data stream over time. At time t, new user data arrives at the model, at this time the model is being trained, the model can see and training samples, where respectively represent the training samples sampled from the training set of the 1st old user, the 2nd old user, …, the t-1th old user, i.e., the old user sample pool; but the old user data and the test data is invisible. Therefore, the training set for training the model can be represented as The test set can be represented as The model is trained by using the training set composed of part of the old user data and the new user data, and after the model training is completed, the test set is used to verify the recognition ability and the recognition accuracy of the model, and according to the recognition ability and the recognition accuracy of the model, it can be known that the gesture recognition model obtained by the method of training the gesture recognition model based on the incremental learning has the recognition performance.

[0034] In order to better illustrate the technical solutions of the present application, the following will be described in detail from the aspects of model structure, training sample, training process and application scene in combination with specific embodiments.

[0035] I. Model structure

[0036] In order to enable the new gesture recognition model to be further trained based on the previously learned knowledge, according to an embodiment of the present application, in one incremental learning, the required model includes an old gesture recognition model and a new gesture recognition model, wherein the old gesture recognition model and the new gesture recognition model both include a feature extractor for extracting gesture features from electromyographic signal data and a classifier for gesture recognition according to gesture features. However, in each incremental learning, only the feature extractor for extracting gesture features from electromyographic signal data is needed for the old gesture recognition model, therefore, the trained old gesture recognition model obtained by the present application can only include (or only use) the feature extractor for extracting gesture features from electromyographic signal data. In addition, the trained old gesture recognition model is the new gesture recognition model obtained by incremental learning in the last incremental learning. And the parameters of the feature extractor in the new gesture recognition model are initialized by using the parameters of the feature extractor in the trained old gesture recognition model.

[0037] According to an embodiment of the present application, as Figure 2As shown, the feature extractor of the gesture recognition model all includes 3 layers of convolutional layers and 4 layers of fully connected layers. In the figure, the first three are convolutional layers, and the last four are fully connected layers (Full connect), which are sequentially connected to each other, and the output of the previous layer is the input of the next layer. Among them, each convolutional layer includes a convolution unit (Conv), a standardization unit (Norm1d), an activation unit (Relu) and a pooling unit (pool). Illustratively, when the electromyographic signal data is input into the feature extractor, first, the first layer of convolutional layer is processed, including using the convolution unit to perform convolution processing on the input electromyographic signal data to obtain first data features, using the standardization unit to standardize the first data features to obtain second data features (usually scaling and translation in the feature dimension to obtain more stable data features), using the activation unit to activate the second data features to obtain third data features (introducing a nonlinear relationship to the stable data features), and using the pooling unit to pool the third data features to obtain fourth data features; then, the fourth data features output by the first layer of convolutional layer are taken as the input of the second layer of convolutional layer, the fourth data features output by the second layer of convolutional layer are taken as the input of the third layer of convolutional layer, and the fourth data features output by the third layer of convolutional layer are taken as the input of the four layers of fully connected layers; after the processing of the four layers of fully connected layers, the gesture features are obtained.

[0038] According to an embodiment of the present application, the classifier can be implemented by using an existing multi-layer perceptron and a Softmax classification layer. Among them, the multi-layer perceptron is used to extract classification features from the gesture features, and the Softmax classification layer is used to determine the confidence of the electromyographic signal data corresponding to the gesture features in each gesture category. It should be understood that other implementation manners can also be used, such as a classification layer implemented based on a Sigmoid function.

[0039] Optionally, in some embodiments of the present application, existing CNN, RNN and LSTM models can be used as (new, old) gesture recognition models to implement the feature extractor and the classification recognition.

[0040] II. Training samples

[0041] Secondly, the technical solution of the present application is to train the feature extractor and the classifier of the new gesture recognition model, so before training, training samples need to be organized.

[0042] In order to enable the new gesture recognition model to obtain incremental learning according to the reserved part of the old user data, it is necessary to obtain the electromyographic signal data of the new user and the electromyographic signal data of the old user. According to an embodiment of the present application, training samples in a sample pool of each old user in a plurality of old users and training samples of a new user are obtained to construct a training set, wherein the sample pool of each old user is a set of representative training samples under each gesture category of the old user, and each training sample includes electromyographic signal data and a label indicating the gesture category to which the electromyographic signal data in the training sample belongs.

[0043] According to an embodiment of the present application, as Figure 3As shown, the set of labels includes finger actions, hand actions, wrist actions, grips, and functional actions or combinations thereof; or, the set of labels includes finger extension, finger flexion, wrist rotation, finger grip, and functional actions thereof or combinations thereof; or, the set of labels includes Index flexion, Index extension, Middle flexion, Middle extension, Ring flexion, Ring extension, Little finger flexion, Little finger extension, Thumb abduction, Thumb adduction, Thumb flexion, Thumb extension, Thumb up, Extension of index and middle, flexion of the others, Flexion of ring and little finger, extension of the others, Thumb opposing base of little finger, Abduction of all fingers, Fingers flexed together in fist, Pointing index, Adduction of extended fingers, Wrist supination (axis: middle finger), Wrist pronation (axis: middle finger), Wrist supination (axis: little finger), Wrist pronation (axis: little finger), Wrist flexion, Wrist extension, Wrist radial deviation, Wrist ulnar deviation, Wrist extension,Wrist extension with closed hand, Large diameter grasp, Small diameter grasp (power grip), Fixed hook grasp, Index finger extension grasp, Medium wrap, Ring grasp, Prismatic four fingers grasp, Stick grasp, Writing tripod grasp, Power sphere grasp, Three finger sphere grasp, Precision sphere grasp, Tripod grasp, Prismatic pinch grasp, Tip pinch grasp, Quadpod grasp, Lateral grasp, Parallel extension grasp, Extension type grasp, Power disk grasp, Open a bottle with a tripod grasp, Turn a screw (grasp the screwdriver with a stick grasp), Cut something (grasp the knife with an index finger extension grasp), or combinations thereof.

[0044] In addition, in actual scenarios, the training set can be extracted from an existing data set to train the feature extractor and the classifier of the new gesture recognition model. According to an embodiment of the present application, the existing data set can use the NinaPro DB-1 data set or the CapgMyo gesture recognition data set, and the existing data set is extracted into a plurality of sub-data sets according to the number of users (for example, all data of one user is a sub-data set), and the plurality of sub-data sets are used as the training set for multiple incremental training of the model; wherein the training set of each incremental training includes: a sub-data set in the previous incremental training (equivalent to an old user sample pool) and another sub-data set (equivalent to a new user training sample) and a label indicating the gesture category of the electromyographic signal data in the sub-data set.

[0045] III. Training process

[0046] Based on the training samples obtained in the second part, the training samples in the training set are used to perform an initial training and multiple incremental training on the gesture recognition model, and after multiple iterations of training, the last training is completed to obtain a new gesture recognition model after incremental learning.

[0047] According to an embodiment of the present application, when the first gesture recognition model is initially trained, the training set includes training samples composed of electromyographic signal data of different users and labels indicating the gesture category to which the training samples belong; the first gesture recognition model is initially trained using the training set, to obtain a trained old gesture recognition model for the first incremental training, and an old user sample pool for incremental training is constructed based on the trained old gesture recognition model. Wherein the electromyographic signal data of different users is obtained by a series of processing of the original electromyographic signal of different users, therefore, in order to make the electromyographic signal data of the user meet the input requirements of the gesture recognition model, it is necessary to first preprocess the collected original electromyographic signal of the user. According to an embodiment of the present application, the original electromyographic signal of the user is collected by using electromyographic sensors, line electromyography and ultrasonic electromyography, and the collected original electromyographic signal is processed by sliding window windowing and filter filtering to obtain preprocessed electromyographic signal data.

[0048] Illustratively, according to one example of the present application, an electromyography sensor is used to collect the original electromyography signal of a user, specifically: 10 electromyography sensors are deployed at different positions of the user's arm, when the user performs a gesture action, the 10 electromyography sensors simultaneously collect electromyography signals, and the 10 collected electromyography signals are spliced to form an electromyography signal data, the length of the electromyography signal data is 300 units; then the electromyography signal data is divided into windows by using a sliding window with a window size of 400 ms and a sliding distance of half the window size of 200 ms, after the window division is completed, the electromyography signal data after the window division is decomposed into data of different frequencies by using a Butterworth filter, and the noise in the electromyography signal data is removed by threshold processing to obtain the preprocessed electromyography signal data. It should be understood that the above is only one illustrative example, and a person skilled in the art can adjust the size and distance of the sliding window, such as adjusting the window size to 300 ms and the sliding distance to 150 ms, etc., to obtain other embodiments. A person skilled in the art can also select other filters for filtering, such as Chebyshev filter, elliptical filter, root mean square filter and adaptive filter, etc., to obtain other embodiments.

[0049] Secondly, since when a specific class is given and the class has a certain number of stored samples, the class with a larger number of samples contains more information, so that the model can more easily learn its complete distribution; in the case of different class classification, when the feature distribution of the majority class and the minority class is similar, the model tends to bias the classification result to the majority class, resulting in a decrease in the prediction accuracy when new data is input into the model. Therefore, in order to make the recognition accuracy of the new gesture recognition model obtained by the incremental learning higher, the preprocessed electromyography signal data also needs to be balanced in class. According to one embodiment of the present application, based on the preprocessed electromyography signal data, the number of electromyography signal data corresponding to different gesture classes is balanced by stratified sampling, and the number balancing includes: the number is equal to a certain extent or the number is different by a few (such as: ±5), or within a certain percentage range (such as: the number is within the range of 96% to 100%), etc. The technical scheme of this embodiment can at least achieve the following beneficial technical effects: by balancing the classes, the number of training samples in different gesture classes is equal, which can improve the decision-making ability of the final incremental learning new gesture recognition model for each gesture class, and then achieve high recognition accuracy.

[0050] In addition, in the prior art, when different users' EMG signal data are recognized by using a deep learning model, the results obtained by the deep learning model cannot explicitly show the features contained in the EMG signal data of different users, because the number of EMG signal data of different users is different. Therefore, in the present application, the number of EMG signal data of different users in the training set is made equal, so that the gesture recognition model learned according to the training samples of different users can well show the features of the EMG signal data of different users, and the recognition ability of the model is enhanced, and the model catastrophic forgetting is overcome.

[0051] Based on the above, the number of EMG signal data belonging to different gesture categories in the training samples of the same user is the same, and the number of training samples of different users is the same.

[0052] Based on the trained old gesture recognition model obtained by the above method for the first time for incremental training, and based on the trained old gesture recognition model, an old user sample pool for incremental training is constructed. In some embodiments of the present application, a previously trained old gesture recognition model can also be obtained to construct an old user sample pool for incremental training. According to an embodiment of the present application, the sample pool of each old user is created in the following way: all training samples of the old user are obtained, and the gesture features of each EMG signal data in all training samples are extracted by using the trained old gesture recognition model; based on the gesture features of each EMG signal data and the gesture category to which it belongs, the feature center under each gesture category is determined; and according to the distance between the gesture features of each gesture category and the feature center under the gesture category, the representative training sample under the gesture category is selected. Illustratively, according to an example of the present application, the gesture features of each EMG signal data in all training samples are extracted by using the new gesture recognition model obtained by the present incremental learning, and are represented as follows:

[0053] V y ←Φ(X i );

[0054] wherein Φ is the feature extractor of the trained old gesture recognition model, X i represents the i-th training sample, V y represents the extracted gesture features.

[0055] According to the gesture features of each electromyogram data in all extracted training samples and the gesture category to which the gesture features belong, the gesture features corresponding to each gesture category are added and averaged to obtain the feature center under each gesture category. Then, it is known that the gesture features corresponding to each gesture category have the same dimension as the feature center under each gesture category. Based on this, the distance between the gesture features of each gesture category and the feature center under the gesture category is calculated to select the representative training sample under the gesture category. The distance between the gesture features of each gesture category and the feature center under the gesture category can be calculated by the following formula:

[0056]

[0057] wherein μ i represents the feature center under the i-th gesture category, V i represents the gesture features of the i-th gesture category. It should be noted that in some embodiments, d(·) represents the Euclidean distance.

[0058] According to the above formula, the smaller the calculated distance is, the closer the gesture features corresponding to the gesture category are to the feature center under the gesture category, which means that the gesture features are representative and similar and can contain unique features in the gesture category. In some embodiments of the present application, based on the calculated distance between the gesture features of each gesture category and the feature center under the gesture category, the results are sorted in order from near to far, and based on the sorted results, the training samples corresponding to the gesture features corresponding to the first k (such as 10, 50, 100, 200 or 300, etc.) results are selected to construct the sample pool of each old user for incremental learning. It should be noted that the above method of constructing the sample pool of the old user can be called a sampling method based on the feature center (or Center of Feature, CoF). The technical solution of this embodiment can at least achieve the following beneficial technical effects: the method of constructing the sample pool of the old user data by calculating the distance between the features and the feature center can retain part of the representative old user data to ensure that the new gesture recognition model can learn knowledge from the old user data, and can help the model consolidate knowledge and overcome the problem of catastrophic forgetting in the prior art.

[0059] The above is the process of initial training, but the core of the present application is the subsequent multiple incremental training, which will be described in detail below. Before introducing the incremental training process, it needs to be explained that in each incremental training, the training set includes the initial training or the old user sample pool constructed after the last incremental training and the new user's electromyographic signal data, which needs to be windowed and filtered to obtain data that meets the input requirements of the gesture recognition model. At the same time, all samples of the new user obtained in each incremental training are processed for class balancing to obtain training samples of the new user, so that the number of electromyographic signal data corresponding to different gesture classes in the training set is balanced, the decision-making ability of the final incremental learning new gesture recognition model for each gesture class is improved, and then a high recognition accuracy is achieved. Therefore, the process of incremental training of the new gesture recognition model based on the obtained training set includes:

[0060] First, the training samples in the training set are input into the trained old gesture recognition model and the new gesture recognition model to extract the gesture features of the electromyographic signal data in each training sample. According to an embodiment of the present application, the training samples in the sample pool of each old user in the obtained multiple old users and the training samples of the new user are input into the trained old gesture recognition model to extract the old gesture features of the electromyographic signal data in each training sample;

[0061] The electromyographic signal data of each training sample is input into the feature extractor of the current new gesture recognition model to extract the new gesture features; the classifier of the new gesture recognition model is used to determine the confidence of each new gesture feature in each gesture class;

[0062] Secondly, according to the classification loss determined by the confidence of each new gesture feature in each gesture class and the corresponding label, and the maximum mean difference loss (MMD) determined by all old gesture features and all new gesture features, the parameters of the feature extractor and the classifier of the new gesture recognition model are updated. Wherein, the initial new gesture recognition model is the new gesture recognition model for the first incremental learning, and the old gesture recognition model in each subsequent incremental learning is the new gesture recognition model after completing the incremental learning.

[0063] When training the current new gesture recognition model, the total loss is determined according to the weighted sum of the determined classification loss and the maximum mean difference loss, and the parameters of the feature extractor and the classifier of the new gesture recognition model are updated according to the gradient backpropagation of the total loss. Wherein, the total loss is determined as follows:

[0064]

[0065] Wherein, represents the classification loss, represents the maximum mean discrepancy loss, and a represents a hyper-parameter.

[0066] The classification loss is determined in the following manner:

[0067]

[0068] where N represents the number of training samples, y i represents the label corresponding to the i-th training sample, represents the confidence of the i-th training sample in each gesture class determined by the classifier.

[0069] The maximum mean discrepancy loss is determined in the following manner:

[0070]

[0071] where, represents the i-th old gesture feature extracted by the old gesture recognition model, represents the i'-th old gesture feature extracted by the old gesture recognition model, represents the j-th new gesture feature extracted by the new gesture recognition model, represents the j'-th new gesture feature extracted by the new gesture recognition model, represents the kernel function of the inner product result of the old gesture feature and the kernel function of the inner product result in the feature space, represents the kernel function of the inner product result of the old gesture feature and the new gesture feature the kernel function of the inner product result in the feature space, represents the kernel function of the inner product result of the new gesture feature and the kernel function of the inner product result in the feature space, and m represents the total number of old feature vectors and the total number of new feature vectors.

[0072] More specifically, the maximum mean discrepancy loss can also be determined in the following manner:

[0073]

[0074] where V o represents the old gesture feature extracted by the old gesture recognition model, represents the i-th old gesture feature extracted by the old gesture recognition model, V n represents the new gesture feature extracted by the new gesture recognition model, represents the j-th new gesture feature extracted by the new gesture recognition model, represents the old gesture feature a mapping function mapping to a high-dimensional space, represents a mapping function mapping to a high-dimensional space. The technical scheme of this embodiment can at least achieve the following beneficial technical effects: the maximum mean discrepancy loss is used to constrain the difference between the old gesture features extracted by the old gesture recognition model and the new gesture features extracted by the new gesture recognition model, thereby constraining the new gesture recognition model, solving the problem of excessive parameter deviation of the new gesture recognition model when learning new data, ensuring the recognition ability of the trained new gesture recognition model, thereby improving the recognition accuracy of the model, and overcoming the problem of catastrophic forgetting of the model.

[0075] Based on the above, the new gesture recognition model of this incremental learning is obtained, and the new gesture model obtained by this incremental learning is used as the old gesture recognition model of the next incremental learning, and each new user in this incremental learning is classified as an old user of the next incremental learning; a sample pool of each old user of the next incremental learning is created using the old gesture recognition model of the next incremental learning. The sample pool of each old user of the next incremental learning is constructed in the manner described above according to the manner of constructing the sample pool of the old user of the first incremental learning after initial training.

[0076] In addition, according to an embodiment of the present application, the process of training the new gesture recognition model using the existing data set to extract the training set in the actual scene includes: first, using a sub-data set as the training set for initial training to train the gesture recognition model, obtaining an old gesture recognition model for incremental training, and using this sub-data set as the old user sample pool of the first incremental training. Each incremental training process includes: using the sub-data set in the initial training or the last incremental training as the old user sample pool and the next sub-data set relative to the last sub-data set as the new user training sample of this incremental training to extract new and old gesture features and the confidence of each new gesture feature in each gesture category, and updating the parameters of the feature extractor and the classifier of the new gesture recognition model according to the classification loss determined by the confidence of each new gesture feature in each gesture category and the corresponding label and the maximum mean discrepancy loss determined by all old gesture features and all new gesture features.

[0077] IV. Application scenarios

[0078] The new gesture recognition model obtained by incremental learning based on the model structure, training sample and training process described above performs gesture recognition on the electromyographic signal data of a user, including:

[0079] T1, obtaining raw electromyographic signals of a user, performing windowing and filtering processing on the raw electromyographic signals to obtain preprocessed electromyographic signal data of the user;

[0080] T2, the pre-processed user's electromyography signal data is recognized by the new gesture recognition model trained by the incremental learning. More specifically, the feature extractor of the new gesture recognition model trained by the incremental learning extracts gesture features of the electromyography signal data, and the classifier of the new gesture recognition model trained by the incremental learning recognizes the confidence of the electromyography signal data in each gesture category according to the gesture features.

[0081] In some embodiments of the present application, the classifier of the new gesture recognition model trained by the incremental learning recognizes the confidence of the electromyography signal data in each gesture category according to the gesture features, and outputs the gesture category to which the electromyography signal data belongs according to the obtained confidence.

[0082] In some other embodiments of the present application, the classifier of the new gesture recognition model trained by the incremental learning can simultaneously output the confidence of the electromyography signal data in each gesture category and the gesture category to which the electromyography signal data belongs according to the gesture features.

[0083] In order to better demonstrate the method of training a gesture recognition model based on incremental learning, the working process of the method will be described from two different angles below by combining the accompanying drawings. Figure 4 and accompanying Figure 5 .

[0084] As shown in Figure 4 , the method of training a gesture recognition model based on incremental learning has the following process:

[0085] Step S1, obtaining the original electromyography signal of a user;

[0086] According to some embodiments of the present application, the original electromyography signal of a user is collected by using an electromyography sensor, a line electromyography or an ultrasonic electromyography.

[0087] Step S2, pre-processing the original electromyography signal;

[0088] According to some embodiments of the present application, the original electromyography signal is subjected to sliding window and filter filtering.

[0089] Step S3, performing class balancing on the pre-processed electromyography signal data;

[0090] According to some embodiments of the present application, the pre-processed electromyography signal data is subjected to class balancing by means of hierarchical sampling, under-sampling and over-sampling, so that the number of electromyography signal data under different categories is the same.

[0091] Step S4, inputting the data subjected to class balancing into an old gesture recognition model to obtain old gesture features.

[0092] Step S5, input the data balanced by class into the new gesture recognition model to obtain new gesture features.

[0093] Step S6, input the new gesture features into the classifier of the new gesture recognition model to obtain a classification result.

[0094] Step S7, perform incremental learning on the new gesture recognition model.

[0095] According to some embodiments of the present application, the maximum mean discrepancy loss is calculated based on the old gesture features and the new gesture features, the classification loss is calculated based on the classification result and the gesture class corresponding to the electromyographic signal data, the total loss is determined according to the weighted sum of the calculated classification loss and the maximum mean discrepancy loss, and the parameters of the feature extractor and the classifier of the new gesture recognition model are updated according to the gradient backpropagation of the total loss, so as to obtain the new gesture recognition model after incremental learning.

[0096] Step S8, construct a sample pool for the next incremental learning.

[0097] According to some embodiments of the present application, the sample pool of the old user for the next incremental learning is constructed based on the sampling method of the feature center.

[0098] As shown in the figure, a method for training a gesture recognition model based on incremental learning has the following process (from left to right, from top to bottom): Figure 5

[0099] Block 1: Obtain raw electromyographic signals of different users and process them to form electromyographic signal time series diagrams.

[0100] As shown in the figure, different arms represent different users, wherein users 1 to T-1 represent old users, and user T represents a new user. Therefore, the first row of blue rectangles and the second row of green rectangles represent the processed electromyographic signal time series diagrams of the old users, and the third row of orange rectangles represents the processed electromyographic signal time series diagram of the new user.

[0101] According to an embodiment of the present application, the raw electromyographic signals can be processed by windowing and filtering.

[0102] Block 2: Perform class balancing processing on the electromyographic signal time series diagram.

[0103] Since the number of electromyographic signal time series diagrams (electromyographic signal data) in each class will affect the final recognition accuracy of the model, the number of corresponding electromyographic signal data under different gesture classes can be made equal by means of hierarchical sampling, under-sampling and over-sampling.

[0104] Block 3: Obtain the data balanced by class for sample replay.

[0105] ​In the figure, the feature center-based sampling method screens the old user data (blue and green), and the data with a red border is the screened old user data.

[0106] Block 4: input the sample pool of the old user and the data of the new user into the feature extractor of the old gesture recognition model to extract old gesture features;

[0107] In the figure, Φ t-1 represents the feature extractor of the old gesture recognition model, and the yellow box is the extracted old gesture feature.

[0108] Block 5: input the sample pool of the old user and the data of the new user into the feature extractor of the new gesture recognition model to extract new gesture features;

[0109] In the figure, Φ t represents the feature extractor of the new gesture recognition model, and the yellow box is the extracted new gesture feature.

[0110] Block 6: calculate the maximum mean difference loss based on the old gesture features and the new gesture features;

[0111] In the figure, Φ t-1 represents the old gesture feature, and Φ t represents the new gesture feature, and MMD is the maximum mean difference loss.

[0112] Block 7: calculate the classification loss based on the output classification result and the label;

[0113] The new gesture features extracted based on the new gesture recognition model pass through the classifier (f) to output the classification result (such as the green long box shown in the figure), which can be abstractly represented as y, and the classification loss is calculated based on the output classification result and the gesture category to which the electromyographic signal data belongs.

[0114] Finally, the total loss is determined by weighted summation based on the calculated classification loss and the maximum mean difference loss, and the parameters of the feature extractor and the classifier of the new gesture recognition model are updated according to the gradient backpropagation of the total loss, obtaining the incrementally learned new gesture recognition model.

[0115] In order to better show the technical effects of the technical scheme of the present application, the technical scheme of the present application will be compared with the existing method through experiments, wherein the existing method includes:

[0116] Finetune: This is a benchmark method, and the model only learns from new user data without reviewing previous user data.

[0117] Joint Train: The training data of the previous user is completely retained, and when the new user data arrives, it is trained together with the old user data as an upper bound of the method.

[0118] iCarl: This method removes the fully connected layer and uses the recent mean example algorithm to sample the samples. In the classification stage, the clustering method is used to classify the samples.

[0119] In this experiment, the data set used is the Ninapro DB-1 data set, which contains 52 gestures performed by 27 healthy subjects, which are divided into 12 basic finger actions, 8 basic hand actions, 9 basic wrist actions and 23 holding and functional actions, and each action is repeated by the subject 10 times, and collected by 10 electrode channels at a sampling frequency of 100Hz, and the first-order Butterworth low-pass filter is used to process the data in the data set. The processed data set is used for gesture recognition by Fine-tune, Joint Train, iCarl and IncreEGR (the method of the present application), and the gesture recognition accuracy results are shown in Table 1:

[0120] Table 1 Comparison of experimental results

[0121]

[0122] Based on Table 1, it can be concluded that the scheme of the present application has a gesture recognition accuracy of 66.70% for 27 subjects in the Ninapro DB-1 data set. The Fine-tune method does not use any incremental learning method, so its accuracy is low. Compared with the Fine-tune method, the scheme of the present application improves the accuracy by more than 53%. Although compared with the JointTrain method, the scheme of the present application reduces the accuracy by about 2.8%, but Joint Train retains and trains all user data, which may consume more space because of retaining all original data, resulting in excessive cost. The present application only retains a small amount of original data, and also achieves a high accuracy, and does not cause a large amount of space consumption and a large cost in the use process. Therefore, according to the comparison from multiple aspects, the scheme of the present application is better than the JointTrain method. Compared with the iCarl method, although the accuracy of the scheme of the present application is slightly lower than that of the iCarl method when identifying the data of 10 subjects, but as the number of subjects increases, the accuracy of the scheme of the present application is higher and higher, and when the number of subjects reaches 27, the accuracy of the scheme of the present application is improved by 1.8% compared with the accuracy of the iCarl method, which shows that the scheme of the present application has an advantage in overcoming long-term forgetting, and can be well applied to the scene of infinite data flow in practice.

[0123] According to one embodiment of the present application, the NinaPro DB-1 dataset can also be used as a test set for verifying the new gesture recognition model after completing incremental learning. According to the recognition accuracy obtained by the above method, it can be known that the gesture recognition model obtained based on the scheme of the present application has good recognition performance.

[0124] In order to further explore the contribution of the feature center-based sampling method to the technical effect of retaining part of the old user data to construct a sample pool and using the maximum mean difference loss to constrain the model in the technical scheme of the present application, the inventors conducted an ablation experiment, and the experimental results are shown in Table 2.

[0125] Table 2 Ablation experiment

[0126]

[0127] In Table 2, CoF represents the feature center-based sampling method, and MMD represents the maximum mean difference loss. Based on Table 2, it can be obtained that the technical scheme of the present application can enable the model after incremental learning to achieve good long-term memory in the actual scenario of continuously increasing the number of users and overcome catastrophic forgetting by constructing a sample pool and using the maximum mean difference loss to constrain the model, which verifies the effectiveness of the technical scheme of the present application and achieves high accuracy.

[0128] According to some embodiments of the present application, a CapgMyo gesture recognition dataset can also be used for comparative experiments.

[0129] In summary, the method for training a gesture recognition model based on incremental learning proposed by the present application can:

[0130] 1) The feature center-based sampling method measures the importance of data by calculating the distance between data features and feature centers to construct a sample pool for each old user data. This can better extract abstract features from old user data and save more representative data, so that the new gesture recognition model can learn the knowledge in the previous task well during the incremental learning process, improve the recognition ability of the new gesture recognition model, and thus improve the accuracy of gesture recognition.

[0131] 2) The present application saves the parameters of the old gesture recognition model and initializes the parameters of the feature extractor of the new gesture recognition model using the parameters of the feature extractor of the old gesture recognition model at the beginning of the new round of incremental learning. This can consolidate the knowledge learned from the old user data and integrate the new user data, so that the new gesture recognition model can overcome catastrophic forgetting.

[0132] 3) In the training of the new gesture recognition model, the maximum mean difference loss calculated by using the old gesture features extracted by the old gesture recognition model and the new gesture features extracted by the new gesture recognition model is used as a regularization term to constrain the new gesture recognition model, which solves the problem that the parameter deviation of the new gesture recognition model is too large when learning new data, and further causes the recognition accuracy of the model to be reduced and the problem of catastrophic forgetting.

[0133] It should be noted that although the above describes each step in a specific order, it does not mean that each step must be performed in the above specific order, in fact, some of the steps can be performed concurrently, or even change the order, as long as the required function can be realized.

[0134] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0135] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a magneto-optical or other optical device, a portable storage device, an memory card, a solid-state memory device, or any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves.

[0136] Embodiments of the application have been described above, with the understanding that these embodiments are exemplary only, and are not restrictive, and are not limited to the disclosed embodiments. Many modifications and variations to the disclosed embodiments will be apparent to those skilled in the art, without departing from the scope and spirit of the described embodiments. The choice of words in this document is intended to best explain the principles of the embodiments, practical application, or improvement to the art in the market, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A method for training a gesture recognition model based on incremental learning, characterized in that: The method comprises the following steps: obtaining a trained legacy gesture recognition model comprising a feature extractor for extracting gesture features from electromyographic signal data; Obtaining an initial new gesture recognition model, which includes a feature extractor for extracting gesture features from electromyographic signal data and a classifier for performing gesture recognition based on the gesture features, wherein parameters of the feature extractor in the initial new gesture recognition model are initialized using parameters of the feature extractor in the trained old gesture recognition model; Obtaining training samples from a sample pool of each of the multiple old users and training samples of the new user to construct a training set, wherein the sample pool of each old user is a set of representative training samples under each gesture category of the old user, and each training sample includes electromyographic signal data and a label indicating the gesture category to which the electromyographic signal data in the training sample belongs; Based on the feature extractor of the trained old gesture recognition model, old gesture features of the electromyographic signal data of each training sample in the training set are respectively extracted; Based on the initial new gesture recognition model, the model is iteratively trained using the training set for multiple times. After the last training, a new gesture recognition model with incremental learning is obtained. Each training includes: Input the electromyographic signal data of each training sample into the feature extractor of the current new gesture recognition model to extract new gesture features, and use the classifier of the new gesture recognition model to determine the confidence of each new gesture feature in each gesture category; updating the parameters of the feature extractor and classifier of the new gesture recognition model according to the classification loss determined by the confidence of each new gesture feature in each gesture category and the corresponding label and the maximum mean difference loss determined according to all old gesture features and all new gesture features; The new gesture recognition model obtained in this incremental learning is used as the old gesture recognition model for the next incremental learning, and each new user in this incremental learning is classified as an old user in the next incremental learning. The old gesture recognition model of the next incremental learning is used to create a sample pool of old users for the next incremental learning.

2. The method according to claim 1, wherein The sample pool for each old user in the next incremental learning is created as follows: Obtain all training samples of the old user, and use the old gesture recognition model to extract gesture features of each electromyographic signal data in all training samples; Determining a feature center for each gesture category based on the gesture features of each electromyographic signal data and the gesture category to which it belongs; According to the distance between the gesture feature of each gesture category and the feature center of the gesture category, representative training samples of the gesture category are selected.

3. The method according to claim 1, wherein During training, the total loss is determined according to the weighted sum of the classification loss and the maximum mean difference loss, and the gradient is calculated according to the total loss and back-propagation is performed to update the parameters of the feature extractor and classifier of the new gesture recognition model.

4. The method according to claim 3, wherein The total loss is determined as follows: ； in, represents the classification loss, It represents the maximum mean difference loss, represents the hyperparameters.

5. The method according to claim 4, characterized in that The maximum mean difference loss is calculated as follows: ； in, It represents the first Gejiu gesture characteristics, It represents the first Gejiu gesture characteristics, It represents the first New gesture features, It represents the first New gesture features, Indicates the calculation of old gesture features and The kernel function of the inner product result in the feature space, Indicates the calculation of old gesture features and new gesture features The kernel function of the inner product result in the feature space, Indicates the calculation of new gesture features and The kernel function of the inner product result in the feature space, It represents the total number of old eigenvectors and the total number of new eigenvectors.

6. The method according to any one of claims 1 to 5, characterized in that The set of tags includes finger movements, hand movements, wrist movements, grips and functional movements or a combination thereof.

7. The method according to any one of claims 1 to 5, characterized in that The set of labels includes finger extension, finger flexion, wrist rotation, finger grasping and their functional movements or their combination.

8. The method according to any one of claims 1 to 5, characterized in that The collection of tags includes index finger flexed, index finger extended, middle finger flexed, middle finger extended, ring finger flexed, ring finger extended, pinky flexed, pinky extended, thumb abducted, thumb adducted, thumb flexed, thumb extended, thumb raised, index and middle finger extended, other fingers flexed, ring and pinky flexed, other fingers extended, thumb to pinky base, all fingers extended, closed fist, index finger pointing, finger adducted, wrist supinated, wrist pronated, wrist supinated, wrist pronated, wrist flexed, wrist extended, radial deviation of wrist, ulnar deviation of wrist, wrist extension Spread, hand grip, large diameter grip, small diameter grip, fixed hook grip, extended index finger grip, middle wrap, ring grip, prismatic four-finger grip, joystick grip, writing three-finger grip, full force grip, three-finger ball grip, precision sphere grip, tripod grip, prismatic pinch grip, fingertip pinch grip, quadripod grip, lateral pinch grip, parallel extension grip, extended grip, full force grip, grasping a bottle cap with a tripod grip to open a water bottle, grasping a screwdriver with a joystick grip to turn a screw, grasping a knife with an extended index finger grip to cut something, or a combination thereof.

9. A gesture recognition method, characterized in that: The method comprises: T1. Obtain the user's electromyographic signal data; T2. Perform gesture recognition on the user's electromyographic signal data using a new gesture recognition model obtained through incremental learning using the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of any one of the methods of claims 1 to 9.

11. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the steps of the method according to any one of claims 1 to 9.

Images