A cross-user myoelectric pattern recognition method that reduces user burden

Abstract

The application discloses a cross-user electromyography pattern recognition method for reducing user burden, which is characterized in that: a few calibration set electromyography sample data of source domain background users and target domain new users are collected, an electromyography gesture classification model FSSDA based on a convolutional neural network is constructed, a positive-negative pair distance loss is designed, a point proxy mode is adopted to optimize the feature distance of target domain new user samples and source domain background user samples in an embedding space, so that the target domain samples are distinguishable in the embedding space, and the trained cross-user model can predict gesture information of new users. The application can significantly reduce the collection and labeling time of calibration data of the model for new users, and simultaneously realize high-accuracy gesture classification performance, thereby promoting the popularization and application of a multi-user electromyography interface.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of signal processing, in particular to a cross-user electromyography pattern recognition method for reducing user burden. BACKGROUND

[0002] Electromyography (EMG) is a bioelectric signal that accompanies muscle contraction and can be used to represent the user's motion state or motion intention. Surface electromyography can be obtained through surface electrodes and reflects the activity intensity and flexion state of the moving joint. Hand motion recognition based on surface electromyography is a very practical technology that can realize the interaction between the user and the external device and help healthy people and disabled people complete daily activities.

[0003] Electromyography pattern recognition is considered a milestone in technological development and can control multiple degrees of freedom. Due to physiological and anatomical differences between different users, the electromyography interface trained by multiple users cannot directly adapt to the unique hand motion pattern of a new user. Most current work requires the new user to provide calibration data (tens to hundreds of samples) for each gesture once or multiple times, and some domain adaptation (DA) methods are used to calibrate the model, such as multi-stream adaptive batch normalization (MS-AdaBN) and adversarial training (AT) to obtain a better gesture recognition accuracy. These methods rely on calculating the distance and similarity between the distributions to learn the differences between the data distributions of the new user and multiple users, and it is difficult to learn these statistical information when there is only one data sample for each gesture. Therefore, the demand for a cross-user model that achieves high performance while having less user burden has not been met. SUMMARY

[0004] The present application is to solve the above-mentioned deficiencies in the prior art, and proposes a cross-user electromyography pattern recognition method for reducing user burden, in order to reduce the electromyography signal acquisition and annotation time of new users, while avoiding negative transfer, so as to realize high-accuracy cross-user gesture recognition.

[0005] To achieve the above-mentioned application purposes, the present application adopts the following technical solutions:

[0006] The cross-user electromyography pattern recognition method for reducing user burden has the characteristics that it includes the following steps:

[0007] Step 1: Collecting, by using a plurality of electromyography electrodes, a plurality of multi-channel electromyography signals generated when a plurality of different subjects perform a plurality of different actions, and performing sliding window, active segment segmentation and feature extraction on the plurality of multi-channel electromyography signals to obtain a training sample data set of a source domain background user wherein, represents the i-th training sample of the source domain, represents the i-th training sample of the source domain corresponding to the real label; n s represents the number of training samples in the training sample data set;

[0008] Collecting a calibration sample data set of a target domain new user wherein, represents the j-th calibration sample of the calibration sample data set D c represents the j-th calibration sample corresponding to the real label; n c represents the number of calibration samples in the calibration sample data set D c

[0009] Constructing a no-label sample data set of the target domain new user wherein, represents the k-th test sample of the no-label sample data set D t t represents the number of test samples in the no-label sample data set D t c < n s ;

[0010] Step 2: Constructing a double-flow electromyography gesture classification network, including two branches, wherein the first branch is composed of two convolutional blocks, an unfolding layer and two fully connected layers in sequence, and each convolutional block contains a convolutional layer, a batch normalization layer and a maximum pooling layer in sequence; the second branch is adding a fully connected layer as the last layer on the basis of the first branch, and the weights of the first branch and the second branch are shared;

[0011] Step 3: Inputting the training sample data set and the calibration sample data set into the double-flow electromyography gesture classification network respectively, the first branch processes any one training batch , and the second fully connected layer outputs the source domain features in the embedding space wherein, f(·) represents a mapping function from the input space to the embedding space; represents the i'-th training sample in the training batch D' s ​​​​​Represents the i′-th training sample The corresponding real label; n′ s The training batch D′ represents s The number of training samples in the dataset;

[0012] The second branch is for any calibration batch After processing, the second fully connected layer outputs the target domain features in the embedding space. in, Indicates the calibration batch D′ c The j′-th calibration sample in the dataset; Represents the j′-th calibration sample The corresponding real label; n′ c Indicates the calibration batch D′ c The number of calibration samples in the sample;

[0013] The last fully connected layer of the second branch After processing, the calibration sample dataset is obtained. Predicted probability of the corresponding action category in, Represents the j′-th calibration sample The predicted probability of the corresponding action category; and from Choose the category with the highest probability as Predicted labels

[0014] Step 4: Construct the positive-negative pair distance loss L using equation (1) P-N :

[0015]

[0016] In equation (1), [·] + =max(·,0), ||·|| F Representing the Frobenius norm, α denotes the marginal distance of separability between positive and negative pairs in the embedding space. Indicates the relationship with the j′-th calibration sample Different types of training batches D′ s The r-th training sample within, Indicates the relationship with the j′-th calibration sample Training batches D′ of the same category s The q-th training sample within, Represents the r-th training sample The corresponding real tags, Represents the q-th training sample The corresponding real tags;

[0017] Step 5: construct the classification loss L using formula (2) CE ;

[0018]

[0019] Step 6: construct the total loss L using formula (3)

[0020] L=L P-N +L CE (3)

[0021] Step 7: based on the training sample data set and the calibration sample data set use the Adam optimizer to train the double-flow myoelectric gesture classification network in batches, and calculate the total loss L, while updating the parameters of the network using the back propagation algorithm, optimizing the feature distance in the embedding space between the calibration sample data set of the new user in the target domain and the training sample data set of the background user in the source domain, narrowing the distance between each calibration sample and the training sample of the same category, and widening the distance between each calibration sample and the training sample of different categories, to obtain the best gesture classification model suitable for the new user;

[0022] Step 8: use the trained best gesture classification model to classify the unlabeled sample data set of the new user in the target domain, to obtain the category label of each test sample.

[0023] The electronic device comprises a memory and a processor, and the memory is used to store a program supporting the processor to execute the cross-user myoelectric pattern recognition method, and the processor is configured to execute the program stored in the memory.

[0024] The computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the cross-user myoelectric pattern recognition method.

[0025] Compared with the prior art, the beneficial effects of the present application are as follows:

[0026] 1、The present application introduces a supervised domain adaptation model FSSDA in a few-shot scenario, which calculates the distribution distance between the source domain and the target domain in a point proxy manner, effectively avoids the negative transfer caused by other domain adaptation methods (i.e. the knowledge learned by the source domain background user will have a negative effect on the prediction of the target domain new user) when a small amount of new user data is insufficient to accurately describe the target domain data distribution, and improves the accuracy of cross-user gesture recognition.

[0027] 2. This invention proposes a positive-negative pair distance loss, which uses a point proxy approach to optimize the feature distance between a small number of new user samples in the target domain and background user samples in the source domain in the embedding space. This loss brings the target domain sample closer to the same type of source domain sample and widens the distance between it and different types of source domain samples, making the target domain samples distinguishable in the embedding space. This avoids the inaccuracy of other domain adaptation methods that directly estimate the distribution of target domain data when there are too few new user samples, thereby achieving low-burden cross-user gesture recognition.

[0028] 3. The advantage of the method of the present invention is that, during the network training process, FSDA even allows a target domain sample to be paired with all source domain samples to learn the distribution relationship between features of different domain samples in the embedding space. It pays more attention to the class relationship between each scarce target domain sample and a large number of source domain samples. The constructed positive-negative pairs can simultaneously optimize the feature distance between each target domain sample point and other similar or dissimilar source domain sample points in each training batch, replacing the direct estimation of the target domain data distribution and achieving better cross-user gesture classification performance. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the electrode device of the present invention;

[0030] Figure 2 This is a schematic diagram of the gesture classification network of the present invention. Detailed Implementation

[0031] In this embodiment, a method for predicting cross-user gestures based on electromyography signals using a deep learning network utilizes only a small amount of calibration data from new users during model training, achieving high classification performance while significantly reducing the user burden. Specifically, the method includes the following steps:

[0032] Step 1: As Figure 1 As shown, multi-channel electromyographic (EMG) signals were collected from M different subjects performing N different actions using multiple EMG electrodes. After performing sliding windowing, active segment segmentation, and feature extraction on the multi-channel EMG signals, a training sample dataset of users with source domain background was obtained. in, This represents the i-th training sample in the source domain. Represents the i-th training sample in the source domain. The corresponding real label; n s This indicates the number of training samples in the training sample dataset;

[0033] Collect calibration sample datasets of new users in the target domain in, Represents the calibration sample dataset D c The j-th calibration sample, Represents the j-th calibration sample Corresponding real label; n c Represents the calibration sample dataset D c The number of calibration samples in the sample;

[0034] Construct an unlabeled sample dataset of new users in the target domain in, Represents the unlabeled sample dataset D t The k-th test sample; n t Represents the unlabeled sample dataset D t The number of test samples in the sample; where n c <<n s ;

[0035] In practice, standard sample data refers to electromyographic signals that have been calibrated for the corresponding hand gestures, and the number of these signals can be set according to the user's needs.

[0036] For example, the specific process for acquiring electromyographic (EMG) signals in this step can be as follows: EMG signals generated by M different subjects performing N different actions are used as standard sample data, where N and N are positive integers not less than 1. For example, M can be 9, N can be 6, and the 6 actions can include: index finger extension, middle finger extension, little finger extension, simultaneous extension of the index and middle fingers, simultaneous extension of all three fingers, and wrist extension. Each action is recorded 10 times, with each instance maintaining a moderate isometric contraction for 5 seconds. Sufficient rest time can be given to the subjects between movements, during which resting EMG signals can be collected. Each completed action is recorded as one trial. During this process, electromyography (EMG) signals are acquired using an electrode device fixed at the subject's sampling point (e.g., the forearm extensor muscles). Continuous EMG signals are acquired from M subjects performing N different actions consecutively. Continuous EMG signals refer to the uninterrupted EMG signals generated between completing one action and performing the next. After acquiring these continuous EMG signals, a preset length of EMG signals from M subjects in a resting state are acquired. Resting EMG signals refer to the EMG signals acquired by the electrode device when the user is not making any gestures. The continuous EMG signals are then segmented using the resting EMG signals to obtain segmented EMG sample data. This segmented EMG sample data represents the EMG signal corresponding to a specific gesture performed by the user. The segmented EMG sample data is used as the standard sample data.

[0037] Among them, the structural types of electrode devices are as follows Figure 1As shown, the electrode device is provided with a square distribution of electromyographic signal collection electrodes. When collecting electromyographic signals, the collection electrodes on the electrode device need to be in contact with the user's skin. The configuration of the electrode device can be configured according to user needs, and the flexible high-density electrode array has p rows, q columns, and a density of D, the values of p and q are both greater than 1, and the value of D is greater than 0. Specifically, p=10, q=10, and D=7mm.

[0038] In the specific process of obtaining the standard sample data provided in the present application, after collecting the electromyographic signals of a batch of gesture actions, the continuous electromyographic signal stream is divided into a certain number of analysis windows using the sliding window technology, the window length is N, and the sliding increment is L. Next, according to the resting state threshold Th, it is judged whether all analysis windows are active segments of muscle contraction. The active segment window is retained, and the resting segment window is removed. Then, three features are extracted from the electromyographic signals of each analysis window of the active segment, which are waveform length (Waveform length, WL), time-varying power spectrum descriptor f1, and time-varying power spectrum descriptor f6. These three features are labeled with the corresponding action task label and used as an electromyographic sample. Any one of the electromyographic samples is denoted as X∈R N×C , where N represents the number of extracted feature types, and C represents the number of channels of the electrode. In order to better reflect the spatial information of the muscle, the electromyographic sample is transformed into the form of an electromyographic image according to the electrode array arrangement, where C row and C col represent the height and width of the electromyographic image, respectively. The resting state threshold Th in the above process can be obtained by adding three times the standard deviation to the average value of all channel electromyographic signals collected in the resting state. N=256ms, L=128ms.

[0039] Step 2: as Figure 2 ​As shown, a dual-stream electromyography (EMG) gesture classification network is constructed, comprising two branches. The first branch consists of two convolutional blocks, one unfolded layer, and two fully connected layers. Each convolutional block contains a convolutional layer, a batch normalization layer, and a max-pooling layer. The convolutional layers extract spatial features of the EMG signals, the batch normalization layer accelerates network training and convergence while preventing overfitting, and the max-pooling layer reduces feature dimensionality. The network then passes through two fully connected layers with ReLU activation functions to obtain the representation of the source domain multi-user data in the embedding space. The second branch adds a fully connected layer as the final layer to the first branch, sharing weights with the first branch. The final fully connected layer uses SoftMax activation functions to obtain the probability values ​​of gestures corresponding to the target domain calibration sample data. The fully connected layer primarily transforms the dimensionality, while the SoftMax function adjusts the network output to probabilities, ensuring their sum equals 1 for easier classification.

[0040] Step 3: Transfer the training sample dataset and calibration sample dataset In the dual-stream electromyography gesture classification network, input is performed in batches. The first branch is used for any training batch. After processing, the source domain features in the embedding space are output by the second fully connected layer. Where f(·) represents the mapping function from the input space to the embedding space; Indicates training batch D′ s The i′th training sample in the dataset, Represents the i′-th training sample The corresponding real label; n′ s Indicates training batch D′ s The number of training samples in the dataset;

[0041] The second branch applies to any calibration batch. After processing, the second fully connected layer outputs the target domain features in the embedding space. in, Indicates calibration batch D′ c The j′-th calibration sample in the dataset; Represents the j′-th calibration sample The corresponding real label; n′ c Indicates calibration batch D′ c The number of calibration samples in the sample;

[0042] The last fully connected layer of the second branch is... After processing, the calibration sample dataset is obtained. Predicted probability of the corresponding action category in, Represents the j′-th calibration sample The predicted probability of the corresponding action category; and from Choose the category with the highest probability as Predicted labels

[0043] Step 4: Construct the positive-negative pair distance loss L using equation (1) P-N :

[0044]

[0045] In equation (1), [·] + =max(·,0), ||·|| F Representing the Frobenius norm, α denotes the marginal distance of separability between positive and negative pairs in the embedding space. Indicates the relationship with the j′-th calibration sample Different types of training batches D′ s The r-th training sample within, Indicates the relationship with the j′-th calibration sample Training batches D′ of the same category s The q-th training sample within, Represents the r-th training sample The corresponding real tags, Represents the q-th training sample The corresponding real tags;

[0046] Step 5: Construct the classification loss L using equation (2) CE ;

[0047]

[0048] Step 6: Construct the total loss L using equation (3):

[0049] L = L P-N +L CE (3)

[0050] Step 7: Based on the training sample dataset and calibration sample dataset The dual-stream electromyography gesture classification network is trained in batches using the Adam optimizer, and the total loss L is calculated. At the same time, the network parameters are updated using the backpropagation algorithm to optimize the feature distance between the calibration sample dataset of new users in the target domain and the training sample dataset of background users in the source domain in the embedding space. The distance between each calibration sample and the training samples of the same category is brought closer, and the distance between each calibration sample and the training samples of different categories is increased, so as to obtain the best gesture classification model suitable for new users.

[0051] Step 8: Using the trained optimal gesture classification model to classify the unlabeled sample dataset of the target domain new user Classification is performed to obtain the class label of each test sample.

[0052] In this embodiment, an electronic device includes a memory for storing a program supporting the processor to execute the above cross-user electromyographic pattern recognition method, and a processor configured to execute the program stored in the memory.

[0053] In this embodiment, a computer readable storage medium has a computer program stored thereon, which, when executed by a processor, performs the steps of the above cross-user electromyographic pattern recognition method.

[0054] In a specific implementation, the FSSDA model is compared with LDA, CNN, Finetune, CCA and CCA-OT methods. The average accuracy and standard deviation of 9 predicted individuals are as shown in Table 1:

[0055] Table 1. Classification performance of different methods

[0056]

[0057] In the context of providing only 5 samples for each gesture of each subject (1 / 8 of one test), FSSDA achieved the best classification performance compared to other cross-user methods.

Claims

1. A cross-user myoelectric pattern recognition method that reduces user burden, characterized by, Comprising the following steps: Step 1: a plurality of electromyography electrodes are used to collect M different subjects to perform N different actions to generate a plurality of multi-channel electromyography signals, and after the plurality of multi-channel electromyography signals are subjected to sliding window, activity segment segmentation and feature extraction, a training sample data set of a source domain background user is obtained Wherein, represents the i-th training sample of the source domain, represents the i-th training sample of the source domain The corresponding real label; n s represents the number of training samples in the training sample data set; Collecting calibration sample dataset of new users in target domain wherein, denotes a calibration sample dataset D c of jth calibration sample of the calibration sample dataset D denotes the jth calibration sample corresponding true label; n c denotes the number of calibration samples in the calibration sample dataset D c ​ constructing a unlabeled sample dataset of new users of the target domain wherein, denotes the unlabeled sample dataset D t the kth test sample; n t denotes the unlabeled sample dataset D t the number of test samples in D c wherein, n s ; Step 2: constructing a dual-stream myoelectric gesture classification network, containing two branches, wherein the first branch is sequentially composed of two convolutional blocks, an unfolding layer and two fully connected layers, and each convolutional block contains a convolutional layer, a batch normalization layer and a max-pooling layer in sequence; the second branch is adding a fully connected layer as the last layer on the basis of the first branch, and the weights of the first branch and the second branch are shared; Step 3: Transfer the training sample dataset and calibration sample dataset Inputting data into the dual-stream electromyography gesture classification network in batches, the first branch for any training batch... After processing, the source domain features in the embedding space are output by the second fully connected layer. Where f(·) represents the mapping function from the input space to the embedding space; The training batch D′ represents s The i′th training sample in the dataset. Represents the i′-th training sample The corresponding real label; n′ s The training batch D′ represents s The number of training samples in the dataset; The second branch processes the calibration batch After processing, the target domain features in the embedding space are output by the second fully connected layer Wherein, D' represents the calibration batch c The j'th calibration sample in the calibration batch D'; Y' represents the corresponding true label of the j'th calibration sample n' represents the number of calibration samples in the calibration batch D'; c D' represents the calibration batch c n' represents the number of calibration samples in the calibration batch D'; fully connected layer of the last layer of the second branch after processing, to obtain a calibration sample data set predicted probability of the corresponding action category wherein, denotes the j'th calibration sample predicted probability of the corresponding action category; and from select the category corresponding to the maximum probability as the predicted label of ​ Step 4: Constructing the positive-negative pair distance loss L using formula (1) P-N : In equation (1), [·] + =max(·,0), ||·|| F Representing the Frobenius norm, α denotes the marginal distance of separability between positive and negative pairs in the embedding space. Indicates the relationship with the j′-th calibration sample Different types of training batches D′ s The r-th training sample within, Indicates the relationship with the j′-th calibration sample Training batches D′ of the same category s The q-th training sample within, Represents the r-th training sample The corresponding real tags, Represents the q-th training sample The corresponding real tags; Step 5: Constructing the classification loss L using formula (2) CE ; Step 6: constructing the total loss L by using formula (3): L = L P-N + L CE (3) Step 7: based on the training sample data set and the calibration sample data set The double-flow myoelectric gesture classification network is trained in batches using an Adam optimizer, and the total loss L is calculated, while the parameters of the network are updated using a backpropagation algorithm, the optimization target is to reduce the feature distance in the embedding space between the calibration sample data set of the new user in the target domain and the training sample data set of the background user in the source domain, to reduce the distance between each calibration sample and the training sample of the same category, and to increase the distance between each calibration sample and the training sample of different categories, so as to obtain the best gesture classification model suitable for the new user. Step 8: Using the trained best gesture classification model to the target domain new user's unlabeled sample dataset Classification is performed to obtain the class label of each test sample.

2. An electronic device comprising a memory and a processor, characterized in that The memory is used for storing a program supporting the processor to execute the cross-user myoelectric pattern recognition method of claim 1, and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to perform the steps of the cross-user myoelectric pattern recognition method of claim 1.

Images