A method and device for myoelectric signal gesture recognition based on deep learning

Abstract

The present invention discloses a method and device for electromyographic signal gesture recognition based on deep learning, the method comprising determining a corresponding feature image according to an electromyographic signal and inputting the feature image into a trained classification model for classification, the classification model comprising an LSTM layer and a convolutional network layer, the LSTM layer comprising two hidden layers, the convolutional network layer comprising five convolutional layers, batch normalization layers and activation functions are used for processing between each convolutional layer, a Dropout layer is used after the third convolutional layer to alleviate overfitting, a hybrid attention mechanism module is used after the first and fourth convolutional layers, and an adaptive maximum pooling layer is used before the fifth convolutional layer. The present invention uses an attention mechanism method to strengthen the prominent information in the electromyographic signal extraction process, especially the significant information in different channels and spaces, so that the network layer is more sensitive to such data, which is conducive to the training and recognition of electromyographic signals that produce larger actions.

Description

Technical Field

[0001] The present invention relates to the technical field of electromyographic signal recognition and processing, and in particular to a method and device for electromyographic signal gesture recognition based on deep learning. Background Art

[0002] Surface electromyography (sEMG) is a bioelectric signal stimulus that can reflect the degree of muscle excitement during biological movement. It is currently widely used in human-computer interaction, prosthetic control, rehabilitation training, etc. Especially in the field of gesture recognition and prosthetic control, as a reliable signal source, compared with traditional digital images, IMU sensor data, data gloves and other signal acquisition methods, electromyography has the advantages of rapid response (generally 30-150ms ahead of human consciousness) and anti-interference.

[0003] The research on gesture recognition methods based on electromyographic signals is mainly divided into two types according to the recognition algorithm: traditional methods and deep learning methods. Among the traditional gesture recognition methods, commonly used are feature recognition methods such as Hidden Markov Model (HMM), K-nearest neighbor (KNN), support vector machine (SVM), random forest (RF), linear discriminant analysis (LDA), etc. This type of method processes different time-frequency domain features extracted from the signal. The recognition algorithm of this method is complex, the deep precision is low, and it is not possible to well explore the potential information of data and gesture movements. The second type of gesture recognition method based on deep learning has become a research hotspot in the field of gesture recognition with the development of artificial intelligence in the 21st century, especially the emergence of large data sets in the information age. Since 1993, neural networks have been used to realize gesture classification from surface electromyographic signal records. In earlier studies, the comparison of classification using CNN exceeded the traditional support vector machine recognition method in terms of accuracy. However, most of the research on this type of method uses complex signal preprocessing methods to find different features of the data as input signal classification, which greatly increases the cost of calculation and the difficulty of software maintenance. It reduces the real-time performance of signal recognition, which makes it difficult to apply gesture recognition methods in real-time applications.

[0004] Therefore, providing a fast, low-computational cost and high-recognition accuracy gesture recognition method is of great significance for the application of gesture recognition in real human-computer interaction. How to balance recognition accuracy and processing cost is a major problem that researchers are currently eager to solve and is also the biggest obstacle to the application of gesture recognition methods in real human-computer interaction. Summary of the invention

[0005] The present invention provides a method and device for myoelectric signal gesture recognition based on deep learning to overcome the problems of large data processing delay and low recognition accuracy in the recognition process of existing gesture recognition technology. It aims to use artificial intelligence technology to achieve high accuracy and robustness in myoelectric signal gesture recognition, while shortening data processing time.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] According to the first technical solution of the present invention, a method for myoelectric signal gesture recognition based on deep learning is provided, the method comprising: determining a corresponding feature image according to the myoelectric signal; inputting the feature image into a trained classification model for classification, the classification model comprising an LSTM layer and a convolutional network layer, the LSTM layer comprising two hidden layers, the convolutional network layer comprising five convolutional layers, batch normalization layers and activation functions are used for processing between each convolutional layer, a Dropout layer is used after the third convolutional layer to alleviate overfitting, a hybrid attention mechanism module is used after the first and fourth convolutional layers, and an adaptive maximum pooling layer is used before the fifth convolutional layer.

[0008] According to the second technical solution of the present invention, a deep learning-based electromyographic signal gesture recognition device is provided, the device comprising a processing unit and a classification model; the classification model comprises an LSTM layer and a convolutional network layer, the LSTM layer comprises two hidden layers, the convolutional network layer comprises five convolutional layers, a batch normalization layer and an activation function are used between each convolutional layer for processing, a Dropout layer is used after the third convolutional layer to alleviate overfitting, a hybrid attention mechanism module is used after the first and fourth convolutional layers, and an adaptive maximum pooling layer is used before the fifth convolutional layer; the processing unit is configured to determine a corresponding feature image according to the electromyographic signal, and output the corresponding feature image to the classification model; the classification model is configured to output a classification result according to the feature image.

[0009] Compared with the prior art, the present invention has the following beneficial effects:

[0010] (1) The data processing process is simplified through simple data processing methods, which reduces the data processing time. The data enhancement method effectively simulates the data jitter and noise interference that may occur during real-time data collection, which is conducive to improving the accuracy of training.

[0011] (2) The use of a combined network model effectively improves the accuracy of EMG gesture recognition. The attention mechanism is used to enhance the prominent information in the EMG signal extraction process, especially the significant information in different channels and spaces, making the network layer more sensitive to such data, which is conducive to the training and recognition of EMG signals that produce larger movements.

[0012] (3) Short-term window division greatly reduces the amount of data and saves training costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A flowchart of a method for myoelectric signal gesture recognition based on deep learning according to an embodiment of the present invention is shown.

[0014] Figure 2 A flowchart of a method for myoelectric signal gesture recognition based on deep learning according to an embodiment of the present invention is shown.

[0015] Figure 3 A window sliding diagram on an electromyographic signal according to an embodiment of the present invention is shown.

[0016] Figure 4 A schematic diagram of stationary wavelet packet decomposition according to an embodiment of the present invention is shown.

[0017] Figure 5 A schematic diagram of a stationary wavelet packet decomposition feature image according to an embodiment of the present invention is shown.

[0018] Figure 6 A classification model training flow chart of a method for myoelectric signal gesture recognition based on deep learning according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0019] The following examples are only for better illustrating the present invention, but the content of the present invention is not limited to the application in the examples. Therefore, those skilled in the art can make non-essential improvements and adjustments to the implementation scheme according to the above content of the invention and apply them to other examples, which are still within the protection scope of the present invention.

[0020] The present invention will now be further described with reference to the accompanying drawings.

[0021] Figure 1 The flowchart of a method for recognizing gestures using electromyographic signals based on deep learning according to an embodiment of the present invention is shown. The embodiment of the present invention provides a method for recognizing gestures using electromyographic signals based on deep learning. Figure 1 As shown, the method starts at step S100, and a corresponding characteristic image is determined according to the electromyographic signal.

[0022] Finally, in step S200, the feature image is input into a trained classification model for classification, wherein the classification model includes an LSTM layer and a convolutional network layer, wherein the LSTM layer includes two hidden layers, and the convolutional network layer includes five convolutional layers, wherein a batch normalization layer and an activation function are used between each convolutional layer for processing, and a Dropout layer is used after the third convolutional layer to alleviate overfitting, and a hybrid attention mechanism module is used after the first and fourth convolutional layers, and an adaptive maximum pooling layer is used before the fifth convolutional layer. The electromyographic signal is identified after classification by the classification model. The present invention uses deep learning technology to avoid artificially constructed features, and can be easily extended to other similar tasks in practical applications.

[0023] When using the classification model for classification, the feature map is input into the model for classification. The classification stage uses a combined network of LSTM-CNN. The network mainly uses the LSTM layer and the Conv2 convolutional network layer for combination. The batch normalization (BatchNorm) layer and the activation function Relu are used between each layer of the convolutional network for processing. The LSTM layer uses two hidden layers, and the output of each hidden layer is 64. After the LSTM output is spliced, the three-channel data enters the convolutional layer. The number of convolutions in the first, second, third, fourth, and fifth layers of the convolutional network layer is 64, 64, 128, 64, and 52 respectively. The convolution kernels of the first four layers are all 3x3, and the convolution kernel of the fifth convolution layer is 1x1. A Dropout layer with a value of 0.3 is used after the third layer to alleviate overfitting, and a hybrid attention mechanism module CBAM is used after the first and fourth convolutional layers. An adaptive maximum pooling layer with an output size of 1 is used before the fifth layer.

[0024] It should be noted that the LSTM layer is a long short-term memory network (LSTM) layer, which is a time recurrent neural network. The use of the LSTM network can well extract the hidden timing information of the electromyographic signal, which is conducive to the classification of characteristic signals such as electromyographic signals that have both time and frequency domains.

[0025] The convolutional network layer is a convolutional neural network. The convolutional neural network is used to classify signals, which has good generalization. Since the electromyographic feature image is expressed as three-dimensional data with channels, length, width and height, the use of a two-dimensional convolutional network can better explore its intrinsic information than a one-dimensional convolutional network. The embodiment of the present invention uses a hybrid attention mechanism module (CBAM) after the first and fourth convolutional layers. That is, the spatial and channel correlation of the electromyographic signal is processed using the hybrid attention mechanism, which effectively improves the recognition accuracy of the data. And CBAM is a lightweight network module with very few additional parameters.

[0026] Therefore, when the method provided by the embodiment of the present invention is used to perform gesture recognition on electromyographic signals, the use process is relatively simple, the maintenance and transplantation of the code are convenient, and the development of online systems has higher availability and robustness.

[0027] In some embodiments, Figure 2 FIG. 1 is a flowchart of a method for myoelectric signal gesture recognition based on deep learning according to an embodiment of the present invention. Figure 2 As shown, the method of determining the corresponding characteristic image according to the electromyographic signal includes the following steps:

[0028] Step S101, low-pass filtering the electromyographic signal. Exemplarily, a first-order 1 Hz high-pass Butterworth filter is used to low-pass filter the electromyographic signal.

[0029] Step S102, standardizing the filtered electromyographic signal;

[0030] Step S103, performing window data division on the standardized electromyographic signal to obtain an original electromyographic image, decomposing the electromyographic signal of each channel in the original electromyographic image into multiple different bands by third-order stationary wavelet packet decomposition, and integrating the electromyographic data of each channel to obtain a corresponding feature image. Figure 3 A window sliding plot on the electromyographic signal is shown. Figure 4 The effect diagram of third-order stationary wavelet packet decomposition is shown. Figure 5 The figure shows the stationary wavelet packet decomposition feature image. Figure 2-Figure 5 As shown, the present invention uses the stationary wavelet packet transform method to significantly improve the data processing speed and recognition accuracy.

[0031] In some embodiments, Figure 6 FIG. 1 shows a classification model training flow chart of a method for recognizing gestures using electromyographic signals based on deep learning according to an embodiment of the present invention. Figure 6 As shown, the classification model is trained through the following steps:

[0032] Step S201, constructing a classification model;

[0033] Step S202, obtaining a first data set, wherein the first data set includes a plurality of preprocessed electromyographic signals;

[0034] Step S203, randomly adding 25-45 dB of random noise to each electromyographic signal in the first data set for data enhancement, to obtain a second data set;

[0035] Step S204, merging the first data set and the second data set to obtain a training data set;

[0036] Step S205: training the classification model using the training data set.

[0037] In the above training process, the present invention creatively uses a method of adding jitter noise to effectively simulate the interference and jitter of the data, and the training result is more robust in recognition.

[0038] The embodiment of the present invention also provides an electromyographic signal gesture recognition device based on deep learning, the device comprising a processing unit and a classification model; the classification model comprises an LSTM layer and a convolutional network layer, the LSTM layer comprises two hidden layers, the convolutional network layer comprises five convolutional layers, a batch normalization layer and an activation function are used for processing between each convolutional layer, a Dropout layer is used after the third convolutional layer to alleviate overfitting, a hybrid attention mechanism module is used after the first and fourth convolutional layers, and an adaptive maximum pooling layer is used before the fifth convolutional layer;

[0039] The processing unit is configured to determine a corresponding feature image according to the electromyographic signal, and output the corresponding feature image to the classification model;

[0040] The classification model is configured to output a classification result based on the feature image.

[0041] In some embodiments, the processing unit is further configured to: perform low-pass filtering on the electromyographic signal; standardize the filtered electromyographic signal; perform window data division on the standardized electromyographic signal to obtain an original electromyographic image, decompose the electromyographic signal of each channel in the original electromyographic image into multiple different bands through third-order stationary wavelet packet decomposition, and integrate the electromyographic data of each channel to obtain a corresponding feature image.

[0042] In some embodiments, the processing unit is further configured to low-pass filter the electromyographic signal using a first-order 1 Hz high-pass Butterworth filter.

[0043] In some embodiments, the device also includes a training unit, which is configured to: construct a classification model; obtain a first data set, wherein the first data set includes multiple preprocessed electromyographic signals; randomly add 25-45dB of random noise to each electromyographic signal in the first data set for data enhancement to obtain a second data set; merge the first data set with the second data set to obtain a training data set; and use the training data set to train the classification model.

[0044] In some embodiments, the convolution numbers of the first to fifth convolutional layers are 64, 64, 128, 64, and 52 respectively, the convolution kernels of the first four convolutional layers are all 3x3, and the convolution kernel of the fifth convolutional layer is 1x1.

[0045] The technical effects of the deep learning-based electromyographic signal gesture recognition device provided in the embodiment of the present invention are basically consistent with those of the previously described method, and will not be repeated here.

[0046] The above description is intended to be illustrative rather than restrictive. For example, the above examples (or one or more of them) can be used in combination with each other. For example, a person of ordinary skill in the art can use other embodiments when reading the above description. In addition, in the above-mentioned specific embodiments, various features can be grouped together to simplify the present disclosure. This should not be interpreted as an intention that a disclosed feature that is not required to be protected is necessary for any claim. On the contrary, the subject matter of the present invention may be less than all the features of a specific disclosed embodiment. Thus, the following claims are incorporated into the specific embodiments as examples or embodiments, wherein each claim is independently used as a separate embodiment, and it is considered that these embodiments can be combined with each other in various combinations or arrangements. The scope of the present invention should be determined with reference to the attached claims and the full scope of equivalent forms granted by these claims.

Claims

1. A method for myoelectric signal gesture recognition based on deep learning, characterized in that: The method comprises: Determine a corresponding characteristic image according to the electromyographic signal; The feature image is input into a trained classification model for classification, wherein the classification model includes an LSTM layer and a convolutional network layer, wherein the LSTM layer includes two hidden layers, and the convolutional network layer includes five convolutional layers, wherein a batch normalization layer and an activation function are used between each convolutional layer for processing, a Dropout layer is used after the third convolutional layer to alleviate overfitting, a hybrid attention mechanism module is used after the first and fourth convolutional layers, and an adaptive maximum pooling layer is used before the fifth convolutional layer; The step of determining the corresponding characteristic image according to the electromyographic signal comprises: Perform low-pass filtering on the electromyographic signal; Normalize the filtered EMG signal; The standardized electromyographic signal is divided into window data to obtain an original electromyographic image. The electromyographic signal of each channel in the original electromyographic image is decomposed into multiple different bands through third-order stationary wavelet packet decomposition, and the electromyographic data of each channel are integrated to obtain a corresponding feature image.

2. The method according to claim 1, characterized in that The EMG signals were low-pass filtered using a first-order 1 Hz high-pass Butterworth filter.

3. The method according to claim 1, characterized in that The classification model is trained by the following method: Build a classification model; Acquire a first data set, wherein the first data set includes a plurality of preprocessed electromyographic signals; Randomly adding 25-45 dB of random noise to each electromyographic signal in the first data set for data enhancement to obtain a second data set; Merging the first data set and the second data set to obtain a training data set; The classification model is trained using the training data set.

4. The method according to claim 1, characterized in that: Among the five convolutional layers, the convolution numbers of the first to fifth convolutional layers are 64, 64, 128, 64, and 52 respectively, the convolution kernels of the first four convolutional layers are all 3x3, and the convolution kernel of the fifth convolutional layer is 1x1.

5. A myoelectric signal gesture recognition device based on deep learning, characterized in that: The device includes a processing unit and a classification model; the classification model includes an LSTM layer and a convolutional network layer, the LSTM layer includes two hidden layers, the convolutional network layer includes five convolutional layers, a batch normalization layer and an activation function are used between each convolutional layer for processing, a Dropout layer is used after the third convolutional layer to alleviate overfitting, a hybrid attention mechanism module is used after the first and fourth convolutional layers, and an adaptive maximum pooling layer is used before the fifth convolutional layer; The processing unit is configured to determine a corresponding feature image according to the electromyographic signal, and output the corresponding feature image to the classification model; The classification model is configured to output a classification result according to the feature image; The processing unit is further configured to: Perform low-pass filtering on the EMG signal: Normalize the filtered EMG signal; The standardized electromyographic signal is divided into window data to obtain an original electromyographic image. The electromyographic signal of each channel in the original electromyographic image is decomposed into multiple different bands through third-order stationary wavelet packet decomposition, and the electromyographic data of each channel are integrated to obtain a corresponding feature image.

6. The device according to claim 5, characterized in that The processing unit is further configured to low-pass filter the electromyographic signal using a first-order 1 Hz high-pass Butterworth filter.

7. The device according to claim 5, characterized in that The apparatus further comprises a training unit, wherein the training unit is configured to: Build a classification model; Acquire a first data set, wherein the first data set includes a plurality of preprocessed electromyographic signals; Randomly adding 25-45 dB of random noise to each electromyographic signal in the first data set for data enhancement to obtain a second data set; Merging the first data set and the second data set to obtain a training data set; The classification model is trained using the training data set.

8. The device according to claim 5, characterized in that Among the five convolutional layers, the convolution numbers of the first to fifth convolutional layers are 64, 64, 128, 64, and 52 respectively, the convolution kernels of the first four convolutional layers are all 3x3, and the convolution kernel of the fifth convolutional layer is 1x1.

Images