Myoelectric gesture recognition method based on lightweight convolutional neural network

By using lightweight convolutional neural networks and deep separable convolution modules in electromyography gesture recognition, combined with cross-stage local networks, the problems of low accuracy and high computing resources in the prior art are solved. Achieve more efficient identification performance and adapt to the needs of mobile devices.

CN114004257BActive Publication Date: 2024-11-22NANJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202111289093.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-02
Publication Date
2024-11-22
Estimated Expiration
2041-11-02

AI Technical Summary

Technical Problem

In the prior art, the electromyography gesture recognition method based on deep learning is low accuracy and consumes a lot of computing resources in multi-classified gesture recognition, making it difficult to meet the practical application needs.

Method used

The lightweight convolutional neural network is adopted, combined with deep separable convolution and cross-stage local networks, and through data preprocessing and feature extraction, it reduces the computational amount and improves recognition performance, including the improvement of deep separable convolution module and label smoothing methods. Use Leaky Relu activation function to improve generalization capabilities.

Benefits of technology

Without reducing the recognition accuracy, the number of parameters and calculations is reduced, the accuracy of gesture recognition is improved, suitable for mobile device application scenarios, and the generalization ability of the network is enhanced through tag smoothing and feature extraction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114004257B_ABST
    Figure CN114004257B_ABST
Patent Text Reader

Abstract

The electromyographic gesture recognition method based on lightweight convolutional neural network extracts time domain features widely used in traditional methods, combines them with original electromyographic data, and characterizes electromyographic signals from multiple angles, which can improve the accuracy of gesture recognition to a certain extent; through a series of operations such as label smoothing and replacement of leaky Relu activation function, the generalization and fitting capabilities of the network are improved, which can further improve the accuracy of gesture recognition; based on lightweight network, deep separable convolution is used instead of traditional convolution, which can reduce the number of parameters and calculations without reducing the recognition accuracy, shorten the training time, and is more suitable for application scenarios such as mobile devices; drawing on the idea of ​​cross-stage local network, local transition layer is adopted, simplified fusion strategy is designed, and the backbone network model is improved, which can achieve further lightweight.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of physiological signal recognition, and aims to design an electromyographic gesture recognition method based on a lightweight convolutional neural network (CNN), which can minimize the computational overhead and improve the algorithm performance while ensuring the accuracy of surface electromyographic signal (sEMG) gesture recognition. Background Art

[0002] Surface electromyography is a relatively complex and weak type of nonlinear bioelectric signal. When the human body makes a movement, the related muscle groups move accordingly and generate various bioelectric signals on the surface of the skin. By conducting corresponding detection and analysis, the electrophysiological characteristics of skeletal muscle during muscle contraction can be studied, so as to distinguish and identify different movement patterns. Due to its advantages of real-time, non-invasive and easy collection, surface electromyography has been widely used in many fields such as human-computer interaction, prosthetic control, clinical diagnosis, medical rehabilitation, etc. in recent years, and has great development potential.

[0003] Similar to other biometric recognition technologies, the steps of sEMG gesture recognition include raw data collection, data preprocessing, feature data extraction, classification and recognition. Among them, classification and recognition is the most critical and important step. Classic methods include linear discriminant analysis, support vector machine, random forest, K nearest neighbor algorithm, hidden Markov model, etc. However, in the actual application scenarios of multi-classification gesture recognition, the recognition accuracy is still not ideal and needs to be improved. In recent years, deep learning technology has been introduced into the relevant research of gesture recognition. Artificial neural networks, convolutional neural networks, recursive neural networks with long short-term memory, residual networks, deep belief networks, etc. have been applied to sEMG gesture recognition by researchers. Among them, convolutional neural networks can easily and quickly extract features of various dimensions, automatically learn the potential features of sEMG, and greatly reduce the manual workload, so they have been widely used. On this basis, domestic and foreign scholars are also constantly trying to fuse different networks, and have proposed a composite convolution-recurrent neural network architecture, a hybrid structure of convolutional long short-term memory-attention mechanism, multi-stream fusion and multi-view deep learning methods, etc. This series of methods based on deep learning have improved the performance of gesture recognition to varying degrees.

[0004] However, the weak signal strength, non-stationarity, and complex structure of the surface electromyographic signal itself, as well as environmental noise, baseline drift, power frequency interference and other factors in the signal acquisition process, will affect the final gesture recognition effect. The accuracy of traditional recognition methods is low, such as the electromyographic gesture recognition method based on multi-stream convolutional neural network disclosed in the prior art CN111898526A and the electromyographic gesture recognition method based on multi-feature fusion CNN disclosed in CN111860410A. At present, similar gesture recognition based on deep learning in the prior art not only consumes more computing resources and time, but also the recognition rate is still unsatisfactory. Summary of the invention

[0005] The electromyographic gesture recognition algorithm proposed in the present invention applies the depthwise separable convolution (DS Conv) in the lightweight network MobileNet to the construction of the gesture recognition network, and draws on the idea of ​​the cross stage partial network (CSPNet) to improve the serially connected depthwise separable convolution module to balance the computational complexity of each layer and maximize the performance of the algorithm without increasing the complexity of the algorithm.

[0006] The method for myoelectric gesture recognition based on lightweight convolutional neural network is characterized in that the method comprises the following steps:

[0007] Step 1, data preprocessing:

[0008] Step 1-1, denoising the original electromyographic data to remove various noises in the signal including baseline drift, electrode displacement, motion artifacts, and power frequency interference, then sampling the data using a sliding window, and converting the processed data into a format;

[0009] Step 1-2, select 5 time domain features for extraction, including mean absolute value MAV, root mean square RMS, number of positive and negative changes of signal slope SSC, number of zero crossings ZC, and waveform length WL, to characterize the surface electromyography signal from multiple angles and combine it with the original data;

[0010] Steps 1-3, introduce label smoothing LS method to prevent overfitting;

[0011] Step 2: Extraction of electromyographic gesture features based on lightweight CNN:

[0012] Build the electromyographic gesture feature extraction model based on the lightweight convolutional neural network CNN, including the input end, traditional convolution layer, depth-separable convolution layer, 2 Csp Blocks, global pooling layer, traditional convolution layer, classifier and output end set in sequence; 2 Csp Blocks are composed of Csp-1 Block and Csp-2 Block; the activation function of the model uses Leaky Relu function;

[0013] Step 3, gesture classification:

[0014] Build the electromyographic gesture classification model based on the lightweight convolutional neural network CNN and connect it with the established feature extraction model. It includes the global pooling layer, Dropout layer, traditional convolution layer, classifier and output end set in sequence. During operation, first perform global average pooling GAP, and inactivate some neurons through Dropout random inactivation operation, then perform 1×1 convolution, and use Softmax classifier to map the result to a real number between (0,1), and perform the final multi-classification task to obtain the possibility of each gesture category. Among them, the category with the maximum probability will be used as the final prediction result of the gesture.

[0015] Furthermore, in step 1-1, first, the baseline drift is processed using the least square method; secondly, the motion artifact is processed using the Butterworth filter; then, the power frequency interference is removed by using a notch filter; and then, different gesture categories are marked and labeled accordingly;

[0016] After the above operations are completed, if the sampling frequency of the electromyographic data is f, and the interception time of the gesture collection is t, then the effective sample points of the rest state are removed, and there are n=f×t in total. Assuming the sliding window length is w and the sliding step length is s, a total of (nw) / s+1 sub-sampled data S are obtained. i , i∈{1,2,…,(tw) / s+1}; if the collected EMG data is multi-channel, the above operation needs to be repeated channel by channel. Suppose the number of electrodes used in the EMG signal collection is C, then the number of channels of the corresponding original EMG data is also C.

[0017] Furthermore, in step 1-2, 5 different time domain features are selected for extraction, and then spliced ​​with the preliminarily preprocessed electromyographic data to obtain more representative data, in preparation for subsequent input into the neural network; the specific extracted time domain features are as follows, where N represents the above sub-sampling data S i The total number of sample points, x i Represents the i-th sample point of the electromyographic signal:

[0018] Mean absolute value MAV:

[0019]

[0020] Root Mean Square RMS:

[0021]

[0022] Number of times the signal slope changes from positive to negative value SSC:

[0023]

[0024]

[0025] Among them, threshold is a threshold parameter set to reduce noise interference;

[0026] Zero crossing times ZC:

[0027]

[0028]

[0029] Waveform length WL:

[0030]

[0031] Furthermore, in steps 1-3, in order to prevent the fitting phenomenon, a smoothing factor parameter ∈ that plays a role similar to noise is artificially set to enhance the loss function value of other labels for the loss function; the specific formula for label smoothing is as follows:

[0032]

[0033] Among them, ∈ is the smoothing factor parameter, K is the number of categories, q'(k) is the new label probability distribution after label smoothing correction, q(k) is the original label distribution, and k is the kth category in the classification category.

[0034] Furthermore, in step 2, the depthwise separable convolution layer is composed of a combination of depthwise convolution and pointwise convolution; the depthwise convolution decomposes the convolution operation by channel, the convolution kernel depth is 1, one channel corresponds to one convolution kernel, so that each channel performs convolution operations of different sizes; the pointwise convolution uses a 1×1 convolution kernel to adjust the number of channels, fuse the features of all different channels, and generate a feature map.

[0035] Furthermore, in step 2, Csp-1 Block performs cross-stage operations; first, the output obtained after the depthwise separable convolution operation is used as input and divided into two parts, including Part 1 and Part 2, and the number of convolution kernels in each part is half of the original input; among them, Part 1 performs a depthwise separable convolution operation alone, and then performs a normal convolution as a local transition processing to maximize the difference in gradient combinations and prevent different layers from learning repeated gradient information; Part 2 no longer performs any additional operations; finally, Part 1 and Part 2 are merged through the Concatenate splicing operation.

[0036] Furthermore, in step 2, Csp-2 Block takes the output obtained after the depthwise separable convolution operation as input, takes the input data as Part 1, performs ordinary convolution on Part 1 to obtain Part 2, and then merges Part 1 and Part 2 through Concatenate to obtain the output.

[0037] The beneficial effects of the present invention are:

[0038] (1) The time domain features widely used in traditional methods are extracted and combined with the original electromyographic data to characterize the electromyographic signals from multiple angles, which can improve the accuracy of gesture recognition to a certain extent.

[0039] (2) Through a series of operations such as label smoothing and replacement of the Leaky Relu activation function, the generalization and fitting capabilities of the network are improved, which can further improve the accuracy of gesture recognition.

[0040] (3) Based on a lightweight network, the deep separable convolution is used instead of the traditional convolution. This can reduce the number of parameters and calculations, shorten the training time, and is more suitable for application scenarios such as mobile devices without reducing the recognition accuracy.

[0041] (4) Drawing on the idea of ​​cross-stage local networks, adopting local transition layers, designing simplified fusion strategies, and improving the backbone network model, further lightweighting can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of data preprocessing in an example of the present invention.

[0043] Figure 2 This is a model architecture diagram based on a lightweight convolutional neural network in an example of the present invention.

[0044] Figure 3 Schematic diagram of depth-wise separable convolution in an example of the present invention.

[0045] Figure 4 It is a schematic diagram comparing the original deeply separable module architecture and the cross-stage fusion strategy in the example of the present invention. DETAILED DESCRIPTION

[0046] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0047] From the perspective of adapting to mobile, embedded and other devices, the present invention proposes an electromyographic gesture recognition method based on a lightweight convolutional neural network. First, in order to increase the feature richness of electromyographic signals and improve the accuracy of gesture recognition, the original electromyographic data is preprocessed, and on this basis, 5 time domain features are extracted to better characterize the electromyographic signals; secondly, in order to reduce the amount of calculation and the number of parameters, the idea of ​​MobileNet is used to introduce deep separable convolution; finally, a cross-stage local network is constructed to further lightweight the architecture. The present invention specifically includes the following steps.

[0048] Step 1: Figure 1 As shown in the preprocessing process, since the surface electromyographic signal is a weak and non-stationary electrical signal, it is very susceptible to various external interferences. Therefore, the first step is to preprocess the original electromyographic signal to remove noise as much as possible. First, the baseline drift is processed using the least squares method; secondly, the motion artifact is processed using the Butterworth filter; then, the power frequency interference is removed through the notch filter; after that, different gesture categories are marked and labeled accordingly.

[0049] After the above operations are completed, the data is sampled using a sliding window, and the processed data is formatted. Specifically, if the sampling frequency of the electromyographic data is f, and the interception time of the gesture is t, then there are n=f×t valid sample points in the rest state after removing them. If the sliding window length is w and the sliding step length is s, then a total of (nw) / s+1 sub-sampled data S can be obtained. i , i∈{1,2,…,(tw) / s+1}. If the collected EMG data is multi-channel, the above operation needs to be repeated channel by channel. Suppose the number of electrodes used in the EMG signal collection is C, then the number of channels of the corresponding original EMG data is also C.

[0050] On this basis, the present invention selects 5 different time domain features for extraction, namely, the average absolute value, the root mean square, the number of changes in the positive and negative values ​​of the signal slope, the number of zero crossings, and the waveform length. Then, it is spliced ​​with the preliminarily preprocessed electromyographic data to obtain more representative data in preparation for subsequent input into the neural network. The specific extracted time domain features are shown below. Among them, N represents the above sub-sampling data S i The total number of sample points, x iRepresents the i-th sample point of the electromyographic signal:

[0051] Mean absolute value MAV:

[0052]

[0053] Root Mean Square RMS:

[0054]

[0055] Number of times the signal slope changes positively or negatively SSC:

[0056]

[0057]

[0058] Among them, threshold is a threshold parameter set to reduce noise interference.

[0059] Zero crossing times ZC:

[0060]

[0061]

[0062] Waveform length WL:

[0063]

[0064] On the basis of the above preprocessing, in order to ensure the generalization ability and adaptability of the model and avoid overfitting, the present invention also introduces a label smoothing method. Specifically, because the present invention adopts One-Hot Encoding, the original label value is only 1 and 0, that is, it can only be selected from the two options of marking the correct category as 1 and marking the wrong category as 0, as shown in the following formula.

[0065]

[0066] Among them, q(k) is the original label distribution, k is the kth category in the classification category, and y is the actual classification category. As a result, the network model will be too "confident" and increasingly biased towards the correct label with a value of 1, causing overfitting. Label smoothing, on the other hand, is to artificially set a smoothing factor parameter ∈ that plays a role similar to noise to enhance the loss function value of other labels for the loss function. Therefore, a certain penalty for accurate classification can improve the generalization ability of the network, thereby improving the accuracy of the network model. The specific formula for label smoothing is as follows:

[0067]

[0068] Among them, ∈ is the smoothing factor parameter with a small value, K is the number of categories, and q'(k) is the new label probability distribution after label smoothing correction.

[0069] Step 2: In this phase, Figure 2 As shown in the figure, the overall model based on the lightweight convolutional neural network is constructed. The input data will first undergo a traditional convolution operation, and then a depth-separable convolution. Then, it will pass through two Csp Blocks in succession. This module consists of Csp-1 and Csp-2 Blocks.

[0070] Among them, the depth-wise separable convolution is as follows Figure 3 As shown, it is composed of a combination of deep convolution and point-by-point convolution. Deep convolution decomposes the convolution operation by channel, the depth of the convolution kernel is 1, and one channel corresponds to one convolution kernel, so that each channel can perform convolution operations of different sizes. Point-by-point convolution is similar to standard convolution. Through the 1×1 convolution kernel, the number of channels is first adjusted, and the features of all different channels are fused to generate a feature map. Through this series of operations, the calculation amount of the convolution part is about 1 / 9 of the original convolution, and the number of parameters and the amount of calculation are greatly reduced. In addition, in terms of activation function, in order to enhance the nonlinear fitting ability of the network, the present invention replaces the activation function from the common Relu function to the Leaky Relu function, and for the part where the input is less than 0, a very small non-zero output value is also assigned, so that the negative axis information will not be completely lost. As a result, the Leaky Relu function not only retains the advantages of the Relu function's convergence speed and fast calculation speed, but also avoids the situation where some neurons no longer update the weights when the input value of the Relu function is negative, thereby improving the recognition rate of the network to a certain extent.

[0071] Based on the above-mentioned depthwise separable convolution, the present invention further applies a cross-stage architecture to optimize the depthwise separable convolution blocks with the same number of convolution kernels and a step size of 1, and designs Csp-1 Block and Csp-2Block. Figure 4 The depth-separable convolution module shown in (a) is the original serially constructed k-layer depth-separable convolution layer dw_conv i , i∈{1,2,...,k},k≥2, if the number of convolution kernels in each layer is the same, and except for the first layer whose step size is 2, the step sizes of the subsequent layers are all 1, then the following k-1 layers are processed as follows Figure 4(b) shows the Csp-1 cross-stage operation processing. First, after a depth-wise separable convolution operation, the output is used as the input of the next step and is divided into two parts (Part 1 and Part 2). The number of convolution kernels in each part is half of the original input. Among them, Part 1 still needs to perform a depth-wise separable convolution operation separately, and then perform a normal convolution as a local transition processing to maximize the difference in gradient combinations and prevent different layers from learning repeated gradient information, while Part 2 no longer performs any additional operation processing. Finally, Part 1 and Part 2 are merged through the Concatenate splicing operation to balance the calculations of each layer and reduce the computational cost.

[0072] At the same time, in order to further simplify the network, the present invention also designs the following Figure 4 The Csp-2Block shown in (c) is more streamlined than the Csp-1 Block. After a depth-wise separable convolution, this module directly uses the data that has undergone a normal convolution as Part 1, and then performs another normal convolution on Part 1 to obtain Part 2, and then directly merges the two through Concatenate to obtain the output, thereby achieving a significant reduction in the amount of calculation and operation.

[0073] Finally, when the original network model continuously applies multiple depth-wise separable convolutional modules, this method Figure 2 As shown, through the application of Csp Block, Csp-1 and Csp-2 modules are used alternately as the final network architecture, thereby achieving the goal of "lightweight".

[0074] Step 3: In the classification stage, in order to further reduce the number of parameters and regularize the structure, the present invention replaces the fully connected layer used in the traditional convolutional neural network with the following operations, such as Figure 2 As shown in . First, a global average pooling is performed, and then a Dropout random inactivation operation is performed to inactivate some neurons with a certain probability to prevent overfitting. Next, a 1×1 convolution is performed to map the output to a specified number of gestures. Finally, Softmax is used to map the result to a real number between (0,1) for final classification to obtain the probability of each gesture category. Among them, the category with the maximum probability will be used as the final prediction result of the gesture.

[0075] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. Myoelectric gesture recognition method based on lightweight convolutional neural network, characterized by: The method comprises the following steps: Step 1, data preprocessing: Step 1-1: De-noise the original EMG data to remove various noises in the signal, including baseline drift, electrode displacement, motion artifacts, and power frequency interference. Then, use a sliding window to sample the data and convert the processed data into a new format. In step 1-2, five time-domain features are selected for extraction, including mean absolute value (MAV), root mean square (RMS), number of positive and negative slope changes (SSC), number of zero crossings (ZC), and waveform length (WL), to characterize the surface electromyography signal from multiple angles and combine it with the original data. Steps 1-3, introduce label smoothing LS method to prevent overfitting; Step 2: Myoelectric gesture feature extraction based on lightweight CNN: Build an EMG gesture feature extraction model based on a lightweight convolutional neural network (CNN), including the input, traditional convolutional layer, depthwise separable convolutional layer, two Csp blocks, a global pooling layer, a traditional convolutional layer, a classifier, and an output. The two Csp blocks consist of Csp-1 and Csp-2 blocks. The activation function of the model uses the LeakyRelu function. The Csp-1 Block performs cross-stage operations. First, the output of the depthwise separable convolution operation is used as input and divided into two parts, including Part 1 and Part 2. The number of convolution kernels in each part is half of the original input. Part 1 performs a depthwise separable convolution operation alone, and then performs a normal convolution as a local transition process to maximize the difference in gradient combinations and prevent different layers from learning repeated gradient information. Part 2 does not undergo any additional operations. Finally, Part 1 and Part 2 are merged through the concatenation operation. Csp-2 Block takes the output of the depthwise separable convolution operation as input, takes the input data as Part 1, performs ordinary convolution on Part 1 to obtain Part 2, and then merges Part 1 and Part 2 through concatenation to obtain the output; Step 3, gesture classification: A myoelectric gesture classification model based on a lightweight convolutional neural network (CNN) is built and connected to the established feature extraction model. It includes a global pooling layer, a Dropout layer, a traditional convolutional layer, a classifier, and an output end, which are set in sequence. During operation, global average pooling (GAP) is first performed, and a Dropout random inactivation operation is performed to inactivate some neurons. Then, a 1×1 convolution is performed, and the result is mapped to a real number between (0,1) using a Softmax classifier. The final multi-classification task is performed to obtain the probability of each gesture category. Among them, the category with the maximum probability will be used as the final prediction result of the gesture.

2. The myoelectric gesture recognition method based on a lightweight convolutional neural network according to claim 1, characterized in that: In step 1-1, first, the baseline drift is processed using the least squares method; second, the motion artifact is processed using a Butterworth filter; and then, the power frequency interference is removed using a notch filter. After that, mark and label the gestures according to their categories. After the above operations are completed, if the sampling frequency of the electromyographic data is f and the interception time of the gesture is t, then the effective sample points of the rest state are removed, and there are n=f×t in total. Assuming the sliding window length is w and the sliding step size is s, a total of (nw) / s+1 sub-sampled data S are obtained. i , i∈{1,2,…,(tw) / s+1}; if the collected EMG data is multi-channel, the above operation needs to be repeated channel by channel. Suppose the number of electrodes used in EMG signal collection is C, then the number of channels of the corresponding original EMG data is also C.

3. The myoelectric gesture recognition method based on a lightweight convolutional neural network according to claim 2, characterized in that: In step 1-2, five different time domain features are selected for extraction, and then spliced ​​with the pre-processed electromyographic data to obtain more representative data, in preparation for subsequent input into the neural network; the specific extracted time domain features are shown below, where N represents the sub-sampled data S i The total number of sample points, x i Represents the i-th sample point of the electromyographic signal: Mean absolute value MAV: Root mean square RMS: Number of positive and negative changes in the signal slope SSC: Among them, threshold is a threshold parameter set to reduce noise interference; Zero crossing times ZC: Waveform length WL:

4. The myoelectric gesture recognition method based on a lightweight convolutional neural network according to claim 1, characterized in that: In steps 1-3, to prevent fitting, a smoothing factor parameter ∈ is artificially set to act like noise to enhance the loss function values ​​of other labels. The specific formula for label smoothing is as follows: Among them, ∈ is the smoothing factor parameter, K is the number of categories, q'(k) is the new label probability distribution after label smoothing correction, q(k) is the original label distribution, and k is the kth category in the classification category.

5. The myoelectric gesture recognition method based on lightweight convolutional neural network according to claim 1, characterized in that: In step 2, the depthwise separable convolution layer is composed of a combination of depthwise convolution and pointwise convolution. The depthwise convolution decomposes the convolution operation by channel, the convolution kernel depth is 1, and one channel corresponds to one convolution kernel, so that each channel performs convolution operations of different sizes. The pointwise convolution uses a 1×1 convolution kernel to adjust the number of channels, fuse the features of all different channels, and generate a feature map.

Citation Information

Cited By

  • Internet of Things equipment response optimization method of lightweight neural network

    CN121078047A