Hand motion estimation method based on fusion of sEMG wavelet transform image and CNN-LSTM network

Abstract

The invention discloses a human hand motion estimation method based on fusion of an sEMG wavelet transform image and a CNN-LSTM network, and the method comprises the following steps: 1, carrying out the preprocessing of a surface electromyogram signal, and recognizing an active segment; 2, processing and visualizing a continuous wavelet transform result of the surface electromyogram signal, and synthesizing an instantaneous image at each sampling moment; 3, constructing a CNN-LSTM neural network model; and step 4, inputting the instantaneous image into a CNN-LSTM neural network model for training so as to predict and estimate the movement of the human hand, and outputting a corresponding prediction result. According to the method, no extra information or manual design of a feature extractor is needed in the human hand motion estimation process, the method can be suitable for few-channel signals, the problems of large data processing amount and high cost are solved under the condition that the prediction result is accurate, and the training time is shortened by constructing a CNN-LSTM neural network model.

Description

Technical Field

[0001] The present invention relates to a method for estimating human hand motion, and in particular to a method for estimating human hand motion based on the fusion of sEMG wavelet transform image and CNN-LSTM network. Background Art

[0002] The main idea of ​​existing methods for hand motion estimation based on surface electromyography signals is to use high-density electrodes to collect electromyography signals and establish a mapping relationship model between surface electromyography signals and continuous variables such as joint angles, joint speeds or joint torques. The main existing models are musculoskeletal models based on biomechanics and regression models. Among them, the musculoskeletal model based on biomechanics can provide a certain theoretical framework and practical guidance for joint angle prediction, but it has problems such as complex calculations, parameter uncertainty, and neglect of dynamic changes. The regression model also has the limitations of feature selection difficulties and linear assumptions, so the more commonly used method is the machine learning method based on neural networks. The existing research level of related technologies has achieved some results in realizing hand motion estimation, but there are still certain shortcomings that need to be further studied: 1) In order to ensure accuracy, recent methods for hand motion estimation focus on using high-density electrodes to collect dozens or hundreds of sEMG signals at the same time, but this will cause various problems such as high cost and large data processing volume. 2) In terms of feature extraction of surface electromyography signals, the existing feature extraction methods require manual feature engineering, which will lead to certain redundancy and limitations in the extracted electromyography information. 3) Current methods usually require a lot of computing resources in steps such as signal processing, feature extraction, and model training, leading to latency issues. Summary of the invention

[0003] The present invention provides a method for estimating hand motion based on the fusion of sEMG wavelet transform image and CNN-LSTM network. The method does not require any additional information or manually designed feature extractors in the process of hand motion estimation, and can be applied to few-channel signals. While ensuring accurate prediction results, the problems of large data processing volume and high cost are solved, and the training time is shortened by constructing a reasonable CNN-LSTM neural network model.

[0004] The objective of the present invention is achieved through the following technical solutions:

[0005] A method for estimating hand motion based on sEMG wavelet transform image and CNN-LSTM network fusion, comprising the following steps:

[0006] Step 1: pre-process the surface electromyography signal and identify the activity segment;

[0007] Step 2: Process and visualize the continuous wavelet transform results of the surface electromyography signal and synthesize the instantaneous image at each sampling moment:

[0008] Step 3: Build a CNN-LSTM neural network model:

[0009] A convolutional neural network model (CNN) and a long short-term memory network (LSTM) are combined to construct a CNN-LSTM neural network model, wherein the first layer of the CNN-LSTM neural network model is an input layer, the second layer is composed of a two-dimensional convolution layer, a normalization layer, a ReLU activation function layer and a maximum pooling layer, the convolution kernel size of the two-dimensional convolution layer is 3×3, the number of output channels is 16, the normalization layer is used to accelerate training and reduce the gradient vanishing problem, and then the ReLU activation function layer is used to activate the nonlinearity, and then the 2×2 maximum pooling layer is connected; the third layer is composed of a two-dimensional convolution layer, a normalization layer and a ReLU activation function layer, the convolution kernel size of the two-dimensional convolution layer is 3×3, the number of output channels is 32, the normalization layer is used to accelerate training and reduce the gradient vanishing problem, and then the ReLU activation function layer is used to activate the nonlinearity; the fourth layer is an expansion layer, which flattens the multidimensional data features into one dimension; the fifth layer is an LSTM layer, the number of output units is 64, and a Dropout layer is connected thereafter to prevent overfitting; finally, a fully connected layer and a regression layer are connected to output a prediction result;

[0010] Step 4: Input the instantaneous image into the CNN-LSTM neural network model for training to predict and estimate the movement of the human hand and output the corresponding prediction results.

[0011] Compared with the prior art, the present invention has the following advantages:

[0012] 1) The present invention solves the problem that the existing methods for estimating hand movements are not accurate enough and that the extracted electromyographic information still has certain redundancy and limitations in the process of feature extraction and selection. The present invention combines continuous wavelet transform images with convolutional neural networks and long short-term memory networks, does not require any additional information or manually designed feature extractors, and reduces the number of layers of the existing model for predicting movement through neural networks, which can not only ensure the prediction effect, but also shorten the training time, and also solve the delay problem to a certain extent.

[0013] 2) The present invention solves the problem that the existing human hand movement estimation method obtains sufficient information to ensure accuracy by collecting dozens or hundreds of sEMG signals. The present invention can be applied to signals with few channels, and solves the problem of large data processing volume and high cost while ensuring accurate prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1It is a flowchart of the method for estimating hand motion based on the fusion of sEMG wavelet transform image and CNN-LSTM network;

[0015] Figure 2 This is the structural diagram of the CNN-LSTM neural network;

[0016] Figure 3 Comparison curves of the true values ​​and predicted values ​​of 10 joints, a-thumb metacarpal joint, b-thumb proximal joint, c-index finger metacarpal joint, d-index finger proximal joint, e-middle finger metacarpal joint, f-middle finger proximal joint, g-ring finger metacarpal joint, h-ring finger proximal joint, i-little finger metacarpal joint, j-little finger proximal joint. DETAILED DESCRIPTION

[0017] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.

[0018] The present invention provides a method for estimating hand motion based on the fusion of sEMG wavelet transform image and CNN-LSTM network. Figure 1 As shown, the method comprises the following steps:

[0019] Step 1: Preprocess the surface electromyography signal and identify the activity segment. The specific steps are as follows:

[0020] Step 1: Filter the original signal with a 50 Hz power frequency noise filter and a 5-450 Hz bandpass filter to improve the signal-to-noise ratio;

[0021] Step 1 and 2: Distinguish the parts with and without movement from the continuous surface electromyography signal, and filter out the parts with movement.

[0022] Step 2: Process and visualize the continuous wavelet transform results of the surface electromyography signal to synthesize the instantaneous image at each sampling moment. The specific steps are as follows:

[0023] Step 21: Perform Morse continuous wavelet transform on each channel, obtain the amplitude of wavelet transform, and draw the original pseudo-color scale map;

[0024] Step 22: Normalize the scale image and apply color mapping, extract the RGB color values ​​corresponding to the three color channels in the mapping, convert the scale image into an RGB image, and save the image in png format;

[0025] Step 2: Splice the images of each channel at each sampling point. Each time, select 12 images (one for each channel) and arrange them in the form of 3 rows and 4 columns to synthesize the instantaneous image at each sampling moment.

[0026] Step 3: Build a CNN-LSTM neural network model:

[0027] The convolutional neural network model (CNN) and the long short-term memory network (LSTM) are combined to construct a CNN-LSTM neural network model, such as Figure 2 As shown, the first layer of the CNN-LSTM neural network model is an input layer of 12×12×3; the second layer is a two-dimensional convolution layer, the size of the convolution kernel is 3×3, the number of output channels is 16, and then the normalization layer is used to accelerate the training to reduce the gradient vanishing problem, and then the ReLU activation function layer is used for nonlinear activation, and then the 2×2 maximum pooling layer is connected; the third layer is a two-dimensional convolution layer, the size of the convolution kernel is 3×3, the number of output channels is 32, and then the normalization layer is used to accelerate the training to reduce the gradient vanishing problem, and then the ReLU activation function layer is used for nonlinear activation; the fourth layer is an expansion layer, which flattens the multidimensional data features into one dimension; the fifth layer is an LSTM layer, the number of output units is 64, and the Dropout layer is connected thereafter to prevent overfitting; finally, the fully connected layer and the regression layer are connected to output the predicted value.

[0028] Step 4: Divide the synthesized instantaneous images into 70% training set and 30% validation set. Use the instantaneous images in the training set as the input of the CNN-LSTM neural network model for training, and use the validation set to adjust the parameters of the model and evaluate the model's capabilities (evaluation indicators are: normalized mean square error, normalized root mean square error, Pearson correlation coefficient and absolute coefficient). Finally, output the predicted estimate of the human hand movement.

[0029] Example:

[0030] This example verifies the method using the data from DB2 in the Ninapro dataset. The surface electromyography data in the selected dataset is obtained by 12 Delsys Trigno electrodes with a sampling frequency of 2000Hz, and the motion data is obtained by the Cyberglove 2 data glove. The specific steps of hand motion estimation are as follows:

[0031] Step 1: Preprocess the surface electromyography signal and identify the activity segment. The specific steps are as follows:

[0032] Step 1: Filter the original signal with a 50 Hz power frequency noise filter and a 5-450 Hz bandpass filter to improve the signal-to-noise ratio;

[0033] Step 1 and 2: Distinguish the parts with and without movement from the continuous surface electromyography signal, and filter out the parts with movement.

[0034] Step 2: Process and visualize the continuous wavelet transform results of the surface electromyography signal and synthesize the instantaneous image at each sampling moment:

[0035] Step 2: Process and visualize the continuous wavelet transform results of the surface electromyography signal to synthesize the instantaneous image at each sampling moment. The specific steps are as follows:

[0036] Step 21: Perform Morse continuous wavelet transform on each channel, obtain the amplitude of wavelet transform, and draw the original pseudo-color scale map;

[0037] Step 22: Normalize the scale image and apply color mapping, extract the RGB color values ​​corresponding to the three color channels in the mapping, convert the scale image into an RGB image, and save the image in png format;

[0038] Step 2: Splice the images of each channel at each sampling point. Each time, select 12 images (one for each channel) and arrange them in the form of 3 rows and 4 columns to synthesize the instantaneous image at each sampling moment.

[0039] Step 3: Build a CNN-LSTM neural network model:

[0040] A convolutional neural network model (CNN) and a long short-term memory network (LSTM) are combined to construct a CNN-LSTM neural network model, wherein the first layer of the CNN-LSTM neural network model is an input layer of 12×12×3; the second layer is a two-dimensional convolution layer, the size of the convolution kernel is 3×3, and the number of output channels is 16, and then the training is accelerated through a normalization layer to reduce the gradient vanishing problem, and then the nonlinear activation is performed through a ReLU activation function layer, and then a 2×2 maximum pooling layer is connected; the third layer is a two-dimensional convolution layer, the size of the convolution kernel is 3×3, and the number of output channels is 32, and then the training is accelerated through a normalization layer to reduce the gradient vanishing problem, and then the nonlinear activation is performed through a ReLU activation function layer; the fourth layer is an expansion layer, which flattens the multidimensional data features into one dimension; the fifth layer is an LSTM layer, the number of output units is 64, and a Dropout layer is connected thereafter to prevent overfitting; finally, the fully connected layer and the regression layer are connected to output the predicted value.

[0041] Step 4: Divide the synthesized instantaneous images into 70% training set and 30% validation set. Use the instantaneous images in the training set as the input of the CNN-LSTM neural network model for training, and use the validation set to evaluate the model's capabilities (evaluation indicators are: normalized mean square error, normalized root mean square error, Pearson correlation coefficient and absolute coefficient). Finally, output the predicted values ​​of the hand joint angles during hand movement.

[0042] In this embodiment, 10 main joints (proximal and metacarpal joints) of grasping movements are selected, and the normalized mean square error, normalized root mean square error, Pearson correlation coefficient and absolute error coefficient are used as evaluation indicators of the prediction results. The average correlation Pearson coefficient of each joint is 0.93, the average absolute coefficient of each joint is 0.84, the average normalized root mean square error of each joint is 0.1307, and the average normalized absolute error of each joint is 0.0962. The present invention improves the accuracy and effect of existing methods for estimating hand movements. The comparison curves of the true values ​​and predicted values ​​of the 10 joints are shown in Figure 2. Figure 3 By comparing the curves of predicted values ​​and true values, it can help evaluate the accuracy and robustness of the prediction model. Figure 3 It can be seen that the predicted value curve is very close to the true value curve, indicating that the model's prediction is relatively accurate and can better simulate the actual motion trajectory.

Claims

1. A method for estimating hand motion based on the fusion of sEMG wavelet transform image and CNN-LSTM network, characterized in that The method comprises the following steps: Step 1: pre-process the surface electromyography signal and identify the activity segment; Step 2: Process and visualize the continuous wavelet transform results of the surface electromyography signal and synthesize the instantaneous image at each sampling moment: Step 3: Build a CNN-LSTM neural network model: Combine the convolutional neural network model with the long short-term memory network to build a CNN-LSTM neural network model; Step 4: Input the instantaneous image into the CNN-LSTM neural network model for training to predict and estimate the movement of the human hand and output the corresponding prediction results.

2. The method for estimating hand motion based on sEMG wavelet transform image and CNN-LSTM network fusion according to claim 1 is characterized in that The specific steps of step one are as follows: Step 1: Filter the original signal with a 50 Hz power frequency noise filter and a 5-450 Hz bandpass filter to improve the signal-to-noise ratio; Step 1 and 2: Distinguish the parts with and without movement from the continuous surface electromyography signal, and filter out the parts with movement.

3. The method for estimating hand motion based on sEMG wavelet transform image and CNN-LSTM network fusion according to claim 1 is characterized in that The specific steps of step 2 are as follows: Step 21: Perform Morse continuous wavelet transform on each channel, obtain the amplitude of wavelet transform, and draw the original pseudo-color scale map; Step 22: Normalize the scale image and apply color mapping, extract the RGB color values ​​corresponding to the three color channels in the mapping, convert the scale image into an RGB image, and save the image in png format; Step 2: The images of each channel of each sampling point are stitched together to synthesize the instantaneous image at each sampling moment.

4. The method for estimating hand motion based on sEMG wavelet transform image and CNN-LSTM network fusion according to claim 3 is characterized in that In the steps 2 and 3, each time the images are stitched together, 12 images are selected and arranged in the form of 3 rows and 4 columns.

5. The method for estimating hand motion based on sEMG wavelet transform image and CNN-LSTM network fusion according to claim 1, characterized in that The first layer of the CNN-LSTM neural network model is the input layer, the second layer is composed of a two-dimensional convolutional layer, a normalization layer, a ReLU activation function layer and a maximum pooling layer, the convolution kernel size of the two-dimensional convolutional layer is 3×3, the number of output channels is 16, the normalization layer is used to accelerate training and reduce the gradient vanishing problem, and then the ReLU activation function layer is used to activate the nonlinearity, and then the 2×2 maximum pooling layer is connected; the third layer is composed of a two-dimensional convolutional layer, a normalization layer and a ReLU activation function layer, the convolution kernel size of the two-dimensional convolutional layer is 3×3, the number of output channels is 32, the normalization layer is used to accelerate training and reduce the gradient vanishing problem, and then the ReLU activation function layer is used to activate the nonlinearity; The fourth layer is the unfolding layer, which flattens the multidimensional data features into one dimension; The fifth layer is the LSTM layer, and the number of output units is 64. In order to prevent overfitting, the Dropout layer is connected to it. Finally, the fully connected layer and the regression layer are connected to output the prediction results.

Images