A finger joint continuous motion estimation method and system based on sEMG

Abstract

The application provides a finger joint continuous motion estimation method and system based on sEMG. The method comprises the following steps: acquiring the arm surface electromyogram signal of finger multi-joint continuous motion and the real-time angle data of the finger joint; performing root mean square value calculation on the time sequence characteristics of the surface electromyogram signal, and constructing a time sequence characteristic vector; performing normalization processing on the time sequence characteristic vector and the real-time angle data of the finger joint respectively, and constructing a training data pair according to the normalized time sequence characteristic vector and the corresponding finger joint real-time angle; taking the time sequence characteristic vector in the training data pair as input and the corresponding finger joint real-time angle as output, and training an improved Transformer neural network model; and performing finger joint motion estimation based on the trained improved Transformer neural network model. The application can realize accurate estimation of finger joint motion, has high operation speed and high estimation precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of biological signals and pattern recognition, in particular, especially to a finger joint continuous motion estimation method and system based on sEMG. BACKGROUND

[0002] With the development of artificial intelligence technology, deep learning algorithms have been widely used in image processing, anomaly detection, text prediction, human-computer collaboration and other fields, greatly improving the efficiency of life and production. Human-computer collaboration (HMC) is usually composed of a system of people and computers (including embedded control systems), in which computers (embedded control devices) are mainly responsible for a large amount of data calculation and part of the inference work, while the insufficient part (such as decision, evaluation and selection, etc.) needs to be responsible for artificial. Not only can it fully exert the flexibility and creativity of people, but also through the mutual cooperation and close cooperation between people and computers, it can efficiently handle more complex problems.

[0003] Electromyography (sEMG) is commonly used in the fields of biomechanics and human-computer interaction to measure muscle activity and calculate motion data of joints, limbs or other body parts. Electromyography is usually recorded with surface electrodes connected to the skin, and then processed and analyzed to estimate the ongoing motion of the target. The common continuous motion estimation method only focuses on improving the accuracy, while ignoring the time-consuming of model estimation, resulting in poor real-time performance of motion evaluation. SUMMARY

[0004] In view of the deficiencies of the prior art, the present application provides a finger joint continuous motion estimation method and system based on sEMG. First, the surface electromyography of the arm key points and the real-time angle of the finger joints of the collected eight kinds of gripping motions are processed, and then the root mean square value of the surface electromyography is extracted as a feature vector for motion estimation, which can quickly and accurately obtain the estimation result.

[0005] The technical means adopted by the present application are as follows:

[0006] A finger joint continuous motion estimation method based on sEMG, comprising the following steps:

[0007] Obtain the arm surface electromyography of the finger multi-joint continuous motion and the real-time angle data of the finger joints, and preprocess the surface electromyography;

[0008] Calculate the root mean square value of the time sequence characteristics of the preprocessed surface electromyography, and construct a time sequence feature vector according to the calculated root mean square value of the time sequence characteristics;

[0009] The temporal feature vector and the real-time angle data of the finger joints are normalized respectively, and training data pairs are constructed based on the normalized temporal feature vector and the corresponding real-time angle of the finger joint.

[0010] The improved Transformer neural network model is trained by using the temporal feature vectors of the training data pairs as input and the corresponding real-time angles of the finger joints as output.

[0011] Finger joint motion estimation is performed based on a trained, improved Transformer neural network model.

[0012] Furthermore, the improved Transformer neural network model is trained, including:

[0013] Encode the temporal feature vector;

[0014] Perform multi-head attention operations on the encoded temporal feature vectors;

[0015] The feedforward neural network is used to compress the features output after multi-head attention operation, and the hidden neurons are filtered through the FNN layer to obtain effective neuron information.

[0016] The improved Transformer neural network model is pre-trained based on a public dataset to obtain the weights of the effective neurons in the FNN layer. The weights of the effective neurons obtained in the pre-training are then loaded into the improved Transformer neural network model. Finally, the neural network model with the loaded weights of the effective neurons is formally trained based on the training dataset.

[0017] Furthermore, a feedforward neural network is used to compress the features output after multi-head attention operations. Hidden layer neurons are filtered through the FNN layer to obtain effective neuron information, including:

[0018] The Bernoulli distribution is used to evaluate the hidden layer neurons in the neural network and to prune unimportant neurons. The Bernoulli distribution is as follows:

[0019] g t ~Ber(sigmoid(α) p ))

[0020] Among them, g t Represents a binary gate unit, α p This indicates the results of the assessment of the number of neurons.

[0021] Furthermore, the improved Transformer neural network model is trained by using the temporal feature vectors from the training data pairs as input and the corresponding real-time angles of the finger joints as output, including:

[0022] After the model is trained, the model output curve is smoothed as follows:

[0023]

[0024] where w is the sliding window, and X is the joint angle predicted by the model output.

[0025] Further, the method further comprises deploying the trained improved Transformer neural network model offline to an ARM device, and measuring the inference time of the model in a window.

[0026] Further, the pre-processed surface electromyography signal is subjected to root mean square value calculation of time sequence characteristics, and a time sequence characteristic vector is constructed according to the calculated root mean square value of the time sequence characteristics, and the method further comprises: performing noise reduction processing on the electromyography signal through a Butterworth filter, and performing sampling alignment processing on the noise-reduced electromyography signal and the joint angle.

[0027] The application further discloses a finger joint continuous motion estimation system based on sEMG, comprising:

[0028] A data acquisition unit is configured to acquire the arm surface electromyography signal of finger multi-joint continuous motion and real-time angle data of finger joints, and pre-process the surface electromyography signal.

[0029] A feature extraction unit is configured to perform root mean square value calculation of time sequence characteristics on the pre-processed surface electromyography signal, and construct a time sequence characteristic vector according to the calculated root mean square value of the time sequence characteristics.

[0030] A normalization unit is configured to perform normalization processing on the time sequence characteristic vector and the real-time angle data of finger joints respectively, and construct a training data pair according to the normalized time sequence characteristic vector and the corresponding finger joint real-time angle.

[0031] A training unit is configured to take the time sequence characteristic vector in the training data pair as input and the corresponding finger joint real-time angle as output, and train an improved Transformer neural network model.

[0032] A motion estimation unit is configured to perform finger joint motion estimation based on the trained improved Transformer neural network model.

[0033] Compared with the prior art, the application has the following advantages:

[0034] This invention extracts root mean square features from the raw electromyography signal, then uses two-dimensional convolution in the multi-head attention mechanism to downsample the key values ​​during the model stage, and uses normalization to reduce feature loss. The FNN layer uses model compression technology to prune neurons with little effect, thereby obtaining more accurate prediction results and faster inference time on ARM architecture devices. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart of a continuous motion estimation method for finger joints based on sEMG according to the present invention.

[0037] Figure 2 This is a flowchart illustrating the prediction deployment process for estimating continuous finger joint movements in this embodiment.

[0038] Figure 3 This is a structural diagram of the prediction algorithm model for estimating continuous finger joint motion in the embodiment.

[0039] Figure 4 for Figure 3 A diagram illustrating the predicted results. Detailed Implementation

[0040] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and in the above drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0042] As Figure 1 shown, the present application provides a finger joint continuous motion estimation method based on sEMG, comprising the following steps:

[0043] S1, acquiring the arm surface electromyogram signal of finger multi-joint continuous motion and the real-time angle data of finger joint, and preprocessing the surface electromyogram signal.

[0044] S2, calculating the root mean square value of time sequence characteristics of the preprocessed surface electromyogram signal, and constructing a time sequence characteristic vector according to the calculated root mean square value of time sequence characteristics. In this step, the sampling frequency after feature extraction needs to be consistent with the sampling frequency of joint angle.

[0045] Specifically, in this step, a Butterworth filter is used to perform noise reduction processing on the electromyogram signal, and the electromyogram signal and the joint angle are aligned.

[0046] The function of the Butterworth filter satisfies:

[0047]

[0048] Wherein, n is the filter order, ω c is the cut-off frequency.

[0049] S3, normalizing the time sequence characteristic vector and the real-time angle data of the finger joint respectively to improve the convergence speed and prediction accuracy of the model. Construct a training data pair according to the normalized time sequence characteristic vector and the corresponding finger joint real-time angle.

[0050] Specifically, the normalization formula is:

[0051]

[0052] Wherein is the sample value after normalization, x max is the maximum value of joint angle, xmin is the minimum value of the sample data set.

[0053] S4, the time series feature vector in the training data pair is taken as input, and the corresponding real-time finger joint angle is taken as output, and the improved Transformer neural network model is trained. Wherein, the improved Transformer neural network model is set as: 1. Replace the self-attention mechanism in Transfromer with an efficient attention mechanism. 2. Prune the hidden layer neurons of the FNN layer in Transfromer to improve the running efficiency of the model. 3. In this embodiment, the improved Transformer neural network model is deployed offline on the Raspberry Pi, and the inference time and computing power of the model are measured. Further, the efficiency of the improved Transformer neural network is reflected in the downsampling of the attention mechanism and the pruning of the FNN layer. The efficiency of the attention mechanism is reflected in that before performing QKV operation, the input features are processed by two-dimensional convolution (conv) for downsampling, thereby reducing the calculation density and improving the calculation efficiency.

[0054] The normalized feature vector and the finger joint angle data are taken as input and output of the improved Transformer neural network respectively, and the model is trained, and finally the prediction result is obtained on the test set, and the saved model is continuously deployed on the ARM device Raspberry Pi to test the inference time, so as to ensure the unity of accuracy and delay. The improved Transformer neural network mainly consists of an encoding layer, an attention mechanism layer and an FNN layer. This step specifically includes:

[0055] First, the time series feature vector is encoded:

[0056]

[0057]

[0058] Where pos represents the value of the electromyographic signal, i represents the position of the electromyographic signal, and l represents the dimension of the electromyographic signal. t (pos,2i) represents the encoding vector of the electromyographic signal at the even position, P t (pos,2i+1) represents the encoding vector of the electromyographic signal at the odd position.

[0059] Then, efficient multi-head attention operation is performed on the encoded vector:

[0060] x c =conv(x)

[0061] x l =LayerNorm(x c )

[0062]

[0063] where x represents the encoded electromyography signal, x c represents the feature vector of the electromyography signal after convolution extraction, d kk is the dimension of the encoded feature, x l represents the normalized feature vector, Q, K, and V represent the results of linear transformation of the input feature vector in the attention mechanism.

[0064] Subsequently, the output feature is calculated by the model compression feedforward neural network:

[0065] FNN(x) = ∑GELU(XW fc1 )W fc2 +b

[0066] where X is the feature vector output by the attention mechanism layer, W fc1 represents the first layer of full connection network, W fc2 represents the second layer of full connection network, the full connection network refers to the linear layer in the neural network, and b represents the bias.

[0067] The unimportant hidden layer neurons are pruned by the FNN layer to improve the running efficiency of the model. This step encodes the operation selection by defining a series of binary gating units. The gating unit is sampled from the Bernoulli distribution and has a learnable parameter theta which can be automatically optimized by gradient.

[0068] In this step, two-dimensional convolution is used for down-sampling, and the ratio of convolution down-sampling is set. According to the actual needs, the key-value pair K and V are processed by multiple down-sampling. Wherein, K and V refer to the Tokens after the Transformer encoding.

[0069] key, value = Conv2D(x)

[0070] where x is the input feature.

[0071] Further, the model training step of the present application further comprises pre-training the model to obtain the weight of the effective neuron of the FNN layer, and then formally training the model. This step first presets a number of hidden layer neurons, and the purpose of pre-training is to find the fine-grained expansion rate of each FNN layer, and then formal training is performed. The formal training loads the fine-grained expansion rate obtained by pre-training into the FNN layer to complete the pruning of the FNN layer.

[0072] Specifically, pre-training refers to a prior training on a large-scale dataset, aiming to train the model weight of the learnable variable of the FNN layer, and read the pre-trained weight in the subsequent task for formal training. The pre-training data is mainly pre-trained on a large scale using data of multiple subjects. (For example, the DB of all individuals in the Ninapro database is used for pre-training.)

[0073] This step performs model compression on the FNN layer, uses a Bernoulli distribution to determine the effect of the hidden layer neurons on the neural network, and prunes the unimportant neurons. Specifically, the binary gate performs automatic optimization of the neural network gradient through the Bernoulli distribution, and the possibility of selecting a certain neuron is determined by sampling in the distribution.

[0074] The Bernoulli distribution is as follows:

[0075] g t ~ Ber (sigmoid (a p ))

[0076] where g t represents a binary gating unit, a p represents the evaluation result of the number of neurons.

[0077] The formal training refers to training directly on a labeled dataset on a specific task, aiming to directly optimize the performance of the model on the task. The formal training reads the weight parameters of the FNN layer in the pre-training, and then trains on a single individual.

[0078] After the model training is completed, the present application further adds a post-processing step, which can smooth the curve, improve the model accuracy, and obtain a lower error value.

[0079]

[0080] where w is a sliding window, and X is the joint angle predicted by the model output.

[0081] S5, based on the trained improved Transformer neural network model, finger joint motion estimation is performed. Further, this step further includes offline deployment of the trained model, deployment of the model on an ARM device, and measurement of the inference time of a window operation.

[0082] In offline deployment on ARM devices, the trained model is saved offline, and the saved model is placed on the burned ARM device Raspberry Pi for offline testing. The test generally measures the inference time of the length of a single window of the electromyographic signal, and measures the accuracy and time consumption of the smoothing model. We use AME (ARM Model Efficient) as the evaluation index:

[0083]

[0084] where Infer_Time is the inference time of an electromyographic signal window input model, and Power refers to the power consumption of the Raspberry Pi running the model.

[0085] The scheme and effect of the application will be further described through a specific application example.

[0086] As shown in Figures 2-4 , the embodiment provides an unsupervised vehicle re-identification method for cross-domain multi-semantics knowledge learning, including the following steps:

[0087] Step 1, an electromyographic device collects sEMG signals of finger joints.

[0088] Step 2, a data glove is used to collect real-time angle information of finger joints.

[0089] The data collected by the data glove is real-time synchronized with the collected sEMG signals of finger joints, and the data is aligned.

[0090] Step 3, a Butterworth filter is used to perform noise reduction processing on the electromyographic signals.

[0091] The function of the Butterworth filter satisfies:

[0092]

[0093] where n is the filter order, and ω c is the cut-off frequency.

[0094] Step 4, time domain feature extraction is performed on the preprocessed electromyographic signals.

[0095] The time domain feature extraction mainly uses the RMS (Root Mean Square) extraction method, which mainly amplifies the force of the electromyographic signal, and its formula is as follows:

[0096]

[0097] where n i represents the current feature value, and represents the average value.

[0098] Step 5, normalize the data extracted from the features to improve the convergence speed and prediction accuracy of the model:

[0099]

[0100] where is the normalized sample value, x max is the maximum value of the joint angle, x min is the minimum value of the sample data set.

[0101] Step 6, normalize the feature vector and finger joint angle data as input and output of the improved Transformer neural network, and finally get the prediction result on the test set, and save the model for continuous deployment on ARM device Raspberry Pi, test its inference time to ensure the accuracy and delay of the unified. The Transformer model is as follows, first encode the time series feature vector:

[0102]

[0103]

[0104] Then perform efficient multi-head attention operation on the encoded vector:

[0105] x c = conv(x)

[0106] x l = LayerNorm(x c )

[0107]

[0108] Then perform model compression feedforward neural network calculation on the output features:

[0109]

[0110] Trim unimportant hidden layer neurons through FNN layer to improve model running efficiency.

[0111] Step 7, save the weights of the trained model.

[0112] Step 8, use Pytorch tools to perform offline processing on the model.

[0113] Step 9, deploy the model offline on Raspberry Pi 4B model, measure the inference time of the model real-time prediction.

[0114] Step 10, get the real-time joint angle prediction result.

[0115] The application further provides a finger joint continuous motion estimation system based on sEMG, comprising:

[0116] a data acquisition unit configured to acquire arm surface electromyography signals of finger multi-joint continuous motion and real-time angle data of finger joints, and pre-process the surface electromyography signals;

[0117] a feature extraction unit configured to perform time sequence feature root mean square value calculation on the pre-processed surface electromyography signals, and construct a time sequence feature vector according to the calculated time sequence feature root mean square value;

[0118] a normalization unit configured to perform normalization processing on the time sequence feature vector and the real-time angle data of the finger joints respectively, and construct a training data pair according to the normalized time sequence feature vector and the corresponding finger joint real-time angle;

[0119] a training unit configured to take the time sequence feature vector in the training data pair as input and the corresponding finger joint real-time angle as output, and train an improved Transform neural network model;

[0120] a motion estimation unit configured to perform finger joint motion estimation based on the trained improved Transform neural network model.

[0121] For the finger joint continuous motion estimation system based on sEMG, since it corresponds to the finger joint continuous motion estimation method based on sEMG in the above embodiment, the description is relatively simple, and the relevant similarities can be seen from the description of the finger joint continuous motion estimation method based on sEMG in the above embodiment, which will not be described in detail here.

[0122] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the application.

Claims

1. A sEMG-based finger joint continuous motion estimation method, characterized in that, The method comprises the following steps: Obtaining arm surface electromyography signals of multi-joint continuous movement of fingers and real-time angle data of finger joints, and preprocessing the surface electromyography signals; Calculating the root mean square value of the time sequence characteristics of the preprocessed surface electromyography signals, and constructing a time sequence characteristic vector according to the calculated root mean square value of the time sequence characteristics; Normalizing the time sequence characteristic vector and the real-time angle data of the finger joints respectively, and constructing a training data pair according to the normalized time sequence characteristic vector and the corresponding real-time angle of the finger joints; Training an improved Transformer neural network model by taking the time sequence characteristic vector in the training data pair as input and the corresponding real-time angle of the finger joints as output, wherein the improved Transformer neural network model comprises an encoding layer, an attention mechanism layer and an FNN layer, the encoding layer is used for encoding processing of the time sequence characteristic vector, the attention mechanism layer is used for multi-head attention operation on the encoded time sequence characteristic vector, and the FNN layer is used for feedforward neural network calculation of model compression on the characteristic vector after attention operation, and the calculation formula is: wherein is a feature vector output by the attention mechanism layer, represents the first layer fully connected network, represents the second layer fully connected network, the fully connected network is a linear layer in the neural network, denotes a bias quantity, Filtering hidden layer neurons through the FNN layer to obtain effective neuron information, including: Using Bernoulli distribution to evaluate the hidden layer neurons to the neural network, and performing pruning operation on unimportant neurons, wherein the Bernoulli distribution is as follows: wherein, represents a binary gating unit, represents a number evaluation result of neurons; Performing finger joint movement estimation based on the trained improved Transformer neural network model.

2. The sEMG-based finger joint continuous motion estimation method according to claim 1, wherein, Training the improved Transformer neural network model, including: Encoding the time sequence characteristic vector; Performing multi-head attention operation on the encoded time sequence characteristic vector; Filtering hidden layer neurons through the FNN layer to obtain effective neuron information; Pre-training the improved Transformer neural network model based on a public data set to obtain the weights of the effective neurons of the FNN layer, loading the weights of the effective neurons obtained through the improved Transformer neural network model, and then formally training the neural network model with the loaded weights of the effective neurons based on a training data set.

3. The sEMG-based finger joint continuous motion estimation method according to claim 1, wherein, Training the improved Transformer neural network model by taking the time sequence characteristic vector in the training data pair as input and the corresponding real-time angle of the finger joints as output, including: After the model training is completed, the following smoothing processing is performed on the model output curve: wherein, is a sliding window, is a model output predicted joint angle.

4. The sEMG-based finger joint continuous motion estimation method according to claim 1, wherein, The method further comprises deploying the trained improved Transformer neural network model offline to an ARM device, and measuring the inference time of one window of the model.

5. The sEMG-based finger joint continuous motion estimation method according to claim 1, wherein, The method further comprises filtering the surface electromyography signals through a Butterworth filter, and sampling and aligning the denoised electromyography signals and joint angles.

6. A sEMG-based finger joint continuous motion estimation system, characterized by, The method further comprises filtering the surface electromyography signals through a Butterworth filter, and sampling and aligning the denoised electromyography signals and joint angles. The data acquisition unit is configured to acquire the arm surface electromyography signals of multi-joint continuous movement of fingers and real-time angle data of finger joints, and pre-process the surface electromyography signals; The feature extraction unit is configured to perform time sequence feature root mean square value calculation on the pre-processed surface electromyography signals, and construct a time sequence feature vector according to the calculated time sequence feature root mean square value; The normalization unit is configured to perform normalization processing on the time sequence feature vector and the real-time angle data of the finger joints respectively, and construct a training data pair according to the normalized time sequence feature vector and the corresponding real-time angle of the finger joints; The training unit is configured to take the time sequence feature vector in the training data pair as input and the corresponding real-time angle of the finger joints as output, and train an improved Transformer neural network model; the improved Transformer neural network model comprises an encoding layer, an attention mechanism layer and an FNN layer, the encoding layer is configured to perform encoding processing on the time sequence feature vector, the attention mechanism layer is configured to perform multi-head attention operation on the encoded time sequence feature vector, and the FNN layer is configured to perform feedforward neural network calculation of model compression on the feature vector after the attention operation, and the calculation formula is: wherein is a feature vector output by the attention mechanism layer, represents the first layer fully connected network, represents the second layer fully connected network, the fully connected network is a linear layer in the neural network, represents a bias amount, The FNN layer is used to screen hidden layer neurons to obtain effective neuron information, including: The Bernoulli distribution is used to evaluate the hidden layer neurons to the neural network, and a pruning operation is performed on unimportant neurons, and the Bernoulli distribution is as follows: wherein, represents a binary gating unit, represents a number evaluation result of neurons; The motion estimation unit is configured to perform finger joint motion estimation based on the trained improved Transformer neural network model.

Images