A hand continuous motion estimation method combining multi-scale convolution and attention mechanism

Abstract

The application is a hand continuous motion estimation method combining multi-scale convolution and attention mechanism, discloses a method for continuously estimating human joint angles based on surface electromyography signals, and belongs to the field of human-computer interaction. The method comprises the following steps: the electromyography signals and joint angle signals of the joint parts of the object to be predicted are pretreated respectively to realize the one-to-one correspondence of the electromyography signal and joint angle signal sampling time; the electromyography signals and joint angle signals in the training set are used as samples for input to construct a prediction model, the prediction model is trained, and a trained prediction model is obtained; the electromyography signals of the object to be predicted are collected in real time, and input into the trained prediction model to predict the joint angle; the angle signal output by the above prediction is processed in real time to obtain the required joint angle signal. Through the application, the motion of the surface electromyography signal mapping to the joint angle is realized in a continuous manner, the real-time performance of continuous estimation is ensured, the prediction accuracy is improved, and the calculation complexity is reduced.

Description

Technical Field

[0001] This invention belongs to the field of biological signal processing technology and relates to a method for estimating continuous hand motion by combining multi-scale convolution and attention mechanisms. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology and neuroscience, analyzing bioelectrical signals of human motor intentions has become more feasible. Surface electromyography (SEMG) has attracted widespread attention due to its rich motor information, ease of acquisition, and non-invasive nature. Therefore, continuous motion estimation based on SEMG has become an important technical route for realizing next-generation human-computer interaction.

[0003] Currently, research on intention recognition mainly falls into two categories: the classification of discrete actions and the recognition of movement patterns, and the direct decoding of motor intention information from the central nervous system, such as joint angles, angular velocities, and torques. While research methods are relatively mature in classification, they can only predict a few discrete limb movements. When using these predictions to control rehabilitation robots, they cannot move as freely as human joints. Ensuring continuous matching of human-robot movements is a prerequisite for safe control of rehabilitation robots. Therefore, estimating continuous joint movements through surface electromyography (SEMG) signals has a positive effect on patient rehabilitation training. This invention uses SEMG signals to identify the angles of multiple joints in the lower limbs. The estimation results can serve as reference inputs for controlling the robot, improving its autonomous adaptability, providing a safe and comfortable training environment for patients, and promoting the development of human-robot interaction. Summary of the Invention

[0004] This invention provides a method for estimating the angles of multiple joints of the hand based on surface electromyography signals, in order to overcome the technical problems of existing continuous motion estimation methods, such as insufficient accuracy and limited range of motion.

[0005] To achieve the above objectives, the technical solution of the present invention is: a method for estimating continuous hand motion combining multi-scale convolution and attention mechanisms, comprising the following steps:

[0006] Step 1: Record the surface electromyography (EMG) signal and joint angle signal of the joint using a preset sampling frequency;

[0007] Step 2: Divide the electromyography (EMG) signals and joint angle signals of the object to be predicted into training set and test set, and perform preprocessing on each set to ensure that the sampling time of the EMG signals and joint angle signals corresponds one-to-one.

[0008] Step 3: Use the electromyography signals and joint angle signals in the training set as samples for input to build a prediction model, train the prediction model, and obtain a trained prediction model.

[0009] Step 4: Collect electromyographic signals of the object to be predicted in real time and input them into the trained prediction model to predict joint angles.

[0010] Step 5: Perform real-time smoothing on the predicted angle signal to obtain the required joint angle signal, thereby achieving continuous estimation of the human joint angle.

[0011] Furthermore, the collected surface electromyography signals and joint angle signals were divided into two sets: the first 60% of the data was used as the training set, and the last 40% of the data was used as the test set.

[0012] Furthermore, the preprocessing of the segmented electromyographic signals includes:

[0013] First, a filter is used to denoise the acquired electromyographic signals;

[0014] Then the joint angle signal is resampled to synchronize it with the surface electromyography signal;

[0015] The electromyographic features in the denoised electromyographic signal are then extracted using the sliding window technique, and these features are normalized to achieve the preprocessing of the electromyographic signal.

[0016] Furthermore, the denoising of the acquired electromyographic signals using filters specifically includes:

[0017] First, the acquired electromyography signals were subjected to notch filtering to remove the 50Hz and its harmonic components.

[0018] Next, Butterworth high-pass filtering is used to remove motion artifacts in the notch filter processed signal in order to eliminate the influence of low-frequency motion of the object to be predicted on the signal.

[0019] Furthermore, the sliding window technique is used to extract features using three methods: zero-order moment, fluctuation expectation, and unbiased standard deviation within the sliding window. The extracted features are then fused, and the fused features are used as electromyographic features.

[0020] Furthermore, the prediction model includes a multi-scale feature extraction module that adaptively extracts the spatial features of the feature signals by extracting the temporal features of the fused feature signals through a channel attention mechanism.

[0021] The output fused signal is sliced, and each slice is treated as a token and linearly projected onto the slice processing module of the model dimension d through an embedding matrix E.

[0022] A three-layer coding module is used to add positional encoding to the mapped slice sequence to capture positional information.

[0023] The Encoder module captures global information from slice sequences with location information.

[0024] The features extracted by the multi-layer encoder are compressed by the average pooling layer, and then mapped to the target joint angle by the linear layer.

[0025] A smoothing module that performs real-time smoothing of the predicted angle signal.

[0026] Furthermore, the prediction model is trained to obtain the trained prediction model. The specific process is as follows:

[0027] Step 3.1: Use convolutional kernels of different scales to extract the temporal features of the fused feature signal, and then use the channel attention mechanism to adaptively extract the spatial features of the feature signal;

[0028] Step 3.2: Slice the output fused signal, and then treat each slice as a token, linearly projecting it onto the dimension d of the model through an embedding matrix E;

[0029] Step 3.3: Add position encoding to the mapped slice sequence to capture position information;

[0030] Step 3.4: Capture global information from the slice sequence with location information;

[0031] Step 3.5: The features extracted by the multi-layer encoder are compressed through an average pooling layer, and then mapped to the target joint angle through a linear layer.

[0032] Furthermore, the channel attention mechanism first uses global average pooling to extract global features, then employs a one-dimensional convolution operation to learn the attention between channels. An adaptive method is used to determine the size of the convolution kernel k, representing the coverage of local cross-channel interactions. The calculation process of k is as follows:

[0033]

[0034] Where |t| odd Let t represent the nearest odd number of t. In this paper, we set γ and b to 2 and 1, respectively.

[0035] Furthermore, the mapped slice sequence employs a learnable one-dimensional position embedding. To capture location information.

[0036] Furthermore, each encoder layer in the three-layer encoding module includes a multi-head attention module and a multilayer perceptron module; the multi-head attention mechanism allows the model to use multiple attention heads to focus on the components of the input sEMG slice in multiple ways, and the multilayer perceptron further filters the features extracted by MSA.

[0037] Furthermore, in step 5, the predicted joint angle signal will be smoothed, and the specific calculation formula is as follows:

[0038]

[0039] s0=y′1+y′2+y′3

[0040] s t =αy′ t +(1-α)s t-1

[0041] Where: y is the output of the linear layer, L is the window length, and α is the smoothing coefficient.

[0042] Beneficial Effects: This invention provides a method for estimating continuous hand motion combining multi-scale convolution and attention mechanisms. First, it extracts and fuses multiple features from the sEMG (semi-EMG) image, providing rich prior knowledge for subsequent model training. Then, the fused feature signal is fed into a multi-scale convolutional fusion module to extract local features at different scales, including temporal and spatial dimensions. Next, the fused signal is sliced ​​and linearly mapped, followed by positional encoding. The sliced ​​sequence with positional information is then fed into an encoder to capture global information, and an average pooling layer is used to compress the feature information. The compressed features are then used to map joint angle signals, and a smoothing module performs smoothing to obtain the final predicted joint angle signal, resulting in a prediction with higher accuracy, stronger robustness, and better real-time performance. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating the surface electromyography signal feature processing and joint angle prediction method in an embodiment of the present invention.

[0045] Figure 2 This is a schematic diagram illustrating the estimation of hand joint angles using preprocessed surface electromyography signals in this invention.

[0046] Figure 3 This is a diagram of the predictive network structure in this invention;

[0047] Figure 4 This is a structural diagram of the channel attention in the convolutional fusion module of this invention;

[0048] Figure 5 The test results are from Experiment 1, which is an example of an implementation of the present invention. Detailed Implementation

[0049] To provide a clearer understanding of the objectives, technical solutions, and advantages of this invention, the embodiments of the invention will be described in detail below with reference to the accompanying drawings. It should be noted that the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, those skilled in the art can obtain other embodiments of the invention without creative work, and these embodiments all fall within the scope of protection of this invention.

[0050] The technical solutions of this application will be described in detail below through different embodiments.

[0051] This embodiment provides a method for estimating the angles of multiple joints in the hand based on surface electromyography signals.

[0052] Figure 1 This is a flowchart illustrating the surface electromyography signal feature processing and joint angle prediction method in this embodiment of the invention; it includes the following steps:

[0053] Step 1: Record the surface electromyography (EMG) signal and joint angle signal of the joint using a preset sampling frequency;

[0054] Step 2: Divide the electromyography (EMG) signals and joint angle signals of the object to be predicted into training set and test set, and perform preprocessing on each set to ensure that the sampling time of the EMG signals and joint angle signals corresponds one-to-one.

[0055] Step 3: Use the electromyography signals and joint angle signals in the training set as samples for input to build a prediction model, train the prediction model, and obtain a trained prediction model.

[0056] Step 4: Collect electromyographic signals of the object to be predicted in real time and input them into the trained prediction model to predict joint angles.

[0057] Step 5: Perform real-time smoothing on the predicted angle signal to obtain the required joint angle signal, thereby achieving continuous estimation of the human joint angle.

[0058] Figure 2This is a schematic diagram illustrating the estimation of hand joint angles using preprocessed surface electromyography signals in this invention.

[0059] Steps 1, 2, 3, 4, and 5 are executed sequentially.

[0060] In a specific implementation example, in step 1, the human joint to be predicted is selected, and electromyography (EMG) signals and joint angle signals of that joint are acquired. The sEMG is recorded using a Delsys Trigno wireless system with 12 wireless sEMG electrodes, and hand movements are measured using a Cyber ​​Glove II with 22 sensors. The sampling rates of the Cyber ​​Glove II and Delsys are 20Hz and 2000Hz, respectively.

[0061] In a specific implementation example, in step 2, we first divide the acquired surface electromyography (EMG) signals and joint angle signals, using the first 60% of the data as the training set and the last 40% as the test set. Then, we perform a series of operations on the divided surface EMG signals, including filtering, feature extraction, and normalization, and synchronize the EMG signals with the angle signals, as detailed below:

[0062] In step 2.1, we perform filtering on the sEMG signal. The electromyography signal denoising specifically includes: first, notch filtering is performed on the acquired electromyography signal to remove the 50Hz and integer multiples of the frequency components in the electromyography signal; then, Butterworth high-pass filtering is used to remove motion artifacts in the notch filtered signal to eliminate the influence of low-frequency motion of the object to be predicted on the signal.

[0063] In step 2.2, we extract features from the sEMG signal and use a sliding window technique to calculate the time-series features of the surface electromyography (sEMG) signal, including the zero-order moment, the expected rate of amplitude change, and the unbiased standard deviation. The sliding window size is set to 100 ms, and the sliding step is set to 0.5 ms to minimize the human-computer interaction delay while preserving the original information of the EMG signal. The formulas for calculating the zero-order moment, the expected rate of amplitude change, and the unbiased standard deviation are as follows:

[0064]

[0065] Where N represents the length of the sliding window, and x[i], i = 1, 2, ..., N represents the electromyographic signal with a time length of N.

[0066]

[0067] Where Δ 2 It represents the second derivative.

[0068]

[0069] in This represents the expected value of the electromyographic sample.

[0070] In step 2.3, after obtaining the processed electromyographic signal features, the maximum-minimum normalization method is used to ensure that the values ​​are between [0,1], and the minimum value is recorded for normalization of the test data. The calculation formula is as follows:

[0071]

[0072] In step 2.4, since the sampling rates of the extracted sEMG and the joint angle signal are different, we use linear interpolation to interpolate the joint angle signal so that it corresponds one-to-one with the sEMG in time.

[0073] Figure 3 This is a diagram of the predictive network structure in this invention;

[0074] The prediction model includes: a multi-scale convolutional fusion module that adaptively extracts the spatial features of the feature signals by using a channel attention mechanism to extract the temporal features of the fused feature signals;

[0075] The output fused signal is sliced, and each slice is treated as a token and linearly projected onto the slice encoding module of the model in dimension d through an embedding matrix E.

[0076] A three-layer Encoder module adds position encoding to the mapped slice sequence to capture position information;

[0077] A capture module that captures global information from slice sequences with location information attached;

[0078] The features extracted by the multi-layer encoder are compressed by the average pooling layer, and then mapped to the target joint angle by the linear layer.

[0079] It also includes a smoothing module, which performs real-time smoothing on the angle signal of the above-predicted output. It is mainly used on the test set and not used during model training.

[0080] Figure 4 This is a structural diagram of the channel attention in the convolutional fusion module of this invention;

[0081] In a specific implementation example, in step 3, the synchronized sEMG feature signal and joint angle signal are divided into 100ms (200 sample points) segments as samples and input into the proposed prediction model (sMCF-PEN). These segments are then processed sequentially through a multi-scale convolutional fusion module, a slice encoding module, an encoder module, and a smoothing module. The specific steps are as follows:

[0082] In step 3.1, the fused electromyographic signal X is first fed into the multi-scale convolutional fusion module. First, local information of the feature signal is extracted in the temporal direction by convolutional kernels of different sizes, with the kernel sizes being 1×3, 1×5, and 1×7, respectively. Then, it is activated by the Elu activation function and then enters the channel attention module ECA to adaptively extract the spatial information between signal channels.

[0083] Furthermore, ECA can enhance the channel features of the input feature map without changing its size. It first uses global average pooling to extract global features, and then employs one-dimensional convolution to learn the attention between channels. The choice of the one-dimensional convolution kernel size is particularly important, as it represents the coverage of local cross-channel interactions. Here, an adaptive method is used to determine the size of the convolution kernel k, calculated as follows:

[0084]

[0085] Where |t| odd Let t represent the nearest odd number of t. In this paper, we set γ and b to 2 and 1, respectively.

[0086] Finally, the information from each feature branch is directly summed to obtain the multi-scale feature X′, calculated as follows:

[0087]

[0088] in (·) represents a convolution operation with C convolution kernels of size n1×1, σ(·) is the ELU activation function, and E represents the ECA channel attention module.

[0089] In step 3.2, before feeding the multi-scale feature fused signal X′ into the encoder, we divide it into several non-overlapping patches, denoted as... Each patch has a length of C, so the number of patches is m = W / C. We then treat each patch as a token. Unlike many Transformer-based models, we don't treat each time step as a token; instead, we aggregate time steps into sEMG subsequence-level patches. This not only enhances locality but also captures comprehensive semantic information unavailable at the point level. Each token is then linearly projected onto the model's dimension d using an embedding matrix E.

[0090] Furthermore, since sEMG signals have a specific order, changing this order may alter the meaning of the input. However, the transformer architecture does not model positional information, thus requiring explicit encoding of the input sequence order. We employ a learnable one-dimensional position embedding matrix. To capture location information. The generated patch sequence with location information can be represented as:

[0091]

[0092] In step 3.3, we feed the patch containing the location information into the encoder. We use N identical encoders to extract relevant information from the sEMG. Each encoder consists of a multi-head self-attention block (MSA) and a multilayer perceptron (MLP). For ease of description, we use Z... l (l=1,2…N) represents the input of each encoder layer.

[0093] Furthermore, in step 3.3, the multi-head attention module consists of multiple self-attention layers. The function of the self-attention layer is to capture the correlation between different vectors in the electromyography feature sequence and aggregate global contextual information to update each component of the sequence. Therefore, for a series of input segments Z... l We first transform the input vector into three distinct vectors: the query vector q, the key vector k, and the value v, each with dimension d. q =d k =d v =D. Vectors from different inputs are packed into three distinct matrices: Q, K, and V. The attention function between the different input vectors is calculated as follows:

[0094]

[0095] Multi-head attention allows a model to focus on the components of the input sEMG patch in different ways using multiple attention heads. Assume there are h heads, each with dimension d. h =D / h. First, we calculate the output of each attention head. i Then, the outputs of multiple attention heads are concatenated, and finally the result is projected into a matrix. This process can be achieved in the following way:

[0096] head i =Attention(Q) i K i V i )

[0097] Multihead=Concat(head1...head h W

[0098] Q i K i V i Let Q, K, V, i = {1, 2, ..., h} be the values ​​in the calculation process of the i-th attention head. It is a linear projection matrix.

[0099] To mitigate network degradation and accelerate training, this module adds skip connections and layer normalization operations, as shown in the following formula:

[0100] Z l =Multihead(LN(Z) l-1 ))+Z l-1

[0101] Furthermore, the output Z of the multi-head attention module l The input Z is fed into a Multilayer Perceptron (MLP), which consists of two linear transformation layers, one leakage layer, and a nonlinear activation function called the Gaussian Error Linear Unit (GELU). The purpose is to map the input Z to a higher-dimensional space, then filter it using the GELU nonlinear function, and finally transform it back to the original dimension. Therefore, an MLP can be described as:

[0102] MLP(Z l )=W2δ(W1Z l )

[0103] Here, W1 and W2 represent the parameter matrices of the two linear layers, and δ represents the GELU activation function. Similar to the multi-head attention module, a residual block and a normalization layer are also added here:

[0104] Z′ l =MLP(LN(Z) l ))+Z l

[0105] In step 3.4, in order to convert the output of step 3.3 into the required joint angle, average pooling is first performed to further compress the features, and then a linear layer is used for prediction to obtain the predicted joint angle y.

[0106] y = Linear(AvgPooling(Z′) l ))

[0107] In a specific implementation example, step 5 involves smoothing the predicted joint angle signals. This module can use a small number of historical joint angles to process some predictions with larger errors, making them more consistent with actual human movement and improving the model's robustness. The specific implementation method is as follows:

[0108]

[0109] s0=y′1+y′2+y′3

[0110] s t =αy′ t +(1α)st-1

[0111] Where y is the output of MCF-PEN, L represents the window length, and α represents the smoothing coefficient. We set L and α to 4 and 0.3, respectively.

[0112] Furthermore, to verify the effectiveness of the continuous motion information prediction model obtained in this embodiment, experimental verification was conducted. All experiments were implemented using the PyTorch framework and validated on the publicly available Ninapro dataset.

[0113] Experiment 1: Ten representative subjects from Ninapro were selected. The Pearson correlation coefficient (CC) between the predicted curve and the actual curve for each subject on the test set was calculated. For example... Figure 5 As shown, the prediction model (sMCF-PEN) proposed in this embodiment is significantly better than other existing algorithms, such as TCN and LSTM models.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some or all of the technical features. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for estimating continuous hand motion combining multi-scale convolution and attention mechanisms, characterized in that, Includes the following steps: Step 1: Record the surface electromyography (EMG) signal and joint angle signal of the joint using a preset sampling frequency; Step 2: Preprocess the electromyography (EMG) signals and joint angle signals of the object to be predicted by collecting them and divide them into training set and test set to achieve a one-to-one correspondence between the sampling time of EMG signals and joint angle signals. Step 3: Use the electromyography signals and joint angle signals in the training set as samples for input to build a prediction model, train the prediction model, and obtain a trained prediction model. Step 4: Collect electromyographic signals of the object to be predicted in real time and input them into the trained prediction model to predict joint angles. Step 5: Perform real-time smoothing on the predicted angle signal to obtain the required joint angle signal, thereby realizing continuous estimation of the human joint angle. The prediction model includes: a multi-scale feature extraction module that extracts the spatial features of the fused feature signals by means of a channel attention mechanism to adaptively extract the spatial features of the feature signals; The output fused signal is sliced, and each slice is treated as a token and linearly projected onto the slice processing module of the model dimension d through an embedding matrix E. A three-layer coding module is used to add positional encoding to the mapped slice sequence to capture positional information. The Encoder module captures global information from slice sequences with location information. The features extracted by the multi-layer encoder are compressed by the average pooling layer, and then mapped to the target joint angle by the linear layer. A smoothing module that performs real-time smoothing of the predicted angle signal.

2. The method as described in claim 1, characterized in that... In step 2, the preprocessing of the segmented electromyographic signals includes: First, a filter is used to denoise the acquired electromyographic signals; Then the joint angle signal is resampled to synchronize it with the surface electromyography signal; The electromyographic features in the denoised electromyographic signal are then extracted using the sliding window technique, and these features are normalized to achieve the preprocessing of the electromyographic signal.

3. The method as described in claim 2, characterized in that... The denoising of the acquired electromyographic signals using filters specifically includes: First, the acquired electromyography signals were subjected to notch filtering to remove the 50Hz and its harmonic components. Next, Butterworth high-pass filtering is used to remove motion artifacts in the notch filter processed signal in order to eliminate the influence of low-frequency motion of the object to be predicted on the signal.

4. The method as described in claim 2, characterized in that... The sliding window technique is used to extract features using three methods: zero-order moment, fluctuation expectation, and unbiased standard deviation within the sliding window. The extracted features are then fused, and the fused features are used as electromyographic features.

5. The method as described in claim 1, characterized in that... The specific process of training the prediction model to obtain the trained prediction model is as follows: Step 3.1: Use convolutional kernels of different scales to extract the temporal features of the fused feature signal, and then use the channel attention mechanism to adaptively extract the spatial features of the feature signal; Step 3.2: Slice the output fused signal, and then treat each slice as a token, linearly projecting it onto the dimension d of the model through an embedding matrix E; Step 3.3: Add position encoding to the mapped slice sequence to capture position information; Step 3.4: Capture global information from the slice sequence with location information; Step 3.5: The features extracted by the multi-layer encoder are compressed through an average pooling layer, and then mapped to the target joint angle through a linear layer.

6. The method as described in claim 1, characterized in that... The channel attention mechanism first uses global average pooling to extract global features, then employs a one-dimensional convolution operation to learn the attention between channels. An adaptive method is used to determine the size of the convolution kernel k, which represents the coverage of local cross-channel interactions. The calculation process of k is as follows: in Let t represent the nearest odd number. , Set it to 1.

7. The method as described in claim 5, characterized in that, The mapped slice sequence employs a learnable one-dimensional position embedding. To capture location information.

8. The method as described in claim 1, characterized in that, Each encoder layer in the three-layer encoding module includes a multi-head attention module and a multi-layer perceptron module. The multi-head attention mechanism allows the model to use multiple attention heads to focus on the components of the input sEMG slice in multiple ways, and the multi-layer perceptron further filters the features extracted by MSA.

9. The method as described in claim 1, characterized in that, In step 5, the predicted output angle signal undergoes real-time smoothing processing. The specific calculation formula is as follows: Where y is the output of the linear layer, and L is the window length. It is the smoothing coefficient.

Images