Lower limb continuous motion estimation method based on muscle synergy and deep learning

Abstract

The present invention belongs to the field of biological signal processing and human-computer interaction, specifically a method for estimating continuous motion of lower limbs based on muscle synergy and deep learning. It comprises the following steps: 1) using an electromyographic sensor to collect the electromyographic signals generated by the unilateral leg muscles, and an inertial sensor to obtain the joint motion angle; 2) pre-processing the electromyographic signals; 3) extracting muscle synergy information from the pre-processed electromyographic data through a non-negative matrix decomposition algorithm; 4) dividing the muscle synergy information into training set data and test set data; 5) establishing a deep bidirectional recurrent neural network; and training with the training set data; 6) inputting the test set data into the trained deep bidirectional recurrent neural network to verify the angle regression effect of the trained deep bidirectional recurrent neural network. The present invention extracts muscle synergy information by matrix decomposition, which significantly reduces the number of electrodes and hardware costs compared to the motion unit decomposition method, and is conducive to improving the scope of application and real-time performance.

Description

Technical Field

[0001] The present invention belongs to the field of biological signal processing and human-computer interaction, and specifically provides a method for estimating continuous motion of lower limbs based on muscle synergy and deep learning. Background Art

[0002] Traditional myoelectric motion estimation methods often use pattern recognition or model-based approaches. Pattern recognition-based methods generally include preprocessing, feature extraction, and classification / regression, enabling gesture recognition or continuous estimation of joint angles. Model-based methods generally use kinematics, dynamics, or skeletal muscle models to model the human musculoskeletal system. These methods require model parameter identification and can be used to estimate motion parameters such as force and angle. These methods estimate human intention at the muscle level or at the myoelectric signal level, but lack in-depth exploration of the underlying neural mechanisms.

[0003] Recently, methods based on motor unit decomposition have emerged to estimate human movement intentions. These methods use high-density array electromyographic (EMG) signals to decompose the firing moments of motor units, known as motor unit spike trains (MUSTs), for gesture recognition or continuous motion estimation. This method decomposes the neural encoding information generated by the muscles and spinal cord during movement to achieve motion estimation. However, this method has significant drawbacks: first, it requires expensive high-density EMG equipment; second, the large number of EMG channels increases the computational power requirements of the system; noise, skin, and fat impedance cause waveform distortion in many motor units, resulting in only 20% to 30% of motor units being correctly decomposed; and finally, MUST algorithms often require extensive matrix operations, further increasing the computational cost and hardware requirements. Summary of the Invention

[0004] In response to the above problems, the purpose of the present invention is to provide a method for realizing movement intention estimation by extracting neural information of the central nervous system using sparse channel electromyographic signals. Muscle synergy is a high-level neural control information that can be obtained by surface electromyography decomposition. The present invention first extracts muscle synergy information from electromyography, and then uses deep learning methods to extract high-level abstract features related to limb movement from the high-level neural information, namely muscle synergy information, to realize the recognition of movement intention. This method has the advantages of a small number of channels, low computational complexity, low hardware requirements and computing power consumption, and has good practical application value.

[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a method for estimating continuous motion of lower limbs based on muscle synergy and deep learning, characterized by comprising the following steps:

[0006] 1) Use electromyographic sensors to collect electromyographic signals generated by unilateral leg muscles during uniform walking, and at the same time, use inertial navigation sensors to obtain joint motion angles;

[0007] 2) Preprocessing the collected electromyographic signals to obtain preprocessed electromyographic data;

[0008] 3) Extract muscle synergy information from the preprocessed EMG data using a non-negative matrix factorization algorithm;

[0009] 4) Divide the muscle synergy information into training set data and test set data;

[0010] 5) Establish a deep bidirectional recurrent neural network; and train the deep bidirectional recurrent neural network using training set data;

[0011] 6) Input the test set data into the trained deep bidirectional recurrent neural network, and verify the angle regression effect of the trained deep bidirectional recurrent neural network by obtaining the root mean square error between the predicted angle and the actual angle collected by the inertial navigation.

[0012] The step 2) comprises the following steps:

[0013] 2-1) Use bandpass filter and notch filter to remove noise interference from EMG signals, and normalize the data by z-score normalization method;

[0014] 2-2) Segment the normalized data using a sliding window with a length of 250 ms and a sliding length of 50 ms to obtain segmented data;

[0015] 2-3) Correct the segmented data to make it non-negative in preparation for the processing of the non-negative matrix decomposition algorithm;

[0016] x 矫正后 =abs(x 矫正前 )

[0017] Wherein, abs() represents the absolute value operation, and x represents the electromyographic data; the corrected electromyographic data is made to have the same format as the data decomposed by the non-negative matrix decomposition algorithm, so that the negative numbers therein are inverted and transformed into positive numbers.

[0018] The corrected format of the electromyographic data is M*T, where M is the number of channels, T is the sampling point length of each sliding window, that is, sampling rate*sliding window time length, and M*T is the non-negative matrix V.

[0019] The step 3) comprises the following steps:

[0020] Decompose the electromyographic data using the non-negative matrix decomposition algorithm and obtain two non-negative matrices W and H, namely:

[0021] V≈WH

[0022] Where V is a non-negative matrix, W is a non-negative matrix of muscle synergy information, in the format of M*N, H is the activation coefficient, in the format of N*T; N is the number of muscle synergies;

[0023] The optimal W and H are obtained by minimizing the Euclidean distance between V and the reconstructed data WH, that is:

[0024]

[0025] Among them, st is the constraint condition, and the muscle synergy information W is solved according to the above formula;

[0026] The dimension of W is set to 2, that is, 2 muscle synergy information is taken.

[0027] The muscle synergy information is divided into training set data and test set data according to a 10-fold cross-validation method.

[0028] In step 5), the deep bidirectional recurrent neural network is established, specifically:

[0029] (1) The network input format of the deep bidirectional recurrent neural network is set to: M*N; where M is the number of electromyographic channels and N is the number of muscle synergies;

[0030] (2) The forward and backward outputs of the bidirectional GRU layer are merged and a DropOut layer with a probability of 0.25, i.e., a DO layer, is added to prevent overfitting;

[0031] (3) Set up three fully connected layers of a deep bidirectional recurrent neural network. Each layer contains a fully connected network FC, a normalized BN operation, and a DO operation. The probability of the DO operation is 0.25. The number of neurons in each layer is 1024, 1024, and 32, respectively, and the ReLU activation function is used.

[0032] (4) Establishing the output layer of a deep bidirectional recurrent neural network to simultaneously output the angles of the hip, knee, and ankle joints;

[0033] (5) Train the deep bidirectional recurrent neural network using training set data.

[0034] The method of training a deep bidirectional recurrent neural network using training set data includes the following steps:

[0035] a. Train the deep bidirectional recurrent neural network using the Adam optimizer, setting the batch size of the training set data in the deep bidirectional recurrent neural network to 128 and the learning rate to 0.001;

[0036] b. Input the training set data into a deep bidirectional recurrent neural network, and the output is the joint angle data, including the angles of the hip joint, knee joint, and ankle joint;

[0037] c. Input the test set data into the deep bidirectional recurrent neural network, and the output is the predicted joint angle data θ i 预测 , including predicted hip, knee, and ankle joint angles.

[0038] The step 6) is specifically as follows:

[0039] The predicted joint angle data θ output by the test set data i 预测 And collect the joint motion angle θ through the inertial sensor i 真实 , get the root mean square error RMSE, that is:

[0040]

[0041] If the root mean square error (RMSE) obtained is smaller than the set control group, the regression effect of the deep bidirectional recurrent neural network after training meets the prediction requirements.

[0042] The present invention has the following beneficial effects and advantages:

[0043] 1. This invention extracts high-level neural control information from the central nervous system level and realizes motion estimation from the neural mechanism level. Compared with classic model-based or pattern recognition methods, it has the advantages of convenient calculation and high interpretability.

[0044] 2. The present invention uses matrix decomposition to extract muscle synergy information. Compared with the motor unit decomposition method, it significantly reduces the number of electrodes and hardware costs, which is conducive to improving the application range and real-time performance.

[0045] 3. The present invention uses 8-channel electrodes to simultaneously achieve synchronous and continuous estimation of the three degrees of freedom of the lower limbs, effectively improving the efficiency and flexibility of motion estimation, and has better practical value in scenarios such as lower limb rehabilitation.

[0046] 4. The present invention uses deep learning methods to decode muscle synergy information and better extract motion-related features from it. Compared with the classical method, the regression error is significantly reduced, the regression accuracy is significantly improved, and the human intention is better decoded. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is the overall operation flow chart of the present invention;

[0048] Figure 2 This is a schematic diagram of the electrode pasting position of the present invention;

[0049] Figure 3 This is a schematic diagram of the joint angle position of the present invention;

[0050] Figure 4 Schematic diagram of the GRU module in the present invention;

[0051] Figure 5 Schematic diagram of the deep regression neural network in the present invention;

[0052] Figure 6 The regression results of the present invention and their comparison with the machine learning method;

[0053] Figure 7 This is a schematic diagram of the angle estimation effect of the present invention; DETAILED DESCRIPTION

[0054] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0055] Figure 1 This is the overall flow chart of this article. The method for estimating lower limb continuous motion based on muscle synergy and deep learning of the present invention includes the following steps:

[0056] 1) Use electromyographic sensors to collect electromyographic signals generated by unilateral leg muscles during uniform walking, and at the same time, use inertial navigation sensors to obtain joint motion angles;

[0057] 2) Preprocessing the collected electromyographic signals to obtain preprocessed electromyographic data;

[0058] 3) Extract muscle synergy information from the preprocessed EMG data using a non-negative matrix factorization algorithm;

[0059] 4) Divide the muscle synergy information into training set data and test set data;

[0060] 5) Establish a deep bidirectional recurrent neural network; and train the deep bidirectional recurrent neural network using training set data;

[0061] 6) Input the test set data into the trained deep bidirectional recurrent neural network, and verify the angle regression effect of the trained deep bidirectional recurrent neural network by obtaining the root mean square error between the predicted angle and the actual angle collected by the inertial navigation.

[0062] First, the skin was wiped with alcohol to remove stains. Noraxon Ultium EMG electrodes were attached to eight muscles in the right leg, and Noraxon myoMotion inertial navigation sensors were placed on the leg and pelvis. The subjects walked at a constant speed of 3.0 km / h, and EMG signals and joint motion angle information were simultaneously collected during walking. The EMG sampling rate was 2000 Hz, and the inertial navigation sampling rate was 100 Hz.

[0063] Preprocessing the collected signals includes: Preprocessing the collected signals, the process is as follows:

[0064] First, the signal is subjected to a bandpass filter and a 50Hz notch filter to remove noise interference. Then, the data is normalized using the z-score normalization method. After that, the data is segmented using a sliding window with a length of 250ms and a sliding length of 50ms. The data is then corrected to meet the non-negativity requirement in preparation for the non-negative matrix factorization (NMF) algorithm. The data obtained in this way meets the prerequisite of NMF non-negativity.

[0065] Among them, the segmented data is corrected to satisfy the non-negativity in preparation for the processing of the non-negative matrix decomposition algorithm;

[0066] x 矫正后 =abs(x 矫正前 )

[0067] Wherein, abs() represents the absolute value operation, and x represents the electromyographic data; the corrected electromyographic data is made to have the same format as the data decomposed by the non-negative matrix decomposition algorithm, so that the negative numbers therein are inverted and transformed into positive numbers.

[0068] The corrected format of the electromyographic data is M*T, where M is the number of channels, T is the sampling point length of each sliding window, that is, sampling rate*sliding window time length, and M*T is the non-negative matrix V.

[0069] Then, NMF is used to decompose the muscle synergy information, a deep regression neural network is designed, and the muscle synergy information is input into the deep network to achieve synchronous regression of the three joint angles of the hip, knee and ankle.

[0070] In this embodiment, the electromyographic data is decomposed by a non-negative matrix decomposition algorithm, and two non-negative matrices W and H are obtained, namely:

[0071] V≈WH

[0072] Where V is a non-negative matrix, W is a non-negative matrix of muscle synergy information, in the format of M*N, H is the activation coefficient, in the format of N*T; N is the number of muscle synergies;

[0073] The optimal W and H are obtained by minimizing the Euclidean distance between V and the reconstructed data WH, that is:

[0074]

[0075] Among them, st is the constraint condition, and the muscle synergy information W is solved according to the above formula;

[0076] The dimension of W is set to 2, that is, 2 muscle synergy information is taken.

[0077] Another solution is to use Kullback-Leibler divergence as the optimization target. The calculation formula is:

[0078] min D(V||WH)stW≥0,H≥0

[0079] in,

[0080]

[0081] Where A in the formula is V, that is, a non-negative matrix, and B is the WH reconstructed data;

[0082] The above two methods can be solved iteratively to obtain the final result.

[0083] The muscle synergy information obtained is divided into training set data and test set data according to the 10-fold cross-validation method.

[0084] Build a deep bidirectional recurrent neural network, specifically:

[0085] (1) The network input format of the deep bidirectional recurrent neural network is set to: M*N; where M is the number of electromyographic channels and N is the number of muscle synergies;

[0086] (2) The forward and backward outputs of the bidirectional GRU layer are merged and a DropOut layer with a probability of 0.25, i.e., a DO layer, is added to prevent overfitting;

[0087] (3) Set up three fully connected layers of a deep bidirectional recurrent neural network. Each layer contains a fully connected network FC, a normalized BN operation, and a DO operation. The probability of the DO operation is 0.25. The number of neurons in each layer is 1024, 1024, and 32, respectively, and the ReLU activation function is used.

[0088] (4) Establishing the output layer of a deep bidirectional recurrent neural network to simultaneously output the angles of the hip, knee, and ankle joints;

[0089] (5) Train the deep bidirectional recurrent neural network using training set data.

[0090] Training a deep bidirectional recurrent neural network using training set data includes the following steps:

[0091] (1) The deep bidirectional recurrent neural network is trained using the Adam optimizer, and the batch size of the training set data in the deep bidirectional recurrent neural network is set to 128, and the learning rate is set to 0.001;

[0092] (2) The training set data is input into a deep bidirectional recurrent neural network, and the output is the joint angle data, including the angles of the hip joint, knee joint, and ankle joint;

[0093] (3) Input the test set data into the deep bidirectional recurrent neural network, and the output is the predicted joint angle data θ i 预测 , including predicted hip, knee, and ankle joint angles.

[0094] The predicted joint angle data θ output by the test set data i 预测 And collect the joint motion angle θ through the inertial sensor i 真实 , get the root mean square error RMSE, that is:

[0095]

[0096] If the root mean square error (RMSE) obtained is smaller than the set control group, the regression effect of the deep bidirectional recurrent neural network after training meets the prediction requirements.

[0097] Figure 2 The diagram below shows the locations of the EMG electrodes. They are located on the vastus lateralis (VL), rectus femoris (RF), vastus medialis (VM), biceps femoris (BF), semitendinosus (ST), tibialis anterior (TA), lateral gastrocnemius (LG), and medial gastrocnemius (MG).

[0098] Using myoelectric sensors and inertial navigation sensors to collect myoelectric signals and joint angles from eight unilateral leg muscles includes the following: Myoelectric sensors are used to collect myoelectric signals from eight unilateral leg muscles, including the vastus lateralis (VL), rectus femoris (RF), vastus medialis (VM), biceps femoris (BF), semitendinosus (ST), tibialis anterior (TA), lateral gastrocnemius (LG), and medial gastrocnemius (MG), numbered 1 to 8. Inertial navigation is used to collect joint kinematic parameters and obtain joint motion angles.

[0099] Figure 3 The angles in the figure are calculated using Noraxon MR3 software using four Noraxon myoMotion inertial sensors placed at the waist, thigh, calf, and foot.

[0100] Figure 4 This is the schematic diagram of the GRU module. A bidirectional recurrent deep neural network is designed for angle estimation. The scheme and parameters are as follows: Using the GRU network module as a unit, a bidirectional recurrent deep neural network is designed. The GRU contains two gates: the update gate z t and reset gate r t , update as follows:

[0101] r t =σ(W r x t +U r h t-1 +b r )

[0102] z t =σ(W z x t +U z h t-1 +b z )

[0103] Among them, σ represents the sigmoid function, W # and U # represents the weight matrix, b # represents the bias term.

[0104] Current status h' t Determined by the following formula:

[0105]

[0106] The output state is determined by the following formula:

[0107]

[0108] Figure 5 It is a diagram of the deep neural network structure.

[0109] Build a deep bidirectional recurrent neural network, specifically:

[0110] (1) The network input format of the deep bidirectional recurrent neural network is set to: M*N; where M is the number of electromyographic channels and N is the number of muscle synergies;

[0111] (2) The forward and backward outputs of the bidirectional GRU layer are merged and a DropOut layer with a probability of 0.25, i.e., a DO layer, is added to prevent overfitting;

[0112] (3) Set up three fully connected layers of a deep bidirectional recurrent neural network. Each layer contains a fully connected network FC, a normalized BN operation, and a DO operation. The probability of the DO operation is 0.25. The number of neurons in each layer is 1024, 1024, and 32, respectively, and the ReLU activation function is used.

[0113] (4) Establishing the output layer of a deep bidirectional recurrent neural network to simultaneously output the angles of the hip, knee, and ankle joints;

[0114] (5) Train the deep bidirectional recurrent neural network using training set data.

[0115] Training a deep bidirectional recurrent neural network using training set data includes the following steps:

[0116] a. Train the deep bidirectional recurrent neural network using the Adam optimizer, setting the batch size of the training set data in the deep bidirectional recurrent neural network to 128 and the learning rate to 0.001;

[0117] b. Input the training set data into a deep bidirectional recurrent neural network, and the output is the joint angle data, including the angles of the hip joint, knee joint, and ankle joint;

[0118] c. Input the test set data into the deep bidirectional recurrent neural network, and the output is the predicted joint angle data θ i 预测 , including predicted hip, knee, and ankle joint angles.

[0119] Figure 6 This is a comparison chart of the regression results of the present invention and its comparison with the machine learning method, using RMSE as the criterion, and the predicted joint angle data θ output by the test set data i 预测 And collect the joint motion angle θ through the inertial sensor i 真实 , get the root mean square error RMSE, that is:

[0120]

[0121] If the root mean square error (RMSE) obtained is smaller than the set control group, the regression effect of the deep bidirectional recurrent neural network after training meets the prediction requirements.

[0122] The control group used two types of machine learning algorithms: linear regression (LR) + mean absolute value (MAV) and multi-layer perceptron (MLP) + logarithmic variance feature (logvar). The preprocessing process was consistent with the method used in the present invention.

[0123] The proposed method achieved errors of 3.780±0.704, 6.335±0.800, and 2.595±0.559 for the hip, knee, and ankle joints, respectively. A significance analysis (ANOVA) was performed, and the p-values satisfied the p-value, indicating that the proposed method significantly outperformed the control group, demonstrating excellent accuracy.

[0124] Figure 7 The figure shows the effect of angle estimation. The black solid line is the reference angle of the limb, and the dotted line is the angle predicted by the model. It can be seen from the figure that the proposed method can accurately achieve continuous motion estimation, showing good practical results.

[0125] The above is an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, expansion, etc. made within the spirit and principle of the present invention are included in the scope of protection of the present invention.

Claims

1. A method for estimating continuous lower limb motion based on muscle synergy and deep learning, characterized in that: The following steps are involved: 1) Use electromyographic sensors to collect electromyographic signals generated by unilateral leg muscles during uniform walking, and at the same time, use inertial navigation sensors to obtain joint motion angles; 2) Preprocessing the collected electromyographic signals to obtain preprocessed electromyographic data; The format of the corrected EMG data is M*T, where M is the number of EMG channels and T is the length of the sampling points of each sliding window. M*T is the non-negative matrix V. 3) Extract muscle synergy information from the preprocessed EMG data using a non-negative matrix factorization algorithm; Decompose the electromyographic data using the non-negative matrix decomposition algorithm and obtain two non-negative matrices W and H, namely: V≈WH Where V is a non-negative matrix, W is a non-negative matrix of muscle synergy information, in the format of M*N, H is the activation coefficient, in the format of N*T; N is the number of muscle synergies; The optimal W and H are obtained by minimizing the Euclidean distance between V and the reconstructed data WH, that is: Among them, st is the constraint condition, and the muscle synergy information W is solved according to the above formula; Set the dimension of W to 2, that is, take 2 muscle synergy information; 4) Divide the muscle synergy information into training set data and test set data; 5) Establish a deep bidirectional recurrent neural network; and train the deep bidirectional recurrent neural network using training set data; 6) Input the test set data into the trained deep bidirectional recurrent neural network, and verify the angle regression effect of the trained deep bidirectional recurrent neural network by obtaining the root mean square error between the predicted angle and the actual angle collected by the inertial navigation.

2. The method for estimating continuous lower limb motion based on muscle synergy and deep learning according to claim 1, characterized in that: The step 2) comprises the following steps: 2-1) Use bandpass filter and notch filter to remove noise interference from electromyographic signals, and normalize the data by z-score normalization method; 2-2) Segment the normalized data using a sliding window with a length of 250 ms and a sliding length of 50 ms to obtain segmented data; 2-3) Correct the segmented data to make it non-negative in preparation for the processing of the non-negative matrix factorization algorithm; x 矫正后 =abs(x 矫正前 ) Wherein, abs() represents the absolute value operation, and x represents the electromyographic data; the corrected electromyographic data is made to have the same format as the decomposition data by the non-negative matrix decomposition algorithm, so that the negative numbers therein are inverted and transformed into positive numbers.

3. The method for estimating continuous motion of lower limbs based on muscle synergy and deep learning according to claim 1, characterized in that: The muscle synergy information is divided into training set data and test set data according to a 10-fold cross-validation method.

4. The method for estimating continuous lower limb motion based on muscle synergy and deep learning according to claim 1, characterized in that: In step 5), the deep bidirectional recurrent neural network is established, specifically: (1) The network input format of the deep bidirectional recurrent neural network is set to: M*N; where M is the number of electromyographic channels and N is the number of muscle synergies; (2) The forward and backward outputs of the bidirectional GRU layer are merged and a DropOut layer with a probability of 0.25, i.e., a DO layer, is added to prevent overfitting; (3) Set up three fully connected layers of a deep bidirectional recurrent neural network. Each fully connected layer contains a fully connected network FC, a normalized BN operation, and a DO operation. The probability of the DO operation is 0.

25. The number of neurons in each fully connected layer is 1024, 1024, and 32, respectively, and the ReLU activation function is used. (4) Establishing the output layer of a deep bidirectional recurrent neural network to simultaneously output the angles of the hip, knee, and ankle joints; (5) Train the deep bidirectional recurrent neural network using training set data.

5. The method for estimating continuous motion of lower limbs based on muscle synergy and deep learning according to claim 4, characterized in that: The method of training a deep bidirectional recurrent neural network using training set data includes the following steps: a. Train the deep bidirectional recurrent neural network using the Adam optimizer, setting the batch size of the training set data in the deep bidirectional recurrent neural network to 128 and the learning rate to 0.001; b. Input the training set data into a deep bidirectional recurrent neural network, and the output is the joint angle data, including the angles of the hip joint, knee joint, and ankle joint; c. Input the test set data into the deep bidirectional recurrent neural network, and the output is the predicted joint angle data θ i 预测 , including predicted hip, knee, and ankle joint angles.

6. The method for estimating continuous motion of lower limbs based on muscle synergy and deep learning according to claim 1, characterized in that: The step 6) is specifically as follows: Predicted joint angle data output by the test set data And collect joint motion angles through inertial sensors Get the root mean square error RMSE, that is: If the root mean square error (RMSE) obtained is smaller than the set control group, the regression effect of the deep bidirectional recurrent neural network after training meets the prediction requirements.

Images