Lower limb joint torque prediction method and system based on feature fusion

Abstract

The invention provides a lower limb joint torque prediction method and system based on feature fusion in the technical field of human body lower limb motion intention prediction, and the method comprises the following steps: S1, obtaining a large amount of historical inertia measurement data and historical electromyogram in a lower limb motion process, preprocessing and labeling the historical inertial measurement data and the historical electromyogram, and then constructing a data set; s2, creating a joint torque prediction model based on an embedding module, a local feature extraction module, a global feature extraction module and a feature fusion module, and setting a loss function of the joint torque prediction model; s3, training a joint torque prediction model through the data set, and continuously optimizing hyper-parameters of the joint torque prediction model in the training process; and S4, predicting the lower limb joint torque by using the trained joint torque prediction model. The method has the advantages that the precision and stability of lower limb joint torque prediction are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of human lower limb movement intention prediction, and in particular to a method and system for predicting lower limb joint torque based on feature fusion. Background Art

[0002] As one of the important parameters of human biomechanics, lower limb joint torque covers a wide range of research fields such as neural control. It can not only help understand how different degrees of arthritis affect walking, but also provide auxiliary torque for powered exoskeletons to reduce muscle consumption, playing an important role in many fields.

[0003] However, due to the complexity of human structure and the diversity of movement, the lower limb joint torque of the human body is difficult to measure directly in vivo. Musculoskeletal modeling is a commonly used method to predict the lower limb joint torque. It converts surface electromyographic signals into muscle activation levels, substitutes them into the musculoskeletal model, calculates muscle forces and joint moment arms, and then obtains the lower limb joint torque. For example, Sartori proposed a neuromusculoskeletal model of human lower limbs, which uses electromyographic signals from 16 muscles to estimate the forces generated by 34 muscle actuators and the joint torques generated by the hip, knee and ankle joints under different contraction conditions; Buongiorno proposed a NMS model for predicting elbow and shoulder torques, and used a linear optimization method to optimize the parameters to improve the performance of the original model. However, the above two methods require many physiological parameters and are usually very time-consuming, have poor real-time performance, and are sensitive to muscle-tendon geometry and subject-specific anthropometric information.

[0004] To address these limitations, early studies combined feature extraction with machine learning algorithms to predict lower limb joint torque. This machine learning algorithm requires fewer features than musculoskeletal models to obtain accurate predictions. Wang et al. used XGBoost based on inertial measurement unit (IMU) to achieve real-time estimation of knee adduction torque; Camargo et al. determined the optimal feature set through forward feature selection and then used machine learning algorithms to predict lower limb joint torque under different environments. Although the above two methods have shown good performance, the high noise and high nonlinear characteristics of the collected human body signals still pose challenges, that is, motion estimation methods based on traditional machine learning algorithms require complex feature extraction processes to obtain meaningful features from raw data, and often have poor robustness and large prediction errors.

[0005] Compared with traditional machine learning algorithms, the advantage of deep learning is that it can automatically extract high-level abstract features from large-scale complex data, and has higher accuracy and better smoothness. Therefore, some deep learning models have been widely used in kinematic estimation. Dorschky et al. proposed a CNN network based on inertial sensors to estimate the knee and ankle torques of subjects; Liang et al. proposed an attention-based CNN-BiLSTM model, which extracts features by using one-dimensional convolution with a convolution kernel size of 1 and uses a multi-head attention structure to capture the long-range dependencies of time series. The model has good performance in both estimation accuracy and computational efficiency; Bin Hossain et al. proposed an imu-based DL-Kinetics-FMNet model, which can accurately estimate the lower limb joint torques under various walking conditions. These studies have shown great potential in predicting lower limb joint torques and have made significant progress in improving prediction accuracy and computational efficiency. However, these studies only extracted single-scale features from the time domain. Attention to a single scale may lead to the loss of key information and affect prediction accuracy, while insufficient attention to global features may lead to insufficient capture of contextual relationships, which in turn leads to unstable prediction results, that is, affecting the stability of the prediction.

[0006] Therefore, how to provide a lower limb joint torque prediction method and system based on feature fusion to improve the accuracy and stability of lower limb joint torque prediction has become a technical problem that needs to be solved urgently. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a method and system for predicting lower limb joint torque based on feature fusion, so as to improve the accuracy and stability of lower limb joint torque prediction.

[0008] In a first aspect, the present invention provides a method for predicting lower limb joint torque based on feature fusion, comprising the following steps:

[0009] Step S1, obtaining a large amount of historical inertial measurement data and historical electromyograms during lower limb movement, and constructing a data set after preprocessing and annotating each of the historical inertial measurement data and historical electromyograms;

[0010] Step S2, creating a joint torque prediction model based on the embedding module, the local feature extraction module, the global feature extraction module and the feature fusion module, and setting a loss function of the joint torque prediction model;

[0011] Step S3, training the joint torque prediction model using the data set, and continuously optimizing the hyperparameters of the joint torque prediction model during the training process;

[0012] Step S4: using the trained joint torque prediction model to predict the lower limb joint torque.

[0013] Furthermore, the step S1 is specifically as follows:

[0014] A large amount of historical inertial measurement data and historical electromyograms during lower limb movement are obtained, each of the historical inertial measurement data is preprocessed to remove outliers, each of the historical electromyograms is preprocessed to enhance the image, the preprocessed historical inertial measurement data and historical electromyograms are annotated with joint torques, and a data set is constructed based on the annotated historical inertial measurement data and historical electromyograms.

[0015] Furthermore, in step S2, the input end of the local feature extraction module is connected to the output end of the embedding module, and the output end is connected to the input end of the feature fusion module; the input end of the global feature extraction module is connected to the output end of the embedding module, and the output end is connected to the input end of the feature fusion module;

[0016] The embedding module is used to map the feature dimensions of the historical inertial measurement data and the historical electromyogram into a high-dimensional vector space using value embedding and position embedding;

[0017] The local feature extraction module is used to capture local features of different time scales from the output of the embedding module; the local feature extraction module is provided with a plurality of local feature extraction units that model different underlying time patterns and a local feature merging unit; the local feature extraction unit captures local sub-features from the output of the embedding module through one-dimensional convolution, and upsamples each local sub-feature through transposed convolution to unify the sequence length; the local feature merging unit merges each local sub-feature after unifying the sequence length through two-dimensional convolution, and outputs the local feature;

[0018] The global feature extraction module is used to capture global features from the output of the embedding module; the global feature extraction module is provided with a feature mixer and a global feature extraction unit using Fourier transform; the feature mixer is used to perform feature mixing on the output of the embedding module and output mixed features; the global feature extraction unit is used to transform the mixed features from the time domain to the frequency domain, and apply the convolution theorem to capture the long-term dependency in the frequency domain, thereby obtaining the global features;

[0019] The feature fusion module is used to perform bit-by-bit sum operation on local features and global features to obtain merged features, and input the merged features into a feedforward neural network for feature fusion to output a lower limb joint torque prediction result.

[0020] Furthermore, the step S3 is specifically as follows:

[0021] Dividing the data set into a training set and a validation set based on a preset ratio, training the joint torque prediction model through the training set until the loss value of the loss function is less than a preset loss threshold, and continuously optimizing the hyperparameters of the joint torque prediction model including at least a learning rate, a batch size, a number of iterations, a convolution kernel size, and a pooling window size during the training process;

[0022] The normalized root mean square error, correlation coefficient and determination coefficient of the trained joint torque prediction model are calculated through the verification set to verify the training effect of the joint torque prediction model.

[0023] Furthermore, the step S4 is specifically as follows:

[0024] The trained joint torque prediction model is deployed to the cloud server to obtain real-time inertial measurement data and real-time electromyography for lower limb joint torque prediction. After preprocessing the real-time inertial measurement data and real-time electromyography, they are input into the joint torque prediction model deployed by the cloud server to obtain lower limb joint torque prediction results. Based on the lower limb joint torque prediction results, the joint torque prediction model is continuously optimized, and the prediction log is recorded in real time.

[0025] In a second aspect, the present invention provides a lower limb joint torque prediction system based on feature fusion, comprising the following modules:

[0026] A data set construction module is used to obtain a large amount of historical inertial measurement data and historical electromyograms during lower limb movement, and to construct a data set after preprocessing and annotating each of the historical inertial measurement data and historical electromyograms;

[0027] A joint torque prediction model creation module is used to create a joint torque prediction model based on the embedding module, the local feature extraction module, the global feature extraction module and the feature fusion module, and set the loss function of the joint torque prediction model;

[0028] A joint torque prediction model training module, used to train the joint torque prediction model using the data set, and continuously optimize the hyperparameters of the joint torque prediction model during the training process;

[0029] The lower limb joint torque prediction module is used to predict the lower limb joint torque using the trained joint torque prediction model.

[0030] Furthermore, the data set construction module is specifically used for:

[0031] A large amount of historical inertial measurement data and historical electromyograms during lower limb movement are obtained, each of the historical inertial measurement data is preprocessed to remove outliers, each of the historical electromyograms is preprocessed to enhance the image, the preprocessed historical inertial measurement data and historical electromyograms are annotated with joint torques, and a data set is constructed based on the annotated historical inertial measurement data and historical electromyograms.

[0032] Further, in the joint torque prediction model creation module, the input end of the local feature extraction module is connected to the output end of the embedding module, and the output end is connected to the input end of the feature fusion module; the input end of the global feature extraction module is connected to the output end of the embedding module, and the output end is connected to the input end of the feature fusion module;

[0033] The embedding module is used to map the feature dimensions of the historical inertial measurement data and the historical electromyogram into a high-dimensional vector space using value embedding and position embedding;

[0034] The local feature extraction module is used to capture local features of different time scales from the output of the embedding module; the local feature extraction module is provided with a plurality of local feature extraction units that model different underlying time patterns and a local feature merging unit; the local feature extraction unit captures local sub-features from the output of the embedding module through one-dimensional convolution, and upsamples each local sub-feature through transposed convolution to unify the sequence length; the local feature merging unit merges each local sub-feature after unifying the sequence length through two-dimensional convolution, and outputs the local feature;

[0035] The global feature extraction module is used to capture global features from the output of the embedding module; the global feature extraction module is provided with a feature mixer and a global feature extraction unit using Fourier transform; the feature mixer is used to perform feature mixing on the output of the embedding module and output mixed features; the global feature extraction unit is used to transform the mixed features from the time domain to the frequency domain, and apply the convolution theorem to capture the long-term dependency in the frequency domain, thereby obtaining the global features;

[0036] The feature fusion module is used to perform bit-by-bit sum operation on local features and global features to obtain merged features, and input the merged features into a feedforward neural network for feature fusion to output a lower limb joint torque prediction result.

[0037] Furthermore, the joint torque prediction model training module is specifically used for:

[0038] Dividing the data set into a training set and a validation set based on a preset ratio, training the joint torque prediction model through the training set until the loss value of the loss function is less than a preset loss threshold, and continuously optimizing the hyperparameters of the joint torque prediction model including at least a learning rate, a batch size, a number of iterations, a convolution kernel size, and a pooling window size during the training process;

[0039] The normalized root mean square error, correlation coefficient and determination coefficient of the trained joint torque prediction model are calculated through the verification set to verify the training effect of the joint torque prediction model.

[0040] Furthermore, the lower limb joint torque prediction module is specifically used for:

[0041] The trained joint torque prediction model is deployed to the cloud server to obtain real-time inertial measurement data and real-time electromyography for lower limb joint torque prediction. After preprocessing the real-time inertial measurement data and real-time electromyography, they are input into the joint torque prediction model deployed by the cloud server to obtain lower limb joint torque prediction results. Based on the lower limb joint torque prediction results, the joint torque prediction model is continuously optimized, and the prediction log is recorded in real time.

[0042] The advantages of the present invention are:

[0043] By acquiring a large amount of historical inertial measurement data and historical electromyograms during lower limb movement, each historical inertial measurement data and historical electromyogram is preprocessed and annotated to construct a data set; then, a joint torque prediction model is created based on an embedding module, a local feature extraction module, a global feature extraction module, and a feature fusion module, and a loss function of the joint torque prediction model is set. The joint torque prediction model is trained through the data set, and the hyperparameters of the joint torque prediction model are continuously optimized during the training process. Finally, the trained joint torque prediction model is used to predict the lower limb joint torque; that is, the lower limb joint torque is predicted by a joint torque prediction model including an embedding module, a local feature extraction module, a global feature extraction module, and a feature fusion module, and the embedding module is used to use value embedding and Position embedding maps the feature dimensions of inertial measurement data and electromyography to a high-dimensional vector space, which facilitates the model to learn and remember the temporal properties of the input sequence; the local feature extraction module is used to capture local features of different time scales from the output of the embedding module to effectively utilize the information of all local features; the global feature extraction module is used to capture global features from the output of the embedding module; the feature fusion module is used to perform bit-by-bit sum operations on local features and global features to obtain merged features, and input the merged features into a feedforward neural network for feature fusion to output the lower limb joint torque prediction results; that is, the present invention not only extracts single-scale features (local features) from the time domain, but also pays attention to global features to effectively capture contextual relationships, ultimately greatly improving the accuracy and stability of lower limb joint torque prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0045] Figure 1 It is a flow chart of a method for predicting lower limb joint torque based on feature fusion of the present invention.

[0046] Figure 2 It is a structural schematic diagram of a lower limb joint torque prediction system based on feature fusion of the present invention.

[0047] Figure 3 It is a schematic diagram of the joint torque prediction model of the present invention.

[0048] Figure 4 It is a schematic diagram of the feature fusion module of the present invention. DETAILED DESCRIPTION

[0049] The technical solution in the embodiments of the present application has the following overall idea: the lower limb joint torque is predicted by a joint torque prediction model including an embedding module, a local feature extraction module, a global feature extraction module and a feature fusion module. The embedding module is used to map the feature dimensions of the inertial measurement data and the electromyogram to a high-dimensional vector space using value embedding and position embedding, so as to facilitate learning and memorizing the temporal properties of the input sequence; the local feature extraction module is used to capture local features of different time scales from the output of the embedding module to effectively utilize the information of all local features; the global feature extraction module is used to capture global features from the output of the embedding module; the feature fusion module is used to perform bit-by-bit sum operations on local features and global features to obtain merged features, and the merged features are input into a feedforward neural network for feature fusion to output the lower limb joint torque prediction results; that is, the present invention not only extracts single-scale features from the time domain, but also pays attention to global features to capture contextual relationships, thereby improving the accuracy and stability of lower limb joint torque prediction.

[0050] Please refer to Figures 1 to 4 As shown, a preferred embodiment of a method for predicting lower limb joint torque based on feature fusion of the present invention comprises the following steps:

[0051] Step S1, obtaining a large amount of historical inertial measurement data (IMU) and historical electromyogram (EMG) during lower limb movement, and constructing a data set after preprocessing and annotating each of the historical inertial measurement data and historical electromyogram;

[0052] Step S2, creating a joint torque prediction model (GLF-Net) based on the embedding module, the local feature extraction module (LFN), the global feature extraction module (GFN) and the feature fusion module, and setting the loss function of the joint torque prediction model; that is, the joint torque prediction model adopts a dual-branch structure of a local feature extraction module and a global feature extraction module, one branch extracts local features with multi-scale representativeness from the time domain perspective, and the other branch extracts global features with overall trend changes from the frequency domain perspective through fast Fourier transform; the feature fusion module is used for feature fusion to learn more diverse features to fit the future real torque curve and improve the prediction accuracy;

[0053] Step S3, training the joint torque prediction model using the data set, and continuously optimizing the hyperparameters of the joint torque prediction model during the training process;

[0054] Step S4: using the trained joint torque prediction model to predict the lower limb joint torque.

[0055] The lower limb joint torque prediction results of the present invention are very close to the real torque value, and can fit the real torque value well at the peak and valley positions. The joint torque prediction model predicts the knee FE, hip AA and hip FE torque of each subject, and maintains a high prediction accuracy in most subjects. In terms of the prediction of the hip and knee joint torque of the human lower limbs, the present invention is superior to the comparison model in all individual subjects.

[0056] The step S1 is specifically as follows:

[0057] A large amount of historical inertial measurement data and historical electromyograms during lower limb movement are obtained, each of the historical inertial measurement data is preprocessed to remove outliers, each of the historical electromyograms is preprocessed to enhance the image, the preprocessed historical inertial measurement data and historical electromyograms are annotated with joint torques, and a data set is constructed based on the annotated historical inertial measurement data and historical electromyograms.

[0058] In the step S2, the input end of the local feature extraction module is connected to the output end of the embedding module, and the output end is connected to the input end of the feature fusion module; the input end of the global feature extraction module is connected to the output end of the embedding module, and the output end is connected to the input end of the feature fusion module;

[0059] The embedding module is used to map the feature dimensions of historical inertial measurement data and historical electromyograms to a high-dimensional vector space using value embedding and position embedding, so as to facilitate the model to learn and memorize the temporal properties of the input sequence; the formula of the embedding module is:

[0060] X emb =sum(VE+PE):x N*C →x N*D ;

[0061] Among them, VE represents value embedding; PE represents position embedding; D represents the feature dimension after mapping; X emb Represents the features mapped to the high-dimensional vector space; x represents the input signal (historical inertial measurement data, historical electromyography); N represents the number of samples of the input signal;

[0062] The local feature extraction module is used to capture local features of different time scales from the output of the embedding module; the local feature extraction module is provided with a plurality of local feature extraction units that model different underlying time patterns and a local feature merging unit; the local feature extraction unit captures local sub-features from the output of the embedding module through one-dimensional convolution, and upsamples each local sub-feature through transposed convolution to unify the sequence length; the local feature merging unit merges each local sub-feature after unifying the sequence length through two-dimensional convolution, and outputs the local feature;

[0063] The local feature extraction module also introduces a nonlinear exponential linear unit (Tanh) as an activation function of nonlinear transformation to speed up the convergence of the model; the dropout rate is set to 0.05 to prevent overfitting during training; the formula of the local feature extraction module is as follows:

[0064] Y L,i =Dropout(Tanh(Conv1d(X emb ) kernel＝i ));

[0065] Y′ L,i =Dropout(Tanh(TransConv1d(Y L,i ) kernel＝i ));

[0066] Y local =Conv2d(Y′ L,i ) kernel＝L*L ;

[0067] Among them, Y L,i represents the local sub-feature before upsampling; Y L ' ,i represents the local sub-feature after upsampling; Y local represents local features; kernel represents the convolution kernel; i represents the size of the convolution kernel; L represents the number of local feature extraction units, that is, the number of branches;

[0068] The global feature extraction module is used to capture global features from the output of the embedding module; the global feature extraction module is provided with a feature mixer and a global feature extraction unit using Fourier transform; the feature mixer is used to perform feature mixing on the output of the embedding module and output mixed features; the global feature extraction unit is used to transform the mixed features from the time domain to the frequency domain, and apply the convolution theorem to capture the long-term dependency in the frequency domain, thereby obtaining the global features;

[0069] The feature mixer includes a layer normalization layer (LN) and a multi-layer perceptron (MLP) for effective feature mixing. That is, the data is first normalized along the feature dimension using LN to ensure that the distribution of each sample feature in the same dimension is similar to reduce the coupling between samples and improve the convergence speed. Then it is passed through the MLP, which consists of two fully connected layers. Finally, the output of the MLP is normalized using LN to facilitate the learning process of the subsequent neural network layer. The calculation process of transforming the mixed features from the time domain to the frequency domain is:

[0070]

[0071] Where X[k] represents the frequency component corresponding to the time domain signal x[n]; N represents the total number of samples of the signal; j represents the imaginary unit; k represents the global convolution kernel; n represents the index of time in x[n];

[0072] Long-term dependencies are captured by performing point-by-point multiplication on X[k] using a trainable global convolution kernel k with the same shape as the input sequence. Finally, for feature fusion with time-based local features, the filtered X[k] is further inverse Fourier transformed to obtain the updated time-domain spatial feature representation (global feature), as follows:

[0073]

[0074] Among them, Y global Represents global features; F -1 represents inverse Fourier transform; represents the Hadamard product, i.e., element-wise multiplication; F(X) represents the conversion of input X into frequency domain representation; K represents the global convolution kernel;

[0075] The feature fusion module is used to perform bit-by-bit sum operation on local features and global features to obtain merged features, and input the merged features into a feedforward neural network for feature fusion to output a lower limb joint torque prediction result.

[0076] The calculation formula of the combined feature is:

[0077] Y=Y global +Y local ;

[0078] The merged features are input into a feedforward neural network, which consists of two fully connected layers (FC). The first layer of FC is followed by a nonlinear activation function ReLU, which maximizes the filtering ability of neurons and accelerates convergence. In order to facilitate model training and enhance the stability of the model, residual splicing is also used to add the merged features directly to the input of the last layer, which not only retains the initial information, but also alleviates the problem of gradient disappearance and improves the learning ability of the network. Layer normalization is also used to reduce the coupling between samples and improve the convergence speed. The process is shown in the following formula:

[0079] Y pred =Norm(Y+FeedForward(Y));

[0080] Among them, Y pred represents the fusion feature obtained by fusion of the merged feature Y;

[0081] In order to obtain the final prediction results of the lower limb joint torque, the projection operation is used to transform Y pred Mapped to the dimensions of the final predicted features:

[0082] Y final-pred =Projection(Y pred ).

[0083] The step S3 is specifically as follows:

[0084] Dividing the data set into a training set and a validation set based on a preset ratio, training the joint torque prediction model through the training set until the loss value of the loss function is less than a preset loss threshold, and continuously optimizing the hyperparameters of the joint torque prediction model including at least a learning rate, a batch size, a number of iterations, a convolution kernel size, and a pooling window size during the training process;

[0085] The validation set is used to calculate the normalized root mean square error (NRMSE), correlation coefficient (PCC), and determination coefficient (R2) between the prediction moment and the inverse dynamic moment of the trained joint torque prediction model to verify the training effect of the joint torque prediction model.

[0086] The step S4 is specifically as follows:

[0087] The trained joint torque prediction model is deployed to the cloud server to obtain real-time inertial measurement data and real-time electromyography for lower limb joint torque prediction. After preprocessing the real-time inertial measurement data and real-time electromyography, they are input into the joint torque prediction model deployed by the cloud server to obtain lower limb joint torque prediction results. Based on the lower limb joint torque prediction results, the joint torque prediction model is continuously optimized, and the prediction log is recorded in real time.

[0088] A preferred embodiment of a lower limb joint torque prediction system based on feature fusion of the present invention includes the following modules:

[0089] A data set construction module is used to obtain a large amount of historical inertial measurement data (IMU) and historical electromyography (EMG) during lower limb movement, and to construct a data set after preprocessing and annotating each of the historical inertial measurement data and historical electromyography;

[0090] The joint torque prediction model creation module is used to create a joint torque prediction model (GLF-Net) based on the embedding module, the local feature extraction module (LFN), the global feature extraction module (GFN) and the feature fusion module, and set the loss function of the joint torque prediction model; that is, the joint torque prediction model adopts a dual-branch structure of a local feature extraction module and a global feature extraction module, one branch extracts local features with multi-scale representativeness from the time domain perspective, and the other branch extracts global features with overall trend changes from the frequency domain perspective through fast Fourier transform; the feature fusion module is used for feature fusion to learn more diverse features to fit the future real torque curve and improve the prediction accuracy;

[0091] A joint torque prediction model training module, used to train the joint torque prediction model using the data set, and continuously optimize the hyperparameters of the joint torque prediction model during the training process;

[0092] The lower limb joint torque prediction module is used to predict the lower limb joint torque using the trained joint torque prediction model.

[0093] The lower limb joint torque prediction results of the present invention are very close to the real torque value, and can fit the real torque value well at the peak and valley positions. The joint torque prediction model predicts the knee FE, hip AA and hip FE torque of each subject, and maintains a high prediction accuracy in most subjects. In terms of the prediction of the hip and knee joint torque of the human lower limbs, the present invention is superior to the comparison model in all individual subjects.

[0094] The data set construction module is specifically used for:

[0095] A large amount of historical inertial measurement data and historical electromyograms during lower limb movement are obtained, each of the historical inertial measurement data is preprocessed to remove outliers, each of the historical electromyograms is preprocessed to enhance the image, the preprocessed historical inertial measurement data and historical electromyograms are annotated with joint torques, and a data set is constructed based on the annotated historical inertial measurement data and historical electromyograms.

[0096] In the joint torque prediction model creation module, the input end of the local feature extraction module is connected to the output end of the embedding module, and the output end is connected to the input end of the feature fusion module; the input end of the global feature extraction module is connected to the output end of the embedding module, and the output end is connected to the input end of the feature fusion module;

[0097] The embedding module is used to map the feature dimensions of historical inertial measurement data and historical electromyograms to a high-dimensional vector space using value embedding and position embedding, so as to facilitate the model to learn and memorize the temporal properties of the input sequence; the formula of the embedding module is:

[0098] X emb =sum(VE+PE):x N*C →x N*D ;

[0099] Among them, VE represents value embedding; PE represents position embedding; D represents the feature dimension after mapping; X emb Represents the features mapped to the high-dimensional vector space; x represents the input signal (historical inertial measurement data, historical electromyography); N represents the number of samples of the input signal;

[0100] The local feature extraction module is used to capture local features of different time scales from the output of the embedding module; the local feature extraction module is provided with a plurality of local feature extraction units that model different underlying time patterns and a local feature merging unit; the local feature extraction unit captures local sub-features from the output of the embedding module through one-dimensional convolution, and upsamples each local sub-feature through transposed convolution to unify the sequence length; the local feature merging unit merges each local sub-feature after unifying the sequence length through two-dimensional convolution, and outputs the local feature;

[0101] The local feature extraction module also introduces a nonlinear exponential linear unit (Tanh) as an activation function of nonlinear transformation to speed up the convergence of the model; the dropout rate is set to 0.05 to prevent overfitting during training; the formula of the local feature extraction module is as follows:

[0102] Y L,i =Dropout(Tanh(Conv1d(X emb ) kernel＝i ));

[0103] Y′ L,i =Dropout(Tanh(TransConv1d(Y L,i ) kernel＝i ));

[0104] Y local =Conv2d(Y′ L,i ) kernel＝L*L ;

[0105] Among them, Y L,i represents the local sub-feature before upsampling; Y′ L,i represents the local sub-feature after upsampling; Y local represents local features; kernel represents the convolution kernel; i represents the size of the convolution kernel; L represents the number of local feature extraction units, that is, the number of branches;

[0106] The global feature extraction module is used to capture global features from the output of the embedding module; the global feature extraction module is provided with a feature mixer and a global feature extraction unit using Fourier transform; the feature mixer is used to perform feature mixing on the output of the embedding module and output mixed features; the global feature extraction unit is used to transform the mixed features from the time domain to the frequency domain, and apply the convolution theorem to capture the long-term dependency in the frequency domain, thereby obtaining the global features;

[0107] The feature mixer includes a layer normalization layer (LN) and a multi-layer perceptron (MLP) for effective feature mixing. That is, the data is first normalized along the feature dimension using LN to ensure that the distribution of each sample feature in the same dimension is similar to reduce the coupling between samples and improve the convergence speed. Then it is passed through the MLP, which consists of two fully connected layers. Finally, the output of the MLP is normalized using LN to facilitate the learning process of the subsequent neural network layer. The calculation process of transforming the mixed features from the time domain to the frequency domain is:

[0108]

[0109] Where X[k] represents the frequency component corresponding to the time domain signal x[n]; N represents the total number of samples of the signal; j represents the imaginary unit; k represents the global convolution kernel; n represents the index of time in x[n];

[0110] Long-term dependencies are captured by performing point-by-point multiplication on X[k] using a trainable global convolution kernel k with the same shape as the input sequence. Finally, for feature fusion with time-based local features, the filtered X[k] is further inverse Fourier transformed to obtain the updated time-domain spatial feature representation (global feature), as follows:

[0111]

[0112] Among them, Y global Represents global features; F -1 represents inverse Fourier transform; represents the Hadamard product, i.e., element-wise multiplication; F(X) represents the conversion of input X into frequency domain representation; K represents the global convolution kernel;

[0113] The feature fusion module is used to perform bit-by-bit sum operation on local features and global features to obtain merged features, and input the merged features into a feedforward neural network for feature fusion to output a lower limb joint torque prediction result.

[0114] The calculation formula of the combined feature is:

[0115] Y=Y global +Y local ;

[0116] The merged features are input into a feedforward neural network, which consists of two fully connected layers (FC). The first layer of FC is followed by a nonlinear activation function ReLU, which maximizes the filtering ability of neurons and accelerates convergence. In order to facilitate model training and enhance the stability of the model, residual splicing is also used to add the merged features directly to the input of the last layer, which not only retains the initial information, but also alleviates the problem of gradient disappearance and improves the learning ability of the network. Layer normalization is also used to reduce the coupling between samples and improve the convergence speed. The process is shown in the following formula:

[0117] Y pred =Norm(Y+FeedForward(Y));

[0118] Among them, Y pred represents the fusion feature obtained by fusion of the merged feature Y;

[0119] In order to obtain the final prediction results of the lower limb joint torque, the projection operation is used to transform Y pred Mapped to the dimensions of the final predicted features:

[0120] Y final-pred =Projection(Y pred ).

[0121] The joint torque prediction model training module is specifically used for:

[0122] Dividing the data set into a training set and a validation set based on a preset ratio, training the joint torque prediction model through the training set until the loss value of the loss function is less than a preset loss threshold, and continuously optimizing the hyperparameters of the joint torque prediction model including at least a learning rate, a batch size, a number of iterations, a convolution kernel size, and a pooling window size during the training process;

[0123] The validation set is used to calculate the normalized root mean square error (NRMSE), correlation coefficient (PCC), and determination coefficient (R2) between the prediction moment and the inverse dynamic moment of the trained joint torque prediction model to verify the training effect of the joint torque prediction model.

[0124] The lower limb joint torque prediction module is specifically used for:

[0125] The trained joint torque prediction model is deployed to the cloud server to obtain real-time inertial measurement data and real-time electromyography for lower limb joint torque prediction. After preprocessing the real-time inertial measurement data and real-time electromyography, they are input into the joint torque prediction model deployed by the cloud server to obtain lower limb joint torque prediction results. Based on the lower limb joint torque prediction results, the joint torque prediction model is continuously optimized, and the prediction log is recorded in real time.

[0126] In summary, the advantages of the present invention are:

[0127] By acquiring a large amount of historical inertial measurement data and historical electromyograms during lower limb movement, each historical inertial measurement data and historical electromyogram is preprocessed and annotated to construct a data set; then, a joint torque prediction model is created based on an embedding module, a local feature extraction module, a global feature extraction module, and a feature fusion module, and a loss function of the joint torque prediction model is set. The joint torque prediction model is trained through the data set, and the hyperparameters of the joint torque prediction model are continuously optimized during the training process. Finally, the trained joint torque prediction model is used to predict the lower limb joint torque; that is, the lower limb joint torque is predicted by a joint torque prediction model including an embedding module, a local feature extraction module, a global feature extraction module, and a feature fusion module, and the embedding module is used to use value embedding and Position embedding maps the feature dimensions of inertial measurement data and electromyography to a high-dimensional vector space, which facilitates the model to learn and remember the temporal properties of the input sequence; the local feature extraction module is used to capture local features of different time scales from the output of the embedding module to effectively utilize the information of all local features; the global feature extraction module is used to capture global features from the output of the embedding module; the feature fusion module is used to perform bit-by-bit sum operations on local features and global features to obtain merged features, and input the merged features into a feedforward neural network for feature fusion to output the lower limb joint torque prediction results; that is, the present invention not only extracts single-scale features (local features) from the time domain, but also pays attention to global features to effectively capture contextual relationships, ultimately greatly improving the accuracy and stability of lower limb joint torque prediction.

[0128] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for predicting lower limb joint torque based on feature fusion, characterized in that: The steps include: Step S1, obtaining a large amount of historical inertial measurement data and historical electromyograms during lower limb movement, and constructing a data set after preprocessing and annotating each of the historical inertial measurement data and historical electromyograms; Step S2, creating a joint torque prediction model based on the embedding module, the local feature extraction module, the global feature extraction module and the feature fusion module, and setting a loss function of the joint torque prediction model; Step S3, training the joint torque prediction model using the data set, and continuously optimizing the hyperparameters of the joint torque prediction model during the training process; Step S4: using the trained joint torque prediction model to predict the lower limb joint torque.

2. A method for predicting lower limb joint torque based on feature fusion as claimed in claim 1, characterized in that: The step S1 is specifically as follows: A large amount of historical inertial measurement data and historical electromyograms during lower limb movement are obtained, each of the historical inertial measurement data is preprocessed to remove outliers, each of the historical electromyograms is preprocessed to enhance the image, the preprocessed historical inertial measurement data and historical electromyograms are annotated with joint torques, and a data set is constructed based on the annotated historical inertial measurement data and historical electromyograms.

3. The method for predicting lower limb joint torque based on feature fusion according to claim 1, characterized in that: In the step S2, the input end of the local feature extraction module is connected to the output end of the embedding module, and the output end is connected to the input end of the feature fusion module; the input end of the global feature extraction module is connected to the output end of the embedding module, and the output end is connected to the input end of the feature fusion module; The embedding module is used to map the feature dimensions of the historical inertial measurement data and the historical electromyogram into a high-dimensional vector space using value embedding and position embedding; The local feature extraction module is used to capture local features of different time scales from the output of the embedding module; the local feature extraction module is provided with a plurality of local feature extraction units that model different underlying time patterns and a local feature merging unit; the local feature extraction unit captures local sub-features from the output of the embedding module through one-dimensional convolution, and upsamples each local sub-feature through transposed convolution to unify the sequence length; the local feature merging unit merges each local sub-feature after unifying the sequence length through two-dimensional convolution, and outputs the local feature; The global feature extraction module is used to capture global features from the output of the embedding module; the global feature extraction module is provided with a feature mixer and a global feature extraction unit using Fourier transform; the feature mixer is used to perform feature mixing on the output of the embedding module and output mixed features; the global feature extraction unit is used to transform the mixed features from the time domain to the frequency domain, and apply the convolution theorem to capture the long-term dependency in the frequency domain, thereby obtaining the global features; The feature fusion module is used to perform bit-by-bit sum operation on local features and global features to obtain merged features, and input the merged features into a feedforward neural network for feature fusion to output a lower limb joint torque prediction result.

4. The method for predicting lower limb joint torque based on feature fusion according to claim 1, characterized in that: The step S3 is specifically as follows: Dividing the data set into a training set and a validation set based on a preset ratio, training the joint torque prediction model through the training set until the loss value of the loss function is less than a preset loss threshold, and continuously optimizing the hyperparameters of the joint torque prediction model including at least a learning rate, a batch size, a number of iterations, a convolution kernel size, and a pooling window size during the training process; The normalized root mean square error, correlation coefficient and determination coefficient of the trained joint torque prediction model are calculated through the verification set to verify the training effect of the joint torque prediction model.

5. The method for predicting lower limb joint torque based on feature fusion according to claim 1, characterized in that: The step S4 is specifically as follows: The trained joint torque prediction model is deployed to the cloud server to obtain real-time inertial measurement data and real-time electromyography for lower limb joint torque prediction. After preprocessing the real-time inertial measurement data and real-time electromyography, they are input into the joint torque prediction model deployed by the cloud server to obtain lower limb joint torque prediction results. Based on the lower limb joint torque prediction results, the joint torque prediction model is continuously optimized, and the prediction log is recorded in real time.

6. A lower limb joint torque prediction system based on feature fusion, characterized in that: Includes the following modules: A data set construction module is used to obtain a large amount of historical inertial measurement data and historical electromyograms during lower limb movement, and to construct a data set after preprocessing and annotating each of the historical inertial measurement data and historical electromyograms; A joint torque prediction model creation module is used to create a joint torque prediction model based on the embedding module, the local feature extraction module, the global feature extraction module and the feature fusion module, and set the loss function of the joint torque prediction model; A joint torque prediction model training module, used to train the joint torque prediction model using the data set, and continuously optimize the hyperparameters of the joint torque prediction model during the training process; The lower limb joint torque prediction module is used to predict the lower limb joint torque using the trained joint torque prediction model.

7. A lower limb joint torque prediction system based on feature fusion as claimed in claim 6, characterized in that: The data set construction module is specifically used for: A large amount of historical inertial measurement data and historical electromyograms during lower limb movement are obtained, each of the historical inertial measurement data is preprocessed to remove outliers, each of the historical electromyograms is preprocessed to enhance the image, the preprocessed historical inertial measurement data and historical electromyograms are annotated with joint torques, and a data set is constructed based on the annotated historical inertial measurement data and historical electromyograms.

8. A lower limb joint torque prediction system based on feature fusion as claimed in claim 6, characterized in that: In the joint torque prediction model creation module, the input end of the local feature extraction module is connected to the output end of the embedding module, and the output end is connected to the input end of the feature fusion module; the input end of the global feature extraction module is connected to the output end of the embedding module, and the output end is connected to the input end of the feature fusion module; The embedding module is used to map the feature dimensions of the historical inertial measurement data and the historical electromyogram into a high-dimensional vector space using value embedding and position embedding; The local feature extraction module is used to capture local features of different time scales from the output of the embedding module; the local feature extraction module is provided with a plurality of local feature extraction units that model different underlying time patterns and a local feature merging unit; the local feature extraction unit captures local sub-features from the output of the embedding module through one-dimensional convolution, and upsamples each local sub-feature through transposed convolution to unify the sequence length; the local feature merging unit merges each local sub-feature after unifying the sequence length through two-dimensional convolution, and outputs the local feature; The global feature extraction module is used to capture global features from the output of the embedding module; the global feature extraction module is provided with a feature mixer and a global feature extraction unit using Fourier transform; the feature mixer is used to perform feature mixing on the output of the embedding module and output mixed features; the global feature extraction unit is used to transform the mixed features from the time domain to the frequency domain, and apply the convolution theorem to capture the long-term dependency in the frequency domain, thereby obtaining the global features; The feature fusion module is used to perform bit-by-bit sum operation on local features and global features to obtain merged features, and input the merged features into a feedforward neural network for feature fusion to output a lower limb joint torque prediction result.

9. A lower limb joint torque prediction system based on feature fusion as claimed in claim 6, characterized in that: The joint torque prediction model training module is specifically used for: Dividing the data set into a training set and a validation set based on a preset ratio, training the joint torque prediction model through the training set until the loss value of the loss function is less than a preset loss threshold, and continuously optimizing the hyperparameters of the joint torque prediction model including at least a learning rate, a batch size, a number of iterations, a convolution kernel size, and a pooling window size during the training process; The normalized root mean square error, correlation coefficient and determination coefficient of the trained joint torque prediction model are calculated through the verification set to verify the training effect of the joint torque prediction model.

10. A lower limb joint torque prediction system based on feature fusion as claimed in claim 6, characterized in that: The lower limb joint torque prediction module is specifically used for: The trained joint torque prediction model is deployed to the cloud server to obtain real-time inertial measurement data and real-time electromyography for lower limb joint torque prediction. After preprocessing the real-time inertial measurement data and real-time electromyography, they are input into the joint torque prediction model deployed by the cloud server to obtain lower limb joint torque prediction results. Based on the lower limb joint torque prediction results, the joint torque prediction model is continuously optimized, and the prediction log is recorded in real time.

Images