Lower limb joint angle prediction method and system based on space-time attention

Abstract

The invention provides a lower limb joint angle prediction method and system based on space-time attention in the technical field of exoskeleton robot control, and the method comprises the steps: S1, collecting a large number of I MU historical signals through a single-node IMU sensor disposed at a shank, carrying out the preprocessing of each IMU historical signal, and constructing a data set; s2, creating a joint angle prediction model based on a multi-scale convolution module, a space-time attention module and a long-term memory module; s3, dividing the data set into a training set, a verification set and a test set based on a preset proportion, training the joint angle prediction model through the training set, verifying the trained joint angle prediction model through the verification set, and testing the joint angle prediction model passing verification through the test set; and S4, performing lower limb joint angle prediction on the joint angle prediction model passing the test. The lower limb joint angle prediction method has the advantages that the lower limb joint angle prediction precision and applicability are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of exoskeleton robot control technology, and in particular to a method and system for predicting lower limb joint angles based on spatiotemporal attention. Background Art

[0002] Exoskeleton robots show great application prospects in enhancing human movement ability and improving rehabilitation training. Exoskeleton robots recognize the user's movement intention through the collected sensor signals, thereby assisting in completing daily activities and rehabilitation training. During exercise, people usually selectively activate the corresponding muscle groups according to different movement tasks, and accurately control and adjust the joint angles to better meet their movement intentions; accurately predicting the changes in the lower limb joint angles can provide exoskeleton robots with reliable predictions of lower limb movement intentions, thereby achieving more natural and coordinated human-machine collaboration.

[0003] For the prediction of lower limb joint angles, the optical motion capture-based method is traditionally used, which is considered the gold standard for evaluating human dynamic function. However, it is limited by the operation of professional equipment and cumbersome manual data processing and can only be applied to specific scenarios. In order to overcome some limitations of traditional methods, inertial measurement units (IMU sensors) are used to predict lower limb joint angles because of their low cost, convenience, and freedom from specific scenario restrictions. Some studies combine neuromusculoskeletal models or computational models with IMU sensors. However, before using IMU sensors to collect data, each specific body part needs to be aligned and calibrated to ensure accurate motion data. Existing alignment and calibration methods are difficult to maintain consistency between different subjects and require a lot of manual settings, which limits their application in daily monitoring.

[0004] Recently, deep learning algorithms based on IMU signals (sensing signals from IMU sensors) have shown great potential in estimating kinematics. Such algorithms can effectively overcome tedious manual settings such as alignment and calibration, and directly extract relevant features from IMU signals for lower limb joint angle prediction. Although the algorithm has achieved good results, it only focuses on modeling the dependency between IMU signals and lower limb joint angles, ignoring the attention to representative features in the spatiotemporal domain and the interference of irrelevant features, which limits the accuracy of lower limb joint angle prediction; in addition, the traditional reliance on multi-node IMU sensors for modeling will be subject to certain limitations in real application scenarios.

[0005] Therefore, how to provide a lower limb joint angle prediction method and system based on spatiotemporal attention to improve the accuracy and applicability of lower limb joint angle prediction has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a method and system for predicting lower limb joint angles based on spatiotemporal attention, so as to improve the accuracy and applicability of lower limb joint angle prediction.

[0007] In a first aspect, the present invention provides a method for predicting lower limb joint angles based on spatiotemporal attention, comprising the following steps:

[0008] Step S1, collecting a large number of IMU historical signals through a single-node IMU sensor set at the calf, and constructing a data set after preprocessing each of the IMU historical signals;

[0009] Step S2, creating a joint angle prediction model based on a multi-scale convolution module, a spatiotemporal attention module, and a long-term memory module;

[0010] Step S3, dividing the data set into a training set, a validation set and a test set based on a preset ratio, training the joint angle prediction model with the training set, validating the trained joint angle prediction model with the validation set, and testing the verified joint angle prediction model with the test set;

[0011] Step S4: deploy the joint angle prediction model after the test, and predict the lower limb joint angles through the deployed joint angle prediction model.

[0012] Furthermore, the step S1 is specifically as follows:

[0013] Through a single-node IMU sensor installed at the calf of different people, a large number of IMU historical signals under different walking conditions are collected at a preset sampling frequency, and the walking conditions are flat ground, treadmill, going upstairs, going downstairs, uphill or downhill;

[0014] After filtering each of the IMU historical signals through a low-pass filter, each of the IMU historical signals is normalized and the joint angles are labeled in turn, and a data set is constructed based on the labeled IMU historical signals.

[0015] Further, in step S2, the multi-scale convolution module is used to extract multi-scale context features from the IMU historical signal, and input the multi-scale context features into the spatiotemporal attention module; the multi-scale convolution module is composed of three dilated convolution blocks with different kernel sizes; each of the dilated convolution blocks is composed of a hole convolution layer, a weight normalization layer, a nonlinear activation function layer, and a regularization layer stack;

[0016] The spatiotemporal attention module is used to extract time domain features and space domain features from multi-scale context features, perform weighted fusion on each of the time domain features and space domain features to obtain fusion features, and input the fusion features into the long-term memory module; the spatiotemporal attention module is composed of a time attention block, a space attention block and a spatiotemporal attention fusion block;

[0017] The long-term memory module is used to extract representative features from the fused features, use a gated recurrent unit to capture long-term dependent features from the representative features, flatten and map the long-term dependent features to output lower limb joint angle prediction results.

[0018] Furthermore, the step S3 is specifically as follows:

[0019] Dividing the data set into a training set, a validation set, and a test set based on a preset ratio, training the joint angle prediction model through the training set, and continuously optimizing the learning rate, initial training rounds, early stopping threshold, convolution kernel size, and optimizer function of the joint angle prediction model during the training process;

[0020] The trained joint angle prediction model is verified by the verification set to determine whether the prediction accuracy of the joint angle prediction model is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training; if yes, the verification passes, and:

[0021] The RMSE index and the PCC index of the joint angle prediction model that have passed the verification are calculated by the test set, and the performance of the joint angle prediction model is tested based on the RMSE index and the PCC index.

[0022] Furthermore, the step S4 is specifically as follows:

[0023] Deploy the joint angle prediction model after the test to the server, set the API interface called by the deployed joint angle prediction model, and set the identity authentication mechanism of the API interface;

[0024] After identity authentication through the identity authentication mechanism, the joint angle prediction model deployed on the server is accessed through the API interface to predict the lower limb joint angles.

[0025] In a second aspect, the present invention provides a lower limb joint angle prediction system based on spatiotemporal attention, comprising the following modules:

[0026] A data set construction module, used to collect a large number of IMU historical signals through a single-node IMU sensor located at the calf, and construct a data set after preprocessing each of the IMU historical signals;

[0027] A joint angle prediction model creation module, used to create a joint angle prediction model based on a multi-scale convolution module, a spatiotemporal attention module, and a long-term memory module;

[0028] A joint angle prediction model training module, for dividing the data set into a training set, a validation set and a test set based on a preset ratio, training the joint angle prediction model with the training set, validating the trained joint angle prediction model with the validation set, and testing the verified joint angle prediction model with the test set;

[0029] The lower limb joint angle prediction module is used to deploy the joint angle prediction model after the test, and predict the lower limb joint angle through the deployed joint angle prediction model.

[0030] Furthermore, the data set construction module is specifically used for:

[0031] Through a single-node IMU sensor installed at the calf of different people, a large number of IMU historical signals under different walking conditions are collected at a preset sampling frequency, and the walking conditions are flat ground, treadmill, going upstairs, going downstairs, uphill or downhill;

[0032] After filtering each of the IMU historical signals through a low-pass filter, each of the IMU historical signals is normalized and the joint angles are labeled in turn, and a data set is constructed based on the labeled IMU historical signals.

[0033] Furthermore, in the joint angle prediction model creation module, the multi-scale convolution module is used to extract multi-scale context features from the IMU historical signal, and input the multi-scale context features into the spatiotemporal attention module; the multi-scale convolution module is composed of three dilated convolution blocks with different kernel sizes; each of the dilated convolution blocks is composed of a hole convolution layer, a weight normalization layer, a nonlinear activation function layer, and a regularization layer stack;

[0034] The spatiotemporal attention module is used to extract time domain features and space domain features from multi-scale context features, perform weighted fusion on each of the time domain features and space domain features to obtain fusion features, and input the fusion features into the long-term memory module; the spatiotemporal attention module is composed of a time attention block, a space attention block and a spatiotemporal attention fusion block;

[0035] The long-term memory module is used to extract representative features from the fused features, use a gated recurrent unit to capture long-term dependent features from the representative features, flatten and map the long-term dependent features to output lower limb joint angle prediction results.

[0036] Furthermore, the joint angle prediction model training module is specifically used for:

[0037] Dividing the data set into a training set, a validation set, and a test set based on a preset ratio, training the joint angle prediction model through the training set, and continuously optimizing the learning rate, initial training rounds, early stopping threshold, convolution kernel size, and optimizer function of the joint angle prediction model during the training process;

[0038] The trained joint angle prediction model is verified by the verification set to determine whether the prediction accuracy of the joint angle prediction model is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training; if yes, the verification passes, and:

[0039] The RMSE index and the PCC index of the joint angle prediction model that have passed the verification are calculated by the test set, and the performance of the joint angle prediction model is tested based on the RMSE index and the PCC index.

[0040] Furthermore, the lower limb joint angle prediction module is specifically used for:

[0041] Deploy the joint angle prediction model that has passed the test to the server, set the API interface called by the deployed joint angle prediction model, and set the identity authentication mechanism of the API interface;

[0042] After identity authentication through the identity authentication mechanism, the joint angle prediction model deployed on the server is accessed through the API interface to predict the lower limb joint angles.

[0043] The advantages of the present invention are:

[0044] A large number of IMU historical signals are collected through a single-node IMU sensor located at the calf, and a data set is constructed after preprocessing; then a joint angle prediction model is created based on a multi-scale convolution module, a spatiotemporal attention module, and a long-term memory module, and the data set is divided into a training set, a validation set, and a test set. The joint angle prediction model is trained through the training set, the trained joint angle prediction model is verified through the validation set, and the joint angle prediction model that passes the verification is tested through the test set. Finally, the joint angle prediction model that passes the test is deployed, and the lower limb joint angle is predicted through the deployed joint angle prediction model; that is, the lower limb joint angle is predicted through the joint angle prediction model equipped with a multi-scale convolution module, a spatiotemporal attention module, and a long-term memory module; the multi-scale convolution module is used to obtain the joint angle prediction model from the IMU. Multi-scale context features are extracted from MU historical signals; the spatiotemporal attention module is used to extract time domain features and space domain features from the multi-scale context features, and weighted fusion of each time domain feature and space domain feature is performed to obtain fusion features; the long-term memory module is used to extract representative features from the fusion features, and the gated recurrent unit is used to capture long-term dependent features from the representative features, and the long-term dependent features are flattened and mapped to output the lower limb joint angle prediction results; that is, the time domain features and space domain features are fully considered in the prediction process, and representative features are further extracted from the fusion features obtained by fusion of time domain features and space domain features, which has stronger feature extraction ability, and the IMU signal is filtered by a low-pass filter before prediction to reduce the interference of irrelevant features, and the data set used to train the joint angle prediction model is based on the IMU sensor acquisition of a single node, which can be applied to more scenarios compared to multi-nodes, and ultimately greatly improves the accuracy and applicability of lower limb joint angle prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0046] Figure 1 It is a flow chart of a method for predicting lower limb joint angles based on spatiotemporal attention of the present invention.

[0047] Figure 2 It is a structural schematic diagram of a lower limb joint angle prediction system based on spatiotemporal attention of the present invention.

[0048] Figure 3 Schematic diagram of the joint angle prediction model of the present invention.

[0049] Figure 4 Schematic diagram of the spatiotemporal attention module of the present invention. DETAILED DESCRIPTION

[0050] The technical solution in the embodiment of the present application has the following overall idea: lower limb joint angle prediction is performed by a joint angle prediction model provided with a multi-scale convolution module, a spatiotemporal attention module and a long-term memory module; the multi-scale convolution module is used to extract multi-scale context features from IMU historical signals; the spatiotemporal attention module is used to extract time domain features and space domain features from the multi-scale context features, and weightedly fuse each time domain feature and space domain feature to obtain a fused feature; the long-term memory module is used to extract representative features from the fused features, and a gated recurrent unit is used to capture long-term dependent features from the representative features, and the long-term dependent features are flattened and mapped to output the lower limb joint angle prediction result; that is, the time domain features and space domain features are fully considered in the prediction process, and representative features are further extracted from the time domain features and space domain features, which has a stronger feature extraction capability, and the IMU signal is filtered by a low-pass filter before prediction to reduce the interference of irrelevant features, and the data set used to train the joint angle prediction model is based on the IMU sensor acquisition of a single node, which can be applied to more scenarios compared to multi-nodes, thereby improving the accuracy and applicability of lower limb joint angle prediction.

[0051] Please refer to Figures 1 to 4 As shown, a preferred embodiment of the method for predicting lower limb joint angles based on spatiotemporal attention of the present invention comprises the following steps:

[0052] Step S1, collecting a large number of IMU historical signals through a single-node IMU sensor set at the calf, and constructing a data set after preprocessing each of the IMU historical signals;

[0053] Step S2, creating a joint angle prediction model (TS-ConvGRU) based on a multi-scale convolution module (MSC module), a spatiotemporal attention module, and a long-term memory module (LM module);

[0054] Step S3, dividing the data set into a training set, a validation set and a test set based on a preset ratio, training the joint angle prediction model with the training set, validating the trained joint angle prediction model with the validation set, and testing the verified joint angle prediction model with the test set;

[0055] Step S4: deploy the joint angle prediction model after the test, and predict the lower limb joint angles through the deployed joint angle prediction model.

[0056] The step S1 is specifically as follows:

[0057] A single-node IMU sensor is installed at the calf of different people to collect a large number of IMU historical signals under different walking conditions at a preset sampling frequency, wherein the walking conditions are flat ground, treadmill, going upstairs, going downstairs, uphill or downhill; the sampling frequency is preferably 200 Hz;

[0058] After filtering each of the IMU historical signals through a low-pass filter (cut-off frequency: 100 Hz, Butterworth order: 6), each of the IMU historical signals is normalized and the joint angles are labeled in turn, and a data set is constructed based on the labeled IMU historical signals. Filtering through a low-pass filter is used to eliminate the interference of motion artifacts and environmental noise.

[0059] In step S2, the multi-scale convolution module is used to extract multi-scale context features from the IMU historical signal, and input the multi-scale context features into the spatiotemporal attention module; the multi-scale convolution module is composed of three dilated convolution blocks with different kernel sizes; each of the dilated convolution blocks is composed of a hole convolution layer, a weight normalization layer, a nonlinear activation function layer, and a regularization layer stack; the hole convolution layer introduces holes in the convolution kernel relative to the conventional convolution, which can significantly increase the size of the receptive field without increasing the number of parameters; the formula of the hole convolution layer is:

[0060]

[0061] Among them, F() represents the output of the hole convolution layer; x represents the IMU historical signal; k represents the total number of IMU historical signals input to the hole convolution layer; f() represents the low-pass filter; s represents the number of the IMU historical signal; seq x-d●s represents a one-dimensional sequence consisting of the input IMU historical signals; d represents the expansion factor;

[0062] The spatiotemporal attention module is used to extract time domain features and space domain features from multi-scale context features, perform weighted fusion on each of the time domain features and space domain features to obtain fusion features, and input the fusion features into the long-term memory module; the spatiotemporal attention module is composed of a time attention block, a space attention block and a spatiotemporal attention fusion block;

[0063] Considering that the use of attention mechanism will inevitably increase the number of network parameters, it is necessary to seek a trade-off between network performance and parameters; SEnet is widely used in various tasks because it can extract dependencies between multiple channels, but SEnet may lose some key local features when performing squeeze operations using global average pooling; in addition, during the excitation operation, the introduction of two fully connected layers to extract dependencies between multiple channels will increase the number of network parameters. In view of this, the present invention chooses to use the temporal attention block and the spatial attention block constructed by one-dimensional convolution operation to replace the traditional SEnet.

[0064] The temporal attention block performs a one-dimensional convolution with S channels on each column of the multi-scale context features, and then performs convolution along different rows of the multi-scale context features to extract temporal domain features:

[0065]

[0066] Among them, T i,j represents the time domain features; S represents the total number of channels; i and j represent the i-th row and j-th column of the input matrix F respectively; T represents the time step of the one-dimensional convolution; KT Represents the size of the time domain convolution kernel; m represents the index of the time domain convolution kernel in the time dimension, that is, the mth position of the time domain convolution kernel in the convolution operation; n represents the channel index of the input matrix F, that is, the nth channel currently being processed; represents the learnable parameter of the i-th channel, that is, the convolution kernel weight at the channel n and the m-th position of the convolution kernel. It is a learnable weight parameter matrix, which represents the convolution weight corresponding to the n-th row and the i-th channel; F (i',n+j) Represents the value of the nth channel and m+j time step in the input matrix F; m+j represents the input element moved by m steps in the time dimension;

[0067] The spatial attention block performs a one-dimensional convolution with a time step of T on each row of the multi-scale context feature, and then performs convolution along different columns of the multi-scale context feature to extract spatial domain features:

[0068]

[0069] Among them, S i,j Represents the spatial domain characteristics; K C Represents the size of the spatial domain convolution kernel; m' represents the index of the spatial domain convolution kernel on the channel, that is, the m'th position of the spatial domain convolution kernel in the convolution operation; n' represents the time step index of the input matrix F, that is, the n'th time step currently processing the input; represents the learnable parameters of the jth channel, that is, the weight parameters at the n'th row and the m'th position of the spatial domain convolution kernel; F( m,+i,n') represents the value of the m'+ith channel and n'th time step in the input matrix F; m+j represents taking data on the m'+ith channel along the channel dimension;

[0070] The formula of the spatiotemporal attention fusion block is:

[0071] F m,n =σ(T m,n ●S m,n );

[0072] Among them, F m,n Represents fusion features; σ() represents the sigmoid activation function;

[0073] The long-term memory module is used to extract representative features from the fused features, use a gated recurrent unit (GRU) to capture long-term dependent features from the representative features, flatten and map the long-term dependent features to output lower limb joint angle prediction results.

[0074] The capture process of the gated recurrent unit is:

[0075] z t =σ(W z x t +U z h t-1 +b z );

[0076] r t =σ(W r x t +U r h t-1 +b r );

[0077]

[0078] Among them, z t represents the activation value of the update gate; W z Represents the update gate input x t The corresponding weight matrix; x t Represents the input vector at the current moment; U z Indicates the last hidden state h of the update gate t-1 The corresponding weight matrix; h t-1 Indicates the hidden state of the previous moment; b z represents the bias term in the update gate; r t Represents the activation value of the reset gate; W r Represents the reset gate input x t The corresponding weight matrix; U r Indicates resetting the hidden state h of the door at the previous moment t-1 The corresponding weight matrix; br represents the bias term in the reset gate; represents the candidate hidden state; W represents the input x in the candidate state t The corresponding weight matrix; U represents h in the candidate state t-1 The corresponding weight matrix; h t Indicates the current hidden state.

[0079] The step S3 is specifically as follows:

[0080] Dividing the data set into a training set, a validation set, and a test set based on a preset ratio, training the joint angle prediction model through the training set, and continuously optimizing the learning rate, initial training rounds, early stopping threshold, convolution kernel size, and optimizer function of the joint angle prediction model during the training process;

[0081] Since the spatiotemporal attention module mainly involves one-dimensional convolution operations, the size of the convolution kernel has a direct impact on the prediction performance. A convolution kernel that is too large may cause the model to pay too much attention to irrelevant features, while a convolution kernel that is too small may not pay attention to the long-term dependencies between features. Therefore, the training process continuously optimizes the size of the convolution kernel; the optimizer function uses Adam; by continuously optimizing the early stopping threshold, overfitting can be avoided during the training process of the joint angle prediction model;

[0082] The trained joint angle prediction model is verified by the verification set to determine whether the prediction accuracy of the joint angle prediction model is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training; if yes, the verification passes, and:

[0083] The RMSE index and the PCC index of the joint angle prediction model that have passed the verification are calculated by the test set, and the performance of the joint angle prediction model is tested based on the RMSE index and the PCC index.

[0084] The step S4 is specifically as follows:

[0085] Deploy the joint angle prediction model that has passed the test to the server, set the API interface called by the deployed joint angle prediction model, and set the identity authentication mechanism of the API interface;

[0086] After identity authentication through the identity authentication mechanism, the joint angle prediction model deployed on the server is accessed through the API interface to predict the lower limb joint angles.

[0087] A preferred embodiment of a lower limb joint angle prediction system based on spatiotemporal attention of the present invention comprises the following modules:

[0088] A data set construction module, used to collect a large number of IMU historical signals through a single-node IMU sensor located at the calf, and construct a data set after preprocessing each of the IMU historical signals;

[0089] A joint angle prediction model creation module is used to create a joint angle prediction model (TS-ConvGRU) based on a multi-scale convolution module (MSC module), a spatiotemporal attention module, and a long-term memory module (LM module);

[0090] A joint angle prediction model training module, for dividing the data set into a training set, a validation set and a test set based on a preset ratio, training the joint angle prediction model with the training set, validating the trained joint angle prediction model with the validation set, and testing the verified joint angle prediction model with the test set;

[0091] The lower limb joint angle prediction module is used to deploy the joint angle prediction model after the test, and predict the lower limb joint angle through the deployed joint angle prediction model.

[0092] The data set construction module is specifically used for:

[0093] A single-node IMU sensor is installed at the calf of different people to collect a large number of IMU historical signals under different walking conditions at a preset sampling frequency, wherein the walking conditions are flat ground, treadmill, going upstairs, going downstairs, uphill or downhill; the sampling frequency is preferably 200 Hz;

[0094] After filtering each of the IMU historical signals through a low-pass filter (cut-off frequency: 100 Hz, Butterworth order: 6), each of the IMU historical signals is normalized and the joint angles are labeled in turn, and a data set is constructed based on the labeled IMU historical signals. Filtering through a low-pass filter is used to eliminate the interference of motion artifacts and environmental noise.

[0095] In the joint angle prediction model creation module, the multi-scale convolution module is used to extract multi-scale context features from the IMU historical signal, and input the multi-scale context features into the spatiotemporal attention module; the multi-scale convolution module is composed of three dilated convolution blocks with different kernel sizes; each of the dilated convolution blocks is composed of a hole convolution layer, a weight normalization layer, a nonlinear activation function layer and a regularization layer stack; the hole convolution layer introduces holes in the convolution kernel relative to conventional convolution, which can significantly increase the size of the receptive field without increasing the number of parameters; the formula of the hole convolution layer is:

[0096]

[0097] Among them, F() represents the output of the hole convolution layer; x represents the IMU historical signal; k represents the total number of IMU historical signals input to the hole convolution layer; f() represents the low-pass filter; s represents the number of the IMU historical signal; seq x-d●s represents a one-dimensional sequence consisting of the input IMU historical signals; d represents the expansion factor;

[0098] The spatiotemporal attention module is used to extract time domain features and space domain features from multi-scale context features, perform weighted fusion on each of the time domain features and space domain features to obtain fusion features, and input the fusion features into the long-term memory module; the spatiotemporal attention module is composed of a time attention block, a space attention block and a spatiotemporal attention fusion block;

[0099] Considering that the use of attention mechanism will inevitably increase the number of network parameters, it is necessary to seek a trade-off between network performance and parameters; SEnet is widely used in various tasks because it can extract dependencies between multiple channels, but SEnet may lose some key local features when performing squeeze operations using global average pooling; in addition, during the excitation operation, the introduction of two fully connected layers to extract dependencies between multiple channels will increase the number of network parameters. In view of this, the present invention chooses to use the temporal attention block and the spatial attention block constructed by one-dimensional convolution operation to replace the traditional SEnet.

[0100] The temporal attention block performs a one-dimensional convolution with S channels on each column of the multi-scale context features, and then performs convolution along different rows of the multi-scale context features to extract temporal domain features:

[0101]

[0102] Among them, T i,j represents the time domain features; S represents the total number of channels; i and j represent the i-th row and j-th column of the input matrix F respectively; T represents the time step of the one-dimensional convolution; KT Represents the size of the time domain convolution kernel; m represents the index of the time domain convolution kernel in the time dimension, that is, the mth position of the time domain convolution kernel in the convolution operation; n represents the channel index of the input matrix F, that is, the nth channel currently being processed; represents the learnable parameter of the i-th channel, that is, the convolution kernel weight at the channel n and the m-th position of the convolution kernel. It is a learnable weight parameter matrix, which represents the convolution weight corresponding to the n-th row and the i-th channel; F (i',n+j) Represents the value of the nth channel and m+j time step in the input matrix F; m+j represents the input element moved by m steps in the time dimension;

[0103] The spatial attention block performs a one-dimensional convolution with a time step of T on each row of the multi-scale context feature, and then performs convolution along different columns of the multi-scale context feature to extract spatial domain features:

[0104]

[0105] Among them, S i,j Represents the spatial domain characteristics; K C Represents the size of the spatial domain convolution kernel; m' represents the index of the spatial domain convolution kernel on the channel, that is, the m'th position of the spatial domain convolution kernel in the convolution operation; n' represents the time step index of the input matrix F, that is, the n'th time step currently processing the input; represents the learnable parameters of the jth channel, that is, the weight parameters at the n'th row and the m'th position of the spatial domain convolution kernel; F( m,+i,n' ) represents the value of the m'+ith channel and n'th time step in the input matrix F; m+j represents taking data on the m'+ith channel along the channel dimension;

[0106] The formula of the spatiotemporal attention fusion block is:

[0107] F m,n =σ(T m,n ●S m,n );

[0108] Among them, F m,n Represents fusion features; σ() represents the sigmoid activation function;

[0109] The long-term memory module is used to extract representative features from the fused features, use a gated recurrent unit (GRU) to capture long-term dependent features from the representative features, flatten and map the long-term dependent features to output lower limb joint angle prediction results.

[0110] The capture process of the gated recurrent unit is:

[0111] z t =σ(W z x t +U z h t-1 +b z );

[0112] r t =σ(W r x t +U r h t-1 +b r );

[0113]

[0114] Among them, z t represents the activation value of the update gate; W z Represents the update gate input x t The corresponding weight matrix; x t Represents the input vector at the current moment; U z Indicates the last hidden state h of the update gate t-1 The corresponding weight matrix; h t-1 Indicates the hidden state of the previous moment; b z represents the bias term in the update gate; r t Represents the activation value of the reset gate; W r Represents the reset gate input x t The corresponding weight matrix; U r Indicates resetting the hidden state h of the door at the previous moment t-1 The corresponding weight matrix; b r represents the bias term in the reset gate; represents the candidate hidden state; W represents the input x in the candidate state t The corresponding weight matrix; U represents h in the candidate state t-1 The corresponding weight matrix; h t Indicates the current hidden state.

[0115] The joint angle prediction model training module is specifically used for:

[0116] Dividing the data set into a training set, a validation set, and a test set based on a preset ratio, training the joint angle prediction model through the training set, and continuously optimizing the learning rate, initial training rounds, early stopping threshold, convolution kernel size, and optimizer function of the joint angle prediction model during the training process;

[0117] Since the spatiotemporal attention module mainly involves one-dimensional convolution operations, the size of the convolution kernel has a direct impact on the prediction performance. A convolution kernel that is too large may cause the model to pay too much attention to irrelevant features, while a convolution kernel that is too small may not pay attention to the long-term dependencies between features. Therefore, the training process continuously optimizes the size of the convolution kernel; the optimizer function uses Adam; by continuously optimizing the early stopping threshold, overfitting can be avoided during the training process of the joint angle prediction model;

[0118] The trained joint angle prediction model is verified by the verification set to determine whether the prediction accuracy of the joint angle prediction model is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training; if yes, the verification passes, and:

[0119] The RMSE index and the PCC index of the joint angle prediction model that have passed the verification are calculated by the test set, and the performance of the joint angle prediction model is tested based on the RMSE index and the PCC index.

[0120] The lower limb joint angle prediction module is specifically used for:

[0121] Deploy the joint angle prediction model that has passed the test to the server, set the API interface called by the deployed joint angle prediction model, and set the identity authentication mechanism of the API interface;

[0122] After identity authentication through the identity authentication mechanism, the joint angle prediction model deployed on the server is accessed through the API interface to predict the lower limb joint angles.

[0123] In summary, the advantages of the present invention are:

[0124] A large number of IMU historical signals are collected through a single-node IMU sensor located at the calf, and a data set is constructed after preprocessing; then a joint angle prediction model is created based on a multi-scale convolution module, a spatiotemporal attention module, and a long-term memory module, and the data set is divided into a training set, a validation set, and a test set. The joint angle prediction model is trained through the training set, the trained joint angle prediction model is verified through the validation set, and the joint angle prediction model that passes the verification is tested through the test set. Finally, the joint angle prediction model that passes the test is deployed, and the lower limb joint angle is predicted through the deployed joint angle prediction model; that is, the lower limb joint angle is predicted through the joint angle prediction model equipped with a multi-scale convolution module, a spatiotemporal attention module, and a long-term memory module; the multi-scale convolution module is used to obtain the joint angle prediction model from the IMU. Multi-scale context features are extracted from MU historical signals; the spatiotemporal attention module is used to extract time domain features and space domain features from the multi-scale context features, and weighted fusion of each time domain feature and space domain feature is performed to obtain fusion features; the long-term memory module is used to extract representative features from the fusion features, and the gated recurrent unit is used to capture long-term dependent features from the representative features, and the long-term dependent features are flattened and mapped to output the lower limb joint angle prediction results; that is, the time domain features and space domain features are fully considered in the prediction process, and representative features are further extracted from the fusion features obtained by fusion of time domain features and space domain features, which has stronger feature extraction ability, and the IMU signal is filtered by a low-pass filter before prediction to reduce the interference of irrelevant features, and the data set used to train the joint angle prediction model is based on the IMU sensor acquisition of a single node, which can be applied to more scenarios compared to multi-nodes, and ultimately greatly improves the accuracy and applicability of lower limb joint angle prediction.

[0125] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for predicting lower limb joint angles based on spatiotemporal attention, characterized in that: The steps include: Step S1, collecting a large number of IMU historical signals through a single-node IMU sensor arranged at the calf, and constructing a data set after preprocessing each of the IMU historical signals; Step S2, creating a joint angle prediction model based on a multi-scale convolution module, a spatiotemporal attention module, and a long-term memory module; Step S3, dividing the data set into a training set, a validation set and a test set based on a preset ratio, training the joint angle prediction model with the training set, validating the trained joint angle prediction model with the validation set, and testing the verified joint angle prediction model with the test set; Step S4: deploy the joint angle prediction model after the test, and predict the lower limb joint angles through the deployed joint angle prediction model.

2. A method for predicting lower limb joint angles based on spatiotemporal attention as claimed in claim 1, characterized in that: The step S1 is specifically as follows: A single-node IMU sensor is installed at the calf of different people to collect a large number of IMU historical signals under different walking conditions at a preset sampling frequency, wherein the walking conditions are flat ground, treadmill, going upstairs, going downstairs, uphill or downhill; After filtering each of the IMU historical signals through a low-pass filter, each of the IMU historical signals is normalized and the joint angles are labeled in turn, and a data set is constructed based on the labeled IMU historical signals.

3. The method for predicting lower limb joint angles based on spatiotemporal attention as claimed in claim 1, characterized in that: In step S2, the multi-scale convolution module is used to extract multi-scale context features from the IMU historical signal, and input the multi-scale context features into the spatiotemporal attention module; the multi-scale convolution module is composed of three dilated convolution blocks with different kernel sizes; each of the dilated convolution blocks is composed of a hole convolution layer, a weight normalization layer, a nonlinear activation function layer, and a regularization layer stack; The spatiotemporal attention module is used to extract time domain features and space domain features from multi-scale context features, perform weighted fusion on each of the time domain features and space domain features to obtain fusion features, and input the fusion features into the long-term memory module; the spatiotemporal attention module is composed of a time attention block, a space attention block and a spatiotemporal attention fusion block; The long-term memory module is used to extract representative features from the fused features, use a gated recurrent unit to capture long-term dependent features from the representative features, flatten and map the long-term dependent features to output lower limb joint angle prediction results.

4. The method for predicting lower limb joint angles based on spatiotemporal attention as claimed in claim 1, characterized in that: The step S3 is specifically as follows: Dividing the data set into a training set, a validation set, and a test set based on a preset ratio, training the joint angle prediction model through the training set, and continuously optimizing the learning rate, initial training rounds, early stopping threshold, convolution kernel size, and optimizer function of the joint angle prediction model during the training process; The trained joint angle prediction model is verified by the verification set to determine whether the prediction accuracy of the joint angle prediction model is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training; if yes, the verification passes, and: The RMSE index and the PCC index of the joint angle prediction model that have passed the verification are calculated by the test set, and the performance of the joint angle prediction model is tested based on the RMSE index and the PCC index.

5. The method for predicting lower limb joint angles based on spatiotemporal attention as claimed in claim 1, characterized in that: The step S4 is specifically as follows: Deploy the joint angle prediction model that has passed the test to the server, set the API interface called by the deployed joint angle prediction model, and set the identity authentication mechanism of the API interface; After identity authentication through the identity authentication mechanism, the joint angle prediction model deployed on the server is accessed through the API interface to predict the lower limb joint angles.

6. A lower limb joint angle prediction system based on spatiotemporal attention, characterized in that: Includes the following modules: A data set construction module, used to collect a large number of IMU historical signals through a single-node IMU sensor located at the calf, and construct a data set after preprocessing each of the IMU historical signals; A joint angle prediction model creation module, used to create a joint angle prediction model based on a multi-scale convolution module, a spatiotemporal attention module, and a long-term memory module; A joint angle prediction model training module, for dividing the data set into a training set, a validation set and a test set based on a preset ratio, training the joint angle prediction model with the training set, validating the trained joint angle prediction model with the validation set, and testing the verified joint angle prediction model with the test set; The lower limb joint angle prediction module is used to deploy the joint angle prediction model after the test, and predict the lower limb joint angle through the deployed joint angle prediction model.

7. A lower limb joint angle prediction system based on spatiotemporal attention as claimed in claim 6, characterized in that: The data set construction module is specifically used for: A single-node IMU sensor is installed at the calf of different people to collect a large number of IMU historical signals under different walking conditions at a preset sampling frequency, wherein the walking conditions are flat ground, treadmill, going upstairs, going downstairs, uphill or downhill; After filtering each of the IMU historical signals through a low-pass filter, each of the IMU historical signals is normalized and the joint angles are labeled in turn, and a data set is constructed based on the labeled IMU historical signals.

8. A lower limb joint angle prediction system based on spatiotemporal attention as claimed in claim 6, characterized in that: In the joint angle prediction model creation module, the multi-scale convolution module is used to extract multi-scale context features from the IMU historical signal, and input the multi-scale context features into the spatiotemporal attention module; the multi-scale convolution module is composed of three dilated convolution blocks with different kernel sizes; each of the dilated convolution blocks is composed of a hole convolution layer, a weight normalization layer, a nonlinear activation function layer and a regularization layer stack; The spatiotemporal attention module is used to extract time domain features and space domain features from multi-scale context features, perform weighted fusion on each of the time domain features and space domain features to obtain fusion features, and input the fusion features into the long-term memory module; the spatiotemporal attention module is composed of a time attention block, a space attention block and a spatiotemporal attention fusion block; The long-term memory module is used to extract representative features from the fused features, use a gated recurrent unit to capture long-term dependent features from the representative features, flatten and map the long-term dependent features to output lower limb joint angle prediction results.

9. A lower limb joint angle prediction system based on spatiotemporal attention as claimed in claim 6, characterized in that: The joint angle prediction model training module is specifically used for: Dividing the data set into a training set, a validation set, and a test set based on a preset ratio, training the joint angle prediction model through the training set, and continuously optimizing the learning rate, initial training rounds, early stopping threshold, convolution kernel size, and optimizer function of the joint angle prediction model during the training process; The trained joint angle prediction model is verified by the verification set to determine whether the prediction accuracy of the joint angle prediction model is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training; if yes, the verification passes, and: The RMSE index and the PCC index of the joint angle prediction model that have passed the verification are calculated by the test set, and the performance of the joint angle prediction model is tested based on the RMSE index and the PCC index.

10. A lower limb joint angle prediction system based on spatiotemporal attention as claimed in claim 6, characterized in that: The lower limb joint angle prediction module is specifically used for: Deploy the joint angle prediction model that has passed the test to the server, set the API interface called by the deployed joint angle prediction model, and set the identity authentication mechanism of the API interface; After identity authentication through the identity authentication mechanism, the joint angle prediction model deployed on the server is accessed through the API interface to predict the lower limb joint angles.

Images