A method for continuous prediction of movement for cerebral palsy patients
By combining TCN and GAT modules to extract multimodal features, using BiLSTM network for bidirectional temporal modeling, and calibrating through multi-subject group correlation loss function and affine transformation, high-precision continuous prediction of movement in cerebral palsy patients is achieved. This solves the problems of poor cross-subject generalization ability and large individual differences in traditional methods, and provides accurate rehabilitation assessment and auxiliary training support.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- UNIV OF SHANGHAI FOR SCI & TECH
- Filing Date
- 2026-02-06
- Publication Date
- 2026-06-09
AI Technical Summary
Existing technologies are insufficient to fully exploit the complementary information of surface electromyography and inertial sensors in cerebral palsy patients, dynamically model muscle synergy, effectively address the correlation between individual differences and pathological fluctuations, and solve specific problems that traditional methods cannot effectively address.
The TCN and GAT modules are used in parallel to extract IMU temporal features and sEMG spatial features respectively. The two types of features are integrated through a multimodal fusion module. The BiLSTM network is used to capture the bidirectional temporal dependence of actions, and prediction is performed through a regression output module. Personalized prediction is achieved by combining a composite loss function of multi-subject group correlation and affine transformation calibration.
It significantly improves the accuracy and continuity of motion prediction for patients with cerebral palsy, providing precise technical support for the assessment of motor rehabilitation and personalized auxiliary training for patients with cerebral palsy, and solving the problems of poor cross-subject generalization ability and large individual differences in traditional methods.
Smart Images
Figure CN122174200A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to biomedical signal processing and artificial intelligence, and in particular to a method for predicting continuous motion in patients with cerebral palsy based on surface electromyography and inertial sensors. Background Technology
[0002] With the development of rehabilitation medicine and intelligent exoskeleton technology, continuous and accurate prediction of lower limb joint angles in children with cerebral palsy has become a core requirement for achieving personalized assistive control and quantitative assessment of motor function.
[0003] Cerebral palsy patients often experience abnormal muscle tone, muscle dyssynergy, and pathological reflexes due to central nervous system damage. This results in highly individualized, time-varying, and fluctuating lower limb joint movement patterns, making traditional discrete movement classification methods inadequate for clinical needs in continuous trajectory prediction and real-time assisted control. Surface electromyography (sEMG), as a non-invasive and real-time acquireable physiological signal, can reflect muscle activation intentions before movement occurs, making it one of the most promising input modalities for recognizing movement intentions in cerebral palsy patients. Combining sEMG with an inertial measurement unit (IMU) can further compensate for the non-stationarity of sEMG during rapid movement or high muscle tone, achieving multimodal complementarity.
[0004] Existing technologies for continuous prediction of lower limb movement in patients with cerebral palsy have the following main shortcomings: First, most existing methods use a single network structure (such as pure CNN, LSTM or Transformer) or a simple feature splicing method, which makes it difficult to simultaneously capture the rich temporal envelope, frequency energy and pose dynamic information in sEMG signals and IMU signals. This results in a significant decrease in prediction accuracy in complex scenarios such as abnormal muscle tone bursts, spasticity or coordinated contraction in cerebral palsy patients.
[0005] Secondly, existing technologies are severely inadequate in modeling muscle synergy. Cerebral palsy patients often exhibit pathological synergistic patterns (such as abnormal co-contraction of the rectus femoris and biceps femoris). Traditional methods process the electromyographic signals of each channel independently or only perform superficial splicing, which cannot effectively uncover the dynamic coupling relationship between multiple muscles, resulting in significant phase lag or waveform distortion in the prediction curve.
[0006] Third, poor cross-subject generalization ability is the biggest bottleneck restricting clinical application. There are significant differences among individual cerebral palsy patients in muscle tone grading (GMFCS IV), degree of muscle atrophy, fitting position error, and abnormal compensation patterns. Existing methods often employ global standardization or global bias correction, resulting in poor generalization ability across subjects. 2Typically, the accuracy drops to 0.6~0.75, and the RMSE is as high as 12°~20° (even worse for patients with cerebral palsy, reaching 37°), which cannot meet the control precision requirements of exoskeletons or rehabilitation robots.
[0007] Therefore, there is an urgent need for a method that can fully exploit the complementary information between surface electromyography signals and inertial signals, effectively model muscle synergy, and be robust to individual differences and pathological fluctuations in the continuous prediction of lower limb joint angles, so as to achieve continuous and accurate prediction of movement in patients with cerebral palsy and provide reliable and practical upper-level control signals for intelligent rehabilitation exoskeleton robots.
[0008] A search revealed Chinese invention patent application publication number CN119318470A, which discloses a method for determining the predicted value of limb neurological rehabilitation status based on multimodal sensor data fusion. This method inputs motion inertial data features, electromyographic signal data features, and text semantic features into a trained multimodal data feature fusion network to obtain multimodal fusion features. These multimodal fusion features are then input into a trained prediction output network to obtain the predicted value of the patient's limb neurological rehabilitation status. The multimodal data feature extraction network includes a time-dependent feature extraction module and a text semantic feature extraction module. Motion inertial data and electromyographic signal data are input into the time-dependent feature extraction module to obtain motion inertial data features and electromyographic signal data features, while the text data from the limb neurological rehabilitation manual is input into the text semantic feature extraction module to obtain text semantic features. This existing patent application lacks dynamic capture of the dynamic coordination of the affected muscle groups in cerebral palsy patients, resulting in weak cross-patient generalization ability and insufficient accuracy in joint angle prediction.
[0009] How to achieve continuous and accurate motion prediction for patients with cerebral palsy has become a technical problem that needs to be solved. Summary of the Invention
[0010] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a method for predicting continuous movement in patients with cerebral palsy.
[0011] The objective of this invention can be achieved through the following technical solutions: According to one aspect of the present invention, a method for predicting continuous motion in patients with cerebral palsy is provided, the method comprising: Acquire sEMG data, IMU data, and true values of joint angles for the subjects, and divide the collected data into training set, validation set, and test set for each subject in chronological order; The collected EMG and IMU data from the subjects were standardized to obtain standardized sequence data for each subject. The standardized sequence data of each subject are input into a trained prediction network, which includes a parallel TCN module and GAT module, a multimodal fusion module, a BiLSTM network, and a regression output module. The TCN module extracts IMU temporal features, and the GAT module extracts sEMG spatial features. The two types of features are fused by the multimodal fusion module and then input into the BiLSTM network for bidirectional temporal modeling to capture the bidirectional temporal dependence of actions. The standardized prediction angle is then output through the regression output module.
[0012] As a preferred technical solution, the TCN module includes three stacked dilated convolutional blocks, with dilation rates of 1, 2, and 4 for each layer, and each dilated convolutional block contains two layers of residual dilated convolution, a ReLU activation function, regularization, and pruning operations.
[0013] As a preferred technical solution, the processing flow of the TCN module includes: transposing the IMU standardized sequence into an input format adapted to one-dimensional convolution, extracting temporal features through three stacked dilated convolutional blocks, and then transposing it back to an IMU hidden feature sequence with the same input time length, i.e., IMU temporal features.
[0014] As a preferred technical solution, the GAT module treats each channel signal of sEMG as a fully connected dynamic graph structure, where channels are regarded as nodes in the graph structure and the collaborative activation between channels is regarded as edges.
[0015] As a preferred technical solution, the processing flow of the GAT module includes: expanding the sEMG normalized sequence from a vector dimension to a "matrix" input form that adapts to linear projection; Linear projection is performed on each channel at each time step, mapping it to a preset hidden dimension; Calculate the attention scores between nodes, and obtain the attention weight matrix by Softmax normalization; The sEMG global feature sequence, i.e. sEMG spatial features, is obtained by weighting and aggregating the features of neighboring nodes based on the attention weight matrix and averaging the features of each node at the same time step.
[0016] As a preferred technical solution, the method further includes the steps of constructing windowed samples and equalizing subject sampling before training the prediction network: constructing "input sequence-regression label" sample pairs in each set using a sliding window, wherein the input sequence is the sEMG and IMU standardized sequence data within the window, and the regression label is the true value of the joint angle at the end of the window; during the training phase, each sample is assigned a sampling weight that is inversely proportional to the number of training samples of the subject to which it belongs, and training samples are extracted based on the sampling weight.
[0017] As a preferred technical solution, the prediction model is trained using a composite loss function based on the correlation of multiple subject groups. This composite loss function includes a basic term and a regularization term, where the basic term is the prediction output. Compared to the actual joint angle The mean squared error between them; the regularization term is a correlation penalty term based on subject grouping.
[0018] As a preferred technical solution, the method further de-standardizes the standardized prediction angle output by the prediction model to obtain an uncalibrated prediction angle. Based on the subject to which the current sample belongs, select the corresponding affine transformation parameters from the calibration parameter table of the corresponding subject, perform an affine transformation on the uncalibrated prediction angle, and obtain the calibrated prediction angle.
[0019] As a preferred technical solution, the process of obtaining the affine transformation parameters for each subject includes: For each subject, the standardized predicted angle sequence and the standardized true angle sequence output by the prediction model on the validation set are collected. Assuming a standardized real perspective From a standardized prediction perspective Approximate representation by affine transformation: , The least squares method was used to estimate the radiotransformation parameters corresponding to subject s, including personalized scaling factors. and offset coefficient ; The affine parameters for each subject are stored as a calibration parameter table.
[0020] As a preferred technical solution, the sEMG data is collected from the tibialis anterior, medial gastrocnemius, rectus femoris, and semitendinosus muscles on the affected side of the subject.
[0021] Compared with the prior art, the present invention has the following beneficial effects: 1) This application addresses the scenario of continuous motion prediction for patients with cerebral palsy. It constructs a dataset by collecting sEMG data, IMU data, and ground truth joint angles from the subjects. The multi-source data is standardized and divided into training, validation, and test sets to ensure the standardization of data input and the effectiveness of model training. The prediction network uses parallel TCN and GAT modules to extract IMU temporal features and sEMG spatial features respectively, achieving accurate feature mining of multimodal data and overcoming the limitation of limited information dimensions in single-modal data. After integrating the two types of features through a multimodal fusion module, a BiLSTM network is used to capture the bidirectional temporal dependence of movements, adapting to the temporal characteristics of continuous motion. This solves the problem that traditional prediction methods can only model unidirectional temporal relationships and cannot fully characterize the laws of continuous motion. Finally, a standardized prediction angle is obtained through a regression output module, significantly improving the accuracy and continuity of continuous motion prediction for patients with cerebral palsy. This provides precise technical support for the assessment of motor rehabilitation and the development of personalized auxiliary training programs for patients with cerebral palsy.
[0022] 2) The TCN module in this invention features an optimized structure. Three stacked dilated convolutional blocks with dilation rates of 1, 2, and 4 progressively expand the receptive field, effectively capturing features at different time scales in IMU time-series data. This allows for the extraction of short-term fine posture features during the movement of cerebral palsy patients while also considering long-term movement trends, thus solving the problem of limited receptive field and inability to cover long-term IMU data features in traditional one-dimensional convolution. The two residual dilated convolutional layers within each dilated convolutional block effectively alleviate the gradient vanishing problem in deep network training, improving the stability and effectiveness of feature extraction. The ReLU activation function introduces non-linear characteristics into feature representation, regularization reduces the risk of model overfitting, and pruning ensures the dimensionality consistency of feature data, resulting in a significant improvement in the quality of the final output IMU time-series features.
[0023] 3) This invention models the signals of each sEMG channel as a fully connected dynamic graph structure, with channels as nodes and inter-channel co-activation as edges, which fits the physiological characteristics of the dynamic coordination of the muscle groups on the affected side of cerebral palsy patients. The fully connected dynamic graph can capture the correlation of dynamic changes between channels in real time, and can better reflect the spatial dynamic characteristics of electromyography signals compared with static graph structures. It effectively improves the accuracy and dynamic adaptability of sEMG spatial feature extraction, and provides electromyography spatial features that are more in line with clinical scenarios for subsequent multimodal fusion.
[0024] 4) This invention uses a composite loss function based on the correlation of multiple subjects to train the model. The mean square error of the basic term can directly constrain the numerical error between the predicted value and the actual joint angle, ensuring the prediction accuracy of a single subject. The regularization term introduces the correlation penalty of multiple subjects. By calculating the correlation coefficient between the predicted sequence and the actual sequence of the same subject, the correlation loss is weighted and superimposed into the overall loss. This design effectively solves the problem of insufficient model generalization caused by large individual differences among cerebral palsy patients, making the model more suitable for rehabilitation assessment scenarios of multiple patients in clinical practice.
[0025] 5) This invention also performs de-standardization and affine transformation calibration on the standardized predicted angles output by the model. First, de-standardization restores the standardized angles output by the model to the numerical range of the actual joint angles, solving the angle offset problem caused by the standardization operation in the preprocessing stage. Then, based on the current subject, personalized affine transformation parameters are selected from the calibration parameter table for calibration. Taking into account the differences in joint range of motion and electromyographic signal amplitude among individual cerebral palsy patients, the systematic bias of the predicted angle is further corrected, so that the final output calibrated angle is more in line with the actual movement state of the patient, significantly improving the clinical applicability of the prediction results and providing more accurate quantitative basis for rehabilitation assessment, assisted training and other scenarios. Attached Figure Description
[0026] Figure 1 This is a flowchart illustrating the method for predicting continuous movement in patients with cerebral palsy according to the present invention. Figure 2 This is a schematic diagram illustrating the principle of dividing the training set, validation set, and test set for each subject in this invention; Figure 3 This is a schematic diagram of the prediction network used for continuous motion prediction in this invention; Figure 4 This is a schematic diagram comparing the actual joint angles with the simulated predicted joint angles over time steps. Figure 5 This is a schematic diagram showing the relationship between the actual joint angle and the predicted joint angle. Detailed Implementation
[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0028] The purpose of this invention is to overcome the problems of insufficient multimodal data fusion, weak individual difference modeling ability, and lack of calibration methods in the existing technology, and to achieve high-precision, robust, and individually adjustable joint angle estimation for cerebral palsy patients.
[0029] This embodiment relates to a method for continuous motion prediction in patients with cerebral palsy. This method utilizes multimodal data from both IMU and EMG, and designs differentiated prediction network structures (TCN + GAT) for different modalities to predict motion. In a multi-subject task, balanced modeling is performed for each subject using a loss function and sampling strategy. After the prediction network is trained, individualized calibration for each subject is achieved through lightweight affine transformation.
[0030] like Figure 1 The method includes the following steps: S1: Subject Data Collection and Segmentation (1) Kinematic data of each subject is collected by wearable IMU sensors to form first time series data to reflect the limb movement state; (2) Collect key lower limb electromyography signals (4-channel sEMG signals reflecting muscle activation state) for each subject through the EMG acquisition module to form second time series data; (3) Collect the true values of joint angles at the corresponding time using a motion capture system to form target time series data; (4) According to the preset subject sequence, splice the entire time series and record the data length corresponding to each subject; (5) For each subject's time period, divide it into training set, validation set, and test set in chronological order, with a ratio of 8:1:1, ensuring that the three sets do not overlap, such as Figure 2 .
[0031] S2: Standardize the collected data according to the subjects' feedback. For each subject, standardized parameters (mean and variance) are fitted only on the training set of that subject for IMU data, sEMG data, and true values of joint angles. Using the fitted standardized parameters, the training set, validation set, and test set of that subject are uniformly standardized to obtain standardized sequence data of IMU data and sEMG data for the corresponding time periods for each subject.
[0032] S3: Windowed Sample Construction and Balanced Subject Sampling Sliding window operations were performed within the training, validation, and test sets for each subject: a preset window length L (200 time steps) and step size s (2 time steps) were used; within each set, a sliding window was used to construct sample sequence pairs, and the window did not cross the set boundary; for each window, the IMU and EMG standardized sequence data of the corresponding time period were used as the input sequence, and the true value of the joint angle at the end of the window was used as the regression label; the corresponding subject identifiers were stored together for subsequent loss calculation and calibration.
[0033] During the training phase: count the number of window samples generated by each subject in the training set; assign a sampling weight to each training sample that is inversely proportional to the number of subject samples; use a weighted random sampler to extract training samples according to the sampling weight, so that the contribution of each subject to parameter updates during training is roughly balanced.
[0034] S4: Constructing a prediction network with a multimodal deep neural network structure A deep neural network is constructed for joint angle regression prediction as the prediction network. For example... Figure 3 The prediction network includes a parallel TCN module (multilayer temporal convolutional network) and GAT module (graph attention network), a multimodal fusion module, and a regression output module.
[0035] 4-1, IMU temporal feature extraction branch (temporal convolutional network: TCN module), its processing includes: 1) Dimensional adaptation: Transpose the input IMU normalized sequence data to (Batch, 9, Time) to bring the channel dimension to the front, adapting to the input format of one-dimensional convolution; 2) Hierarchical Dilated Convolution: A multi-layer one-dimensional convolutional network is employed, with each layer containing dilated convolution, residual connections, and chomp structures. The receptive field is increased by progressively increasing the dilation rate (in this embodiment, three stacked temporal blocks with dilation rates of 1, 2, and 4) to efficiently capture long-term motion dependencies and model long-term dependency information. Chomping eliminates the output offset caused by dilated convolution, while residual connections ensure stability during gradient backpropagation.
[0036] 3) Dimension Restoration: Convert the feature dimensions back to (Batch, Time, D_HID) to obtain the IMU hidden feature sequence with the same time length as the input (D_HID is the hidden layer dimension).
[0037] 4-2, sEMG Spatial Feature Extraction Branch (Graph Attention Encoding Network: GAT Module): The sEMG channel signals are treated as a graph structure, where each sEMG channel is considered a node in the graph, and the co-activation between channels (the synchronous / correlated electrical activity characteristics exhibited by multi-channel sEMG signals during motion) is considered an edge. Specifically, the GAT module constructs a complete graph using each sEMG channel as a graph node. At each time step, attention coefficients are calculated based on the node features, and these attention coefficients are used as edge weights to weight and aggregate the features of each node, thereby obtaining features representing the correlation between channels. The 4-channel EMG at each time step constitutes 4 nodes. The processing includes: 1) Dimension expansion: The sEMG sequence dimension is expanded from (Batch, Time, 4) to (Batch, Time, 4, 1) by unsqueeze(-1) to prepare for node feature projection; 2) Node feature projection: Linear projection is performed on the 4 channel nodes of each time step to map the 1-dimensional original electromyography signal to the preset hidden dimension (hidden_dim) to obtain the node feature vector (Batch, Time, 4, hidden_dim). 3) Graph attention calculation: Based on the graph attention mechanism, attention scores are calculated for nodes as source nodes and target nodes respectively. After activation by LeakyReLU, the scores are normalized to obtain the attention weight matrix between nodes. The features of each node are weighted and summed according to the attention weight matrix to obtain the updated node features. The features of each node at the same time step are averaged to obtain the sEMG global feature sequence at that time step.
[0038] Multichannel sEMG is modeled as a graph structure, and the correlation weights between channels are automatically learned through an attention mechanism to accurately extract the spatial collaborative features of muscle activation.
[0039] 4-3, Multimodal Fusion Module, used for multimodal fusion and bidirectional LSTM: Layer Normalization is performed on the IMU hidden feature sequence and the EMG global feature sequence respectively; after temporal alignment, the two feature sequences are concatenated along the feature dimension and mapped to a fused feature sequence of a unified dimension through a linear transformation; the fused feature sequence is input into a bidirectional Long Short-Term Memory (BiLSTM) network, which performs forward and backward modeling in the temporal dimension, outputting a bidirectional hidden state sequence (Batch, Time, 2×hidden_size) containing forward and backward information. The multimodal fusion module fuses complementary features from the two modalities, and BiLSTM further captures the bidirectional temporal dependencies of actions (e.g., the current joint angle depends on both past movement trends and the intention of subsequent actions).
[0040] 4-4, Regression Output Module, used for temporal attention pooling and regression output, its processing includes: For each time step output of the bidirectional LSTM, the attention score is calculated through a feedforward neural network and normalized by softmax in the time dimension to obtain the attention weight of each time step, so that the model automatically focuses on the time segments with high contribution. The bidirectional hidden states are weighted and summed according to the attention weights to obtain the context feature vector representing the entire time window; By employing a multi-layer fully connected network and non-linear activation, contextual features are mapped to regression outputs of joint angles. The final output is a predicted sequence of joint angles (Batch, Time, 1), which is then processed... Figure 4 The accuracy of the predictions is verified by comparing the true ground truth (GT) versus the simulated prediction curves.
[0041] S5: Training the prediction network based on a composite loss function of multi-subject group correlation. The following composite loss function is used during model training: The basic term is the predicted output. Compared to the actual joint angle The mean square error (MSE) between them; The regularization term is a correlation penalty term based on subject grouping. Predicted and actual values for the same subject within the same batch of samples are grouped; for each subject, the correlation coefficient between the predicted and actual sequences is calculated; and then... The correlation loss function is used as the correlation loss function for this subject; the correlation loss of all subjects is averaged and then weighted and added to the overall loss.
[0042] The total loss can be expressed as: , in, The number of subjects within a batch. This is the correlation loss coefficient, superscript. This represents the s-th subject.
[0043] This loss function improves the trend alignment between the predictions of each subject and the actual curves while ensuring a small overall mean square error, thereby improving the fit consistency in multi-subject scenarios.
[0044] S6: Per-subject affine calibration based on the validation set After the model training is completed and the optimal parameters are selected, output calibration is further performed on each subject to address systematic biases among different subjects.
[0045] 1) De-standardization and validation set prediction: Using the standardized parameters obtained by each subject in step S2, the standardized predicted angle and standardized true angle output by the prediction model in the validation set are respectively de-standardized into physical angle units (such as "degrees"); the subject identifier corresponding to each data is retained to ensure that it can be accurately matched to the specific person in the future.
[0046] 2) Fitting affine transformation parameters for each subject: For each subject, collect the standardized predicted angle sequence and standardized true angle sequence output by the prediction model on the validation set; assuming the true angle... (The calibrated predicted angle, used to approximate the true angle) can be derived from the predicted angle. (The uncalibrated angles directly output by the model) are approximated by affine transformation:
[0047] Estimating the personalized scaling factor for subject s using the least squares method and offset coefficient The affine parameters for each subject are stored as a calibration parameter table.
[0048] 3) Calibration Inference in Testing and Practical Application: During the testing phase or when deployed to a practical system, a standardized prediction angle is first obtained using a trained deep neural network model; then, destandardization is performed using the standardized parameters from step S2 to obtain the uncalibrated prediction angle; finally, based on the subject to which the current sample belongs, the corresponding calibration parameter is selected from the calibration parameter table. and An affine transformation is performed on the uncalibrated predicted angle to obtain the calibrated predicted angle.
[0049] Calibration parameters are calculated individually for each subject to accurately eliminate systematic biases between individuals. This method improves prediction accuracy by individually compensating for systematic biases in different subjects without retraining the deep neural network (parameters are fitted only once on the validation set, and subsequent inference stages can be directly performed by looking up tables). Furthermore, this method is adaptable to multi-subject scenarios (such as rehabilitation assessment and motion capture), avoiding model bias towards specific population movement patterns.
[0050] This embodiment also relates to a method for predicting continuous movement in patients with cerebral palsy, such as... Figure 1 ,include: S1, Data Acquisition: Bipolar Ag / AgCl surface electrodes (2.5 cm electrode spacing) were used to collect surface electromyography (sEMG) signals from the lower limb muscles. Electrodes were applied to the muscle bellies of eight target muscles: bilateral tibialis anterior (TA), medial gastrocnemius (MG), rectus femoris (RF), and semitendinosus (ST). Prior to the experiment, the electrode application sites were cleaned with medical alcohol to remove surface oils and dirt, reduce skin impedance, and ensure good electrode-skin interface contact. EMG signal acquisition was performed using a Noraxon EMG acquisition system at a sampling frequency of 1500 Hz. Simultaneously, a Noraxon inertial measurement unit (IMU) was used to acquire kinematic data of the pelvis, thighs, calves, and feet at a sampling frequency of 100 Hz. Three-dimensional motion capture data was acquired using a 3D motion capture device from a measurement company, using the Helenhayes 26-point model.
[0051] S2, Data Preprocessing and Partitioning: Four-channel sEMG signals from the affected side of cerebral palsy patients were selected, including the tibialis anterior (TA, L / R), medial gastrocnemius (MG, L / R), rectus femoris (RF, L / R), and semitendinosus (ST, L / R). These sEMG signals were sequentially subjected to bandpass filtering (20-450Hz) to remove low-frequency motion artifacts and high-frequency noise, half-wave rectification, and then low-pass filtering (6Hz) for smoothing. Triaxial angle data of the affected thigh, lower leg, and foot acquired by the IMU, as well as joint angle data acquired by the optical motion capture system, were selected and subjected to low-pass filtering (10Hz) to remove high-frequency noise. Subsequently, the sEMG, IMU, and optical capture data were uniformly resampled to 200Hz, and multimodal synchronization was achieved through timestamp alignment.
[0052] Finally, the data from the 10 subjects were concatenated. For each subject's time period, it was divided chronologically into three sets: training, validation, and test, in an 8:1:1 ratio, ensuring no overlap between the three sets. The data partitioning is as follows: Figure 2 As shown.
[0053] S3, Data Standardization: To ensure consistency of feature dimensions, improve model training efficiency, and enhance robustness to outliers, this embodiment standardizes the preprocessed data. Mean centering and standard deviation scaling are used to ensure the data mean is 0 and the standard deviation is 1. For each subject, standardization is applied only to the training set, validation set, and test set of that subject using the fitted standardized parameters for IMU data, EMG data, and ground truth joint angles. Let the feature vector be... , where n is the number of samples, and the standardized mathematical expression is shown in the following formula.
[0054] , , , It is the mean. It is the standard deviation. This is the standardized value. Standardized data. It satisfies the condition that the mean is 0 and the standard deviation is 1.
[0055] S3, Windowed Sample Construction and Balanced Subject Sampling: The standardized IMU sequence and sEMG sequence are used as dual-branch inputs. The IMU input is... This corresponds to the three-axis rotation angles of the thigh, calf, and foot; sEMG input is... This corresponds to four key muscle groups in the lower limbs. Data is divided into 200-frame sliding windows (step size 2), with each sample labeled as the normalized joint angle at the end of the window. .
[0056] S4, Construct a prediction network with a multimodal deep neural network structure, the overall structure of which is as follows: Figure 3 As shown.
[0057] (1) The TCN module is used for IMU temporal feature extraction. The TCN module is built upon dilated causal convolution, achieving a massive receptive field through an exponentially increasing expansion rate while maintaining strict temporal causality. The TCN module consists of three stacked TemporalBlock layers, each containing two layers of residual dilated convolution, ReLU activation, Dropout regularization, and skip connections. The computation of a single TemporalBlock layer is shown in the formula.
[0058] , in, The width of the convolution kernel; Let be the expansion rate of the i-th layer; This represents the input features of the dilated convolution; Here, b represents the kernel weights and b represents the bias; chopping (Chomp 1d) is used to padded the redundant time steps at the ends of the convolution output to ensure the output length matches the input length; residual connections are used to connect the output. ,in This is a 1×1 convolutional mapping used to match the channel dimensions of the residual branch and the main branch. For activation function, This represents the main branch mapping of TemporalBlock (consisting of two layers of dilated one-dimensional convolutions and their nonlinearity and pruning).
[0059] TCN module output characteristics (B represents batch, T represents time step, and D represents dimension) LayerNorm is used to stabilize the feature distribution and reduce the impact of amplitude differences among different subjects / samples on training, thereby obtaining IMU temporal feature sequences for subsequent fusion and temporal modeling.
[0060] (2) GAT module, used for sEMG spatial feature extraction The four channels of the sEMG signal naturally form a fully connected dynamic graph (channels are nodes, and the muscle co-activation relationships between channels are edges), and the co-activation patterns between nodes change with the intention of movement. Traditional CNNs cannot explicitly capture this non-Euclidean relationship, while GAT adaptively allocates the information flow weights between channels through an attention mechanism, which can accurately capture the co-activation patterns of different muscle channels, providing more accurate spatial features for subsequent multimodal fusion.
[0061] For each time step (t), the 4-channel raw values are considered as the initial node embedding. ,in For the first i The initial characteristics of each channel (node), For time step t, the first i The (normalized) sEMG signal values of each channel.
[0062] First, increase the dimensionality using linear projection: , in, For the first l The linear projection weight matrix of the layer, After the first l Node features after layer projection.
[0063] Compute node pairs Attention score: , in, This is the attention parameter vector; For feature splicing operations; Use a leaky ReLU activation function to preserve negative gradients and prevent neuron death; Let i be the attention score of node i to node j. Indicates a node i eigenvectors High-dimensional feature representation after linear projection transformation.
[0064] Normalization yields the attention coefficient: , in, N For nodes i All neighboring nodes, This represents the attention coefficient obtained after normalization.
[0065] Aggregate neighborhood information: , After processing with a single-layer GAT, the average of the features from the four nodes is finally taken to obtain the global sEMG features at time step (t), i.e., the electromyography characterization. The entire GAT module outputs the feature sequences at all time steps. Furthermore, LayerNorm is applied to the feature sequences to stabilize the training process and improve the model's generalization ability.
[0066] (3) Feature fusion and BiLSTM sequence modeling The two branches of features are concatenated and their dimensionality reduced to obtain the fused features: , in, As a feature of fusion, For the output of the TCN module t IMU timing characteristics at any given time For the output of the GAT module t sEMG spatial features at different times t For time steps, and These are the weight matrix and bias, respectively, for learning.
[0067] The fused features are input into a two-layer bidirectional LSTM (128 hidden units): , , , in, Indicates forward LSTM The hidden state at time t is used to capture the temporal dependency between the past and the present. Indicates backward LSTM The hidden state at time t is used to capture the temporal dependency between the present and the future. This represents the hidden state at time t-1. This is to connect the hidden states of the forward and backward directions.
[0068] To enhance focus on key time steps, a temporal attention mechanism is introduced: , , Obtain the context feature vector c : , in, for t The attention score at each time step is used to measure the contribution of that time step to the final prediction. , , These are learnable parameters for the attention mechanism; These are the attention weights after Softmax normalization.
[0069] (4) Output layer context feature vector c The data is fed into a two-layer MLP and the predicted joint angles are output.
[0070] , in, , , , The parameters are learnable for a two-layer MLP, with ReLU as the activation function, and the output is the predicted joint angle in the normalized space. .
[0071] S6, Training using a composite loss function based on the correlation between multiple subject groups: The training sample set is input into the network, and the parameters are updated using the Adam optimization algorithm. The mean squared error between the predicted and actual values is calculated as the base loss. Simultaneously, based on the subject identifiers in the samples, the samples are grouped according to the subjects. For each group, the correlation coefficient between the predicted and actual sequences is calculated, and a model is constructed. The correlation loss is calculated by weighting the base loss and the correlation loss according to predetermined weights and summing them as the total loss, which is then used for backpropagation and parameter updates. During training, an adaptive learning rate adjustment and early stopping strategy are combined to select the model parameters with the minimum loss on the validation set as the final model.
[0072] The selection and optimization of model parameters are among the most important factors affecting the model. These parameters can be divided into architecture parameters and training parameters. For architecture parameters, the number of LSTM layers, the number of LSTM units, and the kernel size are considered. For training parameters, the learning rate, training cycle, activation function, and dropout are considered. Specific model parameters are shown in Table 1.
[0073] Table 1
[0074] Per-subject affine calibration based on the validation set: The final model is used to generate standardized predicted values on the validation sample set, and these values are then inversely transformed into physical angles using the standardized parameters for the corresponding subjects. For each subject, their predicted angle sequence and true angle sequence are collected on the validation set, and a least-squares affine transformation model is established to solve for the parameters. and During the testing phase, the prediction results for each sample are destandardized, and then the corresponding samples are selected based on their subject identifiers. and Perform affine calibration and output the final predicted angle.
[0075] Test results: Taking the knee joint angle as an example, the test results are as follows: Figure 4 , Figure 5 As shown. On a dataset containing 10 patients with cerebral palsy, the predicted knee angle for the test segment was obtained as follows: Original network output: R 2 =0.8738, RMSE= 6.7082°, MAE= 5.3262° Affine calibration applied only: R 2 = 0.8646, RMSE= 6.9486°, MAE= 5.5570° After applying the sample-level selection of this invention: R 2 = 0.8997, RMSE= 5.9883°, MAE= 4.5533° The proportion of those who chose the original prediction was 53.71%.
[0076] Compared to existing literature reports on cross-subject levels, the accuracy is significantly improved, and the calibration process only takes a few seconds, showing clear potential for clinical translation.
[0077] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A method for predicting continuous movement in patients with cerebral palsy, characterized in that, The method includes: We acquired sEMG data, IMU data, and ground truth values of joint angles from the subjects and constructed a dataset. The dataset was then divided into training, validation, and test sets for each subject in chronological order. The collected EMG and IMU data from the subjects were standardized to obtain standardized sequence data for each subject. The standardized sequence data of each subject are input into a trained prediction network, which includes a parallel TCN module and GAT module, a multimodal fusion module, a BiLSTM network, and a regression output module. The TCN module extracts IMU temporal features, and the GAT module extracts sEMG spatial features. The two types of features are fused by the multimodal fusion module and then input into the BiLSTM network for bidirectional temporal modeling to capture the bidirectional temporal dependence of actions. The standardized prediction angle is then output through the regression output module.
2. The method for predicting continuous movement in patients with cerebral palsy according to claim 1, characterized in that, The TCN module includes three stacked dilated convolutional blocks, with dilation rates of 1, 2, and 4 for each layer. Each dilated convolutional block contains two layers of residual dilated convolution, a ReLU activation function, regularization, and pruning operations.
3. The method for predicting continuous movement in patients with cerebral palsy according to claim 2, characterized in that, The processing flow of the TCN module includes: transposing the IMU normalized sequence into an input format adapted to one-dimensional convolution, extracting temporal features through three stacked dilated convolutional blocks, and then transposing it back to an IMU hidden feature sequence with the same input time length, i.e., IMU temporal features.
4. The method for predicting continuous movement in patients with cerebral palsy according to claim 1, characterized in that, The GAT module treats each channel signal of sEMG as a fully connected dynamic graph structure, where channels are regarded as nodes in the graph structure and the co-activation between channels is regarded as edges.
5. The method for predicting continuous movement in patients with cerebral palsy according to claim 4, characterized in that, The processing flow of the GAT module includes: expanding the sEMG normalized sequence from a vector dimension to a "matrix" input form that adapts to linear projection; Linear projection is performed on each channel at each time step, mapping it to a preset hidden dimension; Calculate the attention scores between nodes, and obtain the attention weight matrix by Softmax normalization; The sEMG global feature sequence, i.e. sEMG spatial features, is obtained by weighting and aggregating the features of neighboring nodes based on the attention weight matrix and averaging the features of each node at the same time step.
6. The method for predicting continuous movement in patients with cerebral palsy according to claim 1, characterized in that, The method further includes performing windowed sample construction and subject balanced sampling steps before training the prediction network: constructing "input sequence-regression label" sample pairs in each set using a sliding window, wherein the input sequence is sEMG and IMU standardized sequence data within the window, and the regression label is the true value of the joint angle at the end of the window; During the training phase, each sample is assigned a sampling weight that is inversely proportional to the number of training samples of the subject to which it belongs, and training samples are extracted based on the sampling weight.
7. The method for predicting continuous motion in patients with cerebral palsy according to claim 1, characterized in that, The prediction model is trained using a composite loss function based on the correlation between multiple subject groups. This composite loss function includes a basic term and a regularization term, where the basic term is the prediction output. Compared to the actual joint angle The mean squared error between them; the regularization term is a correlation penalty term based on subject grouping.
8. The method for predicting continuous movement in patients with cerebral palsy according to claim 1, characterized in that, The method also denormalizes the standardized prediction angle output by the prediction model to obtain an uncalibrated prediction angle. Based on the subject to which the current sample belongs, select the corresponding affine transformation parameters from the calibration parameter table of the corresponding subject, perform an affine transformation on the uncalibrated prediction angle, and obtain the calibrated prediction angle.
9. A method for predicting continuous movement in patients with cerebral palsy according to claim 8, characterized in that, The process of obtaining the affine transformation parameters for each subject includes: For each subject, the standardized predicted angle sequence and the standardized true angle sequence output by the prediction model on the validation set are collected. Assuming a standardized real perspective From a standardized prediction perspective Approximate representation by affine transformation: ， The least squares method was used to estimate the radiotransformation parameters corresponding to subject s, including personalized scaling factors. and offset coefficient ; The affine parameters for each subject are stored as a calibration parameter table.
10. A method for predicting continuous movement in patients with cerebral palsy according to claim 1, characterized in that, The sEMG data were collected from the tibialis anterior, medial gastrocnemius, rectus femoris, and semitendinosus muscles on the affected side of the subject.