Exoskeleton robot or active power artificial limb joint angle prediction device and method

Abstract

The invention discloses an exoskeleton robot or active power artificial limb joint angle prediction device and method. According to the method, electromyographic signals carrying neural information and inertial motion unit data carrying kinematics information are fused, information transmitted by spinal nerve and speed information in multi-modal data are utilized, and feature fusion and decoding are carried out; finally, the output is used for controlling intelligent rehabilitation equipment such as a lower limb exoskeleton robot and an active artificial limb. According to the method, the speed adaptability of the existing lower limb motion angle prediction based on the human body surface electromyogram signals is remarkably improved, and the engineering practical performance of the method is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of human gait prediction and intention recognition, specifically to a device and method for predicting joint angles of exoskeleton robots or actively powered prostheses based on the fusion data of surface electromyography signals and inertial sensors. Background Technology

[0002] In recent years, vulnerable groups with lower limb disabilities and mobility impairments have attracted widespread attention from all sectors of society. Intelligent rehabilitation assistive devices such as lower limb exoskeletons and intelligent active prostheses can help them reproduce or partially restore their gait. The system design of these devices mainly includes three areas: mechanical structure, sensor structure layout, and control algorithms. The control algorithm can be broadly divided into upper-level control and lower-level control. The main purpose of upper-level control is to achieve human-computer interaction (HMI), that is, to perceive human intentions through sensor data and generate control commands. The lower-level control is the motor closed-loop control module, responsible for converting the commands output by the upper-level control into specific motor actions. A robust intention recognition algorithm is key to improving the rehabilitation effect of this type of intelligent rehabilitation device.

[0003] Three main intention recognition strategies have been applied in experiments with these intelligent rehabilitation devices: image-based intention recognition, surface electromyography (sEMG)-based intention recognition, and inertial measurement unit (IMU)-based intention recognition. Image-based strategies require exposed cameras within the wearable device, which are easily obstructed and affect aesthetics. sEMG-based intention recognition offers good real-time performance and robustness, but the signal is easily disturbed. Inertial measurement unit (IMU)-based control strategies are common posture sensing solutions in practical applications. This strategy offers signal stability, but it exhibits a lag compared to human movement intentions.

[0004] Research on lower limb intention recognition technology based on surface electromyography signals is still in its early stages. Most existing studies and experiments only focus on the performance of subjects walking at a certain speed, lacking adaptability to speed changes that occur during walking. Predictive models trained on a single speed show a sharp decline in performance when speed conditions change, and such speed changes are happening all the time. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a device and method for predicting joint angles in exoskeleton robots or actively powered prostheses based on multimodal sensing signals. By fusing electromyographic signals carrying neural information and inertial motion unit data carrying kinematic information, and utilizing spinal cord nerve transmission and velocity information from the multimodal data, and performing feature fusion and decoding, the speed adaptability of the lower limb intention recognition algorithm can be improved. The final output can then be used to control intelligent rehabilitation devices such as lower limb exoskeleton robots and actively powered prostheses. This invention combines the advantages of both types of information, improving the speed adaptability of the intention recognition algorithm while expanding the data sources to supplement the acquisition of speed-related features.

[0006] To achieve the above objectives, the technical solution adopted by this invention is as follows: a joint dataset is constructed using real-time acquired sEMG and IMU data. Data sets are collected and trained under different scenarios for different usage environments and different rehabilitation devices. The datasets are preprocessed and used to train various types of deep learning networks. The trained networks are deployed on the control base station of the exoskeleton robot or active prosthesis. The sEMG and IMU signals transmitted during the use of these devices are received in real time. After passing through the network, the angles of the lower limb joints in the future are predicted and output in a very short time. After receiving these angle information, the control base station drives the motor to move to the designated position, thereby assisting the user in movement or neurorehabilitation.

[0007] On the one hand, the present invention proposes a joint angle prediction device for exoskeleton robots or actively powered prostheses, which is characterized by including a signal acquisition unit, a data preprocessing unit, a gait phase analysis unit, a training dataset construction module, and a multimodal neural network model.

[0008] The signal acquisition unit is used to simultaneously acquire surface electromyography (EMG) signals, inertial measurement signals, and true joint angle signals of human limb movement. The data preprocessing unit, connected to the signal acquisition unit, is used to filter the EMG signals, perform attitude calculation on the inertial measurement signals, and synchronize and align the time and frequency of all signals. The gait phase analysis unit, connected to the data preprocessing unit, is used to identify gait cycles based on the foot movement trajectory in the true joint angle signals and normalize the time within the cycle to gait phase. The training dataset construction module is used to segment and organize the signals processed by the data preprocessing unit and the gait phase analysis unit through a sliding window to generate a training dataset. The multimodal neural network model is used to predict gait phase and joint angles based on the input multimodal signals.

[0009] Furthermore, the multimodal neural network model includes: a parallel feature encoding module, a feature fusion and attention module, and a multi-task prediction output module.

[0010] The parallel feature encoding module is used to extract high-order temporal features from each modal input signal respectively; the feature fusion and attention module, connected to the parallel feature encoding module, is used to fuse multimodal features and generate a global context feature vector through a self-attention mechanism; the multi-task prediction output module, connected to the feature fusion and attention module, is used to regress gait phase values ​​based on the global context feature vector, and generate lower limb joint angle prediction values ​​based on the global context feature vector and the predicted gait phase values.

[0011] Furthermore, the signal acquisition unit includes: multiple surface electromyography (SEMG) sensors, multiple inertial measurement units (IMUs), and an optical motion capture system. The SEMG sensors are configured to be attached to the muscle belly of the target muscle group of the limb; the IMUs are configured to be worn on various segments of the limb to acquire triaxial acceleration and triaxial angular velocity; and the optical motion capture system provides true values ​​of joint angles and marker point position data for gait event recognition.

[0012] Furthermore, the parallel feature encoding module includes several parallel encoder branches, each of which sequentially includes: a residual convolutional layer for extracting local temporal features; a bidirectional long short-term memory network layer for capturing forward and backward long-term dependencies of the sequence; and a layer normalization layer.

[0013] Furthermore, the joint angle prediction module concatenates the scalar phase value output by the gait phase prediction module with the global context feature vector to form a phase-aware feature vector, and performs joint angle regression based on this vector.

[0014] Secondly, the present invention also proposes an exoskeleton robot or active power prosthetic system, characterized in that it includes the aforementioned joint angle prediction device, the output of which is connected to the robot's joint actuator controller to provide joint angle setting values.

[0015] Thirdly, the present invention also proposes a method for predicting joint angles of exoskeleton robots or actively powered prostheses, characterized by comprising the following steps: Data acquisition steps: Simultaneously acquire surface electromyography signals, inertial measurement signals, and true joint angle signals obtained by the optical motion capture system during human limb movement; Data preprocessing steps: Filter the acquired surface electromyography signals, perform attitude calculation on the inertial measurement signals to obtain attitude angles, and align the sampling frequencies of all signals; Gait phase calculation steps: Based on the heel trajectory in the true value signal of the joint angle, detect continuous gait cycles and map the time points in each cycle to 0% to 100% of the gait phase; Training data construction steps: The preprocessed and phase-labeled continuous signal is segmented using a sliding window to construct samples. The input features of each sample include the multimodal signal within the window, and the output label includes the true value of the joint angle and the gait phase at the end of the window. Model training steps: Construct and train a multimodal neural network model to learn the mapping relationship from the input features to the output labels; The model processes the signals of each modality through parallel encoding paths, fuses features through an attention mechanism and generates a global vector, first predicts the gait phase, and then combines the prediction results of the phase to output the joint angles. Real-time prediction and control steps: Using the trained multimodal neural network model, the joint angle is predicted online based on the real-time acquired multimodal signals, and the prediction results are sent to the exoskeleton robot controller.

[0016] Furthermore, the attitude calculation employs a complementary filtering algorithm, fusing acceleration and angular velocity information to estimate the three-dimensional attitude angles of the inertial measurement unit.

[0017] Furthermore, in the training data construction step, the step size of the sliding window is smaller than the window length to ensure the continuity of training and inference data over time.

[0018] Furthermore, the input data in the constructed dataset is arranged in chronological order within a selected time window, which conforms to real physiological phenomena and conditions.

[0019] Furthermore, in the model training step, the multimodal neural network model includes: an encoding subnetwork, each subnetwork including a residual convolutional layer and a bidirectional long short-term memory network layer; a multi-head self-attention layer for fusing and weighting the output features of the parallel encoding subnetwork; a gait phase prediction subnetwork for performing phase regression based on the output of the multi-head self-attention layer; and a joint angle prediction subnetwork whose input is the concatenation of the output of the multi-head self-attention layer and the output of the gait phase prediction subnetwork.

[0020] Furthermore, the multimodal data collected in the data acquisition step includes sEMG signals from the muscles (one or more of the following: rectus femoris, vastus lateralis, vastus medialis, biceps brachii, semitendinosus, tibialis anterior, gastrocnemius, and soleus) that drive the legs, hip joints, knee joints, and ankle joints of normal individuals during walking, as well as IMU data (including one or more of the following: acceleration data, angular velocity data, and posture angle data) attached to the surface of these muscles or at the corresponding locations indicated by the rigid exoskeleton.

[0021] Furthermore, the multimodal data collected in the data acquisition step includes sEMG signals from the muscles (one or more of the following: rectus femoris, vastus lateralis, vastus medialis, biceps brachii, semitendinosus, tibialis anterior, gastrocnemius, soleus) of the healthy leg and hip, knee, and ankle joints of a unilateral stroke patient during walking, as well as IMU data (one or more of the following: acceleration data, angular velocity data, posture angle data, etc.) attached to the surface of the relevant muscles of the healthy or affected leg or the corresponding position of the rigid exoskeleton.

[0022] Furthermore, the multimodal data collected in the data acquisition step includes sEMG signals from the residual muscles of patients with above-knee amputations and the muscles related to hip joint drive (one or more of the rectus femoris, vastus lateralis, and vastus medialis) during walking, as well as IMU data (including one or more of acceleration data, angular velocity data, and posture angle data) attached to the residual muscles or the surface of the prosthesis.

[0023] Furthermore, the multimodal data collected in the data acquisition step includes sEMG signals from the residual muscles of patients with below-knee amputations and the muscles (one or more of the rectus femoris, vastus lateralis, and vastus medialis) that drive the hip and knee joints during walking, as well as IMU data (including one or more of acceleration data, angular velocity data, and posture angle data) attached to the residual muscles or the surface of the prosthesis.

[0024] Furthermore, the multimodal data collected in the data acquisition step includes sEMG signals from normal individuals, patients with unilateral or bilateral lower limb stroke, patients with above-knee or below-knee amputations, and muscles related to the swing drive of the shoulder and elbow joints (one or more of the deltoid, trapezius, triceps brachii, biceps brachii, brachioradialis, pronator teres, etc.) during walking, as well as IMU data attached to the surface of the relevant upper limb muscles (including one or more of acceleration data, angular velocity data, and posture angle data).

[0025] Furthermore, the multimodal data collected in the data acquisition step includes a large amount of virtual joint angle data, virtual electromyography data, and virtual IMU data generated based on real data and generative models such as diffusion models and variational autoencoders (VAEs), which are used to train the humanoid robot's lower limb bionic walking strategy.

[0026] Furthermore, the multimodal data collected in the data acquisition step includes fine-tuning of the open-source large model based on the aforementioned data sources, used to train the humanoid robot base large model.

[0027] Furthermore, the joint angle prediction results can be used to drive the motors of a lower limb assistive exoskeleton worn by normal people for position control, so that the joint angle moves to a specified position and provides walking assistance.

[0028] Furthermore, the joint angle prediction results can be used to drive the lower limb rehabilitation exoskeleton motors worn by unilateral and bilateral stroke patients for position control, so that the joint angle can be moved to a specified position, helping patients to perform compliance training or antagonism training, and helping patients with neural remodeling.

[0029] Furthermore, the joint angle prediction results can be used to drive the position control of above-knee amputees wearing active knee prostheses, enabling the knee joint angle to move to a specified position for daily walking and activities.

[0030] Furthermore, the joint angle prediction results can be used to drive the position control of patients with below-knee amputations wearing active ankle prostheses, enabling the ankle joint angle to move to a specified position for daily walking and activities.

[0031] Furthermore, the joint angle prediction results can be used to drive the biomimetic walking of humanoid robots, and the walking strategy of humanoid robots can be trained by collecting data and training models.

[0032] Furthermore, the network model includes existing mainstream deep learning models (CNN, LSTM, GNN, Transformer) and new models composed of their combinations.

[0033] In one embodiment, the surface electromyography signal originates from the muscle belly of the muscles driving the hip, ankle, and knee joints of a normal person's legs. The inertial measurement signal originates from the inertial sensing unit attached to the muscle group driving the hip, ankle, and knee joints of a normal person's legs, specifically including acceleration information and three-dimensional posture angle information. The network is a multi-task cascaded network (CMTM) based on the Attention coding mechanism. The output data is the real-time joint angles of the legs and hip, ankle, and knee joints used to drive the lower limb exoskeleton.

[0034] In one embodiment, the surface electromyography (EMG) signal originates from the muscle belly of the muscles involved in driving the hip, ankle, and knee joints in the healthy leg of a patient simulating a unilateral stroke. The inertial measurement signal originates from the inertial sensing unit attached to the muscle groups involved in driving the hip, ankle, and knee joints in both legs of the healthy leg of a patient simulating a unilateral stroke. Specifically, it includes acceleration information and three-dimensional posture angle information. The network is a multi-task cascaded network (CMTM) based on the Attention coding mechanism. The output data is the real-time joint angles of both legs with the hip, ankle, and knee joints, used to drive the lower limb exoskeleton.

[0035] In one embodiment, the surface electromyography (EMG) signal originates from the muscle belly of the thigh of the unaffected limb of a simulated below-knee amputee patient, which is related to the driving of the hip and knee joints. The multi-channel IMU signal originates from the inertial sensing unit attached to the muscle groups related to the driving of the hip and knee joints of the unaffected leg of a simulated below-knee amputee patient. The signal also originates from the inertial sensor unit placed on the lower leg of the subject at the ankle prosthesis of a simulated below-knee amputee patient. The signal includes acceleration information and three-dimensional posture angle information. The network is a multi-task cascaded network (CMTM) based on the Attention coding mechanism. The output data is the real-time joint angle prediction of the amputee leg knee joint used to drive the active prosthesis.

[0036] In one embodiment, the surface electromyography signal originates from the muscle belly of the muscles involved in the swinging motion of the upper limbs and shoulder and elbow joints of a normal person, and the inertial measurement signal originates from the inertial sensing unit attached to the muscle groups involved in the swinging motion of the legs and hip, ankle, and knee joints of a normal person. Specifically, it includes acceleration information and three-dimensional posture angle information. The network is a multi-task cascaded network (CMTM) based on the attention coding mechanism, and the output data is the real-time joint angles of the legs and hip, ankle, and knee joints used to drive the lower limb exoskeleton.

[0037] Fourthly, the present invention also proposes a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the above-described method.

[0038] Compared with the prior art in this field, the advantages of the present invention are as follows: (1) It can achieve stepless adjustment of intention recognition prediction when walking speed changes, thereby greatly enhancing the speed adaptability of sEMG signal for gait recognition; (2) By using IMU data as a complement, the network's ability to capture the real-time state of the human body is enhanced. sEMG, as an electrical signal that directly reflects the neural commands of the human body, can capture human intentions the fastest. This prediction model has extremely strong robustness. (3) The multimodal fusion method proposed in this invention has excellent performance in the training of different network models; (4) The network model showed stable performance at different test speeds, which was significantly better than the model that only used single modality data. Attached Figure Description

[0039] Figure 1 It is an experimental paradigm diagram (including data collected from the variable speed walking experiment, data processing flow, and the CMTM model proposed in this invention). Figure 2 This is a diagram showing the placement of the surface electromyography (EMG) sensor and the IMU sensor. Figure 3 It is the process of data frequency alignment and dataset construction; Figure 4 The gait cycles captured within 5 seconds and the electromyographic data captured in each cycle are presented. Figure 5 This is a comparison chart of the angles of the three joints of the right leg output and the actual captured values ​​in the control embodiment of the exoskeleton worn by a normal person (Example 1) of the present invention; Figure 6 This is a comparison chart of the angles of the three joints of the right leg output and the actual captured values ​​in the control embodiment (Example 2) of the present invention for wearing an exoskeleton on a unilateral stroke patient; Figure 7 This is a comparison chart of the angles of the three joints of the knee joint output and the actual captured values ​​in the control embodiment (Example 3) of the present invention for patients with knee lower limb amputation; Figure 8 The table compares the performance of the model used in Example 1 with the baseline model in the multi-joint angle prediction task; Figure 9 The performance results of Example 1 at different speeds were compared; Figure 10 A schematic diagram is shown illustrating the prediction of lower limb joint angles during walking using sEMG and IMU data from both the upper and lower limbs. Detailed Implementation

[0040] To make the objectives, features, and advantages of this invention clearer and easier to understand, the specific embodiments of this invention are described in detail below with reference to the accompanying drawings. The following description represents only some, not all, embodiments of this invention in common usage scenarios. Other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are all within the scope of protection of this invention.

[0041] This invention proposes a joint angle prediction device for exoskeleton robots or actively powered prostheses, comprising: a signal acquisition unit, a data preprocessing unit, a gait phase analysis unit, a training dataset construction module, and a multimodal neural network model.

[0042] 1) The signal acquisition unit is used to implement the data acquisition steps: synchronously acquire multimodal data, including surface electromyography signals of human limb movement, inertial measurement signals and true value signals of joint angles.

[0043] Given that human movement is a complex and coordinated process involving muscles throughout the body, and that multiple muscles are involved in the movement, there is a complex mapping inertia between sEMG signals, IMU information, and joint angles during walking. Therefore, the sources of training data and real-time control input data are diverse. Theoretically, electromyographic information of any muscle involved in walking, inertial sensor data, and inertial sensor data from the body surface of the exoskeleton and the surface of the active prosthesis can all serve as information sources to participate in model training and joint angle prediction.

[0044] In practical applications, a surface electromyography (sEMG) sensor is used to collect multi-channel sEMG signals, and an IMU module is used to collect acceleration and angular acceleration data at specific locations. The timestamps of these data are synchronized. sEMG signals are a direct product of the nervous system and can provide the most direct information about human intention. Research has shown that sEMG signals can be detected within a time range of 30-150 ms before the corresponding movement occurs, providing a valuable time window for predicting the user's intention. Simultaneously, motion-assisted or antagonistic training based on user sEMG signals helps in the neural reconstruction and rehabilitation process of patients with nerve injuries. IMU signals can reflect the real-time motion state of the human body or rehabilitation equipment and have been widely used in fields such as smartphone navigation and commercial robots. This invention combines the two to effectively overcome the problems of poor speed adaptability in control strategies relying solely on sEMG and low user participation in control algorithms relying solely on IMU.

[0045] 2) The data preprocessing unit is connected to the signal acquisition unit and is used to implement the data preprocessing steps: filtering the surface electromyography signal, calculating the attitude of the inertial measurement signal, and synchronizing and aligning the time and frequency of all signals.

[0046] 3) The gait phase analysis unit is connected to the data preprocessing unit and is used to implement the gait phase calculation step: based on the foot movement trajectory in the true value signal of the joint angle, the gait cycle is identified and the time within the cycle is normalized to the gait phase.

[0047] 4) The training dataset construction module is used to implement the training data construction and model training steps: It uses a sliding window to segment the preprocessed and phase-labeled continuous signal to construct samples. Each sample's input features include the multimodal signal within the window, and the output label includes the true value of the joint angles and the gait phase at the end of the window. When constructing the dataset, a sliding window method is chosen to segment the data. According to subsequent task requirements, the size of the sliding window can be used to adjust the "field of view" for each prediction, i.e., each time a time window before the current time point can be used. Each time, the window slides backward by an appropriate length. Finally, based on the previous sliding window size and the distance moved each time, the equivalent acquisition frequency of the joint angles in the dataset is calculated, i.e., how many milliseconds are needed to output a joint angle.

[0048] A multimodal neural network model is constructed and trained to learn the mapping relationship from the input features to the output labels. The model processes the signals of each modality through parallel encoding paths, fuses features through an attention mechanism to generate a global vector, first predicts the gait phase, and then combines the prediction results to output the joint angles. During model training and testing, after obtaining the dataset, the training and test sets are divided into training and test sets according to an appropriate ratio. During training, the data order is shuffled, and in each training iteration, multimodal data from a period prior to a specific time point is used to predict the joint angles at the current time point. During testing, predictions are performed in chronological order to check whether the prediction results are consistent with normal human gait.

[0049] Feature encoding methods include, but are not limited to, classic attention mechanism encoding, long short-term memory network sequence encoding, traditional recurrent neural network encoding, and attention mechanism encoding with rotation position encoding.

[0050] 5) The multimodal neural network model is used to realize the real-time prediction and control steps: using the trained multimodal neural network model, the joint angle is predicted online based on the real-time collected multimodal signals, and the prediction result is sent to the exoskeleton robot controller.

[0051] With the development of deep learning technology and the improvement of computing power of modern graphics processing units, experiments have verified that traditional network model structures and their combinations can achieve excellent performance in the task of predicting lower limb joint angles under specific network parameters. For example, convolutional neural network (CNN) models, long short-term memory networks (LSTM), attention-based models, graph neural networks (GNN), and networks composed of them can all perform well after training on appropriate data.

[0052] Example 1, referring to Figure 1The experimental paradigm described in this paper provides an embodiment of the invention in a scenario where a normal person uses a powered exoskeleton. This embodiment uses surface electromyography signals from the muscles driving the knee, hip, and ankle joints of both legs, as well as inertial sensor signals placed in the thigh and calf, as model inputs. The output is the joint angle information of the three joints received by the exoskeleton robot in this scenario, and the speed adaptability of the model is evaluated. The specific implementation process includes: (1) Data Acquisition: In this embodiment, the raw data used includes surface electromyography (sEMG) data of the driving muscles of the knee, hip, and ankle joints of both legs acquired by sEMG sensors, and acceleration and angular acceleration data collected by the IMU acquisition unit. The attachment positions of the above two types of sensors will be determined by... Figure 2 The data shows that a total of 16 sensors were used. The names of the muscles to which the sensors were attached are also marked in the diagram. When attaching the sensors, care should be taken to place them on the muscle belly to ensure signal strength. In addition, there is motion capture data of the knee, hip, and ankle joints of both legs captured during movement using an infrared camera-based motion capture system, as well as the positional data of model points on the skeletal model built during movement. The model's marker points were placed according to the requirements of the Plug-In Gait model. All of the above data acquisition was time-synchronized using a synchronization device, and performed on the same timeline.

[0053] (2) Data preprocessing: After obtaining the raw data, it is necessary to preprocess the data. For sEMG signals, low-pass filtering is performed using a second-order Butterworth filter to retain only signals below 500Hz, which is the frequency band of effective information of electromyography signals. The filtering formula is as follows: (1) (2) Where Y(t) represents the filtered sEMG signal, X(t) is the original acquired sEMG signal, H(t) is the time domain representation of the Butterworth filter, and H(s) in equation (2) is the s-domain expression of a second-order Butterworth filter with a cutoff frequency of 500Hz.

[0054] (2-1) Using acceleration and angular acceleration data, the attitude angles of the IMU unit are calculated using a complementary filtering algorithm. The real-time pitch, yaw, and roll angles of the sensor in the three-dimensional coordinate system are obtained. The formulas for complementary filtering calculations are as follows: (3) (4) (5) in, θ represents the roll angle, θ represents the pitch angle, and ψ represents the yaw angle. The symbol α represents the complementary filter coefficient, which is set to 0.98 in this experiment.

[0055] (2.2) Due to limitations of the acquisition equipment, the initial signal acquisition frequencies are inconsistent. After preliminary processing, all data acquisition frequencies are aligned to 1000Hz through upsampling and downsampling to facilitate dataset construction in step (3). The frequency alignment process is as follows: Figure 3 It is given in the left part.

[0056] (2.3) Using the position of the right heel in the Plug-In Gait skeletal model, the gait phase between the two heel contact points is divided from 0% to 100%. The specific division is based on the following: the first detection of the right heel touching the ground is the start of a gait cycle, recorded as 0%. Then, the cycle ends with the left foot swinging, the right toes leaving the ground, the right foot swinging again, and finally the right heel touching the ground again. The intermediate times are divided into percentiles between 0% and 100%. The division process is as follows: Figure 4 The information is provided in the text.

[0057] (3) Dataset Construction: After data cleaning and frequency alignment, the next step is to construct a usable dataset for training the network model. For ease of use and comparative experiments, an H5 format data file is constructed because this type of data file adopts a directory structure similar to a file system to manage data and allows efficient access to subsets of data.

[0058] (3.1) Data Segmentation: A sliding window with a window length of 20 data points is used to segment the already aligned data, moving 10 data points forward each time. After segmentation, the dataset X has a sample size of 20. 112, where 20 represents the time steps contained in each sample. 112 represents the number of sample features, which consists of a 16-channel sEMG, 16 3 acceleration data and 16 Three pose angles. The dataset Y, used as the supervision label, has a dimension of 1. 19, containing the content of 3 at the last time point in sample X. Information on six joint angles and one gait phase.

[0059] (3.2) Dataset structure: The hierarchical organization of the final H5 data structure file is as follows: The top-level group represents the batch of the experiment, and each subject is tested 4 times. Each experiment is set as an independent batch; the sub-group represents the experimental speed. Each batch is set with 5 different walking speeds, and the data under each walking speed is packaged into a sub-group.

[0060] (4) Model structure and training: (4.1) Parallel coding layer: each input signal ,in C is the sequence length. in The input channel number is [number]. High-dimensional temporal features are extracted using a separate, improved encoder module. The improved encoder modules are described below: (4.1.1) Residual Convolution Module: This module aims to extract local temporal features. The input sequence first passes through a processing unit containing one-dimensional convolution (Conv1d), batch normalization (BatchNorm1d), and the LeakyReLU activation function. The LeakyReLU activation function is defined as follows, used to introduce nonlinearity and alleviate the problem of neuron inactivation: (6) Subsequently, its output undergoes a second similar convolution-normalization unit and is then residually concatenated with the original input. The residual concatenation is calculated as follows: (7) in, This represents the output of two convolutional units. This structure effectively enhances gradient propagation and avoids the vanishing gradient problem during deep network training.

[0061] (4.1.2) Bidirectional Long Short-Term Memory (Bi-LSTM) Network: The output of the residual convolutional module described above is fed into a bidirectional LSTM layer. The LSTM unit learns long-term dependencies in the sequence data through gating mechanisms (input gate, forget gate, output gate). The core computation of an LSTM unit at time step t can be summarized as follows:

[0062] The bidirectional structure processes the sequence from both the forward and backward directions, and stores the hidden states in both directions. By piecing together the information, a more comprehensive contextual information can be obtained.

[0063] (4.1.3) Layer Normalization: To stabilize the training process, layer normalization is applied after the LSTM output. The normalization calculation formula is as follows: (14) (4.2) Feature Vector Encoding: First, the high-dimensional feature sequences output by each encoder are concatenated along the time dimension to form a unified multimodal feature sequence. Then, the fused sequence is input into a multi-head attention encoding network, the calculation formula of which is: (15) Q (query), K (key), and V (value) are all obtained from the input sequence through a linear transformation. It is the dimension of the key vector. This mechanism extracts fused features by calculating the correlation weights between different time steps within the sequence, thereby capturing the intrinsic connections between various sensor modalities.

[0064] Global feature vector generation: The sequence refined by the self-attention module is subjected to average pooling in the time dimension, compressing the sequence information into a global feature vector of fixed dimension. .

[0065] (4.3) Shared Feature Learning and Phase Prediction: The global feature vector is used for learning and prediction. The input is fed into a shared prediction head to learn general high-order features for subsequent multi-task applications. These shared features are further processed by a specialized pacing prediction head, which ultimately outputs a normalized pacing percentage using a Sigmoid activation function. The Sigmoid function maps any real value to the (0, 1) interval, and its formula is: (16) (4.4) Cascaded joint angle prediction: The gait prediction results from (4.3) are concatenated with the shared features to form a "gait-aware" feature vector. This vector is fed into the final angle prediction head, which outputs the angle prediction values ​​for multiple joints. This is the second output of the model.

[0066] In this example, by introducing modules such as residual connections, gated recurrent units, and self-attention mechanisms, and adopting a cascaded multi-task learning architecture, the CMTM model can efficiently fuse multi-source time-series signals and improve the accuracy and robustness of overall gait analysis by leveraging the correlation between tasks.

[0067] In this embodiment, four healthy male subjects were selected, aged 25±2 years, with a height of 175±5cm and a weight of 70±5kg. All subjects signed informed consent forms and clearly understood the experimental procedures.

[0068] During the experiment, see Figure 9 Participants were required to perform variable-speed walking on a treadmill. The treadmill speed was set to five different walking speeds: 1 km / h, 1.5 km / h, 2 km / h, 4 km / h, and 6 km / h. Data was collected for 90 seconds at each speed, and the data was recorded as a set after collecting data at all five speeds. Each participant underwent four sets of the experiment, with a two-minute rest period between sets to reduce the impact of muscle fatigue.

[0069] Throughout the experiment, see Figure 10Three types of data were acquired simultaneously: surface electromyography (sEMG) signals, sampled at a frequency of 2000 Hz; inertial measurement unit (IMU) signals, including acceleration and angular velocity, sampled at a frequency of 148 Hz; and three-dimensional motion capture data, sampled at a frequency of 100 Hz. All data streams were synchronized with a trigger signal to ensure precise timestamp alignment.

[0070] Considering that the effective part of the sEMG signal is concentrated in the frequency range of 20-450Hz, we first perform low-pass filtering on the original acquired signal, using a second-order Butterworth filter with a cutoff frequency of 500Hz. This also removes noise and motion artifacts generated during acquisition.

[0071] Based on the data collected by the IMU, the three-dimensional attitude angle of each sensor is calculated using a formula. Then, the acceleration data, attitude angle data, and sEMG data frequencies are aligned to 1000Hz. Finally, an H5 dataset is constructed for each subject using a sliding window approach.

[0072] Model training: The preprocessed data samples are input into the CMTM model described in this invention for training.

[0073] Baseline Model Comparison: To verify the superiority of the method of this invention, three commonly used models in existing technologies (including CNN-LSTM model, pure CNN model and Attention mechanism-based model) were selected as baselines and trained using the same dataset.

[0074] Performance evaluation: The root mean square error (RMSE) and coefficient of determination (R²) were used as evaluation indicators to compare the prediction accuracy of the model of this invention with three baseline models under variable speed walking conditions for the angles of various joints of the lower limbs (hip joint, knee joint, ankle joint). The average performance of our proposed CMTM model and the speed adaptability of the prediction results at different speeds were evaluated.

[0075] Results Analysis: Experimental results show that, see [link to results]. Figure 8 The CMTM model proposed in this invention has a root mean square error (RMSE) of 2.5479 in the joint angle prediction task at all test speeds, which is significantly lower than the three baseline models (corresponding results are 2.7557, 3.5017, and 3.0055, respectively), and a coefficient of determination (R²) of 0.9028, which is significantly higher than the three baseline models (0.9012, 0.8287, and 0.8692, respectively). This proves that the method of this invention has higher prediction accuracy and stronger speed adaptation capability under variable speed walking conditions.

[0076] Example 2, refer to Figure 1The experimental paradigm described in this invention provides an embodiment for the use of a powered exoskeleton in a unilateral stroke patient. This embodiment uses surface electromyography (EMG) signals from the muscles driving the unilateral knee, hip, and ankle joints, as well as inertial sensor signals placed in the thigh and calf, as model inputs. Unlike Embodiment 1, this embodiment only uses sEMG signals and IMU data from the normal side of the unilateral stroke patient, outputting the joint angle information of the three joints of both legs received by the exoskeleton robot in this scenario. The specific implementation process includes: The data acquisition and corresponding parameter calculation process, as well as the dataset partitioning process, are similar to those in Example 1, and will not be repeated here. The main difference is that we only use data from the right leg as valid input data to simulate the normal side of a unilateral stroke patient. We use unilateral data to control the real-time joint angles of the three joints of the lower limb exoskeleton. The specific experimental process is as follows: In this real-time example, the subject was a 25-year-old adult male. We placed sEMG and IMU sensors on the driving muscles of his right hip, knee, and ankle joints, as well as his thigh and calf. The subject was asked to walk at a relatively slow speed (1 km / h, 1.5 km / h) to reproduce the slow gait of stroke patients. During the walking process, the subject's joint angles and gait cycles were collected.

[0077] Multiple sets of experimental data were collected. After each experiment, the subjects rested for one minute to eliminate muscle fatigue. The collected signals were made into a dataset in the manner described in Example 1, and the dataset was divided into a training set of 80% and a test set of 20%.

[0078] The deep learning model proposed in Example 1 is trained using the training set.

[0079] Test the model output using the test set, compare the error between the actual value and the predicted value, and obtain... Figure 6 In this embodiment, the root mean square error is 3.0480. Example 3, referring to Figure 1 The experimental paradigm described in this invention provides an embodiment in the scenario of using an ankle prosthesis for a patient with a knee-to-hip amputation. This embodiment uses surface electromyography (EMG) signals from the driving muscles of the unilateral knee and hip joints, as well as inertial sensor signals placed at the thigh and ankle prostheses, as model inputs. Unlike embodiments 1 and 2, this embodiment only uses the sEMG signal and IMU data from the residual limb on the amputation side of the unilateral knee-to-hip amputation patient, as well as the IMU data from the prosthesis, to output the joint angle information of the three joints of both legs received by the exoskeleton robot in this scenario. The specific implementation process includes: The data acquisition and corresponding parameter calculation process, as well as the dataset partitioning process, are similar to those in Example 1, and will not be repeated here. The main difference is that we only use data from the amputee's residual limb and IMU data from the prosthesis as valid input data to simulate the control scenario of a below-knee amputee using an ankle prosthesis. We use residual limb data to control the real-time joint angle of the ankle joint of the output ankle prosthesis, thereby controlling the motor output of the powered prosthesis. The specific experimental process is as follows: The subject in this real-time example was a 25-year-old adult male. We set his right leg as a below-knee amputation leg and placed sEMG and IMU sensors at his right hip joint, knee joint driving muscles (only rectus femoris, vastus lateralis, vastus medialis, biceps brachii, and semitendinosus), and thigh. An IMU sensor was placed at the location simulating the installation of his ankle prosthesis (lower leg). The subject was asked to walk at five different speeds (1km / h, 1.5km / h, 2km / h, 4km / h, and 6km / h). During the walking process, the subject's joint angles and gait cycles were collected.

[0080] Multiple sets of experimental data were collected. After each experiment, the subjects rested for one minute to eliminate muscle fatigue. The collected signals were made into a dataset in the manner described in Example 1, and the dataset was divided into a training set of 80% and a test set of 20%.

[0081] The deep learning model proposed in Example 1 is trained using the training set.

[0082] Test the model output using the test set, compare the error between the actual value and the predicted value, and obtain... Figure 7 In this embodiment, the root mean square error is 3.5560. Example 4, refer to Figure 1 The experimental paradigm described in this invention provides an embodiment of the scenario in which normal subjects use an upper and lower limb information-powered exoskeleton. This embodiment uses surface electromyography signals from the muscles driving the bilateral shoulder and elbow joints, as well as inertial sensor signals placed in the thigh and calf, as model inputs, and outputs the joint angle information of the three joints of the two legs received by the exoskeleton robot in this scenario.

[0083] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related system fields, are similarly included within the scope of protection of the present invention.

Claims

1. A joint angle prediction device for an exoskeleton robot or actively powered prosthesis, characterized in that, include: The signal acquisition unit is used to simultaneously acquire surface electromyography signals, inertial measurement signals, and true joint angle signals of human limb movement; A data preprocessing unit, connected to the signal acquisition unit, is used to filter the surface electromyography signal, perform attitude calculation on the inertial measurement signal, and synchronize and align the time and frequency of all signals. The gait phase analysis unit, connected to the data preprocessing unit, is used to identify the gait cycle based on the foot movement trajectory in the true value signal of the joint angle and normalize the time within the cycle into the gait phase. The training dataset construction module is used to segment and organize the signals processed by the data preprocessing unit and the gait phase analysis unit through a sliding window to generate a training dataset. A multimodal neural network model is used to predict gait phase and joint angles based on input multimodal signals.

2. The apparatus according to claim 1, characterized in that, The multimodal neural network model includes: A parallel feature encoding module is used to extract high-order temporal features from each modal input signal; The feature fusion and attention module, connected to the parallel feature encoding module, is used to fuse multimodal features and generate a global context feature vector through a self-attention mechanism; The multi-task prediction output module is connected to the feature fusion and attention module and is used to regress the gait phase value based on the global context feature vector; and generate the angle prediction value of the lower limb joints based on the global context feature vector and the predicted gait phase value.

3. The apparatus according to claim 1, characterized in that, The signal acquisition unit includes: multiple surface electromyography sensors configured to be attached to the muscle belly of the target muscle group of the limb; Multiple inertial measurement units are configured to be worn on various segments of the limbs to collect triaxial acceleration and triaxial angular velocity. An optical motion capture system is used to provide true values ​​of joint angles and marker position data for gait event recognition.

4. The apparatus according to claim 1, characterized in that, The parallel feature encoding module comprises several parallel encoder branches, each encoder branch containing, in sequence: Residual convolutional layers are used to extract local temporal features; Bidirectional long short-term memory network layers are used to capture forward and backward long-term dependencies in sequences; Layer normalization layer.

5. The apparatus according to claim 1, characterized in that, The joint angle prediction module concatenates the scalar phase value output by the gait phase prediction module with the global context feature vector to form a phase-aware feature vector, and performs joint angle regression based on this vector.

6. An exoskeleton robot or actively powered prosthetic system, characterized in that, The device includes a joint angle prediction device as described in any one of claims 1 to 5, wherein the output of the device is connected to a joint actuator controller of a robot or an actively powered prosthesis for providing joint angle setting values.

7. A method for predicting joint angles in an exoskeleton robot or actively powered prosthesis, characterized in that, Includes the following steps: Data acquisition steps: Simultaneously acquire surface electromyography signals, inertial measurement signals, and true joint angle signals obtained by the optical motion capture system during human limb movement; Data preprocessing steps: Filter the acquired surface electromyography signals, perform attitude calculation on the inertial measurement signals to obtain attitude angles, and align the sampling frequencies of all signals; Gait phase calculation steps: Based on the heel trajectory in the true value signal of the joint angle, detect continuous gait cycles and map the time points in each cycle to 0% to 100% of the gait phase; Training data construction steps: The preprocessed and phase-labeled continuous signal is segmented using a sliding window to construct samples. The input features of each sample include the multimodal signal within the window, and the output label includes the true value of the joint angle and the gait phase at the end of the window. Model training steps: Construct and train a multimodal neural network model to learn the mapping relationship from the input features to the output labels; The model processes the signals of each modality through parallel encoding paths, fuses features through an attention mechanism and generates a global vector, first predicts the gait phase, and then combines the prediction results of the phase to output the joint angles. Real-time prediction and control steps: Using the trained multimodal neural network model, the joint angle is predicted online based on the real-time acquired multimodal signals, and the prediction results are sent to the exoskeleton robot controller.

8. The method according to claim 7, characterized in that, In the training data construction step, the sliding window selects an appropriate length and movement length to ensure that the constructed dataset can conform to the time-varying characteristics of gait during human walking and that the time delay of exoskeleton movements is within an acceptable range.

9. The method according to claim 8, characterized in that, The input data in the constructed dataset is arranged in chronological order within a selected time window, which conforms to real physiological phenomena and conditions.

10. The method according to claim 7, characterized in that, In the model training step, the multimodal neural network model includes: The encoding network, each sub-network containing residual convolutional layers and bidirectional long short-term memory network layers; The feature vector fusion network is used to fuse and weight the output features of the encoding subnetwork to generate a fusion vector; A gait phase prediction subnetwork performs phase regression based on the output of the fusion vector; A joint angle prediction subnetwork is provided, whose input is the concatenation of the output of the fused feature vector and the output of the gait phase prediction subnetwork.

11. The prediction method according to claim 7, characterized in that: The surface electromyography (EMG) signals and inertial measurement signals collected during human limb movement in the data acquisition step are as follows: The surface electromyography (EMG) signals and inertial sensor measurement signals of the lower limbs, including the surface EMG signals of the corresponding driving muscles of the hip, knee and ankle joints on both sides, and the inertial sensor signals of the thigh, calf and foot, and the instep, are used for lower limb angle prediction in normal people and lower limb joint angle prediction in unilateral and bilateral stroke patients. Alternatively, surface electromyography (EMG) signals and inertial sensor measurements of the lower limbs, including surface EMG signals of the driving muscles corresponding to the hip, knee, and ankle joints on one side, and inertial sensor signals of the thigh, calf, foot, and instep, can be used for angle prediction in normal individuals and lower limb joint angle prediction in patients with unilateral stroke. Alternatively, surface electromyography signals of the corresponding driving muscles of the hip and knee joints on one or both sides, as well as inertial sensing signals of the thigh and calf or the ankle prosthesis worn by the subject, can be used to predict the ankle angle of below-knee amputees. Alternatively, surface electromyography signals of the driving muscles corresponding to the unilateral or bilateral hip joints, as well as inertial sensing signals from the thigh and the worn assistive prosthesis, can be used to predict knee angles in patients with above-knee amputations. Alternatively, surface electromyography (EMG) signals of the lower limbs and measurement signals of lower limb inertial sensors, including surface EMG signals of the corresponding driving muscles of the hip, knee and ankle joints on both sides, and arm swing inertial sensor signals of one or both sides of the upper limb, are used for lower limb angle prediction in normal individuals and lower limb joint angle prediction in unilateral and bilateral stroke patients. Alternatively, surface electromyography (EMG) signals of the lower limbs and measurement signals from lower limb inertial sensors, including surface EMG signals of the corresponding driving muscles of the hip, knee and ankle joints on one side, and arm swing inertial sensor signals of one side or both sides of the upper limb, can be used for angle prediction in normal individuals and lower limb joint angle prediction in patients with unilateral stroke. Alternatively, the surface electromyography signals of the driving muscles corresponding to the unilateral hip and knee joints of the subject, and the arm swing inertial sensor signals of one of the unilateral and bilateral upper limbs, can be used to predict the ankle angle of the subjects who have undergone below-knee amputation. Alternatively, surface electromyography signals of the hip joint corresponding driving muscles, and arm swing inertial sensor signals of one or both sides of the upper limb, can be used to predict knee angles in patients with above-knee amputations. Alternatively, surface electromyography signals from the main muscle groups involved in arm swinging movements in one or both upper limbs, as well as inertial sensor signals from the thigh, calf, sole, and instep, can be used for angle prediction in normal individuals and for lower limb joint angle prediction in unilateral or bilateral stroke patients. Alternatively, surface electromyography (EMG) signals from the main muscle groups involved in arm swinging movements on one or both upper limbs, EMG signals from the corresponding driving muscles of the hip joints on both sides, and inertial sensor signals from the prosthesis worn by the subject can be used to predict knee angles in patients with above-knee amputations. Alternatively, surface electromyography (EMG) signals from the main muscle groups involved in arm swinging movements on one or both sides of the upper limb, EMG signal groups from the corresponding driving muscles of the hip and knee joints on both sides, and inertial sensor signals from the subject's thigh and the prosthesis are used to predict ankle angles in patients with below-knee amputations. Alternatively, a large amount of virtual joint angle data, virtual electromyography data, and virtual IMU data generated by a diffusion model and a variational autoencoder generative model can be trained based on real data to train a humanoid robot's lower limb bionic walking strategy. Alternatively, the open-source large model can be fine-tuned based on the above data sources to train the large model of the humanoid robot base.

12. The method according to claim 7, wherein the output joint angle information can be used to provide multi-joint angle position control information for a powered lower limb exoskeleton used by normal people and patients with unilateral or bilateral stroke, so that the assistive device moves to the target position according to the user's intention, thereby achieving the effect of assistive or neurorehabilitation. Alternatively, the joint angle information it outputs can be used to generate joint angle position control information for powered lower limb prostheses used by patients with above-knee or below-knee amputations, so that the assistive device can move to the target position according to the user's intention, thereby achieving the effect of controlling walking. or, Its output joint angle information can be used to generate joint angle position control information for powered lower limb prostheses used by patients with above-knee and below-knee amputations, so that the assistive device can move to the target position according to the user's intention, thereby achieving the effect of controlling walking; or, Using the large amount of simulation data generated by the generative model, the joint angle information output by the model trained with the simulation data can be used to train the biomimetic walking control strategy of the humanoid robot, including the control strategy in the virtual environment of the computer and the control strategy in the real physical world.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 7 to 10.

Images