Lower extremity multi-joint angle prediction method, system, device and medium

Abstract

The application provides a lower limb multi-joint angle prediction method, system, device and medium, the method comprises the following steps: collecting multi-modal data of a user, and preprocessing the multi-modal data to obtain a data set; the data set is divided into a training set and a test set, and the training set is embedded to obtain an embedded training set; using the embedded training set, a preset structure double tower model is trained to obtain a lower limb multi-joint angle prediction model; using the test set, the prediction result of the lower limb multi-joint angle prediction model is evaluated to obtain an evaluation result. The application provides a lower limb multi-joint angle prediction method based on a double tower Transformer and multi-modal data to solve the problem that the current lower limb joint angle prediction technology based on multi-modal data ignores muscle and joint coordination, and builds space-time features in the double tower Transformer architecture to improve the prediction performance of the model.

Description

Technical Field

[0001] This invention relates to the field of gait planning technology, and in particular to a method, system, device and medium for predicting the angles of multiple joints in the lower limbs. Background Technology

[0002] Joint angle prediction technology is one of the key technologies for realizing gait planning and active assistance of lower limb exoskeleton robots. It aims to identify and predict human movement intentions in advance of human movement, realize the synchronous movement of the lower limb exoskeleton robot and the wearer, and thus complete gait planning.

[0003] Currently, existing research on joint angle prediction typically applies machine learning techniques to this task. A common approach is to establish a mapping from data sources such as sEMG signals and historical joint angles to future joint angles, thereby achieving continuous joint angle prediction. However, common machine learning techniques are based on manual feature engineering, heavily reliant on expert domain knowledge and lacking generalization ability. With the improvement of computing power, the development of deep learning technology has injected new vitality into joint angle prediction. Neural networks can adaptively extract features and complete prediction tasks, effectively improving prediction accuracy and model adaptability. However, the fundamental anatomical theory of free lower limb bones reveals that human lower limb movement is accomplished by the coordinated action of multiple muscle groups and joints. Existing research lacks modeling of the spatial relationships between muscles and joints, affecting the performance of prediction models.

[0004] Based on the above analysis, given that current lower limb joint angle prediction technologies based on multimodal data neglect the collaborative relationship between muscles and joints, how to provide a method for predicting lower limb multi-joint angles and improve the predictive performance of the model is an urgent technical problem to be solved. Summary of the Invention

[0005] In view of the above problems, the present invention is proposed to provide a method, system, device and medium for predicting the angles of multiple joints of the lower limbs to overcome or at least partially solve the above problems.

[0006] To achieve the above and other related objectives, the present invention provides a method for predicting multiple joint angles of the lower limb, the method comprising:

[0007] Collect multimodal data from users and preprocess the multimodal data to obtain a dataset; wherein the multimodal data includes surface electromyography signal data and lower limb angle sequences;

[0008] The dataset is divided into a training set and a test set, and the training set is embedded to obtain an embedded training set.

[0009] Using the embedded training set, a pre-defined dual-tower model is trained to obtain a lower limb multi-joint angle prediction model.

[0010] The prediction results of the lower limb multi-joint angle prediction model are evaluated using the test set to obtain the evaluation results.

[0011] In one embodiment of the present invention, the collection of user multimodal data includes:

[0012] Multiple surface electromyography (SEMG) sensors are used to collect N-channel SEMG signal data of multiple muscles when a user performs M consecutive lower limb movements, and four inertial measurement units are used to collect the user's lower limb angle sequence; wherein, the multimodal data includes N-channel SEMG signal data and four-channel lower limb angle sequence, M≥1, N≥1.

[0013] In one embodiment of the present invention, the preprocessing of the multimodal data to obtain a dataset includes:

[0014] The N-channel surface electromyography (EMG) signal data are sequentially processed by Butterworth high-pass and low-pass filtering, Butterworth band-stop filtering, discrete wavelet thresholding, and normalization to obtain an N-channel surface EMG signal sequence.

[0015] The 4-channel lower limb angle sequence is preprocessed and solved to obtain a 4-channel joint angle sequence;

[0016] The N-channel electromyography signal sequence and the 4-channel joint angle sequence are clock-aligned and then superimposed in the channel dimension to obtain an N+4 channel dataset.

[0017] The N+4 channel dataset is segmented using a preset sliding window to generate a new dataset.

[0018] In one embodiment of the present invention, the embedding process of the training set to obtain the embedded training set includes:

[0019] By keeping the time dimension priority unchanged in the training set, a time dimension priority dataset is obtained;

[0020] The training set is transposed to obtain a spatial dimension-first dataset;

[0021] Embedding processing is performed on the time-first dataset and the space-first dataset respectively, and the embedded time-first dataset is positionally encoded to obtain the time-embedded dataset and the space-embedded dataset.

[0022] In one embodiment of the present invention, training a preset structured dual-tower model using the embedded training set to obtain a lower limb multi-joint angle prediction model includes:

[0023] The time-embedded dataset is sequentially processed and trained through a multi-head attention layer with masking, a CNN layer, a residual and normalization layer, a feedforward connection layer, and a residual and normalization layer to output a feature vector in the time dimension;

[0024] The spatial embedding dataset is processed and trained sequentially through a multi-head attention layer, a CNN layer, a residual and normalization layer, a feedforward connection layer, and a residual and normalization layer to output a feature vector in the channel dimension.

[0025] The feature vectors in the time dimension and the feature vectors in the channel dimension are aggregated based on the aggregation layer to obtain the merged feature vector.

[0026] The merged feature vector is processed by a fully connected layer to obtain the prediction result. Training stops when a preset number of iterations is reached, and the lower limb multi-joint angle prediction model is output.

[0027] In one embodiment of the present invention, the step of training the temporal embedding dataset sequentially through a masked multi-head attention layer, a CNN layer, a residual and normalization layer, a feedforward connection layer, and a residual and normalization layer to output a temporal dimension feature vector includes:

[0028] The temporal embedding dataset is processed through a masked multi-head attention layer to obtain vectors with long-distance sequence dependencies;

[0029] The vectors with long-distance sequence dependencies are captured by convolution through a CNN layer to capture the local features of the sequence;

[0030] The temporal embedding dataset is residually concatenated with the local features and then standardized to output the feature vectors of the residual and the standardized layer.

[0031] The feature vectors of the residual and the normalized layer are processed through the feedforward connection layer to obtain the feature vector of the feedforward connection layer.

[0032] The feature vectors of the residual and normalized layers are residually concatenated with the feature vectors of the feedforward connection layer and then normalized to output the feature vectors in the time dimension.

[0033] In one embodiment of the present invention, the step of evaluating the prediction results of the lower limb multi-joint angle prediction model using the test set to obtain evaluation results includes:

[0034] The test set is input into the lower limb multi-joint angle prediction model, and the test values ​​are output.

[0035] Obtain the true value corresponding to the test value, and use the test value and the true value to calculate the root mean square error, mean absolute error, coefficient of determination, and explained variance.

[0036] To achieve the above and other related objectives, the present invention provides a lower limb multi-joint angle prediction system, comprising:

[0037] The acquisition module is used to collect multimodal data from users and preprocess the multimodal data to obtain a dataset;

[0038] An embedding module is used to divide the dataset into a training set and a test set, and to perform embedding processing on the training set to obtain an embedded training set.

[0039] The training module is used to train the preset structured double-tower model using the embedded training set to obtain a lower limb multi-joint angle prediction model.

[0040] The evaluation module is used to evaluate the prediction results of the lower limb multi-joint angle prediction model using the test set, and obtain the evaluation results.

[0041] To achieve the above and other related objectives, the present invention provides an electronic device, the electronic device comprising: a memory and a processor;

[0042] The memory is used to store computer programs;

[0043] The processor is used to execute the computer program stored in the memory to enable the electronic device to perform the lower limb multi-joint angle prediction method.

[0044] To achieve the above and other related objectives, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the lower limb multi-joint angle prediction method as described above.

[0045] As described above, the lower limb multi-joint angle prediction method, system, device, and medium of the present invention include: collecting multimodal data from users and preprocessing the multimodal data to obtain a dataset; wherein the multimodal data includes surface electromyography signal data and lower limb angle sequences; dividing the dataset into a training set and a test set, and embedding the training set to obtain an embedded training set; training a pre-defined dual-tower model using the embedded training set to obtain a lower limb multi-joint angle prediction model; and evaluating the prediction results of the lower limb multi-joint angle prediction model using the test set to obtain an evaluation result. The present invention addresses the problem of current lower limb joint angle prediction technologies based on multimodal data neglecting muscle and joint coordination, and provides a lower limb multi-joint angle prediction method based on a dual-tower Transformer and multimodal data, using a dual-tower Transformer architecture to model spatiotemporal features to improve the model's prediction performance. Attached Figure Description

[0046] Figure 1 The diagram shows a flowchart of a method for predicting the angles of multiple joints of the lower limbs in one embodiment of the present invention.

[0047] Figure 2 The diagram shows a flow chart illustrating the attachment positions of the surface electromyography sensor and the IMU in one embodiment of the present invention.

[0048] Figure 3 The diagram shown is a schematic of a dual-tower Transformer architecture according to an embodiment of the present invention;

[0049] Figure 4 The diagram shown is a schematic diagram of lower limb multi-joint angle prediction in one embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of joint angle sequence calculation (right lower limb) in one embodiment of the present invention;

[0051] Figure 6 This is a schematic diagram showing the evaluation results of the test set in one embodiment of the present invention;

[0052] Figure 7 This is a schematic diagram showing the comparison between the predicted curve and the actual curve in one embodiment of the present invention;

[0053] Figure 8 This is a schematic diagram of the functional modules of a lower limb multi-joint angle prediction system according to an embodiment of the present invention;

[0054] Figure 9 The diagram shown is a schematic block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0055] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0056] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0057] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0058] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0059] Unless otherwise stated, the term "multiple" means two or more.

[0060] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0061] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0062] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0063] <1> Butterworth high-pass filter transfer function Where s is a complex frequency domain variable (a variable in the Laplace transform); wc is the cutoff frequency, which can be defined as... T is the time constant of the filter; n is the order of the filter, which determines the slope of the filter at the cutoff frequency.

[0064] <2> Butterworth low-pass filter transfer function Where s is a complex frequency domain variable; wc is the cutoff frequency; and n is the filter order.

[0065] <3> Butterworth bandstop filter transfer function Where s is a complex frequency domain variable; wc is the cutoff frequency; and n is the filter order.

[0066] <4> Discrete wavelet thresholding denoising consists of two parts: discrete wavelet transform and thresholding denoising. The discrete wavelet transform can be expressed as:

[0067] c j,k =∑ n x[n]·ψ j，k [n]

[0068] d j，k =∑ n x[n]·φ j，k [n]

[0069] Among them, c j,k d represents the approximation coefficient of the j-th layer; j,k The j-th detail coefficient represents the j-th layer; x[n] represents a discrete sample of the original signal; ψ j,k [n] represents the wavelet function used to extract the high-frequency components of the signal; φ j,k [n] represents the scaling function used to extract the low-frequency component of the signal; j represents the number of decomposition levels; k represents the index of the wavelet function; after the signal undergoes discrete wavelet transform, a series of wavelet functions are obtained, and then the wavelet functions are thresholded: this can be done using the following formula:

[0070]

[0071] Where coeff represents the original wavelet coefficients, θ represents the set threshold, and Thr(coeff) is the wavelet coefficients after thresholding. Finally, the processed wavelet coefficients are subjected to inverse wavelet transform to reconstruct the denoised signal.

[0072] The technical solutions of the present invention will now be described in detail with reference to the accompanying drawings.

[0073] Please see Figure 1 An embodiment of the present invention provides a method for predicting multiple joint angles of the lower limb, the method comprising the following steps:

[0074] Step S10: Collect the user's multimodal data and preprocess the multimodal data to obtain a dataset; wherein, the multimodal data includes surface electromyography signal data and lower limb angle sequences.

[0075] The multimodal data may include N-channel surface electromyography (EMG) signal data of eight muscles in the user's lower limb, including the extensor digitorum longus (EDL), medial gastrocnemius (MG), tibialis anterior (AT), peroneus longus (PL), vastus medialis (VM), vastus lateralis (VL), rectus femoris (RF), and biceps femoris (BF), as well as four-channel lower limb angle sequences of the user's lower limb, including the sagittal plane of the thigh and calf, the coronal plane of the thigh, and the dorsum of the foot.

[0076] The extensor digitorum longus (EDL), medial gastrocnemius (MG), tibialis anterior (AT), peroneus longus (PL), vastus medialis (VM), vastus lateralis (VL), rectus femoris (RF), and biceps femoris (BF) are a series of important muscles in the human lower limb. These eight muscles play a key role in various lower limb activities such as walking, running, jumping, and balance, and the function of these eight muscles is very important for sports training, rehabilitation therapy, and understanding lower limb movement disorders.

[0077] The dataset can be obtained by clock-aligning two types of data: preprocessed N-channel surface electromyography signal data and 4-channel lower limb angle sequence data, superimposing them along the channel dimension, and then segmenting them through a preset sliding window.

[0078] In the specific implementation, the user's multimodal data can be collected first, including N-channel surface electromyography (EMG) signal data and 4-channel lower limb angle sequences. Then, the N-channel EMG signal data can be preprocessed by noise reduction, filtering, and normalization to obtain the N-channel EMG signal sequence. The 4-channel lower limb angle sequence can be preprocessed and solved to obtain the 4-channel joint angle sequence. Finally, the N-channel EMG signal sequence and the 4-channel joint angle sequence are clock-aligned and superimposed in the channel dimension, and then segmented through a preset sliding window to obtain the dataset.

[0079] Furthermore, in one embodiment, step S10 may include the following sub-steps:

[0080] Sub-step S101 involves using multiple surface electromyography (SEMG) sensors to collect N-channel SEMG signal data of multiple muscles when the user performs M consecutive lower limb movements, and using four inertial measurement units to collect the user's lower limb angle sequence; wherein the multimodal data includes N-channel SEMG signal data and four-channel lower limb angle sequence, M≥1, N≥1.

[0081] In specific implementation, surface electromyography (SEMG) sensors can be attached to the skin surface of eight muscles in the user's lower limbs, including the extensor digitorum longus, medial gastrocnemius, tibialis anterior, peroneus longus, vastus medialis, vastus lateralis, rectus femoris, and biceps femoris. These SEMG sensors can then be used to collect N-channel SEMG signal data of multiple muscles when the user performs M consecutive lower limb movements. Additionally, inertial measurement units (IMUs) can be placed at four locations in the user's lower limbs: the sagittal plane of the thigh and calf, the coronal plane of the thigh, and the dorsum of the foot. These four IMUs can then be used to collect the user's lower limb angle sequence. The multimodal data includes N-channel SEMG signal data and four-channel lower limb angle sequence, where M≥1 and N≥1.

[0082] As an example, surface electromyography (EMG) sensors can be attached to the skin surface of eight muscles in the user's lower limbs: extensor digitorum longus (EDL), medial gastrocnemius (MG), tibialis anterior (AT), peroneus longus (PL), vastus medialis (VM), vastus lateralis (VL), rectus femoris (RF), and biceps femoris (BF). This allows for the acquisition of eight channels of EMG signal data during continuous movement. Additionally, IMUs can be placed at four locations in the user's lower limbs: the sagittal plane of the thigh and calf, the coronal plane of the thigh, and the dorsum of the foot, to acquire four channels of lower limb angular sequences. Gait data acquisition includes three lower limb movements: walking, jogging, and uphill.

[0083] Please see Figure 2 , Figure 2 This is a schematic diagram of the surface electromyography sensor and IMU attachment location; it can be set to a sampling frequency of 1000Hz during data acquisition, with each action repeated 3 times by 3 subjects (i.e., users), each lasting 60 seconds, with a 5-minute interval to avoid muscle fatigue.

[0084] Step S20: Divide the dataset into a training set and a test set, and perform embedding processing on the training set to obtain the embedded training set.

[0085] The training set that has undergone embedding processing can be obtained by embedding the training set through the embedding processing layer.

[0086] In the specific implementation, after obtaining the dataset, the dataset can be divided into a training set and a test set; then the training set can be input into a pre-structured dual-tower model, and the embedding processing layer of the pre-structured dual-tower model can be used to embed the training set to obtain the embedded training set.

[0087] As an example, the dataset can be divided into a training set and a test set in a 7:3 ratio.

[0088] Furthermore, in one embodiment, step S20 may include the following sub-steps:

[0089] Sub-step S201: Keep the time dimension priority of the training set unchanged to obtain the time dimension priority dataset;

[0090] Sub-step S202: Transpose the training set to obtain a spatial dimension-first dataset;

[0091] Sub-step S203 involves embedding the time-priority dataset and the spatial-priority dataset, respectively, and then performing position encoding on the embedded time-priority dataset to obtain the time-embedded dataset and the spatial-embedded dataset.

[0092] Among them, the time-oriented dataset can be obtained by prioritizing the time dimension while keeping the original training set unchanged.

[0093] A spatial dimension-first dataset is obtained by transposing the original training set samples as a matrix, with channels as the horizontal axis and time as the vertical axis.

[0094] A time-embedded dataset can be obtained by embedding and position encoding a time-dimension-first dataset.

[0095] As an example, a fully connected layer is used as the embedding processing layer to perform embedding processing on the time-oriented dataset; since the time dimension needs to maintain strict temporal order in the prediction task, trigonometric function position encoding is also added to the time-dimensional dataset to obtain the time-embedded dataset.

[0096] The location encoding can be achieved using the following formula:

[0097]

[0098] Where PE represents the encoded position vector of a data point; pos represents the position of the data point in the sequence; d model This indicates the length of the sequence to be encoded.

[0099] As an example, set d model Given a value of 500, the position code of the first data point of a sample in this embodiment can be represented as follows:

[0100]

[0101] Spatial embedding datasets can be obtained by embedding spatial dimension-first datasets.

[0102] In the specific implementation, for the time dimension priority, the original training set remains unchanged, resulting in a time-priority dataset. To reduce time complexity and considering the characteristics of electromyography (EMG) signals, a fully connected layer is used as the embedding processing layer. Because the prediction task requires maintaining strict temporal order, trigonometric function positional encoding is added to the time dimension data to obtain a time embedding dataset. For the spatial dimension priority, the original training set samples are transposed as a matrix, with channels as the horizontal axis and time as the vertical axis. Since the order between channels is irrelevant, positional encoding is unnecessary. The data is then embedded through a fully connected layer to obtain a spatial embedding dataset. Thus, by embedding the training set to obtain the time and spatial embedding datasets, the preparation of training data for the pre-defined dual-tower model is completed.

[0103] Step S30: Using the embedded training set, train the preset structure double tower model to obtain a lower limb multi-joint angle prediction model.

[0104] The preset dual-tower model can be a pre-set dual-tower Transformer model, which may include: a tower for modeling the time dimension based on a Transformer encoder architecture and a tower for modeling the spatial dimension based on a Transformer decoder architecture.

[0105] Please see Figure 3 , Figure 3 This is a schematic diagram of the pre-defined dual-tower model architecture. The tower modeling the time dimension is based on a Transformer encoder architecture. It can pass the time-dimension-priority dataset, after embedding and positional encoding, through a masked multi-head self-attention layer to compute the multi-head attention vector representation and concatenate it into an output vector, capturing long-distance dependencies in the input sequence. Following the attention layer is a CNN layer, which compensates for the Transformer model's strength in capturing global features but neglect of local features. After the CNN layer are residual and normalization layers, a feedforward connection layer, and another residual and normalization layer. Figure 4 The time dimension tower in the dual-tower Transformer architecture is the basic structure in Transformer.

[0106] The tower for modeling spatial dimensions is based on the Transformer decoder architecture. It can take the spatial dimension-priority dataset after embedding through a multi-head self-attention layer, followed by a CNN layer, residual and normalization layer, feedforward connection layer, residual and normalization layer.

[0107] The difference between the spatial dimension tower and the temporal dimension tower is that the multi-head attention layer of the spatial dimension tower does not have a mask.

[0108] Finally, the feature vectors output by the two towers (the time dimension tower and the spatial dimension tower) are aggregated by 1D-CNN to obtain the merged feature vector. Then, the merged feature vector is processed based on the fully connected layer to output the 4-channel joint angle prediction result. Training stops when the preset number of iterations is reached, and the lower limb multi-joint angle prediction model is output.

[0109] The training parameters for the preset dual-tower model can be set as follows:

[0110] The training set to test set ratio is 7:3, the number of training iterations (epochs) is set to 400, the learning rate (lr) is set to 0.0002, the batch size is 5, the hidden layer dimension is 2048, the number of attention heads is 6, the optimizer is Adam, the loss function is cross-entropy MSELoss, 10-fold cross-validation is used for training, and training is accelerated using GPU.

[0111] In the specific implementation, a pre-defined dual-tower model can be constructed first. Then, the pre-defined dual-tower model can be iteratively trained using an embedded training set until a pre-defined number of iterations is reached, at which point training stops and a lower limb multi-joint angle prediction model is output. The process of training the pre-defined dual-tower model can include steps S30-S40: the training set can be embedded using the embedding layer of the pre-defined dual-tower model; then, the pre-defined dual-tower model can be iteratively trained using the embedded training set to obtain the lower limb multi-joint angle prediction model.

[0112] In addition, the loss can be calculated in the training of the pre-set dual-tower model, and training can be stopped when the loss function reaches convergence, outputting a lower limb multi-joint angle prediction model.

[0113] Step S40: Use the test set to evaluate the prediction results of the lower limb multi-joint angle prediction model and obtain the evaluation results.

[0114] The evaluation results can be used as evaluation metrics to assess the joint angle prediction performance of the lower limb joint multi-angle prediction model using a test set. These metrics may include, but are not limited to, root mean square error (RESE), mean absolute error (MAE), and coefficient of determination (R²). 2 Evaluation metrics include explanatory variance (EVS) and other metrics. It is important to note that when selecting evaluation metrics, the specific application scenario and data characteristics of the lower limb joint multi-angle prediction model must also be considered.

[0115] The root mean square error (RESE) and mean absolute error (MAE) can provide the magnitude of the prediction error of the lower limb multi-joint angle prediction model.

[0116] Coefficient of determination (R) 2The data and interpretable variance (EVS) can provide information on the goodness of fit of the lower limb multi-joint angle prediction model.

[0117] In practice, after obtaining the lower limb multi-joint angle prediction model, the prediction results of the lower limb multi-joint angle prediction model can be evaluated using a test set to obtain the evaluation results; then the prediction performance of the lower limb multi-joint angle prediction model can be verified based on the evaluation results.

[0118] As an example, please refer to Figure 3 , Figure 3 This is a schematic diagram of a dual-tower Transformer architecture; from Figure 3 The process steps for predicting lower limb multi-joint angles are as follows: First, multimodal data from users is collected and preprocessed to obtain a dataset. Then, the dataset is divided into a training set and a test set. Next, the training set is input into a pre-defined dual-tower model, and the embedding layer of the pre-defined dual-tower model is used to embed the training set (including dual-tower data, time-dimension data, and spatial-dimension data) to obtain an embedded training set. Then, the pre-defined dual-tower model (dual-tower Transformer model) is trained using the embedded training set to obtain a lower limb multi-joint angle prediction model. Finally, the test set is used to evaluate the results of the lower limb multi-joint angle prediction model to assess its predictive performance.

[0119] In this embodiment, multimodal data from users is collected and preprocessed to obtain a dataset. The dataset is then divided into a training set and a test set, and the training set is embedded to obtain an embedded training set. Using the embedded training set, a pre-defined dual-tower model is trained to obtain a lower limb multi-joint angle prediction model. The prediction results of the lower limb multi-joint angle prediction model are evaluated using the test set to obtain an evaluation result. This invention addresses the problem of current lower limb joint angle prediction technologies based on multimodal data neglecting muscle and joint coordination. It provides a lower limb multi-joint angle prediction method based on a dual-tower Transformer and multimodal data, using a dual-tower Transformer architecture to model spatiotemporal features to improve the model's prediction performance.

[0120] Based on the foregoing embodiments, a second embodiment of the lower limb multi-joint angle prediction method of the present invention is proposed. In this embodiment, step S10 may further include the following sub-steps:

[0121] Sub-step A10 involves sequentially passing the N-channel surface electromyography (EMG) signal data through Butterworth high-pass and low-pass filtering, Butterworth band-stop filtering, discrete wavelet thresholding, and normalization to obtain an N-channel EMG signal sequence.

[0122] Butterworth high-pass and low-pass filters can be used to remove high-frequency noise and low-frequency trends from data while retaining signals at intermediate frequencies (i.e., effective frequencies). In this embodiment, a fourth-order Butterworth high-pass and low-pass filter with a frequency range of 20–450 Hz is preferred.

[0123] Butterworth band-stop filtering can effectively remove interference in specific frequency bands of data, including power frequency interference. In this embodiment, a 50Hz Butterworth band-stop filter is preferred.

[0124] Power frequency interference usually refers to interference caused by the power grid frequency; in China, the frequency of power frequency interference is usually 50Hz or 100Hz.

[0125] Discrete wavelet thresholding can be used to remove noise from data while preserving detailed signal information. In this embodiment, the wavelet function is preferably sym7, and the threshold is preferably 0.05.

[0126] Normalization is used to adjust the numerical range of data to a specific interval, usually [0, 1]. It can accelerate the convergence of learning algorithms, prevent gradient vanishing or exploding during training, and help improve the model's generalization ability. It can be expressed using the formula:

[0127]

[0128] Where X represents the original data, X max and X min These represent the minimum and maximum values ​​in the dataset, respectively.

[0129] In a practical implementation, N-channel surface electromyography (EMG) signal data can be sequentially processed through Butterworth high-pass and low-pass filtering, Butterworth band-stop filtering, discrete wavelet thresholding, and normalization to obtain an N-channel EMG signal sequence.

[0130] Sub-step A20 involves preprocessing and solving the 4-channel lower limb angle sequence to obtain a 4-channel joint angle sequence.

[0131] The lower limb angle sequence transmitted back by the IMU requires preprocessing and computation. Please refer to [link / reference needed]. Figure 5 , Figure 5 This is a schematic diagram of the joint angle sequence calculation for the right lower limb. The IMUs located in the sagittal plane of the thigh, the coronal plane of the thigh, the sagittal plane of the lower leg, and the dorsum of the foot are numbered 1, 2, 3, and 4, respectively. The calculation can be performed using the following formula:

[0132] angle1=IMU1 measuring angle+90°

[0133] angle2=IMU2 measuring angle+90°

[0134] angle3=IMU3 measuring angle+90°

[0135] angle4 = IMU4 measuring angle

[0136] Subsequently, the joint angles of the right lower limb in four channels can be calculated. The angles of hip joint coronal abduction, hip joint sagittal plane, knee joint, and ankle joint are a1, a2, a3, and a4, respectively, which can be calculated using the following formula:

[0137] al = aanglel

[0138] a2 = angle2

[0139] a3 = 180° - angle2 + angle3

[0140] a4 = 90° - angle3 + angle4

[0141] In practice, the 4-channel lower limb angle sequence can be preprocessed and solved to obtain the 4-channel joint angle sequence.

[0142] Sub-step A30 involves aligning the N-channel electromyography signal sequence and the 4-channel joint angle sequence clockwise and then superimposing them along the channel dimension to obtain an N+4 channel dataset.

[0143] In the specific implementation, after obtaining the N-channel electromyography signal sequence and the 4-channel joint angle sequence, the N-channel electromyography signal sequence and the 4-channel joint angle sequence can be clock-aligned and superimposed in the channel dimension to obtain an N+4 channel dataset.

[0144] As an example, an 8-channel electromyography signal sequence and a 4-channel joint angle sequence were obtained. After time-dimensional alignment, they were superimposed in the channel dimension to obtain a 12-channel dataset.

[0145] Sub-step A40: Use a preset sliding window to segment the N+4 channel dataset to generate a dataset.

[0146] The preset sliding window can be a sliding window with a pre-set fixed length L and step size J.

[0147] As an example, a sliding window with a fixed length of 1000 and a step size of 50 can be pre-set to segment samples in an N+4 channel dataset.

[0148] In the specific implementation, after obtaining the N+4 channel dataset, a preset sliding window can be used to segment the N+4 channel dataset to generate a new dataset. The dataset label is a 4×1 dimensional vector with a distance of P sampling points from the last data point of the current window sample, representing the 4-channel joint angle at that point.

[0149] As an example, a 12-channel dataset is segmented using a sliding window of fixed length 1000 and step size 50 to form the dataset. The dataset label is a 4×1 dimensional vector, spaced 10 sampling points from the last data point in the current window, representing the 4-channel joint angle at that point.

[0150] In this embodiment, the N-channel surface electromyography (EMG) signal data is sequentially processed through Butterworth high-pass and low-pass filtering, Butterworth band-stop filtering, discrete wavelet thresholding, and normalization to obtain an N-channel EMG signal sequence. The 4-channel lower limb angle sequence is preprocessed and computed to obtain a 4-channel joint angle sequence. The N-channel EMG signal sequence and the 4-channel joint angle sequence are clock-aligned and superimposed along the channel dimension to obtain an N+4 channel dataset. A preset sliding window is used to segment the N+4 channel dataset to generate a dataset. This preprocessing of multimodal data to obtain the dataset lays the foundation for improving the training efficiency and performance of the preset dual-tower model.

[0151] Based on the foregoing embodiments, a third embodiment of the lower limb multi-joint angle prediction method of the present invention is proposed. In this embodiment, step S30 may include the following sub-steps:

[0152] Sub-step S301: The time embedding dataset is sequentially processed and trained through a multi-head attention layer with mask, a CNN layer, a residual and normalization layer, a feedforward connection layer, and a residual and normalization layer to output a feature vector in the time dimension.

[0153] The training set after embedding can include temporal embedding datasets and spatial embedding datasets.

[0154] Residual and normalization layers are used to alleviate the vanishing and exploding gradient problems that may occur during model training. Normalization accelerates the training process and prevents overfitting by normalizing the output.

[0155] In the specific implementation, for the time-dimensional tower, the time-embedded data can be passed through a multi-head attention layer with a mask to obtain a vector with long-distance sequence dependencies. Then, it passes through a CNN layer to capture the local features of the sequence through convolution. In this process, the input data (i.e., the time-embedded data) skips the attention layer and CNN layer through a residual network, and is residually connected with the feature vector output by the CNN layer and standardized. Then, the feature vector is processed by the feedforward connection layer and residually connected with the vector that skipped the layer and standardized to obtain the output of the tower (i.e., the feature vector in the time dimension).

[0156] Further, in one embodiment, sub-step S301 may include the following sub-step:

[0157] Sub-step B10: Process the temporal embedding dataset through a masked multi-head attention layer to obtain a vector with long-distance sequence dependencies;

[0158] Sub-step B20: The vector with long-distance sequence dependencies is passed through a CNN layer to capture the local features of the sequence via convolution.

[0159] Sub-step B30 involves performing a residual concatenation between the temporal embedding dataset and the local features, followed by standardization, to output the feature vectors of the residual and the standardized layer.

[0160] Sub-step B40: The feature vectors of the residual and the normalized layer are processed through the feedforward connection layer to obtain the feature vectors of the feedforward connection layer.

[0161] Sub-step B50 involves performing residual concatenation and standardization on the feature vectors of the residual and standardized layers and the feature vectors of the feedforward connection layer, outputting a feature vector in the time dimension.

[0162] In the specific implementation, the temporal embedding dataset can be processed through a multi-head attention layer with a mask, the attention vector representation of the multi-head attention layer is calculated and concatenated into an output vector, capturing the long-range dependencies of the input sequence, and obtaining a vector with long-range sequence dependencies. Then, it passes through a CNN layer to capture the local features of the sequence through convolution. In this process, the temporal embedding dataset skips the attention layer and CNN layer through a residual network, performs residual concatenation with the local features output by the CNN layer and normalizes them, outputting the feature vectors of the residual and normalized layers. Then, the feature vectors of the residual and normalized layers are processed through a feedforward connection layer to obtain the feature vector of the feedforward connection layer. The feature vector of the feedforward connection layer and the feature vectors of the residual and normalized layers that skipped this layer are residual concatenated and normalized, and finally the temporal dimension feature vector is output.

[0163] Sub-step S302 involves sequentially processing and training the spatial embedding dataset through a multi-head attention layer, a CNN layer, a residual and normalization layer, a feedforward connection layer, and a residual and normalization layer to output a feature vector in the channel dimension.

[0164] In the specific implementation, the processing steps for the spatial embedding dataset and the temporal embedding dataset are basically the same for the channel-dimensional tower. The difference is that the self-attention calculation for the spatial embedding dataset does not require a mask, and the multi-head attention layer with a mask can be modified into a multi-head attention layer. For details of the temporal embedding dataset processing steps, please refer to sub-steps S301 and B10 to B50 above. Finally, the channel-dimensional feature vector is output for the channel-dimensional tower.

[0165] Sub-step S303: Based on the aggregation layer, the feature vectors of the time dimension and the feature vectors of the channel dimension are aggregated to obtain the merged feature vector.

[0166] The aggregation layer, which can employ a 1D-CNN structure, acts as an "aggregation gate" for both time-dimension and channel-dimension feature vectors. It can be implemented using the following formula:

[0167]

[0168] Where F represents the merged feature vector; R represents the dimension of the feature vector output by the dual towers; and These represent the feature vectors output by the temporal dimension Transformer and the channel dimension Transformer, respectively.

[0169] In practical implementation, after both towers output feature vectors in the time-priority and channel-priority modes, 1D-CNN can be used to aggregate the feature vectors output by the two towers to obtain a merged feature vector.

[0170] Sub-step S304: Process the merged feature vector based on the fully connected layer to obtain the prediction result, and stop training when the preset number of iterations is reached, and output the lower limb multi-joint angle prediction model.

[0171] In the specific implementation, after obtaining the merged feature vector, the merged feature vector is processed through a fully connected layer in a learnable manner to output a 4×1 dimensional vector (i.e., the prediction result of 4-channel joint angles). Training stops when the preset number of iterations is reached, and the lower limb multi-joint angle prediction model is output.

[0172] In this embodiment, the temporal embedding dataset is trained sequentially through a masked multi-head attention layer, a CNN layer, a residual and normalization layer, a feedforward connection layer, and another residual and normalization layer to output a temporal feature vector. The spatial embedding dataset is trained sequentially through the same multi-head attention layer, CNN layer, residual and normalization layer, feedforward connection layer, and another residual and normalization layer to output a channel-dimensional feature vector. An aggregation layer is used to aggregate the temporal and channel-dimensional feature vectors to obtain a merged feature vector. A fully connected layer is then used to process the merged feature vector to obtain a prediction result. Training stops after a preset number of iterations, outputting a lower limb multi-joint angle prediction model. Therefore, by inputting the temporal and spatial embedding datasets into a pre-structured dual-tower model for training, the training efficiency and prediction accuracy of the dual-tower model are improved.

[0173] Based on the foregoing embodiments, a fourth embodiment of the lower limb multi-joint angle prediction method of the present invention is proposed. In this embodiment, step S40 may further include the following sub-steps:

[0174] Sub-step S41: Input the test set into the lower limb multi-joint angle prediction model and output the test values;

[0175] Sub-step S42: Obtain the true value corresponding to the test value, and use the test value and the true value to calculate the root mean square error, mean absolute error, coefficient of determination, and explained variance.

[0176] The root mean square error (RESE) measures the difference between predicted and actual values ​​in a prediction task. It evaluates the accuracy of regression or prediction models, providing a quantifiable error metric that numerically represents the average deviation of the model's predictions. It can be expressed using the formula:

[0177]

[0178] Where n is the number of samples; y i This is the i-th true value; Let be the i-th test value; i = 1, 2, ..., n. The smaller the root mean square error (RESE), the higher the prediction accuracy of the model.

[0179] Mean Absolute Error (MAE) can be used to measure the accuracy of a model in a prediction task. It assesses error by the absolute value of the difference between the predicted and actual values, and is equally sensitive to positive and negative errors. It provides an intuitive, unbiased estimate of model error, making it suitable as a fair comparison standard between models. Its formula is:

[0180]

[0181] Where n is the number of samples; y i This is the i-th true value; Let be the i-th test value; i = 1, 2, ..., n. The smaller the mean absolute error (MAE), the higher the prediction accuracy of the model.

[0182] Coefficient of determination (R) 2 This reflects the proportion of model variability to total variability, and can be used to evaluate the predictive ability of a model in a prediction task. It provides an indicator for assessing the potential accuracy of a model when predicting unknown data; it can be expressed by the formula:

[0183]

[0184] Where n is the number of samples; SS res Sum of squared residuals, i.e., the sum of squared errors between the true and predicted values, SS tot The sum of squares is the sum of the squares of the errors between the true values ​​and the average of the test values. The coefficient of determination (R²) is also included. 2 The value of ) ranges from 0 to 1. The closer the value is to 1, the higher the explanatory power of the model.

[0185] Explained variance (EVS) measures the accuracy of a model in regression or prediction tasks, representing the proportion of total variability that the model can explain. Statistically, variance is commonly used to quantify variation and helps understand the extent to which the model can explain changes in the dependent variable through its independent variables; it can be expressed by the formula:

[0186]

[0187] Where ESS is the explained variance; TSS is the total variance, which can be expressed by the formula:

[0188]

[0189] Where n is the number of samples; y i This is the i-th true value; This is the i-th test value; Let be the mean of all true values; i = 1, 2, ..., n. The explained variance (EVS) ranges from 0 to 1, with values ​​closer to 1 indicating higher predictive accuracy.

[0190] In the specific implementation, after obtaining the lower limb multi-joint angle prediction model, the test set can be input into the lower limb multi-joint angle prediction model, and the corresponding test values ​​can be output. Then, the true values ​​corresponding to the test values ​​can be obtained, and the root mean square error, mean absolute error, coefficient of determination, and explained variance can be calculated using the test values ​​and the true values. Thus, the prediction performance of the lower limb multi-joint angle prediction model can be evaluated using these four indicators: root mean square error, mean absolute error, coefficient of determination, and explained variance.

[0191] As an example, please refer to Figure 6-7 , Figure 6 This is a schematic diagram illustrating the evaluation results of the test set. Figure 7 This is a schematic diagram comparing the predicted curve (test value) and the true curve (true value); from Figure 6 As can be seen, the lower limb multi-joint angle prediction method of this invention can achieve relatively accurate prediction results, with a root mean square error (RESE) of 0.0239, a mean absolute error (MAE) of 0.015, and a coefficient of determination (R²) of [missing information]. 2 The variance was 0.9544, and the explained variance (EVS) was 0.958; and from Figure 7 As can be seen from the comparison chart of the actual curve and the predicted curve, the fitting degree is relatively high; in addition, it can be seen that the fitting degree of the curve of the coronal plane of the hip joint is relatively poor, which needs to be optimized in the future.

[0192] It should be noted that in this embodiment, the root mean square error (RESE), mean absolute error (MAE), and coefficient of determination (R²) are used. 2 The calculation of ) and interpretable variance (EVS) relies on the built-in calculation functions of the sklearn package.

[0193] In this embodiment, the test set is input into the lower limb multi-joint angle prediction model, and test values ​​are output. The true values ​​corresponding to the test values ​​are obtained, and the root mean square error, mean absolute error, coefficient of determination, and explained variance are calculated using the test values ​​and the true values. Thus, the predictive performance of the lower limb multi-joint angle prediction model is evaluated using four indicators: root mean square error, mean absolute error, coefficient of determination, and explained variance. This not only assesses the overall prediction error of the lower limb multi-joint angle prediction model on the test set but also evaluates its potential performance when predicting new data, making the prediction model more practical.

[0194] Based on the same inventive concept, the fifth embodiment of this invention also provides a lower limb multi-joint angle prediction system corresponding to the lower limb multi-joint angle prediction method of the foregoing embodiments. Since the principle of the system in the fifth embodiment of this invention for solving the problem is similar to the lower limb multi-joint angle prediction method of the foregoing embodiments of this invention, the implementation of the system can refer to the implementation of the method, and repeated details will not be elaborated further. Please refer to... Figure 8 The lower limb multi-joint angle prediction system of the present invention may include:

[0195] The acquisition module 10 is used to acquire the user's multimodal data and preprocess the multimodal data to obtain a dataset; wherein, the multimodal data includes surface electromyography signal data and lower limb angle sequences;

[0196] Embedding module 20 is used to divide the dataset into training set and test set, and to perform embedding processing on the training set to obtain the embedded training set;

[0197] Training module 30 is used to train the preset structure double tower model using the embedded training set to obtain a lower limb multi-joint angle prediction model.

[0198] Evaluation module 40 is used to evaluate the prediction results of the lower limb multi-joint angle prediction model using the test set, and obtain evaluation results.

[0199] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for predicting the angles of multiple joints of the lower limbs.

[0200] According to the method provided in the embodiments of the present invention, the present invention also provides a computer program product, which includes: computer program code, which, when the computer program code is run on a computer, causes the computer to implement the above-described method for predicting the angles of multiple joints of the lower limbs.

[0201] According to the method provided in the embodiments of the present invention, the present invention also provides an electronic device, specifically, the electronic device includes at least a memory and a processor connected via a bus, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to perform all or part of the steps in the foregoing method embodiments.

[0202] Please see Figure 9 , Figure 9 This is a schematic block diagram of an electronic device provided in an embodiment of the present invention. Figure 9As shown, the computer device includes at least one processor 401, a memory 402, at least one network interface 403, and a user interface 405. The various components in the device are coupled together via a bus system 404. It is understood that the bus system 404 is used to implement communication between these components. In addition to a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 9 The general will label all buses as bus systems.

[0203] The user interface 405 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0204] It is understood that memory 402 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0205] In this embodiment of the invention, the memory 402 is used to store various types of data to support the operation of the electronic terminal 400. Examples of this data include: any executable program for operation on the electronic terminal 400, such as the operating system 4021 and application programs 4022; the operating system 4021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 4022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The implementation of the XX method provided in this embodiment of the invention may be included in the application program 4022.

[0206] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 401 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0207] In an exemplary embodiment, the electronic terminal 400 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to execute the aforementioned method.

[0208] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0209] In summary, this invention addresses the problem that current lower limb joint angle prediction techniques based on multimodal data neglect the collaborative relationship between muscles and joints. It proposes a lower limb multi-joint angle prediction method based on a dual-tower transformer and multimodal data, using a dual-tower transformer architecture to construct spatiotemporal features to improve the model's prediction performance.

[0210] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for predicting the angles of multiple joints in the lower limb, characterized in that, The method includes: Collect multimodal data from users and preprocess the multimodal data to obtain a dataset; wherein the multimodal data includes surface electromyography signal data and lower limb angle sequences; The dataset is divided into a training set and a test set, and the training set is embedded to obtain an embedded training set, including: keeping the time dimension of the training set unchanged to obtain a time-priority dataset; performing matrix transpose on the training set to obtain a spatial dimension-priority dataset; performing embedding on the time-priority dataset and the spatial dimension-priority dataset respectively, and performing position encoding on the embedded time-priority dataset to obtain a time-embedded dataset and a spatial-embedded dataset. Using the embedded training set, a pre-structured dual-tower model is trained to obtain a lower limb multi-joint angle prediction model. This includes: sequentially training the temporal embedded dataset through a masked multi-head attention layer, a CNN layer, a residual and normalization layer, a feedforward connection layer, and another residual and normalization layer to output a temporal dimension feature vector; sequentially training the spatial embedded dataset through a multi-head attention layer, a CNN layer, a residual and normalization layer, a feedforward connection layer, and another residual and normalization layer to output a channel dimension feature vector; aggregating the temporal and channel dimension feature vectors using an aggregation layer to obtain a merged feature vector; processing the merged feature vector using a fully connected layer to obtain a prediction result; stopping training when a pre-set number of iterations is reached, and outputting the lower limb multi-joint angle prediction model. The process involves sequentially training the temporal embedding dataset through a masked multi-head attention layer, a CNN layer, a residual and normalization layer, a feedforward connection layer, and another residual and normalization layer to output a temporal dimension feature vector. This includes: processing the temporal embedding dataset through a masked multi-head attention layer to obtain a vector with long-range sequence dependencies; passing the vector with long-range sequence dependencies through a CNN layer to capture local features of the sequence via convolution; performing a residual concatenation and normalization between the temporal embedding dataset and the local features to output a feature vector from the residual and normalization layer; processing the feature vector from the residual and normalization layer through a feedforward connection layer to obtain a feature vector from the feedforward connection layer; and performing a residual concatenation and normalization between the feature vector from the residual and normalization layer and the feature vector from the feedforward connection layer to output a temporal dimension feature vector. The prediction results of the lower limb multi-joint angle prediction model are evaluated using the test set to obtain the evaluation results.

2. The method according to claim 1, characterized in that, The collected multimodal data from users includes: Multiple surface electromyography (SEMG) sensors are used to collect N-channel SEMG signal data of multiple muscles when a user performs M consecutive lower limb movements, and four inertial measurement units are used to collect the user's lower limb angle sequence; wherein, the multimodal data includes N-channel SEMG signal data and four-channel lower limb angle sequence, M≥1, N≥1.

3. The method according to claim 2, characterized in that, The preprocessing of the multimodal data to obtain the dataset includes: The N-channel surface electromyography (EMG) signal data are sequentially processed by Butterworth high-pass and low-pass filtering, Butterworth band-stop filtering, discrete wavelet thresholding, and normalization to obtain an N-channel surface EMG signal sequence. The 4-channel lower limb angle sequence is preprocessed and solved to obtain a 4-channel joint angle sequence; The N-channel electromyography signal sequence and the 4-channel joint angle sequence are clock-aligned and then superimposed in the channel dimension to obtain an N+4 channel dataset. The N+4 channel dataset is segmented using a preset sliding window to generate a new dataset.

4. The method according to claim 1, characterized in that, The evaluation of the prediction results of the lower limb multi-joint angle prediction model using the test set, and the resulting evaluation results, include: The test set is input into the lower limb multi-joint angle prediction model, and the test values ​​are output. Obtain the true value corresponding to the test value, and use the test value and the true value to calculate the root mean square error, mean absolute error, coefficient of determination, and explained variance.

5. A lower limb multi-joint angle prediction system, characterized in that, The system includes: The acquisition module is used to acquire multimodal data from users and preprocess the multimodal data to obtain a dataset; wherein, the multimodal data includes surface electromyography signal data and lower limb angle sequences; An embedding module is used to divide the dataset into a training set and a test set, and to perform embedding processing on the training set to obtain an embedded training set. The embedding module is used to keep the training set in a time-priority manner to obtain a time-priority dataset; to perform matrix transpose on the training set to obtain a spatial-priority dataset; to perform embedding processing on the time-priority dataset and the spatial-priority dataset respectively, and to perform position encoding on the embedded time-priority dataset to obtain a time-embedded dataset and a spatial-embedded dataset. The training module is used to train the preset structured double-tower model using the embedded training set to obtain a lower limb multi-joint angle prediction model. The training module is further configured to sequentially process the temporal embedding dataset through a masked multi-head attention layer, a CNN layer, a residual and normalization layer, a feedforward connection layer, and a residual and normalization layer to output a temporal dimension feature vector; sequentially process the spatial embedding dataset through a multi-head attention layer, a CNN layer, a residual and normalization layer, a feedforward connection layer, and a residual and normalization layer to output a channel dimension feature vector; aggregate the temporal dimension feature vector and the channel dimension feature vector based on an aggregation layer to obtain a merged feature vector; process the merged feature vector based on a fully connected layer to obtain a prediction result, until a preset number of iterations is reached, at which point training stops and a lower limb multi-joint angle prediction model is output. The training module is further configured to process the temporal embedding dataset through a multi-head attention layer with a mask to obtain vectors with long-range sequence dependencies; pass the vectors with long-range sequence dependencies through a CNN layer to capture local features of the sequence via convolution; perform residual concatenation and normalization on the temporal embedding dataset and the local features to output feature vectors of the residual and normalized layers; process the feature vectors of the residual and normalized layers through a feedforward connection layer to obtain feature vectors of the feedforward connection layer; and perform residual concatenation and normalization on the feature vectors of the residual and normalized layers and the feature vectors of the feedforward connection layer to output feature vectors of the temporal dimension. The evaluation module is used to evaluate the prediction results of the lower limb multi-joint angle prediction model using the test set, and obtain the evaluation results.

6. An electronic device, characterized in that, The electronic device includes: a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory to cause the electronic device to perform the lower limb multi-joint angle prediction method as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by an electronic device, the program implements the lower limb multi-joint angle prediction method according to any one of claims 1 to 4.

Images