Joint angle estimation method dynamically fusing multi-mode information
By combining surface electromyography (EMG) signals and acceleration signals, a dynamic fusion joint angle estimation model is constructed, which solves the problems of insufficient joint angle estimation accuracy and high computational cost in the existing technology. It achieves high-precision and low-cost joint angle estimation, which conforms to real and natural human hand movements.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIDIAN UNIV
- Filing Date
- 2024-04-16
- Publication Date
- 2026-07-28
AI Technical Summary
Existing methods for estimating joint angles based on surface electromyography (EMG) signals have limitations in feature extraction. They cannot simultaneously acquire local and temporal features of EMG signals, have high computational latency, and do not consider hand kinematics information, resulting in insufficient estimation accuracy and high computational cost, and failing to reflect real and natural human hand movements.
By combining surface electromyography (EMG) signals and acceleration signals, a dynamic fusion joint angle estimation model is constructed. Using convolutional neural networks and visual Transformer networks, combined with hand kinematic constraints, a loss function is constructed, and a gating decision network is used for dynamic fusion, which reduces computational cost and improves estimation accuracy and interpretability.
It significantly improves the accuracy and computational efficiency of joint angle estimation, reduces computational costs, makes the estimated values more consistent with real and natural human hand movements, and improves user experience and task execution reliability.
Smart Images
Figure CN118244901B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of signal processing technology, and further relates to a method for estimating hand joint angles, which can be used for hand motion recognition in human-computer interaction. Background Technology
[0002] In human-computer interaction, hand movements are mainly recognized through visual information or surface electromyography (EMG) signals. The implementation processes of these two methods are quite different. With the development of machine learning and deep learning and the changes in application requirements, more and more research is adopting methods based on surface EMG signals.
[0003] Visual information-based hand gesture recognition technology typically uses a high-precision camera to capture hand posture, combined with a hand skeletal model for assistance. This method is easily affected by external environmental factors such as lighting, obstructions, and space size, and the equipment is cumbersome to carry, making it unsuitable for practical applications.
[0004] Surface electromyography (SEMG) signals contain multi-dimensional physiological information about the muscles involved in movement. Furthermore, SEMG signals are highly sensitive to the physiological state of local muscles, unaffected by subjective psychological factors, and can be collected before movement occurs. The acquisition process also offers advantages such as non-invasiveness, real-time capability, and multi-channel processing. In addition, SEMG signals can be acquired using wearable devices, which are portable and do not place a significant burden on the user. Therefore, SEMG signals are undoubtedly an important tool for upper limb movement recognition and assessment, and have attracted widespread attention in the field of human-computer interaction.
[0005] In the research of electromyographic (EMG) prosthetic hands, existing control methods are mainly based on discrete gesture recognition and joint angle estimation. EMG control based on discrete gesture recognition can only switch between limited movements, has low scalability, and places high demands on the user. EMG control based on joint angle estimation, on the other hand, can predict the angles of joints such as the hand and wrist corresponding to EMG signals in real time, better representing the flexibility and continuity of multi-degree-of-freedom movements of the human hand. Currently, in research on joint angle estimation methods based on surface EMG signals, traditional methods mostly use convolutional neural networks (CNNs) or long short-term memory (LSTM) networks for feature extraction. While CNNs can extract local features of the signal, they cannot capture the temporal correlation of the EMG signal, and LSTM networks cannot perform parallel computation. These methods cannot simultaneously extract both local and temporal features of the EMG signal and suffer from high computational latency. To address these issues, some researchers have proposed other deep learning network models to overcome the shortcomings in feature extraction.
[0006] Patent document CN202310643511.4 discloses "a method for estimating continuous hand movements combining multi-scale convolution and attention mechanisms." Its implementation steps are as follows: First, surface electromyography (EMG) signals and joint angle signals at a joint are recorded using a preset sampling frequency. The collected EMG and joint angle signals of the object to be predicted are preprocessed and divided into training and testing sets to ensure a one-to-one correspondence between the sampling times of the EMG and joint angle signals. The EMG and joint angle signals from the training set are used as input samples to construct a prediction model, which is then trained to obtain a well-trained prediction model. The EMG signals of the object to be predicted are collected in real time and input into the trained prediction model to predict the joint angles. The predicted angle signals are then smoothed in real time to obtain the required joint angle signals, thereby achieving continuous estimation of the human joint angles. While this method can use multi-scale convolution to extract signal features at different scales, enriching the feature information and improving the model's accuracy, its implementation is not ideal. However, this method only focuses on the extraction of local features at multiple scales and does not acquire the temporal features of electromyography signals. Furthermore, it does not take into account the computational cost of the model, which may limit the estimation performance of the model and increase the computation time.
[0007] Patent application CN202210246659.X discloses a "method, device, storage medium, and apparatus for estimating hand joint angles." The implementation steps involve acquiring real-time surface electromyography (EMG) signals generated during hand movements of the subject; inputting the EMG signals into a pre-trained hand joint angle estimation model to obtain the estimated joint angles during hand movements. The hand joint angle estimation model includes sequentially connected multi-scale convolutional modules and multi-layer multi-head attention modules. This method considers the temporal features of the EMG signals and the parallel computation of the model. However, this method still only uses EMG signals and does not consider the limitations of single-modal information, because EMG signals are biological signals, exhibiting user variability and having a weak correlation with joint angles.
[0008] Furthermore, since these methods are all based on data-driven models, they do not take into account the relevant mechanisms of hand kinematics and cannot reflect the inherent kinematic information. Therefore, the process of the algorithm calculating joint angles cannot be explained, and the predicted values of joint angles may not conform to the real and natural human hand movements. Summary of the Invention
[0009] The purpose of this invention is to address the shortcomings of the prior art by proposing a dynamic fusion of multimodal information for continuous estimation of joint angles. This method overcomes the limitations of relying solely on electromyographic signal input, improves the accuracy of joint angle prediction, reduces computational costs, increases the interpretability of the joint angle calculation process, and makes the predicted joint angle values more consistent with real and natural human hand movements.
[0010] To achieve the above objectives, the technical approach of this invention is as follows: by adding acceleration signals to electromyographic signals, the limitations of using only electromyographic signal input are overcome; by using a dynamic fusion strategy, the accuracy of joint angle estimation is improved and the computational cost is reduced; by embedding hand kinematic constraints into the process of calculating joint angles, the interpretability is increased, making the calculated joint angle predictions more consistent with real and natural hand movements.
[0011] Based on the above ideas, the technical solution of the present invention includes the following steps:
[0012] (1) Acquire surface electromyography signals, acceleration signals and hand kinematic data, and divide them into training set and test set;
[0013] (2) Construct a dynamic fusion joint angle estimation model:
[0014] (2a) Select existing convolutional neural networks and visual Transformer networks, and input electromyographic signals and acceleration signals into the convolutional neural network. Input the output electromyographic features and acceleration features into the visual Transformer network in three different ways to form three joint angle estimation network models M1, M2 and M3 with different output data.
[0015] (2b) Based on the existing hand kinematic constraints, construct the loss function Loss of the joint angle estimation network model.
[0016] (2c) Input the training set into the three joint angle estimation network models respectively, and minimize their respective loss functions through backpropagation to obtain the three trained joint angle estimation network models M1. ’ M2 ’ M3 ’ ;
[0017] (2d) Concatenate two existing multilayer perceptron models to form a gated decision network, and connect it to three trained joint angle estimation network models respectively to form a three-way parallel dynamic fusion joint angle estimation model.
[0018] (3) Training the dynamic fusion joint angle estimation model:
[0019] (3a) Input the training set into the three-way parallel dynamic fusion joint angle estimation model, and the output of the gating decision network determines which of the three trained joint angle estimation network models to be executed;
[0020] (3b) Loss of existing resources rThe sum of Loss2 of the joint angle prediction loss and the loss function Loss is used as the loss function of the dynamic fusion joint angle estimation model. ’ ;
[0021] (3c) Freeze M1 ’ M2 ’ M3 ’ The parameters, which remain unchanged during training, are used to minimize the loss function Loss through backpropagation. ’ The gating decision network is trained to obtain a well-trained dynamic fusion joint angle estimation model;
[0022] (4) Input the test set into the trained dynamic fusion joint angle estimation model and output the predicted value of the hand joint angle.
[0023] Compared with the prior art, the present invention has the following advantages:
[0024] Firstly, this invention adds an acceleration signal to the surface electromyography signal, forming multimodal data as the input signal. This reduces the user variability of the input signal and enhances the correlation between the input signal and the joint angle, thus overcoming the limitation of using only surface electromyography signals in the prior art.
[0025] Secondly, this invention constructs a dynamic fusion joint angle estimation model and uses resource loss during training. r This improves the accuracy of joint angle estimation while reducing computational costs, thus solving the problems of insufficient accuracy and low computational speed of existing joint angle estimation models.
[0026] Thirdly, this invention utilizes hand kinematics constraints to construct the loss function Loss of the joint angle estimation model, thereby increasing the interpretability of the joint angle estimation model and making the predicted joint angle values more closely match the real and natural human hand movements. This solves the problem in the prior art that the joint angle estimation method cannot reflect the inherent hand kinematic mechanism and that the estimated joint angle values may deviate from the real situation of human hand movements. Attached Figure Description
[0027] Figure 1 This is a flowchart illustrating the overall implementation of the present invention;
[0028] Figure 2 This is a structural diagram of the dynamic fusion joint angle estimation model constructed in this invention;
[0029] Figure 3 This is a sub-flowchart for training the dynamic fusion joint angle estimation model in this invention. Detailed Implementation
[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.
[0031] Reference Figure 1 The implementation steps for this example are as follows:
[0032] Step 1: Obtain surface electromyography signals, acceleration signals, and hand kinematic data, and divide the data into training and testing sets.
[0033] 1.1) Obtain surface electromyography signals, acceleration signals, and hand kinematics data from publicly available websites to form a dataset;
[0034] 1.2) Preprocess the dataset:
[0035] 1.2.1) Extract active segment data from the dataset and downsample the extracted data to 200 Hz;
[0036] 1.2.2) Noise in the downsampled surface electromyography and acceleration signals was removed using a 5-450 Hz bandpass Butterworth filter and a Gaussian filter, respectively. The downsampled hand kinematics data was smoothed using a 2 Hz fourth-order low-pass Butterworth filter.
[0037] 1.2.3) The surface electromyography signal, acceleration signal and hand kinematic data processed in step 1.2.2) are divided into a unified sliding window with a window size of 200ms and a step size of 100ms. The first data in each window of the hand kinematic data is extracted as the true value of the joint angle of that window.
[0038] 1.3) Divide the preprocessed dataset into training and test sets in a 4:2 ratio.
[0039] Step 2: Construct a dynamic fusion joint angle estimation model.
[0040] Reference Figure 2 The specific implementation of this step includes the following:
[0041] 2.1) Construct a convolutional neural network consisting of two identical convolutional structures connected in parallel. Each convolutional structure includes five convolutional layers, followed by a ReLU activation function and a batch normalization layer. The parameter settings for the five convolutional layers are as follows:
[0042] The first convolutional layer has 64 3×1 convolutional kernels with a stride of 1×1;
[0043] The second convolutional layer consists of 64 3×1 convolutional kernels with a stride of 1×1;
[0044] The third convolutional layer consists of 64 3×1 convolutional kernels with a stride of 1×1;
[0045] The fourth convolutional layer has 128 1×1 convolutional kernels with a stride of 2×1;
[0046] The fifth convolutional layer has 128 2×1 convolutional kernels with a stride of 1×1.
[0047] 2.2) A visual Transformer network consisting of a cascaded encoding and regression part is selected. The regression part consists of a fully connected layer with an output dimension of 10, and the encoding part consists of two identical cascaded encoding blocks. The structure of each encoding block is as follows:
[0048] The first is layer normalization;
[0049] The second is multi-head attention, which consists of 6 self-attention blocks connected in parallel;
[0050] The third is Dropout, with a probability of 0.1;
[0051] The fourth is layer normalization;
[0052] The fifth is the multilayer perceptron;
[0053] The sixth is Dropout, with a probability of 0.1;
[0054] 2.3) Inputting electromyography (EMG) and acceleration signals into a convolutional neural network, the output EMG and acceleration features are input into a visual Transformer network in three different ways, forming three joint angle estimation network models M1, M2, and M3 with different output data, where:
[0055] The first method is to first divide the electromyographic signal feature matrix into patches of size 2×N1, and then add position information encoding to all patches, where N1 is the channel dimension of the electromyographic signal feature.
[0056] The second method is to first divide the acceleration signal feature matrix into patches of size 2×N2, and then add position information encoding to all patches, where N2 is the channel dimension of the acceleration signal feature.
[0057] The third approach is to first concatenate the electromyographic signal features and acceleration signal features along the channel dimension, then divide the resulting feature matrix into patches of size 2×N, and finally add position information encoding to all patches, where N=N1+N2.
[0058] 2.4) Constructing the loss function Loss of the joint angle estimation network model:
[0059] 2.4.1) Based on the kinematic constraints of the hand, select the most representative set of constraints in daily hand activities, namely the constraint relationships between the MCP and PIP joints of the four fingers other than the thumb:
[0060] ;
[0061] Where, θ P θ is the angle of the proximal interphalangeal joint. M The angle of the metacarpophalangeal joint;
[0062] 2.4.2) Based on the constraints of the selected hand's kinematics, construct the hand joint constraint loss Loss1:
[0063] ;
[0064] Where MSE is the mean square error calculation function;
[0065] 2.4.3) Calculate the joint angle prediction loss Loss2:
[0066] ;
[0067] Where x is the estimated value of the joint angle and y is the actual value of the joint angle;
[0068] 2.4.4) The loss function of the joint angle estimation network model is formed by adding the hand joint constraint loss and the joint angle prediction loss together:
[0069] .
[0070] 2.5) Train three joint angle estimation network models M1, M2, and M3:
[0071] 2.5.1) Input the training set into three joint angle estimation network models M1, M2, and M3 respectively for forward propagation to generate model predictions;
[0072] 2.5.2) Calculate the loss function Loss and perform backpropagation to calculate the influence of the network parameters on the loss function Loss, i.e. the gradient of the parameters. Then update the parameters in the network through the gradient descent algorithm so that the value of the loss function Loss gradually decreases.
[0073] 2.5.3) Repeat steps 2.5.1) to 2.5.2) until the loss function converges, obtaining the trained joint angle estimation network model M1. ’ M2 ’ M3 ’ ;
[0074] 2.6) Two existing multilayer perceptron models (MLP) are cascaded to form a gated decision network. This example uses, but is not limited to, two-layer perceptrons, where the size of the first layer multilayer perceptron is 1920×128 and the size of the second layer perceptron is 128×3.
[0075] 2.7) The gating decision network is then connected to three pre-trained joint angle estimation network models M1. ’ M2 ’ M3 ’ The three parallel connections form a dynamic fusion joint angle estimation model.
[0076] Step 3: Train the dynamic fusion joint angle estimation model.
[0077] Reference Figure 3 The specific implementation of this step is as follows:
[0078] 3.1) Input the training set into a three-way parallel dynamic fusion joint angle estimation model;
[0079] 3.2) The gating decision network outputs a decision vector, which is a three-dimensional vector. Based on the decision vector, three trained joint angle estimation network models M1 are selected for execution. ’ M2 ’ M3 ’ One of the M in x ’ where x = 1, 2, 3;
[0080] 3.3) Select existing resource loss loss r The formula is as follows:
[0081] ;
[0082] Where g i Given the i-th decision vector of the gating network, C(E) i The computational cost of executing the i-th expert network can be predetermined before training, i=1,2,...,B, where B is the dimension of the decision vector, and in this example, B=3.
[0083] 3.4) Loss of existing resources rAdding the joint angle prediction loss (Loss2) to the loss function of the dynamic fusion joint angle estimation model yields the loss function Loss. ’ :
[0084] Loss ’ =Loss r +Loss2;
[0085] 3.5) Freeze M1 ’ M2 ’ M3 ’ The parameters are used to calculate the loss function. ’ And then propagate in reverse;
[0086] 3.6) Calculate the parameter-pair loss function in the network. ’ The effect of this, i.e., the gradient of the parameters:
[0087] When calculating the gradient, the decision vector g output by the gated decision network... i Since it is a discrete vector, it cannot be differentiated. Therefore, this example uses a method similar to g. i Real-valued vectors g with approximately equal gradients i ’ To replace g i Perform calculations, g i ’ The calculation formula is as follows:
[0088] ;
[0089] Where, G(x) i Let x be the probability of the i-th class, and b be the discrete vector output by the gating decision network. i Let b be a random variable that is independent and identically distributed with respect to the i-th class according to a standard Gumbel distribution. j Let j be a random variable that is independent and identically distributed according to a standard Gumbel distribution with respect to the j-th class. Let be the hyperparameters, i = 1, 2, 3, j = 1, 2, ..., B, where B is the dimension of the decision vector;
[0090] 3.6) Update the parameters in the network using the gradient descent algorithm to make the loss function Loss ’ The value gradually decreases;
[0091] 3.7) Repeat steps 3.1) through 3.6) until the loss function is lost. ’ The model converges, resulting in a well-trained dynamic fusion joint angle estimation model.
[0092] Step 4: Input the test set into the trained dynamic fusion joint angle estimation model and output the estimated values of hand joint angles.
[0093] The effectiveness of this invention can be further illustrated by the following simulation results:
[0094] I. Simulation Conditions
[0095] The hardware test platform for the simulation experiment is: CPU is Intel(R) Core(TM) i7-8700 with a main frequency of 3.2GHz, memory is 16GB, and GPU is NVIDIA GeForce GT 710.
[0096] The software testing platform for the simulation experiment is: Windows 11 operating system and Python 3.7.
[0097] The simulation experiment used the Ninapro DB2 dataset, a publicly available dataset containing 49 hand movements from 40 subjects, including 12 electromyographic (EMG) signals, 36 acceleration signals, and 22 kinematic data from Cyberglove II. During data collection, subjects were asked to repeat specified movements with their right hand, each movement lasting 5 seconds, followed by a 3-second rest. Each movement consisted of 6 repetitions, and the sampling frequency was 2 kHz. The simulation experiment selected 6 hand movements closely related to daily life and estimated the joint angles at 10 key finger positions.
[0098] The simulation uses three existing comparison methods, the sources of which are:
[0099] Shurun Wang et al. proposed a method for estimating hand joint angles in their paper "Continuous Estimation of Human JointAngles From sEMG Using a Multi-Feature Temporal Convolutional Attention-Based Network".
[0100] Yanjuan Geng et al. proposed a method for estimating hand joint angles in their paper "A CNN-Attention Network for Continuous Estimation of Finger Kinematics from Surface Electromyography".
[0101] Chao Wang et al. proposed a method for estimating hand joint angles in their paper "sEMG-based continuous estimation of graspmovements by long-short term memory network".
[0102] II. Simulation Content
[0103] Using the present invention and the three existing methods mentioned above, hand joint angles in the Ninapro DB2 dataset were estimated respectively. The average estimation time for each method is recorded in Table 1, and the average correlation coefficient and average root mean square error between the estimated and actual joint angle values are calculated in Table 2.
[0104] Table 1: Average Estimation Time for Different Methods
[0105] The method of the present invention 55.2ms Existing Method 1 76.7ms Existing Method 2 88.5ms Existing Method 3 83.4ms
[0106] As shown in Table 1, the average estimation time of the present invention is lower than that of existing methods. Specifically, it is reduced by 21.5ms compared to existing method 1, 33.3ms compared to existing method 2, and 28.2ms compared to existing method 3. This significantly improves the efficiency of joint angle estimation, solves the problem of long calculation time of joint angle estimation models in the prior art, and can achieve high real-time performance in practical applications, thus improving the user experience.
[0107] Table 2. Average correlation coefficients and average root mean square errors for different methods.
[0108] The method of the present invention 0.960 ± 0.025 4.25 ± 0.85 Existing Method 1 0.825 ± 0.015 7.93 ± 0.35 Existing Method 2 0.873 ± 0.035 9.65 ± 0.85 Existing Method 3 0.841 ± 0.055 5.89 ± 0.95
[0109] As shown in Table 2, the average correlation coefficient between the estimated joint angles and the actual values of this invention is higher than that of existing methods. Specifically, it is improved by 0.135 compared to existing method 1, 0.087 compared to existing method 2, and 0.119 compared to existing method 3, significantly improving the correlation between the estimated and actual joint angles. Therefore, the estimated joint angles are more consistent with real and natural human hand movements. The root mean square error (RMSE) between the estimated and actual joint angles of this invention is lower than that of existing methods. Specifically, it is reduced by 3.68 compared to existing method 1, 5.4 compared to existing method 2, and 1.64 compared to existing method 3, significantly improving the accuracy of joint angle estimation. This solves the problem of insufficient accuracy in joint angle estimation in existing technologies, which may deviate from real hand movements. In practical applications such as myoelectric prosthetic hand control and rehabilitation assessment, it can achieve more accurate, coherent, and natural hand movement simulation, improving user experience and the reliability of task execution.
[0110] The above results show that the performance of the present invention in estimating hand joint angles is significantly better than that of existing methods. The model calculation speed is faster, and the estimation results are more accurate and realistic. It can be applied to related research based on hand motion recognition to achieve high real-time performance, high accuracy and realistic and natural hand motion simulation, thereby improving the interactive experience and efficiency.
[0111] It should be noted that the step numbers in the specification and claims of this invention are only for the purpose of clearly describing the embodiments of this invention and facilitating understanding, and their order is not limited.
Claims
1. A method for estimating joint angles by dynamically fusing multi-modal information, characterized in that, Includes the following steps: (1) Acquire surface electromyography signals, acceleration signals and hand kinematic data, and divide them into training set and test set; (2) Construct a dynamic fusion joint angle estimation model: (2a) Convolutional neural networks and visual Transformer networks are selected. Electromyographic (EMG) signals and acceleration signals are input into the convolutional neural network. The output EMG and acceleration features are input into the visual Transformer network in three different ways to form three joint angle estimation network models M1, M2, and M3 with different output data. The three different input methods are as follows: The first method is to first divide the electromyographic signal feature matrix into patches of size 2×N1, and then add position information encoding to all patches, where N1 is the channel dimension of the electromyographic signal feature. The second method is to first divide the acceleration signal feature matrix into patches of size 2×N2, and then add position information encoding to all patches, where N2 is the channel dimension of the acceleration signal feature. The third approach is to first concatenate the electromyographic signal features and acceleration signal features along the channel dimension, then divide the resulting feature matrix into patches of size 2×N, and finally add position information encoding to all patches, where N=N1+N2. (2b) Based on the kinematic constraints of the hand, construct the loss function Loss of the joint angle estimation network model; (2c) Input the training set into the three joint angle estimation network models respectively, and minimize their respective loss functions through backpropagation to obtain the three trained joint angle estimation network models M1. ’ M2 ’ M3 ’ ; (2d) Two multilayer perceptron models are cascaded to form a gated decision network, and then connected to three trained joint angle estimation network models to form a three-way parallel dynamic fusion joint angle estimation model. (3) Training the dynamic fusion joint angle estimation model: (3a) Input the training set into the three-way parallel dynamic fusion joint angle estimation model, and the output of the gating decision network determines which of the three trained joint angle estimation network models to be executed; (3b) Loss of resources r Loss2, the sum of the joint angle prediction losses, serves as the loss function for the dynamic fusion joint angle estimation model. ’ ; (3c) Freeze M1 ’ M2 ’ M3 ’ The parameters, which remain unchanged during training, are used to minimize the loss function Loss through backpropagation. ’ The gating decision network is trained to obtain a well-trained dynamic fusion joint angle estimation model; (4) Input the test set into the trained dynamic fusion joint angle estimation model and output the estimated values of hand joint angles.
2. The method according to claim 1, characterized in that, The implementation steps of step (1) include the following: (1a) Obtain surface electromyography signals, acceleration signals, and hand kinematics data from publicly available websites to form a dataset; (1b) Preprocess the dataset: (1b1) Extract active segment data from the dataset and downsample the extracted data to 200 Hz; (1b2) Noise was removed from the downsampled surface electromyography and acceleration signals using a 5-450 Hz bandpass Butterworth filter and a Gaussian filter, respectively. The downsampled hand kinematics data was smoothed using a 2 Hz fourth-order low-pass Butterworth filter. (1b3) The surface electromyography signal, acceleration signal and hand kinematic data processed in step (1b2) are divided into a unified sliding window with a window size of 200ms and a step size of 100ms. The first data of each window in the hand kinematic data is extracted as the true value of the joint angle of that window. (1c) Divide the preprocessed dataset into training and test sets in a 4:2 ratio.
3. The method according to claim 1, characterized in that, The convolutional neural network selected in step (2a) It consists of two identical convolutional structures connected in parallel. Each convolutional structure includes five convolutional layers, followed by a ReLU activation function and a batch normalization layer. The parameter settings for the five convolutional layers are as follows: The first convolutional layer has 64 3×1 convolutional kernels with a stride of 1×1; The second convolutional layer consists of 64 3×1 convolutional kernels with a stride of 1×1; The third convolutional layer consists of 64 3×1 convolutional kernels with a stride of 1×1; The fourth convolutional layer has 128 1×1 convolutional kernels with a stride of 2×1; The fifth convolutional layer has 128 2×1 convolutional kernels with a stride of 1×1.
4. The method of claim 1, wherein, The visual Transformer network selected in step (2a) consists of a cascaded encoding part and a regression part. The regression part consists of a fully connected layer with an output dimension of 10, and the encoding part consists of two cascaded identical encoding blocks. The structure of each encoding block is as follows: The first is layer normalization; The second is multi-head attention, which consists of 6 self-attention blocks connected in parallel; The third is Dropout, with a probability of 0.1; The fourth is layer normalization; The fifth is the multilayer perceptron; The sixth is Dropout, with a probability of 0.
1.
5. The method of claim 1, wherein, In step (2b), the loss function Loss of the joint angle estimation network model is constructed based on the kinematic constraints of the hand. The implementation steps include the following: (2b1) Based on the kinematic constraints of the hand, select the most representative set of constraints in daily hand activities, namely the constraint relationships between the MCP and PIP joints of the four fingers other than the thumb: ； Where, θ P θ is the angle of the proximal interphalangeal joint. M The angle of the metacarpophalangeal joint; (2b2) Based on the constraints of the selected hand kinematics, construct the hand joint constraint loss Loss1: ； Where MSE is the mean square error calculation function; (2b3) Calculate the joint angle prediction loss Loss2: ； Where x is the estimated value of the joint angle and y is the actual value of the joint angle; (2b4) Add the hand joint constraint loss to the joint angle prediction loss to form the loss function: 。 6. The method of claim 1, wherein, In step (2c), the loss function Loss of each network is minimized through backpropagation to obtain three trained joint angle estimation network models M1. ’ M2 ’ M3 ’ The implementation steps include the following: (2c1) Calculate the influence of the network parameters on the loss function Loss during backpropagation, i.e. the gradient of the parameters, and update the network parameters through the gradient descent algorithm so that the value of the loss function Loss gradually decreases. (2c2) Repeat step (2c1) until the loss function converges, and obtain the trained joint angle estimation network model M1. ’ M2 ’ M3 ’ .
7. The method of claim 1, wherein, In step (2d), two multilayer perceptron models are cascaded to form a gated decision network, the structure of which is: a first multilayer perceptron with a size of 1920×128 → a second multilayer perceptron with a size of 128×3.
8. The method according to claim 1, characterized in that, The resource loss in step (3b) r The calculation formula is as follows: ； Among them, g i Given the i-th decision vector of the gating network, C(E) i The computational cost of executing the i-th expert network can be predetermined before training, i=1,2,...,B, where B is the dimension of the decision vector.
9. The method according to claim 1, characterized in that, In step (3c), the loss function Loss is minimized through backpropagation. ’ The gating decision network is trained to achieve the following: (3c1) For cases where the output of the gated decision network is a discrete vector and cannot be differentiated, a real-valued vector g, which is approximately equal to the gradient of the discrete vector, is used during backpropagation. i ’ To replace it for calculation, g i ’ The calculation formula is as follows: ； Where, G(x) i Let x be the probability of the i-th class, and b be the discrete vector output by the gating decision network. i Let b be a random variable that is independent and identically distributed with respect to the i-th class according to a standard Gumbel distribution. j Let j be a random variable that is independent and identically distributed with respect to the j-th class, and follow a standard Gumbel distribution. Let be the hyperparameters, i = 1, 2, 3, j = 1, 2, ..., B, where B is the dimension of the decision vector; (3c2) During backpropagation, calculate the influence of the network parameters on the loss function, i.e., the gradient of the parameters, and update the network parameters using the gradient descent algorithm to make the loss function... ’ The value gradually decreases; (3c3) Repeat step (3c2) until the loss function is lost. ’ The model converges, resulting in a well-trained dynamic fusion joint angle estimation model.