Joint angle estimation method and device, electronic equipment and storage medium

Abstract

The application provides a joint angle estimation method and device, electronic equipment and storage medium, and relates to the technical field of joint angle estimation. The method comprises the following steps: acquiring an electromyographic signal and a joint angle signal of a training target, and generating a training set based on the electromyographic signal and the joint angle signal; training a base neural network model based on the training set to generate a joint angle estimation model; acquiring new action data, wherein the new action data is an electromyographic signal collected for a detection target; performing parameter calibration on the joint angle estimation model based on the new action data, and generating a joint angle estimation result for the detection target through the joint angle estimation model. The application solves the problem of low generalization performance of the joint angle estimation method in the related art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of joint angle estimation, in particular, the present application relates to a joint angle estimation method and device, electronic equipment and storage medium. BACKGROUND

[0002] Surface myoelectric signal driven control of exoskeleton rehabilitation hand is an important means to help patients with hand dysfunction to restore motor function. The core of this technology is the motion intention recognition based on surface myoelectric signal. Due to the physiological and anatomical differences of different users, the myoelectric interface of multiple users and multiple action training cannot directly adapt to the hand motion intention recognition of new users and new actions, which shows serious user dependence in actual application. Therefore, in the cross-user and cross-action scene, cross-user joint angle recognition is needed.

[0003] In the prior art, the cross-user joint angle recognition method can be roughly divided into domain adaptation and domain generalization. The goal of domain adaptation is to transfer the knowledge learned by the neural network model from the source domain to the target domain. In the training phase, the labeled data of the source domain subjects and the unlabeled data of the target domain subjects are used to align the feature distributions of the source domain and the target domain through distance measurement or adversarial methods, so as to obtain a personalized joint angle recognition model. However, the transfer learning method based on domain adaptation is more susceptible to individual differences due to physiological and motion signals. When the individual difference is large, the general domain adaptation method may not achieve satisfactory results, so the joint angle recognition method based on domain adaptation cannot adapt to the cross-user environment and has low universality. Moreover, the requirement of domain adaptation method for target user's unlabeled data in offline training phase will also bring inconvenience to users in practical application.

[0004] The joint angle recognition method based on domain generalization extracts domain-independent features from source domain data as much as possible through some strategies to improve the generalization ability of the model. Based on the domain generalization transfer learning method, the end-to-end trained neural network model is difficult to get rid of the influence of the source domain data distribution under the limitation of the data set size. When the individual difference is large in the cross-user environment, the domain generalization method may not achieve ideal generalization effect, the training process is time-consuming, the user acceptance is low, and the generalization is low.

[0005] From the above, the problem of how to improve the generalization of the joint angle estimation method still needs to be solved. SUMMARY

[0006] The present application provides a joint angle estimation method, device, electronic equipment and storage medium, which can solve the problem of low generalization performance of the joint angle estimation method in related technologies. The technical solution is as follows:

[0007] According to one aspect of this application, a joint angle estimation method is characterized by comprising:

[0008] Acquire electromyographic (EMG) signals and joint angle signals for the training target, and generate a training set based on the EMG signals and joint angle signals;

[0009] The base neural network model is trained based on the training set to generate a joint angle estimation model;

[0010] Acquire new motion data, wherein the new motion data is electromyographic signals collected for the detection target;

[0011] The joint angle estimation model is calibrated based on the new motion data, and the joint angle estimation result for the detected target is generated through the joint angle estimation model.

[0012] According to one aspect of this application, a joint angle estimation method apparatus is characterized by comprising:

[0013] The signal acquisition module is used to acquire electromyographic signals and joint angle signals for the training target, and to generate a training set based on the electromyographic signals and the joint angle signals.

[0014] The training module is used to train the base neural network model based on the training set to generate a joint angle estimation model.

[0015] The data acquisition module is used to acquire new motion data, wherein the new motion data is electromyographic signals acquired for the detection target;

[0016] The result generation module is used to calibrate the parameters of the joint angle estimation model based on the new motion data, and generate joint angle estimation results for the detected target through the joint angle estimation model.

[0017] In one exemplary embodiment, the apparatus further includes:

[0018] The filtering module is used to filter electromyographic signals and joint angle signals.

[0019] A synchronization module is used to perform time synchronization processing on the joint angle signal and the electromyographic signal to synchronize the timing of the joint angle signal and the electromyographic signal.

[0020] The selection module is used to select the action stage of the electromyographic signal and the joint angle signal based on the preset action stage, so as to determine the action stages corresponding to the electromyographic signal and the joint angle signal in time sequence.

[0021] In an example embodiment, the training set comprises multi-channel samples and corresponding joint angle data;

[0022] The training module comprises:

[0023] The segmentation unit is configured to perform signal segmentation processing on the electromyography signals to obtain multi-channel samples comprising at least two sample windows.

[0024] The data generation unit is configured to generate joint angle data corresponding to the multi-channel samples based on the last joint angle value corresponding to the sample windows in the multi-channel samples.

[0025] In an example embodiment, the base neural network model comprises a feature extraction module, a multi-layer perception module, and a nonlinear activation module, wherein the feature extraction module comprises a down-sampling unit and a depth-wise separable attention unit.

[0026] In an example embodiment, the joint angle estimation model comprises a multi-user multi-action neural network model.

[0027] The training module comprises:

[0028] The prediction unit is configured to predict joint angle values of the multi-channel samples in the training set based on the base neural network model to obtain joint angle prediction data.

[0029] The loss value unit is configured to calculate a loss value between the joint angle prediction data and joint angle data corresponding to the multi-channel samples based on a loss function.

[0030] The update unit is configured to update model parameters of the base neural network model based on the loss value, and stop training when the loss function converges to obtain a multi-user multi-action neural network model.

[0031] In an example embodiment, the joint angle estimation model comprises a multi-user multi-action neural network model and a new user new action neural network model.

[0032] The result generation module comprises:

[0033] The initialization unit is configured to initialize a base neural network model based on model parameters of the multi-user multi-action neural network model.

[0034] The parameter tuning unit is configured to tune the base neural network model based on an adversarial transfer learning strategy to obtain a new user new action neural network model.

[0035] The result generation unit is configured to predict joint angle values of the new action data based on the new user new action neural network model to obtain joint angle estimation results.

[0036] In an example embodiment, the parameter adjusting unit comprises:

[0037] The first extracting sub-unit is configured to perform action feature extraction on the training set based on the multi-user multi-action neural network model to obtain first user feature data.

[0038] The second extracting sub-unit is configured to perform action feature extraction on the new action data based on the base neural network model to obtain second user feature data.

[0039] The distance calculating sub-unit is configured to calculate a feature distance between the first user feature data and the second user feature data based on the feature domain discriminator.

[0040] The parameter adjusting sub-unit is configured to adjust model parameters of the base neural network model based on the feature distance, and fix the model parameters and stop training when the feature distance converges, to obtain a new user new action neural network model.

[0041] According to an aspect of the present application, an electronic device comprises at least one processor and at least one memory, wherein the memory has stored thereon computer readable instructions; the computer readable instructions are executed by one or more of the processors, so that the electronic device implements the joint angle estimation method as described above.

[0042] According to an aspect of the present application, a storage medium has stored thereon computer readable instructions, which are executed by one or more processors to implement the joint angle estimation method as described above.

[0043] According to an aspect of the present application, a computer program product comprises computer readable instructions stored in a storage medium, which are read by one or more processors of an electronic device, loaded and executed by the one or more processors, so that the electronic device implements the joint angle estimation method as described above.

[0044] The technical scheme provided by the present application has the beneficial effects that:

[0045] In the above technical scheme, the joint angle estimation model is trained based on the training set to realize joint angle estimation for the detection target, and the parameter calibration of the joint angle estimation model enables the joint angle estimation model to estimate the joint angle of actions not appearing in the training set, thereby improving the accuracy of joint angle estimation and the model generalization. Thus, the problem of low generalization performance of the joint angle estimation method in the related art can be effectively solved. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0047] Figure 1 is a schematic diagram according to an implementation environment involved in the present application;

[0048] Figure 2 is a flowchart of a joint angle estimation method according to an exemplary embodiment;

[0049] Figure 3 is a flowchart of a joint angle estimation method according to another exemplary embodiment;

[0050] Figure 4 is a flowchart of generating a training set according to an exemplary embodiment;

[0051] Figure 5 is Figure 2 is a flowchart of step 230 in the corresponding embodiment in one embodiment;

[0052] Figure 6 is a structural diagram of a joint angle estimation model according to an exemplary embodiment;

[0053] Figure 7 is Figure 2 is a flowchart of step 270 in the corresponding embodiment in one embodiment;

[0054] Figure 8 is Figure 7 is a flowchart of step 273 in the corresponding embodiment in one embodiment;

[0055] Figure 9 is a structural diagram of a dense connection residual module according to an exemplary embodiment;

[0056] Figure 10 is a structural block diagram of a joint angle estimation method device according to an exemplary embodiment;

[0057] Figure 11 is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0058] Embodiments of the present application are described below in detail with reference to the accompanying drawings, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary only, and are used only to explain the present application, and cannot be interpreted as a limitation on the present application.

[0059] Those skilled in the art of the present technology can understand that, unless specifically stated, the singular forms "a", "an", "said" and "the" used herein also include the plural forms. It should be further understood that the use of the phrase "comprising" in the specification of the present disclosure means that the features, integers, steps, operations, elements and / or components exist, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be an intermediate element. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any single unit and all combinations of the associated listed items.

[0060] As described above, the existing technology based on domain adaptation of transfer learning method is more susceptible to individual differences due to physiological and motion signals. When the difference between individuals is large, the universal domain adaptation method may not achieve satisfactory results, so the joint angle recognition method based on domain adaptation cannot adapt to the cross-user environment, and the generalization is low. Moreover, the requirement of domain adaptation method for unlabeled data of target user in the offline training stage will also bring inconvenience to users in practical application. Based on the domain generalization of transfer learning method, under the limitation of the size of the data set, the neural network model trained end to end is difficult to get rid of the influence of the source domain data distribution. When the individual difference is large in the cross-user environment, the domain generalization method may not obtain ideal generalization effect, the training process is time-consuming, the user acceptance is low, and the generalization is low.

[0061] As can be seen from the above, the related art still has the defects of poor efficiency and poor interaction effect of joint angle estimation method.

[0062] Therefore, the joint angle estimation method provided by the present application can effectively improve the accuracy of the joint angle estimation method. Correspondingly, the joint angle estimation method is suitable for a joint angle estimation method device, which can be deployed on an electronic device. The electronic device can be a computer device configured with a von Neumann architecture, for example, the computer device includes a desktop computer, a notebook computer, a server, etc. The electronic device can also be an electronic device with a central control function, for example, the electronic device includes a gateway, etc. The electronic device can also be a portable mobile electronic device, for example, the electronic device includes a smart phone, a tablet computer, etc.

[0063] In order to make the purposes, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0064] Figure 1 A schematic diagram of an implementation environment involved in an image processing method. It should be noted that this implementation environment is only an example adapted to the present application and should not be considered as providing any limitation on the use range of the present application.

[0065] The implementation environment includes a terminal 110 and a server 130.

[0066] Specifically, the terminal 110, which can also be considered as an interactive device, includes but is not limited to a PC, an electromyographic signal acquisition device, and other electronic devices with joint angle estimation functions. For example, the terminal 110 is an ultrasonic array tactile stimulation module.

[0067] The server 130 can be a desktop computer, a notebook computer, a server, or other electronic devices. It can also be a computer cluster composed of multiple servers or even a cloud computing center composed of multiple servers. The server 130 is used to provide background services, such as joint angle estimation services and the like.

[0068] The server 130 and the terminal 110 are pre-established in network communication connection through wired or wireless means, and the data transmission between the server 130 and the terminal 110 is realized through the network communication connection. The transmitted data includes but is not limited to electromyographic signals of target objects and the like.

[0069] In an application scenario, through the interaction between the terminal 110 and the server 130, the terminal 110 photographs and acquires electromyographic signals of a target object, and uploads the electromyographic signals to the server 130 to request the server 130 to provide joint angle estimation services.

[0070] For the server 130, after receiving the electromyographic signals of the target object uploaded by the terminal 110, the joint angle estimation service is called to generate a joint angle estimation model and generate a joint angle estimation result, so as to solve the problem of low generalization performance in the related art.

[0071] Referring to Figure 2 The embodiments of the present application provide a joint angle estimation method, which is applicable to an electronic device, which can be a server 130 in the implementation environment shown in the Figure 1

[0072] ​In the following method embodiments, for ease of description, the execution subject of each step of the method is an electronic device, but this does not constitute a specific limitation.

[0073] like Figure 2 As shown, the method may include the following steps:

[0074] Step 210: Acquire electromyographic signals and joint angle signals for the training target, and generate a training set based on the electromyographic signals and joint angle signals.

[0075] It should be noted that the detection target is the object that provides electromyographic signals for joint angle estimation, while the training target is the object that can provide the same type of electromyographic signals and joint angle signals as the detection target. For example, the training target is one or more subjects, through which the subjects perform different actions, and electromyographic signals and joint angle signals are collected from the subjects simultaneously during the execution of the actions to generate a training set.

[0076] In one possible implementation, the training set includes multi-channel samples and corresponding joint angle data. The multi-channel samples are segmented electromyographic signals, and the joint angle data are the true values ​​of the joint angles corresponding to the multi-channel samples.

[0077] In one possible implementation, a 12-channel electromyography (EMG) acquisition device is used to record EMG signals of hand movements, while a data glove is used to record hand joint angle signals.

[0078] Step 230: Train the base neural network model based on the training set to generate a joint angle estimation model.

[0079] Among them, the basal neural network model is a neural network model that has the ability to predict joint angle data based on electromyographic signals. The basal neural network model is trained with a training set to improve its ability to predict the corresponding joint angle based on the electromyographic signals in the training set, and finally generate a joint angle estimation model with high prediction accuracy.

[0080] In one possible implementation, the joint angle estimation model can be the DSTrans model.

[0081] Step 250: Obtain new motion data.

[0082] The new motion data consists of electromyographic (EMG) signals collected from a target being tested. Specifically, the target being tested can be an object distinct from the training target. The EMG signals in the new motion data are acquired by collecting signals when the target being tested performs a movement not present in the training set.

[0083] At step 270, the joint angle estimation model is calibrated based on the new action data, and a joint angle estimation result of the detection target is generated by the joint angle estimation model.

[0084] The parameter calibration aligns different feature domains of the new action data and the training set in the implicit space through the adversarial transfer learning strategy, so that the joint angle model adapts to the joint angle estimation based on the new action data, and then generates the joint angle estimation result indicating the joint angle of the detection target based on the new action data.

[0085] Through the above process, the joint angle estimation model is trained based on the training set, the joint angle estimation of the detection target is realized, and the joint angle estimation model can estimate the joint angle of the action not appearing in the training set through the parameter calibration of the joint angle estimation model, thereby improving the accuracy of the joint angle estimation and the model generalization.

[0086] In an exemplary embodiment, as shown in Figure 3 After step 210, the method can further include the following steps:

[0087] At step 310, the electromyographic signal and the joint angle signal are filtered.

[0088] It should be noted that there is noise and other interference in the electromyographic signal and the joint angle signal, and the quality of the electromyographic signal and the joint angle signal is improved through targeted filtering processing of the electromyographic signal and the joint angle signal, so that the electromyographic signal and the joint angle signal can meet the training requirements.

[0089] In a possible implementation, a fourth-order Butterworth filter is used to band-pass filter the electromyographic signal to perform baseline correction and remove surface electromyographic signal noise, and a notch filter is used to remove power frequency interference in the electromyographic signal.

[0090] In a possible implementation, a 2Hz zero-phase low-pass filter is used to smooth the joint angle signal to avoid ladder jitter in the signal.

[0091] At step 330, the joint angle signal and the electromyographic signal are time-synchronized to synchronize the timing of the joint angle signal and the electromyographic signal.

[0092] The joint angle signal and the electromyographic signal are generated synchronously in the process of the training target performing the action, so that the collected electromyographic signal and joint angle signal have a mapping relationship in timing by resampling and other time synchronization processing of the joint angle signal and the electromyographic signal.

[0093] In a possible implementation, resampling is performed after the joint angle signal is collected, and the resampling frequency is the same as the electromyographic signal sampling frequency.

[0094] In a possible implementation, a mapping relationship between the electromyography signal and the joint angle signal is established based on a deep separable convolution and a Transformer neural network.

[0095] At step 350, the electromyography signal and the joint angle signal are subjected to action phase selection based on a preset action phase, so as to determine respective action phases to which the electromyography signal and the joint angle signal correspond in time sequence.

[0096] It can be understood that when the training target performs an action, each action can be divided into different phases. For example, when the training target performs a hand movement action, each hand movement action can be divided into a sequence of four phases of grasping, holding, releasing, and resting.

[0097] It should be noted that at this time, each movement phase can be screened and determined according to the main focus of joint angle estimation, and the respective action phases to which the electromyography signal and the joint angle signal correspond in time sequence are determined. For example, only the grasping and releasing phases in which the electromyography signal and the joint angle signal have important electromyography-joint mapping relationships are determined, while the holding and resting phases are abandoned, so that the training process can be concentrated on the grasping and releasing phases, thereby improving the training effect and training efficiency.

[0098] In a possible implementation, the maximum value and the minimum value of the electromyography data and the joint angle data are collected, and normalization processing is performed after the action phase selection of the electromyography signal and the joint angle signal, so as to improve the quality of the training set and improve the operation efficiency.

[0099] Through the above process, the quality of the electromyography signal and the joint angle signal is improved through various processing of the electromyography signal and the joint angle signal, and the training effect and training quality of the subsequent model training process are ensured.

[0100] In an example embodiment, as shown in Figure 4 the process of generating the training set can include the following steps:

[0101] At step 410, the electromyography signal is subjected to signal segmentation processing to obtain a plurality of multi-channel samples including at least two sample windows.

[0102] The signal segmentation processing is to divide the electromyography signal into a plurality of multi-channel samples through a sliding window.

[0103] At step 430, the joint angle data corresponding to the plurality of multi-channel samples is generated based on the last joint angle value corresponding to the sample window in the plurality of multi-channel samples.

[0104] Wherein, after the electromyographic signal is segmented into multiple multi-channel samples, the joint angle vector dimension of the corresponding joint angle signal is used to determine the number of joint angle values in the sample window, and the last joint angle value in each sample window is taken as the true value of the corresponding multi-channel sample to generate a training set.

[0105] Through the above process, the training set is generated based on the electromyographic signal and the joint angle signal, the model training process is ensured, the correspondence between the multi-channel sample and the joint angle data is determined through signal segmentation processing, the accuracy of the training set data is improved, and the model training effect is ensured.

[0106] In an exemplary embodiment, as shown in Figure 5 Step 230 can include the following steps:

[0107] Step 231, based on the base neural network model, the joint angle value of the multi-channel sample in the training set is predicted, and the joint angle prediction data is obtained.

[0108] Specifically, after the multi-channel sample is input into the base neural network model, the base neural network model predicts the joint angle value corresponding to the multi-channel sample, and then obtains the joint angle prediction data.

[0109] In an exemplary embodiment, the base neural network model includes a feature extraction module, a multi-layer perception module and a nonlinear activation module, wherein the feature extraction module includes a down-sampling unit and a depth separable attention unit.

[0110] In a possible implementation, the base neural network model has four feature extraction modules, one multi-layer perception and one nonlinear activation module, each feature extraction model contains a down-sampling part and a depth separable attention part, wherein the number of depth separable attention modules in each feature extraction module is 2-2-6-2, and the feature extraction module is composed of convolution block or self-attention mechanism.

[0111] Step 233, based on the loss function, the loss value between the joint angle prediction data and the joint angle data corresponding to the multi-channel sample is calculated.

[0112] By comparing the joint angle data corresponding to the multi-channel sample with the joint angle prediction data as the true value, the error generated by the base neural network model is quantified as a loss value.

[0113] In a possible implementation, the loss function is a supervised mean square loss function.

[0114] Step 235, based on the loss value, the model parameters of the base neural network model are updated, and when the loss function converges, the training is stopped, and a multi-user multi-action neural network model is obtained.

[0115] In the process of training, whether the loss function converges is used to judge the training effect of the base neural network model, for example, when the loss value no longer stably decreases, that is, the loss function converges, the prediction ability of the base neural network model for joint angle data reaches the strongest, at this time the training is completed, and the multi-user multi-action neural network model is obtained.

[0116] Through the above process, the base neural network model is iterated in the direction of improving performance through training, so that the performance of the base neural network model can be improved through continuous iteration, so as to achieve the maximum effect of predicting joint angle data.

[0117] In an example embodiment, as shown in Figure 7 Step 270 can include the following steps:

[0118] Step 271, initializing the base neural network model based on the model parameters of the multi-user multi-action neural network model.

[0119] It should be noted that, as shown in Figure 6 The joint angle estimation model includes a multi-user multi-action neural network model, a new user new action neural network model and a feature domain discriminator. The structure of the new user new action neural network model is exactly the same as that of the multi-user multi-action neural network model. The model parameters of the trained multi-user multi-action neural network model are used for initialization, that is, the initial parameters of the new user new action neural network model are set to be consistent with the trained multi-user multi-action model parameters.

[0120] Step 273, parameter tuning of the base neural network model based on the adversarial transfer learning strategy, to obtain a new user new action neural network model.

[0121] In an example embodiment, as shown in Figure 8 Step 273 can include the following steps:

[0122] Step 2731, action feature extraction of the training set based on the multi-user multi-action neural network model, to obtain first user feature data.

[0123] Step 2733, action feature extraction of the new action data based on the base neural network model, to obtain second user feature data.

[0124] Step 2735, calculating the feature distance between the first user feature data and the second user feature data based on the feature domain discriminator.

[0125] Specifically, please refer to Figure 6After the multi-user multi-action model extracts features from the training set and the new user new action neural network model extracts features from the new action data, three representative feature data from the multi-user neural network model (red arrow line) and three representative feature data from the new user neural network model (blue arrow line) are sent into the feature area discriminator for feature distance calculation. The feature area discriminator is used to calculate and minimize the feature distance between the first user feature data and the second user feature data during model parameter adjustment.

[0126] As shown in FIG. 6, the main components of the feature area discriminator include a dense connection residual module. Figure 9

[0127] The dense connection residual module is composed of a convolution layer, a dense connection layer, a transition layer and a residual connection. The convolution layer is used to extract features, and the dense connection layer is a direct connection between the convolution layers. After the features are extracted by the convolution layer, the features extracted by each convolution layer are spliced as the input of the following convolution layers, so as to improve the information transmission in the neural network and accelerate the convergence speed of the model. The transition layer in the dense connection residual module uses 1x1 convolution to compress the number of features and compress the network. The residual connection structure in the dense connection residual module gives the network a deeper structure and a richer feature representation.

[0128] Step 2737, based on the feature distance, adjust the model parameters of the base neural network model, until the feature distance converges, fix the model parameters and stop training, and obtain the new user new action neural network model.

[0129] The parameters of the multi-user multi-action neural network model are frozen, and the parameters of the new user new action neural network model and the feature area discriminator are corrected and optimized. Each model parameter in the new user new action neural network model and the feature area discriminator is adjusted by the back propagation algorithm, and the new action data is continuously input for training. Through repeated iteration, the training process of the new user new action neural network model and the feature area discriminator is realized. When the feature area discriminator cannot distinguish the first user feature data and the second user feature data and cannot find the minimum distance, the cross-model knowledge transfer is completed, and the new user new action neural network model is obtained.

[0130] The process of cross-user model transfer learning is trained by the following formulas 1-5:

[0131] Wherein, L total is the total loss function for optimizing the new user new action neural network model and the feature area discriminator.

[0132] L total = L D + L​N (1)

[0133] L D Loss function representing the relative distance between the feature domain of the multi-user multi-action neural network model and the new-user new-action neural network model.

[0134] L D = -log(DFD(F s )) -log(1-DFD(F t ))#(2)

[0135] Where F s represents the feature of the multi-user multi-action neural network model, F t represents the feature of the new-user new-action neural network model. L N represents the loss function of the new-user new-action neural network model.

[0136] L N = L map + L subject #(3)

[0137] Where L map represents the degree of mapping from the feature of the multi-user multi-action neural network model to the feature of the new-user new-action neural network model.

[0138] L map = -log(DFD(F t ))#(4)

[0139] L subject is used to improve the performance of the loss function, making the new-user new-action neural network model more sensitive to the features of the new-action data in the transfer learning process.

[0140]

[0141] Where Nnet(x) represents the estimated joint angle of the model, represents the label of the joint angle, N represents the number of joints, and a is a weighting coefficient.

[0142] In the above process, the new user new action neural network model and the feature domain discriminator are trained in an antagonistic manner, continuously improving the ability of the feature domain discriminator to find the minimum distance and the joint angle value prediction ability of the new user new action neural network model for the electromyographic signal in the new action data. During the training process, the multi-source domain loss is used for back propagation to correct the model, improving the accuracy and generalization of joint angle estimation. The domain discriminator composed of dense residual connections can more effectively extract feature information for cross-action joint angle estimation. Through the deep separable attention module, domain-invariant features can be more effectively extracted, reducing the interference of domain-related features on joint angle estimation and improving the prediction accuracy of the model.

[0143] Step 275, based on the new user new action neural network model, the joint angle value of the new action data is predicted to obtain the joint angle estimation result.

[0144] After the training is completed, the new user new action neural network model can estimate the joint angle of the joint of the detection target in the continuous motion process according to the new action data to generate the joint angle estimation result.

[0145] Through the above process, when estimating the joint angle according to the new action data, the model does not need to be retrained, the model deployment time is short, and the application efficiency is high. The constructed model uses deep separable convolution and attention mechanism for feature extraction. Deep separable convolution can extract spatial feature information, and attention mechanism can help the model pay more attention to and extract important spatio-temporal features. When the data distribution difference between individuals is large, the related features between the source domain action and the target domain new action can be more effectively extracted, the model generalization ability is improved, and accurate recognition of new actions is realized. By learning the information of the source domain background and the target new user new action to correct the parameters, when the data distribution difference between the source domain background user and the new user new action is large, features that are not related to individual differences and action differences can be effectively extracted, and finally the accuracy of joint angle estimation of the general model is improved.

[0146] In an exemplary embodiment, in an application scenario of a joint angle estimation method, the data of 40 users is collected, of which 32 subjects are used as training targets, each subject performs 34 actions, and each action is repeated 8 times to construct a multi-user multi-action model. The remaining 8 subjects are used as detection targets, each subject performs 6 actions, and each action is repeated 8 times as a target domain data set to correct the model. The remaining 4 repetitions are used for model performance testing.

[0147] The skin surface electromyography signals from the finger extensor, finger flexor, biceps brachii, triceps brachii and a circle of muscle 2-6 cm away from the elbow of the forearm are collected, and the sampling frequency is 2000 Hz. After the collection is completed, the electromyography signals are processed, a fourth-order Butterworth filter of 5-450 Hz is selected for band-pass filtering of the electromyography signals, for baseline correction and removal of surface electromyography signal noise, and then a notch filter is used to remove power frequency interference in the surface electromyography signal. Then the electromyography signals are selected for action stage selection, and the action stage includes two stages of gripping and releasing. The selected electromyography signals are normalized for calculation to improve the calculation efficiency.

[0148] The hand joint angle is collected by the data glove, and the sampling frequency is 20 Hz. After the collection is completed, the joint angle signal is processed, and first resampled to 2000 Hz. Then a 2Hz zero-phase low-pass filter is used to smooth the original joint angle signal. Further, the maximum and minimum values of the collected electromyography data and joint angle data are recorded for normalization of the training and test data. The joint angle signal is selected for action stage selection, and the action stage includes two stages of gripping and releasing.

[0149] A sliding window with a width of 112 ms is used, and each time the window is increased by 0.5 ms. The electromyography signal is divided into more multi-channel samples, and the last joint angle value in each sample window is used as the true value to generate training data.

[0150] The DSTrans model is trained based on the training data, and a supervised mean square loss function is used for optimization.

[0151] The parameters of the trained multi-user multi-action neural network model are used to initialize the new user new action model.

[0152] In the process of transfer learning, part of the parameters of the multi-user multi-action neural network model are frozen, while the parameters of the new user new action neural network model and the feature domain discriminator are corrected and optimized to minimize the feature distance between the multi-user feature domain and the new user feature domain. When the feature domain discriminator cannot distinguish between the two feature domains and find the minimum distance, the new user new action neural network model is obtained.

[0153] The new action data collected by the detection target through the new user new action neural network model is used for joint angle estimation to generate joint angle estimation results.

[0154] In the application scenario, the performance of the joint angle estimation model is as shown in the following table:

[0155]

[0156] The table is a performance evaluation of different migration model algorithms in the same hand multi-joint angle estimation task. The input test data set is used to compare the continuous estimated joint angle curve with the actual joint angle curve obtained by the joint angle sensor. The evaluation indicators in the table include Pearson correlation coefficient (CC), root mean square error (RMSE), determination coefficient (R2), mean absolute error (MAE), and mean absolute percentage error (MAPE) as the evaluation criteria for the regression task.

[0157] The details are as follows:

[0158] 1) Pearson correlation coefficient (CC). CC is a statistical measure used to quantify the strength and direction of the linear relationship between two continuous variables, measuring the tightness of the data points around a straight line in a scatter plot. It can show the closeness of the hand kinematics continuous estimation value to the corresponding true value. The calculation formula of CC is as follows:

[0159]

[0160] Where, the value range of CC is between -1 and 1, the closer the value of CC is to 1, the stronger the correlation, the closer the predicted finger trajectory is to the actual trajectory, and the higher the estimation accuracy of the method.

[0161] 2) Root mean square error (RMSE). RMSE is a commonly used indicator to evaluate the accuracy of a prediction model, usually used to measure the average error between the hand kinematics continuous estimation value and the corresponding true value, the smaller the value, the better the performance. The calculation formula of RMSE is as follows:

[0162]

[0163] 3) Determination coefficient (R2). R2 is a statistical measure used to evaluate the goodness of fit of a regression model. As a comprehensive evaluation indicator to measure the overall accuracy of the model, R2 ranges from 0 to 1, defined as the percentage of variation in the true value explained by the estimated value. The larger the R2 value, the better the estimation performance. The determination coefficient formula is as follows:

[0164]

[0165] 4) Mean absolute error (MAE). Calculate the average of the absolute values of the differences between the predicted value and the true value. Similar to the mean square error, but the mean absolute error considers the absolute value of the error, focusing on the size of the error rather than the square of the error. The smaller the value of MAE, the smaller the prediction error of the model, indicating better performance of the model. The calculation method of MAE is as follows:

[0166]

[0167] 5) Mean Absolute Percentage Error (MAPE). Used to measure the percentage difference between the model's predicted values and the actual observed values. MAPE is usually presented in percentage form. The lower the MAPE value, the smaller the prediction error of the model for the sample, and the better the performance of the model. The calculation formula of MAPE is as follows:

[0168]

[0169] In the above formula, N represents the sample size, p i represents the sample points of the predicted finger joint angle, g i represents the sample points of the actual finger joint angle.

[0170] It can be seen from the above process that the transfer learning algorithm used in the present application not only brings higher accuracy to the estimation of hand joint angles for new users and new actions, but also has better generalization performance for new users.

[0171] The following is an apparatus embodiment of the present application, which can be used to execute the joint angle estimation method involved in the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the method embodiment of the joint angle estimation method involved in the present application.

[0172] Please refer to Figure 10 In the embodiments of the present application, a joint angle estimation device 1000 is provided, which includes but is not limited to: a signal acquisition module 1010, a training module 1030, a data acquisition module 1050, and a result generation module 1070.

[0173] The signal acquisition module 1010 is configured to acquire the electromyographic signals and joint angle signals collected for the training target, and generate a training set based on the electromyographic signals and joint angle signals.

[0174] The training module 1030 is configured to train the base neural network model based on the training set, and generate a joint angle estimation model.

[0175] The data acquisition module 1050 is configured to acquire new action data, wherein the new action data is the electromyographic signals collected for the detection target.

[0176] The result generation module 1070 is configured to calibrate the parameters of the joint angle estimation model based on the new action data, and generate a joint angle estimation result for the detection target through the joint angle estimation model.

[0177] It should be noted that the joint angle estimation method and device provided in the above embodiments are only used as an example to illustrate the division of the above functional modules, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the joint angle estimation method and device is divided into different functional modules to complete all or part of the functions described above.

[0178] In addition, the joint angle estimation method and device provided in the above embodiments belong to the same concept, and the specific manner in which each module performs operations has been described in detail in the method embodiments, which will not be repeated here.

[0179] Please refer to Figure 11 In the embodiments of the present application, an electronic device 4000 is provided, which can include a desktop computer, a notebook computer, a server, etc.

[0180] In Figure 11 , the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0181] The data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 can include a channel for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 can be a PCI (Peripheral Component Interconnect, Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture, Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 11 In the above embodiment, only one thick line is used to represent the communication bus 4002, but it does not mean that there is only one bus or only one type of bus.

[0182] Optionally, the electronic device 4000 can also include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception, etc. It should be noted that in actual application, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0183] The processor 4001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic device, transistor logic device, hardware component, or any combination thereof. The processor 4001 can implement or execute various exemplary logical blocks, modules, and circuits described in conjunction with the disclosure. The processor 4001 can also be a combination of computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0184] The memory 4003 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program instructions in the form of instructions or data structures and that can be accessed by the electronic device 400, but is not limited thereto.

[0185] The memory 4003 stores computer-readable instructions, which can be read by the processor 4001 through the communication bus 4002.

[0186] The computer-readable instructions are executed by the one or more processors 4001 to implement the joint angle estimation method in the above embodiments.

[0187] In addition, a storage medium is provided in the embodiments of the present application, and the storage medium stores computer-readable instructions, which are executed by one or more processors to implement the joint angle estimation method as described above.

[0188] The embodiment of the application provides a computer program product, the computer program product includes computer readable instructions, the computer readable instructions are stored in a storage medium, one or more processors of an electronic device read the computer readable instructions from the storage medium, load and execute the computer readable instructions, so that the electronic device implements the joint angle estimation method as above.

[0189] Compared with the related art, the joint angle estimation model is trained based on the training set, the joint angle estimation of the detection target is realized, the joint angle estimation model can estimate the joint angle of the action not appearing in the training set through parameter calibration of the joint angle estimation model, the accuracy and model generalization of the joint angle estimation are improved. Through various processing of the electromyographic signal and the joint angle signal, the quality of the electromyographic signal and the joint angle signal is improved, and the training effect and training quality of the subsequent model training process are ensured. The training set is generated based on the electromyographic signal and the joint angle signal, the model training process is ensured, the correspondence between the multi-channel sample and the joint angle data is determined through signal segmentation processing, the accuracy of the training set data is improved, and the model training effect is ensured. Through training, the performance of the base neural network model is iterated in the direction of improving performance, so that the performance of the base neural network model can be improved through continuous iteration, so as to achieve the maximum effect of predicting joint angle data. Through the adversarial training of the new user new action neural network model and the feature domain discriminator, the ability of the feature domain discriminator to find the minimum distance is continuously improved, and the joint angle value prediction ability of the new user new action neural network model to the electromyographic signal in the new action data, and the multi-source domain loss is used in the training process to correct the model through back propagation, the accuracy and generalization of the joint angle estimation are improved. The domain discriminator composed of dense residual connection can more effectively extract feature information for cross-action joint angle estimation, and the deep separable attention module can more effectively extract domain-invariant features, reducing the interference of domain-related features on joint angle estimation and improving the prediction accuracy of the model. When estimating the joint angle according to the new action data, the model does not need to be retrained, the model deployment time is short, and the application efficiency is high. The constructed model uses deep separable convolution and attention mechanism for feature extraction, the deep separable convolution can extract spatial feature information, and the attention mechanism can help the model more effectively focus on and extract important spatiotemporal features. When the data distribution difference between individuals is large, the related features between the source domain action and the target domain new action can be more effectively extracted, the model generalization ability is improved, and accurate identification of new actions is realized. By learning the information of the source domain background and the target new user new action to correct the parameters, when the data distribution difference between the source domain background user and the new user new action is large, the features that are not much related to individual difference and action difference can be effectively extracted, and finally the accuracy of the joint angle estimation of the general model is improved.

[0190] It should be understood that although the steps in the flowcharts of the drawings are shown in a sequential order following the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated otherwise herein, the execution of the steps is not strictly limited to the order indicated by the arrows, and can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of which is not necessarily sequential, but can be round-robin or alternately executed with at least part of other steps or sub-steps or stages of other steps.

[0191] The above is only some embodiments of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered as the protection scope of the present application.

Claims

1. A method of estimating an angle of a joint, characterized by, The method comprises: acquiring electromyographic signals and joint angle signals of a training target, and generating a training set based on the electromyographic signals and the joint angle signals; training a base neural network model based on the training set to generate a joint angle estimation model; acquiring new action data, wherein the new action data is electromyographic signals collected from a detection target; the detection target is an object different from the training target; the electromyographic signals in the new action data are collected when the detection target performs an action not present in the training set; parameter calibration of the joint angle estimation model based on the new action data, and generation of joint angle estimation results for the detection target by the joint angle estimation model; wherein the joint angle estimation model comprises a multi-user multi-action neural network model; the training set comprises multi-channel samples and corresponding joint angle data; the training of the base neural network model based on the training set to generate the joint angle estimation model comprises: predicting joint angle values of the multi-channel samples in the training set based on the base neural network model to obtain joint angle prediction data; calculating a loss value between the joint angle prediction data and the joint angle data corresponding to the multi-channel samples based on a loss function; updating the model parameters of the base neural network model based on the loss value until the loss function converges, stopping training, and obtaining a multi-user multi-action neural network model; the joint angle estimation model further comprises a new user new action neural network model; the parameter calibration of the joint angle estimation model based on the new action data, and the generation of joint angle estimation results for the detection target by the joint angle estimation model, comprise: initializing the base neural network model based on the model parameters of the multi-user multi-action neural network model; adjusting the base neural network model based on an adversarial transfer learning strategy to obtain a new user new action neural network model; predicting joint angle values of the new action data based on the new user new action neural network model to obtain joint angle estimation results; the adjustment of the base neural network model based on the adversarial transfer learning strategy to obtain a new user new action neural network model comprises: extracting action features of the training set based on the multi-user multi-action neural network model to obtain first user feature data; extracting action features of the new action data based on the base neural network model to obtain second user feature data; calculating a feature distance between the first user feature data and the second user feature data based on a feature domain discriminator; adjusting the model parameters of the base neural network model based on the feature distance until the feature distance converges, fixing the model parameters and stopping training, and obtaining a new user new action neural network model.

2. The joint angle estimation method of claim 1, wherein After acquiring electromyographic signals and joint angle signals of a training target, the joint angle estimation method further comprises: filtering the electromyographic signals and the joint angle signals; time-synchronize the joint angle signal and the electromyography signal to synchronize the timing of the joint angle signal and the electromyography signal; select an action phase of the electromyography signal and the joint angle signal based on a preset action phase to determine each action phase corresponding in timing of the electromyography signal and the joint angle signal.

3. The joint angle estimation method of claim 1, wherein, The generating the training set based on the electromyography signal and the joint angle signal comprises: performing signal segmentation processing on the electromyography signal to obtain a multi-channel sample comprising at least two sample windows; generating joint angle data corresponding to the multi-channel sample based on a last joint angle value corresponding to a sample window in the multi-channel sample.

4. The joint angle estimation method of claim 1, wherein The base neural network model comprises a feature extraction module, a multi-layer perception module, and a nonlinear activation module, wherein the feature extraction module comprises a down-sampling unit and a depth separable attention unit.

5. A joint angle estimation method device for implementing the joint angle estimation method according to any one of claims 1 to 4, characterized in that, The method comprises: a signal acquisition module configured to acquire electromyography signals and joint angle signals for a training target, and generate a training set based on the electromyography signals and the joint angle signals; a training module configured to train a base neural network model based on the training set to generate a joint angle estimation model; a data acquisition module configured to acquire new action data, wherein the new action data is electromyography signals acquired for a detection target; a result generation module configured to calibrate parameters of the joint angle estimation model based on the new action data, and generate a joint angle estimation result for the detection target through the joint angle estimation model.

6. An electronic device, comprising: The method comprises: at least one processor and at least one memory, computer readable instructions are stored on the memory; the computer readable instructions are executed by the at least one processor to enable the electronic device to implement the joint angle estimation method of any one of claims 1 to 4.

7. A storage medium having stored thereon computer readable instructions, characterized in that, the computer readable instructions are executed by the at least one processor to implement the joint angle estimation method of any one of claims 1 to 4.

Images