Joint angle estimation method and device, electronic equipment and storage medium

Abstract

The invention provides a joint angle estimation method and device, electronic equipment and a storage medium, and relates to the technical field of joint angle estimation. The method comprises the following steps: acquiring an electromyographic signal and a joint angle signal collected for a training target, and generating a training set based on the electromyographic signal and the joint angle signal; training a basal neural network model based on the training set to generate a joint angle estimation model; obtaining new action data, wherein the new action data is an electromyographic signal collected for a detection target; and performing parameter calibration on the joint angle estimation model based on the new action data, and generating a joint angle estimation result for the detection target through the joint angle estimation model. According to the invention, the problem of low generalization performance of a joint angle estimation method in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the technical field of joint angle estimation, and specifically, to a joint angle estimation method, device, electronic device and storage medium. Background Art

[0002] The exoskeleton rehabilitation hand driven and controlled by surface electromyography signals is an important means to help patients with hand dysfunction to rebuild their motor functions. The core of this technology is the recognition of movement intention based on surface electromyography signals. Due to the physiological and anatomical differences between different users, the electromyography interface trained by multiple users and multiple actions cannot directly adapt to the hand movement intention recognition of new users and new actions, and shows serious user dependence in practical applications. Therefore, in cross-user and cross-action scenarios, cross-user joint angle recognition is required.

[0003] In the prior art, cross-user joint angle recognition methods can be roughly divided into two categories: domain adaptation and domain generalization. The goal of domain adaptation is to transfer the knowledge learned by the neural network model from the source domain to the target domain. In the training stage, the labeled data of the source domain subjects and the unlabeled data of the target domain subjects are used to align the feature distributions of the source domain and the target domain through distance measurement or adversarial methods, so as to obtain a personalized joint angle recognition model. However, the transfer learning method based on domain adaptation is more susceptible to individual differences in physiological and motor signals. When the differences between individuals are large, the general domain adaptation method may not achieve satisfactory results. Therefore, the joint angle recognition method based on domain adaptation cannot adapt to the cross-user environment and has low versatility. Moreover, the domain adaptation method's requirement for unlabeled data of target users in the offline training stage will also bring inconvenience to users in practical applications.

[0004] The joint angle recognition method based on domain generalization uses some strategies to extract domain-independent features from the source domain data as much as possible to improve the generalization ability of the model. Based on the transfer learning method of domain generalization, under the limitation of data set size, the end-to-end trained neural network model is difficult to get rid of the influence of the source domain data distribution. When there are large individual differences in the cross-user environment, the domain generalization method may not be able to obtain the ideal generalization effect, the training process is time-consuming, the user acceptance is low, and the generalization is low.

[0005] From the above, we can see that the problem of how to improve the generalization of joint angle estimation methods still needs to be solved. Summary of the invention

[0006] The present application provides a joint angle estimation method, device, electronic device and storage medium, which can solve the problem of low generalization performance of the joint angle estimation method in the related art. The technical solution is as follows:

[0007] According to one aspect of the present application, a joint angle estimation method is characterized by comprising:

[0008] Acquire electromyographic signals and joint angle signals for training targets, and generate a training set based on the electromyographic signals and the joint angle signals;

[0009] Training the base neural network model based on the training set to generate a joint angle estimation model;

[0010] Acquire new motion data, wherein the new motion data is an electromyographic signal collected for a detection target;

[0011] The joint angle estimation model is calibrated based on the new motion data, and a joint angle estimation result for the detection target is generated through the joint angle estimation model.

[0012] According to one aspect of the present application, a joint angle estimation method and device is characterized by comprising:

[0013] A signal acquisition module, used to acquire electromyographic signals and joint angle signals for training targets, and generate a training set based on the electromyographic signals and the joint angle signals;

[0014] A training module, used to train the base neural network model based on the training set to generate a joint angle estimation model;

[0015] A data acquisition module, used to acquire new motion data, wherein the new motion data is an electromyographic signal collected for a detection target;

[0016] The result generating module is used to perform parameter calibration on the joint angle estimation model based on the new motion data, and to generate a joint angle estimation result for the detection target through the joint angle estimation model.

[0017] In an exemplary embodiment, the apparatus further comprises:

[0018] A filtering module is used to filter the electromyographic signal and the joint angle signal;

[0019] A synchronization module, used for performing time synchronization processing on the joint angle signal and the electromyographic signal, so as to synchronize the timing of the joint angle signal and the electromyographic signal;

[0020] The selection module is used to select the action stage of the electromyographic signal and the joint angle signal based on the preset action stage, so as to determine the action stages corresponding to the electromyographic signal and the joint angle signal in time sequence.

[0021] In an exemplary embodiment, the training set includes multi-channel samples and corresponding joint angle data;

[0022] The training module comprises:

[0023] A segmentation unit, used for performing signal segmentation processing on the electromyographic signal to obtain a multi-channel sample including at least two sample windows;

[0024] The data generating unit is used to generate the joint angle data corresponding to the multi-channel sample based on the last joint angle value corresponding to the sample window in the multi-channel sample.

[0025] In an exemplary embodiment, the base neural network model includes a feature extraction module, a multi-layer perceptron module and a nonlinear activation module, wherein the feature extraction module includes a downsampling unit and a deep separable attention unit.

[0026] In an exemplary embodiment, the joint angle estimation model includes a multi-user multi-action neural network model;

[0027] The training module includes:

[0028] A prediction unit, configured to predict joint angle values ​​of multi-channel samples in the training set based on the base neural network model to obtain joint angle prediction data;

[0029] A loss value unit, used for calculating a loss value between the joint angle prediction data and the joint angle data corresponding to the multi-channel samples based on a loss function;

[0030] An updating unit is used to update the model parameters of the base neural network model based on the loss value until the loss function converges, then stop training to obtain a multi-user multi-action neural network model.

[0031] In an exemplary embodiment, the joint angle estimation model includes a multi-user multi-action neural network model and a new user new action neural network model;

[0032] The result generation module comprises:

[0033] An initialization unit, configured to initialize a base neural network model based on model parameters of the multi-user multi-action neural network model;

[0034] A parameter adjustment unit, used to adjust the parameters of the base neural network model based on the adversarial transfer learning strategy to obtain a new user and new action neural network model;

[0035] The result generating unit is used to predict the joint angle value of the new action data based on the new user new action neural network model to obtain the joint angle estimation result.

[0036] In an exemplary embodiment, the parameter adjustment unit includes:

[0037] A first extraction subunit is used to extract action features from a training set based on a multi-user multi-action neural network model to obtain first user feature data;

[0038] A second extraction subunit is used to extract action features from the new action data based on the base neural network model to obtain second user feature data;

[0039] A distance calculation subunit, configured to calculate a feature distance between the first user feature data and the second user feature data based on a feature domain discriminator;

[0040] The parameter adjustment subunit is used to adjust the model parameters of the base neural network model based on the feature distance until the feature distance converges, fix the model parameters and stop training to obtain a new user new action neural network model.

[0041] According to one aspect of the present application, an electronic device includes at least one processor and at least one memory, wherein the memory stores computer-readable instructions; the computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the joint angle estimation method as described above.

[0042] According to one aspect of the present application, a storage medium stores computer-readable instructions thereon, and the computer-readable instructions are executed by one or more processors to implement the joint angle estimation method as described above.

[0043] According to one aspect of the present application, a computer program product includes computer-readable instructions, which are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the joint angle estimation method as described above.

[0044] The beneficial effects of the technical solution provided by this application are:

[0045] In the above technical solution, a joint angle estimation model is obtained based on training of the training set to achieve joint angle estimation for the detection target, and the joint angle estimation model is calibrated so that the joint angle estimation model can estimate the joint angles for actions that do not appear in the training set, thereby improving the accuracy of joint angle estimation and the generalization of the model. This can effectively solve the problem of low generalization performance of the joint angle estimation method in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0047] Figure 1 It is a schematic diagram of the implementation environment involved in this application;

[0048] Figure 2 is a flow chart of a joint angle estimation method according to an exemplary embodiment;

[0049] Figure 3 is a flow chart of a joint angle estimation method according to another exemplary embodiment;

[0050] Figure 4 is a flowchart of generating a training set according to an exemplary embodiment;

[0051] Figure 5 yes Figure 2 A flowchart of step 230 in one embodiment corresponding to the embodiment;

[0052] Figure 6 is a structural diagram of a joint angle estimation model according to an exemplary embodiment;

[0053] Figure 7 yes Figure 2 A flowchart of an embodiment corresponding to step 270 in an embodiment;

[0054] Figure 8 yes Figure 7 A flowchart of an embodiment corresponding to step 273 in an embodiment;

[0055] Fig. 9 is a structural diagram of a densely connected residual module according to an exemplary embodiment;

[0056] Fig.10 is a structural block diagram of a joint angle estimation method and device according to an exemplary embodiment;

[0057] Fig.11 The present invention is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0058] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0059] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure refers to the presence of the features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0060] As mentioned above, the transfer learning method based on domain adaptation in the prior art is more susceptible to individual differences because physiological and motor signals are more susceptible to individual differences. When the differences between individuals are large, the general domain adaptation method may not achieve satisfactory results. Therefore, the joint angle recognition method based on domain adaptation cannot adapt to the cross-user environment and has low versatility. Moreover, the domain adaptation method's requirement for unlabeled data of the target user during the offline training phase will also cause inconvenience to users in practical applications. For the transfer learning method based on domain generalization, under the limitation of the data set size, the end-to-end trained neural network model is difficult to get rid of the influence of the source domain data distribution. When the individual differences are large in the cross-user environment, the domain generalization method may not be able to obtain the ideal generalization effect. The training process is time-consuming, user acceptance is low, and generalization is low.

[0061] From the above, it can be seen that the relevant technology still has the defects of poor efficiency and poor interaction effect of the joint angle estimation method.

[0062] To this end, the joint angle estimation method provided in the present application can effectively improve the accuracy of the joint angle estimation method. Accordingly, the joint angle estimation method is applicable to a joint angle estimation method device, and the joint angle estimation method device can be deployed in an electronic device, which can be a computer device configured with a von Neumann architecture, for example, the computer device includes a desktop computer, a laptop computer, a server, etc. The electronic device can also be an electronic device with a central control function, for example, the electronic device includes a gateway, etc. The electronic device can also refer to a portable and mobile electronic device, for example, the electronic device includes a smart phone, a tablet computer, etc.

[0063] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0064] Figure 1 It is a schematic diagram of an implementation environment involved in an image processing method. It should be noted that this implementation environment is only an example adapted to the present invention and cannot be considered as providing any limitation on the scope of use of the present invention.

[0065] The implementation environment includes a terminal 110 and a server 130 .

[0066] Specifically, the terminal 110 can also be considered as an interactive device, including but not limited to a PC, an electromyographic signal acquisition device, and other electronic devices with a joint angle estimation function. For example, the terminal 110 is an ultrasonic array tactile stimulation module.

[0067] The server 130 may be an electronic device such as a desktop computer, a laptop computer, a server, etc., or may be a computer cluster composed of multiple servers, or even a cloud computing center composed of multiple servers. The server 130 is used to provide background services, for example, the background services include but are not limited to joint angle estimation services, etc.

[0068] The server 130 and the terminal 110 establish a network communication connection in advance through wired or wireless means, and data transmission between the server 130 and the terminal 110 is realized through the network communication connection. The transmitted data includes but is not limited to: electromyographic signals of the target object, etc.

[0069] In an application scenario, through the interaction between the terminal 110 and the server 130, the terminal 110 captures and collects electromyographic signals of the target object, and uploads the electromyographic signals to the server 130 to request the server 130 to provide joint angle estimation services.

[0070] For the server 130, after receiving the electromyographic signal of the target object uploaded by the terminal 110, it calls the joint angle estimation service, generates a joint angle estimation model, and generates a joint angle estimation result, so as to solve the problem of low generalization performance existing in related technologies.

[0071] See also Figure 2 The embodiment of the present application provides a joint angle estimation method, which is applicable to an electronic device, which may be Figure 1 The server 130 is shown in an implementation environment.

[0072] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0073] like Figure 2 As shown, the method may include the following steps:

[0074] Step 210, acquiring electromyographic signals and joint angle signals for the training target, and generating a training set based on the electromyographic signals and joint angle signals.

[0075] It should be noted that the detection target is an object that provides electromyographic signals for joint angle estimation, and the training target is an object that can provide electromyographic signals and joint angle signals of the same type as the detection target. For example, the training target is one or more subjects, and the subjects complete different actions. During the process of the subjects performing the actions, the electromyographic signals and joint angle signals are synchronously collected from the subjects to generate a training set.

[0076] In a possible implementation, the training set includes multi-channel samples and corresponding joint angle data, wherein the multi-channel samples are segmented electromyographic signals, and the joint angle data are true values ​​of the joint angles corresponding to the multi-channel samples.

[0077] In a possible implementation, a 12-channel electromyography acquisition device is used to record the electromyography signals of hand movements, and a data glove is used to record the hand joint angle signals.

[0078] Step 230, training the base neural network model based on the training set to generate a joint angle estimation model.

[0079] Among them, the basal neural network model is a neural network model that has the ability to predict joint angle data based on electromyographic signals. The basal neural model is trained through a training set to improve the ability of the basal neural network model to predict the corresponding joint angles based on the electromyographic signals in the training set, and finally generate a joint angle estimation model with high prediction accuracy.

[0080] In a possible implementation, the joint angle estimation model may be a DSTrans model.

[0081] Step 250, obtaining new action data.

[0082] The new motion data is the electromyographic signal collected for the detection target. Specifically, the detection target can be an object different from the training target. The electromyographic signal in the new motion data can be acquired by signal acquisition when the detection target performs an action that does not exist in the training set.

[0083] Step 270, calibrating parameters of the joint angle estimation model based on the new motion data, and generating a joint angle estimation result for the detection target through the joint angle estimation model.

[0084] Among them, parameter calibration aligns the different feature domains of the new motion data and the training set in the implicit space through the adversarial transfer learning strategy, so that the joint angle model adapts to the joint angle estimation based on the new motion data, and then generates the joint angle estimation result indicating the detection target joint angle based on the new motion data.

[0085] Through the above process, a joint angle estimation model is obtained based on training of the training set, and joint angle estimation for the detection target is realized. At the same time, by calibrating the parameters of the joint angle estimation model, the joint angle estimation model can estimate the joint angles of actions that do not appear in the training set, thereby improving the accuracy of joint angle estimation and the generalization of the model.

[0086] In an exemplary embodiment, if Figure 3 As shown, after step 210, the method may further include the following steps:

[0087] Step 310: filter the electromyographic signal and the joint angle signal.

[0088] It should be noted that there is noise and other interference in the electromyographic signals and joint angle signals. The quality of the electromyographic signals and joint angle signals is improved by performing targeted filtering on the electromyographic signals and joint angle signals so that the electromyographic signals and joint angle signals can meet the training requirements.

[0089] In a possible implementation, a fourth-order Butterworth filter is used to perform bandpass filtering on the electromyographic signal to perform baseline correction and remove surface electromyographic signal noise, and then a notch filter is used to remove power frequency interference in the electromyographic signal.

[0090] In one possible implementation, a 2 Hz zero-phase low-pass filter is used to smooth the joint angle signal to avoid step jitter in the signal.

[0091] Step 330 , performing time synchronization processing on the joint angle signal and the electromyographic signal to synchronize the timing of the joint angle signal and the electromyographic signal.

[0092] Among them, the joint angle signal and the electromyographic signal are generated synchronously in the process of the training target performing the action. Therefore, by performing time synchronization processing such as resampling the joint angle signal and the electromyographic signal, the collected electromyographic signal and the joint angle signal have a mapping relationship in time sequence.

[0093] In a possible implementation, resampling is performed after the joint angle signal is acquired, wherein the resampling frequency is the same as the electromyographic signal sampling frequency.

[0094] In one possible implementation, a mapping relationship between electromyographic signals and joint angle signals is established based on a deep separable convolution and a Transformer neural network.

[0095] Step 350 , performing action stage selection on the electromyographic signal and the joint angle signal based on the preset action stage, so as to determine the action stages corresponding to the electromyographic signal and the joint angle signal in terms of timing.

[0096] It can be understood that when the training target performs an action, each action can be divided into different stages. For example, when the training target performs a hand movement action, each hand movement action can be divided into a sequence of four stages: grasp - hold - release - rest.

[0097] It should be noted that at this time, the various motion stages can be screened for the main focus of joint angle estimation and the various action stages corresponding to the electromyographic signals and joint angle signals in timing can be determined. For example, only the grasping and releasing stages where there is an important electromyographic joint mapping relationship between the electromyographic signals and the joint angle signals are determined, while the holding and resting stages are abandoned, so that the training process can focus on the grasping and releasing stages, improving the training effect and training efficiency.

[0098] In one possible implementation, the maximum and minimum values ​​of the electromyographic data and the joint angle data are collected, and after the action stage selection is performed on the electromyographic signals and the joint angle signals, normalization processing is performed to improve the quality of the training set and improve the operation efficiency.

[0099] Through the above process, the quality of the electromyographic signals and joint angle signals is improved by performing multiple processing on the electromyographic signals and joint angle signals, thereby ensuring the training effect and training quality of the subsequent model training process.

[0100] In an exemplary embodiment, if Figure 4 As shown, the process of generating a training set may include the following steps:

[0101] Step 410: perform signal segmentation processing on the electromyographic signal to obtain a multi-channel sample including at least two sample windows.

[0102] The signal segmentation process is to segment the electromyographic signal into multiple multi-channel samples through a sliding window.

[0103] Step 430 , generating joint angle data corresponding to the multi-channel sample based on the last joint angle value corresponding to the sample window in the multi-channel sample.

[0104] Among them, after the electromyographic signal is divided into multiple multi-channel samples, the joint angle vector dimension of the corresponding joint angle signal represents the number of joint angles corresponding to the sample window to determine the value of each joint angle in the sample window, and the last joint angle value in each sample window is used as the true value of the corresponding multi-channel sample to generate a training set.

[0105] Through the above process, a training set is generated based on electromyographic signals and joint angle signals to ensure the model training process. The correspondence between multi-channel samples and joint angle data is determined through signal segmentation processing to improve the accuracy of the training set data and ensure the model training effect.

[0106] In an exemplary embodiment, if Figure 5 As shown, step 230 may include the following steps:

[0107] Step 231 , predicting the joint angle values ​​of the multi-channel samples in the training set based on the base neural network model to obtain joint angle prediction data.

[0108] Specifically, after the multi-channel samples are input into the base neural network model, the base neural network model predicts the joint angle values ​​corresponding to the multi-channel samples, and then obtains the joint angle prediction data.

[0109] In an exemplary embodiment, the base neural network model includes a feature extraction module, a multi-layer perceptron module and a nonlinear activation module, wherein the feature extraction module includes a downsampling unit and a deep separable attention unit.

[0110] In one possible implementation, the base neural network model consists of four feature extraction modules, a multi-layer perceptron and a nonlinear activation module. Each feature extraction model contains a downsampling part and a deep separable attention part. The number of deep separable attention modules in each feature extraction module is 2-2-6-2 respectively. The feature extraction module is composed of convolution blocks or self-attention mechanisms.

[0111] Step 233, calculating the loss value between the joint angle prediction data and the joint angle data corresponding to the multi-channel samples based on the loss function.

[0112] The joint angle data corresponding to the multi-channel samples are taken as the true value and compared with the joint angle prediction data to quantify the error generated by the base neural network model as a loss value.

[0113] In one possible implementation, the loss function is a supervised mean square loss function.

[0114] Step 235, updating the model parameters of the base neural network model based on the loss value until the loss function converges, stopping the training, and obtaining a multi-user multi-action neural network model.

[0115] During the training process, the training effect of the base neural network model is judged by whether the loss function converges. For example, when the loss value no longer decreases steadily, that is, the loss function converges, the base neural network model has the strongest prediction ability for joint angle data. At this time, the training is completed and a multi-user multi-action neural network model is obtained.

[0116] Through the above process, the base neural network model is iterated in the direction of improving performance through training, so that the performance of the base neural network model can be improved through continuous iteration, thereby achieving the maximum effect of predicting joint angle data.

[0117] In an exemplary embodiment, if Figure 7 As shown, step 270 may include the following steps:

[0118] Step 271, initializing the base neural network model based on the model parameters of the multi-user multi-action neural network model.

[0119] It should be noted that if Figure 6 As shown in FIG. 1 , the joint angle estimation model includes a multi-user multi-action neural network model, a new user new action neural network model and a feature domain discriminator. The structure of the new user new action neural network model is exactly the same as that of the multi-user multi-action neural network model. The model parameters of the trained multi-user multi-action neural network model are used for initialization, that is, the initial parameters of the new user new action neural network model are set to be consistent with the parameters of the trained multi-user multi-action model.

[0120] Step 273, adjusting the parameters of the base neural network model based on the adversarial transfer learning strategy to obtain a new user new action neural network model.

[0121] In an exemplary embodiment, if Figure 8 As shown, step 273 may include the following steps:

[0122] Step 2731, extracting action features from the training set based on the multi-user multi-action neural network model to obtain first user feature data.

[0123] Step 2733, extracting action features from the new action data based on the base neural network model to obtain second user feature data.

[0124] Step 2735: Calculate the feature distance between the first user feature data and the second user feature data based on the feature domain discriminator.

[0125] For details, please refer to Figure 6After the multi-user multi-action model extracts features from the training set and the new user new action neural network model extracts features from the new action data, three representative feature data from the multi-user neural network model (indicated by red arrows) and three representative feature data from the new user neural network model (indicated by blue arrows) are respectively sent to the feature region discriminator for feature distance calculation. The feature region discriminator is used to calculate and minimize the feature distance between the first user feature data and the second user feature data during the model parameter adjustment process.

[0126] Among them, Fig. 9 As shown, the main component of the feature region discriminator is the densely connected residual module.

[0127] The densely connected residual module consists of a convolutional layer, a densely connected layer, a transition layer, and a residual connection. The convolutional layer is used to extract features, while the densely connected layer is a direct connection between convolutional layers. After extracting features through the convolutional layer, the features extracted by each convolutional layer are concatenated as the input of each subsequent convolutional layer to improve the information propagation within the neural network, thereby accelerating the convergence of the model. The transition layer in the densely connected residual module uses 1×1 convolution to compress the number of features and compress the network. The residual connection structure in the densely connected residual module gives the network a deeper structure and a richer feature representation.

[0128] Step 2737, adjust the model parameters of the base neural network model based on the feature distance until the feature distance converges, fix the model parameters and stop training to obtain a new user new action neural network model.

[0129] The parameters of the multi-user multi-action neural network model are frozen, and the parameters of the new user new action neural network model and the feature domain discriminator are corrected and optimized. The back propagation algorithm is used to adjust each model parameter in the new user new action neural network model and the feature domain discriminator, and new action data is continuously input for training. The training process of the new user new action neural network model and the feature domain discriminator is realized by repeated iterations. When the feature domain discriminator has difficulty distinguishing the feature data of the first user from the feature data of the second user and cannot find the minimum distance, the cross-model knowledge transfer is completed to obtain the new user new action neural network model.

[0130] The cross-user model transfer learning process is trained using the following formulas 1-5:

[0131] Among them, L total It is the total loss function used to optimize the new user and new action neural network model and feature region discriminator.

[0132] L total =L D +LN #(1)

[0133] L D Represents the loss function of the feature region discriminator, which is used to express the relative distance of the feature domain between the multi-user multi-action neural network model and the new user new action neural network model.

[0134] L D = -log(DFD(F s ))-log(1-DFD(F t ))#(2)

[0135] where F s Represents the characteristics of the multi-user multi-action neural network model, F t Represents the characteristics of the neural network model for new user and new action. N Represents the loss function of the neural network model for new user and new action.

[0136] L N =L map +L subject #(3)

[0137] Where L map Indicates the degree of mapping from the features of the multi-user multi-action neural network model to the features of the new user new action neural network model.

[0138] L map = -log(DFD(F t ))#(4)

[0139] L subject It is used to improve the performance of the loss function and make the new user and new action neural network model more sensitive to the characteristics of the new action data during the transfer learning process.

[0140]

[0141] Where Nnet(x) represents the joint angle estimated by the model, represents the label of the joint angle, N represents the number of joints, and α is a weighting coefficient.

[0142] In the above process, adversarial training is performed between the new user and new action neural network model and the feature domain discriminator to continuously improve the ability of the feature domain discriminator to find the minimum distance and the ability of the new user and new action neural network model to predict the joint angle value of the electromyographic signal in the new action data. In the training process, multi-source domain loss is used to perform back propagation correction on the model, which improves the accuracy and generalization of joint angle estimation. The domain discriminator composed of dense residual connections can more effectively extract feature information for cross-action joint angle estimation, and the deep separable attention module can more effectively extract domain-invariant features, reduce the interference of domain-related features on joint angle estimation, and improve the model prediction accuracy.

[0143] Step 275, predicting the joint angle value of the new action data based on the new user's new action neural network model to obtain a joint angle estimation result.

[0144] After the training is completed, the new user's new action neural network model can estimate the joint angles of the joints of the detection target during continuous movement in real time based on the new action data, and generate joint angle estimation results.

[0145] Through the above process, when estimating joint angles based on new action data, there is no need to retrain the model, the model deployment time is short, and the application efficiency is high. The constructed model uses deep separable convolution and attention mechanism for feature extraction. Deep separable convolution can extract spatial feature information. The attention mechanism can help the model more effectively focus on and extract important spatiotemporal features. When the data distribution between individuals is very different, it can more effectively extract the relevant features between the source domain action and the new action in the target domain, improve the generalization ability of the model, and achieve accurate recognition of new actions. By learning the information of the source domain background and the new action of the target new user to correct the parameters, when the data distribution of the source domain background user and the new action of the new user is very different, it can effectively extract features that have little to do with individual differences and action differences, and finally improve the accuracy of joint angle estimation of the general model.

[0146] In an exemplary embodiment, in an application scenario of a joint angle estimation method, data of 40 users are collected, of which 32 subjects are used as training targets, each subject performs 34 actions, and each action is repeated 8 times to build a multi-user multi-action model. The remaining 8 subjects are used as detection targets, each subject performs 6 actions, and each action is repeated 8 times as the target domain data set to adjust the model parameters, and the remaining 4 repetitions are used for model performance testing.

[0147] The surface electromyographic signals from the extensors, flexors, biceps, triceps and a circle of muscles 2 to 6 cm from the elbow of the forearm are collected at a sampling frequency of 2000 Hz. After the collection is completed, the electromyographic signals are processed, and a fourth-order Butterworth filter with a frequency of 5 to 450 Hz is used to perform bandpass filtering on the electromyographic signals for baseline correction and removal of surface electromyographic signal noise. After that, notch filtering is used to remove power frequency interference in the surface electromyographic signals. After that, the electromyographic signals are selected for action stages, which include grasping and releasing. The selected electromyographic signals are normalized and calculated to improve the calculation efficiency.

[0148] The hand joint angles are collected through data gloves with a sampling frequency of 20Hz. After the collection is completed, the joint angle signals are processed, first resampled to 2000Hz. Then a 2Hz zero-phase low-pass filter is used to smooth the original joint angle signals. The maximum and minimum values ​​of the collected electromyographic data and joint angle data are further recorded to normalize the training and test data. The joint angle signals are subjected to action stage selection, which includes two stages: grasping and releasing.

[0149] A sliding window with a width of 112 ms was used, increasing by 0.5 ms each time, to split the EMG signal into more multi-channel samples. The last joint angle value in each sample window was used as the true value to generate training data.

[0150] The DSTrans model is trained based on the training data and optimized using a supervised mean square loss function.

[0151] The parameters of the trained multi-user multi-action neural network model are used to initialize the new user new action model.

[0152] In the process of transfer learning, the parameters of the multi-user multi-action neural network model are partially frozen, while the parameters of the new user new action neural network model and the feature domain discriminator are corrected and optimized to minimize the feature distance between the multi-user feature domain and the new user feature domain. When the feature domain discriminator has difficulty distinguishing the two feature domains and finding the minimum distance, the new user new action neural network model is obtained.

[0153] The new user's new action neural network model is used to estimate the joint angles of the new action data collected by the detection target and generate the joint angle estimation results.

[0154] In this application scenario, the performance of the joint angle estimation model is shown in the following table:

[0155]

[0156] The table shows the regression performance evaluation of different migration model algorithms in the same hand multi-joint angle estimation task. By inputting the test data set, the continuously estimated joint angle curve is compared with the actual joint angle curve obtained by the joint angle sensor. The evaluation indicators in the table include Pearson correlation coefficient (CC), root mean square error (RMSE), determination coefficient (R2), mean absolute error (MAE) and mean absolute percentage error (MAPE) as the evaluation criteria for this regression task.

[0157] The details are as follows:

[0158] 1) Pearson correlation coefficient (CC). CC is a statistic used to quantify the strength and direction of the linear relationship between two continuous variables, measuring how closely the data points in a scatter plot are around a straight line. It can show how close the continuous estimate of hand kinematics is to the corresponding true value. The calculation formula of CC is as follows:

[0159]

[0160] Among them, the value range of CC is between -1 and 1. The closer the CC value is to 1, the stronger the correlation is, the closer the predicted finger motion trajectory is to the actual trajectory, and the higher the estimation accuracy of the method is.

[0161] 2) Root mean square error (RMSE). RMSE is a common indicator for evaluating the accuracy of prediction models. It is usually used to measure the average error between the continuous estimated values ​​of hand kinematics and the corresponding true values. The smaller the value, the better the performance. The calculation formula of RMSE is as follows:

[0162]

[0163] 3) Coefficient of determination (R2). R2 is a statistic used to evaluate the degree of fit of a regression model. As a comprehensive evaluation indicator for measuring the overall accuracy of the model, R2 ranges from 0 to 1 and is defined as the percentage of variation in the true value explained by the estimated value. The larger the R2 value, the better the performance of the estimate. The formula for the coefficient of determination is as follows:

[0164]

[0165] 4) Mean absolute error (MAE). Calculate the average of the absolute values ​​of the differences between the predicted values ​​and the true values. Similar to mean square error, but the mean absolute error considers the absolute value of the error and focuses more on the magnitude of the error rather than the square of the error. The smaller the value of MAE, the smaller the prediction error of the model, indicating that the performance of the model is better. The calculation method of MAE is as follows:

[0166]

[0167] 5) Mean Absolute Percent Error (MAPE). It is used to measure the percentage difference between the model prediction value and the actual observation value. MAPE is usually presented as a percentage. The lower the MAPE value, the smaller the prediction error of the model for the sample, and the better the model performance. The calculation formula of MAPE is as follows:

[0168]

[0169] In the above formula, N represents the sample size, p i Represents the sample points of the predicted finger joint angles, g i Sample points representing actual finger joint angles.

[0170] From the above process, it can be seen that the transfer learning algorithm used in this application not only brings higher accuracy to the hand joint angle estimation of new users' new actions, but also has better generalization performance for new users.

[0171] The following is an embodiment of the device of the present application, which can be used to execute the joint angle estimation method involved in the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the method embodiment of the joint angle estimation method involved in the present application.

[0172] See also Fig.10 In an embodiment of the present application, a joint angle estimation device 1000 is provided, including but not limited to: a signal acquisition module 1010, a training module 1030, a data acquisition module 1050 and a result generation module 1070.

[0173] Among them, the signal acquisition module 1010 is used to acquire electromyographic signals and joint angle signals for training targets, and generate a training set based on the electromyographic signals and joint angle signals.

[0174] The training module 1030 is used to train the base neural network model based on the training set to generate a joint angle estimation model.

[0175] The data acquisition module 1050 is used to acquire new motion data, wherein the new motion data is the electromyographic signal collected for the detection target.

[0176] The result generation module 1070 is used to calibrate the parameters of the joint angle estimation model based on the new motion data, and generate the joint angle estimation result for the detection target through the joint angle estimation model.

[0177] It should be noted that, when the joint angle estimation method device provided in the above embodiment performs the joint angle estimation method, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the joint angle estimation method device will be divided into different functional modules to complete all or part of the functions described above.

[0178] In addition, the joint angle estimation method device provided in the above embodiment and the embodiment of the joint angle estimation method belong to the same concept, and the specific manner in which each module performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0179] See also Fig.11 In an embodiment of the present application, an electronic device 4000 is provided, and the electronic device 400 may include: a desktop computer, a laptop computer, a server, etc.

[0180] exist Fig.11 In the embodiment, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0181] The data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0182] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0183] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0184] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 400, but is not limited to this.

[0185] Computer-readable instructions are stored in the memory 4003 , and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002 .

[0186] The computer-readable instructions are executed by one or more processors 4001 to implement the joint angle estimation method in the above-mentioned embodiments.

[0187] In addition, an embodiment of the present application provides a storage medium on which computer-readable instructions are stored. The computer-readable instructions are executed by one or more processors to implement the above-mentioned joint angle estimation method.

[0188] A computer program product is provided in an embodiment of the present application. The computer program product includes computer-readable instructions. The computer-readable instructions are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the above-mentioned joint angle estimation method.

[0189] Compared with the related art, the present application realizes obtaining a joint angle estimation model based on training set training, realizes joint angle estimation for detection target, and calibrates the parameters of the joint angle estimation model so that the joint angle estimation model can estimate joint angles for actions that do not appear in the training set, thereby improving the accuracy of joint angle estimation and model generalization. By performing multiple processing on electromyographic signals and joint angle signals, the quality of electromyographic signals and joint angle signals is improved, and the training effect and training quality of the subsequent model training process are guaranteed. A training set is generated based on electromyographic signals and joint angle signals to ensure the model training process, and the corresponding relationship between multi-channel samples and joint angle data is determined through signal segmentation processing, thereby improving the accuracy of training set data and ensuring the model training effect. Through training, the base neural network model is iterated in the direction of improving performance, so that the performance of the base neural network model can be improved through continuous iteration, thereby achieving the maximum effect of predicting joint angle data. Through adversarial training of the new user new action neural network model and the feature domain discriminator, the ability of the feature domain discriminator to find the minimum distance and the ability of the new user new action neural network model to predict the joint angle value of the electromyographic signal in the new action data are continuously improved. In the training process, the multi-source domain loss is used to perform back propagation correction on the model, which improves the accuracy and generalization of joint angle estimation. The domain discriminator composed of dense residual connections can more effectively extract feature information for cross-action joint angle estimation. The deep separable attention module can more effectively extract domain invariant features, reduce the interference of domain-related features on joint angle estimation, and improve the model prediction accuracy. When estimating joint angles based on new action data, there is no need to retrain the model, the model deployment time is short, and the application efficiency is high. The constructed model uses deep separable convolution and attention mechanism for feature extraction. Deep separable convolution can extract spatial feature information. The attention mechanism can help the model more effectively focus on and extract important spatiotemporal features. When the data distribution between individuals is large, the relevant features between the source domain action and the new action in the target domain can be more effectively extracted, the generalization ability of the model can be improved, and the accurate recognition of new actions can be achieved. By learning the information of the source domain background and the new actions of the target new user to correct the parameters, when the data distribution of the source domain background users and the new actions of the new users are very different, features that have little to do with individual differences and action differences can be effectively extracted, ultimately improving the accuracy of joint angle estimation of the general model.

[0190] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0191] The above are only some implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A joint angle estimation method, characterized in that: include: Acquire electromyographic signals and joint angle signals for training targets, and generate a training set based on the electromyographic signals and the joint angle signals; Training the base neural network model based on the training set to generate a joint angle estimation model; Acquire new motion data, wherein the new motion data is an electromyographic signal collected for a detection target; The joint angle estimation model is calibrated based on the new motion data, and a joint angle estimation result for the detection target is generated through the joint angle estimation model.

2. The method according to claim 1, characterized in that After acquiring the electromyographic signal and the joint angle signal for the training target, the method further includes: Filter the electromyographic signal and joint angle signal; Performing time synchronization processing on the joint angle signal and the electromyographic signal to synchronize the timing of the joint angle signal and the electromyographic signal; The action stage selection is performed on the electromyographic signal and the joint angle signal based on the preset action stage to determine the action stages to which the electromyographic signal and the joint angle signal correspond in time sequence.

3. The method according to claim 1, characterized in that The training set includes multi-channel samples and corresponding joint angle data; The generating a training set based on the electromyographic signal and the joint angle signal comprises: Performing signal segmentation processing on the electromyographic signal to obtain a multi-channel sample including at least two sample windows; The joint angle data corresponding to the multi-channel sample is generated based on the last joint angle value corresponding to the sample window in the multi-channel sample.

4. The method according to claim 1, characterized in that The base neural network model includes a feature extraction module, a multi-layer perceptron module and a nonlinear activation module, wherein the feature extraction module includes a downsampling unit and a deep separable attention unit.

5. The method according to claim 1, characterized in that The joint angle estimation model includes a multi-user multi-action neural network model; The training of the base neural network model based on the training set to generate the joint angle estimation model includes: Predicting joint angle values ​​for multi-channel samples in the training set based on the base neural network model to obtain joint angle prediction data; Calculate the loss value between the joint angle prediction data and the joint angle data corresponding to the multi-channel samples based on the loss function; The model parameters of the base neural network model are updated based on the loss value until the loss function converges, and the training is stopped to obtain a multi-user multi-action neural network model.

6. The method according to claim 5, characterized in that The joint angle estimation model includes a multi-user multi-action neural network model and a new user new action neural network model; The step of calibrating parameters of the joint angle estimation model based on the new motion data and generating a joint angle estimation result for the detection target based on the joint angle estimation model includes: Initializing a base neural network model based on model parameters of the multi-user multi-action neural network model; Based on the adversarial transfer learning strategy, the parameters of the base neural network model are adjusted to obtain the neural network model of new users and new actions; Based on the new user's new action neural network model, the joint angle value of the new action data is predicted to obtain a joint angle estimation result.

7. The method according to claim 6, characterized in that The method of adjusting the parameters of the base neural network model based on the adversarial transfer learning strategy to obtain a new user new action neural network model includes: Extracting action features from the training set based on a multi-user multi-action neural network model to obtain first user feature data; Extracting action features from the new action data based on the base neural network model to obtain second user feature data; Calculating a feature distance between the first user feature data and the second user feature data based on a feature domain discriminator; The model parameters of the base neural network model are adjusted based on the feature distance until the feature distance converges, the model parameters are fixed and the training is stopped to obtain a new user and new action neural network model.

8. A method and device for estimating joint angles, characterized in that: include: A signal acquisition module, used to acquire electromyographic signals and joint angle signals for training targets, and generate a training set based on the electromyographic signals and the joint angle signals; A training module, used to train the base neural network model based on the training set to generate a joint angle estimation model; A data acquisition module, used to acquire new motion data, wherein the new motion data is an electromyographic signal collected for a detection target; The result generating module is used to perform parameter calibration on the joint angle estimation model based on the new motion data, and to generate a joint angle estimation result for the detection target through the joint angle estimation model.

9. An electronic device, characterized in that: include: at least one processor and at least one memory, wherein: The memory has computer-readable instructions stored thereon; The computer-readable instructions are executed by one or more of the processors so that the electronic device implements the joint angle estimation method according to any one of claims 1 to 7.

10. A storage medium having computer-readable instructions stored thereon, characterized in that: The computer-readable instructions are executed by one or more processors to implement the joint angle estimation method according to any one of claims 1 to 7.

Images