Electromyographic signal application method and system based on generative adversarial network

Abstract

The invention provides an electromyographic signal application method and system based on a generative adversarial network. The method comprises the steps that a small number of real electromyographic signals are collected and preprocessed; the method comprises the following steps: improving a WGAN-GP model through a Transform model, and constructing a generator and a discriminator in the WGAN-GP model; the real electromyographic signals and the random noise are utilized to train a WGAN-GP model, and a large number of forged electromyographic signals similar to the real electromyographic signals are generated through the trained WGAN-GP model; and performing deep learning on the real electromyographic signals and the forged electromyographic signals, and identifying the current human body motion intention of the user and predicting the future human body motion intention by using a model obtained by deep learning and the collected real-time electromyographic signals of the user. According to the method, data enhancement is carried out on the collected electromyographic signals based on the improved generative adversarial network, the human body motion intention can be accurately recognized and predicted through the expanded electromyographic signals, and the accuracy and intelligence of electromyographic signal application are improved.

Description

Technical Field

[0001] The present application relates to the technical field of electromyogram signal processing, and particularly to an electromyogram signal application method and system based on a generative adversarial network. Background Art

[0002] Electromyogram (EMG) signals mainly originate from the bioelectrical activities in skeletal muscle fibers and are one-dimensional time series signals that are non-linear and non-stationary. Since EMG signals can reflect the contraction and relaxation states of muscles in real time, they have important application values in various fields such as motion control and rehabilitation medicine. For example, the collected EMG signals are used to monitor the physical condition of users, etc.

[0003] However, there are many challenges in the actual application of EMG signals in related technologies. Limited by various practical factors, the number of EMG signals that can be collected is small, and it is difficult to collect pure EMG signals in a non-ideal environment. Furthermore, there are not enough accurate EMG signals in related technologies to improve the accuracy of algorithms, resulting in the inability to accurately achieve the expected application goals.

[0004] Therefore, how to accurately achieve the target function by applying limited EMG signals has become an urgent problem to be solved currently. Summary of the Invention

[0005] The present application aims to solve at least one of the technical problems in related technologies to some extent.

[0006] To this end, the first object of the present application is to propose an electromyogram signal application method based on a generative adversarial network. This method performs data augmentation on the collected electromyogram signals based on an improved generative adversarial network, and the augmented electromyogram signals can be used to accurately identify and predict human motion intentions, improving the accuracy and intelligence of electromyogram signal applications.

[0007] The second object of the present application is to propose an electromyogram signal application system based on a generative adversarial network.

[0008] The third object of the present application is to propose a non-transitory computer-readable storage medium.

[0009] To achieve the above object, the first aspect of the present application is to propose an electromyogram signal application method based on a generative adversarial network, including the following steps:

[0010] Collect a small amount of real electromyogram signals and preprocess the real electromyogram signals;

[0011] Improve the Generative Adversarial Network with Gradient Penalty (WGAN-GP) through the Transformer model, and construct the generator and discriminator in the WGAN-GP model. Among them, the generator and the discriminator contain Transformer layers, and the Transformer layers are used to construct the global dependencies of the electromyogram signals.

[0012] Use the preprocessed real electromyogram signals and random noise to train the constructed WGAN-GP model, and generate a large number of forged electromyogram signals similar to the real electromyogram signals through the trained WGAN-GP model.

[0013] Perform deep learning on the real electromyogram signals and the forged electromyogram signals, and collect the user's real-time electromyogram signals. Use the model obtained by deep learning and the real-time electromyogram signals to identify the user's current human motion intention and predict the human motion intention in the future time period.

[0014] Optionally, in an embodiment of the present application, after performing deep learning on the real electromyogram signals and the forged electromyogram signals, it further includes: integrating the model obtained by deep learning into a wearable exoskeleton device, and collecting the real-time electromyogram signals through the exoskeleton device; after identifying the user's current human motion intention and predicting the human motion intention in the future time period, it further includes: controlling the exoskeleton device to move synchronously with the human body or move in advance according to the current human motion intention and the predicted human motion intention.

[0015] Optionally, in an embodiment of the present application, for using the model obtained by deep learning and the real-time electromyogram signals to identify the user's current human motion intention and predict the human motion intention in the future time period, it includes: extracting features and classifying the real-time electromyogram signals through a deep learning model to identify the current human motion intention; constructing a regression model reflecting the relationship between the electromyogram signals and the human joint angles based on feature classification, inputting the real-time electromyogram signals into the regression model, and obtaining the human motion intention in the future time period predicted by the regression model.

[0016] Optionally, in an embodiment of the present application, the generator includes: a first input layer, a fully connected layer, a first convolutional upsampling layer, a first Transformer layer, a frequency domain filter layer, and a first output layer; wherein, the first input layer is configured to receive the random noise and the class label in the real EMG signal; the fully connected layer is configured to convert the input data into data with a standard dimension; the first convolutional upsampling layer includes multiple upsampling convolutional operation layers, and each upsampling convolutional operation layer includes a spectral normalization function, a residual function, and an activation function, and the first convolutional upsampling layer is configured to gradually increase the signal length; the first Transformer layer includes multiple Transformer modules, and each Transformer module includes a multi-head self-attention layer; the frequency domain filter layer is configured to filter the generated signal in the frequency domain to smooth the generated signal; the first output layer is configured to generate a target signal.

[0017] Optionally, in an embodiment of the present application, the discriminator includes: a second input layer, a second convolutional upsampling layer, a second Transformer layer, and a second output layer; wherein, the second input layer is configured to randomly input the real EMG signal or the forged signal generated by the generator; the second convolutional upsampling layer has the same function as the first convolutional upsampling layer, and the second Transformer layer has the same function as the first Transformer layer; the second output layer is configured to output a score for the current signal, and the score is used to distinguish whether the current signal is a real EMG signal or a forged signal.

[0018] Optionally, in an embodiment of the present application, training the constructed WGAN-GP model includes: inputting the random noise and the class label into the generator to generate multiple forged signals; calculating the discriminator loss according to the score output by the discriminator, and updating the parameters of the discriminator based on the discriminator loss; controlling the generator to regenerate forged signals according to the score output by the discriminator, and calculating the generator loss according to the regenerated forged signals; calculating the frequency domain loss according to the difference in frequency distribution between the real EMG signal and the forged signal; recording the discriminator loss, the generator loss, and the frequency domain loss, and updating the parameters of the generator according to the recorded multiple losses.

[0019] Optionally, in an embodiment of the present application, after preprocessing the real EMG signal, it further includes: performing data augmentation processing on the real EMG signal through various data augmentation methods, and training the WGAN-GP model using the enhanced real EMG signal.

[0020] To achieve the above object, a second aspect of the present application further provides an electromyogram signal application system based on a generative adversarial network, including the following modules:

[0021] An acquisition module, configured to acquire a small amount of real electromyogram signals and preprocess the real electromyogram signals;

[0022] A construction module, configured to improve the generative adversarial network WGAN-GP with gradient penalty through a Transformer model, and construct a generator and a discriminator in the WGAN-GP model, wherein the generator and the discriminator include Transformer layers, and the Transformer layers are used to construct the global dependencies of electromyogram signals;

[0023] A generation module, configured to use the preprocessed real electromyogram signals and random noise to train the constructed WGAN-GP model, and generate a large number of forged electromyogram signals similar to the real electromyogram signals through the trained WGAN-GP model;

[0024] An application module, configured to perform deep learning on the real electromyogram signals and the forged electromyogram signals, acquire the real-time electromyogram signals of the user, and use the model obtained by deep learning and the real-time electromyogram signals to identify the current human motion intention of the user and predict the human motion intention in the future time period.

[0025] To implement the above embodiments, a third aspect embodiment of the present application further provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the electromyogram signal application method based on a generative adversarial network in the first aspect above.

[0026] The technical solutions provided by the embodiments of the present application at least bring the following beneficial effects: Based on the improved generative adversarial network, the present application performs data augmentation on the collected electromyogram (EMG) signals. After collecting only a small amount of EMG signals, diverse and high-quality forged EMG signal data can be generated based on the real signals through the optimized GAN algorithm. Thus, the present application can expand the amount of EMG signal data, simulate and generate learnable EMG signals when the data volume is insufficient, enriching the training data for deep learning. And it ensures the accuracy of the generated EMG signals, reduces the impact of individual differences on the application of EMG signals, and can also protect the privacy of user data. Furthermore, the present application applies the enhanced EMG signals for deep learning, and accurate recognition and prediction of human motion intentions can be achieved using sufficient training data. Thus, it is possible to control the exoskeleton device to perform real-time synchronous motion with the human body, improving the human-computer interaction ability and intelligent level of the exoskeleton device. Therefore, the present application only needs to collect a small amount of real signals to meet the application requirements of EMG signals, enriches the application scenarios of EMG signals, is easy to implement in practical applications, and improves the accuracy, applicability, and intelligence of EMG signal applications.

[0027] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of the embodiments in conjunction with the accompanying drawings, where:

[0029] Figure 1 is a flowchart of a method for applying EMG signals based on a generative adversarial network proposed by an embodiment of the present application;

[0030] Figure 2 is a schematic diagram of the principle of processing EMG signals through a generative adversarial network proposed by an embodiment of the present application;

[0031] Figure 3 is a schematic diagram of the structure of a specific improved generative adversarial network proposed by an embodiment of the present application;

[0032] Figure 4 is a schematic diagram of the structure of a system for applying EMG signals based on a generative adversarial network proposed by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] Embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are intended to explain the present invention, and should not be construed as limiting the present invention.

[0034] It should be noted that when using deep learning technology to learn EMG signals and then identify and predict human motion intentions, but the training data volume is insufficient, EMG signals that can be learned can be simulated and generated based on this application, and then the expanded EMG signals can be used to implement applications such as human motion intention identification and prediction.

[0035] Among them, human muscles are mainly divided into two types: skeletal muscles and smooth muscles. Skeletal muscles are formed by myosin to form muscle fibers, and they undertake most of the movements of the human body. The action potential of the human body mainly transmits information through sodium, calcium, and potassium ions passing through the cell membrane. Every time the muscle contracts and relaxes, it will cause ion movement, thereby generating an electric potential. The superposition of all electrode signals is the EMG signal. That is, the EMG signal mainly comes from the bioelectric activity in skeletal muscle fibers and can reflect the contraction and relaxation state of the muscle in real time. Therefore, different functions in the fields of motion control, rehabilitation medicine, and exoskeleton control systems can be realized by applying EMG signals.

[0036] For example, in the industrial field, some workers often need to perform heavy and dangerous physical labor, such as lifting heavy objects and carrying goods, etc., so they have to maintain the same posture for a long time. By using the method of deep learning of EMG signals on some wearable devices, the physical condition of workers can be effectively monitored, preventing workers from suffering from muscle fatigue or action mistakes, thereby reducing the incidence of work injuries and improving work efficiency. Another example is that in the medical field, the EMG signals after deep learning can be used to monitor the physical condition of patients and predict the physical condition of patients in real time or even in advance, so as to prevent accidents from happening to patients. In addition, in order to reduce the labor intensity of workers and ensure their safety, and to assist some patients with limited mobility in sports medical rehabilitation, etc., wearable exoskeleton devices are becoming an important auxiliary tool in the industrial and medical fields. Further, by accurately monitoring the EMG signals of users, the traditional human-machine interaction method of people moving first and then driving the machine, or people controlling the device to move first and then performing the movement can be abandoned, but the exoskeleton device can sense the motion intention of workers in real time and provide more intelligent assistance and support.

[0037] However, in practical applications, there are many challenges in performing deep learning on EMG signals. First of all, various factors in the non-ideal environment make it relatively difficult to collect pure EMG signals. Weak EMG signals are often affected by various external factors, such as temperature, humidity, and noise, as well as factors such as electrocardiogram signals, individual differences, and muscle fatigue in the human body, which greatly affect the accuracy and stability of collecting EMG signals. If the quality of EMG signals is poor, it will fundamentally affect the accuracy of the discrimination algorithm.

[0038] Secondly, the collection of EMG signals is usually limited by factors such as equipment, time, and personnel. Often, only a limited number of data samples can be collected in the initial stage of the experiment. This makes it lack sufficient training data to improve the accuracy of the algorithm when performing deep learning on EMG signals, resulting in the algorithm being unable to achieve accurate action recognition and control. Finally, as a type of biometric information, EMG signals have a high degree of privacy. The signal data of the previous test subjects involves sensitive information such as personal physical conditions and movement patterns, making data sharing very difficult and further reducing the amount of EMG signal data that can be obtained.

[0039] Traditional EMG signal enhancement methods often rely on simple data transformations, such as adding noise, scaling, and translation. However, these methods are difficult to generate high-quality and diverse EMG signal data and cannot guarantee the consistency of the generated data and the real data in terms of distribution. Therefore, there is an urgent need for an EMG signal data enhancement method that can generate diverse and high-quality forged signals based on real signals, expand the data volume, reduce the influence of individual differences, and at the same time ensure the privacy of the data.

[0040] For this reason, this application proposes an electromyogram signal application method and system based on a generative adversarial network. The generative adversarial network (Generative adversarial network, abbreviated as GAN) is a deep learning model used to generate data similar to real data. Currently, in the field of physiological signals, there is a lack of applications using the GAN model for data enhancement. This application first optimizes the GAN model, so that the optimized GAN model can be used to learn and generate EMG signals, expanding the data volume of EMG signals. Then, the expanded EMG signals are used to achieve relevant application goals.

[0041] Next, a method and system for applying electromyogram signals based on a generative adversarial network proposed in the embodiments of this application will be described with reference to the accompanying drawings.

[0042] Figure 1 It is a flowchart of a method for applying electromyogram signals based on a generative adversarial network proposed in the embodiments of this application. As Figure 1 shown, the method includes the following steps:

[0043] Step S101, collect a small amount of real EMG signals and preprocess the real EMG signals.

[0044] Specifically, first perform initial real signal collection. The EMG signal collection method in related technologies can be used to actually collect a small amount of real EMG signals.

[0045] As an example, a professional EMG signal sensor (such as collecting signals with the Delsys surface EMG acquisition system, etc.) can be used to collect data of an appropriate number of people according to actual needs to obtain accurate and pure EMG signals. The specific number of signals collected is determined according to actual factors such as actual collection conditions and requirements of EMG signal applications.

[0046] Furthermore, preprocess the collected real EMG signals. Since EMG signals have the characteristics of being weak and low-frequency, the values of EMG signals are often small. To ensure the accuracy of subsequent EMG signal applications, before processing the signals through the generative adversarial network combined with the Transformer model in this application, relevant pre-measurements need to be performed on the collected real EMG signals to improve the effectiveness of the collected EMG signals.

[0047] For example, the preprocessing operations can include: high-frequency filtering (such as, a high-order low-pass Butterworth filter can be selected), noise reduction (such as, wavelet denoising, adaptive filtering, etc. can be selected for noise reduction), normalization (the minimum-maximum normalization method between -1 and 1 can be selected), windowing (such as, Hamming window, Hanning window, rectangular window, etc. can be selected), and endpoint detection, etc., so as to ensure the effective frequency of the collected real EMG signals is 0 - 500 Hz through various preprocessing operations, remove the interference signals in the collected signals, and enhance the effective data in the collected signals.

[0048] Step S102, improve the generative adversarial network WGAN-GP with gradient penalty through the Transformer model, and construct the generator and discriminator in the WGAN-GP model.

[0049] Among them, the generator and discriminator contain Transformer layers, and the Transformer layers are used to construct the global dependence relationship of EMG signals.

[0050] To more clearly illustrate the implementation method of improving the generative adversarial network in this application, the generative adversarial network involved in this application will be described in detail below.

[0051] The GAN model consists of a generator and a discriminator, which is a new framework for generative models. It realizes the learning of data distribution through the adversarial process between the generator and the discriminator, getting rid of the complex probability calculations and approximate inferences in traditional generative models. High-quality generation can be achieved only through the adversarial training of two neural networks. Mathematically, the game between the generator and the discriminator is a minimax objective. During the training process, the generator and the discriminator are alternately optimized until the discriminator can no longer distinguish between the generated data and the real data.

[0052] To solve the convergence problem of the traditional GAN model, the Wasserstein distance is adopted in related technologies to measure the difference between the generated distribution and the real distribution, and gradient penalty is introduced to ensure the 1-Lipschitz continuity of the discriminator, thus generating the Wasserstein GAN with gradient penalty (WGAN-GP) model involved in this application. This model improves the training stability and convergence and enhances the quality of the generated samples.

[0053] Among them, the Wasserstein distance is a method for measuring the difference between two probability distributions, that is, the minimum cost required to transform one distribution into another. In the WGAN-GP model, the generator and the discriminator are trained by minimizing the Wasserstein distance. This method is more stable during the optimization process. Especially when the difference between the generated distribution and the real distribution is large, the gradient is still meaningful, thus avoiding the training instability in traditional GANs. After introducing the Wasserstein distance, the discriminator needs to satisfy that the output function is Lipschitz continuous, that is, the norm of its gradient cannot exceed 1. This means that the output of the discriminator must remain somewhat smooth when the input changes. To achieve this, weight clipping is usually required for the discriminator, that is, when updating the discriminator each time, the weights of the discriminator are forced to remain within a specified range. This helps to ensure that the discriminator satisfies Lipschitz continuity and thus stabilizes the training process. However, this operation may limit the ability of the discriminator, making it only able to learn simple functions, resulting in the failure of the model to capture the high-order statistical features of the data and even leading to gradient explosion or disappearance, thereby affecting the generation effect. Therefore, to solve this problem, the WGAN-GP model introduces gradient penalty, by constraining the gradient of the discriminator, that is, adding an additional penalty term to constrain the L2 norm of the gradient of the discriminator at the interpolation points between the generated samples and the real samples to be close to 1. This ensures that the discriminator maintains a reasonable gradient magnitude during the optimization process, thus avoiding unstable phenomena during the training process. Finally, the complete loss function of WGAN-GP is shown in the following formula:

[0054]

[0055] Among them, E represents expectation, D represents discriminator, x represents real data, λ represents penalty coefficient, represents linearly interpolated data, represents the expectation of the generated data distribution, represents the expectation of the real data distribution, represents the expectation of the interpolated data distribution, represents the gradient of the discriminator with respect to the interpolated data.

[0056] When this application processes EMG signals using the WGAN-GP model, as Figure 2 shown, the task of the generator is to generate forged data similar to the real EMG signals. Through adversarial learning with the discriminator, the generator gradually optimizes its parameters so that the generated signals can be close to the real signals in terms of morphology, amplitude, time domain, and frequency domain, thereby effectively supplementing minority-class data and improving the generalization ability of the model, providing diverse and high-quality sample data for the subsequent training of the model. The generator is constructed using a multi-layer neural network structure, including an input layer, a hidden layer, and an output layer. The input layer is used to receive random noise vectors, which serve as the initial input of the generator for generating diverse signals. The hidden layer gradually performs non-linear transformation on the input data through a convolutional neural network.

[0057] The task of the discriminator is to distinguish whether the input signal comes from real data or forged data generated by the generator by learning the feature distributions of real data and generated data. During the training process, the discriminator plays a game with the generator, calculates the Wasserstein distance between real data and generated data, and helps the generator optimize the quality of the generated signals by giving higher scores to real data and lower scores to generated data. The discriminator is constructed using a multi-layer neural network, including an input layer, a hidden layer, and an output layer. The input layer is used to receive EMG signals, which may be real EMG signals or signals forged by the generator. The hidden layer gradually extracts the global features of the signals, calculates the gradient of the discriminator through a gradient penalty mechanism, and constrains the gradient. The output layer outputs a scalar value as the score for the current input data. The higher the score, the closer the data is to the real data distribution; the lower the score, the more likely the data is generated data.

[0058] Furthermore, this application also introduces a Transformer model to improve the above WGAN-GP model, so as to further optimize the EMG signal data generated by the generator and enhance the temporal characteristics of the generated signals. Among them, the Transformer model performs deep learning based on the self-attention mechanism and is an architecture including an encoder and a decoder, which has strong advantages in capturing global relationships in the input sequence. For EMG signals, this application uses the self-attention mechanism of the Transformer model to capture global features and model the global dependencies of EMG signals, which is beneficial for this application to process long-term EMG signals.

[0059] Thus, by improving the WGAN-GP model through the self-attention mechanism of the Transformer, this application can capture the long-term dependencies and local patterns in EMG signals, enabling the generator to generate signals that more accurately conform to the real distribution, and improving the quality and diversity of the generated data. Using the multi-head attention mechanism of the Transformer encoder, by performing attention calculations in parallel through multiple heads, different features in the signal can be simultaneously focused on from multiple perspectives, capturing the global relationships between different time steps in the signal, thereby improving the accuracy and diversity of the generated signals. Therefore, this application effectively solves the deficiencies of traditional generators in time series modeling and further improves the quality and application effect of the generated EMG signals.

[0060] As a possible implementation, the WGAN-GP model constructed by this application in combination with the Transformer model is as Figure 3 shown. When constructing this model, the Transformer layer in the model can be constructed first. The embedded features are processed through multiple stacked Transformer encoders. The multi-head self-attention mechanism is used to capture the global dependencies between different time steps of the signal. When calculating the query matrix Q and the key-value matrix K, convolution operations are performed by adjusting the size of the convolution kernel, ensuring the flexibility of the model. A feed-forward network is used to enhance the feature expression ability of each time step. Residual connections and normalization are used to improve training stability and accelerate convergence. Ensure that the output feature dimension remains consistent with the embedding dimension.

[0061] Furthermore, based on the above improvement method, the generator and discriminator of the WGAN-GP model are constructed. In an embodiment of this application, the constructed generator includes: a first input layer, a fully connected layer, a first convolutional upsampling layer, a first Transformer layer, a frequency domain filter layer, and a first output layer.

[0062] Specifically, the first input layer is used to receive random noise and class labels in real EMG signals. In this application, the noise vector is used as the main input to the generator, representing the random noise in the latent space. The input data received by the first input layer is added as needed. If it is necessary to add the labels in the real EMG signals, the labels need to be mapped to the same dimension as the noise vector through the embedding layer shown in Figure 3 to obtain the label embedding vector. Then, conditioning is performed through the element-wise multiplication of the noise vector and the label embedding vector, enabling the generator to generate signals of specific categories according to the input labels.

[0063] The fully connected layer is used to convert the input data into data of a standard dimension. The fully connected layer can correspond to the Figure 3 hidden layer shown in. First, the input data is expanded to the dimension required for the operation using the fully connected Dense layer, and then the expanded data is reshaped into a three-dimensional tensor through the Reshape operation to serve as the starting point for subsequent convolutional operations.

[0064] The first convolutional upsampling layer includes multiple upsampling convolutional operation layers. Each upsampling convolutional operation layer includes a spectral normalization function, a residual function, and an activation function. The first convolutional upsampling layer is used to gradually increase the signal length.

[0065] Among them, according to the amount of data to be processed, the first convolutional upsampling layer performs convolutional or deconvolutional operations on the data using multiple upsampling convolutional operation layers. At least three upsampling convolutional operation layers are required. Each layer is added with a spectral normalization function (SpectralNormalization) and a residual function as needed, and as shown in Figure 3 it is also added with a LeakyReLU activation function. The first convolutional upsampling layer gradually increases the signal length by setting different strides until the target signal length is reached.

[0066] The first Transformer layer includes multiple Transformer modules. Each Transformer module includes a multi-head self-attention layer. The first Transformer layer uses multiple layers of Transformer modules (at least 3 layers). Each layer sets the multi-head self-attention layer, and the head size and the number of heads are set as needed. As described above, in this application, the first Transformer layer can enhance the modeling ability of the generator, enabling it to capture long-range dependencies of the signals and improve the generation quality.

[0067] The frequency domain filter layer is used to filter the generated signal in the frequency domain to smooth the generated signal. The frequency domain filter layer can be custom-set. Due to the non-linear characteristics of EMG signals, in this application, the frequency domain filter can ensure the smoothness and authenticity of the generated signal.

[0068] The first output layer is used to generate the target signal. As the last layer of the generator, the first output layer is a Conv1D layer with spectral normalization, responsible for generating the final target signal. This final target signal can be the forged electromyogram signal for deep learning applications generated after the model training is completed, or the forged signal input to the discriminator during the training process.

[0069] In an embodiment of the present application, the constructed discriminator includes: a second input layer, a second convolutional upsampling layer, a second Transformer layer, and a second output layer.

[0070] Specifically, the second input layer is used to randomly input the collected real electromyogram signal or the forged signal generated by the generator.

[0071] The function of the second convolutional upsampling layer is the same as that of the first convolutional upsampling layer, and the function of the second Transformer layer is the same as that of the first Transformer layer. That is, the second convolutional upsampling layer also performs convolution or deconvolution operations on the data using multiple upsampling convolutional operation layers according to the amount of data to be processed. At least three upsampling convolutional operation layers are required, and a spectral normalization function and a residual function are added to each layer as needed, and as Figure 3 shown, a LeakyReLU activation function is also added. The second convolutional upsampling layer gradually increases the signal length by setting different strides until the target signal length is reached. The second Transformer layer also includes multiple Transformer modules, and each Transformer module includes a multi-head self-attention layer. The second Transformer layer uses multiple layers of Transformer modules (at least 3 layers), and a multi-head self-attention layer is set for each layer, and the head size and the number of heads are set as needed. In the present application, the second Transformer layer can enhance the modeling ability of the discriminator, enabling it to capture the long-range dependence relationship of the signal and improving the quality of the finally generated forged electromyogram signal.

[0072] The second output layer is used to output a score for the current signal, where the score is used to distinguish whether the current signal is a real electromyogram signal or a forged signal. As the last layer of the discriminator, the second output layer flattens the data features and outputs a scalar value through a fully connected layer. As described above, this scalar represents the true or false score of the current signal.

[0073] Based on the above embodiment, in order to further improve the training effect of the subsequent WGAN-GP model, in an embodiment of the present application, after preprocessing the real electromyogram signal, it further includes: performing data augmentation processing on the real electromyogram signal through various data augmentation methods, and using the enhanced real electromyogram signal to train the WGAN-GP model.

[0074] Specifically, in this embodiment, a data augmentation module can also be constructed. The data augmentation module can be set before the above-mentioned first input layer. When the amount of data of the collected real EMG signals is small, various data augmentation methods such as adding noise, data jitter, time-domain translation, and frequency-domain translation can be selected to perform data augmentation on the collected signals, so as to enrich the amount of real data and improve the training effect.

[0075] Step S103: Use the preprocessed real EMG signals and random noise to train the constructed WGAN-GP model, and generate a large number of forged EMG signals similar to the real EMG signals through the trained WGAN-GP model.

[0076] Specifically, train the constructed WGAN-GP model in the previous step so that the trained WGAN-GP model can generate diverse and high-quality forged EMG signal data. The specific model training process can refer to the training method of the WGAN-GP model in related technologies.

[0077] In an embodiment of the present application, training the constructed WGAN-GP model includes the following steps:

[0078] The first step: Input random noise and class labels into the generator to generate multiple forged signals. This step controls the generator to generate false signals. First, obtain the signals and the class labels corresponding to each signal from the collected real data. Then generate random noise with a normal distribution (other noise distributions can also be selected according to needs). Then, input the random noise and labels into the generator to generate a batch of false signals.

[0079] The second step: Calculate the discriminator loss according to the scores output by the discriminator, and update the parameters of the discriminator based on the discriminator loss. This step calculates the discriminator loss. First, the discriminator outputs the scores of the real signals and false signals calculated. Then, calculate the Wasserstein loss of the discriminator according to the scores. Here, the goal is to maximize the real signal score and minimize the false signal score. Then calculate the gradient penalty term according to the result to ensure the 1-Lipschitz condition of the discriminator. Then update the discriminator parameters.

[0080] The third step: Control the generator to regenerate forged signals according to the scores output by the discriminator, and calculate the generator loss according to the regenerated forged signals. This step calculates the generator loss. First, according to the result output by the discriminator, control the generator to regenerate false signals. Then, calculate the Wasserstein loss of the generator according to the regenerated signals and the initially generated signals. The goal of this step is to maximize the score of the false signals and make them as close as possible to the real signals.

[0081] Step 4: Calculate the frequency-domain loss based on the differences in frequency distribution between the real EMG signals and the forged signals. In this step, the frequency-domain loss is calculated, which is used to measure the difference in frequency distribution between the real signals and the generated forged signals, ensuring that the generated forged signals are not only realistic in the time domain but also consistent in the frequency domain. Further, the EMA mechanism can be used to dynamically adjust the weight of the frequency-domain loss, making it vary smoothly during training, thereby avoiding unstable effects on the training of the generator caused by the frequency-domain loss.

[0082] Step 5: Record the discriminator loss, generator loss, and frequency-domain loss, and update the parameters of the generator based on the recorded multiple losses. In this step, the calculated losses are recorded for subsequent on-demand adjustment of the model parameters. Moreover, to update the parameters of the generator, the gradient of the generator can be calculated, and by combining the gradient of the generator and each loss, the parameters of the generator are updated.

[0083] Thus, through the trained WGAN-GP model of this application, learnable EMG signals can be simulated and generated, and the generated forged EMG signals are relatively similar to the real EMG signals, thereby enriching the EMG data volume with diverse and high-quality forged EMG signals.

[0084] Step S104: Perform deep learning on the real EMG signals and the forged EMG signals, collect the user's real-time EMG signals, and use the model obtained from deep learning and the real-time EMG signals to identify the user's current human motion intention and predict the human motion intention in the future time period.

[0085] Specifically, the collected real EMG signals and the generated forged EMG signals are used for the recognition and prediction of human motion intention. This application only collects EMG signal data of a small number of individuals, and then performs learning and generation of the above generative adversarial network on the real data, which can expand the dataset available for deep learning. Then, deep learning is performed on the collected real EMG signals and the generated forged EMG signals, and various types of models obtained are used to identify the current human motion intention and predict the future human motion intention respectively.

[0086] In an embodiment of this application, using the model obtained from deep learning and the real-time EMG signals to identify the user's current human motion intention and predict the human motion intention in the future time period includes: extracting features and classifying the real-time EMG signals through a deep learning model to identify the current human motion intention; constructing a regression model reflecting the relationship between the EMG signals and the human joint angles based on the feature classification, inputting the real-time EMG signals into the regression model, and obtaining the human motion intention predicted by the regression model in the future time period.

[0087] Specifically, the original real EMG signals and the generated forged EMG signals are used as training data, and a corresponding type of deep learning model is trained through deep learning techniques. For example, the trained deep learning models include: a model combining long short-term memory neural network and support vector machine, a model combining deep convolutional neural network and support vector machine, or a Transformer encoder. The trained deep learning model can be determined according to actual application requirements, and the present application does not limit this.

[0088] Furthermore, based on the trained deep learning model, feature extraction and classification of EMG signals are realized, so that the human motion intention can be recognized according to the EMG signals. For example, after real-time collecting the user's current EMG signals, the current EMG signals are input into the trained deep learning model, and the data features in the real-time EMG signals are extracted through this model, and the real-time EMG signals are classified. For example, the EMG signals are classified into signals of skeletal muscles at different parts of the human body, and signals generated by actions such as contraction or relaxation of skeletal muscles, so that the current motion intention of the human body can be judged according to the category of contraction or relaxation of skeletal muscles at different positions of the human body where the real-time EMG signals belong.

[0089] Further, a regression model between EMG signals and human joint angles is established, and the regression accuracy of predicting human motion by EMG signals can be verified by using this regression model, so as to realize the prediction of human motion intention according to EMG signals.

[0090] For example, using the classification results of the above deep learning model for EMG signals, the corresponding relationships between various EMG signals and different human joint angles are determined, that is, according to the classification results of the deep learning model, the joint angles corresponding to the current EMG signals and the actions performed by the joint angles can be determined, such as contraction and relaxation. Combining the above corresponding relationships between EMG signals and actions of different joint angles to construct a regression model, after inputting the real-time EMG signals into the regression model, the human motion intention in the future time period can be predicted. For example, after inputting multiple real-time EMG signals into the regression model, it is determined that the shoulder joint of a certain arm of the human body is relaxed and the elbow joint is relaxed, and then according to the relaxation angles of the shoulder joint and the elbow joint, it is predicted that the arm will complete an unfolding action within a certain number of seconds in the future.

[0091] Based on the above embodiments, the present application can perform various applications according to the recognized current human motion intention of the user and the predicted human motion intention within a future time period. For example, it can control a wearable exoskeleton device. In an embodiment of the present application, after performing deep learning on real EMG signals and forged EMG signals, it further includes: integrating the model obtained by deep learning into the wearable exoskeleton device, and collecting the real-time EMG signals through the exoskeleton device; after recognizing the current human motion intention of the user and predicting the human motion intention within a future time period, it further includes: controlling the exoskeleton device to move synchronously with the human body or move in advance according to the current human motion intention and the predicted human motion intention.

[0092] Specifically, in this embodiment, the improved deep learning model and regression model in the above embodiments, such as the model obtained by performing deep learning on the extended EMG signals, are applied to the wearable exoskeleton device. Then, the sensors on the exoskeleton device are used to collect the current EMG signals of the user in real time. By using the forward-looking characteristic of the EMG signals, when the human body is about to perform some action postures, the exoskeleton can predict the upcoming actions of the human body according to the generated EMG signals, so as to move in advance or simultaneously with the human body.

[0093] For example, in the above example, when it is predicted that the user's arm will open within a certain number of seconds in the future, at the predicted moment when the arm opens, the corresponding arm component of the exoskeleton device can be controlled to open synchronously to perform the arm-opening action simultaneously with the user. Or, the corresponding arm component of the exoskeleton device can be controlled to open immediately at the current moment, so as to be able to control the exoskeleton device to perform the action in advance before the user opens the arm. Thus, this embodiment can assist some patients with limited mobility to perform the required actions, reduce the effort of the user to perform the actions, and improve the user experience of using the exoskeleton device.

[0094] Therefore, the present application uses EMG signals to identify and predict human motion intentions, which can improve the human-computer interaction level and intelligence level of the exoskeleton.

[0095] In summary, the method for applying electromyogram (EMG) signals based on a generative adversarial network in the embodiments of the present application performs data augmentation on the collected EMG signals based on the improved generative adversarial network. After only a small amount of EMG signals are collected, diverse and high-quality forged EMG signal data can be generated based on the real signals through the optimized GAN algorithm. Thus, this method can expand the amount of EMG signal data, simulate and generate learnable EMG signals when the data volume is insufficient, enriching the training data for deep learning. Moreover, it ensures the accuracy of the generated EMG signals, reduces the influence of individual differences on the application of EMG signals, and can also protect the privacy of user data. Furthermore, this method applies the enhanced EMG signals for deep learning, and accurate recognition and prediction of human motion intentions can be achieved using sufficient training data. Thus, it is possible to control the exoskeleton device to perform real-time synchronous motion with the human body, improving the human-computer interaction ability and intelligent level of the exoskeleton device. Therefore, this method only needs to collect a small amount of real signals to meet the application requirements of EMG signals, enriches the application scenarios of EMG signals, is easy to implement in practical applications, and improves the accuracy, applicability, and intelligence of EMG signal applications.

[0096] To implement the above embodiments, the present application also proposes a system for applying EMG signals based on a generative adversarial network. Figure 4 As shown in the structural schematic diagram of a system for applying EMG signals based on a generative adversarial network proposed in the embodiments of the present application, Figure 4 as shown, the system includes:

[0097] An acquisition module 100, configured to acquire a small amount of real EMG signals and preprocess the real EMG signals.

[0098] A construction module 200, configured to improve the generative adversarial network with gradient penalty (WGAN-GP) through a Transformer model, and construct a generator and a discriminator in the WGAN-GP model, wherein the generator and the discriminator include Transformer layers, and the Transformer layers are used to construct the global dependency relationship of EMG signals.

[0099] A generation module 300, configured to use the preprocessed real EMG signals and random noise to train the constructed WGAN-GP model, and generate a large number of forged EMG signals similar to the real EMG signals through the trained WGAN-GP model.

[0100] An application module 400, configured to perform deep learning on the real EMG signals and the forged EMG signals, collect the real-time EMG signals of the user, and use the model obtained by deep learning and the real-time EMG signals to identify the current human motion intention of the user and predict the human motion intention within a future time period.

[0101] Optionally, in an embodiment of the present application, the application module 400 is further configured to: integrate the model obtained by deep learning into a wearable exoskeleton device, and collect real-time electromyography signals through the exoskeleton device; control the exoskeleton device to move synchronously with the human body or move in advance according to the current human motion intention and the predicted human motion intention.

[0102] Optionally, in an embodiment of the present application, the application module 400 is specifically configured to: extract features and classify the real-time electromyography signals through a deep learning model to identify the current human motion intention; construct a regression model reflecting the relationship between the electromyography signals and the human joint angles based on the feature classification, input the real-time electromyography signals into the regression model, and obtain the predicted human motion intention within a future period by the regression model.

[0103] Optionally, in an embodiment of the present application, the generation module 300 is specifically configured to: input random noise and class labels into a generator to generate a plurality of forged signals; calculate a discriminator loss according to the scores output by the discriminator, and update the parameters of the discriminator based on the discriminator loss; control the generator to regenerate forged signals according to the scores output by the discriminator, and calculate a generator loss according to the regenerated forged signals; calculate a frequency domain loss according to the difference in frequency distribution between the real electromyography signals and the forged signals; record the discriminator loss, the generator loss, and the frequency domain loss, and update the parameters of the generator according to the recorded multiple losses.

[0104] Optionally, in an embodiment of the present application, the construction module 200 is further configured to: perform data augmentation processing on the real electromyography signals through a variety of data augmentation methods, and use the augmented real electromyography signals to train a WGAN-GP model.

[0105] It should be noted that the foregoing explanation of the embodiments of the electromyography signal application method based on the generative adversarial network also applies to the system of this embodiment, and will not be elaborated here.

[0106] In summary, the electromyography signal application system based on the generative adversarial network in the embodiments of the present application performs data augmentation on the collected electromyography signals based on the improved generative adversarial network. After only collecting a small amount of EMG signals, it can generate diverse and high-quality forged EMG signal data based on the real signals through the optimized GAN algorithm. Thus, the system can expand the amount of EMG signal data, simulate and generate learnable EMG signals when the data volume is insufficient, enrich the training data for deep learning. And it ensures the accuracy of the generated EMG signals, reduces the influence of individual differences on the application of EMG signals, and can also protect the privacy of user data.

[0107] To implement the above embodiments, the present application also provides a non-transitory computer-readable storage medium storing a computer program, which when executed by a processor, implements the method for applying electromyogram signals based on a generative adversarial network as described in any one of the embodiments of the first aspect above.

[0108] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0109] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of the present application, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0110] Any process or method description in a flowchart or described in other ways herein can be understood to represent a module, segment, or part of code including one or more executable instructions for implementing a customized logical function or process, and the scope of the preferred embodiments of the present application includes additional implementations, where the functions may be executed in a substantially simultaneous manner or in an opposite order according to the functions involved, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0111] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite ordered list of executable instructions for implementing logical functions, and can be embodied specifically in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection portion having one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable medium on which the program can be printed, as the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpretation, or other suitable processing as necessary, and then storing it in a computer memory.

[0112] It should be understood that various parts of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0113] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method of implementing the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0114] In addition, each functional unit in various embodiments of the present application may be integrated into one processing module, may exist separately as individual physical units, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0115] The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present application.

Claims

1. A myoelectric signal application method based on a generative adversarial network, characterized in that: The following steps are involved: Collecting a small amount of real electromyographic signals and preprocessing the real electromyographic signals; The generative adversarial network WGAN-GP with gradient penalty is improved by the Transformer model, and the generator and discriminator in the WGAN-GP model are constructed, wherein the generator and the discriminator contain a Transformer layer, and the Transformer layer is used to construct the global dependency relationship of the electromyographic signal; The constructed WGAN-GP model is trained by using the preprocessed real electromyographic signal and random noise, and a large number of forged electromyographic signals similar to the real electromyographic signal are generated by the trained WGAN-GP model; The real electromyographic signal and the forged electromyographic signal are subjected to deep learning, and the real-time electromyographic signal of the user is collected. The model obtained by deep learning and the real-time electromyographic signal are used to identify the user's current human body movement intention and predict the human body movement intention in the future time period.

2. The method according to claim 1, characterized in that After the deep learning is performed on the real electromyographic signal and the forged electromyographic signal, the method further includes: Integrate the model obtained by deep learning into a wearable exoskeleton device, and collect the real-time electromyographic signal through the exoskeleton device; After identifying the user's current human body movement intention and predicting the human body movement intention in the future period, the method further includes: According to the current human body movement intention and the predicted human body movement intention, the exoskeleton device is controlled to move synchronously with the human body or to move in advance.

3. The method according to claim 1, characterized in that The method of using the model obtained by deep learning and the real-time electromyographic signal to identify the user's current human body movement intention and predict the human body movement intention in the future period includes: Extracting and classifying features of the real-time electromyographic signal through a deep learning model to identify the current human body movement intention; A regression model reflecting the relationship between the electromyographic signal and the human joint angle is constructed based on feature classification, the real-time electromyographic signal is input into the regression model, and the human body movement intention in the future time period predicted by the regression model is obtained.

4. The method according to claim 1, characterized in that The generator comprises: a first input layer, a fully connected layer, a first convolutional upsampling layer, a first Transformer layer, a frequency domain filter layer and a first output layer; wherein, The first input layer is used to receive the category labels in the random noise and the real electromyographic signal; The fully connected layer is used to convert the input data into data of standard dimension; The first convolution upsampling layer includes a plurality of upsampling convolution operation layers, each of the upsampling convolution operation layers includes a spectral normalization function, a residual function and an activation function, and the first convolution upsampling layer is used to gradually increase the signal length; The first Transformer layer includes a plurality of Transformer modules, each of which includes a multi-head self-attention layer; The frequency domain filter layer is used to filter the generated signal in the frequency domain to smooth the generated signal; The first output layer is used to generate a target signal.

5. The method according to claim 4, characterized in that The discriminator comprises: a second input layer, a second convolution upsampling layer, a second Transformer layer and a second output layer; wherein, The second input layer is used to randomly input the real electromyographic signal or the forged signal generated by the generator; The second convolution upsampling layer has the same function as the first convolution upsampling layer, and the second Transformer layer has the same function as the first Transformer layer; The second output layer is used to output a score for the current signal, wherein the score is used to distinguish whether the current signal is a real electromyographic signal or a forged signal.

6. The method according to claim 5, characterized in that The training of the constructed WGAN-GP model includes: Inputting the random noise and the category label into a generator to generate a plurality of forged signals; Calculating a discriminator loss according to the score output by the discriminator, and updating the parameters of the discriminator based on the discriminator loss; Controlling the generator to regenerate a forged signal according to the score output by the discriminator, and calculating the generator loss according to the regenerated forged signal; Calculating frequency domain loss according to the difference in frequency distribution between the real electromyographic signal and the forged signal; The discriminator loss, the generator loss and the frequency domain loss are recorded, and the parameters of the generator are updated according to the recorded multiple losses.

7. The method according to claim 1, characterized in that After preprocessing the real electromyographic signal, the method further includes: The real electromyographic signal is subjected to data enhancement processing by using a variety of data enhancement methods, and the WGAN-GP model is trained using the enhanced real electromyographic signal.

8. A myoelectric signal application system based on generative adversarial network, characterized in that: Includes the following modules: An acquisition module is used to acquire a small amount of real electromyographic signals and pre-process the real electromyographic signals; A construction module, used to improve the generative adversarial network WGAN-GP with gradient penalty through a Transformer model, and to construct a generator and a discriminator in the WGAN-GP model, wherein the generator and the discriminator include a Transformer layer, and the Transformer layer is used to construct a global dependency relationship of electromyographic signals; A generation module, used to train the constructed WGAN-GP model using the pre-processed real electromyographic signal and random noise, and generate a large number of forged electromyographic signals similar to the real electromyographic signal through the trained WGAN-GP model; The application module is used to perform deep learning on the real electromyographic signal and the fake electromyographic signal, collect the user's real-time electromyographic signal, and use the model obtained by deep learning and the real-time electromyographic signal to identify the user's current human movement intention and predict the human movement intention in the future time period.

9. The system according to claim 8, characterized in that The application module is also used for: Integrate the model obtained by deep learning into a wearable exoskeleton device, and collect the real-time electromyographic signal through the exoskeleton device; According to the current human body movement intention and the predicted human body movement intention, the exoskeleton device is controlled to move synchronously with the human body or to move in advance.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the electromyographic signal application method based on a generative adversarial network as described in any one of claims 1 to 7 is implemented.

Images