An action recognition method and system based on a reconstructed adversarial domain adaptation network

Abstract

The application belongs to the field of artificial intelligence and human-computer interaction, and discloses a motion recognition method and system based on a reconstruction adversarial domain adaptation network. The motion recognition method comprises the following steps: acquiring electromyographic signals generated by a living body when performing a motion; performing data processing on the electromyographic signals through a pre-trained reconstruction adversarial domain adaptation network to obtain a motion category corresponding to the electromyographic signals; the reconstruction adversarial domain adaptation network comprises the following parts: a feature extractor, which is used for extracting deep features of the electromyographic signals; a motion classifier, which is used for identifying the motion category according to the deep features; a domain discriminator, which is used for judging the domain attribute of the deep features; and a domain reconstructor, which is used for reconstructing the deep features into electromyographic signals. The application can solve the problem that the performance of the existing electromyographic motion recognition algorithm greatly decreases or even fails when facing unknown new users, thereby seriously affecting the practicability and adaptability of the algorithm.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and human-computer interaction, specifically relating to an action recognition method and system based on reconstructed adversarial domain adaptive networks. Background Technology

[0002] Electromyography (EMG) signals are important bioelectrical signals generated during human muscle activity. They not only reflect the state of muscle activity but also the user's behavioral intentions. Action recognition technology based on EMG signals has been widely used in human-computer interaction due to its strong expressive power, flexibility, naturalness, and lack of environmental sensitivity. EMG action recognition algorithms play a crucial role in this technology, decoding EMG signals, extracting deep features, and identifying specific user action categories based on these features. The performance of the algorithm directly affects the accuracy, reliability, and adaptability of action recognition. However, because EMG signals are non-stationary and nonlinear bioelectrical signals with weak amplitude, low signal-to-noise ratio, and large individual variability, existing EMG action recognition algorithms do not perform well in real-world applications. Therefore, developing highly accurate, reliable, and adaptable EMG action recognition algorithms is of great practical significance for the further development and commercial application of EMG action recognition.

[0003] Chinese patent CN117695069A discloses a motion signal recognition method based on time-frequency concentration features. This patent analyzes the time-frequency relationship diagram of electromyography (EMG) signals, calculates frequency concentration values, and performs normalization processing to establish a time-frequency concentration feature model. This model can simultaneously combine the frequency and time characteristics of surface EMG signals to more comprehensively reflect different muscle states, significantly improving the accuracy of EMG action recognition. Chinese patent CN117892210A discloses an EMG pattern recognition method robust to electrode offset and signal loss. This patent effectively simulates data after electrode offset through interpolation, random translation, and downsampling. Based on this, it randomly performs masking operations on several channels of data according to actual conditions, making the trained EMG recognition model more generalizable and significantly improving the reliability of EMG action recognition.

[0004] While the aforementioned patents improve the accuracy and reliability of electromyography (EMG) motion recognition algorithms, enabling high-precision EMG motion recognition even in non-ideal environments such as electrode displacement, they fail to address the significant performance degradation or even failure of the algorithms when faced with unknown new users. In practical applications, EMG signals exhibit significant individual variability. Influenced by factors such as gender, age, physiological condition, and muscle fatigue level, even when performing the same movement, different individuals will produce EMG signals with different data distribution structures, making it difficult for EMG recognition algorithms to be applied across users. This problem greatly reduces the practicality of the algorithm and severely limits the commercialization of EMG motion recognition technology. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an action recognition method and system based on a reconstructed adversarial domain adaptive network. This method can solve the problem that existing electromyographic action recognition algorithms experience a significant performance drop or even failure when facing unknown new users, which seriously affects the practicality and adaptability of the algorithms.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] An action recognition method based on reconstructed adversarial domain adaptive networks includes:

[0008] Acquire electromyographic signals generated by an organism when performing actions;

[0009] The electromyographic signals are processed by a pre-trained reconstructed adversarial domain adaptation network to obtain the action category corresponding to the electromyographic signals.

[0010] The reconstructed adversarial domain adaptation network includes:

[0011] Feature extractor: used to extract deep features of the electromyographic signal;

[0012] Action classifier: used to identify action categories based on the deep features;

[0013] Domain discriminator: used to determine the domain attributes of the deep features;

[0014] Domain Reconstructor: Used to reconstruct the deep features into the electromyographic signals.

[0015] Preferably, the training process of the reconstructed adversarial domain adaptation network includes:

[0016] Based on the adversarial training strategy, by minimizing and maximizing the objective function of the reconstructed adversarial domain adaptation network, the model parameters of the feature extractor, action classifier, domain discriminator and domain reconstructor of the reconstructed adversarial domain adaptation network are adversarially optimized. This guides the reconstructed adversarial domain adaptation network to reconstruct the source and target domain data, discarding the inherent data distribution features and extracting the domain-invariant properties of the source and target domains, thereby achieving data distribution alignment in the feature domain.

[0017] Preferably, the objective function of the reconstructed adversarial domain adaptation network is:

[0018]

[0019] in, This represents the cross-entropy loss between the predicted action label output by the action classifier and the action category label of the electromyography (EMG) data sample. The binary cross-entropy loss represents the difference between the domain prediction labels output by the domain discriminator and the domain attribute labels of the electromyography data samples. G represents the mean squared error loss between the reconstructed EMG signal data output by the domain reconstructor and the original EMG data, λ and γ represent the weights corresponding to the binary classification cross-entropy loss and mean squared error loss, respectively. f G represents the feature extractor. y G represents the action classifier. d G represents the neighborhood discriminator. r Represents the domain refactorer, x i Represents electromyography data samples, θ f θ represents the model parameters of the feature extractor. y θ represents the model parameters of the action classifier. d θ represents the model parameters of the domain discriminator. r The model parameters of the domain reconstructor are represented by , where i represents the index of the EMG data sample, N represents the total number of EMG data samples, and y represents the index of the EMG data sample. i d represents the predicted action category for the i-th electromyography (EMG) data sample. i Let x represent the domain attribute predicted for the i-th electromyography data sample. i This represents the reconstructed electromyographic signal data of the i-th electromyographic data sample.

[0020] Preferably, the adversarial training strategy optimizes the model parameters of the feature extractor and action classifier by minimizing the objective function of the reconstructed adversarial domain adaptation network. This process is expressed as follows:

[0021]

[0022] in, and This represents the optimized model parameters for the feature extractor and action classifier.

[0023] Preferably, the adversarial training strategy optimizes the model parameters of the domain discriminator and the domain reconstructor by maximizing the objective function of the reconstructed adversarial domain adaptation network. This process is expressed as follows:

[0024]

[0025] in, and This represents the optimized model parameters of the domain discriminator and domain reconstructor.

[0026] Preferably, the feature extractor consists of three convolutional layers and one fully connected layer connected in series. Each convolutional layer is followed by a batch normalization layer and an activation function. The fully connected layer serves as the output layer of the feature extractor, used to output the feature vector extracted from the electromyographic signal. This feature extraction process is represented as f = G. f (x i ;θ f ), where G f x represents the feature extractor. i Let θ represent the i-th electromyography (EMG) data sample. f represents the model parameters of the feature extractor, and f represents the extracted feature vector.

[0027] Preferably, the action classifier consists of two fully connected layers connected in series. The action classifier performs action recognition based on the features extracted by the feature extractor and outputs a prediction of the action category of the current sample. This action classification process can be represented as y = G. y (f;θ y ), where G y Let f represent the action classifier, θ represent the feature vector output by the feature extractor, and θ represent the feature vector output by the feature extractor. y represents the model parameters of the action classifier, and y represents the predicted action category.

[0028] Preferably, the neighborhood discriminator consists of two fully connected layers connected in series. It is responsible for determining whether the features extracted by the feature extractor belong to the source domain or the target domain, and outputs a neighborhood attribute prediction for the current sample. This neighborhood discrimination process can be represented as d = G. d (f;θ d ), where G d Let f represent the domain discriminator, f represent the feature vector output by the feature extractor, and θ represent the domain discriminator. d This represents the model parameters of the domain discriminator, and d represents the predicted domain attribute.

[0029] Preferably, the neighborhood reconstructor consists of one fully connected layer and three deconvolutional layers connected in series. Each deconvolutional layer is followed by a batch normalization layer and an activation function. The neighborhood reconstructor is responsible for reconstructing the features extracted by the feature extractor and outputting the reconstructed electromyographic signal data. This neighborhood reconstruction process can be represented as follows: Among them, G r Let f represent the feature vector output by the feature extractor, and θ represent the domain reconstructor. r This represents the model parameters of the domain refactorer. This represents the reconstructed electromyographic signal data.

[0030] This invention also provides an action recognition system based on a reconstructed adversarial domain adaptation network, used to implement the action recognition method described above. The action recognition system includes:

[0031] Data acquisition unit: used to acquire electromyographic signals generated by an organism when performing actions;

[0032] Recognition unit: used to process the electromyographic signal through a pre-trained reconstructed adversarial domain adaptation network to obtain the action category corresponding to the electromyographic signal.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The reconstructed adversarial domain adaptation network in this invention includes a feature extractor, an action classifier, a domain discriminator, and a domain reconstructor. The domain reconstructor reconstructs the original electromyography (EMG) data from the deep features output by the feature extractor. An adversarial training strategy is used to adversarially optimize the model parameters of the feature extractor, minimizing its reconstruction capability. This guides the feature extractor to discard inherent data distribution features in the EMG signal data and extract common characteristics from both the source and target domains. This enhances the model's adaptability to EMG data from different users, helping it to better generalize to new user groups and achieve high-precision cross-user EMG action recognition. The domain discriminator determines the domain (source or target domain) from which the deep features output by the feature extractor originate. The adversarial training strategy is used to adversarially optimize the model parameters of the feature extractor, minimizing the discriminative capability of the domain discriminator. This guides the feature extractor to continuously reduce the data distribution differences between the source and target domains, extracting domain-invariant features that are difficult to distinguish. This helps the model achieve better data alignment among different users, enhances its generalization ability to new users, and improves its practicality and adaptability. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the reconstructed adversarial domain adaptation network of the present invention;

[0036] Figure 2(a) is a schematic diagram of the recognition accuracy of the reconstructed adversarial domain adaptation network of the present invention and three other electromyographic action recognition algorithms on the Nina-pro DB6 dataset; Figure 2(b) is a schematic diagram of the recognition accuracy of the reconstructed adversarial domain adaptation network of the present invention and three other electromyographic action recognition algorithms on the 3DC dataset. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Please see Figure 1 The action recognition method based on reconstructed adversarial domain adaptive networks of the present invention includes the following steps:

[0039] Step 1: Collect electromyographic (EMG) signals generated by different users when performing different actions, and divide the collected EMG signals into source domain datasets. and target domain dataset Specifically, an electromyography (EMG) acquisition device is worn on the same designated body part of n+m users to collect EMG signals generated when the users perform k types of movements. The EMG signals collected from the n users are then set as the source domain dataset, and the EMG signals collected from the other m users are set as the target domain dataset. For example, n=8, m=4, k=16, and the body part can be selected from the extensor and flexor muscles of the forearm to collect EMG signals generated when the user makes gestures.

[0040] Step 2: Construct a reconstructed adversarial domain adaptation network. This network model consists of a feature extractor, an action classifier, a domain discriminator, and a domain reconstructor, such as... Figure 1 As shown, during model training, these four components work simultaneously: the feature extractor extracts deep features of the electromyography (EMG) signals from the input data; the action classifier parses the action category corresponding to the EMG signals from the deep features; the domain discriminator determines the domain attribute (source domain or target domain) of the EMG signals based on the deep features; and the domain reconstructor reconstructs the deep features into EMG signal data. During model testing, the domain discriminator and domain reconstructor stop working, leaving only the feature extractor and action classifier to identify the EMG signals collected from new users as the corresponding action categories.

[0041] Step 3: Based on the adversarial training strategy, the network model parameters are adversarially optimized by minimizing and maximizing the objective function of the reconstructed adversarial domain adaptation network. Specifically, the model parameters of the reconstructed adversarial domain adaptation network are updated using stochastic gradient descent. During this process, the gradients of the model parameters generated by the domain discriminator and the domain reconstructor are inverted, and the inverted gradients are used to update the model parameters of the feature extractor. This training strategy enables the feature extractor to minimize the discriminative ability of the domain discriminator and the reconstruction ability of the domain reconstructor, guiding it to discard the inherent data distribution features of the original EMG signal, reduce the data distribution differences between the source and target domains, and extract the common domain-invariant properties in the EMG signal. This helps the model achieve better data alignment among different users, enhances the model's generalization ability to new users, and achieves high-precision cross-user EMG action recognition.

[0042] Step 4: Apply the optimized network model to the electromyography (EMG) signal data of new users to perform cross-user action recognition. By comparing the model's prediction results with the actual action categories, the network model is tested and evaluated. In specific implementation, the reconstructed adversarial domain adaptation network is tested and evaluated using EMG signal data from the publicly available datasets Ninapro DB6 and 3DC. At the same time, three different EMG action recognition algorithms reported in journal literature—Convnet, AdaBN, and DANN—are used for comparative experiments. The test results are shown in Figure 2(a) and Figure 2(b). The test results show that the reconstructed adversarial domain adaptation network of this invention exhibits the highest recognition accuracy on both datasets, indicating that the network has excellent cross-user EMG action recognition capabilities.

[0043] Source domain dataset Electromyography data samples containing action category labels Composition, where x i Let y represent the i-th electromyography data sample in the source domain. i N represents the action category label of the i-th electromyography data sample in the source domain. S This indicates the number of source domain samples; in addition, each source domain sample has a domain attribute label d. i =0.

[0044] Target domain dataset Electromyography data samples without action category labels Composition, where x i N represents the i-th electromyography data sample in the target domain. t This indicates the number of target domain samples; in addition, each target domain sample has a domain attribute label d. i =1.

[0045] The feature extractor consists of three convolutional layers and one fully connected layer connected in series. Each convolutional layer is followed by a batch normalization layer and an activation function. The fully connected layer serves as the output layer of the feature extractor, responsible for outputting the feature vector extracted from the raw electromyography signal. This feature extraction process can be represented as f = G. f (x i ;θ f ), where G f x represents the feature extractor. i Represents electromyography data samples, θ f represents the model parameters of the feature extractor, and f represents the extracted feature vector.

[0046] The action classifier consists of two fully connected layers connected in series. It identifies actions based on features extracted by the feature extractor and outputs a prediction of the action category for the current sample. This action classification process can be represented as y = G. y (f;θ y ), where G y Let f represent the action classifier, θ represent the feature vector output by the feature extractor, and θ represent the feature vector output by the feature extractor. y y represents the model parameters of the action classifier, and y represents the predicted action category.

[0047] The domain discriminator consists of two fully connected layers connected in series. It is responsible for determining whether the features extracted by the feature extractor belong to the source domain or the target domain, and outputs a domain attribute prediction for the current sample. This domain discrimination process can be represented as d = G. d (f;θ d ), where G d Let f represent the domain discriminator, f represent the feature vector output by the feature extractor, and θ represent the domain discriminator. d This represents the model parameters of the domain discriminator, and d represents the predicted domain attribute.

[0048] The neighborhood reconstructor consists of one fully connected layer and three deconvolutional layers connected in series. Each deconvolutional layer is followed by a batch normalization layer and an activation function. The neighborhood reconstructor is responsible for reconstructing the features extracted by the feature extractor and outputting the reconstructed electromyographic signal data. This neighborhood reconstruction process can be represented as follows: Among them, G r Let f represent the feature vector output by the feature extractor, and θ represent the domain reconstructor. r This represents the model parameters of the domain refactorer. This represents the reconstructed electromyographic signal data.

[0049] The objective function for reconstructing the adversarial domain adaptation network is:

[0050]

[0051] in, This represents the cross-entropy loss between the predicted action label output by the action classifier and the action category label of the electromyography (EMG) data sample. The binary cross-entropy loss represents the difference between the domain prediction labels output by the domain discriminator and the domain attribute labels of the electromyography data samples. denoted as the mean squared error loss between the reconstructed electromyographic signal data output by the domain reconstructor and the original electromyographic data, and λ and γ represent the weights corresponding to the binary cross-entropy loss and the mean squared error loss.

[0052] The adversarial training strategy optimizes the model parameters of the feature extractor and action classifier by minimizing the objective function of the reconstructed adversarial domain adaptation network. This process is expressed as:

[0053]

[0054] in, and This represents the optimized model parameters for the feature extractor and action classifier. This represents solving for the parameter θ. f and θ y The process of minimizing the objective function E uses the stochastic gradient descent optimization algorithm.

[0055] The adversarial training strategy optimizes the model parameters of the domain discriminator and the domain reconstructor by maximizing the objective function of the reconstructed adversarial domain adaptation network. This process is expressed as follows:

[0056]

[0057] in, and This represents the optimized model parameters of the domain discriminator and domain reconstructor. This represents solving for the parameter θ. d and θ r The process of maximizing the objective function E is achieved by using the stochastic gradient descent optimization algorithm.

[0058] This invention also provides an action recognition system based on a reconstructed adversarial domain adaptation network, used to implement the action recognition method described above. The action recognition system includes:

[0059] Data acquisition unit: used to acquire electromyographic signals generated by an organism when performing actions;

[0060] Recognition unit: used to process the electromyographic signal through a pre-trained reconstructed adversarial domain adaptation network to obtain the action category corresponding to the electromyographic signal.

[0061] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

[0062] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An action recognition method based on a reconstructed adversarial domain adaptive network, characterized in that, include: Acquire electromyographic signals generated by an organism when performing actions; The electromyographic signals are processed by a pre-trained reconstructed adversarial domain adaptation network to obtain the action category corresponding to the electromyographic signals. The reconstructed adversarial domain adaptation network includes: Feature extractor: used to extract deep features of the electromyographic signal; Action classifier: used to identify action categories based on the deep features; Domain discriminator: used to determine the domain attributes of the deep features; Domain Reconstructor: Used to reconstruct the deep features into the electromyographic signals; The training process of the reconstructed adversarial domain adaptation network includes: Based on the adversarial training strategy, by minimizing and maximizing the objective function of the reconstructed adversarial domain adaptation network, the model parameters of the feature extractor, action classifier, domain discriminator and domain reconstructor of the reconstructed adversarial domain adaptation network are adversarially optimized. This guides the reconstructed adversarial domain adaptation network to reconstruct the source and target domain data, discarding the inherent data distribution features and extracting the domain-invariant properties of the source and target domains, thereby achieving data distribution alignment in the feature domain. The objective function of the reconstructed adversarial domain adaptation network is: in, This represents the cross-entropy loss between the predicted action label output by the action classifier and the action category label of the electromyography (EMG) data sample. The binary cross-entropy loss represents the difference between the domain prediction labels output by the domain discriminator and the domain attribute labels of the electromyography data samples. This represents the mean squared error loss between the reconstructed EMG signal data output by the domain reconstructor and the original EMG data. and The weights represent the cross-entropy loss and mean squared error loss for binary classification. Indicates feature extractor, Represents an action classifier. This represents a domain discriminator. Represents the domain refactorer. This represents a sample of electromyography (EMG) data. This represents the model parameters of the feature extractor. This represents the model parameters of the action classifier. This represents the model parameters of the domain discriminator. This represents the model parameters of the domain refactorer. i This indicates the sequence number of the electromyography (EMG) data sample, and N represents the total number of EMG data samples. y i This represents the predicted action category for the i-th electromyography (EMG) data sample. d i This represents the domain attribute predicted for the i-th electromyography (EMG) data sample. x i This represents the reconstructed electromyographic signal data of the i-th electromyographic data sample; The neighborhood reconstructor consists of one fully connected layer and three deconvolutional layers connected in series. Each deconvolutional layer is followed by a batch normalization layer and an activation function. The neighborhood reconstructor is responsible for reconstructing the features extracted by the feature extractor and outputting the reconstructed electromyographic signal data. This neighborhood reconstruction process can be represented as follows: ,in, Represents the domain refactorer. This represents the feature vector output by the feature extractor. This represents the model parameters of the domain refactorer. This represents the reconstructed electromyographic signal data.

2. The action recognition method based on reconstructed adversarial domain adaptive network according to claim 1, characterized in that, The adversarial training strategy optimizes the model parameters of the feature extractor and action classifier by minimizing the objective function of the reconstructed adversarial domain adaptation network. This process is expressed as: in, and This represents the optimized model parameters for the feature extractor and action classifier.

3. The action recognition method based on reconstructed adversarial domain adaptive network according to claim 1, characterized in that, The adversarial training strategy optimizes the model parameters of the domain discriminator and the domain reconstructor by maximizing the objective function of the reconstructed adversarial domain adaptation network. This process is expressed as follows: in, and This represents the optimized model parameters of the domain discriminator and domain reconstructor.

4. The action recognition method based on reconstructed adversarial domain adaptive network according to claim 1, characterized in that, The feature extractor consists of three convolutional layers and one fully connected layer connected in series. Each convolutional layer is followed by a batch normalization layer and an activation function. The fully connected layer serves as the output layer of the feature extractor, outputting the feature vector extracted from the electromyographic signal. This feature extraction process is represented as follows: ,in, Indicates feature extractor, Indicates the first i One electromyography data sample, This represents the model parameters of the feature extractor. This represents the extracted feature vector.

5. The action recognition method based on reconstructed adversarial domain adaptive network according to claim 1, characterized in that, The action classifier consists of two fully connected layers connected in series. The action classifier identifies actions based on features extracted by the feature extractor and outputs a predicted action category for the current sample. This action classification process can be represented as follows: ,in, Represents an action classifier. This represents the feature vector output by the feature extractor. This represents the model parameters of the action classifier. This indicates the predicted action category.

6. The action recognition method based on reconstructed adversarial domain adaptive network according to claim 1, characterized in that, The domain discriminator consists of two fully connected layers connected in series. It is responsible for determining whether the features extracted by the feature extractor belong to the source domain or the target domain, and outputs a domain attribute prediction for the current sample. This domain discrimination process can be represented as follows: ,in, This represents a domain discriminator. This represents the feature vector output by the feature extractor. This represents the model parameters of the domain discriminator. This indicates the domain attribute of the prediction.

7. An action recognition system based on a reconstructed adversarial domain adaptive network, characterized in that, For implementing the action recognition method according to any one of claims 1-6, the action recognition system comprises: Data acquisition unit: used to acquire electromyographic signals generated by an organism when performing actions; Recognition unit: used to process the electromyographic signal through a pre-trained reconstructed adversarial domain adaptation network to obtain the action category corresponding to the electromyographic signal.

Images