Emg signal gesture recognition method based on adaptive multi-scale dual-flow feature fusion
By using an adaptive multi-scale dual-stream feature fusion method, the problem of underutilization of temporal and frequency domain features in electromyographic signal gesture recognition is solved, achieving high-precision and highly adaptive gesture recognition and improving the application effect of intelligent prostheses.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHONGQING UNIV OF TECH
- Filing Date
- 2024-11-27
- Publication Date
- 2026-08-04
AI Technical Summary
Existing deep learning methods do not fully utilize temporal and frequency domain features in electromyography signal gesture recognition, especially the insufficient mining of frequency domain information, resulting in low gesture recognition accuracy and difficulty in cross-individual recognition, which limits the widespread application of intelligent prostheses.
An adaptive multi-scale dual-stream feature fusion method is adopted. Through electromyography (EMG) preprocessing, temporal feature processing, frequency domain feature processing, bidirectional attention module, and individual adaptation module, the temporal and frequency domain features of EMG signals are processed respectively. The bidirectional attention module establishes a correlation and combines individual information to dynamically adjust feature fusion, thereby improving cross-individual adaptability.
It significantly improves the precision and robustness of gesture recognition, enhances cross-individual recognition capabilities, and achieves high-precision and highly adaptable gesture recognition results.
Smart Images
Figure CN119670000B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of human physiological signal decoding technology, specifically to a method for electromyographic signal gesture recognition based on adaptive multi-scale dual-stream feature fusion. Background Technology
[0002] Due to the loss or damage of limb function, their daily life and work abilities are greatly limited, and they urgently need assistive devices such as intelligent prostheses to restore their ability to take care of themselves and improve their quality of life.
[0003] Surface electromyography (sEMG) signals are reliable physiological signals for prosthetic hand control. They are characterized by their anticipatory generation, meaning that the EMG signal theoretically occurs 30–150 ms before the movement. Simultaneously, the signal intensity of movements with the same surface EMG signal may differ, but the temporal or frequency domain characteristics of the same location are highly similar. Different movements will exhibit differences in the temporal or frequency domain characteristics of the same location. Upper limb sEMG contains rich information on hand movement intentions and is suitable for upper limb movement pattern recognition.
[0004] sEMG-based gesture recognition methods mainly include traditional machine learning methods and deep learning methods. Machine learning methods are relatively effective for gesture classification with a small number of categories, but their performance declines significantly when the number of categories reaches 40-50. Deep learning methods, on the other hand, significantly improve the accuracy of sEMG gesture recognition due to their powerful data fitting capabilities.
[0005] In existing technologies, sEMG gesture recognition algorithms that construct narrow-kernel convolution and feature fusion are based on exploring higher feature extraction capabilities, achieving substantial breakthroughs in recognition performance. Furthermore, by constructing sEMG gesture recognition algorithms that incorporate multi-scale fusion convolution and channel attention, algorithm complexity is reduced while computational efficiency is improved.
[0006] However, current deep neural networks used for gesture recognition do not fully consider the temporal and frequency domain features of surface electromyography (EMG) signals, especially the insufficient mining of frequency domain information. This leads to inadequate discovery of the correlation between temporal and frequency domain features, thus affecting the accuracy of gesture recognition. At the same time, the significant differences in EMG signals between individuals make cross-individual recognition difficult, which has become a major technical bottleneck restricting the widespread application of intelligent prosthetics.
[0007] Therefore, a new solution is needed to address the above problems. Summary of the Invention
[0008] The purpose of this invention is to provide a gesture recognition method based on adaptive multi-scale dual-stream feature fusion of electromyographic signals to solve the problems mentioned in the background art.
[0009] To achieve the above objectives, the present invention provides the following technical solution: a gesture recognition method based on adaptive multi-scale dual-stream feature fusion of electromyographic signals, comprising at least the following steps:
[0010] S1: Sequentially build the electromyography preprocessing module, temporal feature processing module, frequency domain feature processing module, bidirectional attention module, and individual adaptation module;
[0011] S2: The electromyography preprocessing module is used to segment and perform fast Fourier transform processing on the electromyography signal to generate information for the input time-series feature processing module and the frequency domain feature processing module;
[0012] S3: The temporal feature processing module processes the temporal features of surface electromyography (sEMG) signals to capture the temporal dynamics of the electromyography signals, ensuring that the algorithm has agile response capabilities when recognizing rapid movements or short-term changes.
[0013] S4: The frequency domain processing module is used to process the frequency domain feature information of electromyographic signals. The input of the frequency domain processing module is the data after FFT transformation, and a feature vector is generated. Contrastive loss is used for training to optimize the representation of frequency features.
[0014] S5: A bidirectional attention module is used to integrate the temporal feature processing module and the frequency domain feature processing module. By utilizing the complementarity of these two types of information, the correlation between temporal features and frequency domain features is established.
[0015] S6: An individual adaptive module is used to adapt and adjust the model according to specific subject or individual characteristics to improve personalized performance. That is, the individual adaptive module can dynamically adjust the feature fusion between modalities through individual information to achieve cross-subject adaptability and flexible processing of frequency domain features. The individual information includes at least height, age, weight, gender and left or right hand.
[0016] Furthermore, the application of the electromyography preprocessing module in S2 includes at least the following steps:
[0017] The input electromyographic signal is a 400×12 matrix, representing signal data containing 400 time steps and 12 channels;
[0018] The electromyography preprocessing module performs segmentation and fast Fourier transform operations on the electromyography signals. The segmentation operation divides the original data into a subset of 400×6400×6 for the time-domain feature processing branch.
[0019] The Fast Fourier Transform operation transforms the signal into the frequency domain, resulting in a spectrum with a shape of 400×12-400×12, which is used for the frequency domain feature processing branch.
[0020] Furthermore, the input to the temporal feature processing module is the segmented data, which generates feature vectors for contrastive loss to optimize the similarity between similar samples and the difference between dissimilar samples.
[0021] The temporal feature processing module includes multiple sub-modules, including at least a convolution module, a feature fusion module, a multi-stream CNN module, a Mamba module, a StreamFusion module, and a temporal convolutional network (TCN) module.
[0022] Furthermore, the process of the time-series feature processing module includes at least the following steps:
[0023] Input a pair of data points with 400 time steps and 6 channels;
[0024] The input data is first processed through a convolutional layer with 64 kernels, a kernel size of 3, and a stride of 1, to embed the data from a low-dimensional space to a high-dimensional space.
[0025] The convolutional features are fed into the multi-stream CNN module, which splits the data stream into multiple sub-streams and processes them independently to capture information from different time ranges or feature dimensions.
[0026] After passing through the multi-stream CNN module, the data enters the multi-layered Mamba module and StreamFusion module.
[0027] Features from different substreams are fused together using the StreamFusion module to enhance the expressive power of the features. There are three layers of Mamba modules and StreamFusion modules used alternately to gradually fuse features.
[0028] After multiple Mamba and StreamFusion iterations, the features are fed into the temporal convolutional network module, which provides the ability to capture temporal information of long sequence dependencies and is an effective tool for modeling time series.
[0029] The output of the temporal convolutional network module then passes through a max pooling layer to reduce the feature dimension to 64.
[0030] Finally, the feature vector is further processed by a fully connected network (FNN) to obtain the final feature vector.
[0031] Furthermore, the frequency domain processing module includes at least a fully connected layer and a dual-layer GRU, wherein the dual-layer GRU is a gated loop unit.
[0032] Furthermore, the method by which the frequency domain processing module processes the frequency domain characteristics of a signal includes at least the following steps:
[0033] Input 400 time steps and 12 frequency feature channels;
[0034] The input data first passes through a fully connected layer, which expands the feature dimension from 12 to 64 to capture more frequency features;
[0035] The processed features are fed into a two-layer GRU layer with a dimension of 128 to model the sequence dependencies of frequency features;
[0036] The output of the GRU layer passes through a fully connected layer, and the feature dimension is compressed to 64. The output at this point is a feature vector of size 64.
[0037] Furthermore, the output of the bidirectional attention module is trained with FocalLoss to further optimize the model's performance, especially its performance in handling class imbalance problems.
[0038] The bidirectional attention module contains two feedforward neural networks (FNNs) with an initial feature dimension of 64.
[0039] Furthermore, the processing flow of the bidirectional attention module includes at least the following steps:
[0040] The temporal and frequency domain features are respectively passed through their respective FNN layers to obtain feature representations with a dimension of 64;
[0041] After multiple addition and multiplication operations, it is used for the interaction between different features. The addition operation represents feature fusion, and the multiplication operation is the redistribution of weights, similar to the attention mechanism.
[0042] Multiple addition and multiplication operations are designed to balance and enhance the complementarity of the two branch features in order to make full use of time and frequency features;
[0043] The interacting features are passed through a final FNN layer to generate the final feature vector.
[0044] Compared with the prior art, the beneficial effects of the present invention are:
[0045] 1. In order to fully utilize the modal information in sEMG, this invention designs a dual-stream architecture to process temporal and frequency domain features in parallel. This fully utilizes the multi-dimensional information of sEMG signals. The temporal feature processing module is responsible for capturing the temporal dynamics of the signal, ensuring that the algorithm has agile response capabilities when recognizing rapid movements or short-term changes. The frequency domain feature processing module focuses on the frequency domain features of the signal. By selecting appropriate frequency bands through an adaptive feature fusion module, appropriate frequency bands are selected to extract the muscle physiological features reflected in different frequency bands. This structure enables the algorithm not only to effectively cope with the differences in electromyographic signals between individuals, but also to extract key features from the frequency domain information that help to recognize complex gestures, thus significantly enhancing the precision and robustness of gesture recognition.
[0046] 2. To enhance cross-subject recognition capabilities, this invention introduces a cross-subject adaptive feature fusion module. This module dynamically adjusts feature fusion between modalities based on individual information (height, age, weight, gender, left / right hand, etc.). The algorithm effectively bridges the differences in electromyographic features between individuals using an adaptive mechanism, avoiding the insufficient adaptability problem caused by fixed feature fusion strategies in traditional algorithms. Through adaptive processing of frequency domain features, the algorithm can more fully extract deep physiological feature information from sEMG signals, such as recognizing the strength and rhythm features of specific hand gestures in different frequency bands. This flexibility significantly improves the algorithm's performance in complex gestures and subtle movements, enabling it to achieve high accuracy and high adaptability in cross-individual gesture recognition. Attached Figure Description
[0047] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0048] Figure 1 This is a schematic diagram of the adaptive multi-scale dual-stream feature fusion algorithm of the present invention;
[0049] Figure 2 This is a schematic diagram of the time-series feature processing module structure of the present invention;
[0050] Figure 3 This is a schematic diagram of the frequency domain feature processing module structure of the present invention;
[0051] Figure 4 This is a schematic diagram of the bidirectional attention module structure of the present invention;
[0052] Figure 5 This is a schematic diagram of the individual adaptive module structure of the present invention;
[0053] Figure 6 The loss curves for the training and test sets in a gesture recognition experiment of this invention are shown.
[0054] Figure 7 This is the accuracy curve of the training set and test set in a gesture recognition experiment of the present invention. Detailed Implementation
[0055] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0056] Please see Figure 1 - Figure 5 A gesture recognition method based on adaptive multi-scale dual-stream feature fusion of electromyographic signals includes at least the following steps:
[0057] S1: Sequentially build the electromyography preprocessing module, temporal feature processing module, frequency domain feature processing module, bidirectional attention module, and individual adaptation module;
[0058] S2: The electromyography preprocessing module is used to segment and perform fast Fourier transform processing on the electromyography signal to generate information for the input time-series feature processing module and the frequency domain feature processing module;
[0059] S3: The temporal feature processing module processes the temporal features of surface electromyography (sEMG) signals to capture the temporal dynamics of the electromyography signals, ensuring that the algorithm has agile response capabilities when recognizing rapid movements or short-term changes.
[0060] S4: The frequency domain processing module is used to process the frequency domain feature information of electromyographic signals. The input of the frequency domain processing module is the data after FFT transformation, and it generates feature vectors. Contrastive loss is used for training to optimize the representation of frequency features.
[0061] S5: A bidirectional attention module is used to integrate the temporal feature processing module and the frequency domain feature processing module. By utilizing the complementarity of these two types of information, the correlation between temporal features and frequency domain features is established.
[0062] S6: The individual adaptive module is used to adapt and adjust the model according to specific subject or individual characteristics to improve personalized performance. That is, the individual adaptive module can dynamically adjust the feature fusion between modalities through individual information to achieve cross-subject adaptability and flexible processing of frequency domain features. Individual information includes at least height, age, weight, gender and left or right hand.
[0063] The application of the electromyography preprocessing module in S2 includes at least the following steps:
[0064] The input electromyographic signal is a 400×12 matrix, representing signal data containing 400 time steps and 12 channels;
[0065] The electromyography preprocessing module performs segmentation and fast Fourier transform operations on the electromyography signals. The segmentation operation divides the original data into a subset of 400×6400×6 for the time-domain feature processing branch.
[0066] The Fast Fourier Transform operation transforms the signal into the frequency domain, resulting in a spectrum with a shape of 400×12-400×12, which is used for the frequency domain feature processing branch.
[0067] The input to the temporal feature processing module is the segmented data, which generates feature vectors for contrastive loss, optimizing the similarity between similar samples and the difference between dissimilar samples.
[0068] The temporal feature processing module includes multiple sub-modules, including at least convolution, feature fusion, multi-stream CNN, Mamba, StreamFusion, and Temporal Convolutional Network (TCN) modules.
[0069] The time series feature processing module's workflow includes at least the following steps:
[0070] Input a pair of data points with 400 time steps and 6 channels;
[0071] The input data is first processed through a convolutional layer with 64 kernels, a kernel size of 3, and a stride of 1, to embed the data from a low-dimensional space to a high-dimensional space.
[0072] The convolutional features are fed into the multi-stream CNN module, which splits the data stream into multiple sub-streams and processes them independently to capture information from different time ranges or feature dimensions.
[0073] After passing through the multi-stream CNN module, the data enters the multi-layered Mamba module and StreamFusion module.
[0074] Features from different substreams are fused together using the StreamFusion module to enhance the expressive power of the features. There are three layers of Mamba modules and StreamFusion modules used alternately to gradually fuse features.
[0075] After multiple Mamba and StreamFusion iterations, the features are fed into the temporal convolutional network module, which provides the ability to capture temporal information about long sequence dependencies and is an effective tool for modeling time series.
[0076] The output of the temporal convolutional network module then passes through a max pooling layer to reduce the feature dimension to 64.
[0077] Finally, the feature vector is further processed by a fully connected network (FNN) to obtain the final feature vector.
[0078] The frequency domain processing module includes at least a fully connected layer and a dual-layer GRU, where the dual-layer GRU is a gated cyclic unit.
[0079] The method for a frequency domain processing module to process the frequency domain characteristics of a signal includes at least the following steps:
[0080] Input 400 time steps and 12 frequency feature channels;
[0081] The input data first passes through a fully connected layer, which expands the feature dimension from 12 to 64 to capture more frequency features;
[0082] The processed features are fed into a two-layer GRU layer with a dimension of 128 to model the sequence dependencies of frequency features;
[0083] The output of the GRU layer passes through a fully connected layer, where the feature dimension is compressed to 64, resulting in a feature vector of size 64.
[0084] The output of the bidirectional attention module is trained with FocalLoss to further optimize the model's performance, especially its performance in handling class imbalance problems.
[0085] The bidirectional attention module contains two feedforward neural networks (FNNs) with an initial feature dimension of 64.
[0086] The processing flow of the bidirectional attention module includes at least the following steps:
[0087] The temporal and frequency domain features are respectively passed through their respective FNN layers to obtain feature representations with a dimension of 64;
[0088] After multiple addition and multiplication operations, it is used for the interaction between different features. The addition operation represents feature fusion, and the multiplication operation is the redistribution of weights, similar to the attention mechanism.
[0089] Multiple addition and multiplication operations are designed to balance and enhance the complementarity of the two branch features in order to make full use of time and frequency features;
[0090] The interacting features are passed through a final FNN layer to generate the final feature vector.
[0091] Based on the above embodiments, the following verification is proposed:
[0092] This embodiment uses electromyography (EMG) data from the Ninapro DB2, 3, and 4 public datasets. The EMG acquisition frequency is 2kHz, and the number of acquisition channels is 12. A total of 49 hand gestures from 40, 11, and 10 subjects were collected, and each gesture was repeated 6 times.
[0093] In a single-subject application, the model was trained and tested. Each subject underwent a maximum of 200 iterations. In each iteration, the training set was divided into several batches of size 1024. Each batch was sequentially input into the model to obtain a classification result, which was then compared with the label. The focal loss function was used to calculate the loss, and the Adam optimizer was used for backpropagation to update the trainable parameters in the network. The initial learning rate was set to 0.001. Then, in each iteration, after all the training data had been trained, the test set was input into the lightweight convolutional neural network to calculate the classification accuracy. The highest test set accuracy obtained in the last 50 iterations was the gesture recognition accuracy of the algorithm.
[0094] A 49-class gesture recognition experiment was conducted on the data of subject 22 in the Ninapro DB2 dataset. Figure 6 As can be seen, in the early stages of training, the loss values of both the training and test sets decrease rapidly with the increase of the number of iterations. Subsequently, the rate of decrease in the training set loss value gradually slows down, while the test set loss value begins to converge around the 50th iteration. Figure 7 In the early stages of training, the accuracy of both the training set and the test set increased rapidly with the increase of the number of iterations. Subsequently, the growth rate of the training set accuracy gradually slowed down and approached 100%, while the accuracy of the test set stabilized after about the 50th cycle, but still improved slightly, eventually reaching 89.2%.
[0095] The model of this invention was tested on 40, 11, and 10 subjects respectively on the Ninapro DB2, 3, and 4 datasets to recognize 49 types of gestures. The average accuracy of all subjects was 90.03%, 73.83%, and 88.72%, respectively. The accuracy was 2.00%, 3.25%, and 2.81% higher than the state-of-the-art algorithm. The results of the comparison between the model of this invention and other models are shown in Table 1.
[0096] Table 1: Comparison of the performance of the model of this invention with other models
[0097]
[0098] In cross-subject applications, the model of this invention also achieved excellent recognition results. On the Ninapro DB2, 3 and 4 datasets, 49 types of gesture recognition experiments were conducted on 40, 11 and 10 subjects respectively, and the average accuracy of all subjects was 84.93%, 70.33% and 82.42% respectively.
[0099] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
Claims
1. A gesture recognition method based on adaptive multi-scale dual-stream feature fusion of electromyographic signals, characterized in that: At least the following steps are included: S1: Sequentially build the electromyography preprocessing module, temporal feature processing module, frequency domain feature processing module, bidirectional attention module, and individual adaptation module; S2: The electromyography preprocessing module is used to segment and perform fast Fourier transform processing on the electromyography signal to generate information for the input time-series feature processing module and the frequency domain feature processing module; S3: The temporal feature processing module processes the temporal features of the surface electromyography (EMG) signal to capture the temporal dynamics of the EMG signal, ensuring that the algorithm has agile response capabilities when recognizing rapid movements or short-term changes. S4: The frequency domain processing module is used to process the frequency domain feature information of electromyographic signals. The input of the frequency domain processing module is the data after FFT transformation, and a feature vector is generated. The contrastive loss is used for training to optimize the representation of frequency features. S5: A bidirectional attention module is used to integrate the temporal feature processing module and the frequency domain feature processing module. By utilizing the complementarity of these two types of information, the correlation between temporal features and frequency domain features is established. S6: An individual adaptive module is used to adapt and adjust the model according to specific subject or individual characteristics to improve personalized performance. That is, the individual adaptive module can dynamically adjust the feature fusion between modalities through individual information to achieve cross-subject adaptability and flexible processing of frequency domain features. The individual information includes at least height, age, weight, gender and left or right hand.
2. The electromyographic signal gesture recognition method based on adaptive multi-scale dual-stream feature fusion according to claim 1, characterized in that: The application of the electromyography preprocessing module in S2 includes at least the following steps: The input electromyographic signal is a 400×12 matrix, representing signal data containing 400 time steps and 12 channels; The electromyography preprocessing module performs segmentation and fast Fourier transform operations on the electromyography signals. The segmentation operation divides the original data into a subset of 400×6400×6 for the time-domain feature processing branch. The Fast Fourier Transform operation transforms the signal into the frequency domain, resulting in a spectrum with a shape of 400×12-400×12, which is used for the frequency domain feature processing branch.
3. The electromyographic signal gesture recognition method based on adaptive multi-scale dual-stream feature fusion according to claim 1, characterized in that: The input to the temporal feature processing module is the segmented data, which generates feature vectors for comparison loss, optimizing the similarity between similar samples and the difference between dissimilar samples. The temporal feature processing module includes multiple sub-modules, including at least a convolution module, a feature fusion module, a multi-stream CNN module, a Mamba module, a StreamFusion module, and a temporal convolutional network module.
4. The electromyographic signal gesture recognition method based on adaptive multi-scale dual-stream feature fusion according to claim 3, characterized in that: The process of the time-series feature processing module includes at least the following steps: Input a pair of data points with 400 time steps and 6 channels; The input data is first processed through a convolutional layer with 64 kernels, a kernel size of 3, and a stride of 1, to embed the data from a low-dimensional space to a high-dimensional space. The convolutional features are fed into the multi-stream CNN module, which splits the data stream into multiple sub-streams and processes them independently to capture information from different time ranges or feature dimensions. After passing through the multi-stream CNN module, the data enters the multi-layered Mamba module and StreamFusion module. Features from different substreams are fused together using the StreamFusion module to enhance the expressive power of the features. There are three layers of Mamba modules and StreamFusion modules used alternately to gradually fuse features. After multiple Mamba and StreamFusion iterations, the features are fed into the temporal convolutional network module, which provides the ability to capture temporal information of long sequence dependencies and is an effective tool for modeling time series. The output of the temporal convolutional network module then passes through a max pooling layer to reduce the feature dimension to 64. Finally, the feature vector is further processed by a fully connected network to obtain the final feature vector.
5. The electromyography signal gesture recognition method based on adaptive multi-scale dual-stream feature fusion according to claim 1, characterized in that: The frequency domain processing module includes at least a fully connected layer and a dual-layer GRU, wherein the dual-layer GRU is a gated loop unit.
6. The electromyographic signal gesture recognition method based on adaptive multi-scale dual-stream feature fusion according to claim 5, characterized in that: The method for processing the frequency domain features of a signal by the frequency domain processing module includes at least the following steps: Input 400 time steps and 12 frequency feature channels; The input data first passes through a fully connected layer, which expands the feature dimension from 12 to 64 to capture more frequency features; The processed features are fed into a two-layer GRU layer with a dimension of 128 to model the sequence dependencies of frequency features; The output of the GRU layer passes through a fully connected layer, and the feature dimension is compressed to 64. The output at this point is a feature vector of size 64.
7. The electromyography signal gesture recognition method based on adaptive multi-scale dual-stream feature fusion according to claim 1, characterized in that: The output of the bidirectional attention module is trained with FocalLoss to further optimize the model's performance; The bidirectional attention module contains two feedforward neural networks, with an initial feature dimension of 64.
8. The electromyographic signal gesture recognition method based on adaptive multi-scale dual-stream feature fusion according to claim 7, characterized in that: The processing flow of the bidirectional attention module includes at least the following steps: The temporal and frequency domain features are respectively passed through their respective FNN layers to obtain feature representations with a dimension of 64; Multiple addition and multiplication operations are used for the interaction between different features. The addition operation represents feature fusion, while the multiplication operation is the redistribution of weights. Multiple addition and multiplication operations are designed to balance and enhance the complementarity of the two branch features in order to make full use of time and frequency features; The interacting features are passed through a final FNN layer to generate the final feature vector.