Electromyographic signal gesture recognition method based on adaptive multi-scale double-flow feature fusion

Abstract

The invention discloses an electromyographic signal gesture recognition method based on adaptive multi-scale double-flow feature fusion, and relates to the technical field of human physiological signal decoding. In order to make full use of modal information in sEMG, when a network is designed, a double-flow architecture is designed to carry out parallel processing on time sequence features and frequency domain features, multi-dimensional information of sEMG signals is fully used, a time sequence feature processing module is responsible for capturing the time dynamic state of the signals, and a frequency domain feature processing module is responsible for capturing the frequency domain features of the signals. The method ensures that the algorithm has agile response capability when identifying rapid actions or short-time changes, a frequency domain feature processing module focuses on frequency domain features of signals, and a proper frequency band is selected through a self-adaptive feature fusion module, so that muscle physiological features reflected in different frequency bands are extracted, and the recognition accuracy of the muscle physiological features is improved. According to the structure, the algorithm not only can effectively cope with electromyographic signal differences among individuals, but also can extract key features beneficial to recognition of complex gestures from frequency domain information, so that the fineness and robustness of gesture recognition are remarkably enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of human physiological signal decoding, and in particular to a method for myoelectric signal gesture recognition based on adaptive multi-scale dual-stream feature fusion. Background Art

[0002] Due to the loss or damage of limb function, their daily life and work ability are greatly limited. They urgently need assistive devices such as smart prostheses to restore their ability to take care of themselves and improve their quality of life.

[0003] Surface electromyography (sEMG) is a reliable physiological signal for prosthetic hand control. It has the characteristic of being generated ahead of time, that is, the electromyography is theoretically generated 30 to 150 ms before the action occurs. At the same time, the signal strength of the action under the same surface electromyography may be different, but the time domain or frequency domain characteristics of the same position are highly similar. Under different actions, the time domain or frequency domain characteristics of the same position will show differences. Upper limb sEMG contains rich information on hand movement intentions and is suitable for the recognition of upper limb movement patterns.

[0004] sEMG-based gesture recognition methods mainly include traditional machine learning methods and deep learning methods. Among them, machine learning methods have a more effective effect in gesture classification with a small number of categories, but the classification effect decreases significantly when the number of classification categories is 40-50. Deep learning methods have significantly improved the accuracy of sEMG gesture recognition due to their powerful data fitting ability.

[0005] In the existing technology, by constructing a sEMG gesture recognition algorithm with narrow kernel convolution and feature fusion, a substantial breakthrough in recognition performance has been achieved by exploring higher feature extraction capabilities. By constructing a sEMG gesture recognition algorithm with multi-scale fusion convolution and channel attention, the algorithm complexity is reduced while the computational efficiency is improved.

[0006] However, the deep neural networks currently used for gesture recognition do not fully consider the timing and frequency domain characteristics of surface electromyographic signals, especially the insufficient mining of frequency domain information, which leads to insufficient mining of the correlation between timing and frequency domain features and affects the accuracy of gesture recognition. At the same time, due to the large differences in electromyographic signals between individuals, cross-individual recognition is difficult, which has become an important technical shortcoming that restricts the widespread application of intelligent prostheses.

[0007] Therefore, a new solution to the above problems needs to be proposed. Summary of the invention

[0008] The purpose of the present invention is to provide a method for myoelectric signal gesture recognition based on adaptive multi-scale dual-stream feature fusion to solve the problems raised in the above background technology.

[0009] To achieve the above object, the present invention provides the following technical solution: a method for myoelectric signal gesture recognition based on adaptive multi-scale dual-stream feature fusion, comprising at least the following steps:

[0010] S1: build the electromyography preprocessing module, time series feature processing module, frequency domain feature processing module, bidirectional attention module and individual adaptation module in sequence;

[0011] S2: Use the electromyography preprocessing module to segment and perform fast Fourier transform processing on the electromyography signal to generate information for input into the time series feature processing module and the frequency domain feature processing module;

[0012] S3: The timing feature processing module processes the timing features of the surface electromyographic signal (sEMG) to capture the temporal dynamics of the electromyographic signal and ensure that the algorithm has agile response capabilities when identifying fast movements or short-term changes;

[0013] S4: using a frequency domain processing module to process the frequency domain feature information of the electromyographic signal, wherein the input of the frequency domain processing module is the data after FFT conversion, and generates a feature vector, and uses contrastive loss (ContrastiveLoss) for training to optimize the representation of frequency features;

[0014] S5: A bidirectional attention module is used to realize the fusion of the temporal feature processing module and the frequency domain feature processing module, and the complementarity of these two types of information is used to establish the association between the temporal features and the frequency domain features;

[0015] S6: An individual adaptive module is used to adapt the model according to specific subject or individual characteristics to improve personalized performance, that is, the individual adaptive module can dynamically adjust the feature fusion between modalities through individual information to achieve cross-subject adaptability and flexible processing of frequency domain features. The individual information includes at least height, age, weight, gender, and left and right hands.

[0016] Furthermore, the application of the electromyography preprocessing module in S2 at least includes the following steps:

[0017] The input EMG signal is a 400×12400×12 matrix, which means signal data with 400 time steps and 12 channels;

[0018] The EMG preprocessing module performs segmentation and fast Fourier transform operations on the EMG signals. The segmentation operation divides the original data into subsets of 400×6400×6 for the time domain feature processing branch;

[0019] The fast Fourier transform operation converts the signal into the frequency domain, and the resulting spectrum graph has a shape of 400×12400×12, which is used for the frequency domain feature processing branch.

[0020] Furthermore, the input of the temporal feature processing module is the segmented data, generating a feature vector for contrastive loss to optimize the similarity between similar samples and the difference between heterogeneous samples;

[0021] The temporal feature processing module includes multiple sub-modules, and the sub-modules include at least convolution, feature fusion, multi-stream CNN module, Mamba module, StreamFusion module and temporal convolutional network (TCN) module.

[0022] Furthermore, the process of the timing feature processing module includes at least the following steps:

[0023] Input a pair of data with 400 time steps and 6 channels;

[0024] The input data is first processed by a convolutional layer with 64 convolution kernels, a kernel size of 3, and a step size of 1, to embed the data from a low-dimensional space into a high-dimensional space.

[0025] The convolutional features are passed to the multi-stream CNN module, which splits the data stream into multiple sub-streams and processes them independently to capture information in different time ranges or feature dimensions;

[0026] After passing through the multi-stream CNN module, the data enters the multi-layer Mamba module and StreamFusion module

[0027] The features in different sub-streams are fused together using the StreamFusion module to enhance the expressiveness of the features. A total of three layers of Mamba modules and StreamFusion modules are used alternately to gradually fuse the features.

[0028] After multiple Mamba and StreamFusion passes, the features enter the temporal convolutional network module, which provides the ability to capture temporal information of long sequence dependencies and is an effective tool for modeling time series.

[0029] The output of the temporal convolutional network module then passes through the maximum pooling layer (MaxPooling) to reduce the feature dimension to 64;

[0030] Finally, it is further processed by a fully connected network (FNN) to obtain the final feature vector.

[0031] Furthermore, the frequency domain processing module includes at least a fully connected layer and a double-layer GRU, and the double-layer GRU is a gated recurrent unit.

[0032] Furthermore, the method for the frequency domain processing module to process the frequency domain characteristics of a signal comprises at least the following steps:

[0033] Input 400 time steps and 12 frequency feature channels;

[0034] The input data first passes through a fully connected layer to expand the feature dimension from 12 to 64 to capture more frequency features;

[0035] The processed features are passed into a two-layer GRU layer with a dimension of 128 to model the sequence dependency of frequency features;

[0036] The output of the GRU layer passes through a fully connected layer, and the feature dimension is compressed to 64. At this time, the output is a feature vector of size 64.

[0037] Furthermore, the output of the bidirectional attention module is trained with FocalLoss to further optimize the performance of the model, especially the performance of handling the class imbalance problem;

[0038] The bidirectional attention module contains two feedforward neural networks (FNNs) and the initial dimension of the features is 64.

[0039] Furthermore, the processing flow of the bidirectional attention module includes at least the following steps:

[0040] The time series and frequency domain features pass through their respective FNN layers to obtain a feature representation with a dimension of 64;

[0041] After multiple addition and multiplication operations, it is used for interaction between different features. The addition operation represents feature fusion, and the multiplication operation is the redistribution of weights, which is similar to the attention mechanism.

[0042] Multiple addition and multiplication operations are designed to balance and enhance the complementarity of the two branch features to make full use of the time and frequency features;

[0043] The interacted features pass through a final FNN layer to generate the final feature vector.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. In order to make full use of the modal information in sEMG, the present invention designs a dual-stream architecture to process the timing features and frequency domain features in parallel when designing the network, making full use of the multi-dimensional information of the sEMG signal. The timing feature processing module is responsible for capturing the temporal dynamics of the signal to ensure that the algorithm has agile response capabilities when identifying fast movements or short-time changes. The frequency domain feature processing module focuses on the frequency domain characteristics of the signal, and selects the appropriate frequency band through the adaptive feature fusion module to extract the muscle physiological characteristics reflected in different frequency bands. This structure enables the algorithm to not only effectively cope with the differences in electromyographic signals between individuals, but also extract key features from the frequency domain information that are helpful in identifying complex gestures, thereby significantly enhancing the precision and robustness of gesture recognition.

[0046] 2. In order to improve the recognition ability across subjects, the present invention introduces a cross-subject adaptive feature fusion module, which can dynamically adjust the feature fusion between modalities through individual information (height, age, weight, gender, left and right hands, etc.). The algorithm uses an adaptive mechanism to effectively bridge the differences in electromyographic characteristics between different individuals, avoiding the lack of adaptability caused by fixed feature fusion strategies in traditional algorithms. Through adaptive processing of frequency domain features, the algorithm can more fully extract deep physiological feature information in sEMG signals, such as identifying the strength and rhythm characteristics of specific gestures in different frequency bands. This flexibility greatly improves the performance of the algorithm in complex gestures and subtle movements, so that it can still achieve high precision and high adaptability in cross-individual gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0048] Figure 1 Schematic diagram of the adaptive multi-scale dual-stream feature fusion algorithm of the present invention;

[0049] Figure 2 It is a structural schematic diagram of the timing feature processing module of the present invention;

[0050] Figure 3 This is a schematic diagram of the structure of the frequency domain feature processing module of the present invention;

[0051] Figure 4 Schematic diagram of the structure of the bidirectional attention module of the present invention;

[0052] Figure 5 It is a schematic diagram of the structure of the individual adaptive module of the present invention;

[0053] Figure 6 is a loss value curve of a training set and a test set in a certain gesture recognition experiment of the present invention;

[0054] Figure 7 This is the accuracy curve of the training set and the test set in a gesture recognition experiment of the present invention. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0056] See also Figure 1 - Figure 5 The method for myoelectric signal gesture recognition based on adaptive multi-scale dual-stream feature fusion at least includes the following steps:

[0057] S1: build the electromyography preprocessing module, time series feature processing module, frequency domain feature processing module, bidirectional attention module and individual adaptation module in sequence;

[0058] S2: Use the electromyography preprocessing module to segment and perform fast Fourier transform processing on the electromyography signal to generate information for input into the time series feature processing module and the frequency domain feature processing module;

[0059] S3: The timing feature processing module processes the timing features of the surface electromyographic signal (sEMG) to capture the temporal dynamics of the electromyographic signal and ensure that the algorithm has agile response capabilities when identifying fast movements or short-term changes;

[0060] S4: The frequency domain processing module is used to process the frequency domain feature information of the electromyographic signal. The input of the frequency domain processing module is the data after FFT conversion, and a feature vector is generated. The contrast loss is used for training to optimize the representation of the frequency feature.

[0061] S5: A bidirectional attention module is used to realize the fusion of the temporal feature processing module and the frequency domain feature processing module, and the complementarity of these two types of information is used to establish the association between the temporal features and the frequency domain features;

[0062] S6: Use the individual adaptive module to adapt the model according to the specific subject or individual characteristics to improve personalized performance. That is, the individual adaptive module can dynamically adjust the feature fusion between modalities through individual information to achieve cross-subject adaptability and flexible processing of frequency domain features. The individual information includes at least height, age, weight, gender, and left or right hand.

[0063] The application of the EMG preprocessing module in S2 includes at least the following steps:

[0064] The input EMG signal is a 400×12400×12 matrix, which means signal data with 400 time steps and 12 channels;

[0065] The EMG preprocessing module performs segmentation and fast Fourier transform operations on the EMG signals. The segmentation operation divides the original data into subsets of 400×6400×6 for the time domain feature processing branch;

[0066] The fast Fourier transform operation converts the signal into the frequency domain, and the resulting spectrum graph has a shape of 400×12400×12, which is used for the frequency domain feature processing branch.

[0067] The input of the time series feature processing module is the segmented data, which generates a feature vector for contrastive loss to optimize the similarity between similar samples and the difference between heterogeneous samples;

[0068] The temporal feature processing module includes multiple sub-modules, and the sub-modules include at least convolution, feature fusion, multi-stream CNN module, Mamba module, StreamFusion module and temporal convolutional network (TCN) module.

[0069] The process of the time series feature processing module includes at least the following steps:

[0070] Input a pair of data with 400 time steps and 6 channels;

[0071] The input data is first processed by a convolutional layer with 64 convolution kernels, a kernel size of 3, and a step size of 1, to embed the data from a low-dimensional space into a high-dimensional space.

[0072] The convolutional features are passed to the multi-stream CNN module, which splits the data stream into multiple sub-streams and processes them independently to capture information in different time ranges or feature dimensions;

[0073] After passing through the multi-stream CNN module, the data enters the multi-layer Mamba module and StreamFusion module

[0074] The features in different sub-streams are fused together using the StreamFusion module to enhance the expressiveness of the features. A total of three layers of Mamba modules and StreamFusion modules are used alternately to gradually fuse the features.

[0075] After multiple Mamba and StreamFusion passes, the features enter the temporal convolutional network module, which provides the ability to capture temporal information of long sequence dependencies and is an effective tool for modeling time series.

[0076] The output of the temporal convolutional network module then passes through the maximum pooling layer (MaxPooling) to reduce the feature dimension to 64;

[0077] Finally, it is further processed by a fully connected network (FNN) to obtain the final feature vector.

[0078] The frequency domain processing module includes at least a fully connected layer and a double-layer GRU, and the double-layer GRU is a gated recurrent unit.

[0079] The method for the frequency domain processing module to process the frequency domain characteristics of the signal includes at least the following steps:

[0080] Input 400 time steps and 12 frequency feature channels;

[0081] The input data first passes through a fully connected layer to expand the feature dimension from 12 to 64 to capture more frequency features;

[0082] The processed features are passed into a two-layer GRU layer with a dimension of 128 to model the sequence dependency of frequency features;

[0083] The output of the GRU layer passes through a fully connected layer, and the feature dimension is compressed to 64. The output at this time is a feature vector of size 64.

[0084] The output of the bidirectional attention module is trained with FocalLoss to further optimize the performance of the model, especially the performance of handling class imbalance problems;

[0085] The bidirectional attention module contains two feed-forward neural networks (FNNs) and the initial dimension of features is 64.

[0086] The processing flow of the bidirectional attention module includes at least the following steps:

[0087] The time series and frequency domain features pass through their respective FNN layers to obtain a feature representation with a dimension of 64;

[0088] After multiple addition and multiplication operations, it is used for interaction between different features. The addition operation represents feature fusion, and the multiplication operation is the redistribution of weights, which is similar to the attention mechanism.

[0089] Multiple addition and multiplication operations are designed to balance and enhance the complementarity of the two branch features to make full use of the time and frequency features;

[0090] The interacted features pass through a final FNN layer to generate the final feature vector.

[0091] Based on the above embodiment, the following verification is proposed:

[0092] This embodiment uses the electromyographic data of the Ninapro DB2, 3, and 4 public data sets, with an electromyographic acquisition frequency of 2 kHz and 12 acquisition channels. A total of 49 gestures of 40, 11, and 10 subjects are collected, and each action is repeated 6 times.

[0093] The model was trained and tested in a single-subject application, and each subject was trained for a maximum of 200 iterations. In each iteration, the training set was divided into several batches of 1024 in size. Each batch was input into the model of the present invention in turn to obtain the classification result, and compared with the label. The loss was calculated using the focal loss function (FocalLoss), and then the Adam optimizer was used for back propagation to update the trainable parameters in the network, and the initial learning rate was determined to be 0.001. Then, in each iteration, after all the data in the training set were trained, the test set was input into the lightweight convolutional neural network to calculate the classification accuracy. The highest test set accuracy obtained in the last 50 iterations was the gesture recognition accuracy of the algorithm of the present invention.

[0094] A 49-category gesture recognition experiment was conducted on the 22nd subject data of the Ninapro DB2 dataset. Figure 6 It can be seen that in the early stage of training, as the number of iterations increases, the loss values ​​of the training set and the test set both decrease rapidly. Subsequently, the loss value of the training set gradually slows down, while the loss value of the test set begins to converge at about the 50th cycle. Figure 7 In the early stage of training, as the number of iterations increases, the accuracy of the training set and the test set increases rapidly. Then, the growth rate of the training set accuracy gradually slows down and approaches 100%, while the test set accuracy stabilizes after about the 50th cycle, but still increases slightly, finally reaching 89.2%.

[0095] The model of the present invention conducted 49 types of gesture recognition experiments on 40, 11, and 10 subjects on the Ninapro DB2, 3, and 4 data sets, respectively, and the average accuracy of all subjects was 90.03%, 73.83%, and 88.72%, respectively. The accuracy was 2.00%, 3.25%, and 2.81% higher than that of the most advanced algorithm. The comparison results between the model of the present invention and other models are shown in Table 1.

[0096] Table 1: Comparison of the effects of the model of the present invention and other models

[0097]

[0098] The model of the present invention also achieved excellent recognition results in cross-subject applications. 49 types of gesture recognition experiments were conducted on 40, 11, and 10 subjects on the Ninapro DB2, 3, and 4 datasets, respectively, and the average accuracy rates of all subjects were 84.93%, 70.33%, and 82.42%, respectively.

[0099] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A method for myoelectric signal gesture recognition based on adaptive multi-scale dual-stream feature fusion, characterized by: At least the following steps are included: S1: build the electromyography preprocessing module, time series feature processing module, frequency domain feature processing module, bidirectional attention module and individual adaptation module in sequence; S2: Use the electromyography preprocessing module to segment and perform fast Fourier transform processing on the electromyography signal to generate information for input into the time series feature processing module and the frequency domain feature processing module; S3: The timing characteristics of the surface electromyographic signal are processed through the timing feature processing module to capture the temporal dynamics of the electromyographic signal and ensure that the algorithm has agile response capabilities when identifying fast movements or short-time changes; S4: using a frequency domain processing module to process the frequency domain feature information of the electromyographic signal, wherein the input of the frequency domain processing module is the data after FFT conversion, and generates a feature vector, and uses contrast loss for training to optimize the representation of the frequency feature; S5: A bidirectional attention module is used to realize the fusion of the temporal feature processing module and the frequency domain feature processing module, and the complementarity of these two types of information is used to establish the association between the temporal features and the frequency domain features; S6: An individual adaptive module is used to adapt the model according to specific subject or individual characteristics to improve personalized performance, that is, the individual adaptive module can dynamically adjust the feature fusion between modalities through individual information to achieve cross-subject adaptability and flexible processing of frequency domain features. The individual information includes at least height, age, weight, gender, and left and right hands.

2. The method for myoelectric signal gesture recognition based on adaptive multi-scale dual-stream feature fusion according to claim 1 is characterized in that: The application of the electromyography preprocessing module in S2 at least comprises the following steps: The input EMG signal is a 400×12400×12 matrix, which means signal data with 400 time steps and 12 channels; The EMG preprocessing module performs segmentation and fast Fourier transform operations on the EMG signals. The segmentation operation divides the original data into subsets of 400×6400×6 for the time domain feature processing branch; The fast Fourier transform operation converts the signal into the frequency domain, and the resulting spectrum graph has a shape of 400×12400×12, which is used for the frequency domain feature processing branch.

3. The method for myoelectric signal gesture recognition based on adaptive multi-scale dual-stream feature fusion according to claim 1 is characterized in that: The input of the temporal feature processing module is the segmented data, generating a feature vector for contrast loss, optimizing the similarity between similar samples and the difference between heterogeneous samples; The temporal feature processing module includes multiple sub-modules, and the sub-modules include at least convolution, feature fusion, multi-stream CNN module, Mamba module, StreamFusion module and time convolution network module.

4. The method for myoelectric signal gesture recognition based on adaptive multi-scale dual-stream feature fusion according to claim 3 is characterized in that: The process of the timing feature processing module includes at least the following steps: Input a pair of data with 400 time steps and 6 channels; The input data is first processed by a convolutional layer with 64 convolution kernels, a kernel size of 3, and a step size of 1, to embed the data from a low-dimensional space into a high-dimensional space. The convolutional features are passed to the multi-stream CNN module, which splits the data stream into multiple sub-streams and processes them independently to capture information in different time ranges or feature dimensions; After passing through the multi-stream CNN module, the data enters the multi-layer Mamba module and StreamFusion module The features in different sub-streams are fused together using the StreamFusion module to enhance the expressiveness of the features. A total of three layers of Mamba modules and StreamFusion modules are used alternately to gradually fuse the features. After multiple Mamba and StreamFusion passes, the features enter the temporal convolutional network module, which provides the ability to capture temporal information of long sequence dependencies and is an effective tool for modeling time series. The output of the temporal convolutional network module then passes through a maximum pooling layer to reduce the feature dimension to 64; Finally, it is further processed by the fully connected network to obtain the final feature vector.

5. The method for myoelectric signal gesture recognition based on adaptive multi-scale dual-stream feature fusion according to claim 1 is characterized in that: The frequency domain processing module includes at least a fully connected layer and a double-layer GRU, and the double-layer GRU is a gated recurrent unit.

6. The method for myoelectric signal gesture recognition based on adaptive multi-scale dual-stream feature fusion according to claim 5 is characterized in that: The method for processing the frequency domain characteristics of a signal by the frequency domain processing module comprises at least the following steps: Input 400 time steps and 12 frequency feature channels; The input data first passes through a fully connected layer to expand the feature dimension from 12 to 64 to capture more frequency features; The processed features are passed into a two-layer GRU layer with a dimension of 128 to model the sequence dependency of frequency features; The output of the GRU layer passes through a fully connected layer, and the feature dimension is compressed to 64. At this time, the output is a feature vector of size 64.

7. The method for myoelectric signal gesture recognition based on adaptive multi-scale dual-stream feature fusion according to claim 1 is characterized in that: The output of the bidirectional attention module is trained with FocalLoss to further optimize the performance of the model; The bidirectional attention module contains two feedforward neural networks, and the initial dimension of the features is 64.

8. The method for myoelectric signal gesture recognition based on adaptive multi-scale dual-stream feature fusion according to claim 7 is characterized in that: The processing flow of the bidirectional attention module includes at least the following steps: The time series and frequency domain features pass through their respective FNN layers to obtain a feature representation with a dimension of 64; After multiple addition and multiplication operations, it is used for interaction between different features. The addition operation represents feature fusion, and the multiplication operation is the redistribution of weights. Multiple addition and multiplication operations are designed to balance and enhance the complementarity of the two branch features to make full use of the time and frequency features; The interacted features pass through a final FNN layer to generate the final feature vector.

Images