SEMG gesture recognition method based on multi-domain feature fusion

Abstract

The invention discloses an sEMG gesture recognition method based on multi-domain feature fusion, and the method comprises the following steps: collecting and preprocessing a multi-channel sEMG signal, and obtaining a time domain signal, a frequency domain signal and a time-frequency domain signal; respectively extracting fusion features from the time domain and the frequency domain, and extracting spatial domain features from the time domain and the frequency domain; based on a cross attention mechanism, performing feature fusion by taking the time domain-frequency domain fusion feature as a query and taking the spatial domain feature as a key and a value to obtain a multi-domain fusion feature vector; and outputting a gesture classification result according to the vector. According to the method, the spatial dependency relationship between the graph neural network modeling channels is introduced, and the cross attention mechanism is combined to realize multi-domain feature adaptive fusion, so that the gesture recognition accuracy and the feature expression capability are effectively improved, meanwhile, the relatively low calculation complexity is kept, and the method is suitable for real-time deployment of embedded equipment.

Description

Technical Field

[0001] This invention belongs to the fields of human-computer interaction, pattern recognition and deep learning, and specifically relates to an sEMG gesture recognition method based on multi-domain feature fusion. Background Technology

[0002] sEMG signals are bioelectrical signals generated on the skin surface during human muscle activity, containing rich neuromuscular motion information. Decoding sEMG signals to achieve gesture recognition provides a natural, intuitive, and efficient interaction method for human-computer interaction, with broad application prospects in fields such as intelligent prosthetic limb control, virtual reality interaction, and rehabilitation medicine.

[0003] With the rapid development of artificial intelligence technology, deep learning-based sEMG gesture recognition methods have become mainstream. In pursuit of higher recognition accuracy, existing technologies often tend to employ deeper network layers or more complex model structures (such as introducing attention mechanisms or multi-head networks). However, this leads to a sharp increase in the number of model parameters and a significant increase in computational complexity. This trend towards increased complexity poses a severe challenge to the performance and runtime of computing devices, making it difficult to achieve low-power, high-real-time deployment on resource-constrained embedded devices (such as wearable gesture recognition armbands and intelligent prosthetic controllers), greatly limiting its commercialization and application prospects.

[0004] MLP-Mixer is a novel deep learning architecture proposed in recent years. It operates entirely based on Multilayer Perceptrons (MLPs) and matrix multiplication, demonstrating performance comparable to CNNs and Transformers in image classification tasks. This architecture captures spatial features through token-mixed MLPs and achieves feature fusion through channel-mixed MLPs, featuring a concise structure and high computational efficiency. Existing techniques have confirmed the good performance and efficiency of the MLP-Mixer architecture in sEMG gesture recognition tasks.

[0005] However, the channel mixing MLP in MLP-Mixer employs a globally fully connected approach, a design with significant drawbacks: it cannot distinguish between signals from functionally coordinating muscle groups and unrelated muscle groups. Standard channel mixing MLPs mix features from all channels in a "one-size-fits-all" manner, failing to intelligently identify and enhance the global functional dependencies between these coordinating muscle group channels. This "blindness" in utilizing functional connectivity information prevents it from fully leveraging the spatial information contained in sEMG signals, thus limiting further improvements in recognition performance.

[0006] Therefore, a new technical solution is needed to overcome the deficiency of insensitivity to spatial perception of sEMG signals when applying MLP-Mixer to the field of sEMG gesture recognition, and to provide a feasible technical path for achieving high-precision, low-power gesture recognition. Summary of the Invention

[0007] To address the aforementioned issues, this invention discloses an sEMG gesture recognition method based on multi-domain feature fusion. It introduces an innovative spatial perception module and a multi-domain feature cross-fusion mechanism, effectively compensating for the inherent defects in spatial perception when the standard MLP-Mixer architecture is applied to sEMG gesture recognition, thereby improving recognition accuracy while maintaining high inference speed.

[0008] To achieve the above objectives, the technical solution of the present invention is as follows:

[0009] A method for sEMG gesture recognition based on multi-domain feature fusion includes the following steps:

[0010] Step 1, Multi-channel sEMG signal acquisition and preprocessing:

[0011] sEMG signals were acquired using a 12-channel surface electromyography (sEMG) sensor with a sampling rate of 2000 Hz. The acquired raw signals were preprocessed to obtain time-domain, frequency-domain, and time-frequency domain signals.

[0012] Step 2, Multi-domain deep feature extraction:

[0013] Extract time-domain-frequency domain fusion features from the time-domain and frequency-domain signals, and extract spatial domain features from the time-frequency domain signals;

[0014] Step 3, Multi-domain feature fusion:

[0015] A cross-attention mechanism was designed to perform feature fusion using the time-frequency domain fusion features as the query and the spatial domain features as the key and value, to obtain a multi-domain fusion feature vector.

[0016] Step 4: Classification and model training, outputting gesture classification results based on multi-domain fusion feature vectors.

[0017] Furthermore, the preprocessing operation described in step 1 includes the following steps:

[0018] Step 1.1, Signal Segmentation: The continuous sEMG signal is segmented using an overlapping sliding window technique: the window length is set to 200 milliseconds, and the overlap step size is 100 milliseconds. This setting ensures that each data segment contains enough information to reflect gesture features, and also increases the number of training samples through overlap, thereby improving the model's generalization ability.

[0019] Step 1.2, Signal Normalization: To eliminate biases introduced by individual physiological differences, electrode contact impedance variations, and environmental noise, Z-score standardization is performed on the data within each window. This process transforms the data into a standard normal distribution with a mean of 0 and a standard deviation of 1, effectively reducing the negative impact of outliers and scale differences on model training.

[0020] Step 1.3, Frequency Domain to Time-Frequency Domain Conversion: To fully extract information from different forms in the signal, two transformations are performed on the normalized time-domain signal:

[0021] Fast Fourier Transform (FFT): Converts a time-domain signal to the frequency domain and calculates its amplitude spectrum as a frequency domain feature to extract stable frequency distribution characteristics in the signal, which are closely related to different muscle activity patterns.

[0022] Continuous Wavelet Transform (CWT): Using the Mexican Hat wavelet as the mother wavelet and setting the scale parameter to 32, a one-dimensional time-domain signal is converted into a two-dimensional time-frequency spectrum. CWT can provide joint time and frequency distribution information of the signal, overcoming the shortcomings of FFT in time positioning. It is particularly suitable for analyzing non-stationary signals such as sEMG, thus preserving both the time and frequency information of the signal.

[0023] Furthermore, step 2 is detailed below:

[0024] Step 2.1, Time-Domain and Frequency-Domain Feature Extraction: The time-domain signal and the frequency-domain features obtained from the FFT transform are input in parallel into a two-branch network structure with shared weights. This design allows the network to learn a general feature extraction pattern that is effective for both data formats. The processing flow of each branch is as follows:

[0025] Step 2.1.1, CNN Module: The data first passes through a convolutional neural network module. This module consists of three one-dimensional convolutional layers with kernel sizes of [1×3] and increasing channel numbers layer by layer (32, 64, 128), using the LeakyReLU activation function to introduce non-linearity. A residual block is then passed through this module. This residual block is first reduced to 64 channels using a [1,1] convolution, then uses a [3,3] convolution to extract richer local features, and finally uses a [1,1] convolution to restore it to 128 channels. This residual block is then added to the module input via a residual connection. This structure effectively alleviates the vanishing gradient problem in deep networks and enhances the expressive power of local features.

[0026] Step 2.1.2, Patch Partitioning and Embedding: The data processed by the CNN is divided into a series of non-overlapping patches. Each patch is linearly projected through a dedicated Patch-Embedding layer, flattening it and mapping it to a high-dimensional feature space. To further enhance the feature representation at the patch level, this embedding process also incorporates a convolutional layer with a kernel size of [1,3] to recalibrate the features of each patch. Subsequently, a fully connected (Dense) layer uniformly expands the feature dimension to 256 dimensions, preparing it for subsequent input to the MLP-Mixer.

[0027] Step 2.1.3, MLP-Mixer Module: The embedded features are input into the MLP-Mixer module. This module no longer relies on traditional convolutional operations but is based on a Multilayer Perceptron (MLP). It contains two types of layers: Token-Mixing MLP and Channel-Mixing MLP. Token-Mixing MLP applies MLP along the channel dimension to achieve information interaction and fusion between different time points (tokens), capturing global temporal dependencies. Channel-Mixing MLP applies MLP along the time dimension to achieve fusion between features from different channels at the same time point, enhancing the model's feature combination capabilities. The two MLPs are stacked alternately, with LayerNorm applied for normalization before each layer and residual connections, enabling the model to efficiently fuse local and global features.

[0028] Step 2.2, Time-Frequency Domain Feature Extraction: This branch focuses on processing the time-spectrum graph generated by CWT and explicitly models the spatial dependencies between multiple channels to simulate muscle synergy. This includes processing of both time blocks and spatial blocks.

[0029] Step 2.2.1, Time Block Processing: The time spectrum of each channel is first processed through a time block. This time block consists of a one-dimensional convolutional layer (kernel size [1×3]) and a gated linear unit (GLU). The one-dimensional convolution operates along the time dimension to extract high-level temporal dynamic features. The GLU then divides the convolution output into two parts along the channel dimension. One part is gated by the sigmoid function, and the gated signal is multiplied element-wise with the other part to achieve adaptive feature selection while compressing the feature dimension.

[0030] Step 2.2.2, Graph Structure Construction and Spatial Modeling: Channels are treated as nodes in a graph structure. The initial feature vector of each node is composed of the output of its corresponding time-spectrum graph after time-block processing. To control dimensionality, a fully connected layer projects the node features to 256 dimensions. The spatial relationships (edges) between nodes are not predefined but dynamically calculated through a self-attention mechanism. A learnable linear transformation is used to map the node features to query and key vectors respectively. By calculating the dot product of the query and key and applying Softmax normalization, an adjacency matrix is ​​obtained, where the elements of the matrix represent the functional connectivity strength between each pair of channels.

[0031] Step 2.2.3, Graph Convolutional Network (GNN): Utilizing the dynamically constructed adjacency matrix described above, multiple layers of graph convolutional layers are applied to iteratively update node features. The update rule for each layer comprehensively considers the node's own features and the features of its neighboring nodes (weighted summation based on the adjacency matrix weights), thus enabling message passing across the entire graph. A non-linear transformation is then performed using the ReLU activation function. After multiple layers of graph convolution, the node features contain rich local and global spatial context information.

[0032] Global feature aggregation: Finally, global average pooling is used to aggregate all updated node features and compress them into a single, fixed-length global feature vector, which represents the spatial domain features of the multi-channel sEMG signal.

[0033] Furthermore, step 3 is detailed below:

[0034] First, the time-domain and frequency-domain feature vectors extracted by the dual-branch network are concatenated to form a time-frequency fusion feature. Then, using this time-frequency fusion feature as the query and the global feature vector extracted from the spatial domain branch as both the key and value, cross-attention is performed. This mechanism allows the time-frequency feature to proactively "ask" which parts of the spatial domain features are most relevant and complementary to its current representation.

[0035] The calculated attention output is added to the original time-frequency domain fusion features, and then the training process is stabilized through layer normalization. The normalized result is then passed through a feedforward network (FFN), which consists of two fully connected layers and an activation function (GELU) between them, for further nonlinear transformation and feature fusion, ultimately outputting a final fused feature vector that integrates multi-domain information and has extremely strong representational capabilities.

[0036] Furthermore, step 4 is detailed below:

[0037] Step 4.1, Classification Output: The fused feature vector is first further reduced in dimensionality and compressed by global average pooling, and then input into a fully connected layer, which maps the features to the same dimension as the number of gesture categories; finally, a probability distribution vector is output through the Softmax activation function, and the category with the highest probability is taken as the recognition result.

[0038] Step 4.2, Model Training: The standard cross-entropy loss function in classification tasks is used as the optimization objective to measure the difference between the model's predicted probability distribution and the true labels; the optimizer is Adam, with an initial learning rate set to 1e-4; a learning rate decay strategy is adopted during training to finely tune the model parameters in the later stages of training, thereby ensuring the stability of the training process, promoting model convergence, and helping to achieve better generalization performance.

[0039] The beneficial effects of this invention are:

[0040] (1) Significantly improved accuracy of gesture recognition: This invention introduces a spatial perception module based on graph neural networks to explicitly model the functional connectivity and spatial dependencies between multi-channel sEMG signals, effectively overcoming the limitations of traditional methods.

[0041] The MLP-Mixer architecture overcomes the deficiency of "blindly" mixing spatial information. Simultaneously, it utilizes a cross-attention mechanism to achieve deep adaptive fusion of temporal, frequency, and spatial domain features, fully leveraging the complementary advantages of multi-domain information, thereby significantly improving the recognition accuracy of complex gestures.

[0042] (2) Enhanced feature representation capability while maintaining high efficiency: In the feature extraction stage, this invention adopts a dual-branch network with shared weights and a lightweight MLP-Mixer module, inheriting its advantages of simple structure and high computational efficiency. At the same time, through innovative spatial modeling and feature fusion design, the model's ability to extract and fuse key discriminative features in sEMG signals is greatly enhanced without significantly increasing computational complexity, achieving a good balance between performance and efficiency. Attached Figure Description

[0043] Figure 1 This is the overall architecture diagram of the model.

[0044] Figure 2 This is a diagram of the network architecture for time-domain / frequency-domain feature extraction.

[0045] Figure 3 This is a diagram of the network architecture for time-frequency domain feature extraction.

[0046] Figure 4 This is a diagram of a multi-domain feature fusion architecture. Detailed Implementation

[0047] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0048] As shown in the figure, the sEMG gesture recognition method based on multi-domain feature fusion of the present invention includes the following steps:

[0049] Step 1: Multi-channel sEMG signal acquisition and preprocessing

[0050] This implementation acquires raw sEMG signals using a 12-channel electrode array deployed on the surface of the user's forearm muscles. The data acquisition device's sampling rate is set to 2000Hz to satisfy the Nyquist sampling theorem and ensure distortion-free capture of the most effective frequency components of the sEMG signal, up to approximately 1000Hz. The acquired raw signal is 12-channel time-series voltage data. Data preprocessing is performed on the surface electromyography signal of each channel, including four sub-steps: signal segmentation, normalization, frequency domain transformation, and time-frequency domain transformation.

[0051] Step 1.1, Signal Segmentation. As a specific implementation of data preprocessing, an overlapping sliding window technique is used to segment the continuous 12-channel sEMG streaming data. The window length is set to 200ms, and the sliding step size is set to 100ms. This method aims to extract data segments corresponding to specific gestures from long-term sequences, while increasing the number of samples through overlap, thereby improving the efficiency and generalization ability of model training.

[0052] Step 1.2, Signal Normalization. Z-score normalization is used to normalize the data for each channel within each window. This step aims to eliminate signal amplitude deviations caused by individual differences in muscle strength, changes in skin impedance, slight electrode displacement, etc., reducing the impact of abnormal samples on the model. The normalization calculation formula is as follows:

[0053]

[0054] Where X is the original value of a sampling point within the window, μ is the mean of all sample data within the window, and σ is the standard deviation of all sample data within the window. After processing, the data has a mean of 0 and a standard deviation of 1, conforming to a standard normal distribution.

[0055] Step 1.3, Frequency Domain Transformation (FFT). Perform a Fast Fourier Transform (FFT) on the time-domain signal of each normalized channel to extract frequency domain features. Let the signal of a certain channel within the window be x[n], its FFT transform is:

[0056]

[0057] Where N is the window length, f = 0, 1, ..., N-1 is the frequency index; e is a mathematical constant, and j is the imaginary unit; the amplitude spectrum of the transformation result is taken as the frequency domain feature of this channel of the window:

[0058] A[f]=X[f]|

[0059] Step 1.4, Time-Frequency Transform (CWT). A continuous wavelet transform (CWT) is performed on the time-domain signal of each normalized channel to convert the one-dimensional time-domain signal into a two-dimensional time-frequency spectrum, thus preserving both the time and frequency information of the signal. The mother wavelet chosen is the Mexican Hat Wavelet, which is the second derivative of a Gaussian function, exhibiting good localization characteristics in both the time and frequency domains. Being a real function, it eliminates the need to consider phase issues, and its shape has a high similarity to the action potential waveform of the motor unit in the sEMG signal. The scaling parameter is set to 32, ensuring that the constructed scaling sequence covers the effective physiological frequency band of the sEMG signal. The CWT is converted to:

[0060]

[0061] Where a is the scaling parameter, b is the translation parameter, and ψ(t) is the mother wavelet function. * The complex conjugate of the mother wavelet is given, and t is the time variable. After performing CWT on each window of data for each channel, a two-dimensional matrix is ​​obtained. This is the instantaneous spectrogram.

[0062] Step 2, Multi-domain Deep Feature Extraction. Multi-domain deep feature extraction includes extracting high-level deep features from the time domain, frequency domain, and time-frequency domain, respectively.

[0063] Step 2.1, Time-domain and frequency-domain feature extraction: such as Figure 2 The time-domain and frequency-domain data are input into a shared-structure dual-branch network. First, the data is fed into the convolutional part to extract surface features. Then, the data is divided into non-overlapping patches and embedded into features through a patch-embedding layer. Finally, the data passes through MLP layers, including token-mixing MLP and channel-mixing MLP, to achieve feature fusion.

[0064] Step 2.1.1, CNN Module: Contains three one-dimensional convolutional layers, each with a kernel size of [1×3], and the number of channels increases sequentially to 32, 64, and 128. Each convolutional layer is followed by a LeakyReLU activation function. A residual block is appended after the three convolutional layers. This residual block first uses a [1x1] convolution to reduce the number of channels from 128 to 64, then uses a [3x3] convolution to further extract features, and finally uses another [1x1] convolution to restore the number of channels to 128. The input and output of the block are added through residual connections. This structure effectively enhances the representation ability of local features and alleviates the gradient vanishing or gradient exploding problems that may occur in deep networks. Within the residual block, the last two convolutional layers also have a LeakyReLU activation function added after them.

[0065] Step 2.1.2, Patch Embedding: Inspired by ViT, the feature maps extracted by the CNN are divided into N regions of size [P] along the time / frequency dimension. H ,P W The plots are divided into D-dimensional vectors, and each plot is flattened into a D-dimensional vector. The formula is as follows:

[0066] D = P H ×P W

[0067] N = (H × W) / D

[0068] H is the height of the feature map, W is the width of the feature map, and P is the height of the feature map. H P is the height of the patch. W D is the width of the patch, D is the vector dimension of each patch after expansion, and N is the total number of patches.

[0069] To enhance the feature representation capability of each patch, an additional one-dimensional convolutional layer with a kernel size of [1,3] is added during the patch embedding process to recalibrate the features within each patch. Finally, a fully connected layer is used to uniformly expand and project the feature dimensions of each patch to 256 dimensions, forming patch embeddings. These 256-dimensional vectors serve as the input to the subsequent MLP-Mixer module.

[0070] Step 2.1.3, MLP-Mixer Module: MLP-Mixer is an architecture entirely based on Multilayer Perceptron (MLP), abandoning traditional convolution operations and mixing two types of MLP layers. It includes Token-Mixing MLP and Channel-Mixing MLP.

[0071] The Token-Mixing MLP operates on the Patch dimension (sequence length dimension), fixing the channel dimension (256) through matrix transpose, and sharing the same MLP across all Patches. This MLP allows information exchange and fusion between Patch features at different time points (or frequency points), thereby capturing global dependencies within the entire window.

[0072] Channel-Mixing MLP operates on the channel dimension (feature dimension) and fixes the position dimension (number of patches) through matrix transpose. The same MLP is applied independently to the feature vector of each patch. This MLP achieves cross-channel feature fusion and can learn how to combine different feature channels to form a more effective representation.

[0073] Each MLP layer consists of a fully connected layer, a GELU non-linear activation function layer, and another fully connected layer. Layer Norm is applied before the first fully connected layer for normalization to stabilize the training process. Each MLP sub-block is supplemented with residual connections to ensure smooth information propagation and prevent network degradation. Token-Mixing and Channel-Mixing MLPs are stacked alternately multiple times, allowing the model to fully integrate local and global features of the input data.

[0074] Step 2.2 Time-Frequency Domain Feature Extraction. This step aims to extract spatially dependent features from the time-spectrum plot of the multi-channel sEMG signal. For example... Figure 3 First, primary temporal features are extracted and their dimensions are compressed using time blocks. Then, spatial blocks are used to adaptively model the spatial relationships between different channels.

[0075] Step 2.2.1, Temporal Block: This consists of a one-dimensional convolutional layer (CNN) and a gated linear unit (GLU). The aim is to extract high-level temporal dynamic features from the raw temporal spectrum and compress its dimensionality. A one-dimensional convolutional layer (kernel size [1×3]) is used, with the output channel count set to 256, and a non-linear transformation is performed using the LeakyReLU activation function. This convolution slides along the temporal dimension to capture local patterns in the temporal spectrum. The output of the convolutional layer is then split into two parts (128 dimensions each) along the channel dimension. One part is generated by the Sigmoid function to produce a gated signal between 0 and 1, and the other part is element-wise multiplied with this gated signal. This gating mechanism enables adaptive feature selection, allowing the model to dynamically emphasize or suppress certain temporal features while compressing the feature dimension from 256 to 128.

[0076] Step 2.2.2, Constructing the graph structure: The feature maps of the 12 channels after time-block processing are considered as nodes, resulting in a total of 12 nodes (corresponding to the 12 channels). The initial feature vector of each node is the vector flattened from the CWT amplitude spectrum of that channel. To control the feature dimension, a fully connected layer projects the node features to 256 dimensions. A self-attention mechanism is used to dynamically construct the adjacency matrix A∈R. 12×12 , representing the strength of the association between nodes. The formula for calculating the self-attention adjacency matrix is ​​as follows:

[0077]

[0078] Where Q and K are the query and key vectors obtained by linear transformation of node features, and d is the feature dimension. Softmax normalizes by row, so that the sum of the attention weights of each node to all other nodes is 1.

[0079] Step 2.2.3, Graph Convolutional Network (GNN): Using the dynamically constructed adjacency matrix described above, a multi-layer graph convolutional network (GNN) is employed to iteratively update node features and perform message passing. The rules for updating node features at each layer are as follows:

[0080]

[0081] Where A ^ =A+I is the adjacency matrix with self-connections, D ^ H is the degree matrix. (l) For the node features of layer l, W (l) Let σ be the learnable weight matrix for this layer, and σ be the non-linear activation function.

[0082] After multi-layer graph convolution, the features of each node contain information about its neighboring nodes (other functionally related muscle channels). Finally, global average pooling is used to aggregate the features of all 12 nodes, and the average value is taken to output a 256-dimensional global feature vector. This vector comprehensively represents the spatial coordination pattern (spatial domain features) of multi-channel muscle activity, as shown in the following formula:

[0083]

[0084] Step 3, Multi-domain feature fusion: Previously, the three branches output the temporal feature F respectively. t Frequency domain characteristics F f Spatial domain features Fs. This step aims to deeply integrate these three elements. For example... Figure 4 First, the time-domain feature F t and frequency domain features F f spliced ​​into time-domain and frequency-domain fused features F t-f Then, the time-domain and frequency-domain fused features F are fused through a fully connected projection layer.t-f The dimensionality was reduced from 512 to 256 to match the dimensionality of the spatial domain feature Fs. Finally, the dimensionality-reduced time-domain-frequency domain feature Fs was... t-f For the query, spatial domain features Fs are the key and value, respectively. These are input into the cross-attention fusion module to obtain fused features, achieving multi-domain information complementarity. The formula for calculating attention weights is as follows:

[0085]

[0086] Where d k Q represents the dimension of the key vector, where Q is the Query, K is the Key, and V is the Value. The softmax function normalizes the vector row-by-row. This mechanism allows for time-domain to frequency-domain features F. t-f Actively "query" which information in the spatial domain features Fs is the most beneficial supplement to itself, thereby achieving precise feature complementarity.

[0087] The attention output is added to the original features and then normalized by a LayerNorm layer. This step aims to prevent network degradation and stabilize the training process. Finally, the normalized result is input into a feedforward network (FFN), which consists of two fully connected layers and an activation function (GELU) between them. This network further performs nonlinear transformations and feature fusion, ultimately outputting a highly discriminative multi-domain fused feature vector F that integrates temporal, frequency, and spatial information. fused The formula is as follows:

[0088] F fused =FFN(LayerNoorm(X) t +Attention(Q,K,V)))

[0089] Step 4: Classification and model training.

[0090] Step 4.1, Classification Output: The final fused feature vector F fused First, global average pooling is used for further compression and dimensionality reduction, then the input is passed to a fully connected layer for feature mapping. The number of output neurons in this fully connected layer equals the number of categories of the gesture to be recognized. Finally, the output is transformed into a probability distribution vector through a softmax activation function, where each element represents the probability that the input sEMG signal belongs to the corresponding gesture category. The category with the highest probability is taken as the final recognition result.

[0091] Step 4.2, Model Training: The cross-entropy loss function is used as the optimization objective of the model. This function can effectively measure the difference between the predicted probability distribution and the true label. The Adam optimizer is used in the optimization process, with an initial learning rate of 1e-4. To further improve the training effect, a learning rate decay strategy is introduced. This strategy can automatically reduce the learning rate in the later stages of training, which helps stabilize the training process and promote convergence.

[0092] This invention innovatively combines graph neural networks with a dynamic attention mechanism, effectively solving the problem that traditional MLP-Mixer cannot perceive the spatial dependencies of sEMG signals. By constructing a functional connectivity graph and applying graph convolution operations, the model can adaptively learn muscle coactivation patterns, thereby extracting more discriminative spatial domain features. Simultaneously, the introduced cross-attention fusion mechanism can fully exploit the complementary information between temporal, frequency, and spatial domain features, achieving effective integration of multi-level features. Furthermore, this invention maintains low computational complexity while improving recognition accuracy. The MLP-Mixer module in the model ensures efficient feature processing, while the carefully designed network structure effectively controls the overall model parameter count, enabling real-time deployment on embedded devices.

[0093] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A sEMG gesture recognition method based on multi-domain feature fusion, characterized in that: The method comprises the following steps: Step 1, multi-channel sEMG signal acquisition and preprocessing: sEMG signals are collected by a 12-channel surface electromyography signal sensor, the sampling rate is set to 2000 Hz, and the original signals collected need to be preprocessed to obtain time domain signals, frequency domain signals and time-frequency domain signals; Step 2, multi-domain deep feature extraction: time domain-frequency domain fusion features are extracted from the time domain signals and the frequency domain signals, and spatial domain features are extracted from the time-frequency domain signals; Step 3, multi-domain feature fusion: a cross-attention mechanism is designed, the time domain-frequency domain fusion features are taken as queries, and the spatial domain features are taken as keys and values, feature fusion is performed, and a multi-domain fusion feature vector is obtained; Step 4, classification and model training, and outputting a gesture classification result according to the multi-domain fusion feature vector. 2.The sEMG gesture recognition method based on multi-domain feature fusion according to claim 1, characterized in that: The preprocessing operation in step 1 comprises the following steps: Step 1.1, signal segmentation: the continuous sEMG signal is segmented by using an overlapping sliding window technique: the window length is set to 200 milliseconds, and the overlapping step is set to 100 milliseconds; Step 1.2, signal normalization: the data in each window is subjected to Z-score standardization processing; the processing makes the data mean value 0 and the standard deviation 1, and converts it into a standard normal distribution; Step 1.3, frequency domain and time-frequency domain conversion: in order to fully mine different forms of information in the signal, the normalized time domain signal is subjected to two kinds of transformation: Fast Fourier Transform (FFT): the time domain signal is converted to the frequency domain, and the amplitude spectrum is calculated as the frequency domain feature; Continuous Wavelet Transform (CWT): the Mexican Hat wavelet is selected as the mother wavelet, the scale parameter is set to 32, and the one-dimensional time domain signal is converted into a two-dimensional time-frequency spectrum. 3.The sEMG gesture recognition method based on multi-domain feature fusion according to claim 2, characterized in that: The time domain-frequency domain fusion features in step 2 are as follows: Step 2.1, time domain and frequency domain feature extraction: the time domain signal and the frequency domain feature obtained by FFT transformation are input into a double-branch network structure sharing weights in parallel, and the processing procedure of each branch is as follows: Step 2.1.1, CNN module: the data is first input into a convolutional neural network module; the module comprises three one-dimensional convolutional layers, the convolution kernel size is [1x3], the channel number increases layer by layer (32, 64, 128), and a LeakyReLU activation function is used to introduce nonlinearity; then a residual block is connected, the residual block first uses [1,1] convolution to reduce the dimension to 64 channels, then uses [3,3] convolution to extract more local features, and finally uses [1,1] convolution to restore to 128 channels, and adds the module input through the residual connection; Step 2.1.2, Patch division and embedding: The data processed by CNN is divided into a series of non-overlapping patches, and each patch is linearly projected by a dedicated Patch-Embedding layer to flatten and map it to a high-dimensional feature space. To further enhance the feature representation at the patch level, a kernel size of [1, 3] convolution layer is incorporated into the embedding process to recalibrate the features of each patch. Subsequently, a fully connected Dense layer uniformly expands the feature dimension to 256 dimensions, preparing for subsequent input into the MLP-Mixer. Step 2.1.3, MLP-Mixer module: The embedded features are input into the MLP-Mixer module, which includes two types of layers: Token-Mixing MLP and Channel-Mixing MLP. Token-Mixing MLP achieves information interaction and fusion between different time point tokens by applying MLP in the channel dimension, capturing global temporal dependencies. Channel-Mixing MLP achieves the fusion of different channel features at the same time point by applying MLP in the time dimension. The two MLPs are alternately stacked, each preceded by LayerNorm normalization and supplemented by residual connections, allowing the model to integrate local and global features.

4. The sEMG gesture recognition method based on multi-domain feature fusion according to claim 2, characterized in that: The extraction of spatial domain features according to step 2 is as follows: Step 2.2, Time-frequency domain feature extraction: This branch focuses on processing the time-frequency spectrograms generated by CWT and explicitly modeling the spatial dependencies between multiple channels, simulating muscle coordination. It includes time block and spatial block processing. Step 2.2.1, Time block processing: The time-frequency spectrogram of each channel is first processed by a time block. The time block consists of a one-dimensional convolution layer and a Gated Linear Unit (GLU). The one-dimensional convolution operates along the time dimension to extract high-level temporal dynamic features. GLU divides the convolution output into two parts in the channel dimension, one part generates a gating signal through the Sigmoid function, and the other part is multiplied element-wise to achieve adaptive feature selection while compressing the feature dimension. Step 2.2.2, Graph structure construction and spatial modeling: The channels are treated as nodes in a graph structure. The initial feature vector of each node is composed of the output of its corresponding time-frequency spectrogram after time block processing. To control the dimension, a fully connected layer is used to project the node features to 256 dimensions. The node features are mapped to query and key vectors using a learnable linear transformation. By calculating the dot product of the query and key and applying Softmax normalization, an adjacency matrix is obtained. Step 2.2.3, Graph Convolutional Network (GNN): Using the dynamically constructed adjacency matrix, multiple layers of graph convolution layers are applied to iteratively update the node features. The update rule of each layer considers both the node's own features and its neighbor nodes' features, enabling message passing across the entire graph. After that, a ReLU activation function is used for non-linear transformation. ​ Step 2.2.4, global feature aggregation: finally, the global average pooling operation is used to aggregate all the updated node features, compressing them into a single, fixed-length global feature vector, which represents the spatial domain features of the multi-channel sEMG signal.

5. The sEMG gesture recognition method based on multi-domain feature fusion according to claim 3, characterized in that: Step 3 is as follows: First, the time-domain feature vector and the frequency-domain feature vector extracted by the double-branch network are spliced to form a time-frequency domain fusion feature; then, the cross-attention calculation is performed with the time-frequency domain fusion feature as the query and the global feature vector extracted by the spatial domain branch as the key and value; The calculated attention output is added to the original time-frequency domain fusion feature, and then normalized by layer normalization to stabilize the training process; The normalized result is then passed through a feedforward network FFN composed of two fully connected layers and an activation function GELU in between for further nonlinear transformation and feature fusion, finally outputting a final fusion feature vector that integrates multi-domain information and has strong representation ability.

6. The sEMG gesture recognition method based on multi-domain feature fusion according to claim 5, characterized in that: Step 4 is as follows: The fused feature vector is first passed through global average pooling for further dimension reduction and compression, and then input into a fully connected layer that maps the features to the same dimension as the number of gesture categories; a probability distribution vector is output through the Softmax activation function, and the class with the maximum probability is taken as the recognition result.

7. The sEMG gesture recognition method based on multi-domain feature fusion according to claim 5, characterized in that: Step 4 is as follows: The standard cross-entropy loss function in the classification task is used as the optimization objective to measure the difference between the model's predicted probability distribution and the true label; the Adam optimizer is selected with an initial learning rate of 1e-4; and the learning rate decay strategy is used during training.

Images