Gesture recognition method and device based on convolutional spiking neural network
By constructing a three-layer convolutional spiking neural network and combining adaptive optimization strategies with deterministic training, the high computational complexity and high power consumption of deep learning models in electromyography (EMG) signal processing are solved, achieving low-latency and high-precision EMG signal recognition, which is suitable for intelligent prostheses and human-computer interaction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-30
- Publication Date
- 2026-04-03
AI Technical Summary
Existing deep learning models suffer from high computational complexity, high power consumption, and long processing time in electromyography (EMG) signal processing, making it difficult to achieve stable recognition and real-time interaction of high-frequency EMG signals.
A gesture recognition method based on convolutional spiking neural networks is adopted. By constructing a three-layer convolutional spiking neural network, combining a feature extraction layer and a fully connected classification layer, an event-driven spiking mechanism, and an adaptive optimization strategy and deterministic training mechanism, efficient feature extraction and classification of electromyographic signals are achieved.
It achieves low-power, low-latency electromyography signal recognition, with high accuracy and real-time performance, and is suitable for intelligent prostheses and human-computer interaction scenarios.
Smart Images
Figure CN121786549A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to bioelectric signal processing and neuromorphic computing, and more particularly to a gesture recognition method and apparatus based on a convolutional spiking neural network. Background Technology
[0002] With the development of wearable devices and intelligent prosthetic control technology, electromyography (EMG), as a physiological signal reflecting muscle contraction activity, is widely used in gesture recognition and human-computer interaction.
[0003] Traditional deep learning models (such as CNN and LSTM) have high accuracy, but they suffer from high computational complexity, high power consumption, and long processing time in the process of high-frequency electromyography signal processing, which is not conducive to deployment in real-time interactive scenarios.
[0004] Spiking Neural Networks (SNNs) are event-driven neural network models that offer low power consumption and high biological interpretability. However, most existing SNNs have simple structures, insufficient utilization of temporal features, and difficulty in achieving stable recognition on complex multi-channel electromyographic signals.
[0005] Therefore, it is necessary to propose a convolutional spiking neural network gesture recognition method that is structurally efficient, training-stable, and has real-time performance, in order to achieve high-precision, low-latency human-computer interaction signal recognition. Summary of the Invention
[0006] In view of the above problems, this application proposes a gesture recognition method based on convolutional spiking neural networks.
[0007] The gesture recognition method based on convolutional spiking neural networks in this application preprocesses electromyographic signals and inputs them into a convolutional spiking neural network for gesture recognition; wherein the convolutional spiking neural network includes a feature extraction layer and a fully connected classification layer;
[0008] The feature extraction layer consists of three layers. The first feature extraction layer comprises a first convolutional layer, a first batch normalization layer, and a first IFNode layer, connected in sequence. The second feature extraction layer comprises a second convolutional layer, a second batch normalization layer, a second IFNode layer, and a max pooling layer, connected in sequence. The third feature extraction layer comprises a third convolutional layer, a third batch normalization layer, a third IFNode layer, and an adaptive average pooling layer, connected in sequence. The first, second, and third feature extraction layers are connected in sequence. The preprocessed electromyographic signal is input to the first feature extraction layer, and the third feature extraction layer outputs the feature extraction result.
[0009] The fully connected classification layer includes a first fully connected layer connected sequentially, The system consists of a regularization layer, a fourth IFNode layer, a second fully connected layer, and a Softmax activation layer. The first fully connected layer receives the feature extraction result from the feature extraction layer, and the Softmax activation layer outputs the gesture recognition result.
[0010] in, This represents an integral-firing spiking neuron, whose membrane potential update rule is as follows:
[0011]
[0012]
[0013] in For Heaviside step function, Where W is the firing threshold, W is the weight, and Vt is the neuron's value at time t. The membrane potential, Vt-1 is the membrane potential at the previous moment, and St is the membrane potential at the next moment. The impulse output of the neuron, St-1 is the impulse output of the neuron at the previous time step;
[0014] Gradient propagation in convolutional spiking neural networks uses a sigmoid gradient instead of a gradient.
[0015] Preferably, during the training of the convolutional spiking neural network, data sample balancing and enhancement are performed on its sample set; a bidirectional balancing strategy of resting class downsampling and action class oversampling is used to maintain the balance of sample distribution in the sample set.
[0016] Preferably, the following strategy is used when training a convolutional spiking neural network:
[0017] Employ the cross-entropy loss function;
[0018] Using the Adam optimizer, the initial learning rate is... And it decays by 0.5 every 15 iterations.
[0019] Preferably, a fixed random seed and a deterministic cuDNN mode are set.
[0020] Preferably, during training, the following strategies are used to improve model stability:
[0021] Adaptive learning rate adjustment, optimal model weight saving, and batch latency monitoring and optimization result in an average inference latency of <10ms.
[0022] Preferably, the electromyography signal preprocessing includes the following steps:
[0023] Remove DC bias;
[0024] Bandpass filtering of 20–450 Hz to eliminate noise;
[0025] Normalization processing ensures that the signal amplitude of each channel is within the range of interval;
[0026] Sliding window segmentation, the sliding window uses length Step length .
[0027] This application also proposes a gesture recognition device based on a convolutional spiking neural network, which includes an electromyography signal preprocessing unit and a convolutional spiking neural network unit.
[0028] The electromyography signal preprocessing unit is used to preprocess the electromyography signals;
[0029] The convolutional spiking neural network unit is used to process the preprocessed electromyographic signals to obtain gesture recognition results;
[0030] A convolutional spiking neural network unit includes: a feature extraction layer and a fully connected classification layer;
[0031] The feature extraction layer consists of three layers. The first feature extraction layer comprises a first convolutional layer, a first batch normalization layer, and a first IFNode layer connected in sequence. The second feature extraction layer comprises a second convolutional layer, a second batch normalization layer, a second IFNode layer, and a max pooling layer connected in sequence. The third feature extraction layer comprises a third convolutional layer, a third batch normalization layer, a third IFNode layer, and an adaptive average pooling layer connected in sequence. The first, second, and third feature extraction layers are connected in sequence. The preprocessed thickness electromyography signal is input to the first feature extraction layer, and the third feature extraction layer outputs the feature extraction result.
[0032] The fully connected classification layer includes a first fully connected layer connected sequentially, The system consists of a regularization layer, a fourth IFNode layer, a second fully connected layer, and a Softmax activation layer. The first fully connected layer receives the feature extraction result from the feature extraction layer, and the Softmax activation layer outputs the gesture recognition result.
[0033] in, This represents an integral-firing spiking neuron, whose membrane potential update rule is as follows:
[0034]
[0035]
[0036] in For Heaviside step function, Where W is the firing threshold, W is the weight, and Vt is the neuron's value at time t. The membrane potential, Vt-1 is the membrane potential at the previous moment, and St is the membrane potential at the next moment. The impulse output of the neuron, St-1 is the impulse output of the neuron at the previous time step;
[0037] Gradient propagation in convolutional spiking neural network units uses a sigmoid gradient instead of a gradient.
[0038] This invention achieves efficient feature extraction and classification of electromyographic signals by constructing an event-driven neuromorphic computing structure, thereby ensuring recognition accuracy while possessing low power consumption and low latency computing performance. The method employs a stable and reproducible spiking neural network training mechanism, combined with an adaptive optimization strategy to enhance the model's robustness and generalization ability. It supports real-time inference and deployment on embedded devices or neuromorphic hardware platforms, meeting the needs for high-precision, low-power signal recognition in scenarios such as intelligent prostheses, human-computer interaction, and rehabilitation engineering. Attached Figure Description
[0039] Figure 1 This is a flowchart of high-density electromyography gesture recognition according to an embodiment of the present invention.
[0040] Figure 2 This is a structural diagram of the pulsed convolutional neural network according to an embodiment of the present invention.
[0041] Figure 3 This is the training loss curve of the model in an embodiment of the present invention.
[0042] Figure 4 This is the accuracy curve of the model training set and validation set in an embodiment of the present invention.
[0043] Figure 5 This is a schematic diagram illustrating the specific types of gesture recognition according to the present invention. Detailed Implementation
[0044] This invention proposes a gesture recognition method based on a spiking neural network, comprising the following steps:
[0045] S1: Data acquisition and preprocessing of high-density electromyography signals;
[0046] S2: Sample balancing and feature enhancement;
[0047] S3: Construction of a convolutional spiking neural network;
[0048] S4: Model Training and Optimization;
[0049] S5: Verification and Testing.
[0050] Compared with existing technologies, this invention has significant advantages in terms of accuracy, real-time performance, and system stability for electromyography (EMG) signal gesture recognition. First, by introducing a three-layer convolutional spiking neural network, this invention achieves hierarchical feature extraction of EMG signals in both the temporal and spatial domains, effectively capturing subtle differences in various hand gestures and thus significantly improving feature representation capabilities. Second, based on an event-driven spiking mechanism, this invention updates only the neurons that generate spiking signals during the inference phase, greatly reducing redundant computation and energy consumption, resulting in an average single-batch inference latency of less than 10ms, exhibiting excellent real-time performance and low power consumption. Third, by introducing deterministic random seeds and a fixed cuDNN mode during network training, the reproducibility of model training results and algorithm stability are ensured, effectively avoiding performance fluctuations caused by randomness. Furthermore, this invention employs an adaptive learning rate adjustment strategy and an optimal model preservation mechanism, dynamically optimizing the learning rate based on the convergence state during training to prevent overfitting and gradient oscillations, further improving the model's generalization ability and convergence efficiency. In summary, this invention achieves an excellent balance between accuracy, energy efficiency, and stability, providing an efficient and reliable solution for real-time recognition of electromyographic signals in intelligent prostheses, human-computer interaction, and rehabilitation engineering.
[0051] As a further improvement of the present invention, the construction of the convolutional spiking neural network in S2 is as follows:
[0052] The convolutional layer filters the input in the time domain. For the ... A layer, a discrete one-dimensional convolution on the time axis (for each sample and channel), can be written as:
[0053]
[0054] in, For the first Layer convolution kernel length.
[0055] For the first layer, Indicates the input encoding (which can be a continuous value). (or a pulsed sequence).
[0056] For each channel :
[0057]
[0058] in The mean and variance of this channel within the batch. For learnable scales and offsets.
[0059] At any moment :
[0060]
[0061] Distribution (forward):
[0062]
[0063] in It is the Heaviside step function (taking 1 when the independent variable) )).
[0064] The second layer is followed by MaxPool1d(2): max pooling is performed on the time dimension with a step size of 2. If the impulse tensor is in the time domain... Max pooling reduces the pooling time. The following is:
[0065]
[0066] in For the corresponding set of window indices (e.g.) ).
[0067] After the third layer, AdaptiveAvgPool1d(1) is averaged over the time dimension to obtain the global features of each channel:
[0068]
[0069] Here For the index of the final convolutional layer, output (The dimension is then obtained by squeeze(-1) in the code) ).
[0070] The classifier first undergoes a linear transformation:
[0071]
[0072] Dropout will during the training phase A portion of the units are randomly set to zero (with probability). ), denoted as .
[0073] Then, through IFNode (mapping continuous values to pulse behavior): membrane potential and discharge are defined.
[0074]
[0075] (In the implementation of the classifier, the IFNode is usually approximated as a single-step impulse; in the code, it is followed by the last linear layer.)
[0076] Finally, a linear mapping is made to the class logits:
[0077]
[0078] Softmax and cross-entropy loss:
[0079]
[0080] in For one-hot tags.
[0081] because It is not differentiable at the threshold, so an alternative derivative is used during training. replace (That is, approximating the derivative of the step using a continuous function in backpropagation). Commonly used forms include:
[0082] Sigmoid derivative approximation:
[0083]
[0084] in These are hyperparameters. They are used during backpropagation. Alternative .
[0085] We give the weights The gradient representation (convolutional layer) and the form of the classifier weight gradient.
[0086] Top-level (readout) error signal
[0087] Assuming logits are read With cross-entropy loss Then the gradient with respect to logits is:
[0088]
[0089] Weights of the last layer of the classifier :
[0090]
[0091] Reverse of the IFNode inside the classifier: from arrive ;
[0092] Using alternative gradients, the intermediate layers of the classifier adjust their input. Error signal:
[0093]
[0094] in For the first The error signal of the linear output layer. Under IF dynamics, it can be... Calculated via backward time recursion (BPTT):
[0095] Define the membrane potential error signal:
[0096]
[0097] Time recursion relationship (from) forward):
[0098]
[0099] If a non-leaking IF model is used and the reset item is... If the form appears, the impact of the reset on future time points must also be added; under the alternative gradient approximation, the following stable notation is often used:
[0100]
[0101] (Vector / matrix form, (This indicates element-wise multiplication).
[0102] Then there is:
[0103]
[0104] Substituting it into the convolution gradient formula yields the result. The complete gradient accumulation expression.
[0105] After obtaining the gradients for each layer, the parameters are updated using any standard optimizer (such as SGD / Adam).
[0106]
[0107] Adaptive methods such as Adam update weights according to their built-in first and second moment estimation update rules.
[0108] The `reset()` function in the code is used to return all internal states (such as membrane potential) of a `neuron.BaseNode` (IF / LIF node). (Accumulated distribution records, etc.) are reset to zero. Mathematically, this is equivalent to setting the initial membrane electrode position for each sample or mini-batch as follows:
[0109]
[0110] To make the technical means, innovative features, objectives and effects of this invention clearer, the following describes in detail a method for recognizing gestures based on electromyographic signals according to a convolutional spiking neural network (CSNN).
[0111] <Example>
[0112] like Figure 1-4 As shown, the present invention proposes a CSNN-based electromyographic signal gesture recognition method, which includes the following steps:
[0113] S1: Electromyography signal acquisition and preprocessing
[0114] Multi-channel surface electromyography (EMG) signals are synchronously sampled using a high-precision acquisition module, with a sampling frequency of [value missing]. The acquired raw signal is denoted as:
[0115]
[0116] in, For the number of channels, Where is the number of sampling points; E is the entire set of electromyographic signal data collected. Indicates the first A signal vector acquired at each sampling time. This indicates that the signal vector belongs to The real space of dimension, that is, Each channel. The corresponding set of label sequences. for:
[0117]
[0118] in, One indicates a resting state; the rest are gesture categories.
[0119] To improve the training efficiency and signal stationarity of the model, the following preprocessing steps are performed on the signal:
[0120] 1. Remove DC offset (DC Offset Removal);
[0121] 2. Bandpass filtering (20–450 Hz) to eliminate noise;
[0122] 3. Normalization processing to ensure that the signal amplitude of each channel is within the range of interval;
[0123] 4. Sliding window splitting.
[0124] Sliding window uses length Step length Its windowing formula is:
[0125]
[0126]
[0127] When the proportion of non-zero labels in the window At that time, window samples are retained. The final result is the window sample set. .
[0128] S2: Sample Balancing and Feature Enhancement
[0129] Since the number of resting samples is significantly higher than that of action samples, this invention adopts a two-way balancing strategy of "resting sample downsampling + action sample oversampling" to maintain the balance of sample distribution.
[0130] Let the set of resting class samples be... Action-based Then, a portion of the resting samples are randomly retained:
[0131]
[0132] in This represents the downsampling ratio. If there are fewer than 200 action-type samples, random copying will be performed.
[0133]
[0134] The final balanced sample set is obtained:
[0135]
[0136] This strategy effectively avoids time-domain distortion caused by traditional interpolation methods (such as SMOTE).
[0137] S3: Constructing a Convolutional Spike Neural Network (CSNN) model
[0138] The CSNN model structure proposed in this invention is as follows: Figure 2 As shown, it includes three one-dimensional convolutional layers, a spiking neuron layer, and a fully connected classifier. The network structure is as follows:
[0139] Feature extraction layer:
[0140]
[0141] Category layer:
[0142]
[0143]
[0144] in, The membrane potential update rule for an integral-firing spiking neuron is as follows:
[0145]
[0146]
[0147] in For Heaviside step function, This is the discharge threshold.
[0148] To achieve gradient propagation, a sigmoid-type surrogate gradient is used to approximate backpropagation.
[0149] S4: Model Training and Optimization
[0150] Using the cross-entropy loss function:
[0151]
[0152] Using the Adam optimizer, the initial learning rate is... The seed value decreases by 0.5 every 15 iterations. To ensure reproducibility, a fixed random seed and a deterministic cuDNN mode are used. During training, the following strategies are employed to improve model stability:
[0153] Adaptive learning rate adjustment;
[0154] Optimal model weights are saved;
[0155] Batch latency monitoring and optimization (average inference latency < 10ms).
[0156] S5: Experimental Verification
[0157] On the 64-channel sEMG dataset, the CSNN model proposed in this invention achieves a validation set accuracy of over 95% with an average inference latency of less than 10ms under a 500-point downsampling window. It maintains stable performance in multiple experiments, verifying the model's high robustness and reproducibility.
[0158] Functions and effects of the embodiments
[0159] Compared with existing deep learning gesture recognition methods, this invention has significant advantages in feature extraction efficiency, computational energy consumption, recognition accuracy, and system portability. By introducing a three-layer convolutional pulse structure, hierarchical encoding of the spatiotemporal features of electromyographic signals is achieved, effectively enhancing feature representation and action recognition accuracy. Based on an event-driven pulse computation mechanism, this invention updates only the neurons that generate pulses during the inference phase, significantly reducing redundant computation and achieving real-time recognition with low latency and low power consumption. Experimental results show that the validation set accuracy of this method can stably exceed 90% under downsampling window conditions, demonstrating good robustness and generalization ability under different sampling conditions and subjects. Furthermore, by fixing the random seed and enabling deterministic computation mode, the reproducibility of the model training and inference process is ensured.
[0160] This invention is not limited to the above embodiments. Based on the technical solutions disclosed herein, those skilled in the art can make some substitutions and modifications to some of the technical features without creative effort, and all such substitutions and modifications are within the protection scope of this invention.
Claims
1. A gesture recognition method based on a convolutional spiking neural network, which preprocesses electromyographic signals and inputs them into a convolutional spiking neural network for gesture recognition; wherein the convolutional spiking neural network includes a feature extraction layer and a fully connected classification layer; The feature extraction layer consists of three layers. The first feature extraction layer consists of a first convolutional layer, a first batch normalization layer, and a first IFNode layer connected in sequence. The second feature extraction layer consists of a second convolutional layer, a second batch normalization layer, a second IFNode layer, and a max pooling layer connected in sequence. The third feature extraction layer consists of a third convolutional layer, a third batch normalization layer, a third IFNode layer, and an adaptive average pooling layer connected in sequence. The first, second, and third feature extraction layers are connected in sequence. The preprocessed electromyographic signal is input to the first feature extraction layer, and the third feature extraction layer outputs the feature extraction result. The fully connected classification layer includes a first fully connected layer connected sequentially, The system consists of a regularization layer, a fourth IFNode layer, a second fully connected layer, and a Softmax activation layer. The first fully connected layer receives the feature extraction result from the feature extraction layer, and the Softmax activation layer outputs the gesture recognition result. in, This represents an integral-firing spiking neuron, whose membrane potential update rule is as follows: in For Heaviside step function, Where W is the discharge threshold and V is the weight, V is the discharge threshold. t For neurons at time The membrane potential, V t-1 S is the membrane potential at the previous moment. t For a moment Neuron's impulse output, S t-1 This represents the pulse output of the neuron at the previous moment; Gradient propagation in convolutional spiking neural networks uses a sigmoid gradient instead of a gradient.
2. The gesture recognition method based on a convolutional spiking neural network according to claim 1, characterized in that: During the training of a convolutional spiking neural network, data sample balancing and enhancement are performed on its sample set. A two-way balancing strategy of downsampling of resting classes and oversampling of action classes is used to maintain the balance of sample distribution in the sample set.
3. The gesture recognition method based on a convolutional spiking neural network according to claim 1, characterized in that: The following strategy is used when training a convolutional spiking neural network: Employ the cross-entropy loss function; Using the Adam optimizer, the initial learning rate is... And it decays by 0.5 every 15 iterations.
4. The gesture recognition method based on a convolutional spiking neural network according to claim 3, characterized in that: Set a fixed random seed and a deterministic cuDNN mode.
5. The gesture recognition method based on a convolutional spiking neural network according to claim 4, characterized in that: During training, the following strategies are used to improve model stability: Adaptive learning rate adjustment, optimal model weight saving, and batch latency monitoring and optimization result in an average inference latency of <10ms.
6. The gesture recognition method based on a convolutional spiking neural network according to claim 1, characterized in that: Electromyography signal preprocessing includes the following steps: Remove DC bias; Bandpass filtering of 20–450 Hz to eliminate noise; Normalization processing ensures that the signal amplitude of each channel is within the range of interval; Sliding window segmentation, the sliding window uses length Step length .
7. A gesture recognition device based on a convolutional spiking neural network, comprising an electromyography signal preprocessing unit and a convolutional spiking neural network unit; The electromyography signal preprocessing unit is used to preprocess the electromyography signals; The convolutional spiking neural network unit is used to process the preprocessed electromyographic signals to obtain gesture recognition results; A convolutional spiking neural network unit includes: a feature extraction layer and a fully connected classification layer; The feature extraction layer consists of three layers. The first feature extraction layer comprises a first convolutional layer, a first batch normalization layer, and a first IFNode layer connected in sequence. The second feature extraction layer comprises a second convolutional layer, a second batch normalization layer, a second IFNode layer, and a max pooling layer connected in sequence. The third feature extraction layer comprises a third convolutional layer, a third batch normalization layer, a third IFNode layer, and an adaptive average pooling layer connected in sequence. The first, second, and third feature extraction layers are connected in sequence. The preprocessed thickness electromyography signal is input to the first feature extraction layer, and the third feature extraction layer outputs the feature extraction result. The fully connected classification layer includes a first fully connected layer connected sequentially, The system consists of a regularization layer, a fourth IFNode layer, a second fully connected layer, and a Softmax activation layer. The first fully connected layer receives the feature extraction result from the feature extraction layer, and the Softmax activation layer outputs the gesture recognition result. in, This represents an integral-firing spiking neuron, whose membrane potential update rule is as follows: in For Heaviside step function, Where W is the discharge threshold and V is the weight, V is the discharge threshold. t For neurons at time The membrane potential, V t-1 S is the membrane potential at the previous moment. t For a moment Neuron's impulse output, S t-1 This represents the pulse output of the neuron at the previous moment; Gradient propagation in convolutional spiking neural network units uses a sigmoid gradient instead of a gradient.