Sparse attention and distillation-based sEMG hand joint action continuous estimation method
By optimizing the BERT model using sparse attention and distillation techniques, the high computational complexity of the sEMG hand joint motion estimation method is solved, achieving efficient long sequence processing and real-time response, thus expanding the application scope of portable devices.
Patent Information
- Application Number
- CN202511859011.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-10
- Publication Date
- 2026-03-17
AI Technical Summary
Existing BERT-based sEMG hand joint motion estimation methods suffer from high computational complexity and hardware deployment costs when processing long sequence data. Furthermore, they exhibit redundant computations in cross-subject signal processing, failing to meet the response requirements of real-time human-computer interaction. They also lack sufficient optimization of sparsity characteristics, limiting their application in portable devices.
Employing sparse attention and distillation techniques, key queries are filtered through the ProbSparse self-attention mechanism. By combining attention distillation and replica stacking, the model architecture of the Transformer encoder is optimized, reducing time and space complexity and adapting to long sequence sEMG data processing.
It improves computational efficiency by more than 20%, reduces hardware resource consumption by 30%, shortens inference time by 25%-30%, meets the needs of real-time human-computer interaction, is compatible with portable devices, and has performance comparable to or better than the traditional BERT model.
Smart Images
Figure CN121682169A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical field of motion continuity estimation methods, and more particularly to a sEMG hand joint motion continuity estimation method based on sparse attention and distillation. Background Technology
[0002] Surface electromyography (sEMG) signals, as important biosignals reflecting human movement intentions, have significant application value in human-computer interaction scenarios such as prosthetic control and robot-assisted surgery. Deep learning methods have been widely used for sEMG signal decoding. Among them, the BERT model, with its bidirectional attention mechanism, can effectively capture the spatiotemporal features of sEMG signals and shows advantages in cross-subject hand joint estimation. However, existing BERT-based sEMG hand joint motion estimation methods still have significant shortcomings: This type of model uses full self-attention computation, with both time and space complexity being O(L²) (where L is the sequence length). It is difficult to adapt to long sequence sEMG data processing scenarios, and the hardware deployment cost is high. At the same time, there is a lot of redundant computation when processing cross-subject sEMG signals, resulting in low model training and inference efficiency, which cannot meet the response requirements of real-time human-computer interaction. Furthermore, the sparsity characteristics of the attention mechanism are not fully optimized, and the balance between long-range dependency capture and computational efficiency is insufficient, which further limits its application in portable devices.
[0003] Although the sparse attention and self-attention distillation techniques proposed by the Informer model can reduce the complexity to O(LlogL) by filtering key query-key pairs and compressing feature dimensions layer by layer, providing a new approach to optimizing the BERT model, the existing technology has not yet incorporated this sparsification and distillation strategy into the BERT model for sEMG hand joint estimation. As a result, the efficiency defects of the BERT model have not been effectively improved, thus limiting its large-scale application in real-world scenarios. Summary of the Invention
[0004] To address the technical problems mentioned in the background section, this invention provides a continuous estimation method for hand joint movements based on sEMG signals and distillation. This invention addresses the need for continuous estimation of hand joint movements using sEMG signals by integrating the core concepts of sparse attention and distillation from Informer, building upon the traditional BERT model.
[0005] The technical means employed in this invention are as follows: A continuous estimation method for hand joint movements based on sEMG using sparse attention and distillation includes the following steps: Step 1: Data Acquisition and Preprocessing; Acquire sEMG signals with a sampling rate of 2kHz, extract RMS features from the sEMG signals using a sliding window, and then use... The RMS features were normalized; hand joint angle data with a sampling rate of 500Hz were acquired, upsampled to 2kHz and aligned with the sEMG signal, and then processed using... Normalization compresses the hand joint angles to a certain value. between; Step 2: Input projection and position code; (The text abruptly ends here, likely due to an incomplete sentence or a formatting error.) The normalized sEMG signal is projected from 12 dimensions to 128 dimensions through a linear layer, and then added to the position encoding vector to give position information; Step 3: Process the projected data based on the model architecture that integrates ProbSparse self-attention mechanism, attention distillation and copy stacking, and output the continuous angle values of key hand joints; Step 4: Process the 150-dimensional data after stacking the replicas using a regression head to map the 150-dimensional data into 22 hand joint angle data.
[0006] Furthermore, the ProbSparse self-attention mechanism, based on the long-tail distribution characteristics of sEMG signal attention scores, filters the Top-u key queries through a maximum-means metric mechanism; wherein, , Indicates the sampling factor. Indicates the length of the data sample.
[0007] Furthermore, the calculation formula for the maximum-mean metric mechanism is as follows: ; in, Indicates the first One query vector; Indicates the first One key vector; Indicates the dimension of a vector; This indicates the number of key vectors.
[0008] Furthermore, the attention distillation process is as follows: a distillation operation is introduced between the stacked layers of the Transformer encoder, that is, between each attention block of BERT. After the output of each layer is processed by 1D convolution and ELU activation function, the length of the feature sequence is halved by max pooling with a stride of 2, retaining the dominant attention features, while reducing the input dimension of subsequent layers. The distillation formula for attention distillation is: ; in, express The result is calculated after passing through an attention module.
[0009] Furthermore, the replica stacking includes the following steps: Copy I: Take the original data sample with length L=200, and after two attention distillation processes, the feature sequence length is reduced to 100 and 50 respectively; Copy II: Take the last 100 data points from the original data sample, perform attention distillation once, and reduce the feature sequence length to 50. Copy III: Take the last 50 data points from the original data sample, process them only through the ProbSparse self-attention mechanism, and keep the feature sequence length at 50. The output data from the three copies are concatenated along the dimensions, resulting in a total length of 150, which is then used as input for subsequent modules.
[0010] Furthermore, the aforementioned Normalization of laws .
[0011] Furthermore, the formula for the regression head processing is: .
[0012] Compared with the prior art, the present invention has the following advantages: The method of this invention reduces the time complexity from O(L²) to O(L log L) and optimizes the space complexity to O(L log L), improving the computational efficiency of long sequence sEMG data processing by more than 20% and reducing hardware resource consumption by 30%.
[0013] Under the same CPU (Intel i7-10875H) environment, the inference time of the method of this invention is reduced by 25%-30% compared with the traditional BERT model, meeting the response requirements of real-time human-computer interaction (single inference time ≤35ms). Meanwhile, in cross-subject scenarios, the model's correlation coefficient... Normalized root mean square error It performs comparably to the traditional BERT model, but with better stability.
[0014] Meanwhile, the method of the present invention reduces the requirements for hardware, can be adapted to the embedded deployment of portable wearable devices, and expands the application scope in scenarios such as prosthetic control and rehabilitation training. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 This is a diagram of the overall framework of the model of this invention.
[0017] Figure 2 This is a flowchart illustrating the operation of the self-attention distillation module of the present invention.
[0018] Figure 3 This is a schematic diagram of the coefficient attention module structure of the present invention. Detailed Implementation
[0019] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0020] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0021] like Figure 1-3 As shown, this invention provides a continuous estimation method for hand joint movements based on sparse attention and distillation using sEMG, comprising the following steps: Step 1: Data Acquisition and Preprocessing; Acquire sEMG signals with a sampling rate of 2kHz, totaling 12 channels. Extract RMS features from the sEMG signals using a 100ms sliding window (0.5ms step). Law normalization ( The RMS features are processed to amplify low-amplitude effective information and avoid feature distortion caused by linear normalization; hand joint angle data with a sampling rate of 500Hz are acquired, totaling 22 channels, and upsampled to 2kHz to align with the sEMG signal. Normalization compresses the hand joint angles to a certain value. Between these values, avoid large data values from affecting the model.
[0022] Step 2: Input projection and position code; (The text abruptly ends here, likely due to an incomplete sentence or a formatting error.) The normalized sEMG signal is projected from 12 dimensions to 128 dimensions through a linear layer to enhance its feature representation capability, and then added to the position encoding vector to give position information; Step 3: Process the projected data based on the model architecture that integrates ProbSparse self-attention mechanism, attention distillation and copy stacking, and output the continuous angle values of key hand joints.
[0023] In a preferred embodiment, the ProbSparse self-attention mechanism in this application, based on the long-tail distribution characteristics of sEMG signal attention scores, filters the Top-u key queries through a maximum-means metric mechanism; wherein, , Indicates the sampling factor. Indicates the length of the data sample. For each key query, only the attention weights between the key query and all keys are calculated, replacing the traditional full-scale self-attention calculation, reducing the time and space complexity of single-layer attention from O(L²) to O(L log L).
[0024] Standard self-attention calculates attention weights for all queries and keys. Sparse self-attention mechanisms use random sampling. The similarity between query Q and key K is calculated, and the maximum-mean similarity metric is used to select the most similar query. One key query. In this application, the calculation formula for the maximum-mean metric mechanism is: ; in, Indicates the first One query vector; Indicates the first One key vector; Indicates the dimension of a vector; This represents the number of key vectors. This is the maximum. A sparse query matrix is composed of key queries. Then, sparse self-attention calculation is performed: .
[0025] Distillation is introduced between the stacked layers of the Transformer encoder, specifically between each attention block in BERT. After each layer's output undergoes 1D convolution (kernel width = 3) and ELU activation, the feature sequence length is halved using max pooling with a stride of 2. This preserves the dominant attention features while reducing the input dimensionality of subsequent layers. The distillation formula for attention distillation is: ; in, express The result is calculated after passing through an attention module.
[0026] In a preferred embodiment of this application, the copy stacking includes the following steps: Copy I: Take the original data sample with length L=200, and after two attention distillation processes, the feature sequence length is reduced to 100 and 50 respectively; Copy II: Take the last 100 data points from the original data sample, perform attention distillation once, and reduce the feature sequence length to 50. Copy III: Take the last 50 data points from the original data sample, process them only through the ProbSparse self-attention mechanism, and keep the feature sequence length at 50. The output data from the three copies is concatenated along its dimensions, resulting in a total length of 150, which is then used as input for subsequent modules. This approach allows us to focus not only on the overall information of the long sequence but also on its recent details.
[0027] Step 4: Process the 150-dimensional data after stacking the replicas using a regression head, mapping the 150-dimensional data into 22 hand joint angle data. The formula for the regression head processing is: .
[0028] Example 1 As one embodiment of this application, the following steps are included: Data preprocessing: The dataset used was NinaproDB2. sEMG signals and hand joint angle data from 38 participants were selected. The sEMG signal sampling rate was 2kHz, and RMS features were extracted using a 100ms sliding window (0.5ms step). Law normalization ( Processing RMS features amplifies effective information with low amplitude values and avoids feature distortion caused by linear normalization; dividing the training and test sets, and processing data across different subject scenarios... The dataset is divided into two parts: a training set containing data from 27 participants and a test set containing data from 11 participants. The input sequence length is... (Corresponds to 100ms).
[0029] Model parameter settings: number of encoder layers Bullish attention head count Embedded Dimensions ; Sparse attention sampling factor Key query count Distillation module: 1D convolution kernel count = 256, maximum pooling stride = 2, multi-scale replication is... L=100 and L=50; the optimizer is Adam, the initial learning rate is 0.0001, training is performed for 400 epochs, and the learning rate is halved after 200 epochs; the loss function is MSE.
[0030] The effectiveness of this invention was verified by comparing its complexity. When processing long sequences of L=1000, the model of this invention took 12ms for a single forward computation, while the traditional BERT model took 45ms, resulting in a 73.3% reduction in complexity. Performance comparison: In cross-subject testing, the model of this invention had CC=0.843 and NRMSE=0.0835, while the traditional BERT model... With NRMSE = 0.0881, the performance is comparable but more efficient; stability comparison: the joint angle prediction of the model in this invention has an unbiased standard deviation. , It outperforms the traditional BERT model. , ), with smaller fluctuations.
[0031] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the above embodiments of the present invention, the descriptions of each embodiment have their own emphasis; parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the disclosed technical content in the several embodiments provided in this application can be implemented in other ways.
[0032] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A sparse attention and distillation based sEMG hand joint motion continuous estimation method, characterized in that, The method comprises the following steps: Step 1, data collection and preprocessing; collect sEMG signals with a sampling rate of 2 kHz, extract the RMS features in the sEMG signals through a sliding window, and then normalize the RMS features using the law The hand joint angle data with a sampling rate of 500 Hz is collected, up-sampled to 2 kHz to align with the sEMG signal, and normalized to compress the hand joint angle in between 0 and 1. between 0 and 1. Step 2, input projection and position encoding; the normalized sEMG signal is projected from 12 dimensions to 128 dimensions by a linear layer, and then added with a position encoding vector to give position information; Step 2, input projection and position encoding; the normalized sEMG signal is projected from 12 dimensions to 128 dimensions by a linear layer, and then added with a position encoding vector to give position information; Step 3, the model architecture based on fusion ProbSparse self-attention mechanism, attention distillation and copy stacking processes the projected data, and outputs continuous angle values of hand key joints; Step 4, the 150-dimensional data after copy stacking is processed by a regression head, and the 150-dimensional data is mapped into 22 hand joint angle data.
2. The sEMG hand joint motion continuous estimation method based on sparse attention and distillation according to claim 1, characterized in that, The ProbSparse self-attention mechanism is based on the long-tail distribution characteristics of the sEMG signal attention score, and the Top-u key queries are screened through a maximum-mean measurement mechanism. , denotes a sampling factor, denotes a data sample length.
3. The sEMG hand joint motion continuous estimation method based on sparse attention and distillation according to claim 2, characterized in that, The calculation formula of the maximum-mean measure mechanism is: ； wherein, denotes the th query vector; denotes the th key vector; denotes the vector dimension; denotes the number of key vectors.
4. The sEMG hand joint motion continuous estimation method based on sparse attention and distillation according to claim 1, characterized in that, The implementation process of the attention distillation is that a distillation operation is introduced between the stacked layers of the Transformer encoder, that is, between each attention block of Bert, after each layer output is processed by a 1D convolution and an ELU activation function, the feature sequence length is halved by maximum pooling with a step of 2, dominant attention features are retained, and the input dimension of the subsequent layer is reduced; The distillation formula of the attention distillation is: ； wherein represents the result of a layer of attention modules computation.
5. The sEMG hand joint motion continuous estimation method based on sparse attention and distillation according to claim 1, characterized in that, The copy stacking comprises the following steps: Copy I: the original data sample with a length of L=200 is taken, the feature sequence length is sequentially reduced to 100 and 50 after twice attention distillation processing; Copy II: the last 100 data of the original data sample are taken, and the feature sequence length is reduced to 50 after once attention distillation processing; Copy III: the last 50 data of the original data sample are taken, and the feature sequence length is kept to 50 only after ProbSparse self-attention mechanism processing; The output data of the three copies are spliced according to the dimension, and the total length after splicing is 150, serving as the input of a subsequent module.
6. The sEMG hand joint motion continuous estimation method based on sparse attention and distillation according to claim 1, characterized in that, The The law is normalized in .
7. The sEMG hand joint motion continuous estimation method based on sparse attention and distillation according to claim 1, characterized in that, The formula of the regression head processing is: 。