Muscle fatigue state detection method and system based on surface myoelectricity and pulse waves
By converting surface electromyography signals into spatiotemporal graph representations and combining singular spectrum analysis, the interactive multi-head attention mechanism of the Transformer encoder is designed, and the PPG signal is integrated into the sEMG spatiotemporal graph is solved, which is difficult to effectively integrate multimodal signals in the existing technology, and a more accurate muscle fatigue assessment and a more general integration method are achieved.
Patent Information
- Application Number
- CN202510027335.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-08
- Publication Date
- 2025-05-09
AI Technical Summary
The prior art is difficult to effectively integrate multimodal information of surface electromyography and pulse wave signals in muscle fatigue assessment, resulting in limited accuracy of muscle fatigue estimation and lack of a general and effective integration method to adapt to new signal modes.
By converting surface electromyography signals into spatiotemporal graph representations and combining singular spectrum analysis to reduce noise, the interactive multi-head attention mechanism of the Transformer encoder is designed, the PPG signal is integrated into the sEMG spatiotemporal graph, and the deep learning algorithm is used to perform multimodal fusion and feature extraction, and the muscle fatigue classification artificial intelligence model is trained.
Deep fusion of multimodal signals is achieved, factors affecting muscle fatigue are fully considered, the accuracy of muscle fatigue classification is improved, and a general integration method is provided to adapt to new signal modes.
Smart Images
Figure CN119949852A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of surface electromyography and pulse wave signal processing, and more particularly to a muscle fatigue state detection method and system based on surface electromyography and pulse wave. Background Art
[0002] In the field of sports science and health management, accurate assessment of muscle fatigue is of vital importance for preventing sports injuries, optimizing training effects, and protecting individual health. The traditional method of using a single-modality signal to assess muscle fatigue has certain limitations. The multimodal detection technology of surface electromyography and pulse wave has brought new breakthroughs in this field.
[0003] When processing multi-channel sEMG, traditional methods often focus on the analysis of information in a single domain and cannot effectively integrate the time domain and decomposition domain information at the same time. This causes some key information to be ignored during the processing, resulting in an incomplete understanding of sEMG signals, which in turn affects the accuracy of muscle fatigue estimation. For example, when focusing only on time domain features, the pattern changes related to muscle fatigue hidden in the decomposition domain may be missed.
[0004] At present, most studies simply splice the features of the two signals or process them separately and then integrate them when combining sEMG and PPG signals. There is no effective mechanism to deeply integrate the information of the two modalities. This method cannot fully explore the intrinsic relationship between the two signals, resulting in the inability to give full play to the advantages of multimodal information when estimating muscle fatigue. For example, the response information of the cardiovascular system to muscle fatigue reflected in the PPG signal is not organically combined with the electrical activity information of the muscle itself in the sEMG signal, which limits the overall classification accuracy. At the same time, the existing technology lacks a general and effective integration method when facing other related signal modalities that may be added. Once a new modality is introduced, it is often necessary to redesign the entire system architecture to adapt, which is not only time-consuming and labor-intensive, but also easily leads to increased complexity and decreased stability of the system. For example, when trying to combine new signals with sEMG and PPG signals, it is difficult to effectively integrate their information into the muscle fatigue estimation model due to the lack of a suitable integration mechanism.
[0005] In addition, existing models often use a single feature extraction method when processing sEMG spatiotemporal modalities and PPG one-dimensional signal modalities, and are unable to fully exploit the multiple features of the two modalities at the same time. At the same time, traditional machine learning models or simple neural network models are not optimized specifically for the characteristics of multimodal data when processing multimodal data. They find it difficult to effectively learn the complex interactions between different modalities, resulting in an inability to fully utilize the complementarity of multimodal information when classifying or predicting muscle fatigue. Summary of the invention
[0006] The present invention expresses multi-channel sEMG through the proposed space-time diagram, while considering the sEMG time domain and decomposition domain information, avoiding the loss of relevant information, and combining the nonlinear adaptive processing method of singular spectrum analysis to reduce the space-time diagram noise.
[0007] The present invention also uses PPG as auxiliary and supplementary information to enhance the representation of sEMG spatiotemporal graphs and integrate PPG modalities into spatiotemporal graph modalities by designing an interactive multi-head attention mechanism of the Transformer encoder. Since muscle fatigue is closely related to both sEMG and PPG, by combining the features of the two modal signals, the factors affecting muscle fatigue can be fully considered to achieve a more accurate classification effect. If other related signals are added, the information of other related modalities can be asynchronously integrated into the spatiotemporal graph modality through the interactive multi-head attention mechanism, thus achieving a more accurate classification effect through multimodal signal feature extraction. At the same time, the Transformer encoder and LSTM model are combined to fully and simultaneously extract the features of the sEMG spatiotemporal graph modality and the PPG one-dimensional signal modality, and fully learn the features of multiple modalities.
[0008] Specifically, the present invention solves the problems existing in the above background through the following technical solutions:
[0009] One of the purposes of the present invention is achieved by the following technical solution:
[0010] The muscle fatigue state detection method based on surface electromyography and pulse wave includes the following steps:
[0011] S1. Use a multimodal sensor device to obtain the user's surface electromyography (sEMG) and photoplethysmography (PPG) signals during activity.
[0012] S2, preprocessing the surface electromyography (sEMG) and photoplethysmography (PPG);
[0013] S3, converting the surface electromyography signal sEMG into a spatiotemporal diagram;
[0014] S4. Use deep learning algorithms to perform multimodal fusion and feature extraction on processed surface electromyography (sEMG) and photoplethysmography (PPG) signals, and train an artificial intelligence model for muscle fatigue classification.
[0015] S5. Evaluate the user's muscle fatigue status through a muscle fatigue classification artificial intelligence model.
[0016] Furthermore, in S1, the specific steps of obtaining the surface electromyography signal sEMG and photoplethysmography PPG of the user when the user is active include: using the multi-channel surface electrodes of the multimodal sensor to collect N channels of surface electromyography signal sEMG, and using the optical pulse measurement device to collect photoplethysmography PPG; setting the window length L for the amount of data processed at one time and the sliding step size s, and performing data cutting on the surface electromyography signal sEMG and photoplethysmography PPG.
[0017] Furthermore, in S2, the specific steps of preprocessing include: removing original power frequency interference and denoising at the hardware and software levels from the surface electromyography signal sEMG.
[0018] Furthermore, the specific steps of the preprocessing also include: removing the power frequency interference of the original PPG and denoising at the hardware and software levels.
[0019] Furthermore, the photoplethysmography PPG signal after removing the power frequency interference of the original PPG and the denoising at the hardware and software levels is further denoised by performing empirical mode decomposition.
[0020] Furthermore, the surface electromyography signals sEMG of all channels after the original power frequency interference and the hardware and software level denoising are removed from the surface electromyography signals sEMG are decomposed, and for all the N channels, singular value decomposition is used to perform signal decomposition.
[0021] Furthermore, in S3, for all N channels of surface electromyography (sEMG) signals, each channel selects to retain the first R components; a space-time diagram is set, the width of the space-time diagram is the time variable, the height is the channel variable of the surface electromyography (sEMG), and the color channel is R SSA components, so as to obtain a space-time diagram of size L*N*R.
[0022] Furthermore, in S5, the user's data is input into the model trained in S4, the user's muscle fatigue state is fed back through the user interface and the muscle fatigue state classification is output.
[0023] The second object of the present invention is achieved by adopting the following technical solution:
[0024] A muscle fatigue state detection system based on surface electromyography and pulse wave is applied to a muscle fatigue state detection method based on surface electromyography and pulse wave as one of the purposes of the present invention. The muscle fatigue state detection system includes: a processor and a memory, the processor is communicatively connected to the memory, the memory stores a computer program, and the processor executes the above-mentioned muscle fatigue state detection method according to the computer program.
[0025] The third object of the present invention is achieved by adopting the following technical solution:
[0026] A computer-readable storage medium stores program data, and the program data is used to execute the muscle fatigue state detection method described above.
[0027] Compared with the prior art, the present invention has the following beneficial effects:
[0028] 1. The multi-channel sEMG is expressed through the proposed space-time diagram, while considering the sEMG time domain and decomposition domain information, avoiding the loss of relevant information, and combining the nonlinear adaptive processing method of singular spectrum analysis to reduce the noise of the space-time diagram.
[0029] 2. Design the interactive multi-head attention mechanism of the Transformer encoder, use PPG as auxiliary and supplementary information, enhance the representation of the sEMG spatiotemporal graph, and integrate the PPG modality into the spatiotemporal graph modality. Since muscle fatigue is closely related to both sEMG and PPG, by combining the features of the two modal signals, the factors affecting muscle fatigue can be fully considered to achieve a more accurate classification effect. If other related signals are added, the information of other related modalities can be asynchronously integrated into the spatiotemporal graph modality through the interactive multi-head attention mechanism, thus achieving a more accurate classification effect through multimodal signal feature extraction.
[0030] 3. Combine the Transformer encoder and LSTM model to fully and simultaneously extract the features of the sEMG spatiotemporal graph modality and the PPG one-dimensional signal modality, and fully learn the features of multiple modalities.
[0031] 4. In terms of application, the present invention can provide athletes and fitness enthusiasts with more scientific training plans in terms of personalized training guidance, adjust training intensity and methods according to real-time monitoring results, and avoid excessive fatigue and injury; in terms of rehabilitation treatment effect evaluation, it can help rehabilitation therapists better understand the patient's rehabilitation progress and optimize treatment plans; in disease diagnosis and monitoring, it can assist in the diagnosis of some diseases related to muscle function, such as neuromuscular diseases, and monitor the development of the disease and the treatment effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 The present invention is a flow chart of the muscle fatigue state detection method based on surface electromyography and pulse wave. DETAILED DESCRIPTION
[0033] The present invention is further described below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form a new embodiment.
[0034] Embodiment 1
[0035] The present invention provides a preferred embodiment, a muscle fatigue state detection method based on surface electromyography and pulse wave, such as Figure 1 As shown, one of the purposes of the present invention is achieved by the following technical solution:
[0036] One of the purposes of the present invention is achieved by the following technical solution:
[0037] The muscle fatigue state detection method based on surface electromyography and pulse wave includes the following steps:
[0038] S1. Use a multimodal sensor device to obtain the user's surface electromyography (sEMG) and photoplethysmography (PPG) signals during activity.
[0039] S2, preprocessing the surface electromyography (sEMG) and photoplethysmography (PPG);
[0040] S3, converting the surface electromyography signal sEMG into a spatiotemporal diagram;
[0041] S4. Use deep learning algorithms to perform multimodal fusion and feature extraction on processed surface electromyography (sEMG) and photoplethysmography (PPG) signals, and train an artificial intelligence model for muscle fatigue classification.
[0042] S5. Evaluate the user's muscle fatigue status through a muscle fatigue classification artificial intelligence model.
[0043] Furthermore, in S1, the specific steps of obtaining the surface electromyography signal sEMG and photoplethysmography PPG of the user when the user is active include: using the multi-channel surface electrodes of the multimodal sensor to collect N channels of surface electromyography signal sEMG, and using the optical pulse measurement device to collect photoplethysmography PPG; setting the window length L for the amount of data processed at one time and the sliding step size s, and performing data cutting on the surface electromyography signal sEMG and photoplethysmography PPG.
[0044] Furthermore, in S2, the specific steps of preprocessing include: removing original power frequency interference and denoising at the hardware and software levels from the surface electromyography signal sEMG.
[0045] For the original sEMG signal of each channel, the spectrum analysis is performed. For the sEMG signal x of the kth channel ori_sEMG,k (t), the frequency of the interfering sinusoidal signal is calculated using the correction method, and then a cascaded fundamental and harmonic notch filter is designed to remove the power frequency interference of the original sEMG.
[0046] For sEMG with power frequency interference removed, a bandpass filter is designed to set the upper and lower boundary frequencies of the passband fL and f H , and obtain the sEMG signal x after preliminary denoising pre_sEMG,k (t);
[0047] Where H1(f) is the frequency domain representation of the bandpass filter, expressed as
[0048] Furthermore, the specific steps of the preprocessing also include: removing the power frequency interference of the original PPG and denoising at the hardware and software levels.
[0049] For the original PPG signal x ori_PPG (t) Perform spectrum analysis and use the correction method to find the frequency of the interfering sinusoidal signal. Then design a cascaded fundamental and harmonic notch filter to remove the original PPG power frequency interference.
[0050] For the PPG with power frequency interference removed, a bandpass filter is designed and the upper and lower boundary frequencies f′ of the passband are set. L and f′ H , and obtain the PPG signal x after preliminary denoising pre_PPG (t);
[0051] Where H2(f) is the frequency domain representation of the bandpass filter, expressed as
[0052] Furthermore, the photoplethysmography PPG signal after removing the power frequency interference of the original PPG and the denoising at the hardware and software levels is further denoised by performing empirical mode decomposition.
[0053] EMD regards the signal as the sum of a finite number of intrinsic mode functions (IMF). The number of modes does not need to be set manually and depends on the signal. pre_PPG (t) Perform empirical mode decomposition (EMD) to obtain n IMFs, that is, the PPG after power frequency interference and hardware and software level denoising can be expressed as
[0054] where c i is the ith eigenmode function, r n is the nth residual component;
[0055] Set the rules for retaining IMF, and let set I contain all IMFs that meet the rules. Then get the PPG after EMD denoising, expressed as
[0056] Furthermore, the surface electromyography signals sEMG of all channels after the original power frequency interference and the hardware and software level denoising are removed from the surface electromyography signals sEMG are decomposed, and for all the N channels, singular value decomposition is used to perform signal decomposition.
[0057] Decompose the sEMG signals of all channels. For the kth channel, pre_sEMG,k (t) Use singular value decomposition to decompose the signal. The one-dimensional finite time series of length L is expressed as in(.) T As the transpose operator, this signal can be represented by SSA as the sum of its decomposed components.
[0058] Choose the appropriate window length Lag vector x pre_sEMG,i ,get Vector There is 1≤j≤L. Here It is a satisfying These lag vectors are then used as column vectors to form a trajectory matrix X = [X1, X2, …, X κ ], that is:
[0059]
[0060] Note that X is a Hankel matrix. To avoid rank reduction of the trajectory matrix, we need to satisfy therefore Defined as satisfying An integer.
[0061] The singular value decomposition is performed on the covariance matrix C of X, that is, C = cov{X} = XX T , through singular value decomposition, C can be expressed as C = UΣV T .
[0062] Here, U and V are unit orthogonal matrices, and Σ is a diagonal matrix. X is the sum of all singular value decomposition components, denoted by
[0063]
[0064] in are the diagonal elements of the diagonal matrix Σ, and the corresponding eigenvectors form the column vectors of U. i and v i are the i-th eigenvectors of the left matrix U and the right matrix V respectively. σ is the i-th eigenvector satisfying d i >0 when i is the maximum value.
[0065] For each X iPerform de-Hankelization. The i-th component of the signal can be expressed as
[0066] x i =[x 1,i ,x 2,i ,…,x L,i ].
[0067] Furthermore, in S3, for all N channels of surface electromyography (sEMG) signals, each channel selects to retain the first R components; a space-time diagram is set, the width of the space-time diagram is the time variable, the height is the channel variable of the surface electromyography (sEMG), and the color channel is R SSA components, so as to obtain a space-time diagram of size L*N*R.
[0068] Wherein S4 further includes the following steps:
[0069] S41. For the input spatiotemporal graph, use the VisionTransformer model to extract spatiotemporal graph features. The input image is convolved and blocked to obtain a total of b image blocks with dimensions of w*h*R; then a tiling operation is performed to flatten the width and height of the image block, that is, each image block (w*h*R) is converted into a one-dimensional vector of size 1*(w*h*R), and then the b one-dimensional vectors are concatenated to form a two-dimensional vector of size b*(w*h*R). Then the two-dimensional vector is reduced in dimension using a fully connected layer to obtain a two-dimensional feature of c*d, where d is set manually, to obtain a tiled sequence, and then a classification vector is added to the sequence and a position embedding is added to obtain the input sequence. This sequence is input into the Transformer encoder for feature extraction to obtain features, and the output feature dimension is c*d. The Transformer encoder performs feature extraction including self-attention and multi-head attention. The self-attention process is as follows:
[0070] Assume a i is the i-th input vector, q i , k i and v i are query, key and value vectors respectively. i and k j The similarity between them is calculated as:
[0071]
[0072] Among them, d k It is k j The output vector b i By normalizing through the SoftMax function and S(q i ,k j )get Then with the vector vj Multiply them together to get. It can be expressed as:
[0073]
[0074] The above process can be expressed as
[0075]
[0076] Among them, the output matrix B contains the enhanced feature channel data, Attention(·) represents the self-attention operator, and Q, K, and V are q i , k i and v i The query matrix, key matrix and value matrix are composed of.
[0077] Multi-Head Attention (MHA) is an important component of the Transformer encoder. It is composed of multiple self-attentions, and the output F is expressed as:
[0078] F=Multihead(Q,K,V)=Concat(head1,head2,…,head h )W o
[0079] head i =Attention(QW i Q ,KW i K ,VW i V )
[0080] Among them, Multihead(·) represents the multi-head attention operator, Concat(·) is the matrix concatenation function, head i (i=1,…,h) is the output result of the i-th head, h is the number of heads, W o is the linear transformation of the final output, W i Q ,W i K ,W i V are the linear transformations on the i-th head respectively.
[0081] At this point, the feature extraction of the input spatiotemporal graph is completed through the Transformer encoder.
[0082] S42. For the input PPG, a one-dimensional convolutional neural network (1D-CNN) is used for feature extraction. In order to align the number of spatiotemporal graphs and PPG features, the output feature map dimension is c*d.
[0083] S43. Using PPG as auxiliary and supplementary information is beneficial for muscle fatigue estimation. An auxiliary attention mechanism is used to integrate the PPG modality into the spatiotemporal graph modality by enhancing the representation of each spatiotemporal graph. The basic component of the auxiliary attention mechanism adopted is the interactive multi-head attention layer. The representation of the PPG modality in each input segment is used as the query (Q PPG ), the representation of the spatiotemporal graph modality on each input segment is used as the key (K P-sEMG ) and value (V P-sEMG ), enhanced representation of spatiotemporal graph modality P-sEMG←PPG for
[0084]
[0085] The dimension of the feature map output by the interactive multi-head attention layer is c*d.
[0086] S44. Apply a long short-term memory network (LSTM) model to the feature graph. Treat each row in the feature graph as an element of a sequence as the input of the LSTM model, and set the network structure to map the extracted features to the output categories.
[0087] S45. Perform model training, iterate each batch of the training set, and update the model parameters through back propagation and optimizer.
[0088] Furthermore, in S5, the user's data is input into the model trained in S4, the user's muscle fatigue state is fed back through the user interface and the muscle fatigue state classification is output.
[0089] The second object of the present invention is achieved by adopting the following technical solution:
[0090] A muscle fatigue state detection system based on surface electromyography and pulse wave is applied to a muscle fatigue state detection method based on surface electromyography and pulse wave as one of the purposes of the present invention. The muscle fatigue state detection system includes: a processor and a memory, the processor is communicatively connected to the memory, the memory stores a computer program, and the processor executes the above-mentioned muscle fatigue state detection method according to the computer program.
[0091] The third object of the present invention is achieved by adopting the following technical solution:
[0092] A computer-readable storage medium stores program data, and the program data is used to execute the muscle fatigue state detection method described above.
[0093] The above-mentioned embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and substitutions made by technicians in this field on the basis of the present invention shall fall within the scope of protection required by the present invention.
Claims
1. A muscle fatigue state detection method based on surface electromyography and pulse wave, characterized in that: The following steps are involved: S1. Use a multimodal sensor device to obtain the user's surface electromyography (sEMG) and photoplethysmography (PPG) signals during activity. S2, preprocessing the surface electromyography (sEMG) and photoplethysmography (PPG); S3, converting the surface electromyography signal sEMG into a spatiotemporal diagram; S4. Use deep learning algorithms to perform multimodal fusion and feature extraction on processed surface electromyography (sEMG) and photoplethysmography (PPG) signals, and train an artificial intelligence model for muscle fatigue classification. S5. Evaluate the user's muscle fatigue status through a muscle fatigue classification artificial intelligence model.
2. The muscle fatigue state detection method based on surface electromyography and pulse wave according to claim 1 is characterized in that: In S1, the specific steps of obtaining the surface electromyography signal sEMG and photoplethysmography PPG of the user when the user is active include: using the multi-channel surface electrodes of the multimodal sensor to collect N channels of surface electromyography signal sEMG, and using the optical pulse measurement device to collect photoplethysmography PPG; setting the window length L for the amount of data processed at one time and the sliding step size s, and performing data cutting on the surface electromyography signal sEMG and photoplethysmography PPG.
3. The muscle fatigue state detection method based on surface electromyography and pulse wave according to claim 1 is characterized in that: In S2, the specific steps of preprocessing include: removing the original power frequency interference and denoising the surface electromyography signal sEMG at the hardware and software levels.
4. The muscle fatigue state detection method based on surface electromyography and pulse wave according to claim 3 is characterized in that: The specific steps of the preprocessing also include: removing the power frequency interference of the original PPG and denoising at the hardware and software levels.
5. The muscle fatigue state detection method based on surface electromyography and pulse wave according to claim 4 is characterized in that: The photoplethysmography PPG signal after removing the power frequency interference of the original PPG and denoising at the hardware and software levels is further denoised by empirical mode decomposition.
6. The muscle fatigue state detection method based on surface electromyography and pulse wave according to claim 3 is characterized in that: The surface electromyography signals sEMG of all channels after the original power frequency interference and the hardware and software level denoising are removed from the surface electromyography signals sEMG are decomposed. For all the N channels, singular value decomposition is used to perform signal decomposition.
7. The muscle fatigue state detection method based on surface electromyography and pulse wave according to claim 1, characterized in that: In S3, for all N channels of surface electromyography (sEMG) signals, each channel selects to retain the first R components; a space-time diagram is set, the width of the space-time diagram is the time variable, the height is the channel variable of the surface electromyography (sEMG), and the color channel is R SSA components, so as to obtain a space-time diagram of size L*N*R.
8. The muscle fatigue state detection method based on surface electromyography and pulse wave according to claim 1, characterized in that: In S5, the user's data is input into the model trained in S4, the user's muscle fatigue state is fed back through the user interface and the muscle fatigue state classification is output.
9. A muscle fatigue state detection system based on surface electromyography and pulse wave, characterized in that: The performance testing platform of the muscle fatigue state detection system includes a processor and a memory, the processor is communicatively connected to the memory, the memory stores a computer program, and the processor executes the muscle fatigue state detection method based on surface electromyography and pulse wave as described in any one of claims 1-8 according to the computer program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program data, and the program data is used to execute the muscle fatigue state detection method based on surface electromyography and pulse wave as described in any one of claims 1-8.