A multi-channel gesture recognition method and device

Abstract

This invention discloses a multi-channel gesture recognition method and device, belonging to the field of rehabilitation. It collects N-channel electromyographic (EMG) signals during hand movements and real-time pressure signals between the fingertips and the grasping object. The N-channel EMG signals are preprocessed based on the real-time pressure signals. The preprocessed N-channel EMG signals are then integrated into a signal matrix. Features in the time domain, frequency domain, time-frequency domain, and spatial correlation are extracted sequentially from the signal matrix. All features from each time frame are concatenated in chronological order to form a high-dimensional feature matrix. This high-dimensional feature matrix is ​​input into a dynamic spatiotemporal graph convolutional network model. The model performs gesture and force recognition, generating a gesture classification probability distribution and a force grading probability distribution, and outputs the results. Through these steps, complex gestures and gestures of varying forces can be accurately classified and recognized.

Description

Technical Field

[0001] This invention relates to the field of rehabilitation, and in particular to a multi-channel gesture recognition method and apparatus. Background Technology

[0002] Stroke is a neuromuscular pathway disorder caused by obstructed blood supply to the brain, which can lead to functional impairments such as motor, sensory, and language impairments. Hand function rehabilitation training is an important part of the patient's rehabilitation process.

[0003] Existing hand recognition methods are mainly divided into vision-based gesture recognition and gesture recognition based on human electromyography (EMG) signals. Vision-based gesture recognition uses a camera to collect visual information of gestures, eliminating the need for users to wear hardware devices. This non-contact information collection greatly improves the comfort of interaction. However, vision-based gesture recognition is easily affected by lighting conditions. The nature and intensity of the light source, as well as the position of the light source and camera, directly affect the imaging effect. Furthermore, the diversity of gestures is significant; human gestures are three-dimensional and multi-degree-of-freedom, making it difficult for a two-dimensional projection plane to accurately judge complex and varied gestures.

[0004] Existing gesture recognition based on human electromyography (EMG) signals focuses on decoding EMG signals and recognizing movements, but it is difficult to accurately classify complex gestures or gestures of different strengths, especially in the absence of strength recognition functions. Summary of the Invention

[0005] In order to overcome the shortcomings of the prior art, one of the objectives of this invention is to provide a multi-channel gesture recognition method based on human electromyography signals that can accurately classify complex gestures and gestures of different strengths.

[0006] To overcome the shortcomings of the prior art, the second objective of this invention is to provide a multi-channel gesture recognition device based on human electromyography signals that can accurately classify complex gestures and gestures of different strengths.

[0007] One of the objectives of this invention is achieved through the following technical solution:

[0008] A multi-channel gesture recognition method includes the following steps:

[0009] Signal acquisition: Electrodes are placed on the forearm to acquire N-channel electromyographic signals during hand movements, where N is an integer greater than 1; pressure sensors are placed on the fingertips to acquire real-time pressure signals between the fingertips and the grasped object.

[0010] Signal preprocessing: The dynamic cutoff frequency range of the filter is calculated using the real-time pressure signal. Adaptive bandpass filtering is then performed on the electromyography (EMG) signal of each channel based on this frequency range. Wavelet threshold denoising is then applied to the filtered EMG signal of each channel. Multimodal normalization based on pressure value parameter compensation is then performed on the denoised EMG signal of each channel. Finally, the normalized N-channel EMG signals are integrated into a signal matrix.

[0011] Feature extraction: Extract time-domain, frequency-domain, time-frequency domain, and spatially correlated features from the signal matrix in sequence, and concatenate all features of each time frame in chronological order to form a high-dimensional feature matrix;

[0012] Model Construction: A bidirectional LSTM network is used to learn the temporal evolution of hand gestures, and an anatomically constrained muscle connectivity graph is constructed. Multi-channel spatial correlations are captured through graph convolution. Channel-temporal joint attention weights are designed to form a dynamic spatiotemporal graph convolutional network model.

[0013] Output results: The high-dimensional feature matrix is ​​input into the dynamic spatiotemporal graph convolutional network model. The dynamic spatiotemporal graph convolutional network model performs gesture and force recognition, forming a gesture classification probability distribution and a force classification probability distribution. When the probability is greater than or equal to the first threshold, the result is directly output; when the probability is greater than or equal to the second threshold and less than the first threshold, a sliding window weighted voting is triggered; when the probability is less than the second threshold, resampling is triggered.

[0014] Furthermore, in the signal preprocessing step, the lowest frequency in the dynamic cutoff frequency range... highest frequency In the formula, f0 is the reference center frequency, α is the dynamic adjustment coefficient, P(t) is the current pressure value, and P max That is the maximum pressure value.

[0015] Furthermore, in the signal preprocessing step, the wavelet thresholding noise reduction specifically involves: performing multi-level decomposition based on wavelet basis functions, and calculating the threshold for each level of wavelet decomposition. tanh(0.5·SNR j ), where σ j Let N be the standard deviation of the noise at layer j. j SNR is the coefficient length. j As the signal-to-noise ratio dynamic weighting factor, after wavelet decomposition, the high-frequency components of each layer are thresholded.

[0016] Furthermore, the high-frequency components of each layer are thresholded specifically as follows: the absolute value of the high-frequency component is less than λ. j The value of the high-frequency component is set to zero, and the absolute value of the high-frequency component is greater than or equal to λ. j The signal is contracted according to a threshold, preserving the valid signal.

[0017] Furthermore, in the signal preprocessing step, the multimodal normalization based on pressure value parameter compensation specifically refers to: the standardized signal μ EMG σ is the mean of the electromyographic signal. EMG α is the standard deviation of the electromyographic signal, and β is the pressure compensation factor.

[0018] Furthermore, in the signal preprocessing step, the signal matrix is... T is the length of the time series, N is the number of electromyographic signal channels, each row of the signal matrix contains the preprocessed signals of N channels at each time point, and each column of the signal matrix is ​​the complete time series of one channel.

[0019] Furthermore, in the feature extraction step, the extraction of time-domain features specifically involves calculating the mean, variance, and zero-crossing rate of the signal matrix using a sliding window; the extraction of frequency-domain features specifically involves extracting the power spectral density and dividing it into multiple sub-band energy integrals; the extraction of time-frequency-domain features specifically involves adding an electromyography signal spectral tilt compensation factor to improve MFCC coefficient extraction; and the extraction of spatial correlation features specifically involves calculating the inter-channel mutual information entropy. In the formula, I(X) i ,X j p(x) represents the mutual information entropy between the electromyographic signals of channel i and channel j. i ,x j ) represents the joint probability distribution of the electromyographic signals of the i-th channel and the j-th channel.

[0020] Furthermore, in the model construction step, the channel-temporal joint attention weights In the formula h t For the hidden state of LSTM, x c Let W be the feature vector of channel c. h W x v is a trainable parameter, and c′ and t′ represent combinations of other channels and time steps, respectively.

[0021] Furthermore, the multi-channel gesture recognition method also includes an identity recognition step, which specifically involves: identifying the user's identity through user identification or biometrics; when the user is a new user, freezing the underlying network and fine-tuning the top-level classifier to achieve personalized adaptation.

[0022] The second objective of this invention is achieved by the following technical solution:

[0023] A multi-channel gesture recognition device is provided for implementing any of the above-described multi-channel gesture recognition methods. The multi-channel gesture recognition device includes...

[0024] An electrode comprising a release film, hydrogel, medical tape, insulating oil layer, carbon paste circuit, silver chloride circuit, and PET layer, wherein the silver chloride circuit is disposed on the PET layer, the carbon paste circuit, the insulating oil layer, and the hydrogel are disposed on the silver chloride circuit, the insulating oil layer protects the carbon paste circuit and the silver chloride circuit, the hydrogel contacts the skin surface to provide conductivity, the medical tape is located on the insulating oil layer to provide mechanical support, and the electrode is disposed on the forearm to collect electromyographic signals during hand movements;

[0025] A pressure sensor, which is a thin-film pressure sensor, is used to collect real-time pressure signals between the fingertip and the object being grasped.

[0026] The processor identifies gestures and the corresponding force of gestures based on the electromyographic signals and the pressure signals.

[0027] Compared to existing technologies, the multi-channel gesture recognition method of this invention collects N-channel electromyography (EMG) signals during hand movements and real-time pressure signals between the fingertips and the grasped object; calculates the dynamic cutoff frequency range of the filter using the real-time pressure signals, and performs adaptive bandpass filtering on the EMG signals of each channel according to the frequency range; performs wavelet threshold denoising on the filtered EMG signals of each channel, and performs multimodal normalization based on pressure value parameter compensation on the denoised EMG signals of each channel; integrates the normalized N-channel EMG signals into a signal matrix; extracts time-domain, frequency-domain, time-frequency-domain, and spatially correlated features from the signal matrix sequentially, and concatenates all features of each time frame in chronological order to form a high-dimensional feature matrix; and employs a bidirectional... LSTM networks learn the temporal evolution of hand gestures, construct anatomically constrained muscle connectivity graphs, and capture multi-channel spatial correlations through graph convolution. Channel-temporal joint attention weights are designed to form a dynamic spatiotemporal graph convolutional network model. A high-dimensional feature matrix is ​​input into this model, which then performs gesture and force recognition, generating probability distributions for gesture classification and force levels. When the probability is greater than or equal to a first threshold, the result is directly output. When the probability is greater than or equal to a second threshold but less than the first threshold, a sliding window weighted voting is triggered. When the probability is less than the second threshold, resampling is triggered. Through these steps, complex gestures and gestures with varying forces can be accurately classified and recognized. Attached Figure Description

[0028] Figure 1 This is a flowchart of the multi-channel gesture recognition method of the present invention;

[0029] Figure 2 This is a gesture classification diagram of the multi-channel gesture recognition method of the present invention;

[0030] Figure 3This is a classification diagram of different gesture strengths in the multi-channel gesture recognition method of the present invention;

[0031] Figure 4 This is an electromyography (EMG) signal diagram of the multi-channel gesture recognition method of the present invention;

[0032] Figure 5 This is an algorithm framework diagram of the multi-channel gesture recognition method of the present invention;

[0033] Figure 6 This is a model architecture diagram of the multi-channel gesture recognition method of the present invention;

[0034] Figure 7 This is a diagram of the electrode structure of the multi-channel gesture recognition device of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0037] Please see Figure 1 as well as Figure 5 The present invention provides a multi-channel gesture recognition method, comprising the following steps:

[0038] Signal acquisition: Electrodes are placed on the forearm to acquire N-channel electromyographic signals during hand movements, where N is an integer greater than 1; pressure sensors are placed on the fingertips to acquire real-time pressure signals between the fingertips and the grasped object.

[0039] Signal preprocessing: The dynamic cutoff frequency range of the filter is calculated using real-time pressure signals. Adaptive bandpass filtering is then applied to the electromyography (EMG) signals of each channel based on this frequency range. Wavelet threshold denoising is performed on the filtered EMG signals of each channel. Multimodal normalization based on pressure value parameter compensation is then applied to the denoised EMG signals of each channel. Finally, the normalized N-channel EMG signals are integrated into a signal matrix.

[0040] Feature extraction: Extract time-domain, frequency-domain, time-frequency domain, and spatially correlated features from the signal matrix in sequence, and concatenate all features of each time frame in chronological order to form a high-dimensional feature matrix;

[0041] Model Construction: A bidirectional LSTM network is used to learn the temporal evolution of hand gestures, and an anatomically constrained muscle connectivity graph is constructed. Multi-channel spatial correlations are captured through graph convolution. Channel-temporal joint attention weights are designed to form a dynamic spatiotemporal graph convolutional network model.

[0042] Output results: The high-dimensional feature matrix is ​​input into the dynamic spatiotemporal graph convolutional network model. The dynamic spatiotemporal graph convolutional network model performs gesture and force recognition, forming a gesture classification probability distribution and a force classification probability distribution. When the probability is greater than or equal to the first threshold, the result is directly output; when the probability is greater than or equal to the second threshold and less than the first threshold, a sliding window weighted voting is triggered; when the probability is less than the second threshold, resampling is triggered.

[0043] Specifically, such as Figure 2 as well as Figure 3 As shown, gestures are divided into simple gestures and complex gestures. There are 10 simple gestures in total, namely: Index Flex (IF): index finger bent, Index Ext (IE): index finger extended, Middle Flex (MF): middle finger bent, Middle Ext (ME): middle finger extended, Ring Flex (RF): ring finger bent, Ring Ext (RE): ring finger extended, Pinky Flex (PF): little finger bent, Pinky Ext (PE): little finger extended, Thumb Flex (TF): thumb bent, and Thumb Ext (TE): thumb extended. There are 10 complex hand gestures: One (O): gesture for the number 1 (index finger extended, other fingers bent), Two (T): gesture for the number 2 (index and middle fingers extended, other fingers bent), Three (TH): gesture for the number 3 (index, middle, and ring fingers extended, other fingers bent), Four (FO): gesture for the number 4 (all fingers except thumb extended), Five (FI): gesture for the number 5 (palm flat, all fingers extended), Fist (FT): fist, Flat (FL): palm flat, Thumb Up (TU): thumb pointing upwards, Tip pinch (TP): index finger and thumb pinched together, Tripod pinch (TRP): triangle pinch, index, middle, and thumb pinched together.

[0044] Among the complex hand gestures, the three gestures of pinching together the index finger and thumb, pinching together in a triangle with the middle finger, index finger, and thumb, and making a fist, each include three different levels of pressure. These are:

[0045] Pinch 1: Gently pinch the index finger and thumb together, normally pinch together, or pinch together with force (light, medium, or heavy force).

[0046] Pinch 2: Gently pinch, normally pinch, or pinch firmly (light, medium, or heavy pressure).

[0047] Clenching a fist: lightly clenched, normally clenched, or forcefully clenched (light, medium, or heavy force).

[0048] The user's recovery level can be determined by recognizing the type and force of the user's gestures.

[0049] The signal acquisition process is as follows: The electrodes are arranged in a horizontally distributed "T" shape, precisely covering the main muscle groups of the forearm to ensure accurate and stable signal acquisition. The electrodes consist of multiple electrode points covering the main muscle areas of the forearm, including key muscle groups for elbow flexion, pronation and supination, wrist flexion, and finger flexion. The overall dimensions of the electrode patch are 194mm x 126mm, with an electrode diameter of 9mm, ensuring effective adhesion to the forearm and preventing signal loss. In this embodiment, eight electrode points are used, thus generating eight channels of electromyographic signals, such as... Figure 4 As shown. The electrodes are connected to the data acquisition device, and the data is transmitted to the computer in real time at a sampling frequency of 500 Hz.

[0050] The pressure sensor is a thin-film type, positioned between the fingertip and the object being grasped. The specific model is RP-C18.3-LT from Ligan Technology. This thin-film pressure sensor collects pressure parameters between the fingertip and the object, reflecting the pressure magnitude through changes in resistance.

[0051] The specific signal preprocessing steps are as follows:

[0052] The filter's cutoff frequency range is dynamically adjusted based on real-time pressure values ​​to suppress motion artifacts (such as sweat or skin slippage interference) caused by changes in muscle contraction strength, thereby improving signal quality. The filter's dynamic cutoff frequency range is adjusted in real-time by the pressure value, with the lowest frequency... highest frequency In the formula, f0 is the reference center frequency, α is the dynamic adjustment coefficient, P(t) is the current pressure value, and P max This is the maximum pressure value. In this embodiment, f0 = 250Hz, α = 200Hz.

[0053] Wavelet threshold denoising specifically involves:

[0054] Multi-level decomposition is performed based on the Daubechies-4 wavelet basis function, and an improved SUREShrink thresholding algorithm is used for the detail coefficients, achieving a noise suppression ratio of 32.7dB. Specifically, the threshold for each wavelet decomposition level is calculated. In the formula σ j Let N be the standard deviation of the noise at layer j. j SNR is the coefficient length. j As the signal-to-noise ratio dynamic weighting factor, after wavelet decomposition, thresholding is applied to the high-frequency components of each layer. The absolute value of the high-frequency components (detail coefficients) is less than λ. j Setting it to zero, the absolute value of the high-frequency components (detail coefficients) is greater than or equal to λ. j The signal is contracted according to a threshold, preserving the valid signal.

[0055] In the multimodal normalization process, pressure value parameter compensation is introduced for Z-score standardization, specifically: the standardized signal μ EMG σ is the mean of the electromyographic signal. EMG β is the standard deviation of the electromyographic signal, and β is the pressure compensation factor, obtained through the real-time value of the pressure sensor. It is used to suppress signal amplitude fluctuations caused by pressure changes, ensure signal stability, and reduce motion interference. In this embodiment, β = 0.05.

[0056] The signal matrix is T represents the length of the time series, where T = total time × 500. N represents the number of electromyographic signal channels. Each row of the signal matrix contains the preprocessed signals of N channels at each time point, i.e., the preprocessed signals of 8 channels at that time. Each column of the signal matrix corresponds to the complete time series of one channel.

[0057] The feature extraction steps employ a multi-domain joint feature extraction strategy, as shown in Table 1, and specifically include:

[0058] Table 1

[0059]

[0060] Specifically: extracting time-domain features involves calculating the mean, variance, and zero-crossing rate of the signal matrix using a sliding window; extracting frequency-domain features involves extracting the power spectral density and dividing the signal into multiple sub-bands for energy integration; extracting time-frequency-domain features involves adding an electromyography signal spectral tilt compensation factor to improve MFCC coefficient extraction; and extracting spatial correlation features involves calculating the inter-channel mutual information entropy. In the formula, I(X) i ,X j p(x) represents the mutual information entropy between the electromyographic signals of channel i and channel j. i ,x j) represents the joint probability distribution of the electromyographic signals of channel i and channel j. The feature set has a total dimension of 316, which reduces redundant features by 47% compared with traditional methods. Muscle synergy is quantified by mutual information entropy, improving the separability of complex gestures.

[0061] All features of each time frame are concatenated in chronological order to form a high-dimensional feature matrix. Specifically, 316-dimensional features (24+160+104+28) are extracted for each time frame (e.g., a 200ms window) to form a T′×316 matrix.

[0062] Please continue reading. Figure 6 In the model construction step, a dynamic spatiotemporal graph convolutional network model (DST-GCN model) is constructed, employing a dual-branch spatiotemporal feature extraction approach. The dual branches are a temporal branch and a spatial branch. The temporal branch uses a bidirectional LSTM network (128 hidden nodes) to learn the temporal evolution of hand gestures. The spatial branch involves constructing an anatomically constrained muscle connectivity graph (adjacency matrix). d ij (For muscle spacing), multi-channel spatial correlation is captured through graph convolution. Channel-temporal joint attention weights are designed. In the formula h t For the hidden state of LSTM, x c Let W be the feature vector of channel c. h W x v represents a trainable parameter, and c′ and t′ represent combinations of other channels and time steps, respectively. The model building process requires collecting training data and training the model. Specifically, when collecting training data, subjects should be between 18 and 40 years old, with no history of muscle or nervous system diseases, no history of skin allergies, good skin condition suitable for electrode wearing, and basic hand function capable of performing gestures. Subjects with upper limb motor dysfunction, recent (within the past three months) upper limb injuries or surgeries, or allergic reactions to the electrode devices should be excluded. During training data collection, subjects should rest for 3 seconds, perform the action for 5 seconds, and rest for 5 seconds. The number of repetitions is 10 times for each gesture, and 3 sets of data are collected for each action group. Total time = (5 seconds of action + 5 seconds of rest) x 10 repetitions x 3 sets. The experimental environment is a quiet, well-lit laboratory. Subjects should sit in a comfortable chair with their arms resting on a table for stability.

[0063] The multi-channel gesture recognition method also includes an identity recognition step, which is implemented using an incremental learning mechanism. The incremental learning mechanism identifies new users by user identity identifiers (such as login IDs) or biometric features (such as baseline patterns of electromyographic signals). It also uses the method of freezing the weights of the bottom layer network (DST-GCN) and only fine-tuning the top layer classifier to achieve personalized adaptation. The classifier parameters are updated through backpropagation using 50 sets of samples from new users to achieve personalized adaptation. The whole process takes only 3 minutes, which is efficient and accurate.

[0064] The specific steps for outputting the results are as follows: The high-dimensional feature matrix is ​​input into a dynamic spatiotemporal graph convolutional network model. The model performs potential and force recognition and outputs a simple gesture classification probability distribution. (10 types of gestures); probability distribution of complex gesture classification (10 types of gestures); Probability distribution of force levels (Light / Medium / Heavy Level 3). When the main classifier outputs a probability greater than or equal to the first threshold, the result is output directly; when the probability is greater than or equal to the second threshold but less than the first threshold, a sliding window weighted voting is triggered (window length 500ms); when the probability is less than the second threshold, resampling is triggered (new data is collected within 200ms). In this embodiment, the first threshold is 0.85 and the second threshold is 0.6. A real-time visualization interface is used to output the results, specifically displaying the electromyographic signal waveform, feature heatmap, and classification confidence curve.

[0065] This application also discloses a multi-channel gesture recognition device for implementing the multi-channel gesture recognition method. The multi-channel gesture recognition device includes electrodes, a pressure sensor, and a processor. The electrodes collect electromyographic signals during hand movements, and the pressure sensor collects real-time pressure signals between the fingertips and the grasping object. The processor recognizes the gesture and the corresponding force based on the electromyographic signals and the pressure signals.

[0066] Please continue reading. Figure 7 The electrodes include a release film, hydrogel, medical tape, insulating oil layer, carbon paste circuit, silver chloride circuit, and PET layer.

[0067] The release film is the bottom layer, used to protect the adhesive parts of the electrodes from contamination or adhesion before use. The release film is removed during use to expose the hydrogel or adhesive underneath.

[0068] Hydrogels are used to ensure good contact between the electrodes and the skin surface and to conduct electricity, transmitting electromyographic signals to the sensor. The hydrogel's adhesive properties allow the electrodes to adhere to the skin and maintain stable electrode-skin interface conductivity.

[0069] Medical tape provides mechanical support, ensuring the overall structure of the electrodes adheres firmly to the skin. Medical tape is typically waterproof and breathable, ensuring comfort during extended wear and preventing sweat from affecting electrode performance.

[0070] The insulating oil layer is used to protect the conductive materials inside the electrode, prevent external environmental interference with electrode performance, and ensure that the working area of ​​the electrode remains dry and stable.

[0071] Carbon paste circuitry is one of the conductive layers in an electrode, used for signal conduction. Carbon paste possesses excellent conductivity, corrosion resistance, and flexibility, making it a common conductive material in electrodes.

[0072] Silver chloride circuitry, as one of the main conductive materials, can efficiently transmit electromyographic signals and has good signal stability and low noise characteristics. Silver chloride is a commonly used material in electromyographic electrodes and has good biocompatibility.

[0073] The topmost PET layer is a transparent layer that provides physical support for the electrode while maintaining its flexibility. The transparency, flexibility, and durability of PET material ensure that the electrode maintains stable performance in various application scenarios.

[0074] The motor manufacturing process begins with annealing the PET substrate to eliminate internal stress and provide a stable foundation for the conductive circuitry. Specifically, the PET substrate thickness is 35-40 μm, preferably 38 μm. Next, a 6-12 μm thick silver chloride layer is printed onto the PET substrate as the main conductive material, followed by the printing of a conductive carbon paste layer of the same thickness to enhance conductivity and protect the silver paste circuitry. Then, a 10-20 μm thick transparent insulating oil layer is applied to protect the conductive parts from external environmental influences. Afterward, double-sided adhesive and a 120-130 μm thick blue reinforcing sheet are used to enhance the structural strength of the electrode, ensuring its durability and stability during use. A thickness of 125 μm is preferred. A 7-12 mm diameter hydrogel layer is coated onto the conductive area of ​​the electrode to ensure good contact with the skin and signal conduction. A thickness of 9 mm is preferred. Finally, the electrode is precisely die-cut to the designed size (194 mm x 126 mm) to ensure a precise fit to the user's forearm muscle area. The release membrane, as the bottom layer, protects the hydrogel and medical tape components, preventing adhesion or contamination when the electrodes are not in use. The entire manufacturing process emphasizes precise material processing and assembly, ensuring the electrodes' high efficiency and reliability in electromyography (EMG) signal acquisition.

[0075] Compared to existing technologies, the multi-channel gesture recognition method of this invention collects N-channel electromyography (EMG) signals during hand movements and real-time pressure signals between the fingertips and the grasped object; calculates the dynamic cutoff frequency range of the filter using the real-time pressure signals, and performs adaptive bandpass filtering on the EMG signals of each channel according to the frequency range; performs wavelet threshold denoising on the filtered EMG signals of each channel, and performs multimodal normalization on the denoised EMG signals of each channel based on pressure value parameter compensation; integrates the normalized N-channel EMG signals into a signal matrix; extracts time-domain, frequency-domain, time-frequency-domain, and spatially correlated features from the signal matrix sequentially, and concatenates all features of each time frame in chronological order to form a high-dimensional feature matrix; and employs a bidirectional... LSTM networks learn the temporal evolution of hand gestures, constructing anatomically constrained muscle connectivity graphs and capturing multi-channel spatial correlations through graph convolution. Channel-temporal joint attention weights are designed to form a dynamic spatiotemporal graph convolutional network model. A high-dimensional feature matrix is ​​input into this model, which performs potential and force recognition, generating a gesture classification probability distribution and a force level probability distribution. When the probability is greater than or equal to a first threshold, the result is directly output. When the probability is greater than or equal to a second threshold but less than the first threshold, a sliding window weighted voting is triggered. When the probability is less than the second threshold, resampling is triggered. Through these steps, complex gestures and gestures of varying forces can be accurately classified and recognized.

[0076] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These are all equivalent modifications and improvements made to the above embodiments based on the essential technology of the present invention, and all of these fall within the protection scope of the present invention.

Claims

1. A multi-channel gesture recognition method, characterized by, The method comprises the following steps: Signal acquisition: electrodes are arranged on the forearm to collect N-channel electromyography signals during hand movement, N being an integer greater than 1; a pressure sensor is arranged on the finger pulp to collect real-time pressure signals between the finger pulp and the grasped object; Signal preprocessing: the dynamic cutoff frequency range of the filter is calculated through the real-time pressure signal, and the electromyography signals of each channel are adaptively band-pass filtered according to the frequency range; Each channel of the filtered electromyography signals is subjected to wavelet threshold denoising, and each channel of the denoised electromyography signals is subjected to multi-modal normalization based on pressure value parameter compensation; the N-channel electromyography signals after normalization are integrated into a signal matrix; Feature extraction: the signal matrix is sequentially extracted in time domain, frequency domain, time-frequency domain and spatial correlation, and all features of each time frame are spliced in time sequence to form a high-dimensional feature matrix; Model construction: a bidirectional LSTM network is used to learn the time sequence evolution rule of gesture movement, an anatomically constrained muscle connection graph is constructed, and a multi-channel spatial correlation is captured through graph convolution; a channel-time sequence joint attention weight is designed to form a dynamic spatio-temporal graph convolution network model; Result output: the high-dimensional feature matrix is input into the dynamic spatio-temporal graph convolution network model, the dynamic spatio-temporal graph convolution network model performs gesture and force recognition, forms gesture classification probability distribution and force grading probability distribution, and when the probability is greater than or equal to a first threshold value, the result is directly output; when the probability is greater than or equal to a second threshold value and less than the first threshold value, a sliding window weighted voting is triggered; When the probability is less than the second threshold value, re-sampling is triggered.

2. The multi-channel gesture recognition method of claim 1, wherein: In the signal pre-processing step, the lowest frequency of the dynamic cut-off frequency range the highest frequency where f0 is the reference center frequency, a is a dynamic adjustment factor, P(t) is the current pressure value, P max is the maximum pressure value.

3. The method of claim 1, wherein: In the signal preprocessing step, the wavelet threshold denoising is specifically: multi-layer decomposition based on wavelet basis function, calculating the threshold of each layer wavelet decomposition In the formula, σ j is the standard deviation of the jth layer noise, N j is the coefficient length, SNR j is the signal-to-noise ratio dynamic weighting factor, and the high-frequency components of each layer after wavelet decomposition are subjected to threshold processing.

4. The method of claim 3, wherein: Thresholding is applied to the high-frequency components of each layer specifically as follows: the absolute value of the high-frequency component is less than λ. j The value of the high-frequency component is set to zero, and the absolute value of the high-frequency component is greater than or equal to λ. j The signal is contracted according to a threshold, preserving the valid signal.

5. The method of claim 1, wherein: In the signal preprocessing step, the multi-modal normalization based on pressure value parameter compensation is specifically: normalized signal μ EMG is the mean of the electromyographic signal, σ EMG is the standard deviation of the electromyographic signal, and β is a pressure compensation factor.

6. The method of claim 1, wherein: In the signal preprocessing step, the signal matrix is T is the time series length, N is the number of EMG channels, each row of the signal matrix is the preprocessed signal of N channels at each time point, and each column of the signal matrix is the complete time series of one channel.

7. The method of claim 1, wherein: In the feature extraction step, the extracted time domain features are specifically the mean, variance and zero-crossing rate of the signal matrix calculated by sliding window; the extracted frequency domain features are specifically the power spectral density and the integral of the energy of multiple subbands; the extracted time-frequency domain features are specifically the compensation factor of the spectral tilt of the myoelectric signal to improve the extraction of MFCC coefficients; and the extracted spatial correlation features are specifically the mutual information entropy between channels where I(X i ,X j ) represents the mutual information entropy of the myoelectric signals of the i-th channel and the j-th channel, and p(x i ,x j ) represents the joint probability distribution of the myoelectric signals of the i-th channel and the j-th channel.

8. The method of claim 1, wherein: In the constructing model step, the channel-temporal joint attention weight where h t is the LSTM hidden state, x c is the feature vector of channel c, W h ,W x and v are trainable parameters, and c' and t' represent the combination of other channels and time steps, respectively.

9. The method of claim 1, wherein: The multi-channel gesture recognition method further comprises an identity recognition step, which is specifically: the user identity is recognized through a user identity identifier or a biological feature, when the user is a new user, the bottom network is frozen and the top classifier is fine-tuned to realize individual adaptation.

10. A multi-channel gesture recognition apparatus for implementing the multi-channel gesture recognition method according to any one of claims 1-9, characterized by: The multi-channel gesture recognition device comprises An electrode, the electrode comprising a release film, a hydrogel, a medical adhesive tape, an insulating oil layer, a carbon slurry circuit, a silver chloride circuit and a PET layer, the silver chloride circuit being arranged on the PET layer, the carbon slurry circuit, the insulating oil layer and the hydrogel being arranged on the silver chloride circuit, the insulating oil layer protecting the carbon slurry circuit and the silver chloride circuit, the hydrogel being in contact with the skin surface to play a conductive role, the medical adhesive tape being located on the insulating oil layer to provide mechanical support, the electrode being arranged on the forearm to collect electromyography signals during hand movement; A pressure sensor, the pressure sensor being a thin film type pressure sensor, for collecting real-time pressure signals between the finger pulp and the grasped object; A processor, the processor recognizing gestures and the force corresponding to the gestures according to the electromyography signals and the pressure signals.

Images