A gesture recognition method based on electromyographic signal based on dual-branch multi-stream network

Abstract

A method for gesture recognition based on a dual-branch multi-stream network uses the electromyographic signal data obtained by each acquisition electrode as an independent input of the network flow after data preprocessing, and constructs a suitable model architecture to match this input method. Compared with the traditional method of merging data first and then segmenting, this method is simpler in data preprocessing, and can also reduce the time overhead caused by the segmentation operation and improve the operating efficiency of the model; the constructed network model analyzes the intrinsic correlation between the electromyographic signal and the corresponding gesture from the overall and local perspectives, and uses a voting mechanism to combine the outputs of the two branch networks to determine the final recognition result, so that the two branch networks form complementary benefits, make up for their respective inherent defects, and thus improve the accuracy of gesture recognition.

Description

Technical Field

[0001] The present invention belongs to the field of physiological signal recognition, and in particular relates to an electromyographic signal gesture recognition method based on a dual-branch multi-stream network. Background Art

[0002] Surface electromyography (sEMG) is an electrical signal generated when muscles contract. It is an important way to detect muscle activity non-invasively on the body surface. Surface electromyography is closely related to human body movements. By analyzing surface electromyography, researchers can analyze and identify human body movements and behavioral intentions. In recent years, gesture recognition based on surface electromyography has gradually become a cutting-edge research direction in the field of human-computer interaction and has been widely used in rehabilitation medicine, mechanical control, virtual reality and other fields.

[0003] Traditional electromyographic gesture recognition methods generally include four steps: signal acquisition, data preprocessing, feature extraction, and gesture classification. Signal acquisition is to place electrodes on the surface of human skin and collect and record potential change data through the acquisition device connected to the electrodes. In the data preprocessing stage, the original signal is processed by bandpass filtering to eliminate noise and abnormal data. For the processed data, the sliding window technology is generally used to divide the data into equal-length data segments, and each segment of data is sample data for model learning. In the feature extraction stage, a set of feature data is usually obtained by using multiple extraction algorithms on the collected data. This set of feature data is called a feature set. Finally, the feature set can be used to train a classifier after dimensionality reduction. The classifier maps the feature data to one of the given gesture categories, thereby realizing gesture classification. Linear discriminant analysis, support vector machine, and random forest are all commonly used classifiers in electromyographic gesture classification. In recent years, deep learning, as a branch method of machine learning, has shown good performance in the field of electromyographic signal gesture recognition. The algorithm using deep learning not only achieves a lower recognition error rate than traditional methods, but also greatly simplifies the recognition process. It can automatically learn features from training data without manually designing feature set extraction algorithms. At present, mainstream deep learning methods include Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), and the fusion of the two. Through convolution operations, CNN can extract high-dimensional features from input data and classify data based on the features. RNN takes sequence data as input and performs recursive analysis in the direction of sequence evolution. Both types of networks can well fit the sample characteristics of surface electromyography signals, so they are more suitable for processing electromyography gesture recognition problems.

[0004] At present, the processing methods of input samples for myoelectric gesture recognition methods based on deep learning can generally be divided into two categories. One is the overall analysis method, which combines the myoelectric data collected by each electrode into a whole, conducts overall feature learning and analysis of the data through a deep learning network, and then outputs the recognition result. The other is the divide-and-conquer analysis method, which first merges the input data, then divides the merged overall data into multiple equal-sized fragments, and then uses a multi-channel parallel architecture to extract features from each data fragment, and finally uses feature fusion to classify samples. Both methods have certain defects. The overall analysis method ignores the strong correlation between gestures and certain muscle tissues, and the divide-and-conquer analysis method will lose the detailed feature information extracted from the shallow layer as the number of convolution layers increases. These defects restrict the further improvement of recognition accuracy. Summary of the invention

[0005] The present invention proposes a method for electromyographic signal gesture recognition based on a dual-branch multi-stream network. First, the present invention directly corresponds the electromyographic signal data collected by each electrode to an independent input of a network stream for feature extraction and analysis. Secondly, the present invention designs a dual-branch architecture to analyze the correlation between the electromyographic signal and the corresponding gesture category from the overall and local perspectives. On this basis, a voting mechanism is used to combine the outputs of the two branch networks to determine the final gesture recognition result. The present invention specifically includes the following steps.

[0006] A method for myoelectric signal gesture recognition based on a dual-branch multi-stream network, characterized in that the method comprises the following steps:

[0007] Step 1: Obtain electromyographic signal data; record the electromyographic signal data corresponding to the human body completing the gesture action by collecting electrode equipment;

[0008] Step 2: Data preprocessing;

[0009] Step 3: Build a network model for feature extraction and gesture classification; the network model mainly includes 2 branch network modules and 1 voting mechanism module;

[0010] Step 4: Use the collected data to train the constructed network model to obtain the optimal model parameters;

[0011] Step 5: Apply the model; input the electromyographic signal to be recognized into the network model, and the model outputs the gesture recognition result.

[0012] Furthermore, in step 1, data is collected from the individual under test using an electromyographic signal acquisition device at a set sampling rate through an acquisition electrode device, and patch electrodes of the acquisition electrode device are dispersedly placed on the back surface of the hand of the individual under test.

[0013] Furthermore, the preprocessing of step 2 includes the following sub-steps:

[0014] Step 2.1, select steady-state data of a certain time length from the acquired raw electromyographic data as valid data, and perform filtering and denoising processing;

[0015] Step 2.2, use the sliding window method to segment the corresponding channel data collected by each patch electrode, put the channel data in the same time segment into the same set, and attach the corresponding gesture action label, and treat each set-label object as a separate data sample;

[0016] Step 2.3, divide all data samples processed in step 2.2 into training set and test set.

[0017] Further, step 3 includes the following sub-steps:

[0018] Step 3.1, construct a branch network module, including branch network 1 and branch network 2; branch network 1 analyzes the correlation between each muscle and gesture from a local perspective, including multiple parallel network flows, feature fusion layers, multiple fully connected layers and classifiers; each network flow consists of multiple convolutional layers and one long short-term memory (LSTM) layer;

[0019] Branch network 2 analyzes the correlation between EMG data and gestures from a holistic perspective, and includes multiple convolutional layers, multiple fully connected layers, and classifiers.

[0020] Step 3.2, construct a voting mechanism module; this module receives the output of the two branch network modules, and then uses the voting mechanism to calculate and output the final gesture recognition result.

[0021] Furthermore, the gesture recognition process of the branch network 1 is as follows: first, the data sample obtained in step 2.2 is used as the standard input of the network, and the channel data contained therein is input into each network stream respectively; then, the features output by each network stream are merged using the feature fusion layer, and then input into the fully connected layer for dimensionality transformation; finally, the features output by the fully connected layer are received by the classifier, and the gestures are classified according to the features, and the recognition result is output.

[0022] Furthermore, the gesture recognition process of the branch network 2 is as follows: first, the data of each channel in the data sample obtained in step 2.2 is merged into a whole through the feature fusion layer as the standard input of the network; then, the original input is successively subjected to feature extraction of the convolutional layer and dimensional transformation of the fully connected layer, and then input into the classifier; finally, the classifier outputs the gesture recognition result according to the feature information.

[0023] Furthermore, the voting mechanism module receives the classifier outputs of the two branch network modules, and its output format is a vector containing K real numbers, where the K real numbers correspond to the probability of K gestures during collection, and the values ​​are between 0 and 1, and the sum is 1, which is marked as S(i), indicating the probability of the i-th gesture; the voting mechanism is used to combine the outputs of the two branch networks to determine the final result, and the calculation formula is:

[0024]

[0025] Where j represents the number of the branch network, S j (i) represents the probability of the i-th gesture output by the j-th branch network, a j represents the weight coefficient of the j-th branch network, probability(i) is the final probability of the i-th gesture after calculation; the gesture corresponding to the largest one among the K probabilities is taken as the final output result, denoted as g final ,but

[0026] probability(g final )=max(probability(i),i=1...K).

[0027] Furthermore, the step 4 includes the following sub-steps:

[0028] Step 4.1: Use the training set obtained in step 2.3 to train the two branch networks separately, and use the test set to judge the recognition performance of the two branch networks, determine the optimal parameters layer by layer, obtain the network model with the best recognition performance, and save its corresponding parameters;

[0029] Step 4.2: Train the voting mechanism module to determine the optimal weight coefficient.

[0030] Beneficial effects of the present invention:

[0031] (1) This method uses the electromyographic signal data obtained by each acquisition electrode as an independent input to the network flow after data preprocessing, and constructs a suitable model architecture to match this input method. Compared with the traditional "merge data first and then split" method, this method is simpler in data preprocessing, and can also reduce the time overhead caused by the split operation and improve the operation efficiency of the model.

[0032] (2) The network model constructed by this method analyzes the intrinsic correlation between the electromyographic signal and the corresponding gesture from both the overall and local perspectives. At the same time, it uses a voting mechanism to combine the outputs of the two branch networks to determine the final recognition result, so that the two branch networks can form complementary benefits and make up for their respective inherent defects, thereby improving the accuracy of gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is an implementation flow chart of the method for myoelectric signal gesture recognition based on a dual-branch multi-stream network in an embodiment of the present invention.

[0034] Figure 2 Schematic diagram of the model architecture of a dual-branch multi-stream network in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0036] In view of the defects in the existing technology, the present invention analyzes the intrinsic correlation between electromyographic signals and corresponding gestures from the overall and local perspectives through a deep learning network. On this basis, a voting mechanism is introduced to integrate the recognition results of the above two analysis methods, which is specifically achieved through the following steps.

[0037] Step 1: Obtain electromyographic signal data. The electromyographic signal acquisition device is used to collect data from the individual under test at a sampling rate of 100 Hz. The device has 10 patch electrodes, which are scattered on the back of the hand. The individual under test repeats each action 6 times, each time for 12 seconds.

[0038] Step 2 performs preprocessing on the acquired raw EMG data. First, 8 seconds of steady-state data are selected from the 12-second execution time of each gesture as valid data, and the abnormal points in the EMG data are removed by bandpass filtering and denoising. Then, the data collected by the 10 electrodes are segmented using the sliding window method, with a sliding window length of 150 milliseconds and a sliding step length of 40 milliseconds. The data is sorted by electrode number, and the data of each channel in the same time segment are marked as x1, x2, ..., x 10 , and then put them into the same set X, that is, X={x n}, n = (1,2,…,10), number the sets in the order of the time segments, and label each set with a corresponding gesture action label, and use each "set-label" object as a separate data sample. After obtaining all the data samples, the 6 repetitions of the same action are marked as 1 to 6 according to the order of repeated execution of the gesture action, and the repetitions 1 to 4 are divided into the training set, and the repetitions 5 and 6 are divided into the test set.

[0039] Step 3: Build Figure 2 The dual-branch multi-stream network model shown in Figure 1 is implemented based on convolutional neural networks and recurrent neural networks, and mainly consists of three parts: 2 branch network modules and 1 voting mechanism module.

[0040] Branch network 1 analyzes the correlation between each muscle and gesture from a local perspective. In this embodiment, it includes 10 parallel network flows, a feature fusion layer, 2 fully connected layers, and a Softmax classifier. In this embodiment, each network flow is composed of 5 one-dimensional convolutional layers and 1 long short-term memory layer. The channel data in the data sample obtained in step 2 is used as the input of each network flow. The feature fusion layer merges the features output by each network flow and then inputs them into the fully connected layer for dimensional transformation. After receiving the features output by the fully connected layer, the classifier classifies the gestures according to the features and outputs the recognition results.

[0041] Branch network 2 analyzes the correlation between electromyographic data and gestures from a holistic perspective. In this embodiment, it includes 3 two-dimensional convolutional layers, 2 fully connected layers and a Softmax classifier. The data of each channel in the data sample obtained in step 2 is merged into a whole through the feature fusion layer as the input of the network. The original input is extracted by the convolution layer and the dimension is changed by the fully connected layer, and then input into the classifier for gesture classification. Finally, the classifier outputs the recognition result.

[0042] The voting mechanism module receives the Softmax classifier outputs of the two branch network modules. Its output format is a vector containing K real numbers. The K real numbers correspond to the probability of K gestures during collection. The values ​​are between 0 and 1, and the sum is 1. It is marked as S(i), which indicates the probability of the i-th gesture. The voting mechanism is used to combine the outputs of the two branch networks to determine the final result. The calculation formula is:

[0043]

[0044] Where j represents the number of the branch network, S j (i) represents the probability of the i-th gesture output by the j-th branch network, a j represents the weight coefficient of the j-th branch network, and probability(i) is the final probability of the i-th gesture after calculation. The gesture corresponding to the largest one of the K probabilities is taken as the final output result, denoted as g final ,but

[0045] probability(g final )=max(probability(i),i=1...K).

[0046] Step 4 uses the collected data to train the model, obtains the network model with the best recognition performance, and saves its corresponding parameters. First, use the training set to train the two branch networks separately, use the Adam optimizer to calculate the adaptive learning rate of each parameter, and then use the test set to determine the recognition performance of the network model, determine the optimal parameters layer by layer, and save the optimal branch network model obtained by the final training. On this basis, use the training set as a sample, and use the dichotomy method to obtain the optimal values ​​of the weight coefficients a1 and a2 of the two branch networks in the voting mechanism module and save them.

[0047] Step 5: Apply the model. Input the electromyographic signal to be recognized into the network model, filter and denoise the signal data, and then input the data into the two branch networks to obtain the Softmax classifier output. Then use the voting mechanism module to calculate the final recognition result and output it.

[0048] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. A method for myoelectric signal gesture recognition based on a dual-branch multi-stream network, characterized in that: The method comprises the following steps: Step 1: Obtain electromyographic signal data; record the electromyographic signal data corresponding to the human body completing the gesture action by collecting electrode equipment; Step 2: Data preprocessing; Step 3: Build a network model for feature extraction and gesture classification; the network model mainly includes 2 branch network modules and 1 voting mechanism module; Step 3 includes the following sub-steps: Step 3.1, construct a branch network module, including branch network 1 and branch network 2; branch network 1 analyzes the correlation between each muscle and gesture from a local perspective, including multiple parallel network flows, feature fusion layers, multiple fully connected layers and classifiers; each network flow consists of multiple convolutional layers and one long short-term memory (LSTM) layer; Branch network 2 analyzes the correlation between EMG data and gestures from a holistic perspective, and includes multiple convolutional layers, multiple fully connected layers, and classifiers. Step 3.2, construct a voting mechanism module; this module receives the output of the two branch network modules, and then uses the voting mechanism to calculate the final gesture recognition result and output it; Step 4: Use the collected data to train the constructed network model to obtain the optimal model parameters; Step 5: Apply the model; input the electromyographic signal to be recognized into the network model, and the model outputs the gesture recognition result.

2. The method for myoelectric signal gesture recognition based on a dual-branch multi-stream network according to claim 1, characterized in that: In step 1, data is collected from the individual under test using an electromyographic signal acquisition device at a set sampling rate through an acquisition electrode device, and the patch electrodes of the acquisition electrode device are dispersedly placed on the back surface of the hand of the individual under test.

3. The method for myoelectric signal gesture recognition based on a dual-branch multi-stream network according to claim 1, characterized in that: The preprocessing of step 2 includes the following sub-steps: Step 2.1, select steady-state data of a certain time length from the acquired raw electromyographic data as valid data, and perform filtering and denoising processing; Step 2.2, use the sliding window method to segment the corresponding channel data collected by each patch electrode, put the channel data in the same time segment into the same set, and attach the corresponding gesture action label, and treat each set-label object as a separate data sample; Step 2.3, divide all data samples processed in step 2.2 into training set and test set.

4. The method for myoelectric signal gesture recognition based on a dual-branch multi-stream network according to claim 3 is characterized in that: The gesture recognition process of the branch network 1 is as follows: first, the data sample obtained in step 2.2 is used as the standard input of the network, and the channel data contained therein is input into each network stream respectively; then, the features output by each network stream are merged using the feature fusion layer, and then input into the fully connected layer for dimensionality transformation; finally, the features output by the fully connected layer are received by the classifier, and the gestures are classified according to the features, and then the recognition results are output.

5. The method for myoelectric signal gesture recognition based on a dual-branch multi-stream network according to claim 3 is characterized in that: The gesture recognition process of the branch network 2 is as follows: first, the data of each channel in the data sample obtained in step 2.2 is merged into a whole through the feature fusion layer as the standard input of the network; then, the original input is successively subjected to feature extraction by the convolution layer and dimensional transformation by the fully connected layer, and then input into the classifier; finally, the classifier outputs the gesture recognition result according to the feature information.

6. The method for myoelectric signal gesture recognition based on a dual-branch multi-stream network according to claim 1, characterized in that: The voting mechanism module receives the classifier outputs of the two branch network modules, and its output format is a vector containing K real numbers, where the K real numbers correspond to the probability of K gestures during collection, and the values ​​are between 0 and 1, and the sum is 1, which is marked as S(i), indicating the probability of the i-th gesture. The voting mechanism is used to combine the outputs of the two branch networks to determine the final result, and the calculation formula is: Where j represents the number of the branch network, S j (i) represents the probability of the i-th gesture output by the j-th branch network, a j represents the weight coefficient of the j-th branch network, probability(i) is the final probability of the i-th gesture after calculation; the gesture corresponding to the largest one among the K probabilities is taken as the final output result, denoted as g final ,but probability(g final )=max(probability(i),i=1…K)。 7. The method for myoelectric signal gesture recognition based on a dual-branch multi-stream network according to claim 3 is characterized in that: The step 4 includes the following sub-steps: Step 4.1: Use the training set obtained in step 2.3 to train the two branch networks separately, and use the test set to judge the recognition performance of the two branch networks, determine the optimal parameters layer by layer, obtain the network model with the best recognition performance, and save its corresponding parameters; Step 4.2: Train the voting mechanism module to determine the optimal weight coefficient.

Images