A surface electromyography signal classification method based on dual-branch multi-scale multi-stream network

Abstract

The present invention relates to a surface electromyographic signal classification method based on a dual-branch multi-scale multi-stream network, comprising the following steps: obtaining an original electromyographic signal; extracting features from the original electromyographic signal, and inputting the extracted features into a first branch network to obtain a spatial feature map; normalizing the original electromyographic signal, and inputting the normalized information into a second branch network to obtain a temporal feature map; using a fully connected layer to fuse the spatial feature map and the temporal feature map, and classifying through a softmax layer. The beneficial effects of the present invention are as follows: the present invention solves the problem that a single convolution kernel needs to stack networks to improve performance, resulting in an exponential increase in network parameters and computing costs, making it complicated to deploy the model to an application device, and through multi-scale convolution and multi-stream feature fusion, the feature capture capability is improved within the range that network parameters and computing costs can bear, and the network capability is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of surface electromyography signal classification, and more specifically, to a surface electromyography signal classification method based on a double-branch multi-scale multi-stream network. Background Art

[0002] With the advancement of rehabilitation equipment and the continuous development of modern science and technology, the elderly and the disabled, especially those with hand disabilities, have a great desire for normal hand grasping. Bionic prostheses driven by artificial intelligence, machine perception, intelligent control and rehabilitation medicine technology have emerged. The control of the signal source is required for the disabled to control the prosthesis. The human body is full of various signals, which control and represent the various activities and intentions of the human body. The human body has a variety of bioelectric signals, which contain rich information reflecting human behavioral intentions. The bioelectric signals that are currently widely concerned include electromyographic signals (EMG), electrocardiographic signals (ECG), electroencephalographic signals (EEG) and electrooculographic signals (EOG).

[0003] The electromyographic signal represents the sum of subcutaneous motor action potentials generated by muscle contraction. Among many muscle fibers, it represents the superposition of motor unit action potentials (MUAP) in space and time. At present, the main control signal of bionic prostheses comes from surface electromyographic signals (sEMG). Surface electromyographic signals are the comprehensive effect of electrical activity on superficial muscles and neurons on the surface of human skin. They can represent the hand movements and muscle movements performed by the human body at the moment, and to a certain extent reflect the intention of human movements. Surface electromyographic signals contain rich biological information, and the signal collection is usually non-invasive and does not cause any adverse effects on the human body. They are often used in clinical medicine to diagnose neurological functional diseases. In the fields of rehabilitation medicine and clinical medicine, they are often used as control signal sources for bionic prostheses and rehabilitation equipment. Therefore, they have great medical application value.

[0004] However, electromyographic signals are weak bioelectric signals. Effectively extracting electromyographic signal features and realizing accurate classification and recognition of sEMG is a hot topic in the current electromyographic research field. Traditional feature extraction methods vary in form, but most of them use one or more statistical values ​​to represent the information implied by sEMG, which may often lose deeper feature information. Secondly, due to the influence of many environmental factors such as individual differences of subjects and electrode offset, the classification accuracy of traditional classification methods such as Bayesian classification, linear discriminant analysis, and random forests will also be greatly affected. In recent years, with the continuous development of artificial intelligence, research on electromyographic recognition based on deep learning has begun to rise. Some scholars have put electromyographic signals into deep learning models for training and classification, such as convolutional neural networks. However, the existing electromyographic signal recognition models based on deep learning still have the following problems: (1) The performance improvement of a single convolution kernel requires the stacking of networks, which leads to an exponential increase in network parameters and computing costs, making it complicated to deploy the model to application devices; (2) The highly redundant single-scale feature map generated by a single convolution is too limited in the multi-gesture classification task and may ignore the key information in the signal. (3) The model mainly focuses on a single time domain feature or frequency domain feature. There is no effective model that can combine the features of the time domain and frequency domain to achieve a more accurate recognition effect. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the prior art and to propose a surface electromyography signal classification method based on a double-branch multi-scale multi-stream network.

[0006] In a first aspect, a surface electromyography signal classification method based on a dual-branch multi-scale multi-stream network is provided, comprising:

[0007] Step 1: Obtain the original electromyographic signal;

[0008] Step 2: extract features from the original electromyographic signal, and input the extracted features into the first branch network to obtain a spatial feature map; the first branch network includes several branch streams, a multi-scale semantic injection module, a multi-frequency channel attention mechanism module and a fully connected layer;

[0009] Step 3, normalizing the original electromyographic signal, and inputting the normalized information into the second branch network to obtain a time feature map; the second branch network includes a GRU network and a CNN network;

[0010] Step 4: Use a fully connected layer to fuse the spatial feature map and the temporal feature map, and classify them through a softmax layer.

[0011] Preferably, step 2 comprises:

[0012] Step 2.1, extracting features from the original electromyographic signal, and preprocessing the extracted features to obtain an initial feature map;

[0013] Step 2.2, the initial feature map passes through several branch streams of different scales to obtain a deep semantic feature map corresponding to each branch stream; the branch stream includes several batch pooling convolution blocks;

[0014] Step 2.3: Through the multi-scale semantic injection module, the deep semantic feature maps of branch streams of different scales are interacted to obtain interactive feature information;

[0015] Step 2.4, the interactive feature information is passed through a multi-frequency channel attention mechanism module to obtain different frequency information; the different frequency information and the interactive feature information are interacted;

[0016] Step 2.5: Transform the interaction result of step 2.4 through a fully connected layer to obtain a spatial feature map.

[0017] Preferably, step 3 comprises:

[0018] Step 3.1, normalizing the original electromyographic signal;

[0019] Step 3.2: Input the normalized information into the GRU network for training to extract temporal feature information;

[0020] Step 3.3: compress the time feature information in time scale through the CNN network to obtain a time feature map.

[0021] Preferably, in step 2.1, the preprocessing includes enhancing two-dimensionalization.

[0022] Preferably, in step 2, the first branch network further includes a random dropout layer, and the input of the random dropout layer is connected to the output of the fully connected layer.

[0023] In a second aspect, a surface electromyography signal classification system based on a dual-branch multi-scale multi-stream network is provided, which is used to execute any method described in the first aspect, including:

[0024] The first acquisition module is used to acquire the original electromyographic signal;

[0025] A second acquisition module is used to extract features from the original electromyographic signal and input the extracted features into a first branch network to obtain a spatial feature map; the first branch network includes a plurality of branch streams, a multi-scale semantic injection module, a multi-frequency channel attention mechanism module and a fully connected layer;

[0026] A third acquisition module is used to normalize the original electromyographic signal and input the normalized information into a second branch network to obtain a time feature map; the second branch network includes a GRU network and a CNN network;

[0027] The classification module is used to fuse the spatial feature map and the temporal feature map using a fully connected layer and perform classification through a softmax layer.

[0028] According to a third aspect, a computer storage medium is provided, wherein a computer program is stored in the computer storage medium; when the computer program is executed on a computer, the computer executes any method described in the first aspect.

[0029] In a fourth aspect, an electronic device is provided, including:

[0030] Memory, used to store computer programs;

[0031] A processor is used to execute the computer program to implement any method as described in the first aspect.

[0032] The beneficial effects of the present invention are:

[0033] 1. The present invention solves the problem that the performance improvement of a single convolution kernel requires network stacking, which causes the network parameters and computing costs to increase exponentially, making it complicated to deploy the model to application devices. Through multi-scale convolution and multi-stream feature fusion, the feature capture capability is improved within the range of network parameters and computing costs, effectively improving the network's capabilities.

[0034] 2. The dual-branch network, batch pooling convolution module, multi-frequency channel attention mechanism, and multi-scale multi-stream feature fusion module proposed in the present invention pay more attention to extracting features of different scales and features of different domains of input data, and then fusing the outputs of the two through a fully connected layer. The network can learn spatiotemporal features more effectively, which is conducive to improving the recognition accuracy of gestures.

[0035] 3. The dual-branch multi-scale multi-stream network provided by the present invention takes the original signal and the feature map as input, retains the correlation and time dependence of the original time series, and enables the original model to highlight certain local information while mining the key detail information of different scales in the original signal ignored by the convolution operation. The richer detail features can effectively complete the recognition task and improve the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a surface electromyography signal classification method based on a dual-branch multi-scale multi-stream network;

[0037] Figure 2It is a schematic diagram of the structure of the multi-scale multi-stream feature fusion module;

[0038] Figure 3 This is the structure diagram of the multi-frequency channel attention mechanism. DETAILED DESCRIPTION

[0039] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0040] Embodiment 1:

[0041] Embodiment 1 of the present application provides a surface electromyography signal classification method based on a dual-branch multi-scale multi-stream network, such as Figure 1 As shown, including:

[0042] Step 1: Get the original electromyographic signal.

[0043] Specifically, the equipment used for collecting research data is the ELONXI electromyograph developed by the team of the University of Portsmouth in the UK where the inventor is located. The electrode set is equipped with 18 electrodes, 16 of which are used to collect surface electromyographic signals. This experiment uses 16 channels for sampling, the sampling frequency is 1k Hz, and the filtering operation is performed at 50 Hz. 8 healthy volunteers (6 males, 2 females, age: 25±5 years old, height: 175±10cm, weight: 65±10kg) participated in the 3-day data collection. Each person demonstrated 5 gestures in each time period, and the middle 10 seconds of each gesture were recorded. The experiment used the electromyographic data of 7 of them as the training set, and the signal collected by one person as the test set.

[0044] Step 2: extract features from the original electromyographic signal, and input the extracted features into the first branch network to obtain a spatial feature map; the first branch network includes several branch streams, a multi-scale semantic injection module, a multi-frequency channel attention mechanism module and a fully connected layer.

[0045] In addition, in step 2, the first branch network also includes a random dropout layer, and the input of the random dropout layer is connected to the output of the fully connected layer.

[0046] Step 2 includes:

[0047] Step 2.1: extract features from the original electromyographic signal, and preprocess the extracted features to obtain an initial feature map.

[0048] Specifically, in step 2.1, the preprocessing includes enhancing two-dimensionalization.

[0049] Step 2.2: The initial feature map passes through several branch streams of different scales to obtain a deep semantic feature map corresponding to each branch stream; the branch stream includes several batch pooling convolution blocks.

[0050] Specifically, the embodiment of the present application obtains a feature map x of size 15*15 / 16 by strengthening the two-dimensionalization method and inputs it into three branch streams. The input feature map x adopts a multi-stream parallel architecture and passes through Pconv modules (batch pooling convolution blocks) of different sizes of convolution to obtain n features under different receptive fields. ,in Represents the feature output of the i-th scale.

[0051] Table 1 Module parameter configuration table of the first branch network

[0052]

[0053] The branch stream is configured with the parameters shown in Table 1. In addition, in Table 1, Input represents the initial feature map extracted in step 2.1, Pconv Block1-6 represents the batch pooling convolution block in the branch stream. MSFF represents the multi-scale semantic injection module. MFCA represents the multi-frequency channel attention mechanism module. FC represents fully connected, and Dropout represents the random dropout layer.

[0054] Step 2.3: Through the multi-scale semantic injection module, the deep semantic feature maps of branch streams of different scales are interacted with each other.

[0055] Specifically, the feature information obtained from the three branch streams of different scales in S1 Exchange information

[0056] Step 2.4: Pass the interactive feature information through a multi-frequency channel attention mechanism module to obtain different frequency information; interact the different frequency information with the interactive feature information.

[0057] Specifically, for the n features obtained in step 2.3, the multi-frequency channel attention mechanism is used to assign different weight coefficients such as Figure 2 As shown in Figure 2, this process enables the features to be well combined with the frequency component information to obtain the importance of each feature channel, which is expressed as:

[0058]

[0059] in, Represents the relevant parameters of the multi-frequency channel attention of the i-th level feature, Represents the feature map after level i processing.

[0060] Then the multi-scale feature fusion process begins as follows Figure 2 As shown, For example, , , , C represents the i-th scale The height, width and number of channels of each j-th dimension branch flow adjusts the size of the feature map to match and The same size, the process can be expressed as follows:

[0061]

[0062] in , and Represent downsampling, unit mapping and upsampling respectively. Resize to , the range is Then perform normalization to get , sum the convolution output data of each scale. It will also be used in the fusion process of the next level decoder.

[0063] The multi-frequency channel attention mechanism module transforms the interactive feature information into different frequency information through discrete cosine transform (DCT). Specifically, discrete cosine transform (DCT) can convert input data from the spatial domain to the frequency domain. Figure 3 As shown, the two-dimensional DCT transform is shown as follows:

[0064]

[0065] First, the input X is divided into n parts along the channel dimension, resulting in:

[0066]

[0067] Then, different frequency components are used to compress each channel information of the original input image, and n frequency components are selected for frequency analysis in the channel dimension. The result can be expressed as:

[0068]

[0069] in is the vector compressed by the two-dimensional discrete cosine transformer, and [p,q] is the corresponding two-dimensional DCT frequency component index. Then, n pre-processed vectors can be obtained, and the n vectors containing frequency information are vertically spliced ​​to obtain a C×n matrix.

[0070]

[0071] in, Indicates taking the maximum value of each column. Each column is a vector obtained by compressing different frequency components. Taking the maximum value of each column can get the most significant eigenvalue of each channel. Finally, the compressed vector is merged with the original feature map X, and each element of the compressed vector is multiplied with the channel corresponding to the original feature map at the channel level. The formula is as follows:

[0072]

[0073] in Represents the sigmod operation, It means element-wise multiplication. The final result can make full use of different frequency information.

[0074] Step 2.5: Transform the interaction result of step 2.4 through a fully connected layer to obtain a spatial feature map.

[0075] Specifically, a fully connected layer is used to perform dimensional transformation to map the high-dimensional matrix into a one-dimensional feature vector. To avoid overfitting, a random dropout layer is used with a probability of 0.2. Finally, the spatial feature map is obtained. For example, the number of low-frequency components n=16, dropout=0.2, the initial learning rate is 0.001, the Adam algorithm is used to optimize the model parameters, and the cross entropy is the model loss function.

[0076] Step 3: normalize the original electromyographic signal and input the normalized information into the second branch network to obtain a time feature map; the second branch network includes a GRU network and a CNN network.

[0077] Step 4: Use a fully connected layer to fuse the spatial feature map and the temporal feature map, and classify them through a softmax layer.

[0078] Embodiment 2:

[0079] On the basis of Example 1, Example 2 of the present application provides a more specific surface electromyography signal classification method based on a dual-branch multi-scale multi-stream network, including:

[0080] Step 1: Get the original electromyographic signal.

[0081] Step 2: extract features from the original electromyographic signal, and input the extracted features into the first branch network to obtain a spatial feature map; the first branch network includes several branch streams, a multi-scale semantic injection module, a multi-frequency channel attention mechanism module and a fully connected layer.

[0082] Step 3, normalizing the original electromyographic signal, and inputting the normalized information into the second branch network to obtain a time feature map; the second branch network includes a GRU network and a CNN network;

[0083] Step 3 includes:

[0084] Step 3.1, normalize the original electromyographic signal.

[0085] Specifically, the original electromyographic signal is normalized to obtain the sequence , the formula is as follows:

[0086]

[0087] in, represents the mean value of the electromyographic signal sequence, represents the variance of the EMG signal sequence, is the bias value.

[0088] Step 3.2: Input the normalized information into the GRU network for training to extract temporal feature information.

[0089] Specifically, the processed 16×300 one-dimensional sequence y is put into the GRU network for training, and the temporal feature information is extracted to obtain the feature map , expressed as:

[0090]

[0091] The specific parameter settings of the GRU network are: the number of input features is 300, the number of hidden layer nodes is 128, and the number of network layers is 2. The training of the GRU network mainly includes two gate units: update gate and reset gate. The update gate controls how much useful information the current state needs to retain from the historical state and how much new information needs to be accepted from the candidate state. The larger the value of the update gate, the more state information from the previous moment is brought in. The information from the previous moment and the current moment is right-multiplied by the weight matrix and , and then the added data is put into the update gate and then into the sigmoid function.

[0092]

[0093] Reset Gate Used to control candidate status Whether it depends on the state of the previous moment The larger the reset gate is, the more previous state information is written. The information of the previous moment and the current moment are right-multiplied by the weight matrix and , and then the added data is put into the reset gate and then into the sigmoid function.

[0094]

[0095] When the gate signal is received, the state value of the candidate hidden layer unit is To update, the specific operation is as follows:

[0096]

[0097] The final hidden state update method of the network node is:

[0098]

[0099] in The sigmoid function transforms the data into a value in the range of 0-1, acting as a gating signal. For previous information, To reset the gate, is the activation function, , , , , , , , are the gate neuron parameters, learned during training, is element-wise multiplication.

[0100] The feature map The temporal feature information is compressed in time scale by using a one-dimensional CNN with a convolution kernel size of 3 and a step size of 2, and then a one-dimensional CNN with a convolution kernel size of 1 and a step size of 1 is used to increase the feature dimension to obtain a temporal feature map. .

[0101] Step 3.3: Convert the above feature map Then perform two more fully connected layers with 512 units and 128 units to obtain the temporal feature map of the target. .

[0102] Step 4: Use a fully connected layer to fuse the spatial feature map and the temporal feature map, and classify them through a softmax layer.

[0103] Specifically, the fully connected layer is used to fuse the parameters of the two branch network training, and finally the Softmax layer is used for classification to obtain the final classification result, which is expressed as:

[0104]

[0105] The present invention proposes a surface electromyography signal classification method based on a dual-branch multi-scale multi-stream network to solve the surface electromyography signal classification problem. The robustness of the model is enhanced by multi-scale feature extraction and multi-stream structure processing, and the accuracy of gesture recognition is improved. In addition, a multi-frequency channel attention mechanism is introduced, which mines the low-frequency information of electromyography signals by introducing multiple frequency components and mines the connections between the internal information.

[0106] Table 2 Ablation experiment classification performance comparison report of algorithm models

[0107]

[0108] Table 3 Comparison of recognition accuracy of algorithm models before and after improvement

[0109]

[0110] At the same time, the improvements proposed in this application were subjected to ablation experiments and comparative experiments. The network model (MS-ETDTBN) with multi-scale multi-stream feature fusion proposed in this paper performed better in recognition of five electromyographic signal gestures in different situations, namely, hand open (HO), fist (HC), wrist flexion (RF), wrist flexion (WF), and wrist extension (WE), compared with the network model (ETDTBN) that did not use multi-scale multi-stream feature fusion, verifying its effectiveness and feasibility as shown in Tables 2 and 3.

[0111] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be described in detail in this application.

[0112] Embodiment 3:

[0113] On the basis of Example 1, Example 3 of the present application provides a surface electromyography signal classification system based on a dual-branch multi-scale multi-stream network, including:

[0114] The first acquisition module is used to acquire the original electromyographic signal;

[0115] A second acquisition module is used to extract features from the original electromyographic signal and input the extracted features into a first branch network to obtain a spatial feature map; the first branch network includes a plurality of branch streams, a multi-scale semantic injection module, a multi-frequency channel attention mechanism module and a fully connected layer;

[0116] A third acquisition module is used to normalize the original electromyographic signal and input the normalized information into a second branch network to obtain a time feature map; the second branch network includes a GRU network and a CNN network;

[0117] The classification module is used to fuse the spatial feature map and the temporal feature map using a fully connected layer and perform classification through a softmax layer.

[0118] Specifically, the system provided in this embodiment is a system corresponding to the method provided in Embodiments 1 and 2. Therefore, the parts in this embodiment that are the same or similar to Embodiments 1 and 2 can be referenced to each other and will not be repeated in this application.

Claims

1. A surface electromyography signal classification method based on a dual-branch multi-scale multi-stream network, characterized in that: include: Step 1: Obtain the original electromyographic signal; Step 2: extracting features from the original electromyographic signal, and inputting the extracted features into the first branch network to obtain a spatial feature map; The first branch network includes several branch streams, a multi-scale semantic injection module, a multi-frequency channel attention mechanism module and a fully connected layer; step 2 includes: Step 2.1, extracting features from the original electromyographic signal, and preprocessing the extracted features to obtain an initial feature map; Step 2.2, the initial feature map passes through several branch streams of different scales to obtain a deep semantic feature map corresponding to each branch stream; the branch stream includes several batch pooling convolution blocks; Step 2.3: Through the multi-scale semantic injection module, the deep semantic feature maps of branch streams of different scales are interacted to obtain interactive feature information; Step 2.4, the interactive feature information is passed through a multi-frequency channel attention mechanism module to obtain different frequency information; the different frequency information and the interactive feature information are interacted; Step 2.5: Transform the interaction result of step 2.4 through a fully connected layer to obtain a spatial feature map; Step 3, normalizing the original electromyographic signal, and inputting the normalized information into the second branch network to obtain a time feature map; the second branch network includes a GRU network and a CNN network; Step 3 includes: Step 3.1, normalizing the original electromyographic signal; Step 3.2: Input the normalized information into the GRU network for training to extract temporal feature information; Step 3.3, compressing the time feature information in time scale through a CNN network to obtain a time feature map; Step 4: Use a fully connected layer to fuse the spatial feature map and the temporal feature map, and classify them through a softmax layer.

2. The surface electromyography signal classification method based on a dual-branch multi-scale multi-stream network according to claim 1 is characterized in that: In step 2.1, the preprocessing includes enhancing two-dimensionalization.

3. The surface electromyography signal classification method based on a dual-branch multi-scale multi-stream network according to claim 2 is characterized in that: In step 2, the first branch network further includes a random dropout layer, and the input of the random dropout layer is connected to the output of the fully connected layer.

4. A surface electromyography signal classification system based on a dual-branch multi-scale multi-stream network, characterized in that: The method for executing any one of claims 1 to 3 comprises: The first acquisition module is used to acquire the original electromyographic signal; A second acquisition module is used to extract features from the original electromyographic signal and input the extracted features into a first branch network to obtain a spatial feature map; the first branch network includes a plurality of branch streams, a multi-scale semantic injection module, a multi-frequency channel attention mechanism module and a fully connected layer; A third acquisition module is used to normalize the original electromyographic signal and input the normalized information into a second branch network to obtain a time feature map; the second branch network includes a GRU network and a CNN network; The classification module is used to fuse the spatial feature map and the temporal feature map using a fully connected layer and perform classification through a softmax layer.

5. A computer storage medium, characterized in that: The computer storage medium stores a computer program; when the computer program is executed on a computer, the computer executes any one of the methods described in claims 1 to 3.

6. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 3.

Images