A multimodal gesture recognition method based on data-enhanced brain and myoelectric signal fusion

Abstract

The present invention discloses a multimodal gesture recognition method based on data-enhanced brain-electromyographic signal fusion, comprising: preprocessing an electromyographic signal and an electroencephalogram signal to obtain the preprocessed electromyographic signal and the electroencephalogram signal; constructing an electromyographic original signal image based on the preprocessed electromyographic signal, performing data enhancement processing on the electromyographic original signal image to obtain an electromyographic signal enhanced image; constructing an electroencephalogram original signal image based on the preprocessed electroencephalogram signal, performing data enhancement processing on the electroencephalogram original signal image to obtain an electroencephalogram enhanced image; extracting and classifying the features of the electromyographic signal from the electromyographic original signal image and the electromyographic signal enhanced image to obtain an electromyographic signal classification result; extracting and classifying the features of the electroencephalogram original signal image and the electroencephalogram enhanced image to obtain an electroencephalogram signal classification result; performing decision fusion on the electromyographic signal classification result and the electroencephalogram signal classification result to obtain a gesture recognition result.

Description

Technical Field

[0001] The present invention belongs to the field of biomedicine and human-computer interaction technology, and in particular relates to a multimodal gesture recognition method based on data-enhanced brain and myoelectric signal fusion. Background Art

[0002] The physiological electrical signals of the human body can accurately depict the human body's movement intentions, such as brain electroencephalogram signals, myoelectric signals and other neural signals. Gesture recognition technology based on brain electroencephalogram signals can be widely used in industrial production, rehabilitation medicine, virtual reality and other fields. In the field of industrial production, it can realize external mechanical control, reduce the physical exertion of practitioners, and ensure personal safety in high-risk scenarios; in the field of rehabilitation medicine, it can provide support for the disabled through bionic prostheses, exoskeleton robots, etc., to reduce life obstacles; in virtual reality, this technology can be combined with wearable devices to provide solutions for natural and intuitive human-computer interaction interfaces.

[0003] As an electrophysiological signal, EEG (Electroencephalogram) reflects the simplest and most direct state of brain activity. It can be acquired on the scalp surface with very high temporal resolution, non-invasively and at low cost, and the acquisition method is simple and safe. Brain-Computer Interface (BCI) is a communication channel between the brain and the outside world. Through EEG, various thinking activities in the brain can be clearly expressed. Accurately decoding motor intentions is a key component of the BCI system based on motor imagery EEG signals. However, for fine gesture movements, EEG signals obtained by motor imagery alone usually cannot achieve a high classification accuracy. In order to obtain better gesture intention recognition results, actual gesture movements are usually used for electromyographic signal analysis instead of motor imagery, because it is more intuitive, thereby improving the performance of intent parsing. At present, the deep learning network architectures that are widely used in the field of electromyographic gesture recognition mainly include convolutional neural networks and recurrent neural networks. Researchers continue to explore on the basis of these classic network architectures, and propose a variety of electromyographic gesture recognition algorithms by optimizing the configuration parameters of classic networks, combining and building deeper hybrid networks, and introducing the latest theoretical results of deep learning. These algorithms have achieved good recognition performance in their respective application scenarios. According to the core ideas of algorithm design, the relevant algorithms can be divided into four categories: gesture recognition algorithms based on convolutional neural networks, gesture recognition algorithms based on recurrent neural networks, gesture recognition algorithms based on multi-network hybrid architectures, and new gesture recognition algorithms based on other strategies in the field of deep learning.

[0004] In addition, decision-level fusion is also a way to perform accurate gesture intention recognition. Its basic idea is to fuse the classification results after pattern recognition of EEG signals and EMG signals. According to existing research, modal fusion methods can be roughly divided into four types: data fusion / feature-level fusion / decision-level fusion / model-level fusion. Among them, data-level fusion belongs to the low-level fusion, which may bring redundant information and difficulties in feature decoupling; feature set fusion is to extract and construct multiple modal data into corresponding modal features, and then splice them into a feature set that integrates the features of each modality, which can reduce the amount of data processing, but has high requirements on feature quality; decision-level fusion is to fuse the classification results after pattern recognition of EEG signals and EMG signals, which has the best requirements on data and high real-time performance. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a multimodal gesture recognition method based on data-enhanced brain-electromyographic signal fusion. For small sample scenarios, the synergy between brain-electromyographic signals is utilized to fuse the decision levels of brain-electromyographic signals, and the feature recognition accuracy is improved through a hybrid neural network model.

[0006] To achieve the above object, the present invention provides a multimodal gesture recognition method based on data-enhanced brain and myoelectric signal fusion, comprising:

[0007] Acquire electromyographic signals and electroencephalographic signals of the subject when the subject makes different gestures, preprocess the electromyographic signals and electroencephalographic signals, and acquire the preprocessed electromyographic signals and preprocessed electroencephalographic signals;

[0008] Constructing an EMG raw signal image based on the preprocessed EMG signal, performing data enhancement processing on the EMG raw signal image, and obtaining an EMG signal enhanced image; constructing an EEG raw signal image based on the preprocessed EEG signal, performing data enhancement processing on the EEG raw signal image, and obtaining an EEG signal enhanced image;

[0009] Using the original electromyographic signal image and the enhanced electromyographic signal image to perform feature extraction and classification of electromyographic signals, and obtain electromyographic signal classification results; using the original electroencephalographic signal image and the enhanced electroencephalographic signal image to perform feature extraction and classification of electroencephalographic signals, and obtain electroencephalographic signal classification results;

[0010] The electromyographic signal classification result and the electroencephalographic signal classification result are fused to obtain a gesture recognition result.

[0011] According to the multimodal gesture recognition method based on data-enhanced brain-electromagnetic signal fusion provided by the present invention, the method of obtaining the myoelectric signals and electroencephalographic signals of a subject when making different gestures includes: placing electromyographic acquisition electrodes on the muscles of the subject most relevant to hand movements, and recording the myoelectric signals generated by muscle activity when making different gestures; using a sensor element or an electroencephalographic cap to record the electroencephalographic signals generated by muscle activity when the subject makes different gestures.

[0012] According to the multimodal gesture recognition method based on data-enhanced brain and myoelectric signal fusion provided by the present invention, the method for constructing an electromyographic raw signal image based on the preprocessed electromyographic signal includes:

[0013] The preprocessed electromyographic signal is directly converted into an image to obtain each electromyographic picture-label object, and the electromyographic original signal image is constructed.

[0014] According to the multimodal gesture recognition method based on data-enhanced brain and myoelectric signal fusion provided by the present invention, the method for constructing an original EEG signal image based on the preprocessed EEG signal includes:

[0015] The preprocessed EEG signal is divided into several non-overlapping time segments according to the time dimension, and the EEG signal in each time segment is mapped into a two-dimensional space. The time axis is regarded as the horizontal axis of the image, and the signal intensity is regarded as the vertical axis. According to the mapping rules, the signal of each time period is converted into the pixel value of the image, and the original multi-dimensional time series is converted into a series of two-dimensional images, thereby completing the construction of the original EEG signal image.

[0016] According to the multimodal gesture recognition method based on data-enhanced brain and electromyographic signal fusion provided by the present invention, the method for obtaining the electromyographic signal enhanced image and the electroencephalographic signal enhanced image includes: based on supervised data enhancement, the original electromyographic signal image and the original electroencephalographic signal image are cropped and transformed, and then through a conversion step, the original electromyographic signal image and the original electroencephalographic signal image are data segmented using a sliding window, while keeping the sliding window length unchanged, the sliding step size is adjusted to obtain the electromyographic signal enhanced image and the electroencephalographic signal enhanced image.

[0017] According to the multimodal gesture recognition method based on data-enhanced brain and myoelectric signal fusion provided by the present invention, the method of extracting and classifying the features of the myoelectric signal using the original myoelectric signal image and the myoelectric signal enhanced image, and obtaining the myoelectric signal classification result includes:

[0018] The first branch of the three-branch multi-stream multi-feature neural processing network is used to divide the original electromyographic signal image and the enhanced electromyographic signal image into a plurality of sub-images according to the electrode arrangement order, and the plurality of sub-images are input into the multi-stream network in parallel to learn the association between gesture movements and muscle group combinations, and the spatiotemporal features in the original electromyographic signal image and the enhanced electromyographic signal image are extracted using a convolutional neural network and an attention mechanism; the extracted spatiotemporal features are fused, and shallow features, intermediate features in the middle of the multi-stream network, and deep features after convolution training are extracted through convolutional layers, local connection layers, and fully connected layers; the shallow features, the intermediate features, and the deep features are fused and input into a Softmax classifier for recognition and classification to obtain a first classification result;

[0019] The second branch of the three-branch multi-stream multi-feature neural processing network is used to analyze the association between the original electromyographic signal image and the electromyographic signal enhanced image and the gesture action, and the multi-channel shallow features are integrated into a whole, and a single-channel stream is used for learning and analysis to obtain a second classification result;

[0020] The third branch of the three-branch multi-stream multi-feature neural processing network is used to fuse the multi-channel intermediate features into a whole, and a single-channel stream is used for learning and analysis to obtain the third classification result;

[0021] The network fusion structure based on the voting mechanism adopts the voting mechanism to combine the first classification result, the second classification result and the third classification result to determine the final result. The output of each branch network is a vote for the gesture category, and the weighted combination of the probability values ​​output by the three branch networks is the final gesture classification result.

[0022] According to the multimodal gesture recognition method based on data-enhanced brain and myoelectric signal fusion provided by the present invention, the method of extracting and classifying the features of the EEG signal using the original EEG signal image and the EEG signal enhanced image, and obtaining the EEG signal classification result includes:

[0023] A selection-based sparse Transformer model is constructed. The input of the temporal Transformer model is cut into several non-overlapping time slices according to the time dimension and flattened into a one-dimensional vector, which is linearly mapped to the model dimension. The self-attention module in the temporal Transformer model architecture adopts a query-key-value mode, and the output is in the form of a weighted sum of values. A top-k selected sparse Transformer model is adopted. Attention is degraded into sparse attention through top-k selection at the Encoder layer to complete the construction of a selection-based sparse Transformer model.

[0024] The original EEG signal image and the enhanced EEG signal image are input into the selection-based sparse Transformer model to obtain an EEG signal classification result.

[0025] According to the multimodal gesture recognition method based on data-enhanced brain and myoelectric signal fusion provided by the present invention, the method of multimodally fusion of the myoelectric signal classification result and the brain electroencephalogram signal classification result to obtain the gesture recognition result is:

[0026] P(C|O1,O2)∝P(C)P(O1,O2|C)

[0027] P(O1,O2|C)=P(O1|C)P(O2|C)

[0028] C out =argmax(P(C=c)P(O1|C=c)P(O2|C=c))

[0029] Among them, P(C|O1,O2) is a conditional probability model, C represents the result of decision fusion, O1 is the classification result of EEG signal, O2 is the classification result of EMG signal, P(C) is the probability that the decision result is C, P(O1,O2|C) is the joint probability of observing that both EEG signal classification result O1 and EMG signal classification result O2 are correct under the condition of given decision C, P(O1|C) is the probability that the EEG signal classification result is correct under the condition of decision result C, P(O2|C) is the probability that the EMG signal classification result is correct under the condition of decision result C, C out is the decision result of brain-electromyographic signal fusion obtained by calculating the maximum joint probability, P(C=c) is the probability that the decision result is c, P(O1|C=c) is the probability that the EEG signal classification result is correct under the condition of decision result c, P(O2|C=c) is the probability that the EMG signal classification result is correct under the condition of decision result c, and C=c means that the result of decision fusion is c.

[0030] Technical effect of the invention: The invention discloses a multimodal gesture recognition method based on data-enhanced brain-electromyographic signal fusion, which realizes multimodal gesture recognition that integrates brain-electromyographic signals. By exploring the mechanism of coordinated recognition of electromyographic and electroencephalographic signals, the invention breaks the limitation of gesture recognition based on a single traditional electromyographic signal. The invention improves the feature extraction and classification part of the recognition model, which can effectively recognize not only static gestures at a certain moment, but also multiple gestures that are dynamically performed within a continuous period of time. It improves the accuracy while expanding the application scenarios, and makes the requirements for gesture initiators more relaxed, especially providing convenience for people with disabilities. The invention uses data enhancement of small sample signals and trims and transforms the original brain-electromyographic signals based on a supervised data enhancement method to generate new brain-electromyographic images, which can be used as training samples for gesture recognition models together with the original samples, making up for the problem of insufficient training data in small sample scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0032] Figure 1 A schematic diagram of a multimodal gesture recognition method based on data-enhanced brain and myoelectric signal fusion according to an embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram of an electromyographic signal classification model according to an embodiment of the present invention;

[0034] Figure 3 This is an overall architecture diagram of electromyography image processing based on dual-branch multi-stream multi-feature fusion CNN in an embodiment of the present invention;

[0035] Figure 4 This is a schematic diagram of the EEG gesture recognition and analysis process according to an embodiment of the present invention;

[0036] Figure 5 Schematic diagram of the EEG gesture recognition classification model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0038] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] like Figure 1-5 As shown, this embodiment provides a multimodal gesture recognition method based on data-enhanced brain and myoelectric signal fusion, including:

[0040] S1. Obtaining electromyographic signals and electroencephalographic signals of the subject when the subject makes different gestures, preprocessing the electromyographic signals and electroencephalographic signals, and obtaining the preprocessed electromyographic signals and preprocessed electroencephalographic signals;

[0041] S2, constructing an EMG original signal image based on the preprocessed EMG signal, performing data enhancement processing on the EMG signal image, and obtaining an EMG signal enhanced image; constructing an EEG original signal image based on the preprocessed EEG signal, performing data enhancement processing on the EEG signal image, and obtaining an EEG signal enhanced image;

[0042] S3, extracting and classifying the features of the electromyographic signal from the original electromyographic signal image and the electromyographic signal enhanced image, and obtaining the electromyographic signal classification result;

[0043] S4, extracting and classifying the features of the EEG signal from the original EEG signal image and the EEG signal enhanced image to obtain the EEG signal classification result;

[0044] S5. Perform multimodal fusion on the electromyography signal classification results and the electroencephalography signal classification results to obtain the gesture recognition results.

[0045] Further, S1, obtaining the myoelectric signals and electroencephalogram signals of the subjects when making different gestures, preprocessing the myoelectric signals and electroencephalogram signals, and obtaining the preprocessed myoelectric signals and preprocessed electroencephalogram signals, specifically including: the collection and preprocessing of electromyographic signals for gesture recognition is in a collection environment far away from mobile phones, transformers and other devices, and the subjects wear electromyographic and electroencephalogram collection equipment, and make preset action sequences in a prescribed order to synchronously collect continuous electroencephalogram and myoelectric signals. On this basis, by preprocessing the above continuous signals such as filtering, amplification and segmentation, the starting point and end point of the dynamic gesture action are identified, so as to perform dynamic gesture recognition.

[0046] A further embodiment comprises the following steps:

[0047] S11: Synchronously collect the EMG and EEG signals of the experimental subjects when they make different gestures. Select the muscles most related to hand movements for calibration, place electrodes on the surface of human skin, and use the acquisition equipment connected to the electrodes to record the potential change data of the relevant muscle tissue when the limbs complete different movements. The electrodes, amplification and filtering circuits, and trap circuits need to be packaged together, and the EMG signals need to be protected by a shielding layer to reduce the interference of conducted coupling. Configure the EEG signal acquisition equipment, including wearing an EEG cap on the subject's head to record the EEG signals generated by brain activity in a non-invasive manner. Attention should be paid to the synchronous acquisition of EMG and EEG signals to ensure the temporal consistency of the two signals so that they can accurately correspond in subsequent analysis.

[0048] S12: Preprocess the collected electromyographic signal EMG. For repeated gestures during data collection, the data from time t1 to time t2 is intercepted. The amount of data obtained on one channel at this time is N, and the calculation formula is as follows:

[0049]

[0050] Wherein, f represents the sampling frequency of the electrode device, in Hz; the units of t1 and t2 are in milliseconds.

[0051] The data is standardized using the Z-score normalization method, which standardizes the original data based on the mean and standard deviation of the sample data. The calculation formula is as follows:

[0052]

[0053] Where μ represents the mean of a segment of electromyographic signal on the corresponding channel, and σ represents the variance of a segment of electromyographic signal on the corresponding channel. After Z-score normalization, the variance of the new data set is normalized, and each dimension obeys a normal distribution with a mean of 0 and a variance of 1, avoiding the influence of the selection of data of different dimensions on network training.

[0054] The sliding window method is used to divide the EMG signal sequence into multiple data segments, each of which records the changes in the EMG signal within a short time interval. The multi-channel electromyographic data of the sliding window is set to w, the sliding step is s, and the unit is milliseconds. After segmentation, m equal-length data segments can be generated. The format of each segment is l represents the amount of data recorded by a single channel, then

[0055]

[0056] Among them, the sliding step size s in the sliding window method is often smaller than the window length w. This is to allow the generated data segments to overlap and reduce the feature loss caused by segmentation.

[0057] S13: Preprocess the collected EEG signals, including filtering, denoising, and artifact identification and removal, to obtain a data matrix of EEG signals. The frequency of EEG signals is mainly concentrated between 0.1 and 40 Hz. Therefore, the signal is first notched filtered at 50 Hz to eliminate power line interference, and then band-pass filtered from 0.1 to 40 Hz to retain the frequency components related to EEG activity.

[0058] S2, constructing an electromyographic signal image based on the preprocessed electromyographic signal, performing data enhancement processing on the electromyographic signal image, and obtaining an electromyographic signal enhanced image; constructing an electroencephalographic signal image based on the preprocessed electroencephalographic signal, performing data enhancement processing on the electroencephalographic signal image, and obtaining an electroencephalographic signal enhanced image; a further implementation method is to include the following steps:

[0059] S21: EMG signal image construction. The EMG signal fragments after standardization are in two-dimensional data format, which can be directly converted into images. That is, each data fragment It corresponds to an image with a length of c and a width of l. As a training sample for supervised learning, the electromyography images also need to be labeled with corresponding gesture action labels one by one. After completing all preprocessing steps, each "electromyography image-label" object obtained can be used as an independent sample for model training and recognition.

[0060] S22: Construction of EEG signal image, split the continuous EEG signal into k non-overlapping time segments according to the time dimension (k = number of channels, ensure the same input to the attention module). Map the EEG signal in each time segment into two-dimensional space. Consider the time axis as the horizontal axis of the image and the signal intensity as the vertical axis. According to the mapping rules, convert the signal of each time period into the pixel value of the image, thereby converting the original multi-dimensional time series into a series of two-dimensional images. Flatten the two-dimensional image of each time window into a one-dimensional vector and linearly map it to the top-k sparse time Transformer model dimension.

[0061] S23: Based on supervised brain / electromyographic image data enhancement, the original brain / electromyographic signal image is cropped and transformed to generate a new brain / electromyographic image. The specific operation steps are as follows:

[0062] S231: Determine the sliding window length w: The sliding window length determines the sample size extracted each time. This length should be long enough to contain enough EEG / muscle activity information, but not too long to avoid introducing too much redundancy.

[0063] S232: Adjust sliding step size s: The sliding step size is the length of overlap between consecutive samples. By adjusting the step size, the size of the overlapping area can be controlled. A smaller step size will produce more overlap, resulting in more samples, but it may also lead to more redundancy. A larger step size will reduce overlap and increase the diversity of samples, but may lose some continuity information.

[0064] S233: Keep the sample size consistent: In order to ensure that the generated image is consistent with the original brain / EMG image size, the total number of samples needs to be kept unchanged during the cropping process. When the original signal length L is given, the number of samples is:

[0065]

[0066] S234: Experiments and Verification: Determine the optimal sliding window length and step size through experiments, and evaluate the performance of the model under different parameter settings through methods such as cross-validation, so as to select the optimal cropping strategy.

[0067] S3. Extract and classify the features of the original EMG signal image and the enhanced EMG signal image to obtain the EMG signal classification result, including: using the first branch of a three-branch multi-stream multi-feature neural processing network to divide the original EMG signal image and the enhanced EMG signal image into several sub-images according to the electrode arrangement order, inputting the several sub-images into the multi-stream network in parallel, learning the association between gesture movements and muscle group combinations, and using convolutional neural networks and attention mechanisms to extract spatiotemporal features in the original EMG signal image and the enhanced EMG signal image; fusing the extracted spatiotemporal features, extracting shallow features, intermediate features in the middle of the multi-stream network, and deep features after convolution training through convolutional layers, local connection layers, and fully connected layers, and fusing the shallow features, intermediate features, and deep features and inputting them into the Softmax classifier for recognition and classification to obtain the first classification result;

[0068] The second branch of the three-branch multi-stream multi-feature neural processing network is used to analyze the association between the original EMG signal image and the EMG signal enhanced image and the gesture action, and the multi-channel shallow features are integrated into a whole. The single-channel stream is used for learning and analysis to obtain the second classification result.

[0069] The third branch of the three-branch multi-stream multi-feature neural processing network is used to fuse the multi-channel intermediate features into a whole, and a single-channel stream is used for learning and analysis to obtain the third classification result;

[0070] The network fusion structure based on the voting mechanism adopts the voting mechanism to combine the first classification results, the second classification results and the third classification results to determine the final result. The output of each branch network is a vote for the gesture category, and the weighted combination of the probability values ​​output by the three branch networks is the final gesture classification result.

[0071] A further embodiment comprises the following steps:

[0072] S31: Input the labeled EMG training set into the three-branch network module of the multi-stream multi-feature fusion CNN;

[0073] S32: In the first branch, the multi-stream characterization of the electromyographic signal is performed from the perspective of muscle group combination. The electromyographic signals collected by the electrodes with adjacent numbers are combined according to the electrode arrangement, i.e., the electromyographic image Divided into multiple sub-images of equal size And m×n=c. Each sub-image contains data captured by multiple adjacent channels, corresponding to the combination of forearm muscle groups working together. The sub-images obtained by segmentation are input into the multi-stream network in parallel to learn the association between gestures and forearm muscle groups. The number of channel streams in the network is equal to the number of sub-images represented by the multi-stream.

[0074] S33: In the first branch, if Figure 3 As shown in the figure, LSTM represents the long short-term memory network layer, AM represents the attention layer, FC represents the fully connected layer, and G in the G-way fully connected layer represents the total number of gesture categories to be recognized. In order to solve the problem of loss of detailed information during network training, in the iterative process of the convolutional neural network, the deep features extracted from multiple streams are fused, and finally Softmax is used as the gesture classification to obtain the classification result;

[0075] S34: In the second branch, the association between electromyographic signals and gestures is analyzed from a holistic perspective, and the multi-channel shallow features are fused into a whole. Then, a single-channel stream is used for learning and analysis, and Softmax is used as the gesture classifier to obtain the classification result.

[0076] S35: In the third branch, the multi-channel intermediate features are fused into a whole, and then a single-channel stream is used for learning and analysis, and Softmax is used as the gesture classifier to obtain the classification result;

[0077] S36: Network fusion structure based on voting mechanism, using voting mechanism to combine the outputs of three branch networks to determine the final result. The output of each branch network (the output of Softmax classifier) ​​is regarded as a vote for the gesture category. The final gesture classification result is a weighted combination of the probability values ​​output by the three branch networks. The specific calculation process is:

[0078] (1) Define the output of each branch as y i The final output is y final , then the network fusion formula is:

[0079] y final =fusion(y i ,...,y n ) (5)

[0080] (2) According to the definition in (1), the voting mechanism module in this algorithm receives the Softmax classifier output from the two branch network modules. The output format of the Softmax classifier is a vector containing K real numbers, where the K real numbers correspond to the probability of K kinds of gestures, and their sum is 1. Let the probability of the i-th gesture in the j-th network be denoted as s j (i) A voting mechanism is used to combine the outputs of the three branch networks to determine the final result, which is calculated as follows:

[0081]

[0082] (3) The gesture corresponding to the largest of the K probabilities is taken as the final output result and the final voting settlement basis, denoted as g final , the calculation formula is:

[0083] g final =argmax(probablity(i)),i=1...K (7)

[0084] S4, extracting and classifying the features of the EEG signal from the original EEG signal image and the enhanced EEG signal image to obtain the EEG signal classification result, including: dividing the preprocessed EEG signal into several non-overlapping time segments according to the time dimension, mapping the EEG signal in each time segment into a two-dimensional space, regarding the time axis as the horizontal axis of the image, regarding the signal intensity as the vertical axis, converting the signal of each time period into the pixel value of the image according to the mapping rule, converting the original multi-dimensional time series into a series of two-dimensional images, and completing the construction of the EEG signal image. A further implementation method is to include the following steps:

[0085] S41: The EEG signals were preprocessed to remove artifacts using a hybrid method of multivariate empirical mode decomposition (MEMD) and canonical correlation analysis (CCA) (MEMD-CCA);

[0086] S42: Construct a temporal Transformer model. The model input is cut into k non-overlapping time slices according to the time dimension and flattened into a one-dimensional vector (k = number of channels), which is linearly mapped to the model dimension. Figure 5As shown in the figure, the self-attention module in the Transformer model architecture adopts the query-key-value (QKV) mode, and the output is in the form of a weighted sum of values;

[0087] S43: In order to make the extracted spatiotemporal features of the EEG signal more obvious, a sparse Transformer model with top-k selection is used. Attention is degraded into sparse attention through top-k selection in its Encoder layer, that is, the signal that attracts attention is retained and irrelevant signals are deleted. The screening calculation process is as follows:

[0088] (1) Generate the attention score P, expressed as:

[0089]

[0090] (2) Based on the assumption that the larger the score, the greater the relevance, the attention score P is evaluated and a sparse attention masking operation M(·) is performed on P to select the top-k contributing elements. The masking function M(·) is as follows, where t i is the kth largest value in the i-th row, and the first k contributing elements with the highest scores are retained accordingly:

[0091]

[0092] S44: The EEG signals of a period of time rather than the signals at a certain time point are used as the research object and put into the Transformer model for training and verification.

[0093] S5. Perform multimodal fusion on the electromyographic signal classification results and the electroencephalographic signal classification results to obtain the gesture recognition results, including using the Bayesian algorithm to perform decision fusion on the electromyographic signal classification results and the electroencephalographic signal classification results; the calculation method for obtaining the gesture recognition results is:

[0094] P(C|O1,O2)∝P(C)P(O1,O2|C)

[0095] P(O1,O2|C)=P(O1|C)P(O2|C)

[0096] C out =argmax(P(C=c)P(O1|C=c)P(O2|C=c))

[0097] Among them, P(C|O1,O2) is a conditional probability model, C represents the result of decision fusion, O1 is the classification result of EEG signal, O2 is the classification result of EMG signal, P(C) is the probability that the decision result is C, P(O1,O2|C) is the joint probability of observing that both EEG signal classification result O1 and EMG signal classification result O2 are correct under the condition of given decision C, P(O1|C) is the probability that the EEG signal classification result is correct under the condition of decision result C, P(O2|C) is the probability that the EMG signal classification result is correct under the condition of decision result C, C out is the decision result of brain-electromyographic signal fusion obtained by calculating the maximum joint probability, P(C=c) is the probability that the decision result is c, P(O1|C=c) is the probability that the EEG signal classification result is correct under the condition of decision result c, P(O2|C=c) is the probability that the EMG signal classification result is correct under the condition of decision result c, and C=c means that the result of decision fusion is c.

[0098] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multimodal gesture recognition method based on data-enhanced brain and myoelectric signal fusion, characterized in that: The following steps are involved: Acquire electromyographic signals and electroencephalographic signals of the subject when the subject makes different gestures, preprocess the electromyographic signals and electroencephalographic signals, and acquire the preprocessed electromyographic signals and preprocessed electroencephalographic signals; Constructing an electromyographic original signal image based on the preprocessed electromyographic signal, performing data enhancement processing on the electromyographic original signal image, and obtaining an electromyographic signal enhanced image; Constructing an original EEG signal image based on the preprocessed EEG signal, and performing data enhancement processing on the original EEG signal image to obtain an enhanced EEG signal image; The original electromyographic signal image and the enhanced electromyographic signal image are used to extract and classify the features of the electromyographic signal to obtain the electromyographic signal classification result, specifically including: using the first branch of a three-branch multi-stream multi-feature neural processing network to divide the original electromyographic signal image and the enhanced electromyographic signal image into a plurality of sub-images according to the electrode arrangement order, inputting the plurality of sub-images into the multi-stream network in parallel, learning the association between gesture movements and muscle group combinations, and using a convolutional neural network and an attention mechanism to extract the spatiotemporal features in the original electromyographic signal image and the enhanced electromyographic signal image; fusing the extracted spatiotemporal features, extracting shallow features, intermediate features in the middle of the multi-stream network, and deep features after convolution training through convolution layers, local connection layers, and fully connected layers, and fusing the shallow features, the intermediate features, and the deep features and inputting them into a Softmax classifier for recognition and classification to obtain a first classification result; The second branch of the three-branch multi-stream multi-feature neural processing network is used to analyze the association between the original electromyographic signal image and the electromyographic signal enhanced image and the gesture action, and the multi-channel shallow features are integrated into a whole, and a single-channel stream is used for learning and analysis to obtain a second classification result; The third branch of the three-branch multi-stream multi-feature neural processing network is used to fuse the multi-channel intermediate features into a whole, and a single-channel stream is used for learning and analysis to obtain the third classification result; A network fusion structure based on a voting mechanism, wherein the voting mechanism is used to combine the first classification result, the second classification result and the third classification result to determine the final result, the output of each branch network is a vote for the gesture category, and the weighted combination of the probability values ​​output by the three branch networks is the final gesture classification result; Extracting and classifying EEG signals using the original EEG signal image and the enhanced EEG signal image to obtain EEG signal classification results; The electromyographic signal classification result and the electroencephalographic signal classification result are fused to obtain a gesture recognition result.

2. The multimodal gesture recognition method based on data-enhanced brain and myoelectric signal fusion as claimed in claim 1, characterized in that: The method for obtaining the electromyographic signals and electroencephalographic signals of a subject when making different gestures includes: placing electromyographic acquisition electrodes on the muscles of the subject that are most relevant to hand movements, and recording the electromyographic signals generated by muscle activity when making different gestures; using a sensor element or an electroencephalogram cap to record the electroencephalographic signals generated by muscle activity when the subject makes different gestures.

3. The multimodal gesture recognition method based on data-enhanced brain and myoelectric signal fusion as claimed in claim 1, characterized in that: The method for constructing an original electromyographic signal image based on the preprocessed electromyographic signal comprises: The preprocessed electromyographic signal is directly converted into an image to obtain each electromyographic picture-label object, and the electromyographic original signal image is constructed.

4. The multimodal gesture recognition method based on data-enhanced brain and myoelectric signal fusion as claimed in claim 1, characterized in that: The method for constructing an original EEG signal image based on the preprocessed EEG signal comprises: The preprocessed EEG signal is divided into several non-overlapping time segments according to the time dimension, and the EEG signal in each time segment is mapped into a two-dimensional space. The time axis is regarded as the horizontal axis of the image, and the signal intensity is regarded as the vertical axis. According to the mapping rules, the signal of each time period is converted into the pixel value of the image, and the original multi-dimensional time series is converted into a series of two-dimensional images, thereby completing the construction of the original EEG signal image.

5. The multimodal gesture recognition method based on data-enhanced brain and myoelectric signal fusion as claimed in claim 1, characterized in that: The method for obtaining the electromyogram enhanced image and the electroencephalogram enhanced image includes: based on supervised data enhancement, the original electromyogram signal image and the original electroencephalogram signal image are cropped and transformed, and then through a conversion step, a sliding window is used to perform data segmentation on the original electromyogram signal image and the original electroencephalogram signal image, and while keeping the sliding window length unchanged, the sliding step size is adjusted to obtain the electromyogram enhanced image and the electroencephalogram enhanced image.

6. The multimodal gesture recognition method based on data-enhanced brain and myoelectric signal fusion as claimed in claim 1, characterized in that: The method of extracting and classifying the features of the EEG signals by using the original EEG signal image and the enhanced EEG signal image to obtain the EEG signal classification result includes: A selection-based sparse Transformer model is constructed. The input of the temporal Transformer model is cut into several non-overlapping time slices according to the time dimension and flattened into a one-dimensional vector, which is linearly mapped to the model dimension. The self-attention module in the temporal Transformer model architecture adopts a query-key-value mode, and the output is in the form of a weighted sum of values. A top-k selected sparse Transformer model is adopted. Attention is degraded into sparse attention through top-k selection at the Encoder layer to complete the construction of a selection-based sparse Transformer model. The original EEG signal image and the enhanced EEG signal image are input into the selection-based sparse Transformer model to obtain an EEG signal classification result.

7. The multimodal gesture recognition method based on data-enhanced brain and myoelectric signal fusion as claimed in claim 1, characterized in that: The method of performing multimodal fusion of the electromyographic signal classification result and the electroencephalographic signal classification result to obtain the gesture recognition result is: P(C|O1,O2)∝P(C)P(O1,O2|C) P(O1,O2|C)=P(O1|C)P(O2|C) C out =argmax(P(C=c)P(O1|C=c)P(O2|C=c)) Among them, P(C|O1,O2) is a conditional probability model, C represents the result of decision fusion, O1 is the classification result of EEG signal, O2 is the classification result of EMG signal, P(C) is the probability that the decision result is C, P(O1,O2|C) is the joint probability of observing that both EEG signal classification result O1 and EMG signal classification result O2 are correct under the condition of given decision C, P(O1|C) is the probability that the EEG signal classification result is correct under the condition of decision result C, P(O2|C) is the probability that the EMG signal classification result is correct under the condition of decision result C, C out is the decision result of brain-electromyographic signal fusion obtained by calculating the maximum joint probability, P(C=c) is the probability that the decision result is c, P(O1|C=c) is the probability that the EEG signal classification result is correct under the condition of decision result c, P(O2|C=c) is the probability that the EMG signal classification result is correct under the condition of decision result c, and C=c means that the result of decision fusion is c.

Images