Electromyographic signal decoding method based on time-frequency fusion and double-branch network

Abstract

The invention discloses an electroencephalogram and electromyographic signal decoding method based on time-frequency fusion and a double-branch network. The method comprises the steps that an electroencephalogram and electromyographic synchronous experiment normal form is designed; denoising and normalizing the collected electroencephalogram and electromyographic signals; electroencephalogram channels are screened through wavelet coherence analysis, and time-frequency characteristics are obtained through short-time Fourier transform. And fusion is carried out according to the following two modes: stacking is carried out according to channels to obtain global features, and the Rayleigh entropy is utilized to obtain a significant feature frequency band of each channel and splice the significant feature frequency band into a new time-frequency graph. A double-branch network is designed to carry out feature extraction, a time domain residual module and a channel attention module are used to extract features of a spliced time-frequency graph, and a 3D residual module and a ConvLSTM module are used to capture dynamic changes and spatial-temporal features of a stacked time-frequency graph. Finally, the extracted features of the two branches are spliced and fused, error calculation and classified decoding are carried out through a classification module, and the accuracy and stability of electroencephalogram and myoelectricity fusion decoding are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of brain and myoelectric fusion decoding, and is used for multi-modal fusion decoding to enhance the accuracy of action classification. Background Art

[0002] Bioelectric signals are an important carrier of physiological information, containing rich physiological information of the human body. Electroencephalogram (EEG) and surface electromyography (sEMG) are important means of recording the activities of the central motor nervous system of the human body. They are widely used in motor rehabilitation and auxiliary control of patients with neurological diseases such as stroke. Closed-loop rehabilitation based on single EEG or sEMG has made significant progress and has attracted widespread attention from the clinical and academic circles, and has achieved relatively gratifying results. However, there are many disadvantages of single modality signals. EEG signals based on motor imagery have poor recognition accuracy and poor stability due to low signal-to-noise ratio. sEMG signals also have some limitations. Due to the influence of factors such as muscle morphology, muscle fatigue, and muscle activation defects caused by disability, it is difficult to identify different muscle activities.

[0003] The hybrid brain-computer interface (hBCI) that fuses EEG and sEMG signals can effectively improve the problems existing in single-modal signals. The fusion of EEG and sEMG signals can provide a more effective intention recognition interface that directly connects to different parts of the nervous system. EEG signals provide direct observation of brain activity and can capture relevant information about movement intention and conscious activity. sEMG signals provide the characteristics of muscle activity. The fusion of these two signals can provide a more comprehensive understanding of the relationship between the brain and muscles during movement execution. The fusion of EEG-sEMG signals can improve the accuracy and stability of action classification and reduce the possibility of misclassification. This method of comprehensive use of brain and muscle signals is expected to promote progress in the fields of rehabilitation, assistive device control, and bring more possibilities to the fields of human-computer interaction and medical health. How to effectively use EEG and sEM signals for multimodal fusion and practical application is the most concerned issue at present.

[0004] The inventor's prior application, CN117807553A, is a method for fusion decoding of EMG signals based on time-frequency convolution. First, the EMG and EMG signals are synchronously triggered and collected, and the real-time signals are stored to construct a EMG synchronization signal data set; the pre-processed EMG signals are subjected to short-time Fourier transform to obtain the time domain fusion features and frequency domain independent features of the EMG signals respectively; the deep time domain features of the EMG signals are obtained through the designed dual-branch network structure, and the frequency domain attention mechanism is introduced in the frequency domain feature branch to enhance the effectiveness of the EMG signals in exploring different frequency domain components and improve the extraction and expression of EMG frequency domain features; finally, the time domain features and frequency domain features of the EMG signals are spliced ​​and fused in the later stage of the feature layer, and the fused feature vectors are input into two fully connected layers, and finally they are decoded and classified, which effectively improves the accuracy and stability of EMG fusion decoding.

[0005] Patent CN117807553A proposes a technical solution for screening frequency bands containing effective information in the later stage of feature extraction and splicing and fusing these frequency bands. This method can effectively retain the information of electroencephalogram (EEG) and electromyography (sEMG) in the synchronous time domain. However, it fails to fully consider the characteristics of the time-frequency graph in the process of frequency band selection, resulting in the failure to extract more effective frequency bands. At the same time, this method discards a large amount of global information when screening effective frequency bands, limiting the comprehensiveness of the fusion effect. Based on the above problems, the present invention proposes an improved time-frequency fusion method, which aims to retain global information and local information at the same time. Specifically, the method captures global information by stacking time-frequency graphs, and introduces Rayleigh entropy to extract local information with significant features, thereby generating a new time-frequency graph. This fusion strategy can realize the combination of global and local features in the time-frequency domain, improve the effectiveness of feature extraction and the comprehensiveness of information retention. Summary of the invention

[0006] The present invention aims to solve the above problems of the prior art. A method for decoding brain electromyographic signals based on time-frequency fusion and a dual-branch network is proposed. The technical solution of the present invention is as follows:

[0007] A method for decoding brain electromyographic signals based on time-frequency fusion and a dual-branch network comprises the following steps:

[0008] Step 1: Design an experimental paradigm for synchronous acquisition of EEG-sEMG signals based on four-category unilateral arm movements according to EEG motor imagery and muscle execution, including acquisition equipment, experimental process, synchronous trigger device, execution movements and acquisition conditions;

[0009] Step 2: Preprocessing the raw EEG and sEMG data obtained in step 1;

[0010] Step 3: Use wavelet coherence analysis to select the EEG channel with the strongest coherence with the sEMG channel for subsequent fusion;

[0011] Step 4: Based on the screened brain electromyography channels, short-time Fourier transform (STFT) is used to characterize the time-frequency characteristics of EEG and sEMG;

[0012] Step 5: Adopt two different time-frequency fusion strategies, including global fusion based on channel stacking and local feature fusion based on Rayleigh entropy.

[0013] Step 6: Design a 2D branch for feature extraction of the spliced ​​spectrogram. The 2D branch includes a time domain residual module and a channel attention module.

[0014] Step 7: Design a 3D branch for extracting features from the channel stacked spectrum sequence. The 2D branch includes: a 3D residual module and a ConvLSTM module.

[0015] Step 8: Expand the 2D branch features and 3D branch features obtained in step 6 and step 7 respectively into one-dimensional feature vectors, then perform feature concatenation, and use two fully connected layers and a softmax layer to convert the output into classification probability.

[0016] Further, the step 1: design an EEG-sEMG signal synchronous acquisition experimental paradigm based on the four-category action of the unilateral arm, specifically including four actions: clenching fist, opening five fingers, wrist inversion, and wrist abduction; using delsys as the electromyographic acquisition device, selecting 3 electrodes as the number of channels, and placing them at the position of the short wrist flexor, palmaris longus, and ulnar wrist flexor of the right arm, the sampling frequency is set to 1925Hz, and using Brain Products 32-lead EEG cap as the EEG acquisition device, the sampling frequency is set to 250Hz. Specific experimental process: 0-2s, at the beginning of the experiment, the computer screen displays "+" to remind the subject to prepare for motor imagination and action execution; 2-5s, the screen displays the corresponding action picture, at which time the subject performs the action required on the screen and performs the corresponding motor imagination; 5s-8s, the action is completed, rest and wait for the next signal to be sent.

[0017] Furthermore, the step 2: performs corresponding preprocessing on the collected EEG and sEMG signals, specifically including: baseline correction and filtering of the original sEMG signal, using 20-500Hz bandpass filtering, and then using 50Hz notch filtering to eliminate power frequency interference; then extracting the action segment signal from the processed data, and finally downsampling the sEMG signal from 1925Hz to 250Hz in order to synchronize with the EEG frequency for subsequent fusion; firstly, the original data is bandpass filtered at 0.1-40Hz and baseline corrected, and then ICA is used to eliminate the interference of muscle and eye movement artifact signals, and finally the EEG action segment is extracted.

[0018] Furthermore, the step 3 uses wavelet coherence analysis to screen out the EEG channel with the strongest coherence with the sEMG channel, specifically including: using 3 EEG channels to be consistent with the sEMG channel; selecting 3 EEG channels based on the wavelet coherence analysis method, a total of 6 channels of time domain data; wavelet coherence quantifies the similarity of two signals by comparing their relative amplitude and phase relationship in the time-frequency domain; for EEG signal x(t) and sEMG signal y(t), using Morlet wavelet as wavelet basis function to obtain wavelet transform coefficients; therefore, the coherence of EEG signal x(t) and sEMG signal y(t) at scale s and time t can be defined as:

[0019]

[0020] Where S xy (s,t) is the cross-correlation of x(t) and y(t), S xx (s,t),S yy (s, t) are the autocorrelations of x(t) and y(t), respectively; wavelet coherence C xy (s,t) takes values ​​between 0 and 1. The closer the value is to 1, the stronger the similarity between EEG and sEMG signals at this time-frequency point.

[0021] Furthermore, the step 4 uses short-time Fourier transform (STFT) to characterize the time-frequency characteristics of EEG and sEMG based on the screened brain electromyography channels, specifically including:

[0022] The definition of STFT is:

[0023]

[0024] f∈R represents frequency, x(t)∈L 2 (R) represents the original brain myoelectric signal for a long time, τ∈R represents the given time, g(t)∈L 2 (R) represents the window function, and the Hanning window is selected as the window function of the short-time Fourier transform.

[0025] Furthermore, the step 5: a global fusion method based on channel stacking and a local feature fusion method based on Rayleigh entropy specifically includes:

[0026] For the EEG and sEMG time-frequency diagrams of each channel converted in step 4, the time-frequency diagrams of the EEG and sEMG signals are stacked in the channel dimension in the channel stacking manner to obtain a spectrum diagram frame sequence with 6 channels, retaining the global information of each spectrum diagram; the other is to splice the channels in series to fuse the two signal features. The synchronously collected EEG and sEMG signals are consistent in the time dimension, so the spectrum diagrams of the EEG and sEMG signals are spliced ​​together in channel order; the spectrum data fragments with low correlation in each time-frequency diagram are discarded to reduce redundant data in the spectrum features; Rayleigh entropy is used to perform frequency band screening, specifically:

[0027]

[0028] Where p is the normalized probability of the time-frequency diagram, a is the order parameter of the Rayleigh entropy, and the lower the Rayleigh entropy value, the more concentrated the energy of the corresponding signal is, and the more significant the characteristics are. The EEG frequency bands and sEMG frequency bands with significant characteristics are screened out by Rayleigh entropy, and these frequency bands are spliced ​​in the frequency domain dimension according to the channel order, and then a frequency spectrum with fused features is constructed to more comprehensively characterize the characteristic information of EEG and sEMG.

[0029] Furthermore, in step 6, a 2D branch is designed for feature extraction of the spliced ​​spectrogram. The 2D branch includes a time domain residual module and a channel attention module, specifically including:

[0030] The spliced ​​time-frequency map in step 5 is used as the single-frame spectrum map obtained after feature splicing, and a time-domain-residual module is designed to extract image features. First, a time-domain convolution layer with a large-scale convolution kernel is used to extract shallow time-domain features, and then the extracted shallow feature map is input into a two-layer time-domain convolution unit with a small-scale convolution kernel. The designed convolution unit is used to explore deeper time-frequency features. A residual mechanism is introduced in the time-domain convolution layer and the convolution unit to improve the reuse of shallow features. Finally, the obtained features are input into the channel attention module for feature aggregation to enhance cross-channel information exchange.

[0031] Furthermore, in step 7: the stacked time-frequency graph sequence is used as the input of the 3D branch, a 3D convolutional layer with residual connection is used to extract common information between sequences, and then a ConvLSTM module is used to extract long-term spatiotemporal correlation information between features; finally, the 2D branch features and 3D branch features obtained in steps 6 and 7 respectively are expanded into one-dimensional feature vectors, and then feature concatenation is performed, and the output is converted into classification probability using two fully connected layers and a softmax layer.

[0032] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the brain electromyography signal decoding method based on time-frequency fusion and a dual-branch network as described in any one of the items when executing the program.

[0033] A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a method for decoding electroencephalogram (EMG) signals based on time-frequency fusion and a dual-branch network as described in any one of the items.

[0034] The advantages and beneficial effects of the present invention are as follows:

[0035] First, most current methods use decision-level fusion or simple feature-level fusion. These fusion methods ignore the synergy between the two signals and cannot fully utilize their complementary characteristics.

[0036] The main advantage of the present invention is that a method for decoding brain electromyographic signals based on time-frequency fusion and a dual-branch network is proposed. The EEG and sEMG spectrograms after short-time Fourier transform are stacked to retain global features. At the same time, the frequency bands with significant features in each spectrogram are selected based on Rayleigh entropy to splice to form a new time-frequency graph to aggregate local features. A dual-stream hybrid residual network combining attention mechanism and ConvLSTM is designed. The network uses multiple models to construct a dual-branch hybrid structure, aiming to make full use of the global and local features of the spectrogram and optimize the feature representation of motion decoding. The present invention can achieve effective classification of actions for different tasks and different experimental paradigms. The use of this model to fuse EEG signals and sEMG signals can improve the reliability and effectiveness of rehabilitation training for neurological diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The present invention provides a preferred embodiment of a brain electromyography signal decoding method based on time-frequency fusion and a dual-branch network. DETAILED DESCRIPTION

[0038] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0039] The technical solution of the present invention to solve the above technical problems is:

[0040] As shown in the figure, the method for decoding brain electromyographic signals based on global-local time-frequency fusion and a dual-branch network provided in this embodiment includes the following steps:

[0041] Step 1: Design an experimental paradigm for synchronous acquisition based on EEG motor imagery and muscle execution, including acquisition equipment, experimental process, synchronous trigger device, execution action and acquisition status, etc.

[0042] Step 2: Preprocess the raw EEG and sEMG data obtained in step 1.

[0043] Step 3: Use wavelet coherence analysis to select the EEG channel with the strongest coherence with the sEMG channel for subsequent fusion.

[0044] Step 4: Based on the screened EMG channels, short-time Fourier transform (STFT) is used to characterize the time-frequency characteristics of EEG and sEMG.

[0045] Step 5: In order to extract and represent features more effectively, we adopted two different time-frequency fusion strategies, including global fusion based on channel stacking and local feature fusion based on Rayleigh entropy.

[0046] Step 6: Design a 2D branch for feature extraction of the spliced ​​spectrum graph. The 2D branch includes: a time domain residual module and a channel attention module.

[0047] Step 7: Design a 3D branch for extracting features from channel-stacked spectrum sequences. The 2D branch includes a 3D residual module and a ConvLSTM module.

[0048] Step 8: Expand the 2D branch features and 3D branch features obtained in step 6 and step 7 respectively into one-dimensional feature vectors, then perform feature concatenation, and use two fully connected layers and a softmax layer to convert the output into classification probability.

[0049] Further, step 1: design an EEG-sEMG signal synchronous acquisition experimental paradigm based on the four-category action of unilateral arm, including four actions: fist clenching, five fingers opening, wrist inversion, and wrist abduction. We use delsys as the electromyographic acquisition device, select 3 electrodes as the number of channels, and place them at the short wrist flexor, palmaris longus, ulnar wrist flexor and other positions of the right arm, and the sampling frequency is set to 1925Hz. Brain Products 32-lead EEG cap is used as the EEG acquisition device, and the sampling frequency is set to 250Hz. Specific experimental process: 0-2s, at the beginning of the experiment, the computer screen displays "+" to remind the subject to prepare for motor imagination and action execution. 2-5s, the screen displays the corresponding action picture, at which time the subject performs the action required on the screen and performs the corresponding motor imagination. 5s-8s, the action is completed, rest and wait for the next signal to be issued.

[0050] In step 2, the collected EEG and sEMG signals are preprocessed accordingly: the original sEMG signal is baseline corrected and filtered, a 20-500Hz bandpass filter is used, and a 50Hz notch filter is used to eliminate power frequency interference. Then the action segment signal is extracted from the processed data. Finally, in order to synchronize with the EEG frequency for subsequent fusion, the sEMG signal is downsampled from 1925Hz to 250Hz. First, the original data is bandpass filtered at 0.1-40Hz and baseline corrected. Then, ICA is used to eliminate the interference of signals such as muscle and eye movement artifacts, and finally the EEG action segment is extracted.

[0051] For step 3, considering the computational cost and complexity caused by too many EEG channels, and in order to achieve the synchronous fusion of EEG signals and sEMG signals in the experiment, we use 3 EEG channels to keep consistent with the sEMG channels. Based on the wavelet coherence analysis method, 3 EEG channels are selected, with a total of 6 channels of time domain data. Wavelet coherence quantifies the similarity of two signals by comparing their relative amplitude and phase relationship in the time-frequency domain. For EEG signal x(t) and sEMG signal y(t), Morlet wavelet is used as the wavelet basis function to obtain the wavelet transform coefficients. Therefore, the coherence of EEG signal x(t) and sEMG signal y(t) at scale s and time t can be defined as:

[0052]

[0053] Where S xy (s,t) is the cross-correlation of x(t) and y(t), S xx (s,t),S yy (s, t) are the autocorrelations of x(t) and y(t) respectively. Wavelet coherence C xy(s,t) takes values ​​between 0 and 1. The closer the value is to 1, the stronger the similarity between EEG and sEMG signals at this time-frequency point.

[0054] For step 4, the STFT is one of the most widely used time-frequency analysis methods. STFT plays a trade-off between time-based and frequency-based representations. The definition of STFT is:

[0055]

[0056] f∈R represents frequency, x(t)∈L 2 (R) represents the original brain myoelectric signal for a long time, τ∈R represents the given time, g(t)∈L 2 (R) represents the window function. This scheme selects the Hanning window as the window function of the short-time Fourier transform.

[0057] The step 5: for the EEG and sEMG time-frequency diagrams of each channel converted in step 4, the time-frequency diagrams of the EEG and sEMG signals are stacked in the channel dimension in a channel stacking manner to obtain a spectrum diagram frame sequence with 6 channels. This method effectively retains the global information of each spectrum diagram. Another method is to splice the channels in series to fuse the two signal features. The synchronously collected EEG and sEMG signals are consistent in the time dimension, so the spectrum diagrams of the EEG and sEMG signals can be spliced ​​together in channel order. Discard the spectral data fragments with low correlation in each time-frequency diagram to reduce redundant data in the spectral features. Use Rayleigh entropy to perform frequency band screening, specifically:

[0058]

[0059] Where p is the normalized probability of the time-frequency graph, and a is the order parameter of the Rayleigh entropy. The lower the Rayleigh entropy value, the more concentrated the energy of the corresponding signal, and the more significant features it has. The Rayleigh entropy is used to screen out the EEG frequency bands and sEMG frequency bands with significant features, and these frequency bands are spliced ​​in the frequency domain dimension according to the channel order, and then a fusion feature spectrum is constructed to more comprehensively characterize the feature information of EEG and sEMG.

[0060] The step 6 is specifically as follows: the spliced ​​time-frequency graph in step 5 is used as the single-frame spectrum graph obtained after feature splicing, and a time-domain-residual module is designed to extract image features. First, a time-domain convolution layer with a large-scale convolution kernel is used to extract shallow time-domain features, and then the extracted shallow feature map is input into a two-layer time-domain convolution unit with a small-scale convolution kernel. The designed convolution unit is used to explore deeper time-frequency features. A residual mechanism is introduced in the time-domain convolution layer and the convolution unit to improve the reuse of shallow features. Finally, the obtained features are input into the channel attention module for feature aggregation to enhance cross-channel information exchange.

[0061] Step 7: Use the stacked time-frequency graph sequence as the input of the 3D branch, use the 3D convolution layer with residual connection to extract the common information between the sequences, and then use the ConvLSTM module to extract the long-term spatiotemporal correlation information between the features. Finally, the 2D branch features and 3D branch features obtained in steps 6 and 7 are expanded into one-dimensional feature vectors, and then feature concatenation is performed, and the output is converted into classification probability using two fully connected layers and a softmax layer.

[0062] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions.

[0063] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0064] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0065] The above embodiments should be understood to be only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the contents of the present invention, technicians can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A method for decoding brain electromyographic signals based on time-frequency fusion and dual-branch network, characterized in that: The following steps are involved: Step 1: Design an experimental paradigm for synchronous acquisition of EEG-sEMG signals based on four-category unilateral arm movements according to EEG motor imagery and muscle execution, including acquisition equipment, experimental process, synchronous trigger device, execution movements and acquisition conditions; Step 2: Preprocessing the raw EEG and sEMG data obtained in step 1; Step 3: Use wavelet coherence analysis to select the EEG channel with the strongest coherence with the sEMG channel for subsequent fusion; Step 4: Based on the screened brain electromyography channels, short-time Fourier transform (STFT) is used to characterize the time-frequency characteristics of EEG and sEMG; Step 5: Adopt two different time-frequency fusion strategies; Including global fusion method based on channel stacking and local feature fusion method based on Rayleigh entropy; Step 6: Design a 2D branch for feature extraction of the spliced ​​spectrogram. The 2D branch includes a time domain residual module and a channel attention module. Step 7: Design a 3D branch for extracting features from the channel stacked spectrum sequence. The 2D branch includes: a 3D residual module and a ConvLSTM module. Step 8: Expand the 2D branch features and 3D branch features obtained in step 6 and step 7 respectively into one-dimensional feature vectors, then perform feature concatenation, and use two fully connected layers and a softmax layer to convert the output into classification probability.

2. The method for decoding brain electromyographic signals based on time-frequency fusion and dual-branch network according to claim 1, characterized in that: The step 1: design an EEG-sEMG signal synchronous acquisition experimental paradigm based on the four-category action of the unilateral arm, specifically including four actions: fist clenching, five fingers opening, wrist inversion, and wrist abduction; use delsys as the electromyographic acquisition device, select 3 electrodes as the number of channels, and place them at the position of the short wrist flexor, palmaris longus, and ulnar wrist flexor of the right arm, the sampling frequency is set to 1925Hz, and use Brain Products 32-lead EEG cap as the EEG acquisition device, and the sampling frequency is set to 250Hz. Specific experimental process: 0-2s, at the beginning of the experiment, the computer screen displays "+" to remind the subject to prepare for motor imagination and action execution; 2-5s, the screen displays the corresponding action picture, at which time the subject performs the action required on the screen and performs the corresponding motor imagination; 5s-8s, the action is completed, rest and wait for the next signal to be sent.

3. The method for decoding brain electromyographic signals based on time-frequency fusion and dual-branch network according to claim 1, characterized in that: The step 2: performing corresponding preprocessing on the collected EEG and sEMG signals, specifically including: performing baseline correction and filtering on the original sEMG signal, using 20-500Hz bandpass filtering, and then using 50Hz notch filtering to eliminate power frequency interference; then extracting action segment signals from the processed data, and finally downsampling the sEMG signal from 1925Hz to 250Hz in order to synchronize with the EEG frequency for subsequent fusion; firstly performing 0.1-40Hz bandpass filtering on the original data, and performing baseline correction, and then using ICA to eliminate the interference of muscle and eye movement artifact signals, and finally extracting the EEG action segment.

4. The method for decoding brain electromyographic signals based on time-frequency fusion and dual-branch network according to claim 1, characterized in that: The step 3 uses wavelet coherence analysis to screen out the EEG channel with the strongest coherence with the sEMG channel, specifically including: using 3 EEG channels to be consistent with the sEMG channel; selecting 3 EEG channels based on the wavelet coherence analysis method, a total of 6 channels of time domain data; wavelet coherence quantifies the similarity of two signals by comparing their relative amplitude and phase relationship in the time-frequency domain; for EEG signal x(t) and sEMG signal y(t), using Morlet wavelet as the wavelet basis function to obtain wavelet transform coefficients; therefore, the coherence of EEG signal x(t) and sEMG signal y(t) at scale s and time t can be defined as: Where S xy (s,t) is the cross-correlation of x(t) and y(t), S xx (s,t),S yy (s, t) are the autocorrelations of x(t) and y(t), respectively; wavelet coherence C xy (s,t) takes values ​​between 0 and 1. The closer the value is to 1, the stronger the similarity between EEG and sEMG signals at this time-frequency point.

5. The method for decoding brain electromyographic signals based on time-frequency fusion and dual-branch network according to claim 1, characterized in that: The step 4 uses short-time Fourier transform (STFT) to characterize the time-frequency characteristics of EEG and sEMG based on the screened brain electromyography channels, specifically including: The definition of STFT is: f∈R represents frequency, x(t)∈L 2 (R) represents the original brain myoelectric signal for a long time, τ∈R represents the given time, g(t)∈L 2 (R) represents the window function, and the Hanning window is selected as the window function of the short-time Fourier transform.

6. The method for decoding brain electromyographic signals based on time-frequency fusion and dual-branch network according to claim 1, characterized in that: The step 5: a global fusion method based on channel stacking and a local feature fusion method based on Rayleigh entropy, specifically includes: For the EEG and sEMG time-frequency diagrams of each channel converted in step 4, the time-frequency diagrams of the EEG and sEMG signals are stacked in the channel dimension in the channel stacking manner to obtain a spectrum diagram frame sequence with 6 channels, retaining the global information of each spectrum diagram; the other is to splice the channels in series to fuse the two signal features. The synchronously collected EEG and sEMG signals are consistent in the time dimension, so the spectrum diagrams of the EEG and sEMG signals are spliced ​​together in channel order; the spectrum data fragments with low correlation in each time-frequency diagram are discarded to reduce redundant data in the spectrum features; Rayleigh entropy is used to perform frequency band screening, specifically: Where p is the normalized probability of the time-frequency diagram, a is the order parameter of the Rayleigh entropy, and the lower the Rayleigh entropy value, the more concentrated the energy of the corresponding signal is, and the more significant the characteristics are. The EEG frequency bands and sEMG frequency bands with significant characteristics are screened out by Rayleigh entropy, and these frequency bands are spliced ​​in the frequency domain dimension according to the channel order, and then a frequency spectrum with fused features is constructed to more comprehensively characterize the characteristic information of EEG and sEMG.

7. The method for decoding brain electromyographic signals based on time-frequency fusion and dual-branch network according to claim 1, characterized in that: The step 6 designs a 2D branch for feature extraction of the spliced ​​spectrogram. The 2D branch includes: a time domain residual module and a channel attention module, specifically including: The spliced ​​time-frequency map in step 5 is used as the single-frame spectrum map obtained after feature splicing, and a time-domain-residual module is designed to extract image features. First, a time-domain convolution layer with a large-scale convolution kernel is used to extract shallow time-domain features, and then the extracted shallow feature map is input into a two-layer time-domain convolution unit with a small-scale convolution kernel. The designed convolution unit is used to explore deeper time-frequency features. A residual mechanism is introduced in the time-domain convolution layer and the convolution unit to improve the reuse of shallow features. Finally, the obtained features are input into the channel attention module for feature aggregation to enhance cross-channel information exchange.

8. The method for decoding brain electromyographic signals based on time-frequency fusion and dual-branch network according to claim 1, characterized in that: The step 7: using the stacked time-frequency graph sequence as the input of the 3D branch, using a 3D convolutional layer with residual connection to extract common information between sequences, and then using a ConvLSTM module to extract long-term spatiotemporal correlation information between features; finally, the 2D branch features and 3D branch features obtained in step 6 and step 7 are respectively expanded into a one-dimensional feature vector, and then feature concatenation is performed, and the output is converted into a classification probability using two fully connected layers and a softmax layer.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the brain electromyography signal decoding method based on time-frequency fusion and dual-branch network as described in any one of claims 1 to 8 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for decoding electroencephalogram (EMG) signals based on time-frequency fusion and a dual-branch network as described in any one of claims 1 to 8 is implemented.

Images