A brain and myoelectric signal decoding method based on time-frequency fusion and double-branch network

Abstract

The application claims a brain and muscle electrical signal decoding method based on time-frequency fusion and double-branch network, comprising the following steps: designing a brain and muscle electrical synchronous experiment paradigm; carrying out denoising and normalization processing on the collected brain and muscle electrical signals; using wavelet coherence analysis to screen the electroencephalogram channel and using short-time Fourier transform to obtain time-frequency characteristics; fusing in the following two ways: one is to stack by channel to obtain global characteristics, and the other is to use the Renyi entropy to obtain the significant feature band of each channel and splice into a new time-frequency graph; designing a double-branch network to extract features, using a time-domain residual module and a channel attention module to extract the features of the spliced time-frequency graph, using a 3D residual module and a ConvLSTM module to capture the dynamic changes and space-time characteristics of the stacked time-frequency graph; finally, splicing and fusing the features extracted by the two branches, using a classification module to calculate errors and classify decoding, effectively improving the accuracy and stability of the brain and muscle electrical fusion decoding.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of brain-muscle electrical fusion decoding, and is used for enhancing the accuracy of motion classification in multi-modal fusion decoding. BACKGROUND

[0002] Bioelectrical signals are an important carrier of physiological information, containing rich physiological information of the human body. Electroencephalogram (EEG) and surface electromyogram (sEMG) are important means to record the activity of the central motor system of the human body. They are widely used in the motor rehabilitation and auxiliary control of patients with neurological diseases such as stroke. Closed-loop rehabilitation based on single EEG or sEMG has made significant progress and attracted widespread attention from the clinical and academic communities, and has achieved gratifying results. However, single modal signal has many drawbacks. EEG signal based on motor imagery has low signal-to-noise ratio, resulting in poor recognition accuracy and poor stability. sEMG signal also has some limitations. Due to the influence of muscle morphology, muscle fatigue, and muscle activation defects caused by disability, it is difficult to identify different muscle activities.

[0003] Hybrid brain-computer interface (hBCI) that fuses EEG and sEMG signals can effectively improve the problems existing in single modal signal. The fusion of EEG and sEMG signals can provide a more effective intention recognition interface, directly connecting to different parts of the nervous system. EEG signal provides a direct observation of brain activity, which can capture relevant information of action intention and conscious activity, and sEMG signal provides muscle activity features. The fusion of these two signals can provide a more comprehensive understanding of the relationship between brain and muscle during motor execution. The fusion of EEG-sEMG signals can improve the accuracy and stability of motion classification and reduce the possibility of misclassification. This method of comprehensively utilizing brain and muscle signals is expected to promote progress in the fields of rehabilitation, auxiliary device control, and bring more possibilities to human-computer interaction and medical health fields. How to effectively utilize EEG and sEM signals for multi-modal fusion and practical application is the most concerned problem at present.

[0004] Prior application of the inventor, CN117807553A, a brain and muscle electrical signal fusion decoding method based on time-frequency convolution. First, the synchronous trigger collection of brain electrical signals and muscle electrical signals is performed, and the real-time signals are stored to construct a brain and muscle electrical synchronous signal dataset; the preprocessed brain and muscle electrical signals are subjected to short-time Fourier transform to obtain the time domain fusion features and frequency domain independent features of the brain and muscle electrical signals; a double-branch network structure is designed to obtain deep-level time domain features of the brain and muscle electrical signals, and a frequency domain attention mechanism is introduced in the frequency domain feature branch to enhance the effectiveness of the brain and muscle electrical signals in exploring different frequency domain components and improve the extraction and expression of the brain electrical frequency domain features; finally, the time domain features and frequency domain features of the brain and muscle electrical signals are spliced and fused in the late feature layer, and the fused feature vector is input into two fully connected layers, and finally the decoding classification is performed, effectively improving the accuracy and stability of the brain and muscle electrical fusion decoding.

[0005] Patent CN117807553A proposes a technical solution of filtering and splicing the frequency bands containing effective information in the late stage of feature extraction. This method can effectively retain the information of electroencephalogram (EEG) and electromyogram (sEMG) in the synchronous time domain. However, it fails to fully consider the characteristics of time-frequency graph in the frequency band selection process, resulting in the failure to extract more effective frequency bands. At the same time, this method discards a large amount of global information when filtering effective frequency bands, limiting the comprehensiveness of the fusion effect. Based on the above problems, the present invention proposes an improved time-frequency fusion method aimed at retaining both global information and local information. Specifically, this method stacks time-frequency graphs to capture global information and introduces Rayleigh entropy to extract local information with significant features, thereby generating new time-frequency graphs. This fusion strategy can combine global and local features within the time-frequency domain, improving the effectiveness of feature extraction and the comprehensiveness of information retention. SUMMARY

[0006] The present invention aims to solve the problems of the above prior art. A brain and muscle electrical signal decoding method based on time-frequency fusion and double-branch network is proposed. The technical solution of the present invention is as follows:

[0007] A brain and muscle electrical signal decoding method based on time-frequency fusion and double-branch network, comprising the following steps:

[0008] Step 1: Design an EEG-sEMG signal synchronous collection experiment paradigm based on unilateral arm four-class action according to brain electrical motor imagery and muscle execution, including collection equipment, experiment process, synchronous trigger device, execution action and collection condition;

[0009] Step 2: Perform preprocessing operation on the original EEG and sEMG data obtained in step 1;

[0010] Step 3: Use wavelet coherence analysis to screen out the EEG channel with the strongest coherence with the sEMG channel for subsequent fusion;

[0011] Step 4: Based on the screened brain and muscle electrical channels, use short-time Fourier transform (STFT) to represent the time-frequency features of EEG and sEMG;

[0012] Step 5: Take two different time-frequency fusion strategies; including global fusion based on channel stacking and local feature fusion based on ReLu entropy;

[0013] Step 6: Design a 2D branch for feature extraction of the spliced spectrogram, which includes a time domain residual module and a channel attention module;

[0014] Step 7: Design a 3D branch for feature extraction of the channel-stacked spectrogram sequence, which includes a 3D residual module and a ConvLSTM module;

[0015] Step 8: Expand the 2D branch features and 3D branch features obtained in steps 6 and 7 into one-dimensional feature vectors, then perform feature splicing, and use two fully connected layers and a softmax layer to output the classification probability.

[0016] Further, the step 1: design a single-arm four-class action-based EEG-sEMG signal synchronous acquisition experiment paradigm, specifically including clenched fist, five fingers open, wrist inversion, and wrist extension; use delsys as the muscle electrical acquisition device, select 3 electrodes as the channel number, and place them at the right arm wrist flexor carpi ulnaris, palmaris longus, and ulnar flexor carpi ulnaris positions, with a sampling frequency of 1925Hz; use Brain Products 32-channel EEG cap as the EEG acquisition device, with a sampling frequency of 250Hz. The specific experimental process is as follows: 0-2s, when the experiment starts, the computer screen displays "+", reminding the subject to prepare for motor imagery and action execution; 2-5s, the screen displays the corresponding action picture, at which time the subject performs the action required by the screen and performs the corresponding motor imagery; 5s-8s, the action is completed, and the subject rests and waits for the next signal.

[0017] Further, the step 2: the collected EEG and sEMG signals are preprocessed, specifically including: the original sEMG signal is subjected to baseline correction and filtering processing, 20-500Hz band-pass filtering is adopted, and then 50Hz notch filtering is used to eliminate power frequency interference; then the processed data is subjected to action segment signal extraction, and finally in order to be synchronized with the brain electrical frequency and facilitate subsequent fusion, the sEMG signal is down-sampled, and the sampling frequency is down-sampled from 1925Hz to 250Hz; first, the original data is subjected to 0.1-40Hz band-pass filtering and baseline correction, then ICA is used to eliminate the interference of muscle and eye movement artifacts, and finally the EEG action segment is extracted.

[0018] Further, the step 3 utilizes wavelet coherence analysis to screen out the EEG channel with the strongest coherence with the sEMG channel, specifically including: using 3 EEG channels consistent with the sEMG channel; based on the wavelet coherence analysis method, 3 EEG channels are selected, and a total of 6 channel time domain data; the wavelet coherence quantifies the similarity of two signals by comparing the relative amplitude and phase relationship of the signals in the time-frequency domain; for the EEG signal x(t) and the sEMG signal y(t), Morlet wavelet is used as the wavelet base function to obtain the wavelet transform coefficient; therefore, the coherence of the EEG signal x(t) and the sEMG signal y(t) at scale s and time t can be defined as:

[0019]

[0020] Where S xy (s,t) is the cross-correlation of x(t) and y(t), S xx (s,t), S yy (s,t) are the autocorrelations of x(t) and y(t) respectively; the wavelet coherence C xy (s,t) takes a value between 0 and 1, and the value closer to 1 indicates that the EEG and sEMG signals have strong similarity at the time-frequency point.

[0021] Further, the step 4 uses short-time Fourier transform STFT to represent the time-frequency characteristics of EEG and sEMG based on the screened brain muscle electrical channels, specifically including:

[0022] The definition of STFT is:

[0023]

[0024] f∈R represents frequency, x(t)∈L 2 (R) represents a long-time original brain muscle electrical signal, τ∈R represents a given time, g(t)∈L 2 (R) represents a window function, and the Hanning window is selected as the window function of the short-time Fourier transform.

[0025] Further, the step 5: based on the global fusion mode of channel stacking and the local feature fusion mode based on the Renyi entropy, specifically comprising:

[0026] For each channel of the converted EEG and sEMG time-frequency graph in step 4, the time-frequency graph of the EEG and sEMG signal is stacked in the channel dimension according to the channel stacking mode, and a spectrum graph frame sequence with a channel number of 6 is obtained, which retains the global information of each spectrum graph; Another is to concatenate the channel to fuse the features of the two signals, the synchronously collected EEG signal and sEMG signal are consistent in the time dimension, therefore the spectrum graphs of the EEG and sEMG signals are spliced together in the channel order; Discard the spectrum data segment with low correlation in each time-frequency graph to reduce the redundant data in the spectrum feature; use Renyi entropy to screen the frequency band, specifically:

[0027]

[0028] Where p is the normalized probability of the time-frequency graph, and a is the order parameter of the Renyi entropy. The lower the Renyi entropy value, the more concentrated the energy of the corresponding signal, and the more significant features it has. The EEG frequency band and the sEMG frequency band with significant features are selected by Renyi entropy, and these frequency bands are spliced in the frequency domain dimension according to the channel order, and then a fusion feature spectrum graph is constructed to more comprehensively represent the feature information of the EEG and sEMG.

[0029] Further, the step 6 designs a 2D branch for feature extraction of the spliced spectrum graph, and the 2D branch includes a time domain residual module and a channel attention module, specifically comprising:

[0030] The spliced time-frequency graph in step 5 is used as a single-frame spectrum graph after feature splicing, and a time domain-residual module is designed to extract image features; first, a time domain convolution layer with a large-scale convolution kernel is used to extract shallow time domain features, and then the extracted shallow feature map is input into two layers of time domain convolution units with a small-scale convolution kernel. The designed convolution unit is used to explore deeper time-frequency features; a residual mechanism is introduced in the time domain convolution layer and the convolution unit to improve the reuse of shallow features; finally, the obtained features are input into a channel attention module to aggregate features to enhance cross-channel information exchange.

[0031] Further, the step 7: stack the time-frequency map sequence as the input of the 3D branch, use the 3D convolution layer with residual connection to extract the common information between the sequences, and then use the ConvLSTM module to extract the long-term spatio-temporal correlation information between the features; finally, the 2D branch features and the 3D branch features obtained in steps 6 and 7 are respectively unfolded into one-dimensional feature vectors, then the features are spliced, and two fully connected layers and a softmax layer are used to output the classification probability.

[0032] An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the brain and muscle electrical signal decoding method based on time-frequency fusion and double-branch network according to any one of the embodiments when executing the program.

[0033] A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the brain and muscle electrical signal decoding method based on time-frequency fusion and double-branch network according to any one of the embodiments when executed by a processor.

[0034] The advantages and beneficial effects of the present application are as follows:

[0035] First, the current method mostly uses decision layer fusion or simple feature layer fusion, which ignores the synergistic effect between the two signals and cannot fully utilize their complementary characteristics.

[0036] The main advantage of the present application is to provide a brain and muscle electrical signal decoding method based on time-frequency fusion and double-branch network. The EEG and sEMG spectrum maps after short-time Fourier transform are stacked to retain global features, and the characteristic frequency bands in each spectrum map are selected based on the Rényi entropy to splice and form new time-frequency maps to aggregate local features. A double-flow hybrid residual network combining attention mechanism and ConvLSTM is designed. The network adopts a double-branch hybrid structure constructed by multiple models, aiming to fully utilize the global and local features of the spectrum map and optimize the feature representation of motion decoding. The present application can effectively classify actions under different tasks and different experimental paradigms. The use of the model to fuse EEG signals and sEMG signals can improve the reliability and effectiveness of the rehabilitation training of neurological diseases. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The present application provides a preferred embodiment of a brain and muscle electrical signal decoding method based on time-frequency fusion and double-branch network. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described clearly and in detail below with reference to the drawings in the embodiments of the present application. The described embodiments are only a part of the embodiments of the present application.

[0039] The technical scheme of the present application to solve the above technical problems is:

[0040] As shown in the figure, the brain and muscle electrical signal decoding method based on global-local time-frequency fusion and double-branch network provided by the embodiment includes the following steps:

[0041] Step 1: Design a synchronous acquisition experiment paradigm according to the brain electrical motor imagination and muscle execution, including acquisition equipment, experiment process, synchronous triggering device, execution action and acquisition condition, etc.

[0042] Step 2: Perform preprocessing operation on the original EEG and sEMG data obtained in step 1.

[0043] Step 3: Use wavelet coherence analysis to screen out the EEG channel with the strongest coherence with the sEMG channel for subsequent fusion.

[0044] Step 4: Based on the screened brain and muscle electrical channels, use short-time Fourier transform (STFT) to represent the time-frequency features of EEG and sEMG.

[0045] Step 5: In order to more effectively extract and represent the features, we take two different time-frequency fusion strategies. Including global fusion based on channel stacking and local feature fusion based on Rényi entropy.

[0046] Step 6: Design a 2D branch for feature extraction of the spliced spectrogram, including a time domain residual module and a channel attention module.

[0047] Step 7: Design a 3D branch for feature extraction of the channel-stacked spectrogram sequence, including a 3D residual module and a ConvLSTM module.

[0048] Step 8: Expand the 2D branch features and 3D branch features obtained in steps 6 and 7 into one-dimensional feature vectors respectively, then perform feature splicing, and use two fully connected layers and a softmax layer to output the classification probability.

[0049] Further, the step 1: design a single arm four-class action based on the EEG-sEMG signal synchronous acquisition experiment paradigm, including clenched fist, five fingers open, wrist inversion, wrist extension four actions. We use delsys as the myoelectric collection device, select 3 electrodes as the channel number, and place them in the right arm wrist flexor carpi ulnaris, palmaris longus, ulnar flexor carpi and other positions, and the sampling frequency is set to 1925 Hz. The Brain Products 32 lead EEG cap is used as the EEG acquisition device, and the sampling frequency is set to 250 Hz. The specific experimental process is as follows: 0-2s, when the experiment starts, the computer screen displays "+", which is used to remind the subjects to prepare for motor imagination and action execution. 2-5s, the screen displays the corresponding action picture, at this time the subjects perform the action required by the screen and perform the corresponding motor imagination. 5s-8s, the action is completed, rest and wait for the next signal.

[0050] In step 2, the collected EEG and sEMG signals are preprocessed: the original sEMG signal is baseline corrected and filtered, 20-500Hz band-pass filtering is used, and 50Hz notch filtering is used to eliminate power frequency interference. Then the processed data is extracted, and finally in order to synchronize with the EEG frequency and facilitate subsequent fusion, the sEMG signal is down-sampled to 250Hz from 1925Hz. First, the original data is band-pass filtered at 0.1-40Hz and baseline corrected, then ICA is used to eliminate muscle and eye movement artifacts and other signal interference, and finally the EEG action segment is extracted.

[0051] For step 3, considering the computational cost and complexity caused by too many EEG channels, and in order to realize the synchronous fusion of EEG signal and sEMG signal in the experiment, we use 3 EEG channels consistent with the sEMG channel. Based on the wavelet coherence analysis method, 3 EEG channels are selected, a total of 6 channel time domain data. Wavelet coherence quantifies the similarity of two signals by comparing their relative amplitude and phase relationship in the time-frequency domain. For EEG signal x(t) and sEMG signal y(t), Morlet wavelet is used as the wavelet basis function to obtain the wavelet transform coefficient. Therefore, the coherence of EEG signal x(t) and sEMG signal y(t) at scale s and time t can be defined as:

[0052]

[0053] Where S xy (s,t) is the cross-correlation of x(t) and y(t), S xx (s,t), S yy (s,t) are the autocorrelations of x(t) and y(t), respectively. The wavelet coherence C xy(s, t) is valued between 0-1, the value is closer to 1, indicating that the EEG and sEMG signals have strong similarity at this time-frequency point.

[0054] For step 4, the STFT selected is one of the most widely used time-frequency analysis methods, and the STFT plays a trade-off between time-based and frequency-based representations. The definition of STFT is:

[0055]

[0056] f∈R represents the frequency, x(t)∈L 2 (R) represents the original brain muscle signal, τ∈R represents the given time, g(t)∈L 2 (R) represents the window function, and the Hanning window is selected as the window function of the short-time Fourier transform in the scheme.

[0057] The step 5: for each channel of the EEG and sEMG time-frequency diagram converted in step 4, the time-frequency diagram of the EEG and sEMG signal is stacked in the channel dimension in the form of channel stacking, and a spectrum diagram frame sequence with a channel number of 6 is obtained. This method effectively preserves the global information of each spectrum diagram. Another is to splice the two signal features in series to fuse the two signals. The EEG signal and the sEMG signal collected synchronously are consistent in the time dimension, so the spectrum diagrams of the EEG and sEMG signals can be spliced together in the channel order. Discard the low-correlation spectrum data segment in each time-frequency diagram, and reduce the redundant data in the spectrum feature. The Reimann entropy is used for frequency band screening, specifically:

[0058]

[0059] Where p is the normalized probability of the time-frequency diagram, and a is the order parameter of the Reimann entropy. The lower the Reimann entropy value of the region, the more concentrated the energy of the corresponding signal, and the more significant features it has. The EEG frequency band and the sEMG frequency band with significant features are screened out by the Reimann entropy, and these frequency bands are spliced in the frequency domain dimension according to the channel order, and then a spectrum diagram with fused features is constructed to more comprehensively represent the feature information of the EEG and sEMG.

[0060] The step 6 is specifically: taking the spliced time-frequency diagram in step 5 as a single frame spectrum diagram obtained after feature splicing, and a time domain-residual module is designed to extract image features. First, a time domain convolution layer with a large-scale convolution kernel is used to extract shallow time domain features, and then the extracted shallow feature map is input into two layers of time domain convolution units with small-scale convolution kernels, and the designed convolution unit is used to explore deeper time-frequency features. The residual mechanism is introduced in the time domain convolution layer and the convolution unit to improve the reuse of shallow features. Finally, the obtained features are input into a channel attention module to perform feature aggregation to enhance the cross-channel exchange of information.

[0061] The step 7: taking the stacked time-frequency diagram sequence as the input of the 3D branch, using the 3D convolution layer with residual connection to extract the common information between the sequences, and then using the ConvLSTM module to extract the long-term spatio-temporal correlation information between the features. Finally, the 2D branch features and the 3D branch features obtained in steps 6 and 7 are respectively unfolded into one-dimensional feature vectors, then the features are spliced, and two fully connected layers and a softmax layer are used to output the classification probability.

[0062] The system, device, module or unit illustrated in the above embodiments can be specifically implemented by a computer chip or entity, or by a product with certain functions.

[0063] The computer readable medium includes permanent and non-permanent, removable and non-removable media, which can be realized by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition in this paper, computer readable medium does not include transitory computer readable medium, such as modulated data signal and carrier wave.

[0064] It is also to be noted that the terms "comprising", "including", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0065] The above examples are to be understood only as illustrative of the application and not a restriction on the scope of protection of the application. After reading the specification, the skilled person can make various changes or modifications to the application, and these equivalent changes and modifications also fall within the scope defined by the claims of the application.

Claims

1. A brain-muscle electroencephalogram signal decoding method based on time-frequency fusion and a double-branch network, characterized in that, The method comprises the following steps: Step 1: design an EEG-sEMG signal synchronous acquisition experiment paradigm based on unilateral arm four-class action according to electroencephalogram motor imagery and muscle execution, including acquisition equipment, experiment process, synchronous trigger device, execution action and acquisition condition; Step 2: perform pretreatment operation on the original EEG and sEMG data obtained in step 1; Step 3: utilize wavelet coherence analysis to screen out the EEG channels with the strongest coherence with the sEMG channels, for subsequent fusion; Step 4: based on the screened brain and muscle electrical channels, use short-time Fourier transform (STFT) to represent the time-frequency features of the EEG and sEMG; Step 5: adopt two different time-frequency fusion strategies; including a global fusion mode based on channel stacking and a local feature fusion mode based on Swin Entropy; Step 6: design a 2D branch for feature extraction of the spliced spectrum graph, and the 2D branch comprises a time domain residual module and a channel attention module; Step 7: design a 3D branch for feature extraction of the channel-stacked spectrum graph sequence, and the 3D branch comprises a 3D residual module and a ConvLSTM module; Step 8: expand the 2D branch features and the 3D branch features obtained in steps 6 and 7 into one-dimensional feature vectors respectively, then perform feature splicing, and use two fully connected layers and a softmax layer to output the conversion into classification probabilities; The step 5: the global fusion mode based on channel stacking and the local feature fusion mode based on Swin Entropy, specifically comprises: For each channel's EEG and sEMG time-frequency graph converted in step 4, the time-frequency graphs of the EEG and sEMG signals are stacked in the channel dimension according to the channel stacking mode, obtaining a spectrum graph frame sequence with 6 channels, which retains the global information of each spectrum graph; another is to splice and concatenate the two signal features by channel splicing, and the EEG signal and the sEMG signal are consistent in the time dimension during synchronous acquisition, so the spectrum graphs of the EEG and the sEMG signals are spliced together in the order of channels; discard the spectrum data segments with low correlation in each time-frequency graph to reduce the redundant data in the spectrum features; use Swin Entropy to screen frequency bands, specifically: wherein is the normalized probability of the time-frequency map, is the order parameter of the Renyi entropy, the lower the Renyi entropy value of a region, the more concentrated the energy of the corresponding signal, and the more significant features it has; the EEG frequency bands and sEMG frequency bands with significant features are screened out through the Renyi entropy, and these frequency bands are spliced in the frequency domain dimension according to the channel order, and then a fusion feature spectrum is constructed, so as to more comprehensively represent the feature information of EEG and sEMG. The step 6 designs a 2D branch for feature extraction of the spliced spectrum graph, and the 2D branch comprises a time domain residual module and a channel attention module, specifically comprising: The spliced time-frequency graph in step 5 is taken as a single-frame spectrum graph after feature splicing, and a time domain-residual module is designed to extract image features; first, a time domain convolution layer with a large-scale convolution kernel is used to extract shallow time domain features, then the extracted shallow feature map is input into two layers of time domain convolution units with a small-scale convolution kernel, and the designed convolution unit is used to explore deeper time-frequency features; a residual mechanism is introduced in the time domain convolution layer and the convolution unit to improve the reuse of shallow features; finally, the obtained features are input into a channel attention module to aggregate features and enhance cross-channel information exchange; The step 7: the stacked time-frequency map sequence is taken as the input of the 3D branch, common information between sequences is extracted using the 3D convolution layer with residual connection, then the ConvLSTM module is used to extract the long-term spatio-temporal correlation information between features; finally, the 2D branch features and the 3D branch features obtained in the step 6 and the step 7 are respectively unfolded into one-dimensional feature vectors, then feature splicing is performed, and two fully connected layers and a softmax layer are used to output the classification probability.

2. The brain-muscle electroencephalogram signal decoding method based on time-frequency fusion and double-branch network according to claim 1, characterized in that, The step 1: an EEG-sEMG signal synchronous acquisition experiment paradigm based on unilateral arm four-classification action is designed, specifically including four actions of clenched fist, five fingers open, wrist inversion and wrist extension; delsys is used as the electromyography acquisition device, three electrodes are selected as the channel number and placed at the position of the right arm wrist flexor carpi ulnaris, palmaris longus and ulnar flexor carpi, the sampling frequency is set to 1925Hz, and the Brain Products 32-channel electroencephalogram cap is used as the EEG acquisition device, the sampling frequency is set to 250Hz; the specific experiment process is as follows: 0-2s, when the experiment starts, the computer screen displays "+", which is used to remind the subject to prepare for motor imagination and action execution; 2-5s, the screen displays the corresponding action picture, at this time the subject performs the action required by the screen and performs corresponding motor imagination; 5s-8s, the action is completed, rest and wait for the next signal. 3.The brain-muscle electroencephalogram signal decoding method based on time-frequency fusion and double-branch network according to claim 1, wherein, The step 2: the collected EEG and sEMG signals are preprocessed, specifically including: the original sEMG signal is subjected to baseline correction and filtering processing, 20-500Hz band-pass filtering is adopted, and then 50Hz notch filtering is used to eliminate power frequency interference; then the processed data is subjected to action segment signal extraction, and finally in order to synchronize with the electroencephalogram frequency and facilitate subsequent fusion, the sEMG signal is down-sampled to 250Hz from the original sampling frequency of 1925Hz; first, the original EEG data is subjected to 0.1-40Hz band-pass filtering and baseline correction, then ICA is used to eliminate the interference of muscle and eye movement artifacts, and finally the EEG action segment is extracted.

4. The brain-muscle electroencephalogram signal decoding method based on time-frequency fusion and double-branch network according to claim 1, characterized in that, The step 3 uses wavelet coherence analysis to screen out the EEG channel with the strongest coherence with the sEMG channel, specifically including: using 3 EEG channels to keep consistent with the sEMG channel; selecting 3 EEG channels based on the wavelet coherence analysis method, a total of 6 channel time domain data; the wavelet coherence quantifies the similarity of two signals by comparing the relative amplitude and phase relationship of the two signals in the time-frequency domain; for the EEG signal and the sEMG signal , Morlet wavelet is used as the wavelet base function to obtain the wavelet transform coefficient; therefore, the coherence of the EEG signal and the sEMG signal at the scale and the time is defined as: wherein is and the cross-correlation of are and the autocorrelation of; wavelet coherence The value is between 0-1, the closer to 1, the stronger the similarity between the EEG and sEMG signals at this time-frequency point.

5. The method of claim 1, wherein, The step 4: based on the screened brain and muscle electricity channels, the short-time Fourier transform STFT is used to represent the time-frequency features of the EEG and sEMG, specifically including: The definition of STFT is: representing the frequency, representing the raw brain myoelectric signal over a long time, representing a given time, representing a window function, the Hanning window is chosen as the window function for the short-time Fourier transform.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the brain and muscle electricity signal decoding method based on time-frequency fusion and double branch network in any one of claims 1 to 5.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the brain and muscle electricity signal decoding method based on time-frequency fusion and double branch network in any one of claims 1 to 5.

Images