Method for decoding motor imagery electroencephalogram signals based on full-scale filtering double-granularity interaction

Abstract

The application discloses a kind of motion imagination eeg signal decoding methods based on full-scale filtering double granularity interaction, comprising: obtaining motion imagination eeg signal and pretreatment;Frequency decoupling is carried out to filtered motion imagination signal;Based on the time-frequency filter designed, complete global time-domain features and all important local time-domain features are captured;Time convolution and space convolution are applied to map to space with more abundant potential features;Based on double granularity feature interaction module, space coarse-grained information and space fine-grained information are guided to mutually excavate meaningful supplementary information;Full connection layer is applied to output the probability that motion imagination eeg signal belongs to certain motion imagination category, and finally realize the decoding of motion imagination eeg signal.The application creatively realizes the capture of complete global time-domain features and all important local time-domain features in a parameter-friendly manner, and innovatively designs a double granularity feature interaction module to comprehensively and systematically excavate meaningful supplementary information.

Description

Technical Field

[0001] This invention relates to the technical field of decoding motor imagery EEG signals, and in particular to a method for decoding motor imagery EEG signals based on full-scale filtering and dual-granularity interaction. Background Technology

[0002] Brain-computer interface (BCI) systems enable human-computer interaction without muscle intervention by analyzing human brain activity and translating it into commands to control external devices. Electroencephalogram (EEG) signals are favored by researchers due to their non-invasiveness, low cost, and high temporal resolution. In the classic paradigm of EEG-based BCIs, motor imagery allows users to complete tasks autonomously without external stimulation; therefore, BCIs based on motor imagery EEG signals are increasingly attracting research attention. Motor imagery refers to the mental rehearsal of specific limb movements by the subject without actual execution. Related studies have shown that motor imagery induces event-related synchronization and event-related desynchronization in the sensorimotor cortex. Specifically, the μ and β rhythms in the sensorimotor cortex undergo energy modulation. BCI systems based on motor imagery EEG signals have already been widely applied in fields such as medical rehabilitation.

[0003] Brain-computer interface (BCI) systems based on motor imagery EEG signals typically consist of three parts: motor imagery EEG signal acquisition, motor imagery EEG signal decoding, and external devices. Among these, motor imagery EEG signal decoding is crucial for the development of BCI systems. The performance of the decoding method directly affects the effectiveness of the BCI system. Generally, motor imagery-based BCIs consist of two phases: offline training and online testing. In the offline training phase, subjects perform specific motor imagery tasks according to instructions and generate training EEG signals. In the online testing phase, subjects perform specific motor imagery, generate corresponding EEG signals, and directly input them into the trained BCI to control external devices. Because the process of acquiring motor imagery EEG signals is relatively cumbersome, and there are significant differences in the EEG signals generated by subjects performing the same motor imagery task on different days, the motor imagery EEG signal decoding method needs to have sufficient generalization ability.

[0004] Significant progress has been made in research on motor imagery decoding methods by introducing deep learning architectures. Deep learning architectures can automatically learn to extract high-level features from motor imagery EEG signals in an end-to-end manner, eliminating the need for complex preprocessing. Currently, commonly used deep learning-based motor imagery decoding methods can be divided into two categories: those based on multi-scale convolutional neural networks and those based on Transformers. However, both of these deep learning-based methods still have many shortcomings. For multi-scale convolutional neural network-based methods, these methods often neglect the extraction of global temporal features from motor imagery EEG signals, simply applying descriptive statistics layers to extract these features. Furthermore, these methods often use filter banks with fixed receptive fields to extract multi-scale local temporal features, making it difficult to cover all potentially important local temporal features. Moreover, in order to ensure that the decoding method has sufficient generalization ability, each receptive field has multiple filters, which increases the number of trainable parameters. For Transformer-based motion imagery decoding methods, the inherent computing mechanism within Transformer leads to a large number of trainable parameters and computational cost. In addition, these methods often first perform preliminary filtering of the motion imagery EEG signal based on a convolutional neural network before inputting it into a Transformer to extract global temporal features, resulting in these methods often only being able to extract incomplete global temporal features. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and propose a method for decoding motor imagery EEG signals based on full-scale filtering and dual-granularity interaction. This method can capture complete global time-domain features and all important local time-domain features in a parameter-friendly manner based on a designed time-frequency filter. In addition, based on the dual-granularity feature interaction module, a comprehensive and systematic correlation analysis is performed on the information of the two granularities. Then, based on the correlation, the information of the two granularities is guided to complement each other with meaningful information.

[0006] To achieve the above objectives, the technical solution provided by this invention is: a method for decoding motor imagery EEG signals based on full-scale filtering and dual-granularity interaction, comprising the following steps:

[0007] S1: Acquire motor imagery EEG signals and preprocess them to obtain filtered motor imagery EEG signals with specific frequency components;

[0008] S2: Decouple the frequency of the filtered motion imagery EEG signal into a coarse-grained motion imagery EEG signal in the frequency domain and a fine-grained motion imagery signal in the frequency domain;

[0009] S3: Time-frequency filters are designed for coarse-grained and fine-grained motor imagery EEG signals in the frequency domain, respectively. The time-frequency filters first perform frequency domain analysis on the coarse-grained and fine-grained motor imagery EEG signals based on complex-valued neural networks, and then introduce residual structures to supplement temporal fluctuation information, outputting coarse-grained and fine-grained information respectively to achieve full-scale filtering. Among them, full-scale filtering for coarse-grained information can effectively capture complete global temporal features, and full-scale filtering for fine-grained information can effectively capture all important local temporal features.

[0010] S4: Apply temporal convolution, spatial convolution, and average pooling sequentially to coarse-grained and fine-grained information respectively, mapping the coarse-grained and fine-grained information to a space with richer latent features, and outputting spatiotemporal coarse-grained information and spatiotemporal fine-grained information.

[0011] S5: A dual-granularity feature interaction module designed for applications of spatiotemporal coarse-grained information and spatiotemporal fine-grained information. The dual-granularity feature interaction module calculates the temporal view dual-granularity complementary correlation and feature channel view dual-granularity complementary correlation between spatiotemporal coarse-grained information and spatiotemporal fine-grained information based on dual-axis attention. Then, based on the temporal view dual-granularity complementary correlation and feature channel view dual-granularity complementary correlation, it guides the spatiotemporal coarse-grained information and spatiotemporal fine-grained information to mutually mine meaningful supplementary information, and outputs interactive spatiotemporal coarse-grained information and interactive spatiotemporal fine-grained information.

[0012] S6: Add the coarse-grained information and fine-grained information of the interaction spatiotemporal space, and apply the fully connected layer to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category;

[0013] S7: Use the maximum index function to output the final decoding result of the motor imagery EEG signal based on the probability that the motor imagery EEG signal belongs to a certain motor imagery category.

[0014] Furthermore, the specific steps of step S2 are as follows:

[0015] S21: For the input filtered motion imagery EEG signal x raw ∈R C×T Where R represents the real number domain, C represents the number of electrodes collecting motor imagery EEG signals, and T represents the number of sampling points included in the motor imagery EEG signal; the filtered motor imagery EEG signal is transformed into a frequency domain filtered motor imagery EEG signal based on Fast Fourier Transform. in Represents the field of complex numbers. Represents divisibility downwards; frequency domain filtered motor imagery EEG signals represent the time domain patterns contained within the filtered motor imagery EEG signals in a compact and efficient manner. Specifically, each frequency domain component of the frequency domain filtered motor imagery EEG signal corresponds to a specific time domain pattern of the filtered motor imagery EEG signal. The low-frequency components of the frequency domain filtered motor imagery EEG signal represent the global time domain pattern of the filtered motor imagery EEG signal, while the high-frequency components of the frequency domain filtered motor imagery EEG signal represent the local time domain pattern of the filtered motor imagery EEG signal.

[0016] S22: Frequency Masking Frequency domain filtered motion imagery EEG signals are decoupled into coarse-grained frequency domain motion imagery EEG signals. and frequency domain fine-grained motion imagination signal Among them, the frequency domain coarse-grained motor imagery EEG signal retains the complete global time-domain features of the filtered motor imagery EEG signal, while the frequency domain fine-grained motor imagery signal has more refined local time-domain features of the filtered motor imagery EEG signal. The specific calculation process is as follows:

[0017] X coarse =X raw

[0018] X fine =X raw ⊙freq_mask

[0019]

[0020] In the formula, ⊙ represents element-wise multiplication, freq_mask[k] represents the k-th element of the frequency mask, and threshold represents the frequency threshold, which is used as a learnable parameter in the training process.

[0021] Furthermore, the specific steps of step S3 are as follows:

[0022] S31: The designed time-frequency filter, based on a complex-valued neural network, performs frequency domain analysis on both coarse-grained and fine-grained motor imagery EEG signals, outputting coarse-grained frequency domain information. and frequency domain fine-grained information The specific calculation process is as follows:

[0023] X f_coarse =F CVNN_coarse (X coarse ) = X coarse *W CVNN_coarse

[0024] X f_fine =F CVNN_fine (X fine ) = X fine *WCVNN_fine

[0025] In the formula, F CVNN_coarse This represents a complex-valued neural network used to process coarse-grained motor imagery EEG signals in the frequency domain; * represents complex multiplication. F represents the weights of a complex-valued neural network used to process coarse-grained motor imagery EEG signals in the frequency domain. CVNN_fine This represents a complex-valued neural network used to process fine-grained motor imagery EEG signals in the frequency domain. Represents the weights of a complex-valued neural network used to process fine-grained motor imagery EEG signals in the frequency domain;

[0026] S32: The designed time-frequency filter incorporates inverse fast Fourier transform and residual structure to process the input coarse-grained and fine-grained frequency domain information, and outputs coarse-grained information x. f_coarse ∈R C×T and fine-grained information x f_fine ∈R C×T The residual structure is introduced to supplement time-domain fluctuation information; the specific calculation process is as follows:

[0027] x f_coarse =IFFT(X f_coarse )+x raw

[0028] x f_fine =IFFT(X f_fine )+x raw

[0029] In the formula, IFFT represents the inverse fast Fourier transform.

[0030] Furthermore, in step S4, for the input coarse-grained information x f_coarse ∈R C×T and fine-grained information x f_fine ∈R C×T First, using F1 receptive fields of (1,ker) coarse That is, the length is ker. coarse Temporal convolution of a filter with a width of 1 and F1 receptive fields of (1, ker) fine That is, the length is ker. fine Temporal convolutions with a filter width of 1 map coarse-grained and fine-grained information to spaces with richer latent features, respectively. Then, spatial convolutions with F2 filters having a receptive field of (C,1), i.e., length 1, width C, and number of groups F1, compress the information in the spatial dimension and map it to a more expressive space. Finally, convolutions with a receptive field of (1,ker) are applied to the space. pool That is, the length is ker. poolAn average pooling layer with a width of 1 removes redundant features in the temporal dimension and outputs spatiotemporal coarse-grained information. and spatiotemporal fine-grained information in This represents the time domain length after removing redundant features.

[0031] Furthermore, the specific steps of step S5 are as follows:

[0032] S51: For the input spatiotemporal coarse-grained information and spatiotemporal fine-grained information, respectively, use receptive fields with F2 receptive fields (1, ker)... dw That is, the length is ker. dw Depthwise convolutions with a 1-width filter map to a more feature-rich latent space in a parameter-friendly manner and output a temporal view that enhances spatiotemporal coarse-grained information. Enhanced spatiotemporal fine-grained information with temporal domain views

[0033] S52: For the input temporal view enhanced with spatiotemporal coarse-grained information and temporal view enhanced with spatiotemporal fine-grained information, calculate the complementary correlation t_map∈R of the temporal view with dual granularity based on correlation analysis. t×t Based on the complementary correlation analysis of the temporal view at two granularities, the complementary relationship between the temporal view's enhanced spatiotemporal coarse-grained information and the temporal view's enhanced spatiotemporal fine-grained information is analyzed. This guides the two to mutually mine meaningful supplementary information and outputs the temporal view's two granularities supplementary information. The specific calculation process is as follows:

[0034]

[0035] In the formula, softmax represents the softmax function, and T represents the transpose function. Represents matrix multiplication;

[0036] S53: For the input spatiotemporal coarse-grained information and spatiotemporal fine-grained information, first use the transpose function respectively, and then apply the function with t receptive fields (1, ker) respectively. dw That is, the length is ker. dw Depthwise convolutions with a 1-width filter map to a more feature-rich latent space in a parameter-friendly manner and output a feature channel view that enhances spatiotemporal coarse-grained information. Enhanced spatiotemporal fine-grained information with feature channel views

[0037] S54: For the input feature channel view enhancement spatiotemporal coarse-grained information and feature channel view enhancement spatiotemporal fine-grained information, calculate the complementary correlation of the feature channel views at two granularities based on correlation analysis. Based on the complementary correlation analysis of the feature channel view at two granularities, the complementary relationship between the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information enhanced by the feature channel view is analyzed, thereby guiding the two to mutually mine meaningful supplementary information and outputting the feature channel view at two granularities. The specific calculation process is as follows:

[0038]

[0039] S55: After obtaining the temporal view dual-granularity supplementary information and the feature channel view dual-granularity supplementary information, the fused dual-granularity supplementary information is obtained by adding the two together. Then, the fused dual-granularity supplementary information is added to the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information respectively, thereby guiding the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information to mutually mine meaningful supplementary information, and outputting interactive spatiotemporal coarse-grained information. and interactive spatiotemporal fine-grained information The specific calculation process is as follows:

[0040] x fusion_map =T(x t_map )+x f_map

[0041] x inter_coarse =x pst_coarse +x fusion_map

[0042] x inter_fine =x pst_fine +x fusion_map .

[0043] Furthermore, in step S6, for the input interactive spatiotemporal coarse-grained information and interactive spatiotemporal fine-grained information The summation operation is performed, and a fully connected layer is applied to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category. N c The number of types of motor imagery is calculated as follows:

[0044] x result =F linear (x inter_coarse +x inter_fine )

[0045] In the formula, F linear This represents a fully connected layer.

[0046] Furthermore, in step S7, the probability that the motor imagery EEG signal belongs to a certain motor imagery category is determined. The final result of decoding the motor imagery EEG signal is output using the maximum index function:

[0047] result = maxindex(x result )

[0048] In the formula, maxindex represents the maximum value index function.

[0049] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0050] 1. This invention designs a time-frequency filter that can capture global time-domain features and all important local time-domain features based on a complex-valued neural network, thereby achieving full-scale filtering.

[0051] 2. This invention designs a dual-granularity feature interaction module, which calculates the complementary correlation between spatiotemporal coarse-grained information and spatiotemporal fine-grained information based on dual-axis attention, and the complementary correlation between spatiotemporal channel view and feature channel view, thereby guiding spatiotemporal coarse-grained information and spatiotemporal fine-grained information to mutually mine meaningful supplementary information.

[0052] 3. The motor imagery EEG signal decoding method designed in this invention has strong generalization ability. Compared with the mainstream motor imagery EEG signal decoding methods based on multi-scale convolutional neural networks and Transformer-based motor imagery EEG signal decoding methods, it requires only fewer training parameters. In the future, it can be deployed to brain-computer interfaces based on motor imagery EEG signals to improve performance. Attached Figure Description

[0053] Figure 1 This is a framework diagram of the method of the present invention; in the diagram, x raw X is used to filter motion-imagined EEG signals. coarse For coarse-grained motor imagery EEG signals in the frequency domain, X fine For fine-grained motion visualization signals in the frequency domain, x f_coarse For coarse-grained information, x f_fine For fine-grained information, x pst_coarse For spatiotemporal coarse-grained information, x pst_fine For spatiotemporal fine-grained information, x inter_coarse For coarse-grained information in spatiotemporal interaction, x inter_fine For interactive spatiotemporal fine-grained information, For addition, x result is the probability that the motor imagery EEG signal belongs to a certain motor imagery category, and result is the final decoding result of the motor imagery EEG signal.

[0054] Figure 2 This is a schematic diagram of frequency decoupling; in the diagram, x raw X is used to filter motion-imagined EEG signals. coarse For coarse-grained motor imagery EEG signals in the frequency domain, Xfine It is a frequency domain fine-grained motion imagination signal.

[0055] Figure 3 This is a schematic diagram of a time-frequency filter; in the diagram, X coarse For coarse-grained motor imagery EEG signals in the frequency domain, X fine X is a fine-grained motion visualization signal in the frequency domain. f_coarse For coarse-grained information in the frequency domain, X f_fine For fine-grained information in the frequency domain, x raw To filter the motion-imagining EEG signals, x f_coarse For coarse-grained information, x f_fine For fine-grained information, For addition.

[0056] Figure 4 This is a schematic diagram of temporal convolution, spatial convolution, and average pooling; in the diagram, x f_coarse For coarse-grained information, x f_fine For fine-grained information, F1 represents the number of filters in the temporal convolution, (1,ker) coarse ) represents the receptive field of temporal convolution for processing coarse-grained information, (1,ker) fine F1 represents the receptive field of the temporal convolution for processing fine-grained information, F2 represents the number of filters in the spatial convolution, C represents the number of electrodes for acquiring motor imagery EEG signals, and (1,ker) represents the receptive field of the temporal convolution. pool ) represents the receptive field of average pooling, x pst_coarse For spatiotemporal coarse-grained information, x pst_fine This provides spatiotemporal fine-grained information, where groups represents the number of groups.

[0057] Figure 5 This is a schematic diagram of a dual-granularity feature interaction module; in the diagram, x pst_coarse For spatiotemporal coarse-grained information, x pst_fine For spatiotemporal fine-grained information, For addition, F2 is the number of filters in the spatial convolution, (1,ker) dw ) represents the receptive field of the depthwise convolution, T is the transpose function, t is the temporal length after removing redundant features, and x is the receptive field of the depthwise convolution. tdw_coarse To enhance spatiotemporal coarse-grained information in the temporal view, x tdw_fine To enhance the spatiotemporal fine-grained information of the temporal view, x fdw_coarse To enhance spatiotemporal coarse-grained information in the feature channel view, x fdw_fine To enhance the spatiotemporal fine-grained information of the feature channel view, For matrix multiplication, softmax is the softmax function, t_map is the time-domain view of two-granularity complementary correlation, f_map is the feature channel view of two-granularity complementary correlation, and x t_map To supplement information for the time-domain view with dual granularity, x f_mapTo supplement information to the feature channel view at a dual granularity, x fusion_map To integrate dual-granularity supplementary information, x inter_coarse For coarse-grained information in spatiotemporal interaction, x inter_fine This provides fine-grained information for interaction in space and time. Detailed Implementation

[0058] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0059] like Figure 1 As shown in the figure, this embodiment discloses a method for decoding motor imagery EEG signals based on full-scale filtering and dual-granularity interaction, the specific details of which are as follows:

[0060] 1) such as Figure 2 As shown, the specific steps for frequency decoupling are as follows:

[0061] 1.1) For the input filtered motor imagery EEG signal x raw ∈R C×T Where R represents the real number domain, C represents the number of electrodes collecting motor imagery EEG signals, and T represents the number of sampling points included in the motor imagery EEG signal. The filtered motor imagery EEG signal is transformed into a frequency domain filtered motor imagery EEG signal based on Fast Fourier Transform. in Represents the field of complex numbers. This represents divisibility downwards. Frequency-domain filtered motor imagery EEG signals represent the temporal patterns inherent in the filtered motor imagery EEG signals in a compact and efficient manner. Specifically, each frequency component of the frequency-domain filtered motor imagery EEG signal corresponds to a specific temporal pattern of the filtered motor imagery EEG signal. The low-frequency components of the frequency-domain filtered motor imagery EEG signal represent the global temporal pattern of the filtered motor imagery EEG signal, while the high-frequency components represent the local temporal pattern of the filtered motor imagery EEG signal.

[0062] 1.2) Based on frequency masking Frequency domain filtered motion imagery EEG signals are decoupled into coarse-grained frequency domain motion imagery EEG signals. and frequency domain fine-grained motion imagination signal Among them, the frequency domain coarse-grained motor imagery EEG signal retains the complete global time-domain features of the filtered motor imagery EEG signal, while the frequency domain fine-grained motor imagery signal has more refined local time-domain features of the filtered motor imagery EEG signal. The specific calculation process is as follows:

[0063] X coarse =X raw

[0064] X fine =Xraw ⊙freq_mask

[0065]

[0066] In the formula, ⊙ represents element-wise multiplication, freq_mask[k] represents the k-th element of the frequency mask, and threshold represents the frequency threshold, which is used as a learnable parameter in the training process.

[0067] 2) such as Figure 3 As shown, the designed time-frequency filter is used to process the input frequency domain coarse-grained motor imagery EEG signal. and frequency domain fine-grained motion imagination signal Perform full-scale filtering, where Representing the complex field, C represents the number of electrodes used to collect motor imagery EEG signals, and T represents the number of sampling points included in the motor imagery EEG signals. This represents integer division downwards. This allows for the effective capture of complete global time-domain features and all important local time-domain features. The specific operation steps of the time-frequency filter are as follows:

[0068] 2.1) The designed time-frequency filter is based on a complex-valued neural network to perform frequency domain analysis on both coarse-grained and fine-grained motor imagery EEG signals in the frequency domain, and outputs coarse-grained information in the frequency domain. and frequency domain fine-grained information The specific calculation process is as follows:

[0069] X f_coarse =F CVNN_coarse (X coarse ) = X coarse *W CVNN_coarse

[0070] X f_fine =F CVNN_fine (X fine ) = X fine *W CVNN_fine

[0071] In the formula, F CVNN_coarse This represents a complex-valued neural network used to process coarse-grained motor imagery EEG signals in the frequency domain; * represents complex multiplication. F represents the weights of a complex-valued neural network used to process coarse-grained motor imagery EEG signals in the frequency domain. CVNN_fine This represents a complex-valued neural network used to process fine-grained motor imagery EEG signals in the frequency domain. Represents the weights of a complex-valued neural network used to process fine-grained motor imagery EEG signals in the frequency domain;

[0072] 2.2) The designed time-frequency filter incorporates inverse fast Fourier transform and residual structure to process the input frequency domain coarse-grained information and frequency domain fine-grained information, and outputs coarse-grained information x. f_coarse ∈R C×T and fine-grained information x f_fine ∈R C×T R represents the real number field; the residual structure is introduced to supplement the time-domain fluctuation information, and the specific calculation process is as follows:

[0073] x f_coarse =IFFT(X f_coarse )+x raw

[0074] x f_fine =IFFT(X f_fine )+x raw

[0075] In the formula, IFFT represents Inverse Fast Fourier Transform, x raw ∈R C×T This represents filtered motor imagery EEG signals.

[0076] 3) such as Figure 4 As shown, the specific steps of temporal convolution, spatial convolution, and average pooling are as follows:

[0077] For the input coarse-grained information x f_coarse ∈R C×T and fine-grained information x f_fine ∈R C×T Where R represents the real number field, C represents the number of electrodes for collecting motor imagery EEG signals, and T represents the number of sampling points included in the motor imagery EEG signals. First, using electrodes with F1 receptive fields (1, ker...)... coarse That is, the length is ker. coarse Temporal convolution of a filter with a width of 1 and F1 receptive fields of (1, ker) fine That is, the length is ker. fine Temporal convolutions with a filter width of 1 map coarse-grained and fine-grained information to spaces with richer latent features, respectively. Then, spatial convolutions with F2 filters having a receptive field of (C,1), i.e., length 1, width C, and number of groups F1, compress the information in the spatial dimension and map it to a more expressive space. Finally, convolutions with a receptive field of (1,ker) are applied to the space. pool That is, the length is ker. pool An average pooling layer with a width of 1 removes redundant features in the temporal dimension and outputs spatiotemporal coarse-grained information. and spatiotemporal fine-grained information in This represents the time domain length after removing redundant features.

[0078] 4) such as Figure 5 As shown, for the input spatiotemporal coarse-grained information and spatiotemporal fine-grained information Where R represents the real number domain, F² represents the number of filters in the spatial convolution, and t represents the temporal length after removing redundant features. Considering that spatiotemporal coarse-grained and spatiotemporal fine-grained information have temporal and feature channel dimensions, the designed dual-granularity feature interaction module calculates the complementary correlations between the temporal and feature channel views based on biaxial attention. Then, based on the complementary correlations between the temporal and feature channel views, it guides the spatiotemporal coarse-grained and spatiotemporal fine-grained information to mutually mine meaningful complementary information, and outputs interactive spatiotemporal coarse-grained and interactive spatiotemporal fine-grained information. The specific operation steps of the dual-granularity feature interaction module are as follows:

[0079] 4.1) For the input spatiotemporal coarse-grained information and spatiotemporal fine-grained information, respectively, use receptive fields with F2 receptive fields (1, ker) and (2, 1) respectively. dw That is, the length is ker. dw Depthwise convolutions with a 1-width filter map to a more feature-rich latent space in a parameter-friendly manner and output a temporal view that enhances spatiotemporal coarse-grained information. Enhanced spatiotemporal fine-grained information with temporal domain views

[0080] 4.2) For the input temporal view enhanced spatiotemporal coarse-grained information and temporal view enhanced spatiotemporal fine-grained information, calculate the temporal view dual-granularity complementary correlation t_map∈R based on correlation analysis. t×t Based on the complementary correlation analysis of the temporal view at two granularities, the complementary relationship between the temporal view's enhanced spatiotemporal coarse-grained information and the temporal view's enhanced spatiotemporal fine-grained information is analyzed, thereby guiding the two to mutually mine meaningful supplementary information, and outputting the temporal view's two granularities supplementary information. The specific calculation process is as follows:

[0081]

[0082] In the formula, softmax represents the softmax function, and T represents the transpose function. Represents matrix multiplication;

[0083] 4.3) For the input spatiotemporal coarse-grained information and spatiotemporal fine-grained information, firstly, the transpose function is used respectively, and then the functions with t receptive fields (1, ker) are applied respectively. dw That is, the length is ker. dw Depthwise convolutions with a 1-width filter map to a more feature-rich latent space in a parameter-friendly manner and output a feature channel view that enhances spatiotemporal coarse-grained information. Enhanced spatiotemporal fine-grained information with feature channel views

[0084] 4.4) For the input feature channel view enhancement spatiotemporal coarse-grained information and feature channel view enhancement spatiotemporal fine-grained information, calculate the complementary correlation of the feature channel views at two granularities based on correlation analysis. Based on the complementary correlation analysis of the feature channel view at two granularities, the complementary relationship between the enhanced spatiotemporal coarse-grained information and the enhanced spatiotemporal fine-grained information of the feature channel view is analyzed, thereby guiding the two to mutually mine meaningful supplementary information and outputting the feature channel view at two granularities. The specific calculation process is as follows:

[0085]

[0086] 4.5) After obtaining the temporal view dual-granularity supplementary information and the feature channel view dual-granularity supplementary information, the fused dual-granularity supplementary information is obtained by adding the two together. Then, the fused dual-granularity supplementary information is added to the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information respectively, thereby guiding the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information to mutually mine meaningful supplementary information, and outputting interactive spatiotemporal coarse-grained information. and interactive spatiotemporal fine-grained information The specific calculation process is as follows:

[0087] x fusion_map =T(x t_map )+x f_map

[0088] x inter_coarse =x pst_coarse +x fusion_map

[0089] x inter_fine =x pst_fine +x fusion_map

[0090] 5) such as Figure 1 As shown, for the input interactive spatiotemporal coarse-grained information and interactive spatiotemporal fine-grained information Where R represents the real number domain, F2 represents the number of filters in the spatial convolution, and t represents the temporal length after removing redundant features. An addition operation is performed, and a fully connected layer is applied to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category. N c The number of types of motor imagery is calculated as follows:

[0091] x result =F linear(x inter_coarse +x inter_fine )

[0092] In the formula, F linear This represents a fully connected layer.

[0093] 6) such as Figure 1 As shown, the probability of a motor imagery EEG signal belonging to a certain type of motor imagery. N c The number of types of motor imagery is represented by the maximum index function, which outputs the final decoded result of the motor imagery EEG signal:

[0094] result = maxindex(x result )

[0095] In the formula, maxindex represents the maximum value index function.

[0096] The BCIC-IV-2a dataset contains EEG signals from nine participants regarding motor imagery. The motor imagery paradigm in this dataset includes four body parts: left hand, right hand, tongue, and foot. Researchers used 22 electrodes to acquire EEG signals at a sampling rate of 250 Hz. After acquisition, the EEG signals were preprocessed using a bandpass filter with a bandpass frequency of 0.5–100 Hz. Each participant received motor imagery EEG signals from two sessions, with each session containing 288 trials, each trial lasting 4 seconds.

[0097] The BCIC-IV-2b dataset contains EEG signals from nine participants regarding motor imagery. The motor imagery paradigm in this dataset includes two body parts: left and right. Researchers used three electrodes to acquire EEG signals at a sampling rate of 250 Hz. After acquisition, the EEG signals were preprocessed using a bandpass filter with a bandpass frequency of 0.5–100 Hz. Each participant received motor imagery EEG signals from five sessions: the first two sessions each contained 120 trials, and the subsequent three sessions each contained 160 trials. Each trial lasted 4 seconds.

[0098] To verify the effectiveness of the method of this invention, the decoding performance was compared with that of DMSACNN, MSVTNet, EEG-SimpleConv, EISATC-Fusion, FACT-Net, ADFCNN, LightConvNet, IFNet, FBMSNet, and EEG Conformer on two commonly used datasets, BCIC-IV-2a and BCIC-IV-2b, under both session-independent and session-dependent conditions.

[0099] Here, accuracy and kappa are used to represent decoding performance.

[0100] The experimental results are shown in Table 1.

[0101] Table 1 compares decoding performance on the BCIC-IV-2a and BCIC-IV-2b datasets.

[0102]

[0103]

[0104] Experimental results show that the method of this invention outperforms other methods in decoding motor imagery EEG signals. For the BCIC-IV-2a dataset, the accuracy and Kappa of the method of this invention under conversation-independent conditions are 5.17% and 0.07 higher than the best comparative method, respectively. For the BCIC-IV-2b dataset, the accuracy and Kappa of the method of this invention under conversation-independent conditions are 2.66% and 0.05 higher than the best comparative method, respectively, thus verifying the effectiveness of the method of this invention.

[0105] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for decoding motor imagery EEG signals based on full-scale filtering and dual-granularity interaction, characterized in that, Includes the following steps: S1: Acquire motor imagery EEG signals and preprocess them to obtain filtered motor imagery EEG signals with specific frequency components; S2: Decouple the frequency of the filtered motion imagery EEG signal into a coarse-grained motion imagery EEG signal in the frequency domain and a fine-grained motion imagery signal in the frequency domain; S3: Time-frequency filters are designed for coarse-grained and fine-grained motor imagery EEG signals in the frequency domain, respectively. The time-frequency filters first perform frequency domain analysis on the coarse-grained and fine-grained motor imagery EEG signals based on complex-valued neural networks, and then introduce residual structures to supplement temporal fluctuation information, outputting coarse-grained and fine-grained information respectively to achieve full-scale filtering. Among them, full-scale filtering for coarse-grained information can effectively capture complete global temporal features, and full-scale filtering for fine-grained information can effectively capture all important local temporal features. S4: Apply temporal convolution, spatial convolution, and average pooling sequentially to coarse-grained and fine-grained information respectively, mapping the coarse-grained and fine-grained information to a space with richer latent features, and outputting spatiotemporal coarse-grained information and spatiotemporal fine-grained information. S5: A dual-granularity feature interaction module designed for applications of spatiotemporal coarse-grained information and spatiotemporal fine-grained information. The dual-granularity feature interaction module calculates the temporal view dual-granularity complementary correlation and feature channel view dual-granularity complementary correlation between spatiotemporal coarse-grained information and spatiotemporal fine-grained information based on dual-axis attention. Then, based on the temporal view dual-granularity complementary correlation and feature channel view dual-granularity complementary correlation, it guides the spatiotemporal coarse-grained information and spatiotemporal fine-grained information to mutually mine meaningful supplementary information, and outputs interactive spatiotemporal coarse-grained information and interactive spatiotemporal fine-grained information. S6: Add the coarse-grained information and fine-grained information of the interaction spatiotemporal space, and apply the fully connected layer to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category; S7: Use the maximum index function to output the final decoding result of the motor imagery EEG signal based on the probability that the motor imagery EEG signal belongs to a certain motor imagery category.

2. The method for decoding motor imagery EEG signals based on full-scale filtering and dual-granularity interaction according to claim 1, characterized in that, The specific steps for step S2 are as follows: S21: For the input filtered motion imagery EEG signal x raw ∈R C×T Where R represents the real number domain, C represents the number of electrodes collecting motor imagery EEG signals, and T represents the number of sampling points included in the motor imagery EEG signal; the filtered motor imagery EEG signal is transformed into a frequency domain filtered motor imagery EEG signal based on Fast Fourier Transform. in Represents the field of complex numbers. Represents divisibility downwards; frequency domain filtered motor imagery EEG signals represent the time domain patterns contained within the filtered motor imagery EEG signals in a compact and efficient manner. Specifically, each frequency domain component of the frequency domain filtered motor imagery EEG signal corresponds to a specific time domain pattern of the filtered motor imagery EEG signal. The low-frequency components of the frequency domain filtered motor imagery EEG signal represent the global time domain pattern of the filtered motor imagery EEG signal, while the high-frequency components of the frequency domain filtered motor imagery EEG signal represent the local time domain pattern of the filtered motor imagery EEG signal. S22: Frequency Masking Frequency domain filtered motion imagery EEG signals are decoupled into coarse-grained frequency domain motion imagery EEG signals. and frequency domain fine-grained motion imagination signal Among them, the frequency domain coarse-grained motor imagery EEG signal retains the complete global time-domain features of the filtered motor imagery EEG signal, while the frequency domain fine-grained motor imagery signal has more refined local time-domain features of the filtered motor imagery EEG signal. The specific calculation process is as follows: X coarse =X raw X fine =X raw ⊙freq_mask In the formula, ⊙ represents element-wise multiplication, freq_mask[k] represents the k-th element of the frequency mask, and threshold represents the frequency threshold, which is used as a learnable parameter in the training process.

3. The method for decoding motor imagery EEG signals based on full-scale filtering and dual-granularity interaction according to claim 2, characterized in that, The specific steps for step S3 are as follows: S31: The designed time-frequency filter, based on a complex-valued neural network, performs frequency domain analysis on both coarse-grained and fine-grained motor imagery EEG signals, outputting coarse-grained frequency domain information. and frequency domain fine-grained information The specific calculation process is as follows: X f_coarse =F CVNN_coarse (X coarse )=X coarse *W CVNN_coarse X f_fine =F CVNN_fine (X fine )=X fine *W CVNN_fine In the formula, F CVNN_coarse This represents a complex-valued neural network used to process coarse-grained motor imagery EEG signals in the frequency domain; * represents complex multiplication. F represents the weights of a complex-valued neural network used to process coarse-grained motor imagery EEG signals in the frequency domain. CVNN_fine This represents a complex-valued neural network used for processing fine-grained motor imagery EEG signals in the frequency domain. Represents the weights of a complex-valued neural network used to process fine-grained motor imagery EEG signals in the frequency domain; S32: The designed time-frequency filter incorporates inverse fast Fourier transform and residual structure to process the input coarse-grained and fine-grained frequency domain information, and outputs coarse-grained information x. f_coarse ∈R C×T and fine-grained information x f_fine ∈R C×T The residual structure is introduced to supplement time-domain fluctuation information; the specific calculation process is as follows: x f_coarse =IFFT(X f_coarse )+x raw x f_fine =IFFT(X f_fine )+x raw In the formula, IFFT represents the inverse fast Fourier transform.

4. The method for decoding motor imagery EEG signals based on full-scale filtering and dual-granularity interaction according to claim 3, characterized in that, In step S4, for the input coarse-grained information x f_coarse ∈R C×T and fine-grained information x f_fine ∈R C×T First, using F1 receptive fields of (1, ker) coarse That is, the length is ker. coarse Temporal convolution of a filter with a width of 1 and F1 receptive fields of (1, ker) fine That is, the length is ker. fine Temporal convolutions with a filter width of 1 map coarse-grained and fine-grained information to spaces with richer latent features, respectively. Then, spatial convolutions with F2 filters having a receptive field of (C,1), i.e., length 1, width C, and number of groups F1, compress the information in the spatial dimension and map it to a more expressive space. Finally, convolutions with a receptive field of (1,ker) are applied to the space. pool That is, the length is ker. pool An average pooling layer with a width of 1 removes redundant features in the temporal dimension and outputs spatiotemporal coarse-grained information. and spatiotemporal fine-grained information in This represents the time domain length after removing redundant features.

5. The method for decoding motor imagery EEG signals based on full-scale filtering and dual-granularity interaction according to claim 4, characterized in that, The specific steps for step S5 are as follows: S51: For the input spatiotemporal coarse-grained information and spatiotemporal fine-grained information, respectively, use receptive fields with F2 receptive fields (1, ker)... dw That is, the length is ker. dw Depthwise convolutions with a 1-width filter map to a more feature-rich latent space in a parameter-friendly manner and output a temporal view that enhances spatiotemporal coarse-grained information. Enhanced spatiotemporal fine-grained information with temporal domain views S52: For the input temporal view enhancement spatiotemporal coarse-grained information and temporal view enhancement spatiotemporal fine-grained information, calculate the temporal view dual-granularity complementary correlation t_map∈R based on correlation analysis. t×t Based on the complementary correlation analysis of the temporal view at two granularities, the complementary relationship between the temporal view's enhanced spatiotemporal coarse-grained information and the temporal view's enhanced spatiotemporal fine-grained information is analyzed. This guides the two to mutually mine meaningful supplementary information and outputs the temporal view's two granularities supplementary information. The specific calculation process is as follows: In the formula, softmax represents the softmax function, and T represents the transpose function. Represents matrix multiplication; S53: For the input spatiotemporal coarse-grained information and spatiotemporal fine-grained information, first use the transpose function respectively, and then apply the function with t receptive fields (1, ker) respectively. dw That is, the length is ker. dw Depthwise convolutions with a 1-width filter map to a more feature-rich latent space in a parameter-friendly manner and output a feature channel view that enhances spatiotemporal coarse-grained information. Enhanced spatiotemporal fine-grained information with feature channel views S54: For the input feature channel view enhancement spatiotemporal coarse-grained information and feature channel view enhancement spatiotemporal fine-grained information, calculate the complementary correlation of the feature channel views at two granularities based on correlation analysis. Based on the complementary correlation analysis of the feature channel view at two granularities, the complementary relationship between the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information enhanced by the feature channel view is analyzed, thereby guiding the two to mutually mine meaningful supplementary information and outputting the feature channel view at two granularities. The specific calculation process is as follows: S55: After obtaining the temporal view dual-granularity supplementary information and the feature channel view dual-granularity supplementary information, the fused dual-granularity supplementary information is obtained by adding the two together. Then, the fused dual-granularity supplementary information is added to the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information respectively, thereby guiding the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information to mutually mine meaningful supplementary information, and outputting interactive spatiotemporal coarse-grained information. and interactive spatiotemporal fine-grained information The specific calculation process is as follows: x fusion_map =T(x t_map )+x f_map x inter_coarse =x pst_coarse +x fusion_map x inter_fine =x pst_fine +x fusion_map 。 6. The method for decoding motor imagery EEG signals based on full-scale filtering and dual-granularity interaction according to claim 5, characterized in that, In step S6, for the input interactive spatiotemporal coarse-grained information and interactive spatiotemporal fine-grained information The addition operation is performed and a fully connected layer is applied to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category. N c The number of types of motor imagery is calculated as follows: x result =F linear (x inter_coarse +x inter_fine ) In the formula, F linear This represents a fully connected layer.

7. The method for decoding motor imagery EEG signals based on full-scale filtering and dual-granularity interaction according to claim 6, characterized in that, In step S7, the probability that the EEG signal of motor imagery belongs to a certain type of motor imagery is determined. The final result of decoding the motor imagery EEG signal is output using the maximum index function: result=maxindex(x result ) In the formula, maxindex represents the maximum value index function.

Images