Motor imagery electroencephalogram signal decoding method based on full-scale filtering dual-granularity interaction

Abstract

The invention discloses a motor imagery electroencephalogram signal decoding method based on full-scale filtering dual-granularity interaction. The motor imagery electroencephalogram signal decoding method comprises the steps that motor imagery electroencephalogram signals are obtained and preprocessed; performing frequency decoupling on the filtered motor imagery signal; capturing a complete global time domain feature and all important local time domain features based on a designed time frequency filter; mapping to a space with richer potential features by applying time convolution and space convolution; the space-time coarse-grained information and the space-time fine-grained information are guided to mutually mine meaningful supplementary information based on a dual-granularity feature interaction module; and outputting the probability that the motor imagery electroencephalogram signal belongs to a certain motor imagery category by applying a full connection layer, and finally realizing the decoding of the motor imagery electroencephalogram signal. Complete global time domain features and all important local time domain features are captured creatively in a parameter-friendly mode, and a dual-granularity feature interaction module is creatively designed to comprehensively and systematically mine meaningful supplementary information.

Description

Technical Field

[0001] The present invention relates to the technical field of motor imagery EEG signal decoding, and in particular to a motor imagery EEG signal decoding method based on full-scale filtering and dual-granularity interaction. Background Art

[0002] Brain-computer interface (BCI) systems enable muscle-free human-computer interaction by analyzing human brain activity and converting it into commands to control external devices. EEG signals are favored by researchers due to their non-invasiveness, low cost, and high temporal resolution. Among the classic paradigms of BCIs based on EEG signals, motor imagery enables users to complete tasks autonomously without external stimulation. Therefore, BCIs based on motor imagery EEG signals are gaining increasing attention. Motor imagery refers to the mental rehearsal of specific limb movements without actually performing them. Related studies have shown that motor imagery induces event-related synchronization and desynchronization in the sensorimotor cortex. Specifically, the μ and β rhythms in the sensorimotor cortex undergo energy modulation. BCI systems based on motor imagery EEG signals have been widely used in fields such as medical rehabilitation.

[0003] A BCI system based on motor imagery EEG signals typically consists of three components: motor imagery EEG signal acquisition, motor imagery EEG signal decoding, and external devices. Motor imagery EEG signal decoding is crucial for the development of BCI systems based on motor imagery EEG signals. The performance of the decoding method directly impacts the effectiveness of the BCI system. Typically, a BCI based on motor imagery EEG signals consists of two phases: offline training and online testing. During the offline training phase, subjects perform a specific motor imagery task according to instructions, generating training EEG signals. During the online testing phase, subjects perform specific motor imagery and generate corresponding EEG signals, which are directly input into the trained BCI and control the external devices. Because the process of acquiring motor imagery EEG signals is cumbersome and the EEG signals generated by subjects performing the same motor imagery task on different days can differ significantly, the motor imagery EEG signal decoding method must have sufficient generalization capabilities.

[0004] Research on motor imagery decoding methods has made significant progress by introducing deep learning architectures. Deep learning architectures can automatically learn to extract high-level features from motor imagery EEG signals in an end-to-end manner, eliminating the need for complex preprocessing. Currently, commonly used motor imagery decoding methods based on deep learning architectures can be divided into two categories: motor imagery decoding methods based on multi-scale convolutional neural networks and motor imagery decoding methods based on Transformers. However, these two types of motor imagery decoding methods based on deep learning architectures still have many shortcomings. For motor imagery decoding methods based on multi-scale convolutional neural networks, these methods often ignore the extraction of global time-domain features of motor imagery EEG signals and simply apply descriptive statistics layers to simply extract global time-domain features of motor imagery EEG signals. In addition, such methods often use a filter group with a fixed receptive field to extract multi-scale local time domain features, which makes it difficult to cover all potential important local time domain features. In addition, in order to ensure that the decoding method has sufficient generalization ability, each receptive field has multiple filters, which increases the number of trainable parameters. For the Transformer-based motor imagery decoding method, the inherent calculation mechanism of the Transformer leads to a large number of trainable parameters and computational complexity of such methods. Moreover, such methods often first perform preliminary filtering on the motor imagery EEG signals based on a convolutional neural network and then input them into the Transformer to extract global time domain features, resulting in such methods often only being able to extract incomplete global time domain features. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and propose a motor imagery EEG signal decoding method based on full-scale filtering dual-granularity interaction. The method can capture complete global time domain features and all important local time domain features in a parameter-friendly manner based on the designed time-frequency filter. In addition, based on the dual-granularity feature interaction module, a comprehensive and systematic correlation analysis is performed on the information of the two granularities, and then the information of the two granularities is guided to complement each other with meaningful information based on the correlation.

[0006] To achieve the above-mentioned purpose, the present invention provides a technical solution: a method for decoding motor imagery EEG signals based on full-scale filtering and dual-granularity interaction, comprising the following steps:

[0007] S1: Acquire motor imagery EEG signals and preprocess them to obtain filtered motor imagery EEG signals with specific frequency components;

[0008] S2: Decouple the frequency of the filtered motor imagery EEG signal into coarse-grained motor imagery EEG signal in the frequency domain and fine-grained motor imagery signal in the frequency domain;

[0009] S3: Designed time-frequency filters are applied to the frequency-domain coarse-grained motor imagery EEG signals and the frequency-domain fine-grained motor imagery signals, respectively. The time-frequency filters first perform frequency-domain analysis on the frequency-domain coarse-grained motor imagery EEG signals and the frequency-domain fine-grained motor imagery signals based on a complex-valued neural network. Then, a residual structure is introduced to supplement the time-domain fluctuation information, and the coarse-grained information and the fine-grained information are output respectively to achieve full-scale filtering. Among them, the full-scale filtering for the coarse-grained information can effectively capture the complete global time-domain features, and the full-scale filtering for the fine-grained information can effectively capture all important local time-domain features.

[0010] S4: Temporal convolution, spatial convolution, and average pooling are applied to the coarse-grained information and fine-grained information respectively, mapping the coarse-grained information and fine-grained information to a space with richer potential features, and outputting spatiotemporal coarse-grained information and spatiotemporal fine-grained information;

[0011] S5: A dual-granularity feature interaction module designed for the application of spatiotemporal coarse-grained information and spatiotemporal fine-grained information. The dual-granularity feature interaction module calculates the dual-granularity complementary correlation in the time domain view and the dual-granularity complementary correlation in the feature channel view between the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information based on dual-axis attention. Then, based on the dual-granularity complementary correlation in the time domain view and the dual-granularity complementary correlation in the feature channel view, the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information are simultaneously guided to mutually mine meaningful complementary information, and the interactive spatiotemporal coarse-grained information and the interactive spatiotemporal fine-grained information are output;

[0012] S6: Add the interactive spatiotemporal coarse-grained information and the interactive spatiotemporal fine-grained information, and apply a fully connected layer to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category;

[0013] S7: Use the maximum index function to output the final motor imagery EEG signal decoding result according to the probability that the motor imagery EEG signal belongs to a certain motor imagery category.

[0014] Furthermore, the specific operation steps of step S2 are as follows:

[0015] S21: For the input filtered motor imagery EEG signal x raw ∈R C×T , where R represents the real number domain, C represents the number of electrodes collecting motor imagery EEG signals, and T represents the number of sampling points contained in the motor imagery EEG signals; based on the fast Fourier transform, the filtered motor imagery EEG signals are transformed into frequency domain filtered motor imagery EEG signals in represents the complex field, represents downward division; the frequency-domain filtered motor imagery EEG signal represents the time-domain pattern contained in the filtered motor imagery EEG signal in a compact and efficient manner. Specifically, each frequency-domain component of the frequency-domain filtered motor imagery EEG signal corresponds to a specific time-domain pattern of the filtered motor imagery EEG signal, the low-frequency component of the frequency-domain filtered motor imagery EEG signal represents the global time-domain pattern of the filtered motor imagery EEG signal, and the high-frequency component of the frequency-domain filtered motor imagery EEG signal represents the local time-domain pattern of the filtered motor imagery EEG signal;

[0016] S22: Based on frequency mask Decouple the frequency domain filtered motor imagery EEG signal into a coarse-grained motor imagery EEG signal in the frequency domain Sum frequency domain fine-grained motion imagery signal Among them, the coarse-grained motor imagery EEG signal in the frequency domain retains the complete global time domain characteristics of the filtered motor imagery EEG signal, and the fine-grained motor imagery signal in the frequency domain has finer local time domain characteristics of the filtered motor imagery EEG signal. The specific calculation process is as follows:

[0017] X coarse =X raw

[0018] X fine =X raw ⊙freq_mask

[0019]

[0020] Where ⊙ represents element-by-element multiplication, freq_mask[k] represents the kth element of the frequency mask, and threshold represents the frequency threshold, which is used as a learnable parameter in the training process.

[0021] Furthermore, the specific operation steps of step S3 are as follows:

[0022] S31: The designed time-frequency filter performs frequency domain analysis on the frequency domain coarse-grained motor imagery EEG signal and the frequency domain fine-grained motor imagery signal based on the complex-valued neural network, and outputs the frequency domain coarse-grained information and frequency domain fine-grained information The specific calculation process is as follows:

[0023] X f_coarse =F CVNN_coarse (X coarse )=X coarse *W CVNN_coarse

[0024] X f_fine =F CVNN_fine (X fine )=X fine *WCVNN_fine

[0025] Where, F CVNN_coarse represents a complex-valued neural network for processing frequency-domain coarse-grained motor imagery EEG signals, * represents complex number multiplication, represents the weight of the complex-valued neural network used to process the frequency-domain coarse-grained motor imagery EEG signal, F CVNN_fine represents a complex-valued neural network for processing fine-grained motor imagery EEG signals in the frequency domain, represents the weights of the complex-valued neural network used to process frequency-domain fine-grained motor imagery EEG signals;

[0026] S32: The designed time-frequency filter introduces inverse fast Fourier transform and residual structure to process the input frequency domain coarse-grained information and frequency domain fine-grained information, and outputs coarse-grained information x f_coarse ∈R C×T and fine-grained information x f_fine ∈R C×T ; Among them, the residual structure is introduced to supplement the time domain fluctuation information. The specific calculation process is as follows:

[0027] x f_coarse =IFFT(X f_coarse )+x raw

[0028] x f_fine =IFFT(X f_fine )+x raw

[0029] Where IFFT stands for inverse fast Fourier transform.

[0030] Further, in step S4, for the input coarse-grained information x f_coarse ∈R C×T and fine-grained information x f_fine ∈R C×T , first use the F1 receptive field (1,ker coarse ) is ker coarse The temporal convolution of the filter with width 1 has a receptive field of F1 (1, ker fine ) is ker fine The temporal convolution of the filter with a width of 1 maps the coarse-grained information and the fine-grained information to a space with richer potential features. Then, the spatial convolution of the filter with a receptive field of F2 (C, 1) (length 1, width C) and a group number of F1 is used to compress the information in the spatial dimension and map it to a space with stronger expression ability. Finally, the spatial convolution of the filter with a receptive field of (1, ker pool ) is ker poolThe average pooling layer with a width of 1 removes redundant features in the time domain dimension and outputs coarse-grained spatiotemporal information. and spatiotemporal fine-grained information in Represents the time domain length after removing redundant features.

[0031] Furthermore, the specific operation steps of step S5 are as follows:

[0032] S51: For the input spatiotemporal coarse-grained information and spatiotemporal fine-grained information, we use the F2 receptive field (1, ker dw ) is ker dw The depthwise convolution of filters with width 1 is mapped to a feature-rich latent space in a parameter-friendly manner and outputs a temporal view to enhance the spatiotemporal coarse-grained information. and time domain view to enhance spatiotemporal fine-grained information

[0033] S52: For the input temporal view enhanced spatiotemporal coarse-grained information and temporal view enhanced spatiotemporal fine-grained information, calculate the temporal view dual-granularity complementary correlation t_map∈R based on correlation analysis t×t , according to the dual-granularity complementary correlation analysis of the time domain view, the complementary relationship between the time domain view enhanced spatiotemporal coarse-grained information and the time domain view enhanced spatiotemporal fine-grained information is analyzed, thereby guiding the two to mutually mine meaningful complementary information and output the dual-granularity complementary information of the time domain view The specific calculation process is as follows:

[0034]

[0035] In the formula, softmax represents the softmax function, T represents the transpose function, Represents matrix multiplication;

[0036] S53: For the input spatiotemporal coarse-grained information and spatiotemporal fine-grained information, first use the transposition function respectively, and then apply the t receptive fields (1, ker dw ) is ker dw The depthwise convolution of filters with a width of 1 is mapped to a feature-rich latent space in a parameter-friendly way and outputs feature channel views to enhance spatiotemporal coarse-grained information. and feature channel views to enhance spatiotemporal fine-grained information

[0037] S54: For the input feature channel view enhanced spatiotemporal coarse-grained information and the feature channel view enhanced spatiotemporal fine-grained information, calculate the feature channel view dual-granularity complementary correlation based on correlation analysis According to the dual-granularity complementary correlation analysis of the feature channel view, the complementary relationship between the feature channel view enhanced spatiotemporal coarse-grained information and the feature channel view enhanced spatiotemporal fine-grained information is analyzed, thereby guiding the two to mutually mine meaningful complementary information and output the dual-granularity complementary information of the feature channel view. The specific calculation process is as follows:

[0038]

[0039] S55: After obtaining the dual-granularity supplementary information of the time domain view and the dual-granularity supplementary information of the feature channel view, the fused dual-granularity supplementary information is obtained by adding the two together. Then the fused dual-granularity supplementary information is added to the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information respectively, thereby guiding the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information to mutually mine meaningful supplementary information and output interactive spatiotemporal coarse-grained information. and interactive spatiotemporal fine-grained information The specific calculation process is as follows:

[0040] x fusion_map =T(x t_map )+x f_map

[0041] x inter_coarse =x pst_coarse +x fusion_map

[0042] x inter_fine =x pst_fine +x fusion_map .

[0043] Further, in step S6, for the input interactive spatiotemporal coarse-grained information and interactive spatiotemporal fine-grained information Perform the addition operation and apply the fully connected layer to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category N c Represents the number of types of motor imagery. The specific calculation process is as follows:

[0044] x result =F linear (x inter_coarse +x inter_fine )

[0045] Where, F linear Represents a fully connected layer.

[0046] Further, in step S7, the probability that the motor imagery EEG signal belongs to a certain motor imagery category is Use the maximum index function to output the final motor imagery EEG signal decoding result:

[0047] result=maxindex(x result )

[0048] Where maxindex represents the maximum value index function.

[0049] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0050] 1. The present invention designs a time-frequency filter that can fully capture the global time domain features and all important local time domain features based on a complex-valued neural network, thereby achieving full-scale filtering.

[0051] 2. The present invention designs a dual-granularity feature interaction module, which calculates the dual-granularity complementary correlation in the time domain view and the dual-granularity complementary correlation in the feature channel view between the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information based on dual-axis attention, thereby guiding the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information to mine meaningful complementary information from each other.

[0052] 3. The motor imagery EEG signal decoding method designed in the present invention has strong generalization ability. Compared with the mainstream motor imagery EEG signal decoding method based on multi-scale convolutional neural network and the motor imagery EEG signal decoding method based on Transformer, it only requires fewer training parameters. In the future, it can be deployed to brain-computer interfaces based on motor imagery EEG signals to improve performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a framework diagram of the method of the present invention; in the figure, x raw To filter the motor imagery EEG signal, X coarse For frequency domain coarse-grained motion imagery EEG signals, X fine is the frequency domain fine-grained motion imagery signal, x f_coarse is the coarse-grained information, x f_fine is fine-grained information, x pst_coarse is the spatiotemporal coarse-grained information, x pst_fine is the spatiotemporal fine-grained information, x inter_coarse is the interactive spatiotemporal coarse-grained information, x inter_fine For interactive spatiotemporal fine-grained information, is addition, x result is the probability that the motor imagery EEG signal belongs to a certain motor imagery category, and result is the final motor imagery EEG signal decoding result.

[0054] Figure 2 This is a schematic diagram of frequency decoupling; in the figure, x raw To filter the motor imagery EEG signal, X coarse For frequency domain coarse-grained motion imagery EEG signals, Xfine Fine-grained motion imagery signal in the frequency domain.

[0055] Figure 3 It is a schematic diagram of time-frequency filter; in the figure, X coarse For frequency domain coarse-grained motion imagery EEG signals, X fine is the frequency domain fine-grained motion imagery signal, X f_coarse is the coarse-grained information in the frequency domain, X f_fine is the fine-grained information in the frequency domain, x raw To filter the motion imagery EEG signal, x f_coarse is the coarse-grained information, x f_fine For fine-grained information, For addition.

[0056] Figure 4 It is a schematic diagram of time convolution, spatial convolution and average pooling; in the figure, x f_coarse is the coarse-grained information, x f_fine is the fine-grained information, F1 is the number of filters for temporal convolution, (1,ker coarse ) is the receptive field of the temporal convolution for processing coarse-grained information, (1,ker fine ) is the receptive field of the time convolution for processing fine-grained information, F2 is the number of filters for spatial convolution, C is the number of electrodes for collecting motor imagery EEG signals, (1,ker pool ) is the receptive field of the average pooling, x pst_coarse is the spatiotemporal coarse-grained information, x pst_fine is the spatiotemporal fine-grained information, and groups is the number of groups.

[0057] Figure 5 This is a schematic diagram of the dual-granularity feature interaction module; in the figure, x pst_coarse is the spatiotemporal coarse-grained information, x pst_fine is the spatiotemporal fine-grained information, is added, F2 is the number of filters for spatial convolution, (1,ker dw ) is the receptive field of the deep convolution, T is the transposition function, t is the time domain length after removing redundant features, x tdw_coarse Enhance spatiotemporal coarse-grained information for the time domain view, x tdw_fine Enhance spatiotemporal fine-grained information for the time domain view, x fdw_coarse Enhance spatiotemporal coarse-grained information for feature channel view, x fdw_fine Enhance spatiotemporal fine-grained information for feature channel views, is matrix multiplication, softmax is the softmax function, t_map is the dual-granularity complementary correlation of the time domain view, f_map is the dual-granularity complementary correlation of the feature channel view, x t_map Double granularity supplementary information for the time domain view, x f_mapis the dual-granularity supplementary information of the feature channel view, x fusion_map To integrate dual-granularity supplementary information, x inter_coarse is the interactive spatiotemporal coarse-grained information, x inter_fine It is interactive spatiotemporal fine-grained information. DETAILED DESCRIPTION

[0058] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0059] like Figure 1 As shown, this embodiment discloses a method for decoding motor imagery EEG signals based on full-scale filtering and dual-granularity interaction, the details of which are as follows:

[0060] 1) If Figure 2 As shown in the figure, the specific steps of frequency decoupling are as follows:

[0061] 1.1) For the input filtered motion imagery EEG signal x raw ∈R C×T , where R represents the real number domain, C represents the number of electrodes used to collect motor imagery EEG signals, and T represents the number of sampling points contained in the motor imagery EEG signals. Based on the fast Fourier transform, the filtered motor imagery EEG signals are transformed into frequency domain filtered motor imagery EEG signals. in represents the complex field, Represents downward divisibility. The frequency-domain filtered motor imagery EEG signal represents the time-domain pattern contained in the filtered motor imagery EEG signal in a compact and efficient manner. Specifically, each frequency-domain component of the frequency-domain filtered motor imagery EEG signal corresponds to a specific time-domain pattern of the filtered motor imagery EEG signal, the low-frequency component of the frequency-domain filtered motor imagery EEG signal represents the global time-domain pattern of the filtered motor imagery EEG signal, and the high-frequency component of the frequency-domain filtered motor imagery EEG signal represents the local time-domain pattern of the filtered motor imagery EEG signal;

[0062] 1.2) Based on frequency mask Decouple the frequency domain filtered motor imagery EEG signal into a coarse-grained motor imagery EEG signal in the frequency domain Sum frequency domain fine-grained motion imagery signal Among them, the coarse-grained motor imagery EEG signal in the frequency domain retains the complete global time domain characteristics of the filtered motor imagery EEG signal, and the fine-grained motor imagery signal in the frequency domain has finer local time domain characteristics of the filtered motor imagery EEG signal. The specific calculation process is as follows:

[0063] X coarse =X raw

[0064] X fine =Xraw ⊙freq_mask

[0065]

[0066] Where ⊙ represents element-by-element multiplication, freq_mask[k] represents the kth element of the frequency mask, and threshold represents the frequency threshold, which is used as a learnable parameter in the training process.

[0067] 2) If Figure 3 As shown in the figure, the designed time-frequency filter is used to filter the input frequency domain coarse-grained motion imagery EEG signal. Sum frequency domain fine-grained motion imagery signal Perform full-scale filtering, where represents the complex domain, C represents the number of electrodes used to collect motor imagery EEG signals, and T represents the number of sampling points contained in the motor imagery EEG signals. Represents downward division. This effectively captures the complete global time domain features and all important local time domain features. The specific operation steps of the time-frequency filter are as follows:

[0068] 2.1) The designed time-frequency filter performs frequency domain analysis on the frequency domain coarse-grained motor imagery EEG signal and the frequency domain fine-grained motor imagery signal based on the complex-valued neural network, and outputs the frequency domain coarse-grained information and frequency domain fine-grained information The specific calculation process is as follows:

[0069] X f_coarse =F CVNN_coarse (X coarse )=X coarse *W CVNN_coarse

[0070] X f_fine =F CVNN_fine (X fine )=X fine *W CVNN_fine

[0071] Where, F CVNN_coarse represents a complex-valued neural network for processing frequency-domain coarse-grained motor imagery EEG signals, * represents complex number multiplication, represents the weight of the complex-valued neural network used to process the frequency-domain coarse-grained motor imagery EEG signal, F CVNN_fine represents a complex-valued neural network for processing fine-grained motor imagery EEG signals in the frequency domain, represents the weights of the complex-valued neural network used to process frequency-domain fine-grained motor imagery EEG signals;

[0072] 2.2) The designed time-frequency filter introduces inverse fast Fourier transform and residual structure to process the input frequency domain coarse-grained information and frequency domain fine-grained information, and outputs coarse-grained information x f_coarse ∈R C×T and fine-grained information x f_fine ∈R C×T , R represents the real number domain; the residual structure is introduced to supplement the time domain fluctuation information. The specific calculation process is as follows:

[0073] x f_coarse =IFFT(X f_coarse )+x raw

[0074] x f_fine =IFFT(X f_fine )+x raw

[0075] Where IFFT stands for inverse fast Fourier transform, x raw ∈R C×T Represents filtered motor imagery EEG signals.

[0076] 3) If Figure 4 As shown in the figure, the specific steps of temporal convolution, spatial convolution and average pooling are as follows:

[0077] For the coarse-grained input information x f_coarse ∈R C×T and fine-grained information x f_fine ∈R C×T , where R represents the real number domain, C represents the number of electrodes collecting motor imagery EEG signals, and T represents the number of sampling points contained in the motor imagery EEG signals. First, we use a receptive field of (1,ker coarse ) is ker coarse The temporal convolution of the filter with width 1 has a receptive field of F1 (1, ker fine ) is ker fine The temporal convolution of the filter with a width of 1 maps the coarse-grained information and the fine-grained information to a space with richer potential features. Then, the spatial convolution of the filter with a receptive field of F2 (C, 1) (length 1, width C) and a group number of F1 is used to compress the information in the spatial dimension and map it to a space with stronger expression ability. Finally, the spatial convolution of the filter with a receptive field of (1, ker pool ) is ker pool The average pooling layer with a width of 1 removes redundant features in the time domain dimension and outputs coarse-grained spatiotemporal information. and spatiotemporal fine-grained information in Represents the time domain length after removing redundant features.

[0078] 4) If Figure 5 As shown, for the input spatiotemporal coarse-grained information and spatiotemporal fine-grained information Where R represents the real domain, F2 represents the number of filters of spatial convolution, and t represents the time domain length after removing redundant features. Considering that spatiotemporal coarse-grained information and spatiotemporal fine-grained information have time domain dimensions and feature channel dimensions, the designed dual-granularity feature interaction module calculates the dual-granularity complementary correlation of the time domain view and the dual-granularity complementary correlation of the feature channel view based on dual-axis attention in the time domain view and the feature channel view. Then, based on the dual-granularity complementary correlation of the time domain view and the dual-granularity complementary correlation of the feature channel view, the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information are guided to mutually mine meaningful complementary information, and the interactive spatiotemporal coarse-grained information and the interactive spatiotemporal fine-grained information are output. The specific operation steps of the dual-granularity feature interaction module are as follows:

[0079] 4.1) For the input spatiotemporal coarse-grained information and spatiotemporal fine-grained information, we use the F2 receptive field of (1, ker dw ) is ker dw The depthwise convolution of filters with width 1 is mapped to a feature-rich latent space in a parameter-friendly manner and outputs a temporal view to enhance the spatiotemporal coarse-grained information. and time domain view to enhance spatiotemporal fine-grained information

[0080] 4.2) For the input temporal view enhanced spatiotemporal coarse-grained information and temporal view enhanced spatiotemporal fine-grained information, the temporal view dual-granularity complementary correlation t_map∈R is calculated based on correlation analysis t×t According to the dual-granularity complementary correlation analysis of the time domain view, the complementary relationship between the time domain view enhanced spatiotemporal coarse-grained information and the time domain view enhanced spatiotemporal fine-grained information is analyzed, thereby guiding the two to mutually mine meaningful complementary information and output the dual-granularity complementary information of the time domain view. The specific calculation process is as follows:

[0081]

[0082] In the formula, softmax represents the softmax function, T represents the transpose function, Represents matrix multiplication;

[0083] 4.3) For the input spatiotemporal coarse-grained information and spatiotemporal fine-grained information, first use the transposition function respectively, and then apply the t receptive fields (1, ker dw ) is ker dw The depthwise convolution of filters with a width of 1 is mapped to a feature-rich latent space in a parameter-friendly way and outputs feature channel views to enhance spatiotemporal coarse-grained information. and feature channel views to enhance spatiotemporal fine-grained information

[0084] 4.4) For the input feature channel view enhanced spatiotemporal coarse-grained information and feature channel view enhanced spatiotemporal fine-grained information, the feature channel view dual-granularity complementary correlation is calculated based on correlation analysis According to the dual-granularity complementary correlation analysis of the feature channel view, the complementary relationship between the feature channel view enhanced spatiotemporal coarse-grained information and the feature channel view enhanced spatiotemporal fine-grained information is analyzed, thereby guiding the two to mutually mine meaningful complementary information and output the dual-granularity complementary information of the feature channel view. The specific calculation process is as follows:

[0085]

[0086] 4.5) After obtaining the dual-granularity supplementary information of the time domain view and the dual-granularity supplementary information of the feature channel view, the fused dual-granularity supplementary information is obtained by adding the two together. Then the fused dual-granularity supplementary information is added to the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information respectively, thereby guiding the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information to mutually mine meaningful supplementary information and output the interactive spatiotemporal coarse-grained information. and interactive spatiotemporal fine-grained information The specific calculation process is as follows:

[0087] x fusion_map =T(x t_map )+x f_map

[0088] x inter_coarse =x pst_coarse +x fusion_map

[0089] x inter_fine =x pst_fine +x fusion_map

[0090] 5) If Figure 1 As shown, for the input interactive spatiotemporal coarse-grained information and interactive spatiotemporal fine-grained information Where R represents the real domain, F2 represents the number of filters for spatial convolution, and t represents the length of the time domain after removing redundant features. The sum operation is performed and a fully connected layer is applied to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category. N c Represents the number of types of motor imagery. The specific calculation process is as follows:

[0091] x result =F linear(x inter_coarse +x inter_fine )

[0092] Where, F linear Represents a fully connected layer.

[0093] 6) If Figure 1 As shown, the probability that the motor imagery EEG signal belongs to a certain motor imagery category is N c Represents the number of types of motor imagery, and uses the maximum index function to output the final motor imagery EEG signal decoding result:

[0094] result=maxindex(x result )

[0095] Where maxindex represents the maximum value index function.

[0096] The BCIC-IV-2a dataset contains motor imagery EEG signals from nine subjects. The motor imagery paradigm in this dataset covers four areas: the left hand, right hand, tongue, and foot. The researchers used 22 electrodes to collect EEG signals at a sampling rate of 250 Hz. After acquisition, the EEG signals were preprocessed using a bandpass filter with a bandpass frequency of 0.5 to 100 Hz. Each subject's motor imagery EEG signals were collected from two sessions, each consisting of 288 trials, with each trial lasting 4 seconds.

[0097] The BCIC-IV-2b dataset contains motor imagery EEG signals from nine subjects. The motor imagery paradigm in this dataset includes two positions: the left hand and the right hand. The researchers used three electrodes to collect EEG signals at a sampling rate of 250 Hz. After acquisition, the EEG signals were preprocessed using a bandpass filter with a bandpass frequency of 0.5 to 100 Hz. Each subject's motor imagery EEG signals were collected from five sessions: the first two sessions consisted of 120 trials each, and the next three sessions consisted of 160 trials each. Each trial lasted 4 seconds.

[0098] To verify the effectiveness of the proposed method, the decoding performance was compared with that of DMSACNN, MSVTNet, EEG-SimpleConv, EISATC-Fusion, FACT-Net, ADFCNN, LightConvNet, IFNet, FBMSNet and EEG Conformer methods under session-independent and session-dependent conditions on two commonly used datasets, BCIC-IV-2a and BCIC-IV-2b.

[0099] Among them, accuracy and Kappa are used to represent the decoding performance.

[0100] The experimental results are shown in Table 1.

[0101] Table 1 Comparison of decoding performance on the BCIC-IV-2a and BCIC-IV-2b datasets

[0102]

[0103]

[0104] Experimental results show that the motor imagery EEG signal decoding performance of the method of the present invention is better than other methods. For the BCIC-IV-2a dataset, the accuracy and Kappa of the method of the present invention under session-independent conditions are 5.17% and 0.07 higher than those of the best comparison method; for the BCIC-IV-2b dataset, the accuracy and Kappa of the method of the present invention under session-independent conditions are 2.66% and 0.05 higher than those of the best comparison method, verifying the effectiveness of the method of the present invention.

[0105] The above embodiments are preferred implementations of the present invention, but the implementations of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A motor imagery EEG signal decoding method based on full-scale filtering and dual-granularity interaction, characterized in that: The following steps are involved: S1: Acquire motor imagery EEG signals and preprocess them to obtain filtered motor imagery EEG signals with specific frequency components; S2: Decouple the frequency of the filtered motor imagery EEG signal into coarse-grained motor imagery EEG signal in the frequency domain and fine-grained motor imagery signal in the frequency domain; S3: Designed time-frequency filters are applied to the frequency-domain coarse-grained motor imagery EEG signals and the frequency-domain fine-grained motor imagery signals, respectively. The time-frequency filters first perform frequency-domain analysis on the frequency-domain coarse-grained motor imagery EEG signals and the frequency-domain fine-grained motor imagery signals based on a complex-valued neural network. Then, a residual structure is introduced to supplement the time-domain fluctuation information, and the coarse-grained information and the fine-grained information are output respectively to achieve full-scale filtering. Among them, the full-scale filtering for the coarse-grained information can effectively capture the complete global time-domain features, and the full-scale filtering for the fine-grained information can effectively capture all important local time-domain features. S4: Temporal convolution, spatial convolution, and average pooling are applied to the coarse-grained information and fine-grained information respectively, mapping the coarse-grained information and fine-grained information to a space with richer potential features, and outputting spatiotemporal coarse-grained information and spatiotemporal fine-grained information; S5: A dual-granularity feature interaction module designed for the application of spatiotemporal coarse-grained information and spatiotemporal fine-grained information. The dual-granularity feature interaction module calculates the dual-granularity complementary correlation in the time domain view and the dual-granularity complementary correlation in the feature channel view between the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information based on dual-axis attention. Then, based on the dual-granularity complementary correlation in the time domain view and the dual-granularity complementary correlation in the feature channel view, the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information are simultaneously guided to mutually mine meaningful complementary information, and the interactive spatiotemporal coarse-grained information and the interactive spatiotemporal fine-grained information are output; S6: Add the interactive spatiotemporal coarse-grained information and the interactive spatiotemporal fine-grained information, and apply a fully connected layer to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category; S7: Use the maximum index function to output the final motor imagery EEG signal decoding result according to the probability that the motor imagery EEG signal belongs to a certain motor imagery category.

2. The method for decoding motor imagery EEG signals based on full-scale filtering and dual-granularity interaction according to claim 1 is characterized in that: The specific steps of step S2 are as follows: S21: For the input filtered motor imagery EEG signal x raw ∈R C×T , where R represents the real number domain, C represents the number of electrodes collecting motor imagery EEG signals, and T represents the number of sampling points contained in the motor imagery EEG signals; based on the fast Fourier transform, the filtered motor imagery EEG signals are transformed into frequency domain filtered motor imagery EEG signals in represents the complex field, represents downward division; the frequency-domain filtered motor imagery EEG signal represents the time-domain pattern contained in the filtered motor imagery EEG signal in a compact and efficient manner. Specifically, each frequency-domain component of the frequency-domain filtered motor imagery EEG signal corresponds to a specific time-domain pattern of the filtered motor imagery EEG signal, the low-frequency component of the frequency-domain filtered motor imagery EEG signal represents the global time-domain pattern of the filtered motor imagery EEG signal, and the high-frequency component of the frequency-domain filtered motor imagery EEG signal represents the local time-domain pattern of the filtered motor imagery EEG signal; S22: Based on frequency mask Decouple the frequency domain filtered motor imagery EEG signal into a coarse-grained motor imagery EEG signal in the frequency domain Sum frequency domain fine-grained motion imagery signal Among them, the coarse-grained motor imagery EEG signal in the frequency domain retains the complete global time domain characteristics of the filtered motor imagery EEG signal, and the fine-grained motor imagery signal in the frequency domain has finer local time domain characteristics of the filtered motor imagery EEG signal. The specific calculation process is as follows: X coarse =X raw X fine =X raw ⊙freq_mask Where ⊙ represents element-by-element multiplication, freq_mask[k] represents the kth element of the frequency mask, and threshold represents the frequency threshold, which is used as a learnable parameter in the training process.

3. The method for decoding motor imagery EEG signals based on full-scale filtering and dual-granularity interaction according to claim 2, characterized in that: The specific operation steps of step S3 are as follows: S31: The designed time-frequency filter performs frequency domain analysis on the frequency domain coarse-grained motor imagery EEG signal and the frequency domain fine-grained motor imagery signal based on the complex-valued neural network, and outputs the frequency domain coarse-grained information and frequency domain fine-grained information The specific calculation process is as follows: X f_coarse =F CVNN_coarse (X coarse )=X coarse *W CVNN_coarse X f_fine =F CVNN_fine (X fine )=X fine *W CVNN_fine Where, F CVNN_coarse represents a complex-valued neural network for processing frequency-domain coarse-grained motor imagery EEG signals, * represents complex number multiplication, represents the weight of the complex-valued neural network used to process the frequency-domain coarse-grained motor imagery EEG signal, F CVNN_fine represents a complex-valued neural network for processing fine-grained motor imagery EEG signals in the frequency domain, represents the weights of the complex-valued neural network used to process frequency-domain fine-grained motor imagery EEG signals; S32: The designed time-frequency filter introduces inverse fast Fourier transform and residual structure to process the input frequency domain coarse-grained information and frequency domain fine-grained information, and outputs coarse-grained information x f_coarse ∈R C×T and fine-grained information x f_fine ∈R C×T ; Among them, the residual structure is introduced to supplement the time domain fluctuation information. The specific calculation process is as follows: x f_coarse =IFFT(X f_coarse )+x raw x f_fine =IFFT(X f_fine )+x raw Where IFFT stands for inverse fast Fourier transform.

4. The method for decoding motor imagery EEG signals based on full-scale filtering and dual-granularity interaction according to claim 3 is characterized in that: In step S4, for the input coarse-grained information x f_coarse ∈R C×T and fine-grained information x f_fine ∈R C×T , first use the F1 receptive field (1,ker coarse ) is ker coarse The temporal convolution of the filter with width 1 has a receptive field of F1 (1, ker fine ) is ker fine The temporal convolution of the filter with a width of 1 maps the coarse-grained information and the fine-grained information to a space with richer potential features. Then, the spatial convolution of the filter with a receptive field of F2 (C, 1) (length 1, width C) and a group number of F1 is used to compress the information in the spatial dimension and map it to a space with stronger expression ability. Finally, the spatial convolution of the filter with a receptive field of (1, ker pool ) is ker pool The average pooling layer with a width of 1 removes redundant features in the time domain dimension and outputs coarse-grained spatiotemporal information. and spatiotemporal fine-grained information in Represents the time domain length after removing redundant features.

5. The method for decoding motor imagery EEG signals based on full-scale filtering and dual-granularity interaction according to claim 4 is characterized in that: The specific operation steps of step S5 are as follows: S51: For the input spatiotemporal coarse-grained information and spatiotemporal fine-grained information, we use the F2 receptive field (1, ker dw ) is ker dw The depthwise convolution of filters with width 1 is mapped to a feature-rich latent space in a parameter-friendly manner and outputs a temporal view to enhance the spatiotemporal coarse-grained information. and time domain view to enhance spatiotemporal fine-grained information S52: For the input temporal view enhanced spatiotemporal coarse-grained information and temporal view enhanced spatiotemporal fine-grained information, calculate the temporal view dual-granularity complementary correlation t_map∈R based on correlation analysis t×t , according to the dual-granularity complementary correlation analysis of the time domain view, the complementary relationship between the time domain view enhanced spatiotemporal coarse-grained information and the time domain view enhanced spatiotemporal fine-grained information is analyzed, thereby guiding the two to mutually mine meaningful complementary information and output the dual-granularity complementary information of the time domain view The specific calculation process is as follows: In the formula, softmax represents the softmax function, T represents the transpose function, Represents matrix multiplication; S53: For the input spatiotemporal coarse-grained information and spatiotemporal fine-grained information, first use the transposition function respectively, and then apply the t receptive fields (1, ker dw ) is ker dw The depthwise convolution of filters with a width of 1 is mapped to a feature-rich latent space in a parameter-friendly way and outputs feature channel views to enhance spatiotemporal coarse-grained information. and feature channel views to enhance spatiotemporal fine-grained information S54: For the input feature channel view enhanced spatiotemporal coarse-grained information and the feature channel view enhanced spatiotemporal fine-grained information, calculate the feature channel view dual-granularity complementary correlation based on correlation analysis According to the dual-granularity complementary correlation analysis of the feature channel view, the complementary relationship between the feature channel view enhanced spatiotemporal coarse-grained information and the feature channel view enhanced spatiotemporal fine-grained information is analyzed, thereby guiding the two to mutually mine meaningful complementary information and output the dual-granularity complementary information of the feature channel view. The specific calculation process is as follows: S55: After obtaining the dual-granularity supplementary information of the time domain view and the dual-granularity supplementary information of the feature channel view, the fused dual-granularity supplementary information is obtained by adding the two together. Then the fused dual-granularity supplementary information is added to the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information respectively, thereby guiding the spatiotemporal coarse-grained information and the spatiotemporal fine-grained information to mutually mine meaningful supplementary information and output interactive spatiotemporal coarse-grained information. and interactive spatiotemporal fine-grained information The specific calculation process is as follows: x fusion_map =T(x t_map )+x f_map x inter_coarse =x pst_coarse +x fusion_map x inter_fine =x pst_fine +x fusion_map 。 6. The method for decoding motor imagery EEG signals based on full-scale filtering and dual-granularity interaction according to claim 5, characterized in that: In step S6, for the input interactive spatiotemporal coarse-grained information and interactive spatiotemporal fine-grained information Perform the addition operation and apply the fully connected layer to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category N c Represents the number of types of motor imagery. The specific calculation process is as follows: x result =F linear (x inter_coarse +x inter_fine ) Where, F linear Represents a fully connected layer.

7. The method for decoding motor imagery EEG signals based on full-scale filtering and dual-granularity interaction according to claim 6, characterized in that: In step S7, the probability that the motor imagery EEG signal belongs to a certain motor imagery category is Use the maximum index function to output the final motor imagery EEG signal decoding result: result=maxindex(x result ) Where maxindex represents the maximum value index function.

Images