Time-frequency characteristic electroencephalogram decoding method and device

Abstract

The invention provides a time-frequency feature electroencephalogram decoding method and device, and relates to the technical field of electroencephalogram signal processing.The method comprises the steps that firstly, original electroencephalogram signals are obtained through electroencephalogram signal collecting equipment, the original electroencephalogram signals are preprocessed, and preprocessed electroencephalogram signal data E are obtained; 2, inputting the preprocessed electroencephalogram signal data E into a time domain filtering attention branch, and outputting electroencephalogram time domain features; 3, inputting the preprocessed electroencephalogram signal data E into a frequency domain residual convolution branch, and outputting electroencephalogram frequency domain features; step 4, splicing the electroencephalogram time domain feature and the electroencephalogram frequency domain feature to obtain a time-frequency fusion feature F; and 5, inputting the time-frequency fusion feature F into a classification layer, and outputting an interpretation result.

Description

Technical Field

[0001] This application relates to the field of brain signal processing technology, and in particular to a time-frequency characteristic EEG decoding method and apparatus. Background Technology

[0002] Brain-computer interface (BCI) technology enables direct interaction between humans and external devices by decoding electroencephalogram (EEG) signals generated by brain neural activity. This provides revolutionary solutions for rehabilitation of patients with limb disabilities, diagnosis of neurological diseases, and human-machine collaborative control. Through precise interpretation of EEG signals, it is possible to identify users' motor imagery, spelling intentions, and attentional states, constructing natural and direct neural feedback interaction channels. This has significant scientific research value and promising clinical application prospects.

[0003] However, decoding brainwave signals still faces many serious challenges: 1. Complex signal characteristics: EEG signals are weak electrophysiological signals with extremely low signal-to-noise ratios. They are easily affected by noise such as electromuscular signals, eye movement artifacts, and environmental electromagnetic interference, which can cause effective features to be submerged. 2. Significant dynamism and individual differences: Brain neural activity is highly dynamic, and there are significant differences in EEG signal patterns among different individuals. The signal characteristics of the same user may also change in different scenarios, which increases the difficulty of model adaptation. 3. Insufficient utilization of multi-dimensional information: EEG signals contain rich information in the time domain, frequency domain, and spatial domain. Existing methods mostly focus on single-dimensional feature extraction (such as only time domain waveform analysis or frequency domain spectrum analysis), failing to fully model the complex three-dimensional dependency relationship of channel-time-frequency, resulting in the loss of key discriminative features; 4. Limited model generalization ability: Existing technologies mostly rely on shallow feature combinations or simple model fusion, which cannot adaptively mine deep information in the time-frequency map that is crucial for decoding, resulting in low decoding accuracy. Furthermore, the model has weak generalization ability when applied across individuals and scenarios, which seriously restricts the large-scale promotion of BCI technology in real-world scenarios.

[0004] Therefore, how to efficiently extract robust multi-scale time-frequency features from low signal-to-noise ratio EEG signals, fully capture the dynamic changes and complex dependencies of the signals, and construct a high-precision, highly generalizable decoding model has become a key problem that urgently needs to be solved in the current field of brain-computer interfaces. Summary of the Invention

[0005] In view of this, this application provides a time-frequency feature EEG decoding method and apparatus, the main purpose of which is to improve the accuracy of EEG decoding and improve the generalization ability.

[0006] According to one aspect of this application, a time-frequency feature EEG decoding method is provided, comprising the following steps: Step 1: Acquire raw EEG signals using an EEG signal acquisition device, and perform preprocessing operations on the raw EEG signals to obtain preprocessed EEG signal data E; Step 2: Input the preprocessed EEG signal data E into the temporal filtering attention branch and output the EEG temporal features; Step 3: Input the preprocessed EEG signal data E into the frequency domain residual convolution branch and output the EEG frequency domain features; Step 4: Concatenate the EEG time-domain features with the EEG frequency-domain features to obtain the time-frequency fusion feature F; Step 5: Input the time-frequency fusion feature F into the classification layer and output the interpretation result.

[0007] In one implementation, step one involves preprocessing the raw EEG signal to obtain preprocessed EEG signal data E, including: Step 1: Perform bandpass filtering on the raw EEG signal within a preset frequency range; Step 2: Perform electrooculography (EOG) artifact removal on the filtered EEG signal to remove fluctuations caused by eye movements, and obtain the preprocessed EEG signal data E.

[0008] In one implementation, step two involves inputting the preprocessed EEG signal data E into a time-domain filtering attention branch and outputting EEG time-domain features, including: Step 1: Input the preprocessed EEG signal data E into the frequency filtering module, and output the EEG filtered signal R; Step 2: Input the EEG filtered signal R into the multi-scale global attention module to perform temporal relationship modeling, extract temporal dynamic features, and output the temporal encoded representation S; Step 3: Apply one-dimensional convolution and adaptive average pooling operations to the temporal encoded representation S to obtain the temporal channel average feature P; Step 4: Input the time-domain channel average feature P into the fully connected layer, perform feature flattening operation, and output the EEG time-domain feature.

[0009] In one implementation, step 1, which involves inputting the preprocessed EEG signal data E into a frequency filtering module and outputting a filtered EEG signal R, includes: Step 1.1: Perform a Fourier transform operation on the preprocessed EEG signal data E to convert it into a frequency domain signal; Step 1.2: Construct a frequency filter with shared channel parameters; Step 1.3: Multiply the frequency domain signal with the frequency filter to output the frequency domain filtered signal; Step 1.4: Perform an inverse Fourier transform on the frequency domain filtered signal to convert it back to the time domain and obtain the EEG filtered signal R.

[0010] In one implementation, step three involves inputting the preprocessed EEG signal data E into a frequency domain residual convolution branch and outputting EEG frequency domain features, including: Step 1: Perform frequency band decomposition on the preprocessed EEG signal data E, which is decomposed into five signal frequency bands with different characteristics; Step 2: Extract differential entropy features from the five signal frequency bands, project them into a two-dimensional topological map, and merge them to construct a three-dimensional frequency domain tensor M; Step 3: Apply a three-dimensional convolutional layer to the frequency domain three-dimensional tensor M to extract the shallow frequency domain tensor features U; Step 4: Input the frequency domain tensor shallow feature U into the multi-band residual fusion module, extract and fuse the features, and perform adaptive average pooling and fully connected operations to obtain the EEG frequency domain features.

[0011] According to one aspect of this application, a time-frequency feature EEG decoding device is provided, comprising: The preprocessing module is used to acquire raw EEG signals through an EEG signal acquisition device and perform preprocessing operations on the raw EEG signals to obtain preprocessed EEG signal data E. The temporal feature extraction module is used to input the preprocessed EEG signal data E into the temporal filtering attention branch and output the EEG temporal features. The frequency domain feature extraction module is used to input the preprocessed EEG signal data E into the frequency domain residual convolution branch and output the EEG frequency domain features. The feature fusion module is used to concatenate the EEG time-domain features and the EEG frequency-domain features to obtain the time-frequency fusion feature F; The interpretation module is used to input the time-frequency fusion feature F into the classification layer and output the interpretation result.

[0012] In one implementation, the preprocessing module includes: A bandpass filter unit is used to perform bandpass filtering on the raw EEG signal between 0.1 Hz and 50 Hz; The artifact removal unit is used to remove ocular artifacts from the filtered EEG signal, removing fluctuations caused by eye movements, and obtaining the preprocessed EEG signal data E.

[0013] In one implementation, the temporal feature extraction module includes: The frequency filtering unit is used to input the preprocessed EEG signal data E into the frequency filtering module and output the EEG filtered signal R. The temporal modeling unit is used to input the EEG filtered signal R into the multi-scale global attention module to perform temporal relationship modeling, extract temporal dynamic features, and output a temporal encoded representation S. The feature processing unit is used to apply one-dimensional convolution and adaptive average pooling operations to the temporal encoded representation S to obtain the temporal channel average feature P; The time-domain output unit is used to input the average feature P of the time-domain channel into the fully connected layer, perform feature flattening operation, and output the EEG time-domain feature.

[0014] In one implementation, the frequency filtering unit includes: The Fourier transform subunit is used to perform a Fourier transform operation on the preprocessed EEG signal data E, converting it into a frequency domain signal; The filter construction subunit is used to construct frequency filters with shared channel parameters; The product operation subunit is used to perform a product operation between the frequency domain signal and the frequency filter, and output a frequency domain filtered signal. The inverse transform subunit is used to perform an inverse Fourier transform operation on the frequency domain filtered signal, converting it back to the time domain to obtain the EEG filtered signal R.

[0015] In one implementation, the frequency domain feature extraction module includes: The frequency band decomposition unit is used to perform frequency band decomposition on the preprocessed EEG signal data E, which is decomposed into five signal frequency bands with different characteristics. Tensor construction unit is used to extract differential entropy features from the five signal frequency bands, project them into a two-dimensional topological graph, and merge them to construct a frequency domain three-dimensional tensor M. A shallow feature extraction unit is used to extract shallow features U of the frequency domain tensor by applying a three-dimensional convolutional layer to the frequency domain three-dimensional tensor M. The frequency domain output unit is used to input the frequency domain tensor shallow feature U into the multi-band residual fusion module, extract and fuse the features, and perform adaptive average pooling and fully connected operations to obtain the EEG frequency domain features.

[0016] By employing the above technical solution, this application provides a time-frequency feature EEG decoding method and apparatus, which has at least the following technical advantages compared with the prior art: (1) By constructing a parallel architecture of time-domain filtering attention branch and frequency-domain residual convolution branch, the multi-scale features of EEG signals are captured in the time domain through a multi-scale global attention module, and multi-band spatial features are extracted in the frequency domain. This overcomes the problem that the existing technology can only extract time-domain and frequency-domain information, which is insufficient. This application can simultaneously obtain key information in the time domain and frequency domain, and significantly improve the decoding accuracy.

[0017] (2) By embedding a learnable frequency filter block inside the time-domain branch, this application enables adaptive attenuation of the EEG signal spectrum, significantly suppressing non-stationary noise and high-frequency interference, overcoming the problems of fixed filters, low spectrum utilization and poor feature selectivity in the prior art, thus enabling this application to accurately extract important frequency components.

[0018] (3) This application maintains the integrity of five frequency band dimensions by using a three-dimensional convolutional layer in the frequency domain residual convolution branch, and prevents network degradation by using the residual structure in the multi-frequency band residual fusion block. This overcomes the problem of information loss in the process of extracting signal features in the prior art, enabling this application to maintain the integrity of key frequency domain information and significantly improve the utilization rate of frequency domain information.

[0019] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This paper presents a schematic diagram of the overall architecture of the time-frequency feature EEG decoding method based on frequency domain filtering and multi-scale attention provided in this application. Figure 2 A flowchart of the frequency filtering module provided in this application is shown; Figure 3 A schematic diagram of the structure of the multi-scale global attention module provided in this application is shown; Figure 4 This paper shows a schematic diagram of the structure of the multi-band residual fusion module provided in this application; Figure 5 A flowchart of a time-frequency feature EEG decoding method provided in this application is shown. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present application can be combined with each other.

[0022] As analyzed earlier, current EEG signal decoding methods mainly focus on single-dimensional feature extraction, lacking in-depth modeling of the multi-scale time-frequency dynamic characteristics and complex three-dimensional dependencies of channel-time-frequency inherent in EEG signals. Furthermore, in the process of EEG signal decoding, existing methods mostly remain at the level of superficial feature combination or simple model fusion, failing to fully and adaptively mine the deep information of key discriminative regions in the EEG time-frequency map. This limits the accuracy of decoding and the model's generalization ability across individuals and scenarios, hindering the large-scale application and promotion of BCI technology in practical scenarios such as medical rehabilitation and intelligent control.

[0023] Therefore, in view of the shortcomings of existing EEG decoding technology, this application aims to solve the following technical problems: 1. Insufficient utilization of multi-dimensional information: Existing methods mostly focus on feature extraction in a single dimension (time domain or frequency domain), failing to effectively model the complex three-dimensional dependencies of channel-time-frequency in EEG signals, resulting in the loss of time-frequency dynamic characteristics and key discriminative information, and limiting decoding accuracy.

[0024] 2. Weak noise and interference suppression: EEG signals have an extremely low signal-to-noise ratio and are easily affected by non-stationary noise, high-frequency interference and artifacts. Existing methods use fixed filters and cannot adaptively increase or decrease spectral components, making it difficult to accurately extract effective frequency features.

[0025] 3. Information loss during feature extraction: Existing technologies often lose key information in the frequency domain during feature extraction because they fail to fully preserve multi-band dimensions or lack anti-network degradation mechanisms, thus reducing feature utilization.

[0026] 4. Insufficient model generalization ability: Existing methods mostly rely on shallow feature combinations or simple model fusion, which cannot adaptively mine deep discriminative information in time-frequency graphs. This results in weak generalization ability of the model when applied across individuals and scenarios, making it difficult to meet the needs of real-world scenarios.

[0027] See Figure 1 This diagram illustrates the overall architecture of the time-frequency feature EEG decoding method based on frequency domain filtering and multi-scale attention provided in this application. The functional / logical relationships of each part are explained in detail below.

[0028] (a) Preprocessing Raw EEG signals are susceptible to artifacts from eye movements and muscle activity, as well as environmental noise. Preprocessing serves to purify these raw EEG signals. Specifically, bandpass filtering is performed, such as limiting the signal to the effective frequency band of 0.1Hz-50Hz, while removing eye movement artifacts, resulting in preprocessed EEG signal data that provides high-quality input for subsequent time-frequency feature extraction.

[0029] (ii) Temporal filtering attention branch 1. Frequency Filtering Module: This module receives preprocessed EEG signal data, first converts the time-domain signal to the frequency-domain signal using a Fourier transform, then uses a learnable frequency filter with channel-shared parameters to filter the frequency-domain signal, enhancing or attenuating specific frequency components. Finally, it converts the signal back to the time domain using an inverse Fourier transform to obtain the filtered EEG signal. This step adaptively highlights frequency information relevant to the decoding task and suppresses irrelevant interference frequencies.

[0030] See Figure 2 The flowchart of the frequency filtering module is shown. The frequency filtering module utilizes the time-frequency transformation characteristics of the Fourier transform, combined with a learnable frequency filter, to adaptively adjust the spectrum of the EEG signal. This highlights frequency information relevant to the decoding task, suppresses noise and irrelevant frequency interference, and thus obtains a more discriminative filtered EEG signal. Its working process includes the following three steps: 1) Input signal and Fourier transform: preprocessed EEG signal data. As input, the EEG signal first passes through a Fourier transform module to convert the time-domain EEG signal into a digital signal. Convert to the frequency domain signal. The Fourier transform maps waveform information from the time domain to the frequency domain, facilitating signal analysis and processing from the frequency dimension. 2) Frequency domain filtering operation: After obtaining the frequency domain signal, it is filtered by a frequency filter. Perform a product operation. Frequency filter. It is learnable and can adaptively enhance or attenuate specific frequency components in the frequency domain signal according to the needs of the decoding task, achieving precise control of the EEG signal spectrum, filtering out noise and irrelevant frequency interference, and retaining frequency information valuable for decoding. 3) Inverse Fourier Transform to Restore the Time Domain: The frequency domain signal processed by the frequency filter is then converted back to the time domain through the inverse Fourier transform module to obtain the filtered EEG signal. At this time The key frequency components have been highlighted, providing a higher quality input for subsequent time-domain feature extraction.

[0031] 2. Multi-scale Global Attention Module: Inputting EEG filtered signals, the module performs multi-scale modeling of the temporal relationships of the signals. Through attention mechanisms at different scales, it captures the dynamic features of the signals across different time spans, extracts temporal dynamic features that reflect the temporal change patterns, and outputs a temporal coded representation.

[0032] See Figure 3The diagram illustrates the structure of the multi-scale global attention module. This module is a key component in the temporal filtering attention branch, used to capture multi-scale temporal features of EEG signals. It extracts and fuses temporal features at different scales of the EEG signal through operations such as channel upsampling, splitting, multi-branch dilated convolution, concatenation, and channel downsampling. Simultaneously, it utilizes an attention mechanism to enhance the weight of key temporal information and suppress irrelevant information, thereby accurately capturing the multi-scale temporal dynamic characteristics of the EEG signal. Its working process includes the following four steps: 1) Channel Upsampling: The input filtered EEG signal first passes through the channel upsampling module, expanding the channel dimension of the signal to provide a richer channel foundation for subsequent multi-scale feature extraction, enabling the mining of temporal features from different channel perspectives. 2) Splitting and Multi-branch Dilated Convolution: The upsampled signal is split and distributed into multiple parallel branches. Each branch contains a dilated convolution. By setting different dilation rates, the dilated convolution can increase the receptive field without increasing the number of parameters, thereby extracting temporal features at different scales (i.e., different time spans). Multiple parallel branches of dilated convolution extract features from the signal at different scales. 3) Feature concatenation and attention mechanism: The features extracted by the dilated convolution of each branch are concatenated to integrate multi-scale temporal features. Then, an attention mechanism is implemented by multiplying with specific weight factors to weight the concatenated multi-scale features, enhancing the weight of key temporal features and suppressing irrelevant features. 4) Channel downsampling and residual connection: The attention-weighted features are then passed through a channel downsampling module to compress the channel dimension, resulting in a temporal feature that integrates multiple scales and has been enhanced by attention. Finally, through residual connection, the features of the original input signal are fused with the processed features, preserving the original information while enhancing the expressive power of the extracted multi-scale temporal features, outputting the final temporal-domain encoded representation for further processing of subsequent temporal features.

[0033] 3. Subsequent processing (one-dimensional convolution, adaptive average pooling, fully connected layer): One-dimensional convolution is performed on the temporal encoding representation to further extract features; adaptive average pooling is used to compress the feature dimension and retain key information; the fully connected layer flattens the features and finally outputs the EEG temporal features, which focus on the temporal dynamic characteristics of the EEG signal.

[0034] (III) Frequency Domain Residual Convolution Branch 1. Band Decomposition and Feature Extraction: After receiving preprocessed EEG signal data, band decomposition is performed, dividing the signal into five signal bands with different characteristics. Then, differential entropy features are extracted for each band, which can reflect information such as energy distribution within the band. These features are then projected into a two-dimensional topological graph and merged to construct a three-dimensional frequency domain tensor.

[0035] 2. 3D Convolutional Layer: Processes the frequency domain 3D tensor to extract shallow features. 3D convolution can capture features in both spatial and frequency domain dimensions simultaneously, mining information such as spatial distribution within the frequency domain.

[0036] 3. Multi-band Residual Fusion Module: Inputting shallow frequency domain tensor features, this module fuses features from different frequency bands through a residual structure, preventing network degradation and fully integrating multi-band information. Adaptive average pooling and fully connected operations are then performed to output EEG frequency domain features, which reflect the frequency domain characteristics of the EEG signal.

[0037] See Figure 4 This is a schematic diagram of the structure of a multi-band residual fusion module.

[0038] The multi-band residual fusion module is a core component in the frequency domain residual convolution branch used to fuse multi-band features and prevent network degradation. It extracts shallow features from the frequency domain tensor through 3D convolution, compresses the feature dimension using average pooling, utilizes residual blocks to address the degradation problem caused by increasing network depth, and then stabilizes the training process through batch normalization layers. This process is repeated N times to achieve deep fusion and enhancement of multi-band features, providing a higher quality representation for subsequent frequency domain feature output. Its working process includes the following five steps: 1) 3D convolution extracts shallow features: The input frequency domain 3D tensor first passes through a 3D convolutional layer. 3D convolution can simultaneously extract features from the tensor in both spatial and frequency domain dimensions, capturing shallow features such as spatial distribution within the frequency domain, resulting in preliminary shallow features of the frequency domain tensor. 2) Average pooling compresses dimensions: The features output from the 3D convolution are fed into an average pooling layer, which performs downsampling operations on the features, compressing the spatial dimension of the features, reducing subsequent computation, while retaining key information of the features, making the features more representative. 3) Residual Blocks Address Network Degradation: Features after average pooling are input into residual blocks. These blocks learn the residual mapping between input and output through residual connections (i.e., the input is directly added to the output), effectively solving the gradient vanishing and network degradation problems that may occur as network depth increases. This allows the network to mine frequency domain features more deeply and enhances feature expressiveness. 4) Batch Normalization Layer Stabilizes Training: Features processed by residual blocks are input into the batch normalization layer for feature normalization. This makes the feature distribution of each batch more stable, accelerating the network's training convergence speed and improving the model's generalization ability. 5) Repeated Process N Times: The above process, consisting of 3D convolution, average pooling layers, residual blocks, and batch normalization layers, is repeated N times. Through multiple iterations, the frequency domain features are continuously fused and enhanced, ultimately outputting optimized frequency domain features for further processing and output in subsequent frequency domain features.

[0039] (iv) Feature fusion By concatenating the EEG time-domain features from the time-domain filtered attention branch and the EEG frequency-domain features from the frequency-domain residual convolution branch, a time-frequency fusion feature integrating both time and frequency information is obtained. This fusion allows for the simultaneous utilization of both the time-domain dynamics and frequency-domain distribution information of the EEG signal, providing a more comprehensive basis for subsequent accurate decoding.

[0040] (v) Classification layer The system receives time-frequency fusion features, performs classification and other processing, and outputs interpretation results, such as identifying the user's motor imagination intentions and character spelling choices, to achieve the final decoding of EEG signals.

[0041] In short, Figure 1 In the architecture diagram shown, the raw EEG signal first undergoes preprocessing to remove noise, artifacts, and other interference, resulting in cleaner EEG signal data. This data is then simultaneously input into two parallel branch modules: a temporal filtering attention branch and a frequency domain residual convolution branch. The temporal filtering attention branch extracts and processes the temporal features of the EEG signal; the frequency domain residual convolution branch extracts and processes the frequency domain features. Afterward, the two branches output the processed features, which are then integrated into a feature fusion module to obtain a comprehensive feature that incorporates both time and frequency information. Finally, the comprehensive feature is input into a classification layer, where classification and other operations are performed to output the final interpretation result, thus decoding the intent or state implied by the EEG signal.

[0042] As can be seen, this application constructs a parallel architecture of a temporal filtering attention branch and a frequency domain residual convolution branch. In the temporal domain, it captures multi-scale temporal features through a multi-scale global attention module, and in the frequency domain, it extracts multi-band spatial features. Simultaneously, it fully exploits three-dimensional dependencies by fusing time-frequency features, significantly improving decoding accuracy. Specifically, the temporal branch embeds learnable frequency filtering blocks that adaptively attenuate the EEG signal spectrum, accurately suppressing non-stationary noise and high-frequency interference, effectively extracting key frequency components, and improving feature robustness. The frequency domain branch uses three-dimensional convolutional layers to fully preserve the five frequency band dimensions, and the residual structure of the multi-band residual fusion module prevents network degradation, avoiding the loss of key frequency domain information and significantly improving the integrity and utilization of frequency domain features. Furthermore, by deeply modeling the time-frequency dynamic characteristics and adaptively mining key discrimination region information, the model is more adaptable to individual differences and scene changes, and its generalization ability across individuals and scenes is effectively improved, laying the foundation for the large-scale application of brain-computer interface technology in practical scenarios such as medical rehabilitation and intelligent control.

[0043] See Figure 5 The flowchart of a time-frequency feature EEG decoding method provided in this application is shown. Figure 5 The process includes steps one through five, and the detailed process of each step is explained below.

[0044] Step 1: Obtain raw EEG signals through an EEG signal acquisition device, and perform preprocessing operations on the EEG signal data to obtain preprocessed EEG signal data E.

[0045] Specifically, the preprocessing operation on the EEG signal data to obtain preprocessed EEG signal data E includes the following steps: Step 1: Bandpass filter the raw EEG signal within a preset frequency range (e.g., between 0.1Hz and 50Hz); Step 2: Perform oculomotor artifact removal on the filtered EEG signal to remove fluctuations caused by eye movements and other actions, and obtain preprocessed EEG signal data E.

[0046] Step 2: Input the preprocessed EEG signal data E into the temporal filtering attention branch, and output the EEG temporal features. .

[0047] Specifically, the preprocessed EEG signal data E is input into the temporal filtering attention branch, and the EEG temporal features are output. It includes the following steps: Step 1: Input the preprocessed EEG signal data E into the frequency filtering module, and output the filtered EEG signal R; More specifically, the step of inputting the preprocessed EEG signal data E into the frequency filtering module and outputting the filtered EEG signal R includes the following steps: Step 1.1: Perform a Fourier transform operation on the preprocessed EEG signal data E to convert it into a frequency domain signal. ; Step 1.2: Construct a frequency filter with shared channel parameters ; Step 1.3: Convert the frequency domain signal With frequency filter Perform a product operation to output a frequency-domain filtered signal. ; Step 1.4: Filter the signal in the frequency domain Perform an inverse Fourier transform to convert back to the time domain and obtain the EEG filtered signal R.

[0048] Step 2: Input the EEG filtered signal R into the multi-scale global attention module to perform temporal relationship modeling, extract temporal dynamic features, and output the temporal encoded representation S; Step 3: Apply one-dimensional convolution and adaptive average pooling operations to the temporal encoded representation S to obtain the temporal channel average feature P; Step 4: Input the time-domain channel average feature P into the fully connected layer, perform feature flattening operation, and output the EEG time-domain feature. .

[0049] Step 3: Input the preprocessed EEG signal data E into the frequency domain residual convolution branch, and output the EEG frequency domain features. ; Specifically, the preprocessed EEG signal data E is input into the frequency domain residual convolution branch, and the output is the EEG frequency domain features. It includes the following steps: Step 1: Decompose the preprocessed EEG signal data E into five signal frequency bands with different characteristics. Step 2: Extract differential entropy features from the five signal frequency bands mentioned in Step 1, project them into a two-dimensional topological map, and merge them to construct a three-dimensional frequency domain tensor M. Step 3: Apply a 3D convolutional layer to the frequency domain 3D tensor M to extract the shallow frequency domain tensor features U; Step 4: Input the shallow frequency domain tensor feature U into the multi-band residual fusion module to extract and fuse features, and perform adaptive average pooling and fully connected operations to obtain the frequency domain features. .

[0050] Step 4: Combine the EEG time-domain features described in Step 2 and Step 3 With EEG frequency domain characteristics Perform splicing to obtain the time-frequency fusion feature F; Step 5: Input the time-frequency fusion feature F into the classification layer and output the interpretation result. .

[0051] The above process will be illustrated with a specific example below.

[0052] Suppose we want to use this method to decode the EEG signals corresponding to a user's intention to "imagine movement of the left hand".

[0053] Step 1: Acquisition and Preprocessing of Raw EEG Signals 1. First, use an EEG signal acquisition device (such as a multi-channel EEG cap) to collect the raw EEG signals generated when the user imagines moving their left hand. The raw EEG signals are mixed with various noises, such as electromagnetic interference from the surrounding environment, as well as electrical signals generated by the user's slight eye movements.

[0054] 2. Next, preprocessing is performed: Bandpass filtering: The raw EEG signal is bandpass filtered between 0.1 Hz and 50 Hz. Since the effective frequency components of the EEG signal associated with "imagining left hand movement" are usually within this range, filtering can remove unwanted signals at high frequencies (such as electromagnetic interference above 50 Hz) and low frequencies (such as slow baseline drift below 0.1 Hz).

[0055] Ocular artifact removal: Since users may experience slight eye movements while imagining movement, ocular artifacts may occur. By using ocular artifact removal algorithms, such as independent component analysis (ICA), the fluctuations caused by eye movements are identified and removed, resulting in preprocessed EEG signal data E. This E is now purer and better reflects the neural electrical activity of "imagining left hand movement".

[0056] Step 2: Temporal filtering and attention branch extraction of EEG temporal features 1. Frequency filtering module processing: Fourier Transform: The preprocessed EEG signal data E is subjected to a Fourier transform, converting it from the time domain to the frequency domain. In the frequency domain, the distribution of different frequency components can be seen more clearly, making it easier to adjust the frequencies related to "imagining left hand movement".

[0057] Constructing and applying frequency filters: Building frequency filters with shared channel parameters This filter can learn frequency patterns associated with "imagining left-hand movement" through training. The frequency domain signal is then compared with... By performing a product operation, useful frequency components are enhanced and irrelevant frequency components are suppressed to obtain a frequency domain filtered signal.

[0058] Inverse Fourier Transform: Performing an inverse Fourier transform on the frequency domain filtered signal converts it back to the time domain, yielding the EEG filtered signal R. At this point, R highlights the time-domain waveform characteristics associated with "imagining left hand movement."

[0059] 2. Multi-scale global attention module modeling: The EEG filtered signal R is input into the multi-scale global attention module. This module captures the temporal relationship of R over different time spans through attention mechanisms at different scales. For example, it captures the dynamic change pattern of the EEG signal over time during the process of "imagining the movement of the left hand," extracts the temporal dynamic features, and outputs the temporal encoded representation S.

[0060] 3. One-dimensional convolution and adaptive average pooling: Apply one-dimensional convolution to S to further extract features; then perform adaptive average pooling to compress the feature dimension, retain key information, and obtain the temporal channel average feature P.

[0061] 4. Fully connected layer flattening: Input P into the fully connected layer and perform feature flattening to obtain the EEG temporal features. This feature focuses on the temporal dynamics of "imagining left-hand movement".

[0062] Step 3: Extracting EEG frequency domain features using frequency domain residual convolution branching. 1. Band decomposition: The preprocessed EEG signal data E is decomposed into five signal bands with different characteristics, each containing EEG signal components in different frequency ranges.

[0063] 2. Differential Entropy Feature Extraction and 3D Tensor Construction: Differential entropy features are extracted from each of the five signal frequency bands. Differential entropy reflects information such as energy distribution within each frequency band. These features are then projected into a two-dimensional topological map and merged to construct a frequency domain 3D tensor M, which contains spatial distribution information of multiple frequency bands.

[0064] 3. Extracting shallow features through 3D convolution: Applying a 3D convolutional layer to M extracts shallow features U in the frequency domain tensor. 3D convolution can capture features related to "imagining left hand movement" in both spatial and frequency domain dimensions.

[0065] 4. Multi-band Residual Fusion: Input U into the multi-band residual fusion module. This module fuses features from different frequency bands through residual structures, preventing network degradation and fully integrating multi-band information. Adaptive average pooling and fully connected operations are then performed to obtain EEG frequency domain features. This feature reflects the frequency domain characteristics of "imagining left-hand movement".

[0066] Step 4: Time-Frequency Feature Fusion EEG temporal characteristics With EEG frequency domain characteristics The data is then spliced ​​together to obtain the time-frequency fusion feature F. F contains both the temporal dynamics and frequency distribution information of "imagining left hand movement," providing a more comprehensive characterization of the EEG features corresponding to this intention.

[0067] Step 5: The classification layer outputs the interpretation results. The time-frequency fusion feature F is input into the classification layer. The classification layer classifies and judges F based on the trained model, and finally outputs the interpretation result, that is, it recognizes that the user's intention at this time is "to imagine the movement of the left hand".

[0068] In summary, the time-frequency feature EEG decoding method provided in this application has at least the following technical advantages compared with the prior art: (1) By constructing a parallel architecture of time-domain filtering attention branch and frequency-domain residual convolution branch, the multi-scale features of EEG signals are captured in the time domain through a multi-scale global attention module, and multi-band spatial features are extracted in the frequency domain. This overcomes the problem that the existing technology can only extract time-domain and frequency-domain information, which is insufficient. This application has the advantage of being able to acquire key information in both time and frequency domains at the same time, which significantly improves the decoding accuracy.

[0069] (2) By embedding a learnable frequency filter block inside the time-domain branch, this application enables adaptive attenuation of the EEG signal spectrum, significantly suppressing non-stationary noise and high-frequency interference, overcoming the problems of fixed filters, low spectrum utilization and poor feature selectivity in the prior art, thus giving this application the advantage of being able to accurately extract important frequency components.

[0070] (3) This application maintains the integrity of five frequency band dimensions by using a three-dimensional convolutional layer in the frequency domain residual convolution branch, and prevents network degradation by using the residual structure in the multi-frequency band residual fusion block. This overcomes the problem of information loss in the process of extracting signal features in the prior art, and makes this application have the advantages of maintaining the integrity of key information in the frequency domain and significantly improving the utilization rate of frequency domain information.

[0071] Corresponding to the above method, this application also provides a time-frequency feature EEG decoding device, comprising: The preprocessing module is used to acquire raw EEG signals through an EEG signal acquisition device and perform preprocessing operations on the raw EEG signals to obtain preprocessed EEG signal data E. The temporal feature extraction module is used to input the preprocessed EEG signal data E into the temporal filtering attention branch and output the EEG temporal features. The frequency domain feature extraction module is used to input the preprocessed EEG signal data E into the frequency domain residual convolution branch and output the EEG frequency domain features. The feature fusion module is used to concatenate the EEG time-domain features and the EEG frequency-domain features to obtain the time-frequency fusion feature F; The interpretation module is used to input the time-frequency fusion feature F into the classification layer and output the interpretation result.

[0072] The preprocessing module includes: A bandpass filter unit is used to perform bandpass filtering on the raw EEG signal between 0.1 Hz and 50 Hz; The artifact removal unit is used to remove ocular artifacts from the filtered EEG signal, removing fluctuations caused by eye movements, and obtaining the preprocessed EEG signal data E.

[0073] The time-domain feature extraction module includes: The frequency filtering unit is used to input the preprocessed EEG signal data E into the frequency filtering module and output the EEG filtered signal R. The temporal modeling unit is used to input the EEG filtered signal R into the multi-scale global attention module to perform temporal relationship modeling, extract temporal dynamic features, and output a temporal encoded representation S. The feature processing unit is used to apply one-dimensional convolution and adaptive average pooling operations to the temporal encoded representation S to obtain the temporal channel average feature P; The time-domain output unit is used to input the average feature P of the time-domain channel into the fully connected layer, perform feature flattening operation, and output the EEG time-domain feature.

[0074] The frequency filtering unit includes: The Fourier transform subunit is used to perform a Fourier transform operation on the preprocessed EEG signal data E, converting it into a frequency domain signal; The filter construction subunit is used to construct frequency filters with shared channel parameters; The product operation subunit is used to perform a product operation between the frequency domain signal and the frequency filter, and output a frequency domain filtered signal. The inverse transform subunit is used to perform an inverse Fourier transform operation on the frequency domain filtered signal, converting it back to the time domain to obtain the EEG filtered signal R.

[0075] The frequency domain feature extraction module includes: The frequency band decomposition unit is used to perform frequency band decomposition on the preprocessed EEG signal data E, which is decomposed into five signal frequency bands with different characteristics. Tensor construction unit is used to extract differential entropy features from the five signal frequency bands, project them into a two-dimensional topological graph, and merge them to construct a frequency domain three-dimensional tensor M. A shallow feature extraction unit is used to extract shallow features U of the frequency domain tensor by applying a three-dimensional convolutional layer to the frequency domain three-dimensional tensor M. The frequency domain output unit is used to input the frequency domain tensor shallow feature U into the multi-band residual fusion module, extract and fuse the features, and perform adaptive average pooling and fully connected operations to obtain the EEG frequency domain features.

[0076] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A time-frequency feature EEG decoding method, characterized in that, Includes the following steps: Step 1: Acquire raw EEG signals using an EEG signal acquisition device, and perform preprocessing operations on the raw EEG signals to obtain preprocessed EEG signal data E; Step 2: Input the preprocessed EEG signal data E into the temporal filtering attention branch and output the EEG temporal features; Step 3: Input the preprocessed EEG signal data E into the frequency domain residual convolution branch and output the EEG frequency domain features; Step 4: Concatenate the EEG time-domain features with the EEG frequency-domain features to obtain the time-frequency fusion feature F; Step 5: Input the time-frequency fusion feature F into the classification layer and output the interpretation result.

2. The method according to claim 1, characterized in that, In step one, the raw EEG signal is preprocessed to obtain preprocessed EEG signal data E, including: Step 1: Perform bandpass filtering on the raw EEG signal within a preset frequency range; Step 2: Perform electrooculography (EOG) artifact removal on the filtered EEG signal to remove fluctuations caused by eye movements, and obtain the preprocessed EEG signal data E.

3. The method according to claim 1, characterized in that, In step two, the preprocessed EEG signal data E is input into the time-domain filtering attention branch, and the output EEG time-domain features include: Step 1: Input the preprocessed EEG signal data E into the frequency filtering module, and output the EEG filtered signal R; Step 2: Input the EEG filtered signal R into the multi-scale global attention module to perform temporal relationship modeling, extract temporal dynamic features, and output the temporal encoded representation S; Step 3: Apply one-dimensional convolution and adaptive average pooling operations to the temporal encoded representation S to obtain the temporal channel average feature P; Step 4: Input the time-domain channel average feature P into the fully connected layer, perform feature flattening operation, and output the EEG time-domain feature.

4. The method according to claim 3, characterized in that, In step 1, the preprocessed EEG signal data E is input into the frequency filtering module, and the output EEG filtered signal R includes: Step 1.1: Perform a Fourier transform operation on the preprocessed EEG signal data E to convert it into a frequency domain signal; Step 1.2: Construct a frequency filter with shared channel parameters; Step 1.3: Multiply the frequency domain signal with the frequency filter to output the frequency domain filtered signal; Step 1.4: Perform an inverse Fourier transform on the frequency domain filtered signal to convert it back to the time domain and obtain the EEG filtered signal R.

5. The method according to claim 1, characterized in that, In step three, the preprocessed EEG signal data E is input into the frequency domain residual convolution branch, and the output EEG frequency domain features include: Step 1: Perform frequency band decomposition on the preprocessed EEG signal data E, which is decomposed into five signal frequency bands with different characteristics; Step 2: Extract differential entropy features from the five signal frequency bands, project them into a two-dimensional topological map, and merge them to construct a three-dimensional frequency domain tensor M; Step 3: Apply a three-dimensional convolutional layer to the frequency domain three-dimensional tensor M to extract the shallow frequency domain tensor features U; Step 4: Input the frequency domain tensor shallow feature U into the multi-band residual fusion module, extract and fuse the features, and perform adaptive average pooling and fully connected operations to obtain the EEG frequency domain features.

6. A time-frequency characteristic EEG decoding device, characterized in that, include: The preprocessing module is used to acquire raw EEG signals through an EEG signal acquisition device and perform preprocessing operations on the raw EEG signals to obtain preprocessed EEG signal data E. The temporal feature extraction module is used to input the preprocessed EEG signal data E into the temporal filtering attention branch and output the EEG temporal features. The frequency domain feature extraction module is used to input the preprocessed EEG signal data E into the frequency domain residual convolution branch and output the EEG frequency domain features. The feature fusion module is used to concatenate the EEG time-domain features and the EEG frequency-domain features to obtain the time-frequency fusion feature F; The interpretation module is used to input the time-frequency fusion feature F into the classification layer and output the interpretation result.

7. The apparatus according to claim 6, characterized in that, The preprocessing module includes: A bandpass filter unit is used to perform bandpass filtering on the raw EEG signal between 0.1 Hz and 50 Hz; The artifact removal unit is used to remove ocular artifacts from the filtered EEG signal, removing fluctuations caused by eye movements, and obtaining the preprocessed EEG signal data E.

8. The apparatus according to claim 6, characterized in that, The time-domain feature extraction module includes: The frequency filtering unit is used to input the preprocessed EEG signal data E into the frequency filtering module and output the EEG filtered signal R. The temporal modeling unit is used to input the EEG filtered signal R into the multi-scale global attention module to perform temporal relationship modeling, extract temporal dynamic features, and output a temporal encoded representation S. The feature processing unit is used to apply one-dimensional convolution and adaptive average pooling operations to the temporal encoded representation S to obtain the temporal channel average feature P; The time-domain output unit is used to input the average feature P of the time-domain channel into the fully connected layer, perform feature flattening operation, and output the EEG time-domain feature.

9. The apparatus according to claim 8, characterized in that, The frequency filtering unit includes: The Fourier transform subunit is used to perform a Fourier transform operation on the preprocessed EEG signal data E, converting it into a frequency domain signal; The filter construction subunit is used to construct frequency filters with shared channel parameters; The product operation subunit is used to perform a product operation between the frequency domain signal and the frequency filter, and output a frequency domain filtered signal. The inverse transform subunit is used to perform an inverse Fourier transform operation on the frequency domain filtered signal, converting it back to the time domain to obtain the EEG filtered signal R.

10. The apparatus according to claim 6, characterized in that, The frequency domain feature extraction module includes: The frequency band decomposition unit is used to perform frequency band decomposition on the preprocessed EEG signal data E, which is decomposed into five signal frequency bands with different characteristics. Tensor construction unit is used to extract differential entropy features from the five signal frequency bands, project them into a two-dimensional topological graph, and merge them to construct a frequency domain three-dimensional tensor M. A shallow feature extraction unit is used to extract shallow features U of the frequency domain tensor by applying a three-dimensional convolutional layer to the frequency domain three-dimensional tensor M. The frequency domain output unit is used to input the frequency domain tensor shallow feature U into the multi-band residual fusion module, extract and fuse the features, and perform adaptive average pooling and fully connected operations to obtain the EEG frequency domain features.

Images