An electroencephalogram signal decoding method based on time domain and frequency domain signal fusion

Abstract

The application discloses a kind of electroencephalogram signal decoding method based on time domain and frequency domain signal fusion, it is related to brain-computer interface and electroencephalogram signal processing field, comprising: build the brain-computer interface data acquisition environment based on encoding modulation visual evoked potential, utilize multichannel electroencephalogram acquisition equipment to obtain the original electroencephalogram observation signal set of subject under multiple category encoding stimulation, the present application can adaptively filter non-stationary noise, fully exploit time-frequency complementary features, retain signal high-order statistical information and fine-grained timing structure, significantly improve the accuracy and robustness of electroencephalogram signal decoding.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interface and EEG signal processing technology, specifically to an EEG signal decoding method based on the fusion of time-domain and frequency-domain signals. Background Technology

[0002] Brain-computer interface (BCI) technology can establish a direct communication channel between the human brain and external devices without relying on peripheral nerves and muscle tissue, and has broad application prospects in medical rehabilitation, assistive control for the disabled, and human-computer interaction. Among them, BCI systems based on coded modulation visual evoked potentials (c-VEP) have become a research hotspot in the field of brain-computer interfaces due to their advantages of high information transmission rate and low user training cost.

[0003] However, the actual acquired EEG signals are usually extremely weak in amplitude (microvolt level) and accompanied by various non-stationary noises such as electromyography artifacts, eye movement interference, and power frequency noise. The original signal-to-noise ratio is low, which poses a great challenge to accurate decoding. Traditional EEG signal decoding methods, such as linear methods like canonical correlation analysis (CCA) and linear discriminant analysis, are unable to capture the nonlinear characteristics of EEG signals and cannot effectively process the dynamic evolution information in the time domain and the phase structure information in the frequency domain at the same time. Existing deep learning methods based on convolutional neural networks (CNNs) can extract nonlinear features, but they are limited by the receptive field size of the convolutional kernel and cannot capture the long-range contextual dependencies of long-sequence EEG signals.

[0004] Furthermore, existing mainstream EEG decoding models still suffer from the following core defects: First, most adopt fixed-weight denoising or feature fusion strategies, which cannot adaptively modulate according to the real-time intensity of non-stationary noise, easily leading to excessive smoothing of signals in strong noise segments and loss of effective features in weak noise segments; Second, in frequency domain feature extraction, most only utilize amplitude spectrum information, ignoring phase information that is crucial for visual evoked potentials, or directly perform equal-weighted splicing of the real and imaginary parts of the spectrum, resulting in the dilution of effective discriminative features; Third, in the feature dimensionality reduction and decoding stages, most models only use global average pooling (GAP) to extract first-order statistical features (mean), resulting in serious loss of higher-order statistical information such as the energy distribution shape and fluctuation amplitude of the signal. At the same time, the pooling process is prone to losing fine-grained temporal topology that is crucial for classification, ultimately limiting the decoding accuracy of EEG signals in complex noise environments.

[0005] In the prior art, Chinese patent CN121365242A discloses "A Method for Recognizing Open-Set EEG Signals Based on Discriminant Feature Learning." This method achieves intra-class compactness and inter-class dispersion of closed-set features by constructing discriminant feature learning regularization and edge distribution regularization, thereby reducing the risk of misclassification of unknown EEG signals in open-set scenarios. However, this method mainly focuses on optimizing the distribution boundary of the feature space through a loss function, neglecting the in-depth mining of the complex temporal dynamic evolution and frequency-domain phase information of the EEG signal itself; moreover, it does not involve adaptive denoising for non-stationary noise and collaborative fusion of multi-view features, and cannot fully utilize the time-frequency complementary characteristics to further improve decoding accuracy.

[0006] Therefore, there is an urgent need to design a novel EEG signal decoding method to solve the problems existing in the above-mentioned technologies and achieve high-precision and robust decoding of EEG signals in complex noise environments. Summary of the Invention

[0007] In view of the above-mentioned shortcomings of the existing technology, the present invention provides a method for decoding EEG signals based on the fusion of time-domain and frequency-domain signals, which can effectively solve the problems of the existing technology.

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions; This invention discloses a method for decoding electroencephalogram (EEG) signals based on the fusion of time-domain and frequency-domain signals, comprising: S1: Build a brain-computer interface data acquisition environment based on coded modulation visual evoked potentials, and use multi-channel EEG acquisition equipment to acquire the raw EEG observation signal set of subjects under multi-category coded stimulation; S2: Construct a time-frequency co-manifold projection space, and adaptively denoise and reconstruct the original EEG observation signal set through a time-series manifold denoising model based on residual dynamics and a confidence modulation mechanism. Extract time-domain features and phase-preserving frequency-domain features respectively and perform adaptive weighted fusion. After adaptive redistribution, generate a time-frequency joint semantic representation. S3: Input the time-frequency joint semantic representation into the pre-normalized convolutional enhanced Transformer encoder, capture long-range contextual dependencies through a multi-head self-attention mechanism, extract local features by combining a convolutional feedforward network, and output a deep context encoding tensor. S4: Construct a heterogeneous dual-stream topology-aligned decoding head, input the deep context coding tensor into the translation-invariant statistical branch and the total order topology preservation branch respectively, and extract the global high-order statistical representation and fine-grained temporal structure representation containing first-order and second-order moments respectively. S5: Align global high-order statistical representations with fine-grained temporal structure representations through cross-spatial feature mapping, and output the final EEG intent decoding result based on posterior probability inference.

[0009] Furthermore, S1 specifically includes: S11: The m-sequence is used as a pseudo-random coding sequence to modulate the visual stimulus target. The sampling rate is set to 250Hz, the stimulus sequence length is 126 bits, and the screen refresh rate is 75Hz. S12: Using the OpenBCI EEG acquisition system as the multi-channel EEG acquisition device, EEG signals are acquired at the preset sampling rate in S11; The electrode arrangement scheme of the multi-channel EEG acquisition device follows the international 10-20 system, selecting the channel of the occipital visual area as the signal input channel and the electrode at the earlobe as the reference electrode.

[0010] Furthermore, the signal input channels of the occipital lobe visual region include channels O1, Oz, O2, PO3, POz, and PO4; the reference channels corresponding to the reference electrode include channels A1 and A2.

[0011] Furthermore, S2 specifically includes: S21: Construct a temporal manifold denoising model, perform temporal modeling and adaptive correction on the original EEG observation signal set, and complete adaptive denoising and reconstruction by introducing residual projection and confidence modulation mechanism to obtain the reconstructed signal; S22: Construct a temporal feature extraction branch, perform serialized manifold embedding on the reconstructed signal, combine dynamic position coding, capture the transient dynamic evolution of EEG signals in the time dimension, and extract the temporal features; S23: Construct a phase-preserving complex spectral feature space, perform a real fast Fourier transform on the reconstructed signal along the time dimension, extract the real and imaginary parts of the spectrum and concatenate them to preserve phase information, introduce a complex spectrum adaptive weighting mechanism based on channel attention, perform nonlinear mapping through a one-dimensional convolutional neural network with ReLU activation function and normalization layer, and then align to the sequence length through linear interpolation operation to extract the phase-preserving frequency domain features; S24: Concatenate the time-domain features and the phase-preserving frequency-domain features in the channel dimension to obtain concatenated time-frequency features. Introduce a time-frequency channel attention mechanism based on signal quality to the concatenated time-frequency features to complete adaptive weight redistribution and weighted fusion. Generate a joint time-frequency semantic representation through linear projection.

[0012] Furthermore, step S21 specifically includes: using a bidirectional gated recurrent unit to perform temporal modeling on the original EEG observation signal set; constructing a bidirectional contextual latent space through parallel computation of forward and backward latent state sequences; merging the bidirectional latent states through feature dimension splicing operations; gating modulation of the forgetting and updating of the information flow through update and reset gates; mapping the spliced ​​bidirectional features back to the original channel dimension through a linear projection layer; introducing a point-by-point confidence modulation mechanism; using a confidence-gated network to evaluate the noise intensity of the current time step point by point; using the evaluation results as projection weights to dynamically modulate the residual projection; adding the modulated projection results to the original input signal to output the reconstructed signal.

[0013] Furthermore, the prenormalized convolutional enhanced Transformer encoder consists of four stacked encoder layers, each of which includes a multi-head self-attention module and a convolutional feedforward network module. S3 specifically includes: S31: For the l-th layer encoder, l=1,2,3,4, the time-frequency joint semantic representation is first processed by layer normalization and multi-head self-attention module. The processing result is residually connected with the input feature to obtain intermediate feature; S32: The intermediate features are processed by the layer normalization and convolutional feedforward network module. The processing result is residually connected with the intermediate features to obtain the output features of the encoder of this layer. The output features of the 4th layer encoder are the deep context coding tensor.

[0014] Furthermore, the multi-head self-attention module comprises eight attention heads that are computed in parallel, and its specific computation steps include: S311: Map the normalized input features to query matrix, key matrix and value matrix respectively; S312: Calculate the scaled dot product attention score based on the query matrix and the key matrix, and obtain the attention weight after normalization; S313: The value matrix is ​​weighted and summed using attention weights. The outputs of the eight attention heads are concatenated and then linearly transformed to obtain the final output of the multi-head self-attention module, which is the context feature tensor that has been computed by multi-head self-attention and incorporates long-range context dependencies.

[0015] Furthermore, S4 specifically includes: S41: Through the translation-invariant statistical branch, the deep context coding tensor is processed by the pooling operator to extract a global high-order statistical representation that is translation-invariant to the time dimension. The global high-order statistical representation includes first-order statistical moments and second-order statistical moments, which are used to capture the overall energy distribution shape and fluctuation amplitude of the deep context coding tensor signal. S42: By using the total order topology preservation branch, the tensor vectorization operator is used to preserve the complete fine-grained temporal topology of the deep context-coded tensor signal, and the fine-grained temporal structure representation is extracted. S43: The global high-order statistical representation and the fine-grained temporal structure representation are projected onto the shared semantic space through a heterogeneous mapping layer.

[0016] Furthermore, S43 specifically includes: S431: For the translation-invariant statistical branch, the first-order statistical moments and the second-order statistical moments are concatenated along the feature dimension and mapped through a fully connected layer to obtain a global high-order statistical representation projected onto the shared semantic space; the first-order statistical moments are the mean of the signal sequence and the second-order statistical moments are the standard deviation of the signal sequence. S432: For the total order topology-preserving branch, the three-dimensional feature tensor is flattened into a one-dimensional vector, and after dimensionality reduction mapping by a fully connected layer, a fine-grained temporal structure representation projected onto the shared semantic space is obtained.

[0017] Furthermore, S5 specifically includes: S51: The global high-order statistical representation projected to the shared semantic space and the fine-grained temporal structure representation are spliced ​​and fused in the channel dimension to obtain the fused feature vector, thus completing the cross-space feature mapping alignment. S52: The fused feature vector is fed into the classification layer, and the posterior probability of each category of stimulus target corresponding to the input sample is calculated based on the Softmax function. The category with the highest probability is selected as the final EEG intention decoding result output.

[0018] Furthermore, the core hyperparameter configuration of the method's operation process is as follows: The batch size was set to 32, the EEG signal sequence length was set to 420, the number of input EEG channels was set to 8, the feature embedding dimension was set to 128, the single-head dimension of the multi-head self-attention module was set to 16, the hidden layer dimension of the convolutional feedforward network was set to 256, and the dropout probability of random deactivation was set to 0.5.

[0019] Compared with the known prior art, the technical solution provided by this invention has the following beneficial effects: This invention designs a temporal manifold denoising model based on residual dynamics and innovatively introduces a point-by-point confidence modulation mechanism. This mechanism can dynamically evaluate the noise intensity at each time step according to the hidden state of the bidirectional gated loop unit and assign differentiated denoising projection weights to non-stationary noise (such as sudden eye movement and electromyography artifacts) in different time periods. This not only achieves effective elimination of strong noise, but also avoids excessive smoothing of pure physiological signal waveforms, thus solving the core problem of poor adaptability of traditional fixed-weight denoising methods. This invention constructs a time-frequency co-manifold projection space and introduces a complex spectrum adaptive weighting and time-frequency channel cross-attention redistribution mechanism. Unlike traditional methods that only utilize amplitude spectrum or concatenate real and imaginary parts with equal weights, this invention preserves phase information completely by retaining both the real and imaginary parts of the spectrum, and dynamically learns the discrimination weights of amplitude and phase information through an adaptive weighting mechanism. At the same time, during time-frequency feature fusion, it can dynamically adjust the contribution ratio of time-domain and frequency-domain features according to the real-time quality of the signal. When a feature in one dimension is severely contaminated by noise, it automatically increases the weight of another stable feature dimension, effectively overcoming the problem that effective features are easily diluted by extreme noise, and greatly improving the robustness of the model in complex environments. This invention designs a pre-normalized convolution-enhanced Transformer encoder. The pre-normalized structure ensures the gradient stability during deep network training, solving the problem of gradient vanishing that is common in deep Transformers. By using a multi-head self-attention mechanism, it breaks through the bottleneck of limited receptive fields in traditional CNNs, enabling efficient capture of long-range contextual dependencies in long-sequence EEG signals. At the same time, by combining a convolution-enhanced feedforward neural network, it supplements the ability to perceive local detailed features, enabling the model to comprehensively understand the overall trend and local subtle changes in EEG signals, significantly improving the depth and richness of feature representation. This invention designs a heterogeneous dual-stream topology-aligned decoding head, which overcomes the limitation of traditional global average pooling, which only extracts the first moment and easily loses data distribution information. By simultaneously extracting the mean (first moment) and standard deviation (second moment), the model can more comprehensively capture the overall energy distribution and fluctuation amplitude of the signal. At the same time, the fine-grained temporal topology structure of the signal is completely preserved through tensor vectorization branching, preventing the loss of key temporal patterns during pooling. The complementary fusion of the two features significantly enhances the model's ability to distinguish subtle EEG differences and its generalization performance, effectively improving the decoding accuracy in complex scenarios. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0021] Figure 1 This is a flowchart illustrating a method for decoding electroencephalogram (EEG) signals based on the fusion of time-domain and frequency-domain signals. Figure 2 This is a detailed structural diagram of an EEG signal decoding method based on the fusion of time-domain and frequency-domain signals provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0023] The present invention will be further described below with reference to embodiments.

[0024] Example: This embodiment presents a method for decoding electroencephalogram (EEG) signals based on the fusion of time-domain and frequency-domain signals, such as... Figure 1 As shown, it includes: S1: Set up the data acquisition environment and obtain the raw EEG observation signal set. A brain-computer interface data acquisition environment based on coded modulation visual evoked potentials was established. Multi-channel EEG acquisition equipment was used to obtain the raw EEG observation signal set of the subjects under multi-category coded stimuli. The specific implementation process is as follows: S11: The m-sequence is used as a pseudo-random coding sequence to modulate the visual stimulus target. The EEG signal sampling rate is set to 250Hz, the stimulus sequence length is 126 bits, and the screen refresh rate of the visual stimulus presentation is 75Hz. In this embodiment, multiple visual stimulus targets are independently grayscale encoded and modulated by the m-sequence, so that different stimulus targets correspond to a unique pseudo-random coding sequence, inducing the subject's occipital visual area to produce an EEG response with specific temporal and spectral characteristics.

[0025] S12: Using the OpenBCI EEG acquisition system as a multi-channel EEG acquisition device, the subject's EEG signals were acquired at the 250Hz sampling rate preset in S11. The electrode arrangement of the multi-channel EEG acquisition device followed the international 10-20 system, selecting the O1, Oz, O2, PO3, POz and PO4 channels of the occipital visual cortex as signal input channels, and selecting the A1 and A2 electrodes on the subject's earlobe as reference electrodes.

[0026] In this embodiment, the acquired raw EEG observation signal set is stored in tensor form, with tensor dimensions of [batch size, sequence length, number of EEG channels], where the batch size is 32, the sequence length is 420, and the number of EEG channels is 8.

[0027] S2: Construct a time-frequency co-manifold projection space and generate a time-frequency joint semantic representation. A time-frequency co-manifold projection space is constructed. The original EEG observation signal set is adaptively denoised and reconstructed using a time-series manifold denoising model based on residual dynamics and a confidence modulation mechanism. Temporal features and phase-preserving frequency features are extracted and adaptively weighted and fused. After adaptive redistribution, a joint time-frequency semantic representation is generated, including the following sub-steps: S21: Construct a temporal manifold denoising model based on residual dynamics, use bidirectional gated recurrent units to perform temporal modeling and adaptive correction on the original EEG observation signal set, construct forward and backward contextual latent spaces, use gating mechanisms to adaptively adjust the forgetting and updating of information flow, introduce residual projection and point-by-point confidence modulation mechanisms to make the projection intensity dynamically change with the noise confidence of the time step, and complete the adaptive denoising and reconstruction of the original EEG observation signal set.

[0028] For the input vector at time step t, its hidden state update is determined by both the forward and backward processes, and the specific calculation is as follows: ; in, The original input EEG signal tensor, For batch size, For sequence length, Number of brainwave channels; This represents the input feature vector at time t; , These represent the update gate and the reset gate, used to control the retention and forgetting of information. For hidden layer dimensions; All are learnable weight matrices. The corresponding bias vector is used to capture the distribution characteristics of non-stationary noise such as electromyography artifacts and eye movement interference; and These represent the Sigmoid activation function and the hyperbolic tangent activation function, respectively. It represents the Hadamardi (or Hadama) stack; and These represent the forward and backward hidden state vectors at time step t, respectively. The complete set of bidirectional hidden state sequences is denoted as: and .

[0029] The calculation process for point-by-point noise confidence based on a bidirectional hidden state sequence is as follows: ; in, and These are the weight matrix and bias vector of the confidence-gated network, respectively. This represents the model's confidence assessment of the noise intensity at the current time step. A higher value indicates higher signal purity and lower noise intensity at the current time step. This represents a concatenation operation along the feature dimension.

[0030] Finally, the denoised and reconstructed signal output is: ; in, This is the weight matrix of the linear projection layer, used to weight the concatenated bidirectional features (dimension 1). Mapping back to the original channel dimension ensures that the filtered signal maintains the same tensor shape as the original input, thereby achieving residual adaptive correction of features.

[0031] S22: Construct a temporal feature extraction branch, perform serialized manifold embedding on the denoised and reconstructed signal, and combine dynamic position coding to capture the transient dynamic evolution of the EEG signal in the time dimension, extracting temporal features. The calculation formula is as follows: ; in, Features in the time domain; This is the weight matrix of the TokenEmbedding layer, with an input dimension of 8 and an output dimension of 8. This is used to map the input signal to a high-dimensional feature space; It is a learnable location encoding vector with dimensions (420, 128). By dynamically learning the location weights at different stimulus time periods, it compensates for the time drift of EEG signals and is used to supplement the location information of the sequence.

[0032] S23: Constructing the phase-preserving complex spectral feature space, firstly, for the corrected signal... Perform a real-number Fast Fourier Transform along the time dimension (dim=1) to extract the real part of the spectrum. With the imaginary part The sequences are then concatenated to preserve phase information. To establish the intrinsic correlation between the real and imaginary parts, a complex spectrum adaptive weighting mechanism based on channel attention is introduced. Subsequently, a one-dimensional convolutional neural network with ReLU activation function and normalization layer is used for nonlinear mapping, and the sequences are aligned to the sequence length through linear interpolation. Frequency domain features are obtained. The calculation formula is as follows: ; ; ; ; in, Includes one-dimensional convolutional layers, Activation function and normalization layer ( The kernel size is set to 7 to extract deep frequency domain features from spectral data containing phase information; This indicates a feature concatenation operation (dim=1); This represents the real-number Fast Fourier Transform, used to transform the corrected time-domain signal. Transform to the complex frequency domain; This indicates the extraction of the real part of the complex spectrum, used to characterize the amplitude information of the signal; This indicates the extraction of the imaginary part of the complex spectrum, used to characterize the phase information of the signal; This represents a linear interpolation operation; The initial complex spectrum features after splicing. To pass The adaptive weight matrix learned by one-dimensional convolution; Use the Sigmoid activation function; This represents the weighted complex spectrum features after processing by the adaptive weighting mechanism for complex spectra; This represents an element-wise multiplication operation. Targeting the characteristic of specific frequency and phase coupling in visually evoked EEG, this mechanism can dynamically allocate the discrimination weights of the real part (amplitude information) and the imaginary part (phase information), overcoming the problem of effective feature dilution caused by direct equal-weighted splicing.

[0033] The weighted frequency domain features are aligned to the original sequence length through linear interpolation, resulting in phase-preserving frequency domain features.

[0034] S24: Concatenate the time-domain features and the phase-preserving frequency-domain features in the channel dimension to obtain concatenated time-frequency features. Introduce a time-frequency channel attention mechanism based on signal quality to the concatenated time-frequency features to complete adaptive weight redistribution and weighted fusion. Generate a joint time-frequency semantic representation through linear projection.

[0035] First, the time-domain features and frequency-domain features are concatenated in the channel dimension (dim=2): ; in, Represents time-domain characteristics; Represents frequency domain characteristics; This indicates a feature concatenation operation with a channel dimension of 2; This represents the time-frequency joint characteristics after cascading.

[0036] An adaptive weight redistribution mechanism based on signal quality time-frequency features is introduced to calculate the attention weights of the time-frequency channels: ; ; ; in, Represents time-domain characteristics; Represents frequency domain characteristics; This indicates a feature concatenation operation with a channel dimension of 2; This is the weight matrix of the linear fusion layer, with an input dimension of 2×128 and an output dimension of 128. The time-frequency joint features after cascading; Global statistics used to extract joint features; This is a multilayer perceptron network used to calculate the attention weights for the time-frequency channels. This mechanism can dynamically adjust the contribution ratio of time-domain and frequency-domain features according to the actual quality of the input signal. When a feature in one dimension suffers severe non-stationary contamination, it automatically increases the weight of another stable feature, significantly enhancing the robustness of the fusion architecture.

[0037] Finally, a joint time-frequency semantic representation is generated through a linear projection layer: ; in, This is the weight matrix of the linear fusion layer, with an input dimension of 2×128 and an output dimension of 128; This is the final time-frequency joint semantic representation of the output; This represents the time-frequency joint feature tensor after dynamic weighting processing via the time-frequency channel attention mechanism; This represents the bias vector of the linear fusion layer.

[0038] S3: Enhances the Transformer encoder based on pre-normalized convolution, outputting a deep context encoding tensor. The time-frequency joint semantic representation is input into the pre-normalized convolutional enhanced Transformer encoder. A multi-head self-attention mechanism captures long-range contextual dependencies, and a convolutional feedforward network extracts local features, outputting a deep context-encoding tensor. The pre-normalized convolutional enhanced Transformer encoder consists of four stacked encoder layers. Each encoder layer contains a multi-head self-attention module and a convolutional feedforward network module, and includes the following sub-steps: S31: For the l-th layer encoder, l=1,2,3,4, the time-frequency joint semantic representation is first processed by layer normalization and multi-head self-attention module. The processing result is residually concatenated with the input features to obtain intermediate features. The calculation formula is as follows: ; in, This is a random deactivation operation with a dropout probability of 0.5 to prevent overfitting of the neural network. It is a multi-head self-attention mechanism, responsible for efficient sequence modeling; It is a normalization layer used to stabilize the training process; Indicates the first The input feature tensor of the layer encoder, when When, it is the time-frequency joint semantic representation of the input; when At that time, it was the first The output feature tensor of the layer encoder.

[0039] The multi-head self-attention module contains eight attention heads that are computed in parallel. Its specific computation steps include: S311: Map the normalized input features to the query matrix, key matrix, and value matrix respectively: ; ; ; in, This is the normalized input for this layer; These are the corresponding learnable linear projection weight matrices, with both the input and output dimensions being 128.

[0040] S312: Calculate the scaled dot product attention score based on the query matrix and the key matrix, and obtain the attention weights after normalization. ; ; ; in, These represent the query matrix, key matrix, and value matrix obtained from the input feature mapping, respectively. This represents the calculated scaled dot product attention score matrix; This represents the attention weight matrix after processing with the normalized exponential function; This represents the feature representation of the final output of the current single attention head; This represents a matrix multiplication operation; =128 / 8=16 (8 is the number of attention heads) represents the key vector dimension of a single attention head. This is a scaling factor used to prevent gradient vanishing; Indicates matrix transpose; This represents the normalized exponential function, normalized along the key vector dimension (dim=-1), where dim=-1 corresponds to the key sequence length dimension. Normalization along the last dimension ensures that the attention weights are reasonably allocated at each time step, avoiding feature confusion caused by normalization across the first dimension.

[0041] S313: Concatenate the outputs of the 8 attention heads and obtain the final output through a linear transformation: ; in, Indicates the first The output of each attention head has dimensions of (32, 420, 16). The output transformation weight matrix has an input dimension of 128 and an output dimension of 128.

[0042] S32: Intermediate Features After hierarchical normalization ( The layer consists of a feedforward neural network module (FFN) and a residual connection with intermediate features to obtain the output features of that layer. The calculation formula is as follows: ; in, That is, the time-frequency joint representation obtained in step S2; The output of the fourth encoder layer is used as the final high-dimensional encoded feature; the hidden layer dimension of the feedforward neural network (FFN) is set to 256; this structure, which places layer normalization at the beginning, ensures the stability of deep network training.

[0043] S4: Construct a heterogeneous dual-stream topology-aligned decoding head and extract dual-branch feature representations. A heterogeneous dual-stream topology-aligned decoding head is constructed. The deep context-coded tensor is input into the translation-invariant statistical branch and the total-order topology-preserving branch, respectively, to extract global high-order statistical representations and fine-grained temporal structure representations containing first and second-order moments. The process includes the following sub-steps: S41: By using the translation-invariant statistical branch, the pooling operator is used to process the deep context coding tensor to extract the global high-order statistical representation that is translation-invariant to the time dimension. The global high-order statistical representation includes first-order statistical moments and second-order statistical moments, which are used to capture the overall energy distribution shape and fluctuation amplitude of the deep context coding tensor signal. S42: By using the total order topology preservation branch, the tensor vectorization operator is used to preserve the complete fine-grained temporal topology of the deep context-coded tensor signal, and the fine-grained temporal structure representation is extracted. S43: The global high-order statistical representation and the fine-grained temporal structure representation are projected onto the shared semantic space through a heterogeneous mapping layer, specifically including: S431: For the translation-invariant statistical branch, the first-order and second-order statistical moments are concatenated along the feature dimension and mapped through a fully connected layer to obtain a global high-order statistical representation projected onto the shared semantic space. The calculation formula is as follows: ; ; ; In the formula: The three-dimensional feature tensor output by the encoder; express At time step eigenvectors; The first-order statistical moment characterizes the average energy level of the signal sequence; These are second-order statistical moments, characterizing the fluctuation amplitude and the degree of dispersion of the signal sequence; This represents a concatenation operation along the feature dimension; This represents the feature vector output by the global statistical branch, with dimensions (32, 256). This represents the activation function; and The weight matrix and bias vector of the fully connected layer in the GAP branch are used; by synchronously extracting and fusing higher-order statistics, the loss of data distribution shape information caused by traditional global average pooling (GAP) operation is effectively overcome.

[0044] S432: For the total order topology-preserving branch, the three-dimensional feature tensor is flattened into a one-dimensional vector, and after dimensionality reduction mapping by a fully connected layer, a fine-grained temporal structure representation projected onto the shared semantic space is obtained. The calculation formula is as follows: ; in, This represents the feature vector output by the tensor vectorization branch, with dimensions (32, 256). This represents the ReLU activation function; This represents the weight matrix of the fully connected layer in the tensor vectorization branch, used to reduce the dimensionality of the flattened features; For vectorization operators, the three-dimensional tensor Flattened into a one-dimensional vector, it preserves the complete temporal dependencies; The three-dimensional feature tensor output by the encoder; This is the bias vector for the tensor vectorized fully connected layer branch.

[0045] S5: Cross-spatial feature mapping alignment, outputting EEG intent decoding results. Aligning global high-order statistical representations with fine-grained temporal structure representations through cross-spatial feature mapping, and outputting the final EEG intention decoding result based on posterior probability inference, includes the following sub-steps: S51: The global high-order statistical representation projected onto the shared semantic space and the fine-grained temporal structure representation are concatenated and fused along the channel dimension to obtain a fused feature vector, thus completing the cross-space feature mapping alignment. ; in, This indicates a feature concatenation operation, used to merge two feature vectors along the channel dimension.

[0046] S52: The fused feature vector is fed into the classification layer. The posterior probability of each category of stimulus target corresponding to the input sample is calculated based on the Softmax function. The category with the highest probability is selected as the final EEG intention decoding result output. The calculation formula is as follows: ; in, The total number of categories of stimulus targets; and These represent the weight vector and bias scalar corresponding to the i-th class in the classification layer, respectively. The function is an exponential function; the denominator is a normalization constant to ensure that the sum of the predicted probabilities of all categories is 1. In this embodiment, the final model outputs a probability distribution matrix with dimensions (32, 6), and the category with the highest probability is selected as the decoding result of the EEG intent.

[0047] This embodiment achieves adaptive filtering of non-stationary noise through the above steps, fully explores the time-frequency complementary features of EEG signals, and completely preserves the high-order statistical information and fine-grained temporal structure of the signal. It effectively solves the core problems of poor denoising adaptability, loss of phase information, insufficient long-range dependency capture, and loss of feature information in traditional methods, and significantly improves the accuracy and robustness of EEG signal decoding in complex noise environments.

[0048] like Figure 2 As shown, the network structure flow provided by this invention is as follows: First, the data enters the time-frequency feature extraction and fusion stage (left side of the figure). The raw EEG signal is denoised by Bi-GRU and reconstructed with confidence modulation, then split into two paths: the left path sequentially undergoes dynamic position encoding and temporal manifold embedding to generate a time-domain feature vector; the right path undergoes RFFT transformation and concatenation of the real and imaginary parts, followed by adaptive weighting of the complex spectrum, and then extracts a frequency-domain feature vector through two layers of 1×1 one-dimensional convolution (with ReLU activation in between). The time-domain and frequency-domain feature vectors are then concatenated, and after adaptive weight redistribution through channel attention, a joint time-frequency semantic representation is generated through a linear fusion layer.

[0049] Subsequently, the features enter the convolutional enhanced Transformer encoding stage (middle of the figure). The time-frequency joint semantic representation first passes through a normalization layer, a multi-head self-attention module, a Dropout layer, and then undergoes the first residual connection; it then enters a convolutional feedforward network, which consists of a normalization layer (LayerNorm), two 1×1 one-dimensional convolutional layers (containing ReLU activation and Dropout in the middle), and finally outputs deep encoded features through a second residual connection.

[0050] Finally, the features enter the heterogeneous dual-stream decoding and output stage (right side of the figure). After the encoded features are split, they are processed in parallel: one stream extracts global high-order features through a global high-order statistical module that simultaneously extracts the mean and standard deviation, a fully connected layer, and ReLU activation; the other stream extracts local features through tensor flattening, a fully connected layer, and ReLU activation. After the two feature streams are fused, they sequentially pass through batch normalization (BatchNorm), ReLU activation, a final classification fully connected layer, and a softmax layer, ultimately outputting the EEG intention decoding result.

[0051] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for decoding electroencephalogram (EEG) signals based on the fusion of time-domain and frequency-domain signals, characterized in that, include: S1: Build a brain-computer interface data acquisition environment based on coded modulation visual evoked potentials, and use multi-channel EEG acquisition equipment to acquire the raw EEG observation signal set of subjects under multi-category coded stimulation; S2: Construct a time-frequency co-manifold projection space, and adaptively denoise and reconstruct the original EEG observation signal set through a time-series manifold denoising model based on residual dynamics and a confidence modulation mechanism. Extract time-domain features and phase-preserving frequency-domain features respectively and perform adaptive weighted fusion. After adaptive redistribution, generate a time-frequency joint semantic representation. S3: Input the time-frequency joint semantic representation into the pre-normalized convolutional enhanced Transformer encoder, capture long-range contextual dependencies through a multi-head self-attention mechanism, extract local features by combining a convolutional feedforward network, and output a deep context encoding tensor. S4: Construct a heterogeneous dual-stream topology-aligned decoding head, input the deep context coding tensor into the translation-invariant statistical branch and the total order topology preservation branch respectively, and extract the global high-order statistical representation and fine-grained temporal structure representation containing first-order and second-order moments respectively. S4 specifically includes: S41: Through the translation-invariant statistical branch, the deep context coding tensor is processed by the pooling operator to extract a global high-order statistical representation that is translation-invariant to the time dimension. The global high-order statistical representation includes first-order statistical moments and second-order statistical moments, which are used to capture the overall energy distribution shape and fluctuation amplitude of the deep context coding tensor signal. S42: By using the total order topology preservation branch, the tensor vectorization operator is used to preserve the complete fine-grained temporal topology of the deep context-coded tensor signal, and the fine-grained temporal structure representation is extracted. S43: The global high-order statistical representation and the fine-grained temporal structure representation are projected to the shared semantic space through a heterogeneous mapping layer; Specifically, S43 includes: S431: For the translation-invariant statistical branch, the first-order statistical moments and the second-order statistical moments are concatenated along the feature dimension and mapped through a fully connected layer to obtain a global high-order statistical representation projected onto the shared semantic space; the first-order statistical moments are the mean of the signal sequence and the second-order statistical moments are the standard deviation of the signal sequence. S432: For the total order topology preservation branch, the three-dimensional feature tensor is flattened into a one-dimensional vector, and after dimensionality reduction mapping by a fully connected layer, a fine-grained temporal structure representation projected onto the shared semantic space is obtained. S5: Align global high-order statistical representations with fine-grained temporal structure representations through cross-spatial feature mapping, and output the final EEG intent decoding result based on posterior probability inference.

2. The EEG signal decoding method based on time-domain and frequency-domain signal fusion according to claim 1, characterized in that, S1 specifically includes: S11: The m-sequence is used as a pseudo-random coding sequence to modulate the visual stimulus target; S12: Using the OpenBCI EEG acquisition system as the multi-channel EEG acquisition device, the channel of the occipital visual area is selected as the signal input, and the electrode at the earlobe is selected as the reference electrode to acquire EEG signals at a preset sampling rate. The signal input channels of the occipital lobe visual region include channels O1, Oz, O2, PO3, POz, and PO4; the reference channels corresponding to the reference electrode include channels A1 and A2.

3. The EEG signal decoding method based on time-domain and frequency-domain signal fusion according to claim 1, characterized in that, S2 specifically includes: S21: Construct a temporal manifold denoising model, perform temporal modeling and adaptive correction on the original EEG observation signal set, and complete adaptive denoising and reconstruction by introducing residual projection and confidence modulation mechanism to obtain the reconstructed signal; S22: Construct a temporal feature extraction branch, perform serialized manifold embedding on the reconstructed signal and combine it with dynamic position coding to extract temporal features; S23: Construct a phase-preserving complex spectral feature space, perform a real-number fast Fourier transform on the reconstructed signal, extract and splice the real and imaginary parts of the spectrum to preserve phase information, introduce a complex spectrum adaptive weighting mechanism based on channel attention, and align it through nonlinear mapping and linear interpolation of a one-dimensional convolutional neural network to extract phase-preserving frequency domain features. S24: Concatenate the time-domain features and the phase-preserving frequency-domain features in the channel dimension to obtain concatenated time-frequency features. Introduce a time-frequency channel attention mechanism based on signal quality to the concatenated time-frequency features to complete adaptive weight redistribution and weighted fusion. Generate a joint time-frequency semantic representation through linear projection.

4. The EEG signal decoding method based on time-domain and frequency-domain signal fusion according to claim 3, characterized in that, Step S21 specifically includes: using a bidirectional gated recurrent unit to perform temporal modeling on the original EEG observation signal set; constructing a bidirectional contextual latent space through parallel computation of forward and backward latent state sequences; merging the bidirectional latent states through feature dimension splicing operations; gating modulation of the forgetting and updating of the information flow through update and reset gates; mapping the spliced ​​bidirectional features back to the original channel dimension through a linear projection layer; introducing a point-by-point confidence modulation mechanism; using a confidence-gated network to evaluate the noise intensity of the current time step point by point; using the evaluation results as projection weights to dynamically modulate the residual projection; adding the modulated projection result to the original input signal to output the reconstructed signal.

5. The EEG signal decoding method based on time-domain and frequency-domain signal fusion according to claim 1, characterized in that, The prenormalized convolutional enhanced Transformer encoder consists of four stacked encoder layers, each of which contains a multi-head self-attention module and a convolutional feedforward network module. S3 specifically includes: S31: For the l-th layer encoder, l=1,2,3,4, the time-frequency joint semantic representation is first processed by layer normalization and multi-head self-attention module. The processing result is residually connected with the input feature to obtain intermediate feature; S32: The intermediate features are processed by the layer normalization and convolutional feedforward network module. The processing result is residually connected with the intermediate features to obtain the output features of the encoder of this layer. The output features of the 4th layer encoder are the deep context coding tensor.

6. The EEG signal decoding method based on time-domain and frequency-domain signal fusion according to claim 5, characterized in that, In step S31, the multi-head self-attention module processing specifically includes: S311: Map the normalized time-frequency joint semantic representation into a query matrix, a key matrix, and a value matrix, respectively; S312: Calculate the scaled dot product attention score based on the query matrix and the key matrix, and obtain the attention weight after normalization; S313: The value matrix is ​​weighted and summed using attention weights. The outputs of multiple attention heads are concatenated and then linearly transformed to obtain the final output of the multi-head self-attention module, namely the context feature tensor.

7. The EEG signal decoding method based on time-domain and frequency-domain signal fusion according to claim 1, characterized in that, The computational logic for the fine-grained temporal structure representation follows: ； ； ； In the formula: The three-dimensional feature tensor output by the encoder; express At time step eigenvectors; The first-order statistical moment characterizes the average energy level of the signal sequence; These are second-order statistical moments, characterizing the fluctuation amplitude and the degree of dispersion of the signal sequence; This represents a concatenation operation along the feature dimension; This represents the feature vector output by the global statistical branch; This represents the activation function; and Here are the weight matrix and bias vector of the fully connected layer in the GAP branch; L=420 represents the length of the time series.

8. The EEG signal decoding method based on time-domain and frequency-domain signal fusion according to claim 1, characterized in that, S5 specifically includes: S51: The global high-order statistical representation projected to the shared semantic space and the fine-grained temporal structure representation are spliced ​​and fused in the channel dimension to obtain the fused feature vector, thus completing the cross-space feature mapping alignment. S52: The fused feature vector is fed into the classification layer, and the posterior probability of each category of stimulus target corresponding to the input sample is calculated based on the Softmax function. The category with the highest probability is selected as the final EEG intention decoding result output.

Images