Tone electroencephalogram decoding method and system based on spatiotemporal attention collaborative mechanism

Abstract

This invention proposes a method and system for tone-based EEG decoding based on a spatiotemporal attention coordination mechanism, belonging to the field of EEG decoding technology. The method includes: acquiring EEG signals from subjects to obtain raw EEG data; preprocessing the raw EEG data to obtain standardized EEG data; dividing the standardized EEG data into training and testing sets, inputting them into an ECA-MTCNet decoding model for training; optimizing the training process by adjusting model parameters to obtain the trained ECA-MTCNet decoding model; inputting the Chinese tone EEG signal to be decoded into the trained ECA-MTCNet decoding model, and outputting the corresponding tone classification result. This achieves high-precision and objective decoding of Chinese tone EEG signals, providing an effective auditory assessment tool for infants, people with language disorders, and other groups unable to cooperate with subjective auditory observation, filling a technological gap in the field of objective auditory assessment of Chinese tones.

Description

Technical Field

[0001] This invention belongs to the field of EEG decoding technology, and in particular relates to a method and system for early non-subjective hearing assessment based on spatiotemporal attention coordination mechanism using tone EEG. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Mandarin Chinese is a typical tonal language, and accurate perception of tones directly affects verbal communication ability. Since tonal features are the semantic features of the basic units of Mandarin pronunciation, and because these features are relatively abstract, and subjective testing usually relies on the subject's educational level and cognitive ability, objective auditory testing and assessment based on EEG evoked by tonal sample sounds has significant application value for infants, children, and people with language disorders who cannot cooperate with subjective auditory observation. However, the EEG signals evoked by Mandarin tones present decoding and detection challenges, including extremely low signal-to-noise ratios, numerous variations in physical quantities, and significant individual differences in brain networks. Traditional EEG analysis and modeling methods struggle to comprehensively and effectively extract the distinctive features of tones, leading to insufficient predictive decoding accuracy and affecting the test results.

[0004] In the existing technology, EEG signal decoding and detection models based on convolutional neural networks have improved the classification and decoding performance of various EEG detections to a certain extent, but the processing techniques used for EEG detection tasks are different.

[0005] EEG induced by Chinese tone features is an EEG induced by objective sound events outside the body. Unlike the spontaneous EEG of motor imagery and subjective brain cognition detection in brain-computer interfaces, its detection has the unique complexity of short-duration, multi-channel EEG caused by the appearance of tone features and the multi-dimensional changes of sound signals in energy, pitch and spectral structure. It is also accompanied by other invalid interference signals in the brain. Therefore, the corresponding EEG has unique and rich information in terms of temporal sequence and lead space.

[0006] To address this, while existing technologies have incorporated deep network structures and attention mechanisms, these are mostly general-purpose structures that have not been optimized for the dynamic characteristics of Chinese tone-induced EEG signals over time and space. As a result, existing models and systems are unable to adapt to high-performance decoding of non-subjective tone-induced EEG signals. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, this invention provides a tone EEG decoding method and system based on a spatiotemporal attention coordination mechanism, which can simultaneously achieve spatial noise suppression and multi-temporal spatiotemporal feature extraction for Chinese tone EEG decoding.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] Firstly, a tone-based EEG decoding method based on a spatiotemporal attention coordination mechanism is disclosed.

[0010] include:

[0011] EEG signals were collected from the subjects to obtain raw EEG data, and the raw EEG data was preprocessed to obtain standardized EEG data.

[0012] Standardized EEG data were divided into training and testing sets and input into the ECA-MTCNet decoding model for training. The training process was optimized by adjusting the model parameters to obtain the trained ECA-MTCNet decoding model.

[0013] The ECA-MTCNet decoding model includes a front-end feature extraction module, a sliding window augmentation module, a dual attention co-coding module, and a classification output module.

[0014] The front-end feature extraction module is used to process the input standardized EEG data, extract the primary spatiotemporal features of the EEG signal, and realize adaptive calibration of channel weights.

[0015] The sliding window augmentation module is used to segment the feature sequence output by the front-end feature extraction module into multiple overlapping sub-windows to achieve data augmentation.

[0016] The spatiotemporal attention co-coding module is used to capture the multi-temporal spatiotemporal dependent features of EEG signals;

[0017] The classification output module is used to concatenate and fuse the encoded features and output the classification result;

[0018] The EEG signal of Chinese tone to be decoded is input into the trained ECA-MTCNet decoding model, and the corresponding tone classification result is output.

[0019] As a further technical solution, the front-end feature extraction module includes a temporal convolutional layer, a channel depth convolutional layer, a spatial convolutional layer, an efficient channel attention (ECA) module, and an average pooling layer.

[0020] The temporal convolutional layer takes a standardized EEG tensor as input and performs convolution operations along the time dimension using a one-dimensional convolution kernel. By simulating the function of a bandpass filter, it can extract time-frequency features of different frequency bands from the raw EEG signal.

[0021] Channel depthwise convolutional layers are used to perform depthwise separable convolutions on the output of the previous layer;

[0022] Spatial convolutional layer: Convolution is performed along the electrode channel dimension to fuse spatial information from the whole brain;

[0023] The efficient channel attention (ECA) module is embedded between the spatial convolutional layer and the average pooling layer to perform channel weight calibration at the source of feature extraction;

[0024] Average pooling layer: used to downsample the feature map.

[0025] As a further technical solution, the high-efficiency channel attention (ECA) module processes the input data, including:

[0026] Global average pooling is performed on the feature map output by the spatial convolutional layer to obtain the channel descriptor;

[0027] Based on the number of channels, the size k1 of the one-dimensional convolution kernel is determined using an adaptive algorithm;

[0028] A one-dimensional convolution of size k1 is used to perform local cross-channel interactive computation on the channel descriptor;

[0029] Channel attention weights are generated by an activation function, and then weighted and fused with the feature map output by the spatial convolutional layer to obtain the calibrated feature map.

[0030] As a further technical solution, the dual attention co-coding module includes a multi-head self-attention sub-module and a temporal convolutional TCN sub-module;

[0031] The multi-head self-attention submodule is used to map the sub-feature sequences output by the sliding window to query vector Q, key vector K, and value vector V, respectively; and calculates the attention weights of each sub-feature sequence by scaling dot product attention.

[0032] The Temporal Convolutional (TCN) submodule employs a dilated causal convolution structure.

[0033] As a further technical solution, when collecting the subject's EEG signals, an improved passive Oddball EEG evoked paradigm is designed, using FHz pure tones as standard stimuli and the four tones of a certain Chinese vowel syllable (level, rising, falling-rising, and falling) as deviation stimuli. Each experimental block contains n stimuli, with a stimulation duration of T1 milliseconds, a stimulation interval of T2 milliseconds, and a sampling frequency of F1 Hz to collect the subject's EEG signals.

[0034] As a further technical solution, the raw EEG data is preprocessed, specifically including: filtering the raw EEG data using a Butterworth bandpass filter;

[0035] The filtered data was converted into a whole-brain average reference; it was then segmented according to a time window from a milliseconds before stimulus presentation to b milliseconds after presentation.

[0036] Baseline correction was performed using the data from a milliseconds before stimulation as the baseline.

[0037] Independent component analysis algorithm was used to remove artifacts from electrooculography (EOG) and electromyography (EMG).

[0038] Secondly, a tone-based EEG decoding system based on a spatiotemporal attention coordination mechanism is disclosed, including:

[0039] The EEG data acquisition and preprocessing module is configured to: acquire the subject's EEG signals to obtain raw EEG data, and preprocess the raw EEG data to obtain standardized EEG data;

[0040] The ECA-MTCNet decoding model building module is configured to: divide standardized EEG data into training and testing sets, input them into the ECA-MTCNet decoding model for training, optimize the training process by adjusting the model parameters, and obtain the trained ECA-MTCNet decoding model.

[0041] The ECA-MTCNet decoding model includes a front-end feature extraction module, a sliding window augmentation module, a dual attention co-coding module, and a classification output module.

[0042] The front-end feature extraction module is used to process the input standardized EEG data, extract the primary spatiotemporal features of the EEG signal, and realize adaptive calibration of channel weights.

[0043] The sliding window augmentation module is used to segment the feature sequence output by the front-end feature extraction module into multiple overlapping sub-windows to achieve data augmentation.

[0044] The dual attention co-coding module is used to capture multi-temporal spatiotemporal dependent features of EEG signals;

[0045] The classification output module is used to concatenate and fuse the encoded features and output the classification result;

[0046] The tone classification module is configured to input the EEG signals of Chinese tones to be decoded into the trained ECA-MTCNet decoding model and output the corresponding tone classification results.

[0047] The above one or more technical solutions have the following beneficial effects:

[0048] The technical solution of this invention employs the following steps: constructing a database of Mandarin four-tone evoked EEG data; preprocessing the EEG data; constructing an ECA-MTCNet decoding model based on efficient channel attention and multi-head spatiotemporal attention collaboration; inputting the preprocessed EEG data into the model for training; and using the trained model to decode and classify Mandarin tone EEG signals. This invention achieves adaptive calibration of channel weights through efficient channel attention at the front end of the ECA-MTCNet decoding model, suppressing noise channel interference. Combined with multi-head spatiotemporal attention at the back end of the ECA-MTCNet decoding model to capture multi-temporal dependent features, it ensures the decoding accuracy of Mandarin tone EEG signals. It can be used for non-subjective auditory perception assessment of any population, including infants and hearing-impaired individuals, solving the problem of insufficient decoding accuracy caused by inadequate extraction of spatiotemporal features of Mandarin tone-evoked EEG signals.

[0049] This invention focuses on the early assessment of objective auditory perception, employing a passive Oddball paradigm where subjects do not need to actively respond. It aims to accurately classify discrete Chinese tone categories using the ECA-MTCNet model. For passively evoked tone perception assessment, the ECA-MTCNet model captures the unique multi-temporal spatiotemporal dependence of tones. It processes event-related potentials (ERPs) induced by external physical sound (tone) stimuli, whose signal characteristics change with sound attributes. Technically, the ECA module is combined with TCN to capture these multi-temporal spatiotemporal dependence features. This application is based on non-invasive scalp electroencephalography (EEG), with a portable and non-invasive device, making it particularly suitable for widespread early clinical hearing screening and diagnosis in infants and individuals with language disorders.

[0050] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0051] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0052] Figure 1 This is a schematic diagram of an experimental paradigm for an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of the model architecture of an embodiment of the present invention. Detailed Implementation

[0054] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0055] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0056] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0057] Terminology Explanation:

[0058] ECA: Efficient Channel Attention;

[0059] MTCNet: Multi-head Temporal Convolutional Network;

[0060] Passive Oddball: An improved EEG evoked paradigm that induces event-related potentials (ERPs), particularly the P300 component, in the brain by presenting standard stimuli at high frequencies and deviated stimuli at low frequencies, for assessing auditory perception processing.

[0061] Example 1

[0062] This embodiment discloses a tone-based EEG decoding method based on a spatiotemporal attention coordination mechanism, including:

[0063] Step 1: Constructing a Chinese four-tone evoked EEG database: A passive Oddball auditory evoked paradigm was designed, using a pure tone (single-frequency tone) of F=440Hz as the standard stimulus and the four tones of the Chinese single vowel / a / (high level tone, rising tone, falling-rising tone, and falling tone) as the deviation stimuli. During the experiment, subjects were required to watch a silent movie to remain awake and avoid moving around as much as possible, while EEG signals were collected simultaneously to obtain raw data.

[0064] EEG data preprocessing: The raw EEG data is subjected to bandpass filtering, rereference, segmentation, baseline correction and artifact removal to obtain standardized EEG data.

[0065] Step 2: Construct the ECA-MTCNet decoding model: The model includes a front-end feature extraction module, a sliding window augmentation module, a dual attention co-coding module, and a classification output module.

[0066] The front-end feature extraction module is used to extract primary spatiotemporal features of EEG signals and adaptively calibrate the channel weights of the extracted features. Finally, it outputs the calibrated feature sequence as the input of the sliding window augmentation module.

[0067] The sliding window data augmentation module aims to effectively augment data by performing overlapping slicing on the feature sequences output from the front end. By setting a fixed window length and movement step size, it transforms samples into multiple time-series feature sequences containing local temporal information within fixed windows. This not only alleviates the overfitting problem caused by the limited size of EEG samples but also enhances the model's ability to capture the local dynamic features of tone-based EEG signals.

[0068] The dual attention co-encoding module is used to capture multi-temporal spatiotemporal dependent features of the augmented feature sequence.

[0069] The classification output module is used to concatenate and fuse the encoded features and output the classification result.

[0070] Regarding model training: Standardized EEG data are divided into training and testing sets, which are then input into the ECA-MTCNet decoding model for training. The training process is optimized by adjusting the model parameters.

[0071] Step 3: EEG Decoding and Classification: Input the EEG signal of the Chinese tone to be decoded into the trained ECA-MTCNet decoding model, and output the corresponding tone classification result.

[0072] In one implementation example, step one involves acquiring the subject's Mandarin four-tone evoked EEG signals according to a modified passive Oddball EEG evoked paradigm. The parameters of the modified passive Oddball EEG evoked paradigm are set as follows: the ratio of standard to biased stimuli is 7:3; each experimental block contains n=100 stimuli; the stimulation duration is T1=500ms; and the stimulation interval is T2=1000ms. A wireless EEG acquisition system is used for signal acquisition, with a sampling frequency of F1=1000Hz, and electrode placement follows the international 10-20 system standard. Existing commercial acquisition equipment is used for EEG data acquisition. In this embodiment, the NeuSen W64 wireless EEG acquisition system from Boruikang is specifically used, which includes an intelligent synchronization center, a wireless EEG amplifier, a multi-parameter synchronizer, and a wet electrode EEG cap to achieve high-quality synchronous signal acquisition.

[0073] Regarding data preprocessing, the raw EEG data was filtered, rereferenced, segmented, baseline corrected, and artifact removed. Specifically, the process involved: using a Butterworth bandpass filter to filter the raw EEG data; converting the filtered data into a whole-brain average reference to eliminate potential bias introduced by a single reference electrode and highlight the true spatial distribution characteristics of the EEG signals; segmenting the EEG data according to a time window from a=100ms before stimulus presentation to b=2000ms after presentation to form each sample in the dataset; using the data from a=100ms before stimulus presentation as the baseline for baseline correction; and using Independent Component Analysis (ICA) to remove electrooculography (EOG) and electromyography (EMG) artifacts.

[0074] The above preprocessing steps aim to preserve the complete temporal information of tone perception and remove noise. The resulting standardized EEG data will be used as a tensor input to the subsequent ECA-MTCNet model for the extraction of deep spatiotemporal features.

[0075] The reason for segmentation: Each EEG sample is segmented from a=100 ms before the arrival of the stimulus time corresponding to the label to b=2000 ms after the arrival of the stimulus time, forming an EEG sample corresponding to the label. The core role of whole-brain average reference (CAR) is to eliminate the systematic bias introduced by a single physical reference electrode (such as Cz or mastoid) during acquisition and to suppress common-mode noise. This is crucial for the deep learning model in this example because it can significantly improve the spatial signal-to-noise ratio of the signal, ensuring that subsequent spatial convolutional layers can accurately extract the real brain region topological features, rather than artifacts from the reference electrode.

[0076] Regarding correction: Time dimension: The set segmented window (e.g., -100ms to 2000ms) is much longer than the single stimulus duration T1=500ms, which can fully cover the entire process from the early auditory evoked response (N1-P2) to the late perceptual response (MMN, P300), avoiding the loss of late perceptual features due to the window being too short.

[0077] Frequency dimension: The low-frequency cutoff frequency of 0.1Hz preserves the slow wave components associated with tone profile tracking.

[0078] In one implementation example, the ECA-MTCNet decoding model is constructed in step two. The front-end feature extraction module consists of a temporal convolutional layer, a channel depth convolutional layer, a spatial convolutional layer, an efficient channel attention ECA module, and an average pooling layer.

[0079] The temporal convolutional layer takes a standardized EEG tensor as input and performs convolution operations along the time dimension using a large one-dimensional convolution kernel (e.g., 1×64). By simulating the function of a bandpass filter, it can extract time-frequency features of different frequency bands (e.g., Alpha and Beta waves) from the raw EEG signal.

[0080] The standardized EEG tensor is: ;

[0081] For the number of channels, To determine the number of time points, a large one-dimensional convolution kernel is used to perform convolution along the time dimension. R Let represent the set of real numbers, indicating that each element in the matrix is ​​a real number. To extract time-frequency features of specific frequency bands from raw EEG signals, a temporal convolutional layer was designed. Let Represents the standardized EEG tensor XThe voltage amplitude located in channel c at time t+i is calculated using the following formula:

[0082] Formula for calculating temporal convolutional layers:

[0083] in, , representing the filtering characteristics of the output of the f-th filter on the c-th channel. The temporal convolution kernel length, such as 64. This is the bias term. Time filter weights. This represents the i-th weight coefficient of the f-th temporal convolution kernel. In signal processing, it acts as a digital bandpass filter to extract specific frequency band features from the raw EEG signal. In this embodiment, the input tensor... X Padding of the same size was applied so that the time dimension index t of the convolution output could cover... The scope should be maintained to ensure consistency in the time scale of the feature maps. Represents the standardized EEG tensor X The voltage amplitude located in the c-th channel at time t+i.

[0084] Channel-level depthwise convolutional layers are used to perform depthwise separable convolution on the output of the previous layer. The process involves independently convolving each feature map channel without inter-channel feature fusion. This significantly reduces the number of model parameters and computational complexity; it also decouples spatial and temporal features, preventing overfitting during training on small sample data. Let the depth multiplier be M, and the kernel length be... The output of the channel depth convolutional layer is defined as follows:

[0085]

[0086] Wherein, the kernel length is , This represents the output value from the previous temporal convolutional layer: the response of the f-th feature map at time point t+τ in channel c. ) represents the weights of the m-th depthwise convolutional kernel corresponding to channel c at delay τ. This is the bias term; the convolution stride and padding can be set according to the network structure, for example, a stride of 1 and padding of the same size. Here, t is the index of the time dimension. L represents the total time length of the input EEG feature sequence. The key point is that since the above operations are performed independently for each channel c and do not introduce linear combinations between channels, this layer does not perform inter-channel feature fusion. This reduces the number of parameters and computational cost while lowering the risk of overfitting, and provides more discriminative channel-level features for subsequent cross-channel / spatial fusion.

[0087] Spatial convolutional layer: Convolution is performed along the electrode channel dimension to fuse whole-brain spatial information. The processing involves weighted summation of information from all electrode channels, learning the spatial topological relationships of different electrode locations, and extracting spatial distribution features of brain regions related to tone perception. Let the number of spatial convolution output channels be S(…). Then, the output of the spatial fusion convolutional layer can be defined as:

[0088]

[0089] in, This is the output of the deep convolutional layer. This represents the spatial contribution weight of electrode channel c under frequency band f, depth mode m, and the s-th spatial fusion channel. Physically, the model learns the spatial topological relationships of different brain regions through this parameter, such as identifying the activation level of frontal or occipital electrodes in a specific task. This is the bias term. The formula, through a summation operation, achieves a weighted summation over all input channels c, thereby mapping the electrode spatial distribution into high-dimensional abstract spatial features. This means that the output of this layer no longer includes the electrode channel dimension c, but retains the temporal variation characteristics of the f-th frequency band and the m-th depth mode by fusing information from all electrodes.

[0090] Average pooling layer: Used to downsample the feature map. Its process involves calculating the average value within a local window as the output. This layer reduces data dimensionality while preserving key feature information, decreasing subsequent computation and granting the model a degree of time-translation invariance, making it insensitive to minor temporal deviations in tone.

[0091] Average pooling layer formula: , ;

[0092] This corresponds to the feature sequence output from the previous level. Here, f represents the frequency band, m represents the depth mode, and s represents the spatial fusion channel. Represents the index of the time point involved in the current pooling calculation. k: represents the time offset index within the pooling window; P and : Represents the pooling window size and stride, respectively. : Represents the new time index after downsampling; : Downsampled output features. This feature retains the physical feature attributes extracted from the previous layer, including frequency f, depth pattern m, and spatial distribution channels s, and is compressed only on the time axis.

[0093] The efficient channel attention (ECA) module is embedded between the spatial convolutional layer and the average pooling layer to perform channel weight calibration at the feature extraction source. The working process of the efficient channel attention (ECA) module is as follows:

[0094] (1) Perform global average pooling on the feature map output by the spatial convolutional layer to obtain the channel descriptor;

[0095] Regarding the generation of channel descriptors:

[0096] First, obtain the feature map output by the spatial convolutional layer. Because the spatial convolutional layer performs S spatial dimension fusion processing on the M depth pattern features output from the previous layer, the feature map has the following characteristics in each frequency band f: Each feature channel has a logical dimension represented as: , where L is the total length of the time series as mentioned above.

[0097] Next, global average pooling is performed on the feature map. Specifically, the average value of the data for each feature channel is calculated along the time dimension, using the following formula:

[0098] in, This represents the aggregated features across f frequency bands, m deep modes, and the s-th spatial fusion channel. : The feature map output by the spatial convolutional layer. L: Represents the length of the feature map in the time dimension; This indicates that global average pooling is used to aggregate along the entire time axis t, thereby eliminating the time dimension and extracting the global distribution information of each channel.

[0099] After this operation, the original spatiotemporal feature map is compressed into a dimension of [dimensional value missing]. The vector Z. This vector Z is the channel descriptor, which eliminates the fluctuations in the time dimension and only retains the global importance information of each feature channel, which combines the depth pattern and spatial location.

[0100] Channel descriptor definition: It is a channel with one dimension. eigenvectors, where The total number of feature channels output by the spatial convolutional layer is composed of M deep pattern features and S spatial fusion channels; this descriptor is a multi-dimensional spatiotemporal feature map. The global aggregation over the time dimension, each value represents the average response intensity of the corresponding feature channel (a combination of a specific pattern and a specific spatial distribution) within the entire time window.

[0101] (2) Based on the number of feature channels The size k1 of the one-dimensional convolution kernel is determined by an adaptive algorithm.

[0102] (3) A one-dimensional convolution of size k1 is used to perform local cross-channel interactive computation on the channel descriptors; the formula for calculating k1 is as follows:

[0103]

[0104] in, This represents the bias of a linear mapping. For hyperparameters, Indicates the kernel size With the number of feature channels Nonlinear mapping between them.

[0105] 4) Generate channel attention weights using the Sigmoid activation function, and then perform weighted fusion with the feature map output by the spatial convolutional layer to obtain the calibrated feature map.

[0106] After determining the optimal interaction range k1, the ECA module performs local cross-channel interaction using one-dimensional convolution. It uses a one-dimensional convolution of size k1 to operate on the channel descriptor Z, calculates the interaction information between channels, and maps the convolution output to normalized weight coefficients in the (0, 1) interval using the Sigmoid activation function. The specific formula for generating the weights is shown below:

[0107]

[0108] in, This represents a one-dimensional convolution operation with a kernel size of k1, which aggregates information from each channel with its k1 neighboring channels. The weights... It acts as a feature filter. Addressing the issues of low signal-to-noise ratio and high noise in irrelevant channels in EEG signals, this weighting mechanism automatically identifies feature channels containing key tone information and assigns them high weights (close to 1), while suppressing irrelevant channels containing electrooculography (EOG) or electromyography (EMG) noise and assigning them weights close to 0. This "recalibration" process significantly improves the spatial signal-to-noise ratio of the signal at the feature extraction source.

[0109] The weighted fusion here refers to channel-wise multiplication (Hadamard Product), which generates weights. Feature maps output by spatial convolutional layers Perform channel-by-channel weighted fusion (Scale).

[0110] The calculation formula is as follows:

[0111]

[0112] The attention weight vector generated earlier In this context, the weight values ​​correspond to the specific feature channels (frequency band f, depth mode m, spatial channel s). This scalar is broadcast along the time axis t, scaling the entire feature sequence. : The calibrated enhanced feature map. This step achieves adaptive enhancement of key EEG frequency bands and spatial patterns (weights approaching 1) and suppresses noise channels (weights approaching 0).

[0113] Finally, the calibrated feature map is obtained. Then enter the next level.

[0114] In one implementation example, the dual-attention co-coding module includes a multi-head self-attention submodule and a temporal convolutional TCN submodule.

[0115] Multi-head self-attention submodule (number of attention heads: head, key vector dimension: d) k The working process is as follows: the sub-feature sequences output by the sliding window are mapped to query vector Q, key vector K and value vector V respectively; the attention weight of each sub-feature sequence is calculated by scaling dot product attention.

[0116] The sub-feature sequences here correspond to the dynamic response of EEG signals at different time steps. The calculation of Q, K, and V essentially analyzes the correlation between tone-evoked EEG potentials at different times.

[0117] Specifically, the sub-feature sequence is actually a window slice of the feature map output by the previous layer sliding on the time axis.

[0118] The sliding window size is set to J, and in this embodiment, J=2. Each sub-feature sequence is a truncated portion of the complete feature map, with dimensions of [missing value]. The local time series matrix is ​​then projected into Q, K, V vectors to compute local attention.

[0119] Perceptual decoding of Mandarin tone EEG relies heavily on the brain's tracking of the complete auditory process, including the "fall-rise" pitch change of the third tone and the corresponding "loud-weak-loud" volume change. Traditional convolution can only extract local features, while multi-head self-attention mechanisms can capture multi-temporal temporal dependencies, connecting EEG features scattered at different time points to reconstruct complete perceptual features of tone EEG signals, significantly improving the ability to distinguish complex tone EEG (such as the second tone and the third tone).

[0120] This process achieves a deep integration with the time-varying characteristics of EEG signals: by analyzing the interrelationships between EEG features at different times, i.e., multi-temporal dependence, the model can transcend the limitations of local convolutional receptive fields and capture the unique dynamic evolution patterns of Chinese tone EEG from a global perspective, such as distinguishing between the potential change patterns of "full rise" and "first fall then rise", thereby accurately identifying tone categories.

[0121] The Temporal Convolutional (TCN) submodule adopts a dilated causal convolutional structure (with n1 residual blocks) and solves the gradient vanishing problem in deep networks through residual connections.

[0122] The dilated causal convolutional structure receives sub-feature sequences generated by a sliding window. The processing mechanism employs a dilated causal convolution structure. The network contains four stacked residual blocks, with the dilation factor d increasing exponentially with the number of layers (1, 2, 4, 8). This structure utilizes interval sampling to cover the entire tone period (e.g., several hundred milliseconds) without increasing the number of parameters, strictly adhering to temporal causal logic. Output flow: The deep temporal features encoded by the TCN are transmitted to the classification output module. In this module, the features from all sub-windows are concatenated and fused, and finally, the tone category probability is output through a softmax layer.

[0123] The perceptual decoding of Mandarin tone-evoked EEG relies on relatively complete, full-time dynamic potential changes, such as tone perception-following responses within hundreds of milliseconds. The dilated convolution structure of TCN can exponentially expand the receptive field without increasing the number of parameters, enabling it to cover the complete tone cycle in the EEG signal. Causal convolution ensures that the model's prediction at time t1 depends only on historical EEG data prior to time t1, conforming to the temporal causal logic of human brain processing auditory information, thus accurately capturing the multi-temporal evolution of tone perception.

[0124] Specifically, considering the characteristics of long duration and wide feature span of Chinese tone EEG signals, dilated convolution is used to exponentially expand the temporal receptive field of the model, enabling it to "see" the complete contour of tone evoked potentials. At the same time, causal convolution ensures that feature extraction strictly follows the physical laws of EEG signals over time, that is, the current output is only affected by historical information, preventing the leakage of future information, thereby achieving accurate encoding of multi-temporal spatiotemporal dependent features of tone.

[0125] In one implementation example, the parameters for model training are set as follows: the Adam optimizer is used, the initial learning rate is c, the batch size is batch, the loss function is the classification cross-entropy loss function with L2 regularization, and the regularization coefficient is e; the number of training epochs is h, and an early stopping mechanism is used during training. If the validation set loss does not decrease for h1 consecutive epochs, training is terminated, and the optimal model weights are saved. When training ends, the weights of the trained model are the optimal parameters of the subject's tone EEG signal decoding model (ECA-MTCNet), and the optimal accuracy of the tone EEG signal is output.

[0126] More detailed examples:

[0127] Step (1): Construct a database of four tones of the Chinese / a / syllable evoked EEG.

[0128] Three right-handed subjects with normal hearing, aged 23-31 years, all of whom spoke Mandarin Chinese as their native language, were selected. A modified passive Oddball EEG evoked paradigm was used, such as... Figure 1 As shown, the standard stimulus was a pure tone at F=440Hz, and the biased stimulus was the four tones of the Mandarin vowel / a / : first tone, second tone, third tone, and fourth tone. The ratio of standard stimulus to biased stimulus was 7:3. Each experiment contained 5 blocks, each block had n=100 stimuli, the stimulus duration T1=500ms, and the stimulus interval T2=1000ms. EEG signals were acquired using the NeuSen W64 wireless EEG acquisition system with a sampling frequency F1=1000Hz, and electrode placement followed international standard 10... The system used a 20-system standard, selecting 59 EEG channels, 4 EOG channels for artifact monitoring, and 1 ECG channel for simultaneous acquisition. The experiment was conducted in a soundproof room, with subjects wearing in-ear headphones and watching a silent film to maintain passive hearing.

[0129] Step (2): EEG data preprocessing.

[0130] Preprocessing based on the MATLAB platform and EEGLAB toolbox:

[0131] 2.1) Filtering: A 4th-order Butterworth bandpass filter with a frequency range of 0.1~100Hz is used to remove low-frequency drift and high-frequency electromyographic noise;

[0132] 2.2) Rereference: Convert the data into a whole-brain average reference to eliminate systematic bias from a single reference electrode;

[0133] 2.3) Segmentation: The EEG segments were segmented according to the time window from a=100ms before stimulus presentation to b=2000ms after presentation, resulting in EEG segments of 2100ms in length.

[0134] 2.4) Baseline correction: Baseline correction was performed on each EEG segment using the data from a=100ms before stimulation as the baseline;

[0135] 2.5) Artifact Removal: Independent Component Analysis (ICA) algorithm was used to identify and remove artifacts from electrooculography (EOG) and electromyography (EMG) data to obtain standardized EEG data with dimensions of (600, 59, 2100), corresponding to the number of samples × number of channels × number of sampling points.

[0136] Step (3): Construct ECA The MTCNet decoding model consists of a front-end feature extraction module, a sliding window augmentation module, a spatiotemporal attention co-coding module, and a classification output module. The specific structure is as follows: Figure 2 As shown, the working process is as follows:

[0137] 3.1) Front-end feature extraction module: This module consists of a temporal convolutional layer, a channel depthwise convolutional layer, a spatial convolutional layer, an ECA module, and an average pooling layer. The ECA module is embedded between the spatial convolutional layer and the average pooling layer. Based on the number of channels (59), it uses an adaptive algorithm to determine the kernel size k1=3, generates channel attention weights, and calibrates the feature map. The average pooling layer reduces the dimensionality of the feature map.

[0138] The temporal convolutional layer uses f=64 convolutional kernels with a length of L. t A 64-dimensional convolutional kernel. This layer acts as a learnable filter bank, using a large convolutional kernel to perform frequency domain filtering on the raw EEG signal to extract time-frequency amplitude features at different frequency bands. It can automatically discover and lock specific frequency components strongly correlated with Chinese tone perception, overcoming the limitations of manually preset filtering frequency bands.

[0139] The channel-wise deep convolutional layer has a depth multiplier of M=4. This layer performs four independent deep convolution operations on each temporal feature map output from the previous layer, thereby expanding the feature dimension by four times to extract richer and more refined spectral features for each base frequency band. Utilizing a depth-wise convolution structure achieves feature decoupling, significantly reducing computational cost and the number of parameters while maintaining the breadth of feature extraction. This effectively solves the problem of overfitting in traditional convolutional networks on small-sample EEG data.

[0140] The number of output channels in the spatial convolutional layer is set to S=32. This layer extracts the spatial topological patterns of EEG signals by weighting and combining all electrode signals through cross-channel convolution operations. It acts as a learnable spatial filter, effectively addressing the volumetric conduction effect of EEG signals, removing redundant information between electrodes, and enhancing the feature representation of key brain regions.

[0141] The average pooling layer uses a pooling kernel size of P=(20,1). This layer downsamples the feature map by calculating the average value within a local time window. While compressing data dimensionality and reducing computational complexity, it also imparts time-shift invariance to the input signal through smoothing operations, improving robustness to different speech rates or latency jitter.

[0142] 3.2) Sliding window augmentation module: The feature sequence output from the front end is divided into 12 overlapping sub-windows, each with a length of 2, to augment the data and expand the effective training samples.

[0143] 3.3) Dual Attention Co-coding Module: This module includes a multi-head self-attention submodule and a TCN submodule. The TCN submodule uses n1=4 residual blocks with dilation factors of 1, 2, 4, and 8 respectively, and a convolution kernel size of 4. It expands the receptive field through dilated causal convolution to solve the problem of long sequence dependencies.

[0144] In this implementation example, the multi-head self-attention submodule is configured with 2 attention heads and the key vector dimension is d. k =8. This module directly captures the long-distance correlation between any two time points in the sequence under different representation dimensions by projecting the input sequence onto two independent feature subspaces and computing the self-attention matrix in parallel across the entire time domain.

[0145] The above methods overcome the physical limitations of the local receptive field of convolutional neural networks, and can effectively capture long-span temporal dependencies in Chinese tone-induced EEG signals, such as the potential changes in the beginning and end stages of tone association, and prevent the loss of key multi-temporal features.

[0146] In this implementation example, the receptive field is expanded by dilated causal convolution. Specifically, as the number of network layers increases, the dilation factor grows exponentially (1, 2, 4, 8), allowing the convolutional kernel to cover a longer time span through interval sampling without increasing the number of parameters.

[0147] The above structure can efficiently construct a global receptive field that is sufficient to cover the complete tone evoked potential time history (>500ms), thereby solving the problem that traditional convolutional networks cannot capture long-distance temporal dependencies (such as the potential correlation between the tone start and end stages) due to the limited receptive field, while avoiding the defect of recurrent neural networks being difficult to train in parallel.

[0148] 3.4) Classification output module: The encoded features of each sub-window are concatenated and output as the probability distribution of four tones through a fully connected layer and a Softmax activation function.

[0149] Step (4) Model Training: The standardized EEG data were divided into training and test sets according to 5-fold cross-validation, and the model was trained using the TensorFlow 2.4 framework. The Adam optimizer was set with an initial learning rate of c=0.001, a batch size of batch=40, and the loss function was the classification cross-entropy loss function with L2 regularization, with a regularization coefficient of e=0.009. The training epochs were h=100, and an early stopping mechanism was adopted. If the validation set loss did not decrease for h1=10 consecutive epochs, the training was terminated, and the optimal model weights were saved.

[0150] Step (5) EEG decoding and classification: Input the EEG signal of the Chinese tone to be decoded into the trained ECA. The MTCNet model outputs the tone category corresponding to the signal. Experimental results are shown in Table 1. The model achieves an average classification accuracy of 83.22%, a 25.68% improvement over the traditional EEGNet model. The model exhibits significantly improved decoding performance and good robustness.

[0151] Table 1 Classification results of EEGNet and ECA-MTCNet

[0152] Subjects Subject 1 Subject 2 Subject 3 average EEGNet 62.38% 63.69% 46.55% 57.54% ECA-MTCNet 84.17% 83.50% 82.00% 83.22%

[0153] This embodiment proposes a tone-based EEG decoding method and system based on a spatiotemporal attention collaborative mechanism, effectively solving the problem of insufficient decoding accuracy caused by inadequate extraction of spatiotemporal features from traditional models of Chinese tone-evoked EEG signals. A standardized Chinese four-tone evoked EEG database is independently constructed, providing high-quality data support for the decoding task. The innovatively designed ECA-MTCNet model uses an efficient channel attention (ECA) module at the front end to achieve adaptive calibration of channel weights, accurately suppressing irrelevant channel noise interference. At the back end, it combines multi-head self-attention and a temporal convolutional network (TCN) to capture multi-temporal spatiotemporal dependent features, forming a dual collaborative decoding mechanism of spatial denoising and temporal aggregation.

[0154] Experimental verification shows that the average classification accuracy of this method is significantly better than that of existing mainstream models. The technical solution provided in this embodiment achieves high-precision and objective decoding of Chinese tone EEG signals, providing an effective auditory assessment method for infants, children, language-disordered individuals, and other groups who cannot cooperate with subjective hearing observation. It fills the technical gap in the field of objective auditory assessment of Chinese tone and has important application value and promotion prospects in the fields of clinical hearing diagnosis and auditory rehabilitation monitoring.

[0155] Example 2

[0156] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0157] Example 3

[0158] The purpose of this embodiment is to provide a computer-readable storage medium.

[0159] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.

[0160] Example 4

[0161] The purpose of this embodiment is to provide a tone-based EEG decoding system based on a spatiotemporal attention coordination mechanism, including:

[0162] The EEG data acquisition and preprocessing module is configured to: acquire the subject's EEG signals to obtain raw EEG data, and preprocess the raw EEG data to obtain standardized EEG data;

[0163] The ECA-MTCNet decoding model building module is configured to: divide standardized EEG data into training and testing sets, input them into the ECA-MTCNet decoding model for training, optimize the training process by adjusting the model parameters, and obtain the trained ECA-MTCNet decoding model.

[0164] The ECA-MTCNet decoding model includes a front-end feature extraction module, a sliding window augmentation module, a dual attention co-coding module, and a classification output module.

[0165] The front-end feature extraction module is used to process the input standardized EEG data, extract the primary spatiotemporal features of the EEG signal, and realize adaptive calibration of channel weights.

[0166] The sliding window augmentation module is used to segment the feature sequence output by the front-end feature extraction module into multiple overlapping sub-windows to achieve data augmentation.

[0167] The dual attention co-coding module is used to capture multi-temporal spatiotemporal dependent features of EEG signals;

[0168] The classification output module is used to concatenate and fuse the encoded features and output the classification result;

[0169] The tone classification module is configured to input the EEG signals of Chinese tones to be decoded into the trained ECA-MTCNet decoding model and output the corresponding tone classification results.

[0170] This embodiment's sub-solution, through the design and improvement of the passive Oddball EEG evoked paradigm, constructs a standardized Chinese four-tone evoked EEG database, providing high-quality data support for Chinese tone EEG decoding and verifying the data's separability. The ECA constructed by this embodiment's sub-solution... The MTCNet model introduces an efficient channel attention (ECA) module, which adaptively calibrates channel weights at the feature extraction source, effectively suppressing noise interference from irrelevant channels and improving feature quality. Combined with the dual attention synergy mechanism of multi-head self-attention and temporal convolutional neural networks (TCN), it can accurately capture multi-temporal spatiotemporal dependent features of EEG signals, significantly improving decoding accuracy. The methods and systems described in this embodiment are suitable for objective auditory assessment of people with hearing impairments, and have significant clinical application value, especially for infants and young children who cannot cooperate with subjective hearing observation, and for people with cognitive impairments.

[0171] Example 5

[0172] The purpose of this embodiment is to provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the methods and functions involved in any of the above embodiments.

[0173] The steps and methods involved in the apparatus of the above embodiments correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0174] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0175] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A tone-based EEG decoding method based on a spatiotemporal attention coordination mechanism, characterized by: include: EEG signals were collected from the subjects to obtain raw EEG data, and the raw EEG data was preprocessed to obtain standardized EEG data. Standardized EEG data were divided into training and testing sets and input into the ECA-MTCNet decoding model for training. The training process was optimized by adjusting the model parameters to obtain the trained ECA-MTCNet decoding model. The ECA-MTCNet decoding model includes a front-end feature extraction module, a sliding window augmentation module, a dual attention co-coding module, and a classification output module. The front-end feature extraction module is used to process the input standardized EEG data, extract the primary spatiotemporal features of the EEG signal, and realize adaptive calibration of channel weights. The front-end feature extraction module includes a temporal convolutional layer, a channel depth convolutional layer, a spatial convolutional layer, an efficient channel attention (ECA) module, and an average pooling layer. The temporal convolutional layer takes a standardized EEG tensor as input and performs convolution operations along the time dimension using a one-dimensional convolution kernel. By simulating the function of a bandpass filter, it can extract time-frequency features of different frequency bands from the raw EEG signal. Formula for calculating temporal convolutional layers: in, , representing the filtering characteristics of the output of the f-th filter on the c-th channel. Temporal convolution kernel length; temporal filter weights This represents the i-th weight coefficient of the f-th temporal convolution kernel; in signal processing, it acts as a digital bandpass filter to extract specific frequency band features from the raw EEG signal, applying the input tensor... X Padding of the same size was applied so that the time dimension index t of the convolution output could cover... Range, maintaining consistency in the time scale of feature maps; Represents the standardized EEG tensor X The voltage amplitude located in the c-th channel at time t+i; Channel depthwise convolutional layers are used to perform depthwise separable convolutions on the output of the previous layer; The output of a channel depth convolutional layer is defined as follows: Wherein, the kernel length is , This represents the output value from the previous temporal convolutional layer: the response of the f-th feature map at time point t+τ in channel c; ) represents the weights of the m-th depthwise convolutional kernel corresponding to channel c at delay τ. This is the bias term; the convolution stride and padding can be set according to the network structure; where t is the index of the time dimension. L represents the total length of the input EEG feature sequence; the key point is that since the above operation is performed independently for each channel c and does not introduce linear combination between channels, this layer does not perform feature fusion between channels, which can reduce the number of parameters and computation while reducing the risk of overfitting, and provide more discriminative channel-level features for subsequent cross-channel / spatial fusion. Spatial convolutional layer: Convolution is performed along the electrode channel dimension to fuse spatial information from the whole brain; The output of a spatially fused convolutional layer can be defined as: in, This is the output of the deep convolutional layer. This represents the spatial contribution weight of electrode channel c under frequency band f, depth mode m, and the s-th spatial fusion channel. Physically, the model learns the spatial topological relationships of different brain regions through this spatial contribution weight. This is the bias term; the formula achieves a weighted summation of all input channels c through a summation operation, thereby mapping the electrode spatial distribution into high-dimensional abstract spatial features; This means that the output of this layer no longer contains the electrode channel dimension c, but retains the temporal variation characteristics of the f-th frequency band and the m-th depth mode by fusing the information of all electrodes. The efficient channel attention (ECA) module is embedded between the spatial convolutional layer and the average pooling layer to perform channel weight calibration at the source of feature extraction; The high-efficiency channel attention (ECA) module processes the input data, including: Global average pooling is performed on the feature map output by the spatial convolutional layer to obtain the channel descriptor; Based on the number of channels, the size k1 of the one-dimensional convolution kernel is determined using an adaptive algorithm; A one-dimensional convolution of size k1 is used to perform local cross-channel interactive computation on the channel descriptor; Channel attention weights are generated by an activation function, and the channel attention weights are then weighted and fused with the feature map output by the spatial convolutional layer to obtain the calibrated feature map. Average pooling layer: used to downsample the feature map; Average pooling layer formula: , ; : Corresponds to the feature sequence output from the previous level; where f represents the frequency band, m represents the depth mode, and s represents the spatial fusion channel. P represents the index of the time point participating in the current pooling calculation; k represents the time offset index within the pooling window; P and : Represents the pooling window size and stride, respectively. : Represents the new time index after downsampling; : Downsampled output features; these features retain the physical features extracted from the previous layer, including frequency f, depth pattern m, and spatial distribution channels s, and are compressed only on the time axis; The sliding window augmentation module is used to segment the feature sequence output by the front-end feature extraction module into multiple overlapping sub-windows to achieve data augmentation. The dual attention co-coding module is used to capture multi-temporal spatiotemporal dependent features of EEG signals; The dual-attention co-coding module includes a multi-head self-attention sub-module and a temporal convolutional TCN sub-module; The multi-head self-attention submodule is used to map the sub-feature sequences output by the sliding window to query vector Q, key vector K, and value vector V, respectively; and calculates the attention weights of each sub-feature sequence by scaling dot product attention. The temporal convolutional TCN submodule adopts a dilated causal convolutional structure; The classification output module is used to concatenate and fuse the encoded features and output the classification result; The EEG signal of Chinese tone to be decoded is input into the trained ECA-MTCNet decoding model, and the corresponding tone classification result is output.

2. The tone EEG decoding method based on spatiotemporal attention coordination mechanism as described in claim 1, characterized in that, When collecting the subjects' EEG signals, a passive Oddball auditory evoked paradigm was designed, using a pure tone of F Hz as the standard stimulus and four tones of a certain Chinese vowel monosyllabic syllable (level, rising, falling-rising, and falling-falling tone) as the deviation stimulus, while simultaneously collecting the subjects' EEG signals.

3. The tone EEG decoding method based on spatiotemporal attention coordination mechanism as described in claim 1, characterized in that, Preprocessing of the raw EEG data includes filtering the raw EEG data using a Butterworth bandpass filter. The filtered data was converted into a whole-brain average reference; it was then segmented and labeled according to a time window from a milliseconds before stimulus presentation to b milliseconds after presentation. Baseline correction was performed using the data from a milliseconds before stimulation as the baseline. Independent component analysis algorithm was used to remove artifacts from electrooculography (EOG) and electromyography (EMG).

4. A tone-based EEG decoding system based on a spatiotemporal attention coordination mechanism, characterized by: The tone EEG decoding method based on spatiotemporal attention coordination mechanism as described in any one of claims 1-3 includes: The EEG data acquisition and preprocessing module is configured to: acquire the subject's EEG signals to obtain raw EEG data, and preprocess the raw EEG data to obtain standardized EEG data; The ECA-MTCNet decoding model building module is configured to: divide standardized EEG data into training and testing sets, input them into the ECA-MTCNet decoding model for training, optimize the training process by adjusting the model parameters, and obtain the trained ECA-MTCNet decoding model. The ECA-MTCNet decoding model includes a front-end feature extraction module, a sliding window augmentation module, a dual attention co-coding module, and a classification output module. The front-end feature extraction module is used to process the input standardized EEG data, extract the primary spatiotemporal features of the EEG signal, and realize adaptive calibration of channel weights. The sliding window augmentation module is used to segment the feature sequence output by the front-end feature extraction module into multiple overlapping sub-windows to achieve data augmentation. The dual attention co-coding module is used to capture multi-temporal spatiotemporal dependent features of EEG signals; The classification output module is used to concatenate and fuse the encoded features and output the classification result; The tone classification module is configured to input the EEG signals of Chinese tones to be decoded into the trained ECA-MTCNet decoding model and output the corresponding tone classification results.

5. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 3.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method described in any one of claims 1-3.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it performs the steps of the method described in any one of claims 1-3 above.

Images