Auditory attention decoding method and system based on blink sequence
By combining EEG and EEG signals with a multimodal deep convolutional neural network and utilizing blink sequences and speech envelope features, the problem of low decoding accuracy and high latency in existing technologies has been solved, achieving low-latency and high-precision auditory attention decoding.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-06
- Publication Date
- 2026-03-13
AI Technical Summary
Existing auditory attention decoding methods suffer from low decoding accuracy, high latency, and high computational cost in complex auditory environments, and fail to effectively utilize the multimodal information of EEG and EEG signals.
A multimodal deep convolutional neural network is used to combine EEG and EEG signals to achieve auditory attention decoding through blink sequences and speech envelope features. This includes data preprocessing, blink time point recognition, speech envelope extraction, and cross-modal similarity calculation of neural behavior representation.
It significantly improves decoding accuracy, reduces latency, maintains stable performance, and enables low-latency, high-precision auditory attention object recognition.
Smart Images
Figure CN121662036A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of brain-computer interface technology, specifically relating to an auditory attention decoding method and system based on blink sequences. Background Technology
[0002] Normal-hearing (NH) listeners exhibit exceptional selective attention to target speech in complex auditory environments, such as the classic "cocktail party" scenario, enabling them to precisely focus their attention amidst competing sounds. In contrast, hearing-impaired (HI) listeners face significant challenges in speech perception in such environments, even with the use of hearing aids (HA). A key factor contributing to these difficulties is the limitation of current hearing aid technology, which cannot effectively identify the target speech stream from multiple candidate speech streams.
[0003] In auditory selective attention tasks, listeners' neural responses show enhanced temporal modulation of the envelope of the speech being focused on, while modulation of the envelope of the speech not being focused on is weaker. Based on this neurophysiological foundation, researchers have developed methods for identifying target speech through non-invasive neural recordings, such as magnetoencephalography (MEG) and electroencephalography (EEG). These methods aim to reconstruct the speech envelope from neural responses using a neural decoder. This framework operates by quantifying the correlation between the decoded speech envelope and the original speech envelope (target speech and interfering speech); a statistically significant correlation indicates the speech being focused on by the listener—a process known as auditory attention decoding (AAD). Although AAD methods, especially those employing deep neural networks (DNNs), have shown high decoding performance on some publicly available EEG datasets, they still face significant challenges when applied to real-world scenarios. They often struggle to achieve robust performance across diverse datasets. Developing effective AAD techniques remains a complex and ongoing challenge.
[0004] In complex auditory environments, listeners continuously process multimodal stimuli (e.g., auditory and visual) while simultaneously generating multimodal responses, such as neurophysiological activities (e.g., cortical oscillations) and behavioral patterns (e.g., blinking). Current AAD methods rely solely on single-modal neural data, neglecting the potential utility of behavioral responses. In contrast, in the relevant field of brain-computer interfaces (BCIs), behavioral responses have been systematically integrated into discriminative biomarkers, with significant progress. For example, the multimodal fusion of neural and behavioral signals has proven effective in emotion recognition, fatigue detection, and sleep staging. Clearly, leveraging multimodal information to enhance AAD could be an effective solution; however, research on how to select and utilize additional information in this context remains very limited.
[0005] Eye-movement behavior shows particular potential as an attentional biomarker in speech perception. Blinking is consistent with language structure and synchronizes with speech breaks during narrative comprehension. Holtze et al. further demonstrated that blinking can track target speech when listening to competing speech. They found that NH listeners were more likely to blink during pauses in target speech than during pauses in masked speech. Therefore, it can be hypothesized that blinking is more likely to occur during pauses in target speech than during pauses in masked speech; in other words, the temporal sequence of blinks shows a stronger consistency with the pause sequence of target speech. This suggests that the consistency between blinking and speech can be used for auditory attention decoding. Summary of the Invention
[0006] The purpose of this invention is to propose an auditory attention decoding method and system based on blink sequences to solve the problems of low decoding accuracy, large latency and high computational load in existing auditory attention decoding. It aims to achieve low latency, high accuracy and robust auditory attention object recognition by using multimodal joint input of EEG and EEG signals.
[0007] To achieve the above objectives, the present invention adopts the following technical solution.
[0008] An auditory attention decoding method based on blink sequences includes the following steps: 1) Collect and preprocess EEG and EEG signals to obtain clean EEG data; 2) Identify blink time points in the vertical electrooculogram (EOG) signal and convert the identified blink time points into a sparse binary blink time sequence; 3) Perform band filtering and Hilbert transform on the speech signal to be tested to obtain the analytical signal; perform nonlinear amplitude compression and low-pass filtering on the analytical signal, calculate the speech envelope signal of each frequency band, and average the speech envelope signals of each frequency band to obtain the overall speech envelope signal. 4) Input clean EEG data, sparse binarized blink time series and overall speech envelope signal into a multimodal deep convolutional neural network to generate neural behavior representations and stimulus representations corresponding to each speech stream. Calculate brain-speech cross-modal similarity based on neural behavior representations and stimulus representations, map it to attention probability through a fully connected layer, and train the network to optimize parameters to achieve automatic recognition of listener attention speech streams.
[0009] Further, the data preprocessing in step 1) includes: downsampling the EEG and EOS signals, performing 2-8 Hz offline bandpass filtering on the EEG signals, and using independent component analysis to remove artifact signals.
[0010] Further, the identification of blink time points in step 2) includes: detecting the local trough minimum value of the vertical electrooculogram signal, performing moving average smoothing and time differentiation on the signal, and determining the sample points where the time derivative turns from negative to positive and the signal amplitude exceeds and falls below the average value plus or minus a certain standard deviation in the forward and backward fixed windows as blink time points.
[0011] Further, in step 2), the identified blink time points are converted into sparse binary time series, including initializing the all-zero sequence and inserting unit pulses at each identified blink position.
[0012] Furthermore, in step 3), bandpass filtering includes bandpass filtering of the speech signal using a filter bank with a center frequency in the range of 150~8000 Hz and uniformly distributed according to the equivalent rectangular bandwidth.
[0013] Furthermore, in step 3), nonlinear amplitude compression includes performing nonlinear amplitude compression on the analytic signal after Hilbert transform and performing low-pass filtering on the compressed signal to extract the slowly changing envelope component.
[0014] Further, in step 4), the multimodal deep convolutional neural network includes a spatial convolutional layer and three dilated convolutional blocks. The first dilated convolutional block generates a neural behavior representation, and the second and third dilated convolutional blocks generate stimulus representations for two speech streams, respectively. A modified linear unit is used as a nonlinear activation function after the dilated convolutional blocks.
[0015] Further, step 4) involves calculating the brain-speech cross-modal similarity, which includes calculating the similarity between the neural behavioral representation and the stimulus representation for each speech stream.
[0016] Furthermore, in step 4), training the multimodal deep convolutional neural network includes dividing the training dataset using the leave-one-out method, optimizing the network parameters through batch iterative training, and ending the training using an early stopping strategy.
[0017] An auditory attention decoding system based on blink sequences, comprising: The data preprocessing module is used to collect and preprocess EEG and EEG signals to obtain clean EEG data. The blink detection module is used to identify blink time points in the vertical electrooculogram signal and convert the identified blink time points into a sparse binary blink time sequence. The speech envelope extraction module is used to perform band filtering and Hilbert transform on the speech signal under test to obtain the analytical signal; to perform nonlinear amplitude compression and low-pass filtering on the analytical signal, to calculate the speech envelope signal of each frequency band, and to average the speech envelope signals of each frequency band to obtain the overall speech envelope signal. The deep neural network decoding module is used to input clean EEG data, sparse binarized blink time series, and overall speech envelope signal into a multimodal deep convolutional neural network to generate neural behavior representations and stimulus representations corresponding to each speech stream. Based on the neural behavior representations and stimulus representations, brain-speech cross-modal similarity is calculated, mapped to attention probabilities through fully connected layers, and the network is trained to optimize parameters to achieve automatic recognition of listener-attentive speech streams.
[0018] The present invention has achieved the following beneficial effects.
[0019] 1. This invention can achieve auditory attention decoding using only electrooculography data and the envelope of the candidate speech.
[0020] 2. This invention significantly improves decoding accuracy by using multimodal input of EEG and EEG signals, while maintaining stable performance even with low latency.
[0021] 3. This invention extracts temporal features based on deep convolutional neural networks, effectively capturing long-term dependent features and achieving automatic recognition of attentional speech streams.
[0022] 4. This invention employs blink sequence detection and sparse binarization representation, combined with speech envelope features, to provide effective input to the neural network and improve the model's decoding performance.
[0023] 5. The present invention exhibits good generalization ability under different time windows, verifying the synergistic advantage of brain-eye joint signals in representing the direction of attention.
[0024] 6. This invention can be applied to hearing aids, cognitive interaction systems, and brain-computer interfaces to achieve real-time and robust auditory attention decoding. Attached Figure Description
[0025] Figure 1 This is a diagram illustrating the consistency between blinking and speech.
[0026] Figure 2 This is an architectural diagram of the multimodal deep convolutional neural network of the present invention.
[0027] Figure 3 This is a graph showing the accuracy of auditory attention decoding within a 5-second window.
[0028] Figure 4 This is a graph showing the accuracy of auditory attention decoding within a 10-second window. Detailed Implementation
[0029] To make the various technical features, advantages, or effects of the present invention more apparent and understandable, detailed descriptions are provided below through embodiments.
[0030] This invention provides a method for decoding auditory attention based on blink sequences, comprising the following specific steps.
[0031] 1. Data preprocessing.
[0032] This embodiment uses the Peking University Hearing-Impaired Auditory Attention Dataset (PKU-HIAAD), which includes 19 hearing-impaired (HI) subjects. During electroencephalogram (EEG) acquisition, subjects were required to perform an auditory selective attention task, i.e., continuously focus on a pre-designated speaker. Each trial provided a 60-second auditory stimulus signal, including simultaneously played male and female Mandarin speech streams.
[0033] Both electroencephalogram (EEG) and electrooculogram (EOG) signals were recorded at a sampling rate of 500 Hz and filtered in real-time using an online bandpass filter ranging from 0.15 to 100 Hz. Subsequently, the EEG and EOG signals were uniformly downsampled to 64 Hz, and the EEG signal was further subjected to an offline bandpass filter ranging from 2 to 8 Hz. To remove artifacts from the EEG, trial-specific independent component analysis (ICA) was used for artifact separation and removal, resulting in cleaner EEG data.
[0034] 2. Blink detection.
[0035] Blinking events are identified by detecting local trough minima in the vertical electrooculogram (VEOG) signal, such as... Figure 1 The P region is indicated by the red vertical line. To improve detection stability, a 70-millisecond moving average filter is first applied to the VEOG signal for smoothing, and then the smoothed signal is differentiated over time.
[0036] When a sample point meets the following two conditions, it is determined to be a blink time point: (1) the time derivative changes from negative to positive; (2) the signal amplitude is higher than +1.5 times the standard deviation of the mean in the forward 100 ms window and lower than -1.5 times the standard deviation of the mean in the backward 100 ms window.
[0037] To represent eye-tracking events as a time-series format suitable for neural network input (consistent with speech envelope or EEG signal formats), a zero-based time-series array is initialized after a blink is detected, and unit pulses are inserted at each identified blink location to generate a sparse, binarized time-series signal. This sequence can then serve as one of the input features for subsequent neural network decoding models.
[0038] 3. Extraction of speech envelope signal.
[0039] For each preprocessed clean speech signal, it is first subjected to band filtering by a bandpass filter bank. The center frequency of this bandpass filter bank is in the range of 150~8000 Hz and is uniformly distributed according to the equivalent rectangular bandwidth (ERB) scale (see Glasberg & Moore, 1990).
[0040] ERB Scale and Frequency The correspondence is defined by the following formula: in, For the first The ERB number of each filter indicates the position of the frequency band in the ERB scale; For the first The passband center frequency of each filter, in kHz.
[0041] Subsequently, the output signal of each filter... Perform a Hilbert transform to obtain the analytic signal. Considering the nonlinear compression characteristics of the human cochlear system (see Plack et al., 2008), for analytical signals... The nonlinear amplitude compression process (0.3 power transformation) is calculated as follows: in, For signal The Hilbert transform result, This is the bandpass filtered output signal after nonlinear compression.
[0042] Next, the compressed signal Apply a low-pass filter operator with a cutoff frequency of 8 Hz To extract its slowly changing envelope component, the calculation formula is: in, It is a low-pass filter operator. For the first The voice envelope signal corresponding to each frequency band.
[0043] Finally, the envelope signals of all frequency bands are averaged sample by sample to obtain the final overall speech envelope signal. : in, For the overall speech envelope signal, This represents the total number of bands in the filter bank.
[0044] Through the above steps, an envelope signal reflecting the time-domain energy variation characteristics of speech can be obtained, providing temporal input features for subsequent multimodal decoding models.
[0045] 4. Use a multimodal deep convolutional neural network to decode auditory attention.
[0046] Auditory attention decoding methods based on deep neural networks (DNNs) have demonstrated excellent performance in recent years. To verify the effectiveness of multimodal fusion, this invention designs a multimodal deep convolutional neural network architecture that combines electroencephalogram (EEG) and eye movement (ES) signals, and compares its performance with models using only a single modality (EEG or ES). The system simultaneously extracts temporal physiological features from EEG and ES, as well as speech envelope signal features, and performs cross-modal alignment in a unified feature space. The model output indicates whether the speech stream currently being attended to by the listener corresponds to speech envelope 1 or speech envelope 2. Considering the low latency requirements of practical hearing aids or attention enhancement devices, the system uses decision windows of 5 seconds and 10 seconds for inference, respectively.
[0047] The overall architecture of a multimodal deep convolutional neural network is as follows: Figure 2As shown, the input biological signals (EEG and ES) are first subjected to channel-level feature projection through a spatial convolution layer (8 channels, 1 kernel size) to capture cross-channel spatial dependencies. Subsequently, the signals pass through three dilated convolution blocks with kernel sizes of 3, 9, and 27, respectively, each containing 16 channels. This dilated convolution structure expands the receptive field while maintaining computational efficiency, thus effectively extracting long-term dependent features.
[0048] Furthermore, for the speech signal portion, independent but structurally identical dilated convolutional networks were used to extract the temporal envelope features of speech stream 1 and speech stream 2, respectively. A rectified linear unit (ReLU) was applied as the non-linear activation function after all dilated convolutional layers. The first dilated convolutional block generated the neural behavior representation, denoted as . The second and third dilated convolutional blocks share parameters and generate stimulus representations for speech stream 1 and speech stream 2, respectively, denoted as... and Cross-modal similarity is quantified by calculating cosine similarity: in, For the first Stimulus features of a speech stream are represented. ; The representation of neural behavioral features obtained by joint encoding of EEG and ES; Let L2 be the norm of the vector.
[0049] The similarity results are passed through a 16-to-1 fully connected layer and mapped to attention probabilities. and , respectively, indicate the possibility that the listener pays attention to speech stream 1 or speech stream 2.
[0050] Finally, optimization is performed using the cross-entropy loss function, the expression of which is as follows: in, For the total loss function, For the first The true label of each speech stream (with a value of 0 or 1, indicating whether it is the listener's attention target). This represents the corresponding probability predicted by the model.
[0051] For example, when the listener pays attention to speech stream 1, the label is taken. The model minimizes the loss through backpropagation. By gradually optimizing the parameters, automatic decoding of auditory attention objects can be achieved.
[0052] To prevent overestimation of accuracy due to temporal autocorrelation, this invention employs a "leave-one-subject-out" method for data partitioning. Specifically, in each experiment, all samples collected from one subject are used as the test set, while samples from the remaining subjects are used for training and validation. The batch size during training is 128, and the maximum number of training epochs is 100. Early stopping is triggered when the validation accuracy fails to improve within 10 consecutive epochs.
[0053] The optimizer uses the AdamW algorithm, with a learning rate set to... The weight decay factor is 0.01. All models share a consistent hyperparameter configuration, are implemented based on the PyTorch framework, and are trained and validated on high-performance computing nodes equipped with A800 GPUs.
[0054] Through the joint learning mechanism of the aforementioned multimodal deep neural network, the system can effectively capture brain-eye signal features related to auditory attention, and achieve real-time and robust decoding of the target speech for the listener's attention.
[0055] The following is a test result and analysis of auditory attention decoding based on the above method.
[0056] To evaluate the decoding performance of the proposed multimodal deep neural network under different time windows, auditory attention decoding tests were conducted for time windows of 5 seconds and 10 seconds. Leave-one-out cross-validation was used in all tests to ensure the objectivity and generalization ability of the results.
[0057] Figure 3 The results show the accuracy of auditory attention decoding within a 5-second window. It is evident that the multimodal model using combined EEG and ES inputs performs best, achieving an average accuracy of 65.2% (standard error = 1.8%) under purely auditory conditions and 72.1% (standard error = 1.2%) under audiovisual conditions. In contrast, the models using only ES input have accuracies of 58.3% ± 1.7% (auditory conditions) and 63.8% ± 1.6% (auditory conditions), while the models using only EEG input have accuracies of 63.3% ± 1.7% and 69.7% ± 1.2%, respectively. This demonstrates that the multimodal fusion model significantly outperforms the single-modal model under both conditions, indicating a complementary relationship between EEG and ES signals in attention recognition.
[0058] Figure 4The results show the accuracy of auditory attention decoding within a 10-second window. As the time window lengthens, the overall performance of all models improves. The multimodal model achieves an average accuracy of 70.3% (standard error = 2.2%) under auditory conditions and 78.7% (standard error = 1.8%) under audiovisual conditions, significantly higher than the single-modal input case. Specifically, the model using only ES achieves 59.7% ± 2.3% (audio) and 69.0% ± 2.1% (audiovisual), while the model using only EEG achieves 67.2% ± 2.3% and 74.3% ± 1.9%.
[0059] Comprehensive analysis shows that as the time window increases from 5 seconds to 10 seconds, the decoding accuracy improves by an average of 5% to 7%, indicating that a longer time window helps the model capture stable auditory attention features. The multimodal EEG-ES fusion model significantly outperforms the single-modal model under all conditions, verifying the synergistic advantage of brain-eye joint signals in representing the direction of attention. The accuracy under visual and auditory conditions is generally higher than that under auditory conditions, indicating that visual information can enhance the neural relevance of auditory attention, thereby further improving decoding performance. Therefore, the multimodal deep neural network model proposed in this invention can achieve high auditory attention recognition accuracy under low latency windows and has application potential in hearing aids, cognitive interaction systems, and brain-computer interfaces.
[0060] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention should be covered within the protection scope of the present invention, which is defined by the claims.
Claims
1. A method for decoding auditory attention based on blink sequences, characterized in that, Includes the following steps: 1) Collect and preprocess EEG and EEG signals to obtain clean EEG data; 2) Identify blink time points in the vertical electrooculogram (EOG) signal and convert the identified blink time points into a sparse binary blink time sequence; 3) Perform band filtering and Hilbert transform on the speech signal to be tested to obtain the analytical signal; perform nonlinear amplitude compression and low-pass filtering on the analytical signal, calculate the speech envelope signal of each frequency band, and average the speech envelope signals of each frequency band to obtain the overall speech envelope signal. 4) Input clean EEG data, sparse binarized blink time series and overall speech envelope signal into a multimodal deep convolutional neural network to generate neural behavior representations and stimulus representations corresponding to each speech stream. Calculate brain-speech cross-modal similarity based on neural behavior representations and stimulus representations, map it to attention probability through a fully connected layer, and train the network to optimize parameters to achieve automatic recognition of listener attention speech streams.
2. The method as described in claim 1, characterized in that, The data preprocessing in step 1) includes: downsampling the EEG and EOS signals, performing 2-8 Hz offline bandpass filtering on the EEG signals, and using independent component analysis to remove artifact signals.
3. The method as described in claim 1, characterized in that, Step 2) The identification of blink time points includes: detecting the local trough minimum value of the vertical electrooculogram signal, performing moving average smoothing and time differentiation on the signal, and determining the sample points where the time derivative turns from negative to positive and the signal amplitude exceeds and falls below the average value plus or minus a certain standard deviation in the forward and backward fixed windows as blink time points.
4. The method as described in claim 1, characterized in that, Step 2) converts the identified blink times into a sparse binary time series, including initializing the all-zero sequence and inserting a unit pulse at each identified blink position.
5. The method as described in claim 1, characterized in that, Step 3) involves bandpass filtering of the speech signal using a filter bank with a center frequency in the range of 150~8000 Hz and uniformly distributed according to the equivalent rectangular bandwidth.
6. The method as described in claim 1, characterized in that, Step 3) Nonlinear amplitude compression includes performing nonlinear amplitude compression on the analytic signal after Hilbert transform, and performing low-pass filtering on the compressed signal to extract the slowly changing envelope component.
7. The method as described in claim 1, characterized in that, In step 4), the multimodal deep convolutional neural network includes a spatial convolutional layer and three dilated convolutional blocks. The first dilated convolutional block generates a neural behavior representation, and the second and third dilated convolutional blocks generate stimulus representations for two speech streams, respectively. A modified linear unit is used as a non-linear activation function after the dilated convolutional blocks.
8. The method as described in claim 1, characterized in that, Step 4) Midbrain-speech cross-modal similarity calculation includes calculating the similarity between the neural behavioral representation and the stimulus representation of each speech stream.
9. The method as described in claim 1, characterized in that, Step 4) involves training a multimodal deep convolutional neural network, which includes dividing the training dataset using the leave-one-out method, optimizing the network parameters through batch iterative training, and ending the training using an early stopping strategy.
10. An auditory attention decoding system based on blink sequences, used to perform the method according to any one of claims 1-9, characterized in that, include: The data preprocessing module is used to collect and preprocess EEG and EEG signals to obtain clean EEG data. The blink detection module is used to identify blink time points in the vertical electrooculogram signal and convert the identified blink time points into a sparse binary blink time sequence. The speech envelope extraction module is used to perform band filtering and Hilbert transform on the speech signal under test to obtain the analytical signal; to perform nonlinear amplitude compression and low-pass filtering on the analytical signal, to calculate the speech envelope signal of each frequency band, and to average the speech envelope signals of each frequency band to obtain the overall speech envelope signal. The deep neural network decoding module is used to input clean EEG data, sparse binarized blink time series, and overall speech envelope signal into a multimodal deep convolutional neural network to generate neural behavior representations and stimulus representations corresponding to each speech stream. Based on the neural behavior representations and stimulus representations, brain-speech cross-modal similarity is calculated, mapped to attention probabilities through fully connected layers, and the network is trained to optimize parameters to achieve automatic recognition of listener-attentive speech streams.