An electroencephalogram speech generation method and system fusing soft wavelet multi-scale feature extraction and task decoupling

Abstract

The application discloses a kind of fusion soft wavelet multiscale feature extraction and task decoupling electroencephalogram speech generation method and system, the method first collects the electroencephalogram signal of subject reading and records speech, respectively pre-processing, obtain pre-processed electroencephalogram feature and pre-processed Mel spectrum diagram.Secondly, the pre-processed electroencephalogram feature is extracted using soft wavelet thinning and multiscale hollow convolution feature extractor, after position coding and global feature extraction, global electroencephalogram feature is obtained.Then construct bridge mechanism, realize neuroacoustic signal alignment for global electroencephalogram feature, generate alignment electroencephalogram feature, generate Mel spectrum diagram by task decoupling multi-branch spectrum generation decoder.Lastly, through iterative harmonic phase reconstruction vocoder, generate speech, while combining pre-processed Mel spectrum diagram to construct joint loss function to train optimization.The present application improves spectral matching degree and restoration accuracy, greatly improves the intelligibility and naturalness of electroencephalogram decoding speech.

Description

Technical Field

[0001] This invention belongs to the field of brain-computer interfaces and relates to a method and system for generating brain-computer speech that integrates soft wavelet multi-scale feature extraction and task decoupling. Background Technology

[0002] Electroencephalogram (EEG) signals are the physical manifestations of the synchronized electrical activity of neuronal groups in the cerebral cortex at macroscopic or microscopic scales. In terms of acquisition methods and spatial resolution, EEG signals are not limited to traditional non-invasive scalp EEG (EEG), but also broadly encompass invasive electrocorticography (ECoG), stereotactic electroencephalography (sEEG), and local field potentials (LFP) of deep brain regions obtained through surgical implantation. These signals record the potential changes of the central nervous system at different spatiotemporal resolutions, containing extremely rich neural information such as cognition, emotion, and motor intentions, and are the core medium for brain-computer interface systems, neuropsychiatric diseases, and cognitive science research. However, EEG signals typically exhibit strong nonlinearity, non-stationarity, and low signal-to-noise ratio characteristics, making them highly susceptible to interference from physiological artifacts and external electromagnetic noise.

[0003] Research into decoding semantic information from electroencephalogram (EEG) signals originates from in-depth exploration of the human brain's language processing mechanisms. By analyzing the brain's neural activity in response to language stimuli, the semantic encoding mechanisms within the brain can be revealed. Studies have shown that the human brain's ability to rapidly understand linguistic information and generate corresponding language expressions demonstrates its complex processing capabilities. EEG signal decoding provides a new tool for studying human cognitive processes, contributing to an understanding of how the brain processes higher cognitive functions such as language, memory, and emotion.

[0004] For patients suffering from severe speech impairments due to diseases such as stroke and amyotrophic lateral sclerosis (ALS), EEG-based speech decoding technology holds promise for constructing a "neural bypass" that directly translates their linguistic intentions into audible speech or text, fundamentally restoring their natural and efficient communication abilities and significantly improving their quality of life and dignity. EEG signals provide scientists with an unprecedented window into how the brain dynamically encodes various elements of speech (such as phonemes, tones, and prosody) during natural vocalization, silent reading, and even imagined speech. This is crucial for understanding the uniquely human ability to speak.

[0005] However, current research on decoding speech from EEG signals faces several key challenges, severely limiting the intelligibility and clarity of the generated speech. EEG signals contain long-range semantic dependencies across time steps, but EEG datasets are sparse and have small sample sizes. Traditional networks, by increasing the receptive field through convolutional kernels, are prone to overfitting. EEG signals are highly aliased and non-stationary, containing both macroscopically smooth background potential trends and highly volatile, transient neural responses. Traditional temporal aggregation and denoising operations often employ low-pass filtering effects. While stabilizing background potential trends, they inevitably smooth out transient neural changes highly correlated with microscopic timbre and consonant transitions, leading to overly smoothed reconstructed spectra and unclear speech.

[0006] EEG-based speech decoding is essentially a cross-modal transformation task, namely, transforming a high-dimensional, sparse, and spatiotemporally complex neural signal modality into an acoustic or language sequence modality. Many existing methods employ an end-to-end "direct mapping" paradigm, attempting to forcibly establish a mapping from EEG signals to speech features using a single model. However, there is a fundamental mismatch in the structure of the feature spaces between these two modalities.

[0007] The overall linear trend of the speech mel spectrum reflects the fundamental frequency trend, syllable prosody, and core semantics, serving as a crucial guarantee for the intelligibility of generated speech. Strict linear mapping constraints are necessary to avoid global distortion. Meanwhile, spectral edge details (consonant transient boundaries, fine harmonic structure, and formant contours) determine speech clarity and naturalness, relying on nonlinear local enhancement features. These two aspects inherently conflict: global linear constraints suppress detail enhancement, while the nonlinear operations required for detail restoration easily disrupt global linearity, ultimately leading to a dilemma of "preserving the overall picture at the expense of details, or enhancing details at the expense of the overall picture," severely limiting the quality of speech generated by EEG decoding. Summary of the Invention

[0008] This invention proposes a method and system for EEG speech generation that integrates soft wavelet multi-scale feature extraction and task decoupling. The method employs soft wavelet refinement and a multi-scale dilated convolution feature extractor. Its core functions are: using pooling and standard convolution to stabilize the background smoothing trend and resolution benchmark, and using heterogeneous dilated convolution to lightweightly expand the temporal receptive field; the soft wavelet refinement branch uses temporal difference to simulate wavelet decomposition and, through learnable gating, dynamically captures transient neural mutations associated with speech details, perfectly balancing macroscopic noise reduction and high-fidelity representation of microscopic features. Based on this, the method introduces a task-decoupled multi-branch spectrum generation decoder. The trend path and main decoding branch lock the core topology and macroscopic energy trend of the spectrum through linear mapping, stabilizing the global linear benchmark; the residual branch uses a neural guidance network to repair the edge harmonic structure and locally restricts the nonlinear contrast enhancement operation within the residual. Without compromising global linearity, it perfectly restores speech formants and timbre texture, achieving a synergistic optimization of linear fidelity and detail restoration. The Pearson correlation coefficient (PCC) and time-frequency glimpse index (STGI) of the Mel spectrogram generated by this method are superior to existing methods, and the generated audio has good intelligibility.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows: In one aspect, this invention provides a method for generating brainwave speech that integrates soft wavelet multi-scale feature extraction and task decoupling, comprising the following steps: Step S1: Collect EEG signals from the subject as they read words or sentences aloud, and simultaneously record their speech; preprocess the EEG signals to obtain preprocessed EEG features, and preprocess the speech to obtain preprocessed speech and Mel spectrograms.

[0010] Step S2: The preprocessed EEG features are extracted using soft wavelet refinement and a multi-scale dilated convolution feature extractor to obtain multi-scale EEG features. Considering the characteristic of EEG signals simultaneously containing long-range semantic dependencies and local transient neural responses, a 5-branch parallel Inception architecture is adopted: First, an average pooling branch extracts background smoothing features to suppress random noise; second, a 1×1 convolution preserves the original resolution baseline features to avoid loss of core information; third and fourth, 3×1 dilated convolutions with dilation rates of 2 and 4 effectively expand the temporal receptive field coverage while limiting the number of parameters; fifth, the core soft wavelet refinement branch separates high and low frequency components through differential simulation of stationary wavelet decomposition, and then performs dynamic fusion with learnable sigmoid gating to accurately capture neural transient mutations related to speech details; after concatenation of the multi-branch features, CBAM attention enhancement is applied to output multi-scale EEG features.

[0011] Step S3: Perform global feature extraction. After location encoding of the multi-scale EEG features, the global EEG features with regular dimensions and rich global semantic information are output sequentially through bidirectional Mamba (capturing long-range dependencies), Conformer (enhancing local-global semantics), and bidirectional Mamba.

[0012] Step S4: Cross-modal bridging layer achieves neuroacoustic signal alignment. A bridging mechanism of "neural self-enhancement - acoustic prior guidance - cross-modal alignment - gated fusion" is constructed: Global EEG features are first enhanced through multi-head self-attention, resulting in enhanced global EEG features. Then, a learnable acoustic prior query that perfectly matches the input temporal sequence and dimension is introduced as an anchor point. Cross-modal attention is performed using the acoustic prior as the query and the enhanced global EEG features as the key / value to obtain aligned features. Finally, the global EEG features and the aligned features are fused through learnable gated residuals to generate aligned EEG features. This achieves precise alignment of neural modalities to the acoustic space without additional annotation, solving the problems of cross-modal semantic drift and temporal misalignment.

[0013] Step S5: The multi-branch spectrum generation decoder, decoupled from the task, generates the Mel spectrogram. Addressing the inherent contradiction between overall linear trend fidelity and spectral edge detail restoration in speech spectrum generation—that is, the linear constraint of the global trend inhibits detail enhancement, while the nonlinear operations required for detail enhancement easily disrupt global linearity—three parallel decoupled paths are split: the trend path generates an overall linear trend of the spectrum through global average pooling and linear layers, ensuring global trend fidelity through linear mapping throughout; the main decoding branch generates a linear main spectrogram through 2D convolution, fusing with the trend path to lock in a global linear baseline; the residual branch repairs spectral edges and fine harmonic structures through neural-guided U-Net, performing local contrast enhancement only on residual components to restore details without disrupting global linearity; finally, the multi-branch fusion outputs a high-quality Mel spectrogram with accurate global trend and clear edge details, achieving synergistic optimization of linear trend fidelity and detail restoration.

[0014] Step S6: The Mel spectrogram generates speech using an iterative harmonic phase reconstruction vocoder.

[0015] Preferably, in step S2, the process of soft wavelet thinning and multi-scale dilated convolution feature extractor is as follows: Targeting preprocessed EEG features This invention uses a five-branch parallel soft wavelet thinning and multi-scale dilated convolution feature extractor to achieve comprehensive feature representation: Baseline branches (Branch 1 & 2): Utilize average pooling to capture the low-frequency trend of the background field potential, and through... Convolution preserves the identity mapping of the original resolution.

[0016] Temporal Dependency Branch (Branch 3 & 4): Introduces dilated convolution (dilation rate) This design effectively expands the temporal receptive field coverage without increasing parameters.

[0017] Soft wavelet refinement branch (Branch 5): Simulates wavelet decomposition filters using predefined convolutional kernel groups to refine EEG features. Decomposed into low-frequency approximate components and With high-frequency detail components : in, and This represents a fixed-weight or learnable-weight convolution operator constructed based on wavelet basis functions (such as Haar or Daubechies). and This represents network parameters.

[0018] To achieve nonlinear refinement of features, the embodiment introduces a learnable gated weight matrix. This is used to dynamically adjust the contribution of the two types of components in the final representation. Based on EEG characteristics... Generate gate weights : in, The Sigmoid activation function maps the weights to... interval; and This is the learnable parameter matrix for this branch. This is a convolution operation.

[0019] Finally, this branch outputs refined time-frequency features through a weighted fusion method. : This represents element-wise multiplication. The gating mechanism dynamically adjusts its weights based on the neural input at the current time step. During periods of signal stability, the gating biases towards low-frequency components, adaptively suppressing high-frequency random discharge noise; during abrupt changes in pronunciation or syllable transitions, the gating momentarily biases towards high-frequency components, allowing details to pass through. This achieves dynamic denoising and local enhancement at the feature level.

[0020] The soft wavelet refinement branch and the features output from the other four branches are concatenated along the channel dimension to form a multi-scale feature matrix. Subsequently, the fused features are recalibrated for spatial and channel-dimensional importance using a convolutional block attention model (CBAM) to generate multi-scale EEG features. This is to ensure the accuracy of subsequent neural speech decoding.

[0021] Preferably, the specific steps of the task-decoupled multi-branch spectrum generation decoder in step S5 are as follows: Decoder receives aligned EEG features ,in For batch size, For the time-series splicing window dimension, As a feature dimension, the number of frequency bands in the target Mel spectrum is This step resolves the inherent contradiction between global trend fidelity and detail restoration through a three-branch decoupling architecture: Linear trend branch: Global average pooling (GAP) and linear transformation are used to capture the macroscopic trend of speech energy distribution, ensuring that the generated Mel spectrogram is consistent with the target speech in terms of overall energy magnitude, preventing the baseline shift caused by nonlinear prediction runaway, and guaranteeing the global linear trend.

[0022] in, , These are the learnable weights for linear transformations.

[0023] The spatially conserved main decoding branch is responsible for reconstructing the core topological structure of the spectrum. Spatially conserved convolution refers to maintaining the consistency of the mapping between neural features in the time-frequency dimension through specific convolution strides and padding, avoiding spectral distortion caused by significant feature stretching. This will align EEG features. Remodeling into 2D spatial features .

[0024] By maintaining resolution convolution kernels Perform mapping to generate the main spectrum. : in This is for trend weighting. It provides a foundation with the correct geometric structure for subsequent detailed supplementation.

[0025] Neurally Guided Contrast Enhancement Residual Branch: Through mathematical transformation, high-dimensional, nonlinear neural features are mapped to the acoustic spectral space, serving as a priori corrector to repair local details in the output of the main decoding branch.

[0026] First, through a fully connected layer... Perform a linear projection, mapping it to the feature latent space, to generate Dimensions The physical significance of this step is to explore the conversion of the electrical signal intensity of the brainwave channels into an energy distribution related to the spectral components. Remodeling The convolutional kernel generates neural guidance signals. .

[0027] Will Remodeling Same dimension, and Concatenate along the channel dimension (axis=-1): At this time It has two channels: acoustic contours (from the main decoding branch of the decoder) and neural constraints (from the projection of the original signal from the brain). This design allows for the correction of blurred frequencies generated by the main decoding branch by referencing the frequencies of the sound the brain originally intended to emit when repairing the spectrum.

[0028] Subsequently, convolution is used to capture local spectral smoothness, resulting in features. Hollow convolution expands the receptive field, capturing long-range harmonic relationships to obtain features. Features are combined using residual connections and convolution. and The fusion process outputs the final repaired 128-dimensional spectrum vector: Finally, adaptive boosting weights are generated using local max pooling for the spectrum vector. After contrast enhancement, compared with the main spectrum The final spectrum of the fused output .

[0029] In another aspect, the present invention also provides a brainwave speech generation system that integrates soft wavelet multi-scale feature extraction and task decoupling, comprising the following modules: The EEG signal module is used to collect the EEG signals of the subject reading aloud and record the speech simultaneously; the EEG signals are preprocessed to obtain preprocessed EEG features, and the speech is preprocessed to obtain a preprocessed Mel spectrogram.

[0030] The multi-scale EEG feature module is used to extract features from preprocessed EEG features using soft wavelet thinning and a multi-scale dilated convolution feature extractor to obtain multi-scale EEG features.

[0031] The global EEG feature module is used to encode the location of multi-scale EEG features and then output the global EEG features through bidirectional Mamba, Conformer and bidirectional Mamba in sequence.

[0032] The aligned EEG feature module is used to construct a bridging mechanism to align neuroacoustic signals for global EEG features and generate aligned EEG features.

[0033] The Mel spectrogram module generates Mel spectrograms based on aligned EEG features and through a task-decoupled multi-branch spectrogram generation decoder.

[0034] The EEG speech generation module is used to generate speech from the Mel spectrogram generated by the decoder through an iterative harmonic phase reconstruction vocoder. At the same time, it combines the preprocessed Mel spectrogram to construct a joint loss function for training and optimization.

[0035] Compared with the prior art, the beneficial effects of the present invention are reflected in: 1. The soft wavelet thinning and multi-scale dilated convolution feature extractor proposed in this invention has significant advantages over existing feature extraction methods, accurately adapting to the characteristics of EEG signals. Compared to fixed wavelet transform and ordinary dilated convolution, it can learn soft wavelet gating to dynamically separate high and low frequency components, adaptively matching the feature requirements of stable signal segments and transient abrupt segments, accurately restoring the high-frequency neural responses corresponding to speech details, while suppressing noise. The final extracted features have significantly improved discriminative power and anti-interference ability, laying a solid foundation for high-quality speech decoding.

[0036] 2. The acoustic prior-guided cross-modal bridging mechanism proposed in this invention addresses the modal heterogeneity gap in EEG-speech cross-modal conversion, offering significant performance and engineering advantages over existing methods. Compared to the mainstream end-to-end "direct mapping" paradigm, this mechanism avoids forcibly fitting the feature spaces of neural and acoustic heterogeneous modalities with a single model, fundamentally avoiding the semantic drift and temporal misalignment defects commonly found in traditional methods. Using learnable acoustic priors as alignment anchors, this mechanism combines neural self-reinforcement, cross-modal attention, and gated residual fusion to achieve precise alignment from the neural space to the acoustic space while fully preserving the original core neural information, significantly improving the semantic consistency, temporal matching, and intelligibility of decoded speech.

[0037] 3. The task-decoupled multi-branch spectrum generation decoder of this invention overcomes the inherent contradiction of traditional single-branch decoders—"preserving global linear trends at the expense of details, and enhancing details at the expense of global linearity"—offering significant advantages over existing methods. Through a three-branch parallel task-decoupled architecture, global trend fidelity preservation, main spectrogram generation, and detail enhancement are completely decoupled; the linear trend branch performs a linear mapping throughout, anchoring core global features such as speech fundamental frequency and prosody to ensure speech intelligibility; local detail enhancement is performed only on residual components that do not affect the global trend, losslessly restoring consonant boundaries and fine harmonic structures. Ultimately, this achieves synergistic optimization of global trends and local details, improving spectrum matching accuracy and restoration precision, and significantly enhancing the clarity and naturalness of EEG-decoded speech. Attached Figure Description

[0038] Figure 1 A schematic diagram illustrating the framework of the method of the present invention is shown. Detailed Implementation

[0039] To make the technical means, inventive features, objectives, and effects of the invention easier to understand, the following is a detailed explanation. Figure 1 The invention is further illustrated below. However, the invention is not limited to the embodiments described below.

[0040] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0041] Step S1: While the subject reads words or sentences aloud, the subject's EEG signals are collected and their speech is recorded simultaneously. The EEG signals are first detrended, and then high-gamma frequency components are obtained through bandpass filtering. At the same time, a notch filter is used to remove 50Hz power frequency noise and its harmonic interference. After extracting the signal envelope through Hilbert transform, overlapping windows are created with a time window of 0.05s and a frame shift of 0.01s, and the average amplitude within the window is calculated. A sliding window strategy with a model order of 4 and a step size of 5 is used to fuse the feature information of the current time window and the adjacent time windows to capture the temporal dependencies of multiple time intervals and fully represent the dynamic changes of the signal over time. Finally, the preprocessed EEG features are obtained by Z-score standardization. , dimension These correspond to the number of time frames, the dimension of the time-series splicing window, and the number of channels, respectively. The audio signal was first downsampled to 16kHz to obtain preprocessed speech. A 128-dimensional Mel spectrogram was extracted using a Hanning window with overlapping time windows of the same duration and frame shift. The dataset was divided into two parts: a training set (80%) and a test set (20%).

[0042] Step S2: The preprocessed EEG features are extracted using soft wavelet thinning and a multi-scale dilated convolution feature extractor to obtain multi-scale EEG features. This process is then applied to the preprocessed EEG features. This method uses a five-branch parallel soft wavelet thinning and multi-scale dilated convolution feature extractor to achieve comprehensive feature representation: Step S21, Baseline Branch (Branch 1 & 2): The baseline branch is composed of parallel one-dimensional average pooling paths and standard linear projection paths, aiming to construct a global semantic baseline for multi-scale feature extraction.

[0043] Branch 1 utilizes a one-dimensional average pooling operator (GAP) with a kernel size of 3 to perform mean sampling of a local window along the time axis. Its physical significance lies in suppressing transient non-stationary noise in the EEG signal through a low-pass filtering mechanism, thereby stabilizing the extraction of the low-frequency evolution trend of the background field potential. Subsequently, it combines... Convolution performs feature projection to achieve precise alignment along the channel dimensions. Branch 2 directly applies the standard... The convolution operator performs a linear transformation on the original input sequence, serving as an identity mapping path to ensure the initial reorganization of spatial features without sacrificing temporal resolution. This dual-branch architecture effectively addresses the problem of easily lost weak neural features in deep networks by collaboratively representing macroscopic evolutionary features and high-fidelity original features, providing a robust reference foundation for subsequent nonlinear refinement modules.

[0044] Step S22, Dilated Convolution Branches (Branch 3 & 4): In response to the inherent characteristics of high sparsity and limited sample size of EEG data, and to address the issues of increased parameter count and increased risk of overfitting with small samples caused by increasing the convolution kernel to expand the receptive field in traditional deep neural networks, a one-dimensional dilated convolution architecture is introduced in branches 3 and 4 to achieve a balance between the receptive field and the parameter size.

[0045] Dilated convolution modifies the sampling interval of the input signal by inserting holes between the parameters of the convolution kernel. For an input sequence... The output of a one-dimensional dilated convolution The definition is as follows: in, For size convolution kernel, This refers to the dilation rate. This mechanism does not increase the learnable weights. Under the premise of equivalent receptive field Expanded to: .

[0046] Branch 3: Setting the expansion rate While maintaining the convolution kernel With its lightweight design, its equivalent receptive field reaches 5 time steps, aiming to capture mid-frequency fluctuations in neural oscillation signals. Branch 4: Setting the expansion rate. 4. At this point, the equivalent receptive field expands to 9 time steps, enabling full temporal coverage of the entire temporal stitching window with extremely low parameter cost.

[0047] Step S23, Soft Wavelet Refinement Branch (Branch 5): Simulates a wavelet decomposition filter using a predefined set of convolution kernels, decomposing the original signal into low-frequency approximate components and... With high-frequency detail components : Component decomposition: Simulating Haar wavelet decomposition, calculating the difference of the sequence to extract high-frequency components: Subsequently, feature mapping was performed on the original signal and the high-frequency components respectively to generate low-frequency approximate components. With high-frequency detail components : in, Denotes the exponential linear unit activation function, and BN denotes the batch normalization layer. , These are the learnable convolution kernel parameters.

[0048] To achieve nonlinear refinement of features, this embodiment introduces a learnable gated weight matrix. This is used to dynamically adjust the contribution of the two types of components in the final representation. The formula for generating the gating weights is as follows: in, The sigmoid activation function maps the weights to... interval; and This is the learnable parameter matrix for this branch. This is a convolution operation.

[0049] Finally, this branch outputs refined time-frequency features through a weighted fusion method. : This represents element-wise multiplication. The gating mechanism dynamically adjusts its weights based on the neural input at the current time step. During periods of signal stability, the gating biases towards low-frequency components, adaptively suppressing high-frequency random discharge noise; during abrupt changes in pronunciation or syllable transitions, the gating momentarily biases towards high-frequency components, allowing details to pass through. This achieves dynamic denoising and local enhancement at the feature level.

[0050] Step S24, Cross-Dimensional Feature Fusion and Attention Enhancement: After completing the feature extraction of the above five branches, perform the fusion operation. Feature Concatenation: Concatenate the outputs of each branch along the channel dimension to form a multi-scale aggregated feature. Spatial-Channel Attention Enhancement (CBAM): ... The input convolutional block attention module (CBAM) weights important neural features through channel attention and spatial attention mechanisms to generate multi-scale EEG features. : in, and These represent channel and spatial attention mappings, respectively. This mechanism ensures that the model can focus on the neural firing patterns most relevant to speech reconstruction in subsequent stages.

[0051] Step S3: Perform global feature extraction. First, the multi-scale EEG features are positionally encoded, and then serially stacked using two-level bidirectional Mamba blocks and a single-level Conformer block. The bidirectional Mamba blocks model forward and backward sequences, efficiently capturing global long-range temporal dependencies and solving the gradient vanishing problem in long sequences. The inserted Conformer block uses a Macaron structure of "feedforward-self-attention-convolution-feedforward" to strengthen local contextual association and global semantic consistency, making up for the shortcomings of Mamba in capturing local features. Finally, the output is a global EEG feature with regular dimensions and rich global semantic information. ; Step S4: Achieving neuroacoustic signal alignment via a cross-modal bridging layer. After obtaining global EEG features, this embodiment achieves precise alignment of neural signals to the acoustic representation space through a cross-modal bridging layer.

[0052] Step S41: Neural Feature Self-Enhancement. Utilizing multi-head self-attention (MHSA) mechanisms to enhance the input global EEG features. , For batch size, For the time-series splicing window dimension, Using this as a feature dimension, we perform internal correlation mining to enhance the consistency of neural signal representation in the spatial dimension.

[0053] This step uses feature self-enhancement to extract discriminative neural discharge patterns from EEG signals.

[0054] Step S42: Acoustic Prior Guidance and Cross-Modal Alignment. To map neural features to the acoustic semantic space, this embodiment introduces a set of learnable acoustic prior queries. The query vector serves as the anchor point, and the acoustic envelope distribution characteristics in the latent space are automatically learned through training.

[0055] The alignment process employs a cross-modal attention mechanism. Acoustic priors are used as the basis for this. For querying, to obtain enhanced neural features As key and value: Through this operation, the model forces neural features to find the neural activation component that best matches the acoustic prior at each time step, thereby achieving a non-linear mapping from "EEG semantics" to "acoustic semantics".

[0056] Step S43, Residual Gated Fusion. To prevent excessive loss of original neural information during cross-modal alignment, this embodiment constructs a gated residual fusion mechanism based on learnable weights. Gated weight generation: Utilizing... The activation function dynamically generates gate vectors based on the aligned features. This is used to balance the retention ratio of raw neural information and alignment features: in, The sigmoid activation function maps the weights to... interval; and For learnable parameter matrix, Bimodal fusion: In the early stages of training when cross-modal alignment is low, through... The path retains the physical properties of the original neural features; as training progresses, The tendency is to select more acoustically representative aligned EEG features. The final output is an aligned EEG feature that combines neurophysical integrity with acoustic semantic relevance. This provides accurate feature inputs for the subsequent generation of high-fidelity spectrograms.

[0057] In step S5, a task-decoupled multi-branch spectrum generation decoder is used. The decoder receives aligned EEG features. ,in For batch size, For the time-series splicing window dimension, The feature dimension is [value], and the target Mel spectrum flattening dimension is [value]. This step resolves the inherent contradiction between global trend fidelity and detail restoration through a three-branch decoupling architecture: Step S51, Linear Trend Branch: The macroscopic trend of speech energy distribution is captured by global average pooling (GAP) and linear transformation to ensure that the generated Mel spectrogram is consistent with the target speech in terms of overall energy magnitude, prevent the reference shift caused by nonlinear prediction runaway, and ensure the global linear trend.

[0058] in, , These are the learnable weights for linear transformations.

[0059] Step S52, Spatial Conservation Main Decoding Branch: This branch is responsible for reconstructing the core topological structure of the spectrum. Spatial conservation convolution refers to maintaining the consistency of the mapping of neural features in the time-frequency dimension through specific convolution strides and padding, avoiding spectral distortion caused by large-scale feature stretching. This will align the EEG features... Remodeling into 2D spatial features By using resolution-preserving convolutional kernels The mapping process first involves... Convolution is used for cross-channel feature fusion; subsequently, a kernel size of [missing value] is employed. Step size is The valid convolution smoothly reduces the time axis from 9 steps to 8 steps, achieving temporal feature fusion without destroying the spatial structure; linear activation is used. Convolution, acting as the output mapping, preserves the linear change in spectral energy, thus stabilizing the Pearson correlation coefficient. This involves branching the linear trend. Inject space-conserving main decoding branch to generate main spectrum This allows the model to quickly grasp the global trend of the spectrum in the early stages of training, ensuring that the reconstructed spectrum is highly correlated with the target signal macroscopically, thereby alleviating the problem of drastic fluctuations in PCC during the early stages of deep network training. A simplified formula is as follows: in This is for trend weighting. It provides a foundation with the correct geometric structure for subsequent detailed supplementation.

[0060] Step S53, Neural-guided adaptive enhancement residual branch: Through mathematical transformation, high-dimensional, nonlinear neural features are mapped to the acoustic spectral space, serving as a priori corrector to repair local details in the output of the main decoding branch.

[0061] To correct potential semantic shifts in the main decoding branch, this embodiment introduces a neural-guided UNet.

[0062] First, through a fully connected layer... Perform a linear projection, mapping it to the feature latent space, to generate Dimensions The physical significance of this step is to explore the conversion of the electrical signal intensity of the brainwave channels into an energy distribution related to the spectral components. Remodeling The convolutional kernel generates neural guidance signals. .

[0063] This operation precisely aligns the time step from 9 steps to 8 steps of the target spectrum, ensuring strict synchronization between the neural constraint signal and the acoustic signal in the time domain.

[0064] Will Remodeling Same dimension, and Concatenate along the channel dimension (axis=-1): At this time It has two channels: acoustic contours (from the main decoding branch of the decoder) and neural constraints (from the projection of the original signal from the brain). This design allows for the correction of blurred frequencies generated by the main decoding branch by referencing the frequencies of the sound the brain originally intended to emit when repairing the spectrum.

[0065] Subsequently, residual features are extracted using the insulated blocks of the UNet structure. Local smoothing capture: Local time-frequency correlations are extracted through standard convolution. Long-range harmonic modeling: Introducing dilated convolution ( Expanding the sensory field: These features are fused using residual connections and convolutions to output the final repaired 128-dimensional spectral vector: To further improve the objective clarity of the spectrum, Adaptive enhancement processing is performed. First, energy centers within the neighborhood are identified using local max-pooling: Calculate the offset between the current pixel and the local energy center, and generate enhancement weights using Sigmoid gating. : Final Enhanced Residual Term Expressed as: in, This indicates element-wise multiplication. This mechanism imparts a 10% local gain to the residual, making the ridges in the speech spectrum sharper.

[0066] Ultimately, the main spectrum that maintains a linear trend will be preserved. With enhanced detail residuals Perform additive fusion to output a high-fidelity reconstructed spectrum. .

[0067] Step S6: To achieve high-fidelity constraints on the predicted spectrogram, this method constructs a multi-dimensional joint loss function that integrates multi-scale time-frequency features, global structural similarity, and local dynamic differences. Let the preprocessed Mel spectrogram be denoted as... The model predicts the Mel spectrum as follows: ,in For time frames, The number of frequency bands is given. The specific calculation steps and implementation methods of the joint loss function are as follows: Step S61, Multi-scale Time-Frequency Distance Loss. To constrain the energy distribution of the spectrum at different time resolutions, a multi-scale average pooling mechanism is first introduced. (The remaining text appears to be incomplete and requires further context.) Define the average pooling operator on the one-dimensional time axis. ,in The elements of the pooled matrix are calculated as follows: (Zero padding is performed on sequence elements that exceed the boundary). Let the spectral features at different scales be denoted as... Based on the above multi-scale characteristics, the spectral convergence (SC) loss and the magnitude loss (Magnitude Loss, Mag) are calculated: Spectral convergence loss: constrains the relative approximation of the predicted spectrum to the true spectrum in terms of energy norm. in, It is the Frobenius norm. This is a constant used to prevent division by zero from reaching a minimum value.

[0068] Absolute amplitude loss: Absolute amplitude error between constrained points: in, It is an L1 norm.

[0069] The final multi-scale time-frequency distance loss is the average of the three scales: pass{ With multi-scale temporal pooling, the model can not only align the microscopic single-frame frequency band energy, but also match the macroscopic prosodic envelope of speech in a larger temporal receptive field, effectively preventing excessive smoothing of the predicted spectrum.

[0070] Step S62, Global Structural Similarity Loss (SSIM Loss) will and It is treated as a two-dimensional single-channel grayscale image, and its topological features are constrained by the structural similarity index (SSIM).

[0071] The structural similarity index within a local window is defined as follows: in Let these be the local mean, variance, and covariance within the Gaussian sliding window, respectively. The global SSIM loss is defined as: Global structural similarity loss treats the spectrogram as a whole image, forcing the model to learn and preserve the unique global two-dimensional texture structure of the speech signal (such as harmonic lines of vowels and broadband noise blocks of consonants), which significantly improves the visual and auditory structural integrity of synthesized speech.

[0072] Step S63, Local First-Order Difference Loss. For fine edges in the speech spectrum that determine timbre and intelligibility (such as formant jumps and transient consonants), a first-order difference gradient constraint is introduced: Frequency domain difference (resonant constraint): Calculates the gradient matrix along the frequency axis. : .

[0073] The loss items are: Time-domain difference (transient change constraint): Calculate the gradient matrix along the time axis. : The loss items are: The total difference loss is the sum of the two items mentioned above: Differential operation is equivalent to a two-dimensional high-pass filter. It enhances the sharpness of the edges of the resonant peaks in the frequency domain (avoiding frequency axis blurring). This constraint improves the accuracy of transient transitions such as onset and end of sounds in the time domain. This significantly improves the objective intelligibility metric for predicted speech.

[0074] Step S64: Final Joint Loss Function. This is the total joint loss function used to guide network backpropagation and parameter updates. Configured as a weighted sum of the above items: Where 0.2 is the balancing weight for structural loss. The invention constructs... A three-dimensional supervised constraint network consisting of "macroscopic energy envelope, global topology, and microscopic edge texture" was formed. This multi-dimensional collaborative optimization mechanism completely solves the technical problems commonly encountered in EEG decoding, such as over-smoothing of predicted spectra and loss of high-frequency transient details, and achieves high-fidelity conversion of EEG to speech features.

[0075] Step S7: The Mel spectrogram generates speech using an iterative harmonic phase reconstruction vocoder.

[0076] The results of generating Mel spectrograms on the test set are shown in Table 1.

[0077] Table 1 Comparison of Model Reconstruction Mel-spectrum Plot Performance

[0078] As shown in Table 1, compared with linear regression, the average PCC of all participants in this example increased from 0.7208 to 0.9667. Compared with CNN, the average PCC and average STGI increased from 0.9109 and 0.8950 to 0.9667 and 0.9516, respectively. Furthermore, compared with NeuroIncept Decoder in 2025, the average PCC and average STGI improved by 0.0323 and 0.0501, respectively. The model performance has been significantly improved.

[0079] In another aspect, the present invention also provides an EEG speech generation system that integrates soft wavelet multi-scale feature extraction and task decoupling, for implementing the aforementioned EEG speech generation method, comprising an EEG signal module, a multi-scale EEG feature module, a global EEG feature module, an aligned EEG feature module, a Mel spectrogram module, a Mel spectrogram module, and an EEG speech generation module: The EEG signal module is used to collect the EEG signals of the subject reading aloud and record the speech simultaneously; the EEG signals are preprocessed to obtain preprocessed EEG features, and the speech is preprocessed to obtain a preprocessed Mel spectrogram.

[0080] The multi-scale EEG feature module is used to extract features from the preprocessed EEG features using soft wavelet thinning and a multi-scale dilated convolution feature extractor to obtain multi-scale EEG features.

[0081] The global EEG feature module is used to encode the location of multi-scale EEG features and then output the global EEG features sequentially through bidirectional Mamba, Conformer and bidirectional Mamba.

[0082] The aligned EEG feature module is used to construct a bridging mechanism to align neuroacoustic signals for global EEG features and generate aligned EEG features.

[0083] The Mel spectrogram module generates Mel spectrograms based on aligned EEG features and through a task-decoupled multi-branch spectrogram generation decoder.

[0084] The EEG speech generation module is used to generate speech from the Mel spectrogram generated by the decoder through an iterative harmonic phase reconstruction vocoder, and simultaneously combine the preprocessed Mel spectrogram to construct a joint loss function for training and optimization.

[0085] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A brainwave speech generation method integrating soft wavelet multi-scale feature extraction and task decoupling, characterized in that, Includes the following steps: Step S1: Collect the EEG signals of the subject reading aloud and record the speech simultaneously; Preprocessing of EEG signals yields preprocessed EEG features, and preprocessing of speech yields preprocessed Mel spectrograms. Step S2: The preprocessed EEG features are extracted using soft wavelet thinning and a multi-scale dilated convolution feature extractor to obtain multi-scale EEG features; Step S3: After location encoding of the multi-scale EEG features, output the global EEG features sequentially through bidirectional Mamba, Conformer and bidirectional Mamba. Step S4: Construct a bridging mechanism to align neuroacoustic signals based on global EEG features and generate aligned EEG features: Step S5: Based on aligned EEG features, generate Mel spectrograms using a multi-branch spectrogram generator decoder with task decoupling; Step S6: The Mel spectrogram generated by the decoder is used to generate speech through the iterative harmonic phase reconstruction vocoder. At the same time, a joint loss function is constructed by combining the preprocessed Mel spectrogram and then trained and optimized.

2. The EEG speech generation method according to claim 1, which integrates soft wavelet multi-scale feature extraction and task decoupling, is characterized in that... In step S2, the implementation process of the soft wavelet thinning and multi-scale dilated convolution feature extractor is as follows: Targeting preprocessed EEG features A comprehensive feature representation is achieved by using a five-branch parallel soft wavelet refinement and a multi-scale dilated convolution feature extractor. The first and second branches are the baseline branches: they utilize average pooling to capture the low-frequency trend of the background field potential, and through... Convolution preserves the identity mapping of the original resolution; The third and fourth branches are temporally dependent branches: dilated convolutions are introduced to expand the receptive field; Soft wavelet refinement branch: Simulates wavelet decomposition filters using predefined convolutional kernel groups to refine EEG features. Decomposed into low-frequency approximate components With high-frequency detail components : Introducing a learnable gating weight matrix This is used to dynamically adjust the contribution of the two types of components in the final representation, based on EEG characteristics. Generate gate weights Finally, the refined time-frequency features are output through a weighted fusion method. : This represents element-wise multiplication; The soft wavelet refinement branch and the features output from the other four branches are concatenated along the channel dimension to form a multi-scale feature matrix. Subsequently, the fused features are recalibrated in terms of spatial and channel dimensions using a convolutional block attention model (CBAM) to generate multi-scale EEG features. .

3. The EEG speech generation method according to claim 2, which integrates soft wavelet multi-scale feature extraction and task decoupling, is characterized in that... The specific implementation of step S4 is as follows: the global EEG features are enhanced by multi-head self-attention to obtain enhanced global EEG features. Then, a learnable acoustic prior query that is completely matched with the input time sequence and dimension is introduced as an anchor point. Cross-modal attention is performed with the acoustic prior as the Query and the enhanced global EEG features as the Key / Value to obtain aligned features. Finally, aligned EEG features are generated by fusing enhanced global EEG features with learnable gated residuals.

4. The EEG speech generation method integrating soft wavelet multi-scale feature extraction and task decoupling according to claim 3, characterized in that, Step S5 is implemented as follows: Decoder receives aligned EEG features Mel spectrum diagrams are generated using a three-branch decoupled architecture: Linear trend branches: By capturing the macroscopic trend of speech energy distribution through global average pooling and linear transformation, features are obtained. ; The main decoding branch of spatial conservation: aligning EEG features Remodeling into 2D spatial features ; By maintaining resolution convolution kernels Perform mapping to generate the main spectrum : in Trend weighting; Neural-guided contrast enhancement residual branch: Through mathematical transformation, neural features are mapped to the acoustic spectral space to obtain the restored spectral vector. ; Finally, adaptive boosting weights are generated using local max pooling for the spectrum vector. After contrast enhancement, compared with the main spectrum The final spectrum is output by addition and fusion. .

5. The EEG speech generation method according to claim 4, which integrates soft wavelet multi-scale feature extraction and task decoupling, is characterized in that... The neurally guided contrast enhancement residual branch is specifically implemented as follows: firstly, it is passed through a fully connected layer... Perform a linear projection to map it onto the feature latent space to generate ;Will Remodeling The convolutional kernel generates neural guidance signals. ; Then Remodeling Same dimension, and By splicing along the channel dimension, we obtain... At this time It has two channels: acoustic contouring and neural constraint; Subsequently, convolution is used to capture local spectral smoothness, resulting in features. Hollow convolution expands the receptive field, capturing long-range harmonic relationships to obtain features. Features are combined using residual connections and convolution. and The spectrum vector is fused and output as the final repaired spectrum vector. .

6. A brainwave speech generation system integrating soft wavelet multi-scale feature extraction and task decoupling, used to implement the brainwave speech generation method according to any one of claims 1 to 5, characterized in that, Includes the following modules: The EEG signal module is used to collect the EEG signals of the subject reading aloud and to record the speech simultaneously; Preprocessing of EEG signals yields preprocessed EEG features, and preprocessing of speech yields preprocessed Mel spectrograms. The multi-scale EEG feature module is used to extract features from the preprocessed EEG features using soft wavelet thinning and a multi-scale dilated convolution feature extractor to obtain multi-scale EEG features. The global EEG feature module is used to encode the location of multi-scale EEG features and then output the global EEG features through bidirectional Mamba, Conformer and bidirectional Mamba in sequence. The aligned EEG feature module is used to construct a bridging mechanism to align neuroacoustic signals based on global EEG features and generate aligned EEG features. The Mel spectrogram module generates Mel spectrograms based on aligned EEG features through a task-decoupled multi-branch spectrogram generation decoder. The EEG speech generation module is used to generate speech from the Mel spectrogram generated by the decoder through an iterative harmonic phase reconstruction vocoder. At the same time, it combines the preprocessed Mel spectrogram to construct a joint loss function for training and optimization.

Images