Speech reconstruction method and device based on intracranial neural electrical signals and electronic equipment

Abstract

The present application relates to a kind of speech reconstruction methods, device and electronic equipment based on intracranial nerve electric signal, the method comprises: the neural feature time sequence stream corresponding to the intracranial nerve electric signal continuously collected by the language-related brain area of subject is acquired;Neural feature time sequence stream is input to a flow causal decoding model, to generate corresponding speech acoustic feature time sequence stream in real time;Wherein, flow causal decoding model is the causal deep neural network model trained based on sample data, and sample data includes sample neural signal feature sequence and its corresponding sample speech acoustic feature sequence;Speech acoustic feature time sequence stream is integrated into speech waveform stream and output.The present application realizes real-time speech synthesis of millisecond level delay by flow causal decoding model and dynamic block strategy, solves the high delay problem of existing scheme, simultaneously avoids auditory feedback interference by causal mask mechanism, can adapt to the application demand in silent scene.

Description

Technical Field

[0001] This invention relates to the fields of brain-computer interface and biomedical signal processing technology, and in particular to a speech reconstruction method, device and electronic device based on intracranial neural electrical signals. Background Technology

[0002] In the field of brain-computer interface speech decoding, offline processing based on whole-sentence data is commonly used to improve the accuracy of synthesized speech. A typical approach aligns whole-sentence speech using dynamic time warping algorithms and introduces speech organ motion data as intermediate constraints to assist decoding. This approach often employs network structures that allow the use of complete sequence context information, such as bidirectional recurrent neural networks or models with global attention mechanisms. However, this approach has inherent drawbacks in practical applications: because the model can access and utilize neural signals from future moments when decoding the current moment's speech, these future signals may contain auditory cortical feedback information generated by the speaker hearing their own voice. This causes the model's learned mapping to confuse the brain's motor feedforward control signals for speech production with the auditory feedback response signals. When this technology is applied to silent speech scenarios where no sound can be produced, or in noisy environments where auditory feedback is missing or distorted, the model's decoding performance significantly degrades due to the absence of the relied-upon feedback signals. Therefore, the core problem of causal confusion in the decoding process in existing solutions limits their robustness and practicality in real-world scenarios. Summary of the Invention

[0003] Therefore, it is necessary to address the core problem of causal confusion in the decoding process of existing solutions, which limits their robustness and practicality in real-world scenarios, and to provide a speech reconstruction method, device, and electronic device based on intracranial neural electrical signals.

[0004] This invention provides a speech reconstruction method based on intracranial neural electrical signals, the method comprising: Acquire the temporal flow of neural features corresponding to the continuous intracranial neural electrical signals collected from the language-related brain regions of the subjects; The neural feature time-series stream is input into a streaming causal decoding model to generate the corresponding speech acoustic feature time-series stream in real time. The streaming causal decoding model is a causal deep neural network model trained on sample data, which includes sample neural signal feature sequences and their corresponding sample speech acoustic feature sequences. The model is trained by constraining it to predict the sample speech acoustic features at the current moment based only on the sample neural signal features at the current moment and historical moments in the input sequence. The temporal stream of the acoustic features of the speech is synthesized into a speech waveform stream and output.

[0005] In one embodiment, the neural feature temporal stream is obtained by fusing multi-band neural features, specifically including: High-frequency envelope features in the 70-150 Hz band and low-frequency waveform features in the 0.5-30 Hz or 0.5-4 Hz bands are extracted in parallel from the intracranial neural electrical signals, and the two are spliced ​​together in the channel dimension.

[0006] In one embodiment, the streaming causal decoding model implements the constraint by introducing a causal masking mechanism in the self-attention computation layer of its neural network, the causal masking being used to physically shield neural signal feature information from future time steps.

[0007] In one embodiment, the streaming causal decoding model is a model trained on a generative adversarial network architecture, wherein the generator is used to perform the mapping of the neural feature temporal stream to the speech acoustic feature temporal stream, and the architecture includes a multi-scale discriminator for distinguishing the authenticity of acoustic features output by the generator at different resolutions.

[0008] In one embodiment, the backbone network of the streaming causal decoding model is composed of stacked streaming Conformer modules containing causal mask multi-head self-attention layers and convolutional layers.

[0009] In one embodiment, the streaming causal decoding model is trained using a dynamic block strategy, that is, data blocks of random duration are used during the training phase, and data blocks of preset fixed duration are used to stream the neural feature time-series stream during the inference phase.

[0010] In one embodiment, the speech corresponding to the acoustic feature sequence of the sample speech is synthesized speech generated through the following process: collecting reference speech of the target user; training a timbre cloning model based on the reference speech; and using the timbre cloning model to synthesize speech with the timbre of the target user based on a preset text.

[0011] In one embodiment, the intracranial neuroelectrical signals are acquired by any one of the intracranial electrodes selected from cortical EEG electrode arrays, stereotactic EEG electrodes, and microelectrode arrays.

[0012] In one embodiment, the speech content corresponding to the reconstructed speech waveform stream includes a single standard language speech, and covers cross-language and cross-dialect speech information, and has the potential to extend to non-language speech information.

[0013] The present invention also provides a speech reconstruction device based on intracranial neural electrical signals, the device comprising: The neural signal acquisition and preprocessing module is used to acquire the temporal stream of neural features corresponding to the intracranial neural electrical signals continuously acquired from the language-related brain regions of the subject; A streaming causal decoding module is used to input the neural feature time-series stream into a streaming causal decoding model to generate the corresponding speech acoustic feature time-series stream in real time. The streaming causal decoding model is a causal deep neural network model trained on sample data, which includes sample neural signal feature sequences and their corresponding sample speech acoustic feature sequences. The model is trained by constraining it to predict the sample speech acoustic features at the current moment based only on the sample neural signal features at the current moment and historical moments in the input sequence. The speech synthesis module is used to synthesize the temporal stream of the speech acoustic features into a speech waveform stream and output it.

[0014] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the speech reconstruction method based on intracranial neural electrical signals as described above.

[0015] The aforementioned speech reconstruction method, device, and electronic device based on intracranial neural electrical signals acquire a temporal stream of neural features corresponding to continuously collected intracranial neural electrical signals from language-related brain regions of the subject. This temporal stream is then input into a streaming causal decoding model to generate a corresponding temporal stream of speech acoustic features in real time. The decoding model is a causal deep neural network model trained on sample data. It is trained by constraining the model to predict the acoustic features of the current sample speech based solely on the neural signal features of the current and historical time samples in the input sequence. This physically shields the input of neural signals from future time moments during the model inference stage, ensuring that the decoding process strictly relies on the brain's motor feedforward control commands. This effectively solves the causal confusion problem caused by existing non-causal decoding models that utilize neural signals containing future auditory feedback. Furthermore, by streaming continuous neural signals and generating acoustic feature streams in real time, this method eliminates the need for offline alignment and processing of entire speech sentences, achieving millisecond-level low-latency synthesis from neural signals to speech waveforms. This overcomes the high latency and inability to interact in real time inherent in existing technologies. Furthermore, this scheme directly establishes a mapping relationship from neural features to speech acoustic features through an end-to-end deep neural network, avoiding the introduction of intermediate constraints such as speech organ movement data and complex inference processes. While simplifying the system structure and improving versatility and robustness, it combines an advanced neural network architecture to ultimately synthesize speech with higher naturalness and clarity, thus providing a real-time, reliable, and natural means of speech communication reconstruction for aphasic patients. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart of a speech reconstruction method based on intracranial neural electrical signals, as an example. Figure 2 Flowchart of a streaming causal EEG speech reconstruction system; Figure 3 This is a schematic diagram of a speech reconstruction device based on intracranial neural electrical signals, according to one embodiment. Figure 4 This is an internal structural diagram of an electronic device according to one embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The following is combined with Figures 1-4 This invention describes a speech reconstruction method, apparatus, and electronic device based on intracranial neural electrical signals.

[0020] like Figure 1 and Figure 2 As shown, in one embodiment, a speech reconstruction method based on intracranial neural electrical signals aims to establish an end-to-end mapping from brain neural activity to speech output, achieving low-latency, high-fidelity real-time speech synthesis. The method includes the following steps: Step S110: Obtain the temporal stream of neural features corresponding to the intracranial neural electrical signals continuously collected from the language-related brain regions of the subject.

[0021] Specifically, subjects typically refer to patients with implanted intracranial electrodes, whose language-related brain regions include, but are not limited to, Broca's area, the motor cortex, or the sensorimotor cortex—areas closely related to language production. Intracranial neuroelectrical signals are the electrical activity signals of brain neuronal groups directly recorded by implanted electrodes (such as ECoG electrocorticography arrays). Neural feature temporal streams refer to the continuously changing feature sequences extracted from the raw electrical signals after preprocessing (such as filtering and downsampling). These feature sequences reflect the neurodynamic processes of the brain during the generation of language intentions. Because the signals are continuously acquired, the corresponding neural feature temporal streams possess continuous temporal attributes, providing a data foundation for subsequent streaming processing.

[0022] Step S120: The neural feature time-series stream is input into the streaming causal decoding model to generate the corresponding speech acoustic feature time-series stream in real time. The streaming causal decoding model is a causal deep neural network model trained on sample data. The sample data includes sample neural signal feature sequences and their corresponding sample speech acoustic feature sequences. The model is trained by constraining it to predict the sample speech acoustic features at the current moment based solely on the sample neural signal features from the current and historical moments in the input sequence.

[0023] Specifically, streaming refers to a model's ability to process continuously input data streams and output results simultaneously with or after a very short delay upon receiving data, without waiting for the entire data acquisition process to complete, thus achieving real-time processing. Causality means that when calculating the output at the current moment, the model's computational logic is physically restricted to using only the input data from the current moment and previous historical moments, strictly shielding it from data information from future moments. This causal constraint ensures that the model simulates the brain's feedforward motor commands for generating speech, rather than the auditory feedback after hearing one's own voice. During the training phase, a sample dataset containing sample neural signal feature sequences (input) and sample speech acoustic feature sequences (labels) is constructed, and the model is forced to focus only on the current frame and previous input frames when predicting the label of the current frame, thereby training a deep neural network model that conforms to causal constraints. The temporal stream of speech acoustic features refers to the intermediate feature sequence output by the model that characterizes the acoustic properties of speech (such as Mel spectrum, fundamental frequency, etc.), and these features will be used in subsequent waveform synthesis.

[0024] Step S130: Synthesize the temporal stream of speech acoustic features into a speech waveform stream and output it.

[0025] Specifically, this step is typically performed by a vocoder. The vocoder receives a sequence of acoustic features generated in real time by a decoding model and converts it into a playable time-domain speech waveform stream. Because the input acoustic features are streaming, the output speech waveform stream is also continuously generated, thus achieving real-time conversion from neural signals to speech waveforms. The final output speech waveform stream can be played through a speaker, thereby helping aphasic patients regain their ability to communicate verbally.

[0026] This embodiment achieves millisecond-level low-latency real-time speech synthesis through a streaming causal decoding model and a dynamic block processing mechanism, meeting the patient's interactive need for "speaking as soon as they want to speak." At the same time, through strict causal constraints, auditory feedback interference in future signals is physically shielded, enabling the model to maintain high robustness in silent speech or noisy environments, thus solving the problem of performance degradation of existing non-causal models in real-time applications.

[0027] In one embodiment, the neural feature time-series stream is obtained by fusing multi-band neural features, specifically including: extracting high-frequency envelope features of the 70-150Hz band and low-frequency waveform features of the 0.5-30Hz or 0.5-4Hz band in parallel from intracranial neural electrical signals, and splicing the two together in the channel dimension.

[0028] Specifically, after acquiring continuous intracranial neural electrical signals from the subject's language-related brain regions, this embodiment employs parallel processing to extract features from two complementary frequency bands in order to construct input features that comprehensively reflect the brain's language control mechanism. First, for the high-frequency band (70-150Hz), the analytical amplitude, i.e., the high-frequency envelope feature, is extracted using signal processing methods such as Hilbert transform. This band is commonly referred to as the High-Gamma band, and its physical significance lies in reflecting the firing intensity and synchronization degree of local neuronal populations. It primarily encodes the articulatory motion information of language and makes a crucial contribution to the accuracy and clarity of speech content. Second, for the low-frequency band (0.5-30Hz or 0.5-4Hz), the original waveform after bandpass filtering is retained, i.e., the low-frequency waveform feature. This band corresponds to slow cortical potentials. Although its specific mechanism is still under investigation, existing evidence suggests that it is correlated with higher cognitive functions such as prosody and pitch variation in speech, providing macroscopic information about speech rhythm and intonation.

[0029] After extracting the two types of features mentioned above, this embodiment concatenates them along the channel dimension. Assuming the effective physical number of channels of the original intracranial electrodes is N, the high-frequency envelope features form a matrix of dimension [N,T], and the low-frequency waveform features form a matrix of dimension [N,T], where T is the number of time frames. By stacking along the channel dimension, an expanded input feature matrix of dimension [2N,T] is formed. The physical significance of this concatenation method lies in preserving the differences in spatial topology between the two signals; that is, high-frequency and low-frequency features may correspond to different cortical functional areas in spatial distribution. By expanding the channel dimension rather than simply superimposing numerical values, the deep neural network model can learn and utilize the complementary neurodynamic information contained in these two frequency bands. Experimental data shows that compared to using only a single feature from the High-Gamma band, this multi-band fusion strategy can significantly improve the model's decoding performance of speech content and prosody, thereby improving the naturalness and intelligibility of the reconstructed speech.

[0030] In one embodiment, the streaming causal decoding model implements constraints by introducing a causal masking mechanism into the self-attention computation layer of its neural network. The causal masking is used to physically shield neural signal feature information from future time steps. Meanwhile, the backbone network of the streaming causal decoding model is constructed from stacked streaming Conformer modules containing causal masking multi-head self-attention layers and convolutional layers.

[0031] Specifically, to construct a decoding model capable of simultaneously capturing local temporal features and global long-range dependencies while strictly satisfying causal constraints, this embodiment employs a streaming Conformer architecture as the backbone network. This backbone network consists of multiple stacked streaming Conformer modules; for example, the number of stacked layers N can be set to 8 or more, and the specific number of layers can be adjusted according to model complexity and data volume. Each streaming Conformer module internally adopts a "sandwich" stacked structure, sequentially containing a feedforward neural network module, a multi-head self-attention module, a convolutional module, and a second feedforward neural network module.

[0032] In the multi-head self-attention module, a causal masking mechanism is introduced to achieve streaming causal processing. When calculating the self-attention score, the dot product of the query matrix and the key matrix is ​​typically used to obtain the attention score matrix. In this embodiment, a causal masking matrix is ​​superimposed on this score matrix. This causal masking matrix is ​​designed as an upper triangular matrix (with or without a diagonal, depending on whether the current time step is allowed to focus on itself), where positions corresponding to future time steps are filled with negative infinity (or minimum values). After Softmax normalization, the weights at these positions approach zero, thus physically masking the neural signal features from future time steps. This means that when calculating the output at the current time step, the model can only aggregate the input features from the current and historical time steps, strictly preventing information leakage and ensuring the causality of the decoding process.

[0033] Convolutional modules typically consist of one-dimensional deep convolutional layers and pointwise convolutional layers. Deep convolutional layers extract features from local neighborhoods along the time axis, capturing short-term patterns of change in neural signals; pointwise convolutional layers are used for feature mixing between channels. By combining convolutional modules with self-attention modules, this architecture leverages the local inductive bias of convolution to capture high-frequency details of local neural activity, while utilizing the global receptive field of the self-attention mechanism to model long-distance associations in speech sequences (such as coarticulation relationships between syllables). This combination significantly improves the ability to model complex neurodynamic signals while maintaining streaming processing and causal constraints, thereby enhancing the accuracy and naturalness of speech reconstruction.

[0034] In one embodiment, the streaming causal decoding model is a model trained on a generative adversarial network architecture, wherein the generator is used to perform the mapping of the neural feature temporal stream to the speech acoustic feature temporal stream, and the architecture includes a multi-scale discriminator for distinguishing the authenticity of acoustic features output by the generator at different resolutions.

[0035] Specifically, to address the issue of traditional regression models easily leading to overly smoothed spectra in speech reconstruction, resulting in synthesized speech sounding blurry and lacking high-frequency texture details, this embodiment employs a Generative Adversarial Network (GAN) architecture to train the streaming causal decoding model. This architecture mainly consists of a generator and a discriminator. The generator is the streaming causal Conformer network described in the previous embodiment, responsible for mapping the input neural feature time-series to the predicted speech acoustic feature time-series (e.g., Mel spectrogram). The discriminator uses a multi-scale discriminator (MSD) structure, designed to distinguish between genuine and fake acoustic features output by the generator from different resolutions and receptive fields, thereby forcing the generator to produce more realistic speech details.

[0036] The multi-scale discriminator comprises multiple sub-discriminators operating in parallel, such as a first-level discriminator, a second-level discriminator, and a third-level discriminator. During training, the acoustic features output by the generator (i.e., the predicted Mel spectrogram) and the acoustic features of the real sample speech (i.e., the ground truth) are simultaneously input into the multi-scale discriminator. For the first-level discriminator, it directly processes the acoustic features at the original resolution, focusing on capturing high-frequency texture details in the spectrum, such as the fricatives of consonants or rapid changes in formants, which are crucial for the clarity and naturalness of speech. For the second- and third-level discriminators, the input acoustic features are first downsampled using operations such as average pooling, reducing them to different resolutions (e.g., 1 / 2 and 1 / 4 of the original size). The downsampled features have a larger receptive field, allowing these discriminators to focus on the global structure and macroscopic consistency of the spectrogram, such as the prosodic contours and energy distribution of the entire speech segment.

[0037] Through this coarse-to-fine multi-scale discrimination mechanism, the discriminator can simultaneously constrain the accuracy of the generator in both local details and global structure. During the adversarial training process, if the generator's output spectrum exhibits local ambiguity or excessive smoothness, the first-level discriminator can easily identify it as "fake." If the generator's output spectrum has incoherence or prosodic errors in its global structure, the downsampled discriminator will penalize it. This multi-scale adversarial loss signal propagates back to the generator, forcing it to not only fit the overall contour of realistic speech but also generate rich texture information in high-frequency details. Experiments show that the streaming causal decoding model trained with this multi-scale discriminator significantly outperforms traditional models trained with only L1 loss in terms of Mel-Cepstral Distortion (MCD) metrics, particularly in consonant intelligibility and background noise suppression, thus significantly improving the naturalness and intelligibility of the reconstructed speech.

[0038] In one embodiment, the streaming causal decoding model employs a dynamic block-based training strategy, which uses data blocks of random duration during the training phase and streams the neural feature time-series data blocks of preset fixed duration during the inference phase.

[0039] Specifically, the dynamic block segmentation strategy is the key mechanism in this embodiment for achieving a balance between low-latency streaming decoding and high-precision modeling. This strategy employs differentiated data processing logic during the model's training and inference phases. During training, to enhance the generalization ability and robustness of the streaming causal decoding model, the system uses data blocks of random duration as input. This means that when constructing training batches, the system does not always extract neural feature sequences of fixed length, but rather randomly samples block lengths within a preset duration range (e.g., 10 milliseconds to 300 milliseconds). This randomness forces the model to learn to extract effective neural features from local context windows of different lengths, thereby adapting to the temporal variations of different syllables and words in the speech signal. By exposing the model to varying context lengths during training, the model can better capture the nonlinear mapping relationship between neural signals and speech features, avoiding overfitting to a specific fixed window length, and thus improving the decoding accuracy of the model in complex speech scenarios.

[0040] During the inference phase, to meet the low-latency requirements of real-time interaction, the system switches to streaming processing of the neural feature time-series stream using preset fixed-duration data blocks. Specifically, the system divides the continuously acquired neural feature time-series stream into consecutive data blocks of a preset fixed duration (e.g., 50 milliseconds or 100 milliseconds). Whenever the accumulated data volume of a data block meets the preset duration requirement, that data block is immediately input into the streaming causal decoding model for computation. Because the data block length is fixed and short, the model can complete feature extraction and acoustic feature prediction within a very short time allowed by computing resources, and immediately output the results before processing the next data block. This "frame-by-frame" streaming processing mode eliminates the buffering time of waiting for the entire signal to be acquired, thus achieving millisecond-level end-to-end latency. Furthermore, since the model has already learned to adapt to different contexts during the training phase through a random block partitioning strategy, it can still maintain high decoding accuracy when using fixed-duration blocks during the inference phase, without significant performance degradation due to the fixed context window. By differentiating the training and inference phases, this embodiment effectively finds the optimal balance between computational resource consumption, decoding latency, and reconstruction accuracy.

[0041] In one embodiment, the speech corresponding to the acoustic feature sequence of the sample speech is synthesized speech generated through the following process: collecting reference speech of the target user; training a timbre cloning model based on the reference speech; and using the timbre cloning model to synthesize speech with the timbre of the target user based on a preset text.

[0042] Specifically, when constructing the training dataset for a streaming causal decoding model, obtaining high-quality sample speech acoustic feature sequences as ground truth is crucial. However, in real-world clinical applications, directly collecting subjects' speech as training labels often faces numerous challenges. For example, for patients with dysarthria such as amyotrophic lateral sclerosis (ALS), their own pronunciation may be unclear, making them difficult to use as accurate training targets. Furthermore, the intraoperative acquisition environment often suffers from background noise interference, resulting in a low signal-to-noise ratio and affecting model training performance. To address these challenges, this embodiment proposes a timbre-cloning-based synthetic speech generation scheme to construct high-quality training samples.

[0043] The process begins by collecting reference speech from the target user. For patients who retain some vocal ability, speech data of them clearly reading specific texts can be collected; for patients who have completely lost their speech, pre-illness recordings can be used as a reference source. The collected reference speech, after preprocessing, is used to train a voice cloning model. This voice cloning model can utilize existing end-to-end text-to-speech methods. (TTS) architectures, such as the VITS (Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech) model. The VITS model combines the advantages of variational inference (VAE) and generative adversarial networks (GAN), enabling it to efficiently extract and learn personalized acoustic attributes of the target user, such as timbre and breathing rhythm, from a small amount of reference speech.

[0044] After training the timbre cloning model, the system uses this model to synthesize speech with the target user's timbre based on preset experimental text content. The preset text typically covers a rich vocabulary and sentence structure required for model training. Because the synthesis process is completed in the digital domain, the generated speech has extremely high purity and fidelity, completely eliminating environmental background noise, and the pronunciation is standard and clear. Subsequently, the system extracts sample speech acoustic feature sequences (such as Mel spectrograms) from these synthesized speech samples as training labels for the streaming causal decoding model.

[0045] This embodiment not only solves the training data quality problem caused by unclear patient pronunciation or environmental noise, but also provides the model with a unified, standardized, and high-fidelity baseline ground truth. This synthetic speech-based training strategy can effectively reduce the learning difficulty of mapping neural signals to acoustic features, improve the convergence speed of the streaming causal decoding model and the accuracy of reconstructed speech, and ensure that the final output speech waveform stream highly reproduces the patient's own voice characteristics in terms of timbre.

[0046] In one embodiment, intracranial neuroelectrical signals are acquired by any one of the intracranial electrodes selected from cortical EEG electrode arrays, stereotactic EEG electrodes, and microelectrode arrays.

[0047] Specifically, the speech reconstruction method provided by this invention has broad hardware adaptability, compatible with different types of invasive neural signal acquisition devices to cover diverse clinical application needs. Electrocorticometry (ECoG) electrodes are typically placed directly on the surface of the cerebral cortex, capable of recording local field potentials over a large area, with high signal-to-noise ratio and spatial resolution, suitable for large-area signal acquisition covering the language motor cortex and sensory cortex. Stereotactic electroencephalography (SEEG) electrodes are implanted into deep brain structures via minimally invasive puncture, suitable for recording neural activity in deep brain regions such as the basal ganglia and hippocampus, and have unique advantages for decoding language emotion processing or deep motor circuits. Microelectrode arrays (such as UtahArray) can penetrate the cortex to record single neuron firing or local field potentials, providing extremely high-precision neural signals, suitable for scenarios requiring fine decoding of specific small nuclei. This invention, through a unified streaming causal decoding framework, can process signals acquired by the above-mentioned different electrodes. Only the preprocessing parameters (such as sampling rate and filtering range) need to be adjusted according to the electrode type to achieve the mapping from neural signals to speech, thus avoiding the limitation of the technical solution by specific hardware forms.

[0048] In one embodiment, the speech content corresponding to the reconstructed speech waveform stream includes a single standard language speech, and covers cross-language and cross-dialect speech information, and has the potential to extend to non-language speech information.

[0049] Specifically, the decoding target of this invention is the underlying motor control rules of the brain over the vocal organs (such as lips, tongue, palate, and larynx), rather than phonemes or grammatical features based on specific linguistics. Therefore, this method is not limited to any particular language. Experimental verification shows that this model is not only applicable to the reconstruction of standard languages ​​such as Mandarin, but can also faithfully reproduce the speech of various dialects and even other languages ​​(such as English), demonstrating highly reliable cross-linguistic universality. Furthermore, based on the aforementioned decoding mechanism of the underlying control rules of the full-frequency expression of the human vocal system, this solution further possesses the expanded space for reconstructing non-linguistic vocal information. For example, in scenarios involving fine adjustment of melody, pitch, and resonance, such as singing or humming, the streaming causal decoding model of this invention, by capturing temporal changes in high-frequency and low-frequency neural features, also has the capability to reconstruct vocal content with a sense of melody or specific rhythmic information. This comprehensive decoding architecture across languages ​​and modalities provides a more complete technical path to meet the diverse communication needs of patients in complex social environments.

[0050] In a preferred embodiment, a collaborative optimization method for EEG decoding speech synthesis is provided. This embodiment addresses the technical obstacles faced by existing technologies in achieving a synergistic improvement in low latency, high accuracy, and high naturalness through a series of interrelated improvement schemes.

[0051] Specifically, during the training phase, an adaptive dynamic segmentation strategy based on neural activity intensity is employed to overcome the fragmentation problem caused by random segmentation on pre-activation signals of neurons before speech initiation. This strategy first processes the input High-Gamma neural features... Calculate its short-time energy sequence ,in For the number of electrodes, This represents the number of time steps. Based on this energy sequence, the global mean is calculated. with standard deviation And set dynamic thresholds accordingly. During training, the system monitors the energy value at every moment in real time. and in accordance with the rules Dynamically adjust the block length, where , When energy exceeds a threshold, it indicates a high-intensity region of neural activity, typically corresponding to the start of speech or stress. In this case, short blocks are used to precisely capture rapidly changing neural patterns; conversely, long blocks are used to utilize broader contextual information to improve decoding stability. To smoothly handle potential discontinuities at the boundaries of high and low activity regions, this method further introduces an overlapping block strategy with a 50% overlap rate and performs linear weighted fusion on the outputs of the overlapping regions. This series of operations enables the model to more accurately locate and utilize neural preparatory potentials before the start of speech, thereby reducing the delay in determining the speech start point during decoding and improving the accuracy of first-word decoding. Experiments show that compared to uniform random blocks, this strategy reduces the speech start delay by 42% while improving the first-word recognition accuracy by 18.5%.

[0052] At the model architecture level, this embodiment designs a spatially-temporally separated causal Conformer module to better handle the irregular spatial distribution and spatiotemporal dynamics of intracranial electrode signals. This module decouples traditional convolution operations into two specialized sub-layers. The first is a spatial convolution sub-layer, which constructs a graph structure using the two-dimensional physical coordinates of the electrodes. This graph connects only those with a physical distance less than [a certain value]. Electrode pairs, thus forming a Graph Laplace matrix. Spatial features are obtained through graph convolution operations. Extraction is performed, among which For degree matrix, For learnable spatial filter weights, The activation function is nonlinear. This design forces the model to aggregate only information from spatially adjacent electrodes, which aligns with the physiological principle of functional localization in the cerebral cortex and helps maintain the spatial topological consistency of features. The second layer is the temporal causal self-attention sublayer, which, based on spatial features, strictly applies a causal self-attention mechanism along the temporal dimension. ,in These are query, key, and value matrices, respectively. The dimension of the key vector; For causal masking matrix, causal mask Defined as This ensures that at any point in time... The output relies only on current and past information, eliminating the leakage of future information, which is crucial for real-time streaming decoding. Finally, a learnable gating parameter is used. Spatial features With time series characteristics Perform dynamic fusion: This split design avoids the spatial topology information destruction and parameter redundancy problems caused by treating electrodes as regular grids for fully connected attention calculations. As a result, while maintaining decoding accuracy, it significantly reduces the false activation rate across cortical regions (experiments show a 35% reduction) and the number of model parameters (experiments show a 40% reduction).

[0053] To coordinate the processing of heterogeneous information carried by different frequency bands in neural signals (such as prosody encoded by LFS in the low-frequency band and pronunciation details encoded by HG in the high-frequency band), this embodiment proposes a multi-band progressive fusion and gating modulation mechanism. This method constructs a dual-branch parallel processing network: the low-frequency branch uses LFS features... As input, long-term prosodic contour features are extracted through a multi-layer causal temporal convolutional network. The high-frequency branch is characterized by HG. As input, the aforementioned spatial-temporal separation Conformer module extracts refined pronunciation content features. The features from the two branches are not simply concatenated at the input layer, but rather fused at a high-level representation level. This fusion is achieved through an adaptive gating system. To achieve this, the gating is determined by the characteristics of two branches: ,in The Sigmoid function is used. The final fused features are calculated as follows: The key lies in gating. It is a dynamic weight that changes over time and can be automatically adjusted according to the speech characteristics at the current moment: at the beginning of speech or in consonant segments, gating tends to assign features to the pronunciation content. Higher weighting is applied to improve the accuracy of content decoding; prosodic features are assigned at the end of sentences or in vowel extensions. Higher weights are applied to improve the prosodic naturalness of synthesized speech. This phased, dynamically weighted fusion strategy effectively avoids interference between high and low frequency information caused by early fusion, thereby reducing Mel-spectral distortion in objective metrics and improving the naturalness of speech in subjective listening experience, especially showing a significant improvement in intonation restoration for tonal languages.

[0054] Furthermore, to improve the robustness of the model in real-world, non-ideal environments and optimize computational efficiency, this embodiment provides an online electrode channel selection method based on causal gradients. After model training, validation set data is used to evaluate the actual contribution of each electrode channel to the decoding output. The contribution evaluation is based on the gradient magnitude of the loss function with respect to the input features. Calculations are performed, in which, For the first The gradient magnitude of each electrode, Let Mel-Cepstral Distortion be the loss function. For the first Each electrode in time The input features. To ensure the causality of the evaluation, gradient calculation is strictly limited to the current time step. and a previous causal window This excludes electrodes that might be misjudged as important due to future information leaks (e.g., electrodes that only respond to auditory feedback). Based on this, a spatiotemporal saliency score is calculated for each electrode. ,in It is a time-weighted factor used to emphasize contributions during speech activity. During the inference phase, a differentiable soft masking mechanism is introduced to modulate the input features online: ,in For the Sigmoid function, For the preset threshold, This refers to the temperature coefficient. During the training and fine-tuning phase, a smaller coefficient is used. The value preserves the differentiability of the mask; during final inference, a maximal value is used. The value makes the mask approach a binary state of 0 or 1, thus achieving deterministic channel selection. Furthermore, during online decoding, the system can periodically recalculate gradients and update saliency scores using recent data, thereby adaptively tracking the non-stationary characteristics of neural signals caused by biological drift or environmental changes. This method effectively suppresses interfering electrode signals unrelated to motor intent, especially in scenarios lacking auditory feedback, such as silent vocalization, significantly improving decoding accuracy and stability.

[0055] This embodiment utilizes the synergistic application of techniques such as adaptive dynamic block training, spatial-temporal separation modeling, multi-band feature fusion, dynamic block training, causal Conformer temporal modeling, and multi-scale adversarial training to work together on the EEG speech decoding system, achieving a balanced and significant improvement in multiple dimensions including latency, accuracy, naturalness, and robustness. Experimental results show that the system effectively achieves low-latency streaming conversion from continuous neural signals to high-fidelity speech waveforms. The multi-band fusion and the introduction of adversarial networks bring quantifiable improvements in objective indicators (such as Mel-spectral distortion) and subjective naturalness. More importantly, based on a strictly causal architecture, this solution not only exhibits high accuracy in current speech decoding tasks but also theoretically possesses the potential to maintain decoding robustness in environments without auditory feedback or noisy environments (such as silent speech). This provides an effective and comprehensive end-to-end solution to address the technical challenge of simultaneously achieving low latency, high fidelity, and interference resistance in existing technologies.

[0056] The speech reconstruction device based on intracranial neural electrical signals provided by the present invention will be described below. The speech reconstruction device based on intracranial neural electrical signals described below and the speech reconstruction method based on intracranial neural electrical signals described above can be referred to in correspondence.

[0057] like Figure 3 As shown, in one embodiment, a speech reconstruction device based on intracranial neural electrical signals includes a neural signal acquisition and preprocessing module 310, a streaming causal decoding module 320, and a speech synthesis module 330.

[0058] The neural signal acquisition and preprocessing module 310 is used to acquire the neural feature time-series flow corresponding to the intracranial neural electrical signals continuously acquired from the language-related brain regions of the subject.

[0059] The streaming causal decoding module 320 is used to input the neural feature time-series stream into the streaming causal decoding model to generate the corresponding speech acoustic feature time-series stream in real time. The streaming causal decoding model is a causal deep neural network model trained on sample data. The sample data includes sample neural signal feature sequences and their corresponding sample speech acoustic feature sequences. The model is trained by constraining it to predict the sample speech acoustic features at the current moment based only on the sample neural signal features at the current moment and historical moments in the input sequence.

[0060] The speech synthesis module 330 is used to synthesize the temporal stream of the speech acoustic features into a speech waveform stream and output it.

[0061] Figure 4 This example illustrates a schematic diagram of the physical structure of an electronic device, which can be a smart terminal. Its internal structure diagram can be as follows: Figure 4As shown, the electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements the speech reconstruction method based on intracranial neural electrical signals according to any of the above embodiments.

[0062] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the electronic device to which the present invention is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0063] On the other hand, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements the speech reconstruction method based on intracranial neural electrical signals of any of the above embodiments.

[0064] In another aspect, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, it implements the speech reconstruction method based on intracranial neural electrical signals according to any of the above embodiments.

[0065] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0066] By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0067] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0068] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A speech reconstruction method based on intracranial neural electrical signals, characterized in that, The method includes: Acquire the temporal flow of neural features corresponding to the continuous intracranial neural electrical signals collected from the language-related brain regions of the subjects; The neural feature time-series stream is input into a streaming causal decoding model to generate the corresponding speech acoustic feature time-series stream in real time. The streaming causal decoding model is a causal deep neural network model trained on sample data, which includes sample neural signal feature sequences and their corresponding sample speech acoustic feature sequences. The model is trained by constraining it to predict the sample speech acoustic features at the current moment based only on the sample neural signal features at the current moment and historical moments in the input sequence. The temporal stream of the acoustic features of the speech is synthesized into a speech waveform stream and output.

2. The speech reconstruction method based on intracranial neural electrical signals according to claim 1, characterized in that, The neural feature time-series stream is obtained by fusing multi-band neural features, specifically including: High-frequency envelope features in the 70-150 Hz band and low-frequency waveform features in the 0.5-30 Hz or 0.5-4 Hz bands are extracted in parallel from the intracranial neural electrical signals, and the two are spliced ​​together in the channel dimension.

3. The speech reconstruction method based on intracranial neural electrical signals according to claim 1, characterized in that, The streaming causal decoding model implements the constraint by introducing a causal masking mechanism in the self-attention computation layer of its neural network, the causal masking being used to physically shield neural signal feature information from future time steps.

4. The speech reconstruction method based on intracranial neural electrical signals according to claim 1 or 3, characterized in that, The streaming causal decoding model is a model trained on a generative adversarial network architecture. Its generator is used to perform the mapping of the neural feature temporal stream to the speech acoustic feature temporal stream, and the architecture includes a multi-scale discriminator for distinguishing the authenticity of acoustic features output by the generator at different resolutions.

5. The speech reconstruction method based on intracranial neural electrical signals according to claim 1, characterized in that, The backbone network of the streaming causal decoding model is composed of stacked streaming Conformer modules containing causal mask multi-head self-attention layers and convolutional layers.

6. The speech reconstruction method based on intracranial neural electrical signals according to claim 1, characterized in that, The streaming causal decoding model is trained using a dynamic block strategy, that is, data blocks of random duration are used during the training phase, and data blocks of preset fixed duration are used to stream the neural feature time-series stream during the inference phase.

7. The speech reconstruction method based on intracranial neural electrical signals according to claim 1, characterized in that, The speech corresponding to the acoustic feature sequence of the sample speech is synthesized speech generated through the following process: collecting reference speech from the target user; A timbre cloning model is trained based on the reference speech; the timbre cloning model is then used to synthesize speech with the target user's timbre based on a preset text.

8. The speech reconstruction method based on intracranial neural electrical signals according to claim 1, characterized in that, The intracranial neuroelectrical signals are acquired by any one of the intracranial electrodes selected from cortical EEG electrode arrays, stereotactic EEG electrodes, and microelectrode arrays.

9. The speech reconstruction method based on intracranial neural electrical signals according to claim 1, characterized in that, The reconstructed speech waveform stream by the method contains speech content corresponding to a single standard language speech, and also covers speech information across languages ​​and dialects, and has the potential to extend to non-language speech information.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the speech reconstruction method based on intracranial neural electrical signals as described in any one of claims 1 to 8.

Images