High-density electrocorticography-based method for decoding chinese tonal sentences and use thereof
By decoding Chinese tonal sentences using a modular deep learning method, the difficulty of decoding Chinese tonal sentences in the existing technology has been solved, efficient speech synthesis has been achieved, and the communication ability of aphasia patients has been improved.
Patent Information
- Application Number
- PCT/CN2025/080719
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-03-20
- Filing Date
- 2025-03-05
- Publication Date
- 2025-09-25
AI Technical Summary
Existing technologies are unable to effectively decode Chinese tonal sentences, resulting in low communication efficiency for aphasic patients, especially those whose native language is a tonal language, who are unable to efficiently achieve speech synthesis.
A modular approach is adopted, and deep learning is used to process high-density cortical EEG signals. A speech movement starting point detector, basic syllable decoder, basic tone decoder and language model are established respectively, and the hidden Markov model is combined to decode Chinese tone sentences.
It has achieved the direct synthesis of tonal language sentences from human intracranial neural activity, providing efficient rehabilitation treatment plans for Chinese-speaking aphasia patients and improving the speed and accuracy of speech synthesis.
Smart Images
Figure CN2025080719_25092025_PF_FP_ABST
Abstract
Description
A Chinese tone sentence decoding method based on high-density electrocortical imaging and its application Technical Field
[0001] The present invention belongs to the field of biomedical engineering technology and relates to clinical EEG monitoring (language brain-computer interface) technology. Specifically, it relates to a method for decoding Chinese sentences using high-density cortical EEG based on EEG monitoring technology. This method utilizes high-density cortical EEG and targets the neuroanatomical and physiological characteristics of speech initiation, tones, and basic syllables in native Chinese speakers. It can directly synthesize the speech of monosyllabic characters in tonal languages from intracranial neural activity. This method can provide a potential rehabilitation treatment option for native Chinese speakers with aphasia. Background Art
[0002] According to data, in clinical research and practice, anarthria refers to the partial or complete loss of articulation, which can be seen in conditions such as stroke, laryngeal cancer, and amyotrophic lateral sclerosis (ALS); aphasia refers to the inability to understand language and communicate after damage or injury to the language function area of the brain, and is commonly seen in conditions such as trauma, brain tumors, and brainstem stroke. These two disorders encompass the vast majority of clinical language disorders. Clinical research and practice also show that some patients with anarthria and aphasia are lucid and retain partial or almost complete language ability; however, language impairment not only significantly impacts patients' quality of life but also places a significant burden on society and families. Recent studies have indicated that the total incidence of severe language impairment due to various major brain diseases in my country exceeds one million people per year. In the past, the treatment of language dysfunction mostly relied on rehabilitation training for patients. However, due to the small number of professional speech rehabilitation trainers and the complexity of brain diseases that cause language dysfunction in patients, rehabilitation treatment of language dysfunction requires more technical support to benefit patients.
[0003] The recent emergence of brain-computer interfaces (BCIs) has provided new opportunities for the treatment of language disorders. Prior to 2019, these patients could only rely on eye movements and limited muscle movement to manipulate a cursor to select letters for spelling, or utilize non-invasive speech-based brain-computer interfaces for direct letter selection and spelling (SSVEP). However, spelling speeds were slow, averaging approximately 15-20 words per minute. For patients whose native language is a tonal language, additional tone selection was required, further slowing down the process and hindering effective communication. In 2019, researchers in the United States and Europe respectively achieved high-density cortical EEG-speech synthesis of limited short sentences and words in native English-speaking patients, creating a historical precedent for the direct output of speech signals by invasive language brain-computer interfaces; in 2021, Professor Shenoy of Stanford University in the United States achieved the first direct spelling of English letters using an invasive brain-computer interface; in 2022, Japanese researchers expanded the high-density cortical EEG language brain-computer interface work to Japanese, and achieved the use of ECoG to piece together short sentences and output text using limited Japanese phrases; in 2023, the research team of this application used high-density cortical EEG to decode the four tones of the two syllables ma and mi and achieved speech synthesis of monosyllabic characters; these works have laid a solid foundation for the development of cortical EEG-language brain-computer interfaces, but the above work did not involve the decoding of Chinese tonal sentences.
[0004] Pitch and tone are crucial for accurate human semantic expression. In non-tonal languages like English, pitch changes serve to emphasize specific sentence components and express questions. In tonal languages, however, tone changes directly influence the meaning of words. For example, the Chinese character "zhi" can mean "branch," "job," "stop," and "wisdom" depending on the tone, while "qiji" can mean "chance" or "miracle." Phrases and short sentences can be considered different permutations of "syllable + tone." Mandarin Chinese has only 410 basic syllables, which, when combined with four tones, form over 1,300 tonal syllables, encompassing approximately 6,000 commonly used Chinese characters. Furthermore, tonal Chinese sentences differ significantly from non-tonal languages in terms of neurolinguistic mechanisms and natural language processing. Furthermore, the encoding of tones within sentences differs from that of individual words. Therefore, no research has yet examined tones within Chinese sentences. Therefore, achieving EEG decoding of Chinese tonal sentences cannot be achieved by applying decoding models for non-tonal languages, nor can it directly use research results on monosyllabic tone decoding. Therefore, the aforementioned technical field urgently needs to develop a high-density EEG-based decoding solution for Chinese tonal sentences, taking into account the neurolinguistic characteristics of native Chinese speakers and the tonal language characteristics of Chinese.
[0005] Based on the current status of the prior art, the inventors of this application intend to provide a Chinese tone sentence decoding method based on high-density cortical electroencephalography. Summary of the Invention
[0006] The purpose of the present invention is to provide a Chinese tone sentence decoding method based on high-density cortical electroencephalography to overcome the shortcomings of the prior art based on the current status of the prior art.
[0007] The technical solution of the present invention is to adopt a modular approach and use deep learning to separately process and establish a speech movement starting point detector, a basic syllable decoder, a basic tone decoder and a language model.
[0008] Specifically, the technical solution of the present invention is:
[0009] The onset detector is designed using a CNN-GRU cascade model. The model's input layer is a high-frequency gamma signal from a high-density cortical EEG syllable-response electrode. The input layer size is N × T, where N is the number of responding electrodes and T is always 200 (0.5s duration). The first step of the model consists of a one-dimensional convolution-pooling network with N convolutional input channels to ensure that all convolution operations are performed within a single electrode. The convolution kernel length is 3, the convolution stride is 1, and the convolution is followed by pooling with a pooling kernel length of 2. The pooling result is then connected to three layers of bidirectional gated recurrent units (GRUs). Each layer is inactivated with a 50% dropout to prevent rapid overfitting. The final time-step features of the final GRU layer are projected through a fully connected layer to a binary classification output layer (speech movement state and rest state). The network uses cross-entropy loss (CEL) as the loss function, weighted according to the duration of speech movement and rest in the training set. The schematic diagram of the model is shown in Figure 1.
[0010] In the present invention, the basic syllable decoder is also designed using the CNN-GRU sequence model: the input layer of the model is the high-frequency γ signal of the high-density cortical EEG syllable separation electrode, and the input layer size is N×T, where N is the number of electrodes and T is always equal to 480 (1.2s time, that is, the time window from 0.4s before the starting point to 0.8s after the starting point of the detection). The model consists of an initial convolutional layer, C layer A series of one-dimensional convolution-pooling modules and R layer The initial convolution kernel size is (N, filter_time_length), and the convolution step size is ST conv , after which the convolution kernel length of each convolution module is FT conv , the convolution step size is ST convEach convolutional module also incorporates a two-dimensional batch normalization, an ELU activation function, and a random dropout layer with a dropout value of D to prevent the model from overfitting. The final time-step features of the bidirectional GRU are projected to the ten-class output layer through a fully connected layer. The average integration results of the ten sub-model output layers are then softmax-transformed to obtain the final output probability. A schematic diagram of the model is shown in Figure 2.
[0011] In this paper, the tone decoder and syllable decoder have the same architecture, but the specific parameter values and final output result are four-category classification instead of ten-category classification, which is different from the basic syllable decoder. The loss function is weighted according to the distribution of the four tones in the task corpus.
[0012] In this paper, a language model is constructed using a hidden Markov model. The Peking University CCL Chinese database is used to count the transition frequencies between phrases. Transitions that repeat more than 512 times are removed and normalized by taking the ninth root to construct a transition probability matrix. Transition types that do not appear in the corpus but do appear in the decoding task are incorporated into the transition probability matrix. The Viterbi algorithm is used to search for the optimal path and output the final Chinese tonal sentence decoding results.
[0013] This invention provides a new and reliable method for decoding Chinese sentences based on high-density electrocortical (EEG) data. By leveraging the neuroanatomical and physiological characteristics of tones and basic syllables in native Chinese speakers' brains and combining them with language models, it facilitates the direct synthesis of tonal language sentences from intracranial neural activity, providing a potential rehabilitation treatment for native Chinese speakers with aphasia. Compared to existing technologies, this invention is the first to achieve decoding of Chinese tonal sentences based on high-density EEG data. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1. Schematic diagram of the onset detection model.
[0015] Figure 2. Schematic diagram of a basic syllable decoder.
[0016] Figure 3 is a schematic diagram of Example 1.
[0017] Figure 4, decoding accuracy of this method. DETAILED DESCRIPTION
[0018] Example 1
[0019] The onset detector is designed using a CNN-GRU cascade model. The model's input layer is a high-frequency gamma signal from a high-density cortical EEG syllable-response electrode. The input layer size is N × T, where N is the number of responding electrodes and T is always 200 (0.5s duration). The first step of the model consists of a one-dimensional convolution-pooling network with N convolutional input channels to ensure that all convolution operations are performed within a single electrode. The convolution kernel length is 3, the convolution stride is 1, and the convolution is followed by pooling with a pooling kernel length of 2. The pooling result is then connected to three layers of bidirectional gated recurrent units (GRUs). Each layer is inactivated with a 50% dropout to prevent rapid overfitting. The final time-step features of the final GRU layer are projected through a fully connected layer to a binary classification output layer (speech movement state and rest state). The network uses cross-entropy loss (CEL) as the loss function, weighted according to the duration of speech movement and rest in the training set. The schematic diagram of the model is shown in Figure 1.
[0020] The basic syllable decoder is designed using a CNN-GRU sequence model: the input layer of the model is the high-frequency γ signal of the high-density cortical EEG syllable separation electrode. The input layer size is N×T, where N is the number of electrodes and T is always equal to 480 (1.2s time, that is, the time window from 0.4s before the starting point to 0.8s after the starting point of the detection). The model consists of an initial convolutional layer, C layer A series of one-dimensional convolution-pooling modules and R layer The initial convolution kernel size is (N, filter_time_length), and the convolution step size is ST conv , after which the convolution kernel length of each convolution module is FT conv , the convolution step size is ST conv Each convolutional module also incorporates a two-dimensional batch normalization, an ELU activation function, and a random dropout layer with a dropout value of D to prevent the model from overfitting. The final time-step features of the bidirectional GRU are projected to the ten-class output layer through a fully connected layer. The average integration results of the ten sub-model output layers are then softmax-transformed to obtain the final output probability. A schematic diagram of the model is shown in Figure 2.
[0021] The tone decoder shares the same architecture as the syllable decoder, with the only difference being the parameter values and the final output, which is a four-category classification instead of a ten-category classification, as with the basic syllable decoder. The loss function is weighted based on the distribution of the four tones in the task corpus.
[0022] The language model was built using a hidden Markov model. The Peking University CCL Chinese database was used to count the frequency of transitions between phrases. Transitions that repeated more than 512 times were removed and normalized by taking the ninth root to construct a transition probability matrix. Transition types that did not appear in the corpus but did appear in the decoding task were incorporated into the transition probability matrix. The Viterbi algorithm was used to search for the optimal path and output the final Chinese tonal sentence decoding results.
[0023] Example 2
[0024] The present invention will be further described below with reference to the accompanying drawings ( FIG3 ):
[0025] Because the tumor is close to the functional area of the brain, the subject needs to undergo awakening surgery to achieve tumor resection. During the awakening surgery, the surgeon places a high-density cortical EEG electrode in the surgical area (frontal-parietal area or temporal-parietal area) after the dura mater is suspended. Under the guidance of the text prompts on the slide, the patient reads aloud ten Chinese sentences with tones ranging from 5 to 8 words in length. Ten sentences are read aloud once and considered as a group. The position of the sentences in each group is randomly arranged, and each patient recites a total of 16 groups. During the recitation, a microphone is used to synchronously record the patient's voice and EEG signals. The EEG signal is then Hilbert transformed and downsampled to 400Hz, and the hig hgamma frequency band of 70 to 150Hz is extracted using Gaussian filtering.
[0026] The high-gamma (70-150 Hz) EEG was aligned with syllables. The period from 1.8 s before the onset of the first word of each sentence was extracted as silence, and the EEG amplitude during this period was used as the baseline. The high-gamma response was then calculated for each time point (480 in total) from 0.4 s before the onset of speech to 0.8 s after the onset of speech. Two-sample t-tests were performed against the baseline data (Bonferroni correction for electrode number and duration). Electrodes were considered speech-responsive if P < 0.01 for 100 ms. All EEG segments from 0.4 s before the onset of speech to 0.8 s after the onset of speech were averaged, and the averages of the same time points across different syllables were used for one-way ANOVA. Electrodes were selected for tone separation if P < 0.05 (Bonferroni correction for electrode number and duration) was used as the significance criterion for at least half of the time points within the 100 ms continuous period. The screening method of tone separation electrodes is the same as that of syllable separation electrodes, but the time window extends from 0.5s before phonation to 0.5s after phonation.
[0027] The optimization algorithm for the onset detection model is Adam, with a batch size of 1024 and a learning rate of 0.001. Early stopping with a patient of 10 based on the validation set loss function is used to avoid overfitting. The output layer is one-hot in two states (voicing or silent). The test method is four leave-one-out tests on four trials of the test set.
[0028] The optimization algorithm for both the syllable and tone decoder networks was Adam, with a batch size of 8 and a learning rate of 0.0005. Early stopping with a patient of 50 based on the validation set loss function was used to avoid overfitting. The output layer consisted of 10 one-hot syllables or 4 tones. The test method was four leave-one-out tests on four trials of the test set, with the output being the probability of 10 syllables or 4 tones, respectively. The highest probability was taken as the result.
[0029] This method included three subjects, achieving onset detection AUCs of 0.97, 0.94, and 0.96, respectively. The average syllable decoding accuracy reached 59%, and the average tone decoding accuracy reached 46%. The final single-word decoding accuracy reached 76%, with subject 3 achieving a word error rate of 0.21. The model's prediction accuracy significantly improved compared to traditional architectures based solely on CNNs and RNNs.
[0030] In summary, the Chinese monosyllabic word decoding and speech synthesis method based on high-density cortical EEG of the present invention is the first in the world to directly synthesize the speech of monosyllabic words in a tonal language from the neural activity in the human cranium.
Claims
1. A Chinese tone sentence decoding method based on high-density cortical electroencephalography, characterized in that: It includes: Using a deep learning-based method, we target the distribution characteristics of brain functional areas of tones and basic syllables in native Chinese speakers, implement Chinese sentence decoding using word-by-word decoding and splicing, and design a modular language starting point detector, tone syllable decoder, and language model.
2. The Chinese tone sentence decoding method based on high-density cortical electroencephalography according to claim 1, characterized in that: The starting point detector is used to decode the time point of speech initiation from high-density cortical EEG. It is composed of 1DCNN and GRU units in series, and is used to determine the state (voice / rest) of the time point corresponding to the sliding window, thereby realizing the starting point detection of speech movement to adapt to non-uniform speed speech decoding.
3. The Chinese tone sentence decoding method based on high-density cortical electroencephalography according to claim 1, characterized in that: The basic syllable decoder is used to decode the syllable information of Chinese sentences from high-density cortical EEG. It is composed of an initial convolution block, a 1DCNN block, and a GRU block connected in series. Parameters such as the number and size of CNN and GRU blocks, as well as the convolution time step, are individually fine-tuned for each patient. An integrated model is used to enhance the stability of the prediction model.
4. The Chinese tone sentence decoding method based on high-density cortical electroencephalography according to claim 1, characterized in that: The basic tone decoder is used to decode the tone information of Chinese sentences from high-density cortical EEG. It consists of an initial convolution block, a 1DCNN block, and a GRU block connected in series. Parameters such as the number and size of CNN and GRU blocks, as well as the convolution time step, are individually fine-tuned for each patient. An integrated model is used to enhance the stability of the prediction model.
5. The Chinese tone sentence decoding method based on high-density cortical electroencephalography according to claim 1, characterized in that: The language model constructs an n-gram transfer matrix using a large-scale commonly used Chinese language database and uses a hidden Markov model and a Viterbi decoder to select the optimal combination. This is used to obtain semantic information from prior knowledge and assist in sentence decoding.
6. The method according to claim 2, characterized in that The speech movement onset detector is designed using a CNN-GRU series model: the input layer of the model is the high-frequency gamma signal of the syllable response electrode of the high-density cortical EEG, and the input layer size is N×T, where N is the number of response electrodes and T is always equal to 200 (0.5s time); The first step of the model is a one-dimensional convolution-pooling network with N convolution input channels to ensure that all convolution operations are within a single electrode. The convolution kernel length is 3 and the convolution stride is 1. After convolution, it is pooled with a pooling kernel of length 2. The pooling result is connected to three layers of bidirectional gated recurrent units. Each layer is added with a random inactivation layer with a dropout of 50% to prevent the model from overfitting too quickly. The terminal time step features of the last GRU layer are projected to the binary classification (speech movement state and resting state) output layer through the fully connected layer; The network uses cross entropy loss as the loss function and is weighted according to the duration of speech movement and rest in the training set.
7. The method according to claim 3, characterized in that The basic syllable decoder is designed using a CNN-GRU sequence model: the input layer of the model is the high-frequency gamma signal from the high-density cortical EEG syllable separation electrode, and the input layer size is N×T, where N is the number of electrodes and T is always equal to 480 (1.2s time, i.e., the time window from 0.4s before the onset to 0.8s after the onset of detection); The model consists of an initial convolutional layer, C layer A series of one-dimensional convolution-pooling modules and R layer The layer is composed of bidirectional GRU modules sequentially; The initial convolution kernel size is (N, filter_time_length), and the convolution step size is ST conv , after that, the convolution kernel length of each convolution module is FT conv , the convolution step size is ST conv , each convolution module also adds a two-dimensional batch normalization, an ELU activation function and a random loss layer with dropout D to prevent the model from overfitting too quickly; The terminal time step features of the bidirectional GRU are projected to the ten-class output layer through a fully connected layer. The average integration results of the output layers of the ten sub-models are transformed by softmax to obtain the final output probability.
8. The method according to claim 5, characterized in that The language model is established using a hidden Markov model. The transition frequencies between phrases are counted using a Chinese database. Transition frequencies that are repeated more than 512 times are removed and normalized by taking the ninth root to establish a transition probability matrix. Transition types that do not appear in the corpus but appear in the decoding task are incorporated into the transition probability matrix. The Viterbi algorithm is used to search for the best path and output the final Chinese tone sentence decoding result.
9. Use of the method according to claim 1 in preparing a device for treating speech dysfunction.