Alternating current auxiliary system for generating Chinese text based on electroencephalogram
By constructing an EEG signal classification system based on a multi-layer attention mechanism encoding model and a pre-trained large language model, pinyin letters are generated and converted into Chinese text, which solves the problem of low efficiency of Chinese text generation in the existing technology and realizes an efficient communication assistance solution.
Patent Information
- Application Number
- CN202510119256.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-24
- Publication Date
- 2025-10-10
AI Technical Summary
Existing technologies make it difficult to efficiently generate EEG signal models for Chinese text, and the accuracy of most models is less than 40%. There is a lack of datasets and models for Chinese.
A multi-layer attention mechanism encoding model and a pre-trained large language model are used to build an EEG signal classification system. Pinyin letters are generated through EEG signals and converted into Chinese text using the Pinyin input method. Combined with the contrast loss function training model, a Chinese text generation system based on EEG and letter imagination is constructed.
It achieves efficient and highly accurate Chinese text generation, provides communication assistance solutions for patients with stroke, aphasia and ALS, and improves communication efficiency.
Smart Images
Figure CN120763679A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of electroencephalogram signal processing, and particularly relates to an exchange auxiliary system based on electroencephalogram generated Chinese text. BACKGROUND
[0002] Normal people can convey their inner intentions through various ways such as language, body movements and expressions. However, for special groups such as stroke-induced paralysis, aphasia and patients with amyotrophic lateral sclerosis, they face difficulties in communicating with the surrounding people, although their brain thinking speed is no different from that of normal people. They have difficulty in conveying their thoughts and intentions through language, leading to many inconveniences and misunderstandings, as if they are exiled to an "island". Paralysis and aphasia have brought great trouble to the life and socialization of patients, and have also brought great burden to the family and society.
[0003] For a long time, understanding and decoding the human brain has been an intriguing research field, and numerous studies have been devoted to exploring human emotions, cognition, consciousness, sleep and other activities using electroencephalogram (EEG). However, EEG signals are susceptible to interference, and they are complex spatiotemporal sequences containing many noises and nonlinear components. At the same time, human language is also very complex, with a large vocabulary and complex syntax and syntax structure. Therefore, how to establish the mapping relationship between electroencephalogram signals and text, how to design an effective experimental scheme to collect reliable data, and how to build a model that can directly generate open-vocabulary text are all problems that need to be further studied and solved.
[0004] Many studies have used EEG signals of imagined speech to conduct EEG decoding studies. Some studies have explored the EEG signals of imagined phonemes, vowels, and syllables, such as "ba" and / ku / , and the imagination process of the vowels / a / , / i / , and / u / . In 2015, Zhao et al. classified seven speech / syllables into categories. In 2017, Nguyen et al. analyzed two sets of word imagination signals of different lengths, short words including "in", "out", and "up", and long words including "cooperate" and "independent". In 2018, Jahangiri et al. analyzed four phonetic structures / ba / , / fo / , / le / , and / ry / . In 2020, some studies analyzed the EEG signals of five vowels produced during imagined speech. In addition, some studies involved imagining words, such as the Chinese characters "left" and "one", and the Spanish words "up", "down", "left", "right", and "select". In 2020, Lee et al. constructed a closed-word imaginary dataset containing 12 English words or phrases, including "ambulance," "clock," and "hello," and conducted EEG signal classification studies. Although these studies were based on small closed-word vocabularies, they demonstrated that EEG signals can recognize and classify various imaginary speech units, providing a foundation for decoding EEG into text.
[0005] In the past two years, the development of large language models has made it possible to generate open-vocabulary text from EEG signals. In 2022, Wang Z. et al. proposed a method for decoding electroencephalogram (EEG) to open-vocabulary text using a pre-trained language model (BART), and explored the zero-shot sentiment classification task for the first time. This method was named EEG-to-Text. In 2023, four Chinese researchers from the University of Technology Sydney released the DeWave model at the 37th Conference on Neural Information Processing Systems (NeurIPS2023). Its accuracy was 3 to 6 percentage points higher than that of previous models such as EEG-to-Text, Wave2Vec, BENDR, and SCL. In 2023, Feng X. et al. from Harbin Institute of Technology proposed a semantic-aware contrastive learning strategy C-SCL, which calibrated the subject-dependent EEG representation and reduced the impact of individual differences in subjects on the decoded EEG signals.
[0006] Currently, most EEG-based text models focus on simple classification of a few closed-ended letters, phonemes, or words, with limited research on directly generating Chinese text from EEG. The DeWave and EEG-to-Text models, released in 2023, offer models for generating open-word text from EEG, but these models are still in their early stages of exploration, with accuracy rates generally below 40%. Furthermore, these two models only provide methods for generating open-word text from EEG in English; datasets and models for generating Chinese text from EEG are currently lacking. Therefore, building an efficient model for generating Chinese text from EEG is an urgent challenge. Summary of the Invention
[0007] In response to the problems of the prior art, the present invention provides a communication assistance system that generates Chinese text based on EEG.
[0008] A communication assistance system based on electroencephalogram signals, comprising:
[0009] The input module is configured to input an EEG signal;
[0010] The EEG signal classification module is configured to input the EEG signal into a classification model to obtain a classification result of pinyin letters;
[0011] The classification model includes an EEG signal encoder and a letter decoder, the EEG signal encoder is selected from a multi-layer attention mechanism encoding model, and the letter decoder is selected from a pre-trained large language model.
[0012] Preferably, the EEG signals are EEG signals collected when the subject does not move his vocal organs and hands and keeps imagining writing or pronouncing pinyin letters.
[0013] Preferably, in the training of the letter decoder, the EEG signal encoder is used to encode the EEG signal to construct the embedding vector z t ; Use the text data encoder to encode the pinyin text corresponding to the EEG signal and construct the embedding vector z e ;
[0014] Contrastive loss is used as the loss function, and the expression is:
[0015]
[0016] Among them, Cos_sim() represents cosine similarity.
[0017] Preferably, the text data encoder is selected from a multi-layer attention mechanism encoding model.
[0018] Preferably, the self-pretrained large language model is selected from the BERT model.
[0019] The present invention also provides a communication assistance method based on EEG signals, which is implemented by applying the above-mentioned communication assistance system based on EEG signals, comprising: inputting the EEG signals into a classification model to obtain a classification result of pinyin letters;
[0020] The classification model includes an EEG signal encoder and a letter decoder, the EEG signal encoder is selected from a multi-layer attention mechanism encoding model, and the letter decoder is selected from a pre-trained large language model.
[0021] The present invention also provides a communication assistance system for generating Chinese text based on EEG, comprising:
[0022] The above-mentioned EEG-based communication assistance system;
[0023] The pinyin input method module is configured to convert the classification results of pinyin letters into Chinese text through the pinyin input method.
[0024] The present invention also provides a communication assistance method for generating Chinese text based on EEG, which uses the above-mentioned communication assistance system for generating Chinese text based on EEG, including: inputting the EEG signal into a classification model to obtain a classification result of pinyin letters; converting the classification result of pinyin letters into Chinese text through a pinyin input method;
[0025] The classification model includes an EEG signal encoder and a letter decoder, the EEG signal encoder is selected from a multi-layer attention mechanism encoding model, and the letter decoder is selected from a pre-trained large language model.
[0026] The present invention also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program is used to implement the above-mentioned communication assistance system based on EEG signals, or to implement the above-mentioned communication assistance method based on EEG signals, or to implement the above-mentioned communication assistance system based on EEG to generate Chinese text, or to implement the above-mentioned communication assistance method based on EEG to generate Chinese text.
[0027] The present invention constructs a system and method for generating pinyin letters based on EEG imagery, and further generating open-ended Chinese text using a pinyin input method. The method and system of the present invention offer the advantages of high efficiency and accuracy, and are the first to achieve open-ended Chinese text generation based on EEG signals. The method and system of the present invention are suitable for special populations, such as those with stroke-induced paralysis, aphasia, and amyotrophic lateral sclerosis (ALS), providing a feasible solution for their communication needs and promising application prospects.
[0028] Obviously, based on the above contents of the present invention, according to common technical knowledge and customary means in this field, without departing from the above basic technical ideas of the present invention, other various forms of modifications, replacements or changes can be made.
[0029] The following further describes the above content of the present invention in detail through specific embodiments in the form of examples. However, this should not be construed as limiting the scope of the above subject matter of the present invention to the following examples. All technologies implemented based on the above content of the present invention fall within the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 This is the technical roadmap of Example 1 of the present invention. DETAILED DESCRIPTION
[0031] It should be noted that the algorithms for data collection, transmission, storage and processing steps not specifically described in the embodiments, as well as the hardware structures, circuit connections, etc. not specifically described can all be implemented through the disclosed content of the prior art.
[0032] Example 1 Communication assistance system and method for generating open text based on EEG signals
[0033] The EEG signal communication auxiliary system of this embodiment is used to process the EEG signals of the subject, classify them, and obtain the prediction results of the phonetic letters that the subject is imagining. The EEG signals are collected when the subject does not move the pronunciation organs and hands and keeps imagining the writing or pronunciation of the phonetic letters. A 64-channel electroencephalogram (EEG) cap is used to record EEG signals using the international 10-10 system arrangement of active Ag / AgCl electrodes. Data preprocessing is performed using MATLAB software.
[0034] The technical route of this embodiment is as follows Figure 1 As shown, the system includes:
[0035] The input module is configured to input an EEG signal;
[0036] The EEG signal classification module is configured to input the EEG signal into a classification model to obtain a classification result of pinyin letters;
[0037] The classification model includes an EEG signal encoder and a letter decoder, the EEG signal encoder is selected from a multi-layer attention mechanism encoding model, and the letter decoder is selected from a pre-trained large language model BERT model.
[0038] Specifically, the classification model construction and training process of this embodiment is as follows:
[0039] (1) EEG signal encoding: Use the multi-layer attention mechanism encoding model Multi-layer TransformerEncoder (MLE) as the encoder to encode the EEG signal e and construct the embedding vector z e The expression is:
[0040] z e = ε e (e)
[0041] wherein ε e (e) denotes the encoder.
[0042] (2) Text data encoding: using MLE as the encoder, the pinyin letter text data t corresponding to the electroencephalogram signal e is encoded to construct the embedding vector z t . The expression is:
[0043] z t = ε t (t)
[0044] wherein ε t (t) denotes the encoder.
[0045] (3) Training letter decoding model: using contrastive loss as the loss function to train the model. The loss function expression is:
[0046]
[0047] wherein Cos_sim() denotes the cosine similarity.
[0048] First, the positive and negative sample pairs are constructed. The correct electroencephalogram signal-text embedding vector pair is the positive sample pair, and the incorrect electroencephalogram signal-text embedding vector pair is the negative sample pair. Then, the cosine similarity of all sample pairs is calculated as the contrastive loss function to train the model, thereby constructing the electroencephalogram signal decoding classification model corresponding to the letter.
[0049] (4) Constructing an open Chinese text model based on electroencephalogram signals generated by letter writing or reading imagination. Based on the above trained electroencephalogram signal encoder ε e and the text decoder ε t , a vector quantization variational autoencoder VQ-VAE (i.e., a classification model) is constructed, and a VQ-VAE is further used to construct an open Chinese vocabulary text model for electroencephalogram signal-generated Chinese pinyin description.
[0050] Since there is currently no ready-to-use data set available for training the above classification model, the present embodiment further plans to design experiments and collect data. The main goal is to collect electroencephalogram signals for 26 English letters written or pronounced. The specific experimental scheme is as follows:
[0051] 1. Subject recruitment
[0052] 20-40 healthy subjects are recruited. The subjects must meet the following conditions: no history of neurological diseases or language disorders, no hearing or visual impairment, and no use of drugs within 12 hours before the experiment.
[0053] 2. Experimental Materials
[0054] EEG signals were recorded using a 64-channel electroencephalogram (EEG) cap with active Ag / AgCl electrodes arranged according to the international 10-10 system. Data preprocessing was performed using MATLAB software.
[0055] 3. Experimental process
[0056] The experiment is divided into three parts: imagining speech pronunciation, imagining letter spelling, and imagining letter writing. Each experiment contains 12-50 words and a rest period. The subjects need to silently or imagine the pronunciation or writing of the specified words or letters without moving their vocal organs or hands. The three types of experiments are as follows:
[0057] (1) Imagine writing and pronouncing letters: The subjects imagine writing and pronouncing 26 letters without moving their hands or using their vocal organs to make sounds.
[0058] (2) Imagined letter spelling stage: The subject imagines spelling out the pronunciation of a given word or letter without moving the articulatory organs or making any sound.
[0059] (3) Imagining the pronunciation of the selected word: The subject silently imagines the pronunciation of the designated word but does not move the vocal organs or make any sound.
[0060] The cued words or letters were randomly assigned in each experiment to maintain consistency in the imagery and obtain a relatively large amount of high-quality data. The imagery phase was followed by a 3-second relaxation phase. Words / phrases were selected from a basic vocabulary used by paralysis / aphasia patients in hospitals worldwide, such as "ambulance, hello, help me, sunshine, pain, stop, thank you, toilet, TV, water, yes," and so on.
[0061] This experiment aims to further investigate the EEG signals generated by imagining text in different paradigms to explore their impact on brain-computer interface (BCI) decoding performance. The collected data can be directly used to build a stable EEG model for generating Chinese text, providing a stable and convenient communication tool for paralyzed and aphasic patients whose native language is Chinese, thus filling a technological gap in this field.
[0062] Example 2 Communication Assistance System and Method for Generating Chinese Text Based on EEG
[0063] The system of this embodiment adds a pinyin input method module on the basis of embodiment 1, and is configured to convert the classification results of pinyin letters into Chinese text through the pinyin input method.
[0064] As can be seen from the above embodiments, the present invention provides an EEG-based communication assistance system and method that can generate phonetic letters from EEG signals generated by imaginary spelling or writing, and further generate open Chinese text using a phonetic input method. The system and method of the present invention are characterized by efficient and accurate generation of Chinese text, greatly assisting the communication needs of patients with stroke-induced paralysis, aphasia, and ALS, and has promising application prospects.
Claims
1. A communication assistance system based on EEG signals, characterized in that: include: The input module is configured to input an EEG signal; The EEG signal classification module is configured to input the EEG signal into a classification model to obtain a classification result of pinyin letters; The classification model includes an EEG signal encoder and a letter decoder, the EEG signal encoder is selected from a multi-layer attention mechanism encoding model, and the letter decoder is selected from a pre-trained large language model.
2. The EEG-based communication assistance system according to claim 1, characterized in that: The EEG signals are EEG signals collected when the subject does not move his vocal organs and hands and keeps imagining writing or pronouncing pinyin letters.
3. The EEG-based communication assistance system according to claim 1, characterized in that: In the training of the letter decoder, the EEG signal encoder is used to encode the EEG signal to construct the embedding vector z t. ; Use the text data encoder to encode the pinyin text corresponding to the EEG signal and construct the embedding vector z e. ; Contrastive loss is used as the loss function, and the expression is: Among them, Cos_sim() represents cosine similarity.
4. The EEG-based communication assistance system according to claim 3, characterized in that: The text data encoder is selected from a multi-layer attention mechanism encoding model.
5. The EEG-based communication assistance system according to claim 1, characterized in that: The self-pretrained large language model is selected from the BERT model.
6. A communication assistance method based on EEG signals, characterized in that: The communication assistance system based on EEG signals according to any one of claims 1 to 5 is implemented, comprising: inputting the EEG signals into a classification model to obtain a classification result of pinyin letters; The classification model includes an EEG signal encoder and a letter decoder, the EEG signal encoder is selected from a multi-layer attention mechanism encoding model, and the letter decoder is selected from a pre-trained large language model.
7. A communication assistance system based on EEG to generate Chinese text, characterized in that: include: The communication assistance system based on EEG signals according to any one of claims 1 to 5; The pinyin input method module is configured to convert the classification results of pinyin letters into Chinese text through the pinyin input method.
8. A communication assistance method for generating Chinese text based on EEG, characterized in that: The communication assistance system for generating Chinese text based on EEG according to claim 7 comprises: inputting the EEG signal into a classification model to obtain a classification result of pinyin letters; converting the classification result of pinyin letters into Chinese text through a pinyin input method; The classification model includes an EEG signal encoder and a letter decoder, the EEG signal encoder is selected from a multi-layer attention mechanism encoding model, and the letter decoder is selected from a pre-trained large language model.
9. A computer-readable storage medium, characterized in that A computer program is stored thereon, which is used to implement the communication assistance system based on EEG signals as described in any one of claims 1 to 5, or to implement the communication assistance method based on EEG signals as described in claim 6, or to implement the communication assistance system based on EEG-generated Chinese text as described in claim 7, or to implement the communication assistance method based on EEG-generated Chinese text as described in claim 8.