Brain-computer interface interaction system and method for inputting Chinese characters based on handwriting imagination

Abstract

The invention relates to a brain-computer interface interaction system and method for inputting Chinese characters based on handwriting imagination, and belongs to the technical field of brain-computer interfaces. The system comprises a user interaction module which is used for recording and storing information such as names, genders and ages of subjects and selecting and setting experimental paradigms of handwriting imagination tasks; according to the electroencephalogram data acquisition module, a subject is guided by a handwriting imagination experiment normal form to carry out a corresponding handwriting imagination task, and offline electroencephalogram signals are acquired for multiple times; the model training module is used for constructing an off-line classification model through the handwritten imaginary electroencephalogram signals collected for multiple times; the on-line identification module is used for classifying the signals through the model and carrying out on-line classification identification; and the Chinese character input method module is connected with a character API (Application Program Interface) in an open input method and generates Chinese characters according to the classified strokes or pinyin. The invention provides a novel brain-computer interaction mode based on noninvasive brain imaging for disabled people who lose a handwriting function.

Description

Technical Field

[0001] The invention relates to a brain-computer interface interaction system based on handwriting imagination input of Chinese characters, belonging to the technical field of brain-computer interfaces. Background Art

[0002] The text-oriented motor imagery brain-computer interface does not rely on external stimulation. It can achieve daily communication through the patient's spontaneous imagination, which is of great significance for helping patients who have lost their language and mobility due to neurological diseases. Therefore, it is important to provide optional communication methods for patients or disabled people who have lost their handwriting function, or healthy people who cannot write by hand in specific scenarios. There are some strategies for auxiliary communication, such as BCI with steady-state visual evoked potential (SSVEP-BCI). It is worth noting that although SSVEP-BCI currently has high performance, the user experience and satisfaction still need to be further improved. This type of BCI essentially must have flickering external visual stimulation, so it is inseparable from visual attention and is bound to visual stimulation and attention. Strictly speaking, it is not the result of internal psychological activities. Even more novel is the handwriting imagination BCI (HWI-BCI) input character. This work (Willett FR, Nature, 2021) is to invasively implant microelectrode arrays in the cortex to collect signals, and achieve high-performance brain-to-text communication through handwriting imagination of English characters and real-time neural decoding. The study used a recursive neural network decoding method to translate the neural activity of the motor cortex into text in real time, and a subject with severe spinal cord injury achieved high-speed typing. This study provides a new approach for BCI and proves the feasibility of accurately decoding handwriting imagination after years of paralysis. However, invasive BCI based on internal mental imagination target character input has poor safety, practicality and promotion, and faces great challenges. Summary of the invention

[0003] In order to overcome the defects of the above-mentioned prior art, the present invention provides a new brain-computer interface interaction system based on handwritten imagination input of Chinese characters for disabled people who have lost their handwriting function, and directly converts the internal mentally imagined target characters into text output through a decoding algorithm.

[0004] The technical solution of the present invention is: in the first aspect, the present invention provides a brain-computer interface interaction system based on handwriting imagination input of Chinese characters, the system comprising:

[0005] User interaction module: used to record and save subject information, select and set the experimental paradigm of the handwriting imagination task;

[0006] EEG data acquisition module: used to guide the corresponding handwriting imagination task through the experimental paradigm of the handwriting imagination task, and collect offline handwriting imagination EEG signals;

[0007] Model training module: used to build an offline handwriting imagination character classification model through the collected handwriting imagination EEG signals;

[0008] Online recognition module: used to perform online classification and recognition of new handwritten imaginary EEG signals through the trained offline handwritten imaginary character classification model;

[0009] Chinese character input method module: used to connect to the text API in the open input method to generate Chinese characters or pinyin from classified strokes or pinyin.

[0010] Furthermore, the subject information includes the subject's name, gender, and age information.

[0011] Furthermore, the experimental paradigms of the handwriting imagination task include two types, a handwriting stroke imagination experimental paradigm and a handwriting pinyin imagination experimental paradigm; the handwriting stroke imagination experimental paradigm includes two types, one is the handwriting imagination of the basic strokes of Chinese characters "horizontal", "vertical", "left stroke", "right stroke" and "falling stroke"; the other is the handwriting imagination of 26 Chinese pinyin letters.

[0012] Furthermore, in the EEG data acquisition module, when the subject imagines writing corresponding characters according to the paradigm prompts, the EEG signals of the subject's brain are synchronously recorded.

[0013] Furthermore, in the online recognition module, the real-time EEG signal is classified and recognized online based on the trained offline handwritten imagined character classification model to obtain the imagined corresponding handwritten strokes.

[0014] Furthermore, in the online recognition module, the trained offline handwritten imaginary character classification model uses a filter bank common space pattern FBCSP to extract features.

[0015] Furthermore, in the Chinese character input method module, the classified strokes or pinyin are used as parameters to send a request to the input method API, and the API will return candidate pinyin or characters and present them to the user through the screen.

[0016] In a second aspect, the present invention provides a brain-computer interface interaction method based on handwriting imagination input of Chinese characters, and the specific steps of the method are as follows:

[0017] Step 1. Record and save the subject information, select and set up the experimental paradigm of the handwriting imagination task;

[0018] Step 2: Conduct the corresponding handwriting imagination task through the experimental paradigm of the handwriting imagination task, and collect the offline handwriting imagination EEG signals;

[0019] Step 3, construct an offline handwriting imagination character classification model through the collected handwriting imagination EEG signals;

[0020] Step 4: Use the trained offline handwriting imaginary character classification model to perform online classification and recognition of new handwriting imaginary EEG signals;

[0021] Step 5. Connect to the text API in the open input method to generate Chinese characters or pinyin from the classified strokes or pinyin.

[0022] In a third aspect, the present invention provides a processor for running a program, wherein the program, when running, executes the brain-computer interface interaction method based on handwritten imagination input of Chinese characters described in the second aspect above.

[0023] In a fourth aspect, the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the brain-computer interface interaction method based on handwritten imagination input of Chinese characters as described in the second aspect above.

[0024] The beneficial effects of the present invention are as follows: the present invention guides the corresponding handwriting imagination task through the experimental paradigm of the handwriting imagination task, and collects offline handwriting imagination EEG signals; constructs an offline handwriting imagination character classification model through the collected handwriting imagination EEG signals; classifies the offline handwriting imagination EEG signals through the trained offline handwriting imagination character classification model, and performs online classification and recognition; connects to the text API in the open input method, and generates Chinese characters or pinyin from the classified strokes or pinyin; the present invention provides a new brain-computer interaction method based on non-invasive brain imaging for disabled people who have lost their handwriting function. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of the brain-computer interface interaction system for handwriting and imagining Chinese character input according to the present invention;

[0026] Figure 2 This is a structural block diagram of the brain-computer interface interaction system for handwriting and imagining Chinese character input of the present invention;

[0027] Figure 3 A single experimental flow chart of the present invention;

[0028] Figure 4 Schematic diagram of EEG preprocessing;

[0029] Figure 5 FIG. 4 is a schematic diagram of the FBCSP of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0031] Example 1: Figure 1-Figure 5 As shown, a brain-computer interface interaction system based on handwriting imagination input of Chinese characters, the system comprises:

[0032] User interaction module: used to record and save subject information, select and set the experimental paradigm of the handwriting imagination task; including subject name, gender, and age information;

[0033] EEG data acquisition module: Figure 3 As shown, it is used to guide the corresponding handwriting imagination task through the experimental paradigm of the handwriting imagination task, and collect offline handwriting imagination EEG signals; the experimental paradigm of the handwriting imagination task includes two types, the handwriting stroke imagination experimental paradigm and the handwriting pinyin imagination experimental paradigm; the handwriting stroke imagination experimental paradigm includes two types, one is the handwriting imagination of the basic strokes of Chinese characters "horizontal", "vertical", "left stroke", "right stroke", and "fall"; the other is the handwriting imagination of 26 Chinese pinyin letters. When the subject imagines writing the corresponding characters according to the paradigm prompts, the subject's brain EEG signals are synchronously recorded.

[0034] Model training module: used to build an offline handwriting imagination character classification model through the collected handwriting imagination EEG signals;

[0035] Online recognition module: used to perform online classification and recognition of new handwritten imaginary EEG signals through the trained offline handwritten imaginary character classification model; based on the trained offline handwritten imaginary character classification model, online classification and recognition of real-time EEG signals are performed to obtain the corresponding imagined handwritten strokes. The trained offline handwritten imaginary character classification model uses the filter bank common space pattern FBCSP for feature extraction.

[0036] Chinese character input method module: used to connect to the text API in the open input method, and generate Chinese characters or pinyin from the classified strokes or pinyin. In the Chinese character input method module, the classified strokes or pinyin are used as parameters to send a request to the input method API, and the API will return candidate pinyin or characters, which will be presented to the user through the screen.

[0037] In a second aspect, the present invention provides a brain-computer interface interaction method based on handwriting imagination input of Chinese characters, and the specific steps of the method are as follows:

[0038] Step 1. Record and save the subject information, select and set up the experimental paradigm of the handwriting imagination task;

[0039] Step 2: Conduct the corresponding handwriting imagination task through the experimental paradigm of the handwriting imagination task, and collect the offline handwriting imagination EEG signals;

[0040] Step 3, construct an offline handwriting imagination character classification model through the collected handwriting imagination EEG signals;

[0041] Step 4: Use the trained offline handwriting imaginary character classification model to perform online classification and recognition of new handwriting imaginary EEG signals;

[0042] Step 5. Connect to the text API in the open input method to generate Chinese characters or pinyin from the classified strokes or pinyin.

[0043] In a third aspect, the present invention provides a processor for running a program, wherein the program, when running, executes the brain-computer interface interaction method based on handwritten imagination input of Chinese characters described in the second aspect above.

[0044] In a fourth aspect, the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the brain-computer interface interaction method based on handwritten imagination input of Chinese characters as described in the second aspect above.

[0045] Embodiment 2, a brain-computer interface interaction method based on handwriting imagination input of Chinese characters, the specific steps of the method are as follows:

[0046] Step 1. Record and save the subject information, select and set up the experimental paradigm of the handwriting imagination task;

[0047] Step 2, such as Figure 3 As shown, the subject plays a specific character writing animation in stroke order on the computer screen, and the subject follows the guidance of the animation to imagine writing the character stroke by stroke, and collects offline handwriting imagination EEG signals;

[0048] Step 3, if Figure 4 As shown, the collected handwriting imagination EEG signals are preprocessed, and an offline handwriting imagination character classification model is constructed through the collected handwriting imagination EEG signals;

[0049] Step 4, such as Figure 5 As shown, the new handwriting imaginary EEG signals are classified and recognized online through the trained offline handwriting imaginary character classification model;

[0050] Step 5. Connect to the text API in the open input method to generate Chinese characters or pinyin from the classified strokes or pinyin.

[0051] Further, a schematic diagram of an example of the overall system of the present invention is as follows Figure 1As shown. The user interaction module records and saves the subject information, selects and sets the experimental paradigm of the handwriting imagination task. During the experiment, a computer screen is placed in front of the subject, and the experimental paradigm is used to guide the subject to perform the corresponding handwriting imagination task. The subject follows the guidance of the writing animation to imagine handwriting the character. In the process of the subject handwriting a specific character, the subject's brain signal is synchronously recorded, and the collected handwriting imagination EEG signal is feature extracted through the filter bank common space pattern (FBCSP), and an offline handwriting imagination character classification model is constructed. The handwriting imagination EEG signal is feature extracted through the filter bank common space pattern (FBCSP), and online classification and recognition are performed through the offline handwriting imagination character classification model, and the text API in the open input method is connected to generate Chinese characters or pinyin from the classified strokes or pinyin.

[0052] The specific embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A brain-computer interface interactive system based on handwriting imagination input of Chinese characters, characterized by: The system comprises: User interaction module: used to record and save subject information, select and set the experimental paradigm of the handwriting imagination task; EEG data acquisition module: used to guide the corresponding handwriting imagination task through the experimental paradigm of the handwriting imagination task, and collect offline handwriting imagination EEG signals; Model training module: used to build an offline handwriting imagination character classification model through the collected handwriting imagination EEG signals; Online recognition module: used to perform online classification and recognition of new handwritten imaginary EEG signals through the trained offline handwritten imaginary character classification model; Chinese character input method module: used to connect to the text API in the open input method to generate Chinese characters or pinyin from classified strokes or pinyin.

2. The brain-computer interface interactive system based on handwriting imagination input of Chinese characters according to claim 1 is characterized in that: The subject information includes the subject's name, gender, and age.

3. The brain-computer interface interactive system based on handwriting imagination input of Chinese characters according to claim 1 is characterized in that: The experimental paradigms of the handwriting imagination task include two types, a handwriting stroke imagination experimental paradigm and a handwriting phonetic imagination experimental paradigm; The handwriting stroke imagination experimental paradigm includes two types: one is the handwriting imagination of the basic strokes of Chinese characters, namely, "horizontal", "vertical", "left stroke", "right stroke" and "falling stroke"; the other is the handwriting imagination of 26 Chinese phonetic letters.

4. The brain-computer interface interactive system based on handwriting imagination input of Chinese characters according to claim 1 is characterized in that: In the EEG data acquisition module, when the subject imagines writing corresponding characters according to the paradigm prompts, the EEG signals of the subject's brain are synchronously recorded.

5. The brain-computer interface interactive system based on handwriting imagination input of Chinese characters according to claim 1 is characterized in that: In the online recognition module, the real-time EEG signal is classified and recognized online based on the trained offline handwritten imaginary character classification model to obtain the corresponding imagined handwritten strokes.

6. The brain-computer interface interactive system based on handwriting imagination input of Chinese characters according to claim 1 is characterized in that: In the online recognition module, the trained offline handwritten imaginary character classification model uses the filter bank common space pattern FBCSP to extract features.

7. The brain-computer interface interactive system based on handwriting imagination input of Chinese characters according to claim 1 is characterized in that: In the Chinese character input method module, the classified strokes or pinyin are used as parameters to send a request to the input method API, and the API will return candidate pinyin or characters and present them to the user through the screen.

8. A brain-computer interface interaction method based on handwriting imagination input of Chinese characters, characterized by: The specific steps of the method are as follows: Step 1. Record and save the subject information, select and set up the experimental paradigm of the handwriting imagination task; Step 2: Conduct the corresponding handwriting imagination task through the experimental paradigm of the handwriting imagination task, and collect the offline handwriting imagination EEG signals; Step 3, construct an offline handwriting imagination character classification model through the collected handwriting imagination EEG signals; Step 4: Use the trained offline handwriting imaginary character classification model to perform online classification and recognition of new handwriting imaginary EEG signals; Step 5. Connect to the text API in the open input method to generate Chinese characters or pinyin from the classified strokes or pinyin.

9. A processor, characterized in that: The processor is used to run a program, wherein when the program is running, the brain-computer interface interaction method based on handwritten imagination input of Chinese characters as described in claim 8 is executed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the brain-computer interface interaction method based on handwritten imagination input of Chinese characters as described in claim 8.

Images