Electroencephalogram decoding method and application thereof

Abstract

The invention discloses an electroencephalogram language decoding method and application thereof, and the method comprises the following steps: opening recording software, playing and recording the pronunciation of a character needing to be recorded, and obtaining a recording file, the pronunciation being the sound made by artificial intelligence; 2, performing Fourier transform on the sound recording file to obtain an amplitude-frequency diagram, selecting frequencies corresponding to a plurality of peak values in a frequency range of 1-1100Hz according to the amplitude-frequency diagram so as to complete audio decoding and obtain N decoding frequencies, writing the N decoding frequencies obtained in the step 2 into a sound production program from small to large, and playing for each decoding frequency for 1-20s, and opening recording software for recording, finally executing a sound production program, and storing the recording software to obtain an electroencephalogram voice frequency file. The playing of the electroencephalogram speech file obtained by the electroencephalogram speech decoding method has a regulation and control effect on sleep, and the electroencephalogram speech decoding method is not limited by languages and can decode multiple languages.

Description

Technical Field

[0001] The present invention belongs to the technical field of electroencephalogram (EEG) signal processing, and particularly relates to a method for EEG language decoding and its application. Background Art

[0002] The problem of brain science is one of the basic scientific problems faced by human society, and it is an area that needs to be further explored for humans to understand nature and themselves. The brain-computer interface is one of the effective exploration means. A brain-computer interface (BCI) is a direct communication channel established between the brain and external devices. Its signals come from the central nervous system and do not rely on the peripheral nerve and muscle systems during transmission. It is commonly used to assist, enhance, or repair the sensory-motor functions of the human body or improve the human-computer interaction ability.

[0003] The collected EEG signals are processed and decoded to achieve direct interaction and control between the human brain and the computer. Currently, EEG signal decoding still belongs to the difficult and bottleneck problems in the field of brain science, and it is also a hot research topic for the brain-computer interface to truly achieve human-computer interaction. After the EEG signals are decoded, they can be used for information transmission such as intelligence and commercial secrets, or for the communication of conscious patients without communication ability in clinical practice, or for the monitoring of brain signals of unconscious patients, such as the monitoring of brain functions of vegetative patients and one-way communication. Currently, most of the EEG signal decoding methods are based on machine learning in artificial neural networks of artificial intelligence, which have disadvantages such as limited decoding languages, limited sources of EEG data collection, and invasive EEG collection devices.

[0004] Therefore, designing an accurate, efficient, non-invasive, and cross-language EEG language decoding method has important research significance for the fields of brain-computer interface, neuroscience research, medical diagnosis, etc. Summary of the Invention

[0005] Aiming at the deficiencies of the prior art, the purpose of the present invention is to provide a method for EEG language decoding, which solves the problem of limited languages.

[0006] Another purpose of the present invention is to provide a semantic stimulation method.

[0007] The purpose of the present invention is achieved by the following technical solutions.

[0008] A method for EEG language decoding includes the following steps:

[0009] Step 1: Open the recording software, play and record the pronunciation of the text to be recorded, and obtain a recording file, where the pronunciation is the voice emitted by artificial intelligence;

[0010] In step 1, before recording, adjust the sampling rate, number of channels, and bit depth of the recording software: the sampling rate is 8, 11.025, 16, 22.05, 32, 44.1, 48, 64, 88.2, 96, 176.4, or 192 kHz, the number of channels can be selected as mono, stereo, 5.1, or custom, and the bit depth is 8, 16, 24, or 32 (floating point) bits.

[0011] In step 1, the language of the text is one of English, Chinese, and French.

[0012] In step 1, the text is a single character / word.

[0013] In step 1, the recording software is software with recording functionality, such as: Adobe Audition CC, Rainbow Office Software, or the built-in recorder in Win 10.

[0014] Step 2, perform a Fourier transform on the recording file to obtain an amplitude-frequency diagram. Select the frequencies corresponding to multiple peaks within the frequency range of 1 - 1100 Hz based on the amplitude-frequency diagram to complete audio decoding and obtain N decoded frequencies. Among them, the decoded frequencies avoid the power frequency and its multiple frequencies, and N = 3 - 8;

[0015] In step 2, the frequency of performing the Fourier transform on the recording file is the same as the sampling rate in step 1.

[0016] In step 2, the power frequency is 50 Hz.

[0017] Step 3, write the N decoded frequencies obtained in step 2 into the sound generation program in ascending order. The playback time of each decoded frequency is 1 - 20 s. Then open the recording software to record. Finally, execute the sound generation program. After the recording software saves the recording, an electroencephalogram voice frequency file is obtained.

[0018] Application of the electroencephalogram voice frequency file obtained by the above electroencephalogram language decoding method in regulating sleep.

[0019] A semantic stimulation method, including the following steps:

[0020] S1, select one or more locations on the forehead and / or temporal lobe of the test subject and place an electrode at each location. Play the electroencephalogram voice frequency file. Based on the multi-frequency auditory steady-state evoked potential (MASSR) paradigm, collect electroencephalogram signals while playing the electroencephalogram voice frequency file. Among them, mark the electroencephalogram signals at the end of the sound generation of each decoded frequency;

[0021] In S1, the electrode is one of a wet electrode, a dry electrode, and a semi-dry electrode.

[0022] In S1, the positions where the electrodes are placed by the test subject are one or more of the frontal lobe, prefrontal lobe, and temporal lobe in the 10-20 international standard lead system.

[0023] In S1, an electroencephalogram audio file is played through headphones. The test subject needs to wear headphones and keep their eyes closed when collecting electroencephalogram signals.

[0024] In S1, the sampling rate for collecting electroencephalogram signals is 256, 512, 1024, 2048, 4096 Hz.

[0025] S2: Each segment of the electroencephalogram signals collected in S1 is preprocessed in sequence to obtain N preprocessed electroencephalogram data. Among them, the preprocessing includes band-pass filtering.

[0026] In S2, the frequency of the band-pass filtering is 1 - 1100 Hz. The frequency of the band-pass filtering avoids the power frequency and its multiple frequencies and includes the decoding frequency.

[0027] S3: For each preprocessed electroencephalogram data obtained in S2, frequency domain features are extracted through Fourier transform to obtain the frequency and the amplitude corresponding to that frequency. A graph is plotted with the frequency as the X-axis and the amplitude as the Y-axis. Peak detection is performed on each graph and it is judged that: if the amplitude satisfies the 3-fold signal-to-noise ratio condition and a peak appears at the decoding frequency corresponding to this graph, then the signal recognition is successful.

[0028] In S3, the sampling rate of the Fourier transform is the same as the sampling rate of collecting electroencephalogram signals.

[0029] In the above technical solution, the length of the Fourier transform is the product of the acquisition time and the sampling rate, and the length of the Fourier transform is preferably a power of 2.

[0030] In the above technical solution, when the length of the Fourier transform is not a power of 2, an integer greater than and close to the power of 2 is selected.

[0031] Application of the semantic stimulation method in monitoring the brain function of patients in a vegetative state. When the signal recognition is successful for all N graphs, it means that the patient in a vegetative state has consciousness.

[0032] Application of the semantic stimulation method in information transmission. After the signal recognition is successful for all N graphs, the text of the electroencephalogram audio file is output, and the information transmission is successful.

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0034] 1. Playing the electroencephalogram audio file obtained by the electroencephalogram language decoding method of the present invention has a regulatory effect on sleep.

[0035] 2. The semantic stimulation method of the present invention is suitable for a variety of electrodes.

[0036] 3. The method for decoding electroencephalogram language of the present invention is not restricted by language types and can decode multiple languages. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a parameter setting diagram of the recording software;

[0038] Figure 2 It is the audio waveform of the sound emitted by artificial intelligence (when the text is "sleep");

[0039] Figure 3 It is the amplitude-frequency diagram when the text is "sleep";

[0040] Figure 4 It is the amplitude-frequency diagram of Chinese "sleep" (left side) and French "sleep" (right side);

[0041] Figure 5 It is the electroencephalogram signal diagram collected, where 1 to 19 respectively represent the electroencephalogram signals obtained from 19 tentacles of the electrode in contact with the skin;

[0042] Figure 6 It is a photo of collecting electroencephalogram signals;

[0043] Figure 7 It is a program for peak searching and judgment for each diagram;

[0044] Figure 8 It is four "diagrams plotted with frequency as the X-axis and amplitude as the Y-axis";

[0045] Figure 9 It is the time-domain diagram of spindle waves of high-density electrodes, where 1 to 19 respectively represent the time-domain diagrams obtained from 19 tentacles of the high-density electrode in contact with the skin;

[0046] Figure 10 It is the amplitude-frequency diagram of the sleep spindle wave of the 1st tentacle in the high-density electrode. DETAILED DESCRIPTION OF THE INVENTION

[0047] The technical solution of the present invention will be further described below in conjunction with specific embodiments.

[0048] The drugs and instruments involved in the following embodiments are as follows:

[0049] Traditional wet electrode: Ag / AgCl electrode (Beijing Qingjing Electronic Technology Company);

[0050] The Grael EEG device is a 32-channel electroencephalogram acquisition device of the Greal model, from Neuroscan Company, Australia;

[0051] In the following embodiments, the recording software is Adobe Audition CC2019.

[0052] The formula for calculating the accuracy rate P1 of each antenna of the high-density electrode is: P1 = M / N * 100%, where M is the number of successfully recognized signals of each antenna of the high-density electrode, and N is the number of decoding frequencies corresponding to the electroencephalogram voice frequency file.

[0053] The formula for calculating the accuracy rate P2 of the high-density electrode: P2 = the sum of the accuracy rates P1 of 19 antennas of the high-density electrode divided by 19. The semantic stimulation method is used to repeat the test 10 times, and the average value of the accuracy rates P2 obtained in these 10 times is defined as the following "average accuracy rate".

[0054] Example 1

[0055] A method for electroencephalogram language decoding, comprising the following steps:

[0056] Step 1, first find the pronunciation of the text to be recorded. The pronunciation is the voice issued by artificial intelligence. Open the recording software, select "File - New - Audio File", and adjust the sampling rate, channel and bit depth of the recording software before recording. The sampling rate is 44100Hz, the channel is selected as mono, and the bit depth is selected as 32 (floating point) bits, as Figure 1 shown; play the pronunciation of the text to be recorded and record it using the recording software to obtain a recording file, which is saved in the ".wav" format.

[0057] Step 2, perform a Fourier transform on the recording file obtained in Step 1 to obtain an amplitude-frequency diagram. Select all (N) peak frequencies within the frequency range of 1 - 1000Hz according to the amplitude-frequency diagram to complete audio decoding and obtain N decoding frequencies (the decoding frequencies avoid the power frequency (50Hz) and multiples of the power frequency to avoid affecting the signal during testing). Among them, the function for reading the audio when performing the Fourier transform on the recording file is audioread (the program for performing the Fourier transform is written by the programming software MATLABR2019a), and the sampling rate is 44100Hz.

[0058] Step 3, write the N decoding frequencies obtained in Step 2 into the voice program in ascending order. The playing time of each decoding frequency is 5s. Then open the recording software to record, and finally execute the voice program and save the recording software record to obtain an electroencephalogram voice frequency file (the electroencephalogram voice frequency file is in the ".wav" format). Among them, the function of the voice program is sound (the program is written by the programming software MATLAB R2019a).

[0059] In the above step 1, the language of the text is English, Chinese, or French. When it is Chinese, the text is a single character; when it is English and French, the text is a single word. The text in step 1 uses 35 cases: Chinese (Simplified Chinese) and English's "one, two, three, four, five, six, seven, eight, nine, left, right, exhale, inhale, laugh, good, beautiful", and Chinese, English, and French's "sleep".

[0060] When the text is the English word "sleep", the audio waveform of the sound emitted by the artificial intelligence is as Figure 2 shown. N = 4, and the N decoding frequencies are arranged in ascending order as 272 Hz, 486 Hz, 544 Hz, 835 Hz. The amplitude-frequency diagram is as Figure 3 shown.

[0061] When the text is the Chinese word "sleep", the amplitude-frequency diagram is as Figure 4 shown in "Chinese" in

[0062] When the text is the French word "sleep", the amplitude-frequency diagram is as Figure 4 shown in "French" in

[0063] Example 2

[0064] A semantic stimulation method, comprising the following steps:

[0065] S1, dip a small amount of abrasive cream with a lint-free cloth and wipe the mastoid behind the ear of the test subject. The function of the abrasive cream is to remove the cutin layer on the surface of the mastoid. Apply 0.2 mL of conductive paste on two traditional wet electrodes - Ag / AgCl respectively to fill the concave parts of the traditional wet electrodes - Ag / AgCl. Then fix the two traditional wet electrodes - Ag / AgCl on the two mastoid positions behind the ear (A1, A2) with medical tape. One of the two traditional wet electrodes - Ag / AgCl serves as a reference electrode, and the other serves as a ground electrode. Wipe the area above the eyebrow bone Fp1 of the test subject with abrasive cream, and then wipe it with 0.5 μL of an aqueous NaCl solution with a concentration of 0.01 mol / L. After wiping, place a high-density electrode (for the preparation method of the high-density electrode, refer to Example 1 of Patent Publication No. CN116019455A) on the wiped Fp1 and fix it with a headband, as Figure 6 shown.

[0066] Play the electroencephalogram (EEG) audio file obtained in Example 1 as a stimulus file. Based on the multi-frequency auditory steady-state evoked potential (MASSR) paradigm, use a Grael EEG device and Curry 8 software to test and collect EEG signals while playing the EEG audio file (the subject wears headphones and keeps eyes closed during EEG signal collection, observes the computer screen, adjusts the impedance value, starts collecting EEG signals when the impedance value is below 10 kΩ, plays the EEG audio file through headphones, mark the EEG signals at the end of each decoded frequency's sound emission, and the collected EEG signals are in the ".cdt" format, which is converted to the ".cnt" format.). Among them, mark the EEG signals at the end of each decoded frequency's sound emission, the number of electrodes of the high-density electrode is 19, and the sampling rate of collecting EEG signals is 4096 Hz.

[0067] When the text in Example 1 is "sleep", the EEG signals collected after being stimulated by the obtained EEG audio file are as Figure 5 shown (simultaneously test the 19 electrodes at Fp1 to obtain 19 EEG signals).

[0068] S2. Successively preprocess each segment of the EEG signals marked in S1 to obtain N preprocessed EEG data (that is, one segment of EEG signal corresponding to each decoded frequency becomes one preprocessed EEG data) one by one. Among them, the preprocessing includes band-pass filtering, and the frequency of the band-pass filtering is 1 - 1100 Hz. The frequency of the band-pass filtering avoids the power frequency (50 Hz) and its multiple frequencies and includes the decoded frequency. For example, when the text is "sleep", the frequencies of the band-pass filtering for the decoded frequencies of 272 Hz, 486 Hz, 544 Hz, and 835 Hz are 260 - 280 Hz, 480 - 490 Hz, 530 - 548 Hz, and 820 - 840 Hz respectively; the range of the band-pass filtering can be adjusted according to one's own requirements.

[0069] S3. Extract the frequency-domain features of each preprocessed EEG data obtained in S2 through Fourier transform (convert the EEG signal from the time domain to the frequency domain and extract the frequency-domain features) to obtain the frequency and the amplitude corresponding to that frequency, and draw a graph with the frequency as the X-axis and the amplitude as the Y-axis (one graph is obtained from one preprocessed EEG data). Perform peak searching and judgment on each graph: If the amplitude meets the 3-fold signal-to-noise ratio condition and a peak appears at the decoded frequency corresponding to this graph, then the signal recognition is successful.

[0070] In the above S3, the length of the Fourier transform is the product of the acquisition time and the sampling rate, the sampling rate of the Fourier transform is 4096 Hz, and the length of the Fourier transform is 32768.

[0071] When the text is "sleep", the procedure for performing peak searching and judgment on each graph is as Figure 7 shown.

[0072] When the Chinese character in Example 1 is "sleep", N "graphs with frequency as the X-axis and amplitude as the Y-axis" are drawn as follows Figure 8 shown.

[0073] The semantic stimulation method can be applied in many fields, such as: vegetative brain function monitoring or information transmission. When the semantic stimulation method is applied to vegetative brain function monitoring, when N average signals are successfully recognized, the vegetative brain is conscious. The semantic stimulation method is applied to information transmission. The semantic stimulation method can be used to stimulate the test subject wearing high-density electrodes. When N average signals are successfully recognized, the receiver can obtain the text corresponding to the EEG audio file of the stimulated test subject through the output.

[0074] When the Chinese characters in Example 1 are "sleep, right, left, exhale, inhale, beautiful, laugh, good", the average accuracy rates of the text in the EEG audio file output by S3 are 88.16%, 90.42%, 90.26%, 92.42%, 89.68%, 85.09%, 93.10% and 78.95% respectively; when the Chinese characters in Example 1 are "sleep, right, left, exhale, inhale, beautiful, laugh, good", the average accuracy rates of the text in the EEG audio file output by S3 are 88.56%, 83.95%, 91.26%, 89.39%, 87.17%, 79.01%, 89.54% and 90.26% respectively.

[0075] Example 3

[0076] Spindle waves are narrow-band bursts of activity that dominate the EEG (electroencephalogram) signal in one or more scalp regions. Sleep spindles are oscillatory activity in the 11-16 Hz frequency range and are the primary indicator of NREM (non-rapid eye movement) stage 2 sleep. Sleep spindles have a well-defined structure and are often preceded by a signal feature called a K-complex, which consists of a short negative deflection, followed by a large positive deflection, and then a large negative deflection.

[0077] When the text is the English word "sleep", the EEG speech audio file obtained in Example 1 is played to the patient with sleep disorder. During the playback, the patient keeps his eyes closed and the EEG signal is collected at the frontal lobe or temporal lobe. The collected EEG signal is processed in the time domain and frequency domain respectively to obtain its time domain graph ( Figure 9 ) and amplitude-frequency diagram ( Figure 10 , with an amplitude of about 0.6μV). Figure 9 and Figure 10 It can be seen that during the auditory stimulation phase, club-shaped spindles appeared, the duration of the spindles was 2s, and no spindles appeared during the non-stimulation phase ( Figure 9The part on the left side of the dotted line). The appearance of spindle waves indicates that the EEG voice frequency file obtained by the EEG decoding method has a regulatory effect on sleep.

[0078] Example 4

[0079] When the high-density electrodes in the semantic stimulation method in Example 2 are replaced with wet electrodes or EEG caps, the same technical effects as those in Example 2 can be obtained.

[0080] The above has made an exemplary description of the present invention. It should be noted that without departing from the core of the present invention, any simple deformation, modification, or equivalent replacement that can be made by those skilled in the art without creative labor falls within the protection scope of the present invention.

Claims

1. A method for electroencephalogram language decoding, characterized in that, It includes the following steps: Step 1: Open the recording software, play and record the pronunciation of the text to be recorded to obtain a recording file, where the pronunciation is the voice emitted by artificial intelligence; Step 2: Perform Fourier transform on the recording file to obtain an amplitude-frequency diagram. Select the frequencies corresponding to multiple peaks within the frequency range of 1 - 1100 Hz according to the amplitude-frequency diagram to complete audio decoding and obtain N decoding frequencies, where the decoding frequencies avoid power frequency and multiples of the power frequency; Step 3: Write the N decoding frequencies obtained in Step 2 into the sound generation program in ascending order. The playing time of each decoding frequency is 1 - 20 s. Then open the recording software to record, and finally execute the sound generation program and save the recording of the recording software to obtain an electroencephalogram voice frequency file.

2. The method according to claim 1, wherein In Step 1, before recording, adjust the sampling rate, channel, and bit depth of the recording software: the sampling rate is 8, 11.025, 16, 22.05, 32, 44.1, 48, 64, 88.2, 96, 176.4, or 192 kHz, and the bit depth is 8, 16, 24, or 32 bits.

3. The method according to claim 1, characterized in that In Step 1, the language of the text is one of English, Chinese, and French.

4. The method according to claim 1, wherein In Step 1, the text is a single character / word.

5. The method according to claim 1, wherein In Step 2, the frequency of performing Fourier transform on the recording file is the same as the sampling rate in Step 1.

6. Application of the electroencephalogram voice frequency file obtained by the method for electroencephalogram language decoding as claimed in claim 1 in regulating sleep.

7. A semantic stimulation method, characterized in that, It includes the following steps: S1: Select one or more locations on the forehead and / or temporal lobe of the test subject and place an electrode at each location. Play the electroencephalogram voice frequency file obtained by the method for electroencephalogram language decoding as claimed in claim 1, and collect electroencephalogram signals while playing the electroencephalogram voice frequency file. Among them, mark is made on the electroencephalogram signals at the end of the sound emission of each decoding frequency; S2: Perform preprocessing on each segment of the electroencephalogram signals marked in S1 in sequence to obtain N preprocessed electroencephalogram data, where the preprocessing includes band-pass filtering; S3: Extract frequency domain features of each preprocessed electroencephalogram data obtained in S2 through Fourier transform to obtain the frequency and the amplitude corresponding to the frequency. Draw a graph with the frequency as the X-axis and the amplitude as the Y-axis, and perform peak seeking and judgment on each graph: If the amplitude meets the 3-fold signal-to-noise ratio condition and a peak appears at the decoding frequency corresponding to the graph, the signal recognition is successful.

8. The semantic stimulation method according to claim 7, characterized in that, In S1, the electrode is one of wet electrode, dry electrode, and semi-dry electrode; In S1, the positions where the test subject places the electrodes are one or more of the frontal lobe, prefrontal lobe, and temporal lobe in the 10 - 20 international standard lead system; In S1, play the electroencephalogram voice frequency file through headphones. The test subject needs to wear headphones and keep the eyes closed when collecting electroencephalogram signals; In S1, the sampling rate of collecting electroencephalogram signals is 256, 512, 1024, 2048, 4096 Hz; In S2, the frequency of the band-pass filtering is 1 - 1100 Hz. The frequency of the band-pass filtering avoids the power frequency and multiples of the power frequency and includes the decoding frequency; In S3, the sampling rate of the Fourier transform is the same as the sampling rate of collecting electroencephalogram signals; The length of the Fourier transform is the product of the acquisition time and the sampling rate, and the length of the Fourier transform is preferably a power of 2; when the length of the Fourier transform is not a power of 2, an integer greater than and close to the power of 2 is selected.

9. The application of the semantic stimulation method according to claim 7 in the monitoring of the brain function of a vegetative patient, characterized in that, When the signals of all N graphs are successfully recognized, the vegetative brain is conscious.

10. The application of the semantic stimulation method according to claim 7 in information transmission, characterized in that, After the signals of all N graphs are successfully recognized, the text of the electroencephalogram voice file is output, and the information transmission is successful.

Images