Thought typing method and brain-computer interface system based on visually evoked broadband response

Abstract

The present invention discloses a mind typing method and brain-computer interface system based on visually evoked broadband response, which relates to the field of brain-computer interface research. By displaying a keyboard used for mind typing on a display device, a brightness sequence is randomly generated for each character on the keyboard, and each character switches between two opposite presentation states according to its own brightness sequence; the user is allowed to observe the keyboard, and the user's electroencephalogram or brain magnetoencephalogram signals are collected and preprocessed at the same time; the preprocessed electroencephalogram or brain magnetoencephalogram signals are input into a decoder based on a convolutional neural network for processing, and a predicted stimulation sequence is output, and the stimulation sequence is compared with the brightness sequence of all characters on the keyboard, and the character corresponding to the most similar stimulation sequence is used as the character predicted by the decoder. It is capable of extracting and processing multi-channel, high-throughput electromagnetic signals of whole-brain neural activity in real time, and performing high-speed and accurate decoding and feedback on the user's mind typing process.

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface research, and specifically to a method and a brain-computer interface system for decoding a visual stimulus stream based on a visual evoked broadband response design using a neural network decoder to achieve direct interaction with a keyboard through thoughts and closed-loop feedback regulation. Background Art

[0002] The brain-computer interface (Brain-Computer Interface) directly establishes a communication link between the human brain and external devices through steps such as signal acquisition, signal processing, feature engineering, and device control, thereby constructing a biological signal closed loop of "brain-computer" interaction. As the human brain electromagnetic signal acquisition equipment continues to develop in the direction of miniaturization and portability, the brain-computer interface has become a hot spot for scientific research and industrial layout internationally. The development of more practical brain-computer interface clinical applications, more accurate brain-computer interface recognition algorithms, and more portable brain-computer interface signal acquisition equipment has become an important area for various countries to occupy the commanding heights of core strength in the field of brain science.

[0003] Among the many applications of brain-computer interfaces, the most researched technology is mind typing. Specifically, by decoding the electromagnetic signals of the human brain, the user's typing intention is identified and fed back to the screen, forming a "decoding-feedback" brain-computer interface closed loop. Brain-computer interface mind typing technology does not simply use the electromagnetic signals of the human brain to decode thoughts, but uses certain cognitive neuroscience principles to decode typing intentions in the interaction between the human brain and the screen keyboard.

[0004] In traditional mind typing design, two paradigms are often used:

[0005] The first is a paradigm designed for mind typing based on the P300 event-related potential (Event Related Potentials P300). P300 is a positive waveform with a significant amplitude in the EEG / EMG signal 300ms after the stimulus is presented, which is related to the human brain's processing of novel stimuli. Based on this principle, the classic P300 paradigm for mind typing is designed as follows. A 6×6 square stimulus interface is presented on the screen, with a letter or number in each square. The squares in each row and column light up in turn, and the EEG / EMG signals are recorded at the same time. When the characters in the user's mind light up, a large P300 component will be caused in the EEG / EMG signal. By lighting up the above rows and columns in turn, combined with the amplitude of the P300 component, the characters in the user's mind can be located. This method is easy to operate, but the characteristics of the experimental paradigm itself result in a low information transmission rate; since the electromagnetic signals of the human brain are decoded at the single trial level, the signal-to-noise ratio is low; at the same time, P300 is an endogenous encoding of stimuli and does not fully utilize the rich properties of visual stimulation.

[0006] The second is a mind typing design paradigm based on Steady State Visual Evoked Potentials (SSVEP). When looking at a visual stimulus of a fixed frequency, the visual cortex of the brain will produce a continuous response related to the stimulus frequency (at the fundamental frequency or multiple of the stimulus frequency), which is SSVEP. An essential difference between SSVEP and P300 potential is that the former is an exogenous visual evoked potential. Therefore, it is possible to control the SSVEP response by changing the visual properties of the exogenous stimulus, and to encode each letter on the screen keyboard through different flashing frequencies, thereby achieving mind typing. This experimental paradigm is currently widely used in brain-computer interfaces, but it still has major limitations.

[0007] The system encodes the letters on the keyboard based on the periodic brightness changes of visual stimuli, and extracts and recognizes SSVEP signals through Fourier analysis and wavelet analysis. However, it has the following limitations: First, the selection of SSVEP frequency is subject to the refresh rate of the display. In actual application, the refresh rate of the computer display used is relatively low (60Hz), and the display refresh rate needs to be an integer multiple of the SSVEP frequency, otherwise frame loss or frame drop will occur, affecting subsequent signal processing; second, the human visual cortex has a high signal-to-noise ratio in response to periodic flickering stimuli in the 11-19Hz frequency band, and this frequency band is relatively narrow, which limits the number of letters on the keyboard; third, this paradigm requires a period of EEG / MEG signals for spectrum analysis, does not fully utilize the time domain information of the signal, and increases the delay in recognizing characters, reducing the recognition speed; fourth, different frequencies encode the letters on the keyboard, which easily cause crosstalk between frequencies, resulting in recognition failure; fifth, in actual operation, stroboscopic stimulation below 15Hz gives users a strong sense of flickering, which reduces user comfort to a certain extent. Sixth, traditional SSVEP data analysis only uses relatively basic signal processing techniques, such as Fourier analysis and wavelet analysis, and fails to fully utilize artificial intelligence technology to empower brain-computer interfaces.

[0008] In addition to the above two mind typing paradigms and their variants, there is currently a mind typing brain-computer interface paradigm designed using principles such as motor imagery. However, in terms of speed and accuracy, it is not as good as the two widely used mind typing paradigms on the market, so it will not be discussed here.

[0009] To sum up, in the field of brain-computer interface mind typing, a new design paradigm needs to be proposed, which can overcome the problems of poor signal-to-noise ratio, low signal transmission rate, and poor user experience existing in the traditional paradigm to the greatest extent, and use the latest achievements in the cutting-edge field of artificial intelligence to empower the design of mind typing brain-computer interface. Summary of the invention

[0010] In order to overcome the problems of the prior art, the inventors propose a mind typing method and brain-computer interface interaction system based on visual evoked spread-spectrum response (VESR), which can extract and process multi-channel, high-throughput electromagnetic signals of whole-brain neural activity in real time, and perform high-speed and accurate decoding and feedback on the user's mind typing process.

[0011] The technical solution adopted by the present invention is:

[0012] A method of mind typing based on visually induced broadband response comprises the following steps:

[0013] A keyboard for mind typing is displayed on a display device, wherein each character on the keyboard has two opposite presentation states;

[0014] Randomly generating a brightness sequence corresponding to the two opposite presentation states for each character on the keyboard, and each character switches between the two opposite presentation states according to the respective brightness sequence;

[0015] Allowing the user to observe the keyboard, while collecting and preprocessing the user's electroencephalogram (EEG) or magnetic brain signals;

[0016] The preprocessed EEG or MEG signals are input into a decoder based on a convolutional neural network for processing, and a predicted stimulus sequence is output. The stimulus sequence is compared with the brightness sequence of all characters on the keyboard, and the character corresponding to the most similar stimulus sequence is used as the character predicted by the decoder.

[0017] Preferably, the two opposite presentation states are light and dark states, or display and disappearance states.

[0018] Preferably, the brightness sequence is generated by random sampling through a pseudo-random algorithm or the like, and the brightness sequence is represented as a 0-1 sequence.

[0019] Preferably, the generated brightness sequence is converted into frequency domain information through Fourier transform, the power of each frequency band is normalized in the frequency domain, and then converted back into a brightness sequence for stimulation through inverse Fourier transform.

[0020] Preferably, the preprocessing includes: performing power frequency noise filtering, bandpass filtering and downsampling on the EEG or MEG signal of each channel.

[0021] Preferably, the high frequency cut-off frequency of the bandpass filter is the refresh rate of the display device, and the frequency of signal downsampling is an integer multiple of the refresh rate of the display device.

[0022] Preferably, the method for comparing the stimulation sequence with the brightness sequence of all characters on the keyboard is: respectively calculating the Pearson correlation coefficient between the stimulation sequence and the brightness sequence of all characters on the keyboard, and taking the stimulation sequence with the largest correlation coefficient as the most correlated stimulation sequence.

[0023] A brain-computer interface system for mind typing based on visually induced broadband response, comprising:

[0024] A stimulus presentation system is used to display a keyboard used for mind typing on a display device, wherein each character on the keyboard has two opposite presentation states; a brightness sequence corresponding to the two opposite presentation states is randomly generated for each character on the keyboard, and each character switches between the two opposite presentation states according to the respective brightness sequence;

[0025] Brain magnetic signal acquisition equipment, used to collect the user's EEG or brain magnetic signals;

[0026] A real-time analysis workstation is used to pre-process the collected EEG or MEG signals, input the pre-processed EEG or MEG signals into a decoder based on a convolutional neural network for processing, output a predicted stimulus sequence, compare the stimulus sequence with the brightness sequence of all characters on the keyboard, and use the character corresponding to the most relevant stimulus sequence as the character predicted by the decoder;

[0027] An external feedback system is used to feed back the predicted characters to the stimulus presentation system.

[0028] Preferably, the display device of the stimulus presentation system is a liquid crystal display or an LED array with adjustable brightness and flashing frequency; the electroencephalographic signal acquisition device is a electroencephalographic cap; and the external feedback system includes one or more of a feedback circuit, a speech synthesizer, a screen, and a prosthesis.

[0029] Preferably, the real-time analysis workstation adopts GPU acceleration optimization.

[0030] The advantages of the present invention are:

[0031] 1) Use a wide-band brightness sequence to modulate the stimulus and flash it on the screen, that is, the change in stimulus brightness presents the characteristics of random white noise in time. This kind of stimulus is conducive to simultaneously detecting the brain's response to changes in all frequency bands of stimuli, rather than being limited to a specific frequency.

[0032] 2) Stimulus presentation is no longer restricted by the refresh rate of the display. By encoding the brightness of the letters when the display refreshes each frame, the visual properties of the stimulus are fully utilized, so that the EEG / MEG signals contain more information related to the stimulus encoding, which is conducive to the use of artificial intelligence decoders to recognize thoughts.

[0033] 3) The maximum change frequency of the stimulus sequence is 60 Hz, which exceeds the human eye's perception threshold for flicker frequency. In actual use, users perceive screen flickering much weaker than the traditional SSVEP-based stimulus sequence design, and the user experience is better.

[0034] 4) The AI-based decoder can decode the brightness sequence of letters in the user's mind frame by frame in real time. In actual operation, the data 250ms after the stimulus is presented can be used to decode the user's typing intention, with high information transmission rate, low delay and high decoding accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of a brain-computer interface system for mind typing based on visually induced broadband response according to the present invention;

[0036] Figure 2 A schematic diagram of a keyboard displayed by a stimulus presentation system of the present invention;

[0037] Figure 3 A schematic diagram of a stimulus brightness sequence generated by a stimulus presentation system of the present invention;

[0038] Figure 4 A flow chart of a method of mind typing based on visually induced broadband response according to the present invention;

[0039] Figure 5 This is a schematic diagram of the decoding effect of a mind-typing brain-computer interface system based on visually induced broadband response according to the present invention. DETAILED DESCRIPTION

[0040] In the following description, the brain-computer interface system of the present invention is further described through specific embodiments, so that those skilled in the art can have a more thorough understanding of the features and advantages of the present invention. It should be noted that what is described below is only a representative typical application. Obviously, the present invention is not limited to any specific structure, function, device and method described herein, and may also have other implementations, or a combination of other implementations. The software / hardware modules described in the present invention or shown in the drawings can also be flexibly adjusted as needed.

[0041] This embodiment provides a brain-computer interface system for mind typing based on visually induced broadband response, and its hardware structure is as follows: Figure 1 The specific instructions are as follows.

[0042] The stimulus presentation system includes a display device for presenting a keyboard used for mind typing, wherein the brightness sequence of each character in the keyboard is modulated using a random broadband brightness sequence. The display device of the stimulus presentation system includes but is not limited to a liquid crystal display or an LED array with modulatable brightness and flashing frequency.

[0043] The electroencephalogram (EMG) signal acquisition device, which can be specifically an electroencephalogram (EEG) cap, is worn on the user's head to acquire the user's real-time EEG / EMG signals and transmit them to a real-time analysis workstation.

[0044] The real-time analysis workstation pre-processes the received EEG / MEG signals and has a built-in EEG / MEG signal decoder based on a convolutional neural network to decode the user's thoughts in real time. The real-time analysis workstation uses GPU acceleration optimization to realize artificial intelligence-based EEG / MEG signal analysis and real-time decoding of the user's thoughts.

[0045] The external feedback system feeds back the user's thoughts to the stimulus presentation system based on the decoding results. The system consists of all external devices that receive the decoding results of the human brain electromagnetic signals from the real-time analysis workstation, such as speech synthesizers, screens, prostheses, etc.

[0046] The processing process of this system is roughly divided into: 1. The stimulus presentation device provides a stimulus brightness sequence based on visually evoked broadband response; 2. The real-time analysis workstation pre-processes and decodes the received EEG / MEG signals in real time. Figure 4 The specific instructions are as follows.

[0047] S1: Arrange the keyboard used for mind typing on the display device (such as Figure 2 As shown in the figure, each character is displayed in a square. Each character has two presentation states: "light" and "dark" (or "displayed" and "disappeared"). For the convenience of description, the "light" or "displayed" state is 1, and the "dark" or "disappeared" state is 0.

[0048] S2: Generate a random 0-1 brightness sequence for all characters on the keyboard through a pseudo-random algorithm. The generation of the random 0-1 sequence can continuously extract numbers between 0 and 1 with equal probability. Preferably, the brightness sequences of all characters are independent of each other, for example, through exhaustive method or other suitable methods, the correlation coefficient between the brightness sequences of each character is less than 0.01, and the significance of the correlation coefficient is greater than 0.05 (indicating that there is no significant correlation between the brightness sequences, and they are considered to be independent of each other). Figure 2 The possible situations of stimulation sequence within 1s are shown, where the keyboard has 38 keys, the screen refresh rate is 60Hz, and the screen refreshes 60 frames within 1s, and each frame corresponds to a presentation state of the character. Preferably, after generating a random brightness sequence, the brightness sequence is converted into frequency domain information through Fourier transform, and the power of each frequency band is normalized in the frequency domain, and then converted back to the brightness sequence for stimulation through inverse Fourier transform, so as to ensure that the energy of the brightness sequence for stimulation is the same at different frequencies and conforms to the characteristics of broadband. Figure 3 shown.

[0049] S3: Let the user observe the keyboard and pre-process the collected EEG / MEG signals. The basic steps include: for each channel signal, filter out 50Hz power frequency noise, bandpass filter (low frequency: 1Hz, high frequency: 60Hz), and downsample to 600Hz. It should be noted that these frequency parameters can be adjusted according to actual conditions, where the high frequency cutoff frequency of the bandpass filter is the display refresh rate, and the signal downsampling frequency should be an integer multiple of the display refresh rate (e.g. 600Hz). The settings of these parameters can usually achieve better results in practical applications.

[0050] S4: The preprocessed EEG / MEG signals and stimulus sequences are used as input and output, respectively, and a decoder based on a convolutional neural network is used for training and verification. Specifically, in the test phase, the user's EEG / MEG signals are input to a decoder based on a convolutional neural network, and a predicted stimulus sequence is output. Subsequently, the Pearson correlation coefficient is calculated between the predicted stimulus sequence and the actual brightness sequence of all characters, and the character corresponding to the stimulus sequence with the largest correlation coefficient is taken as the character predicted by the decoder, such as Figure 5 shown.

[0051] The following is a specific application example. In this example, the system needs to be pre-trained before use. It can be used normally only after the training is completed. It is generally divided into two parts: the training stage and the actual application stage.

[0052] Training phase

[0053] The user wears a brain magnetic cap equipped with an AM probe in a magnetically shielded environment. The position coil on the brain magnetic cap records the user's head movement in real time to correct and compensate for the impact of head movement on the brain magnetic signal.

[0054] A mind-typing keyboard based on broadband brightness sequence modulation is presented on the stimulus presentation system, and a fixation point is presented at the same time. The fixation point is presented in the center of each key in turn, and the duration of presentation is 5 seconds in each key, and then disappears for 1.5 seconds, and then the fixation point is presented in the next key. During this process, the user needs to focus on the fixation point, and blink within 1.5 seconds of each disappearance of the fixation point and quickly move the gaze to the next key. One fixation point traversal is considered one training, and the training is repeated 4 times.

[0055] The actual number of MEG channels is 306. To reduce computational overhead, 25 magnetometer data located in the occipital lobe can be selected for training. Taking the case of 25 MEG channels, 5s presentation time for each key, 38 keyboard characters, 4 training repetitions, 0.25s decoding time window, 60Hz screen refresh rate, and 600Hz MEG signal sampling rate as an example, 25×(5×600)×38×4 data and (5×60)×38×4 stimulation sequence matrix can be obtained. After preprocessing the MEG signal, the signal is reorganized into a 25×(5×600×38×4)×(0.25×600) matrix with a 0.25s time window, and the stimulation sequence is upsampled to a 5×600×38×4 matrix. The former is used as input and the latter as output, and is sent to the convolutional neural network for training. During training, the performance of the neural network can be examined by cross-validation.

[0056] The purpose of the training phase is to provide training data for the AI ​​decoder so that the decoder learns parameters suitable for a specific user. After the AI ​​decoder is trained, the decoder can be directly used by the user to perform mind typing activities.

[0057] Practical application stage

[0058] In actual use, if the user wants to type a certain letter, he needs to stare at the letter on the screen. If the decoding time window is set to 0.25s, then the minimum time the user stares at a certain letter is 0.5s. After preprocessing, the user's EEG / EMG signals are transmitted to the trained neural network to obtain a prediction of the stimulation pattern within the time period of 0.25s to 0.5s. If the screen refresh rate is 60Hz, the decoder will output 0.25×60=150 predicted brightness within this time period. Perform Pearson correlation (or AND operation) on this predicted sequence and the brightness sequence of all letters within 0.25s to 0.5s respectively, and take the character corresponding to the maximum correlation coefficient or the maximum value of the AND operation, which is the predicted character in the user's mind.

[0059] At present, the accuracy (ACC) of the SSVEP mind typing technology based on the improved method on the market can reach 100%, and the information transfer rate (ITR) is 150 to 200 bits / min. In actual application, the accuracy of the mind typing of the present invention can reach up to 100% (there are individual differences in the accuracy between users), and the information transfer rate can reach 210 bits / min, which has advantages in speed and accuracy over the common mind typing paradigm on the market.

[0060] The above embodiments are only illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Any person skilled in the art may modify and change the above embodiments without violating the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be as listed in the claims.

Claims

1. A method of mind typing based on visually induced broadband response, characterized in that: The following steps are involved: A keyboard for mind typing is displayed on a display device, wherein each character on the keyboard has two opposite presentation states; Randomly generating a brightness sequence corresponding to the two opposite presentation states for each character on the keyboard, and each character switches between the two opposite presentation states according to the respective brightness sequence; Allowing the user to observe the keyboard, while collecting and preprocessing the user's electroencephalogram (EEG) or magnetic brain signals; The preprocessed EEG or MEG signals are input into a decoder based on a convolutional neural network for processing, and a predicted stimulus sequence is output. The stimulus sequence is compared with the brightness sequence of all characters on the keyboard, and the character corresponding to the most similar stimulus sequence is used as the character predicted by the decoder.

2. The method according to claim 1, characterized in that The two opposite presentation states are light and dark, or display and disappearance.

3. The method according to claim 1, characterized in that The brightness sequence is generated by random sampling using a pseudo-random algorithm, and the brightness sequence is represented as a 0-1 sequence.

4. The method according to claim 1, characterized in that The generated brightness sequence is converted into frequency domain information through Fourier transform, the power of each frequency band is normalized in the frequency domain, and then converted back into the brightness sequence for stimulation through inverse Fourier transform.

5. The method according to claim 1, characterized in that The preprocessing is as follows: power frequency noise filtering, bandpass filtering and downsampling of the EEG or MEG signals of each channel.

6. The method according to claim 5, characterized in that The high frequency cut-off frequency of the bandpass filter is the refresh rate of the display device, and the frequency of signal downsampling is an integer multiple of the refresh rate of the display device.

7. The method according to claim 1, characterized in that The method for comparing the stimulus sequence with the brightness sequence of all characters on the keyboard is: calculating the Pearson correlation coefficient between the stimulus sequence and the brightness sequence of all characters on the keyboard respectively, and taking the stimulus sequence with the largest correlation coefficient as the most correlated stimulus sequence.

8. A brain-computer interface system for mind typing based on visually induced broadband response, characterized in that: include: A stimulus presentation system, used for displaying a keyboard used for mental typing on a display device, wherein each character on the keyboard has two opposite presentation states; Randomly generating a brightness sequence corresponding to the two opposite presentation states for each character on the keyboard, and each character switches between the two opposite presentation states according to the respective brightness sequence; Brain magnetic signal acquisition equipment, used to collect the user's EEG or brain magnetic signals; A real-time analysis workstation is used to pre-process the collected EEG or MEG signals, input the pre-processed EEG or MEG signals into a decoder based on a convolutional neural network for processing, output a predicted stimulus sequence, compare the stimulus sequence with the brightness sequence of all characters on the keyboard, and use the character corresponding to the most relevant stimulus sequence as the character predicted by the decoder; An external feedback system is used to feed back the predicted characters to the stimulus presentation system.

9. The system according to claim 8, characterized in that The display device of the stimulus presentation system is a liquid crystal display or an LED array with adjustable brightness and flashing frequency; the electroencephalographic signal acquisition device is a electroencephalographic cap; and the external feedback system includes one or more of a feedback circuit, a speech synthesizer, a screen, and a prosthesis.

10. The system according to claim 8, characterized in that The real-time analysis workstation is optimized with GPU acceleration.

Images