Visual brain-computer interface control method and device

Abstract

The invention discloses a visual brain-computer interface control method and device, and the method comprises the steps: outputting a first control signal and a second control light signal which correspond to preset frequencies at the same time, are visible light of different frequencies, and are located in the same preset region; human body electroencephalogram signals generated based on real-time synchronous stimulation of two signals are collected, SSVEP features and P300 features are extracted, and a preliminary control instruction and a verification mark are generated respectively. Verifying the consistency of the preliminary control instruction and the verification mark, giving up the preliminary control instruction when the preliminary control instruction is inconsistent with the verification mark, and generating an output control instruction corresponding to the preset area when the preliminary control instruction is consistent with the verification mark; the visual brain-computer interface control device using the method comprises a microcontroller, a light-emitting device, a signal acquisition device and a signal processing device. The brain-computer interface control method and device can significantly improve the classification reliability and reduce the misjudgment rate, and are suitable for being integrated in brain-computer interface products.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of brain-computer interface, and particularly relates to a visual brain-computer interface control method and device. BACKGROUND

[0002] A brain-computer interface is an external device control system, which can obtain a corresponding control signal through the coding and decoding of the brain cortex signal. By using this technology, people can directly control external devices through the brain. The brain-computer interface system based on the scalp electroencephalogram has a high event resolution, and its hardware system is stable, convenient to use, and low in price, so it has been widely studied and applied. The traditional electroencephalogram-induced brain-computer interface mainly uses a single electroencephalogram feature signal. Steady-state visual evoked potential (SSVEP), P300, and event-related desynchronization (ERD) features are three signal paradigms with better effects in the current electroencephalogram brain-computer interface. Although the brain-computer interface based on the above three paradigms can obtain a relatively stable feature induction effect, the judgment accuracy is poor, the information transmission rate is low, and it is still difficult to meet the needs of practical applications. In recent years, hybrid paradigm brain-computer interface has been widely concerned as a new direction of the development of brain-computer interface technology. The hybrid paradigm brain-computer interface is a combination of a sub-brain-computer interface system and other human-computer interface systems, so as to output control commands faster and more accurately. Among them, the pure hybrid paradigm brain-computer interface system combines multiple sub-brain-computer interface systems together, and uses two types (or multiple types) of electroencephalogram feature signals to complete the target task. Among the three classic paradigms, visual evoked P300 and SSVEP can achieve a high information transmission efficiency and are more widely used. However, both of them need visual stimulation provided by the outside world. Since the types of visual stimulation required by P300 and SSVEP are different, and the visual pathway of the human body is single, it is difficult to induce P300 and SSVEP at the same time.

[0003] In the prior art, the combination of P300 paradigm and SSVEP paradigm usually adopts the same stimulation light signal, the human body observes the same flickering monochromatic light source or visual symbol to generate a special electroencephalogram. After the acquisition device collects the electroencephalogram, the electroencephalogram is separated to obtain the SSVEP signal and the P300 signal, and then the SSVEP signal and the P300 signal are mixed to obtain the control instruction. The disadvantage of this method is that the optimal stimulation light source characteristics of the human brain to generate the SSVEP signal and the P300 signal are different, and the same light source stimulation cannot simultaneously generate the optimal SSVEP signal and the P300 signal. Moreover, although the mixed processing of the SSVEP signal and the P300 signal can improve the accuracy compared with the single paradigm signal processing, there is still a risk of misjudgment, and this risk of misjudgment may cause safety hazards when controlling peripheral devices of the brain-computer interface. The series analysis of the SSVEP signal and the P300 signal takes a long time, which also causes a long response delay of the system.

[0004] The above information is presented as background information only to assist with an understanding of the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above might be applicable as prior art with regard to the present disclosure. SUMMARY

[0005] Embodiments of the present disclosure solve the problems of poor light source stimulation effect, low accuracy, and long system delay in the mixed paradigm brain-computer interface of SSVEP and P300. A visual brain-computer interface control method is provided, which simultaneously stimulates by two light sources, processes the SSVEP signal and the P300 signal respectively, and generates a control instruction together. A visual brain-computer interface control device capable of implementing this method is also provided.

[0006] A first aspect of the present disclosure provides a visual brain-computer interface control method, comprising: simultaneously outputting a first control signal corresponding to a preset frequency and a second control signal corresponding to a fixed frequency; emitting a first visible light signal of the preset frequency based on the first control signal and emitting a second visible light signal of the fixed frequency based on the second control signal, the first visible light signal and the second visible light signal being located in the same preset area; collecting a human electroencephalogram signal generated by real-time synchronization stimulation based on the first visible light signal and the second visible light signal; extracting an SSVEP feature based on the electroencephalogram signal, and generating a preliminary control instruction based on the SSVEP feature; extracting a P300 feature based on the electroencephalogram signal, and generating a verification mark based on the P300 feature; verifying the consistency of the preliminary control instruction and the verification mark, discarding the preliminary control instruction when the preliminary control instruction and the verification mark are inconsistent, and generating an output control instruction corresponding to the preset area when the preliminary control instruction and the verification mark are consistent.

[0007] For example, in at least one embodiment, the plurality of preset regions are capable of emitting the first visible light signal and the second visible light signal; the first visible light signal and the second visible light signal are both light signals generated by LED light sources; the preset frequencies at which the first visible light signal is emitted in each of the preset regions are different; and the preset frequency range is 7-30 HZ.

[0008] For example, in at least one embodiment, the SSVEP feature is extracted using an FFT frequency domain analysis method, and the P300 feature is extracted using a peak value detection method in a time range, and the steps of extracting the SSVEP feature and extracting the P300 feature are processed in parallel.

[0009] For example, in at least one embodiment, the second control signal includes random flickering in a certain period in a pseudo-random sequence, and each random flickering event is time-labeled by a specific character marker transmitted through serial communication; and in the P300 feature extraction, time synchronization between the second control signal and the electroencephalogram signal is achieved based on the time label.

[0010] For example, in at least one embodiment, the method further includes the step of controlling an external device based on the generated output control instruction.

[0011] A second aspect of the present disclosure provides a visual brain-computer interface control device, comprising: a microcontroller, a light emitting device, a signal acquisition device, and a signal processing device; wherein the microcontroller is configured to output a light source control signal, the light source control signal including a first control signal corresponding to a preset frequency and a second control signal corresponding to a fixed frequency; the light emitting device is connected with the microcontroller and includes a first light source and a second light source arranged in the same preset region, the first light source emits a first visible light signal based on the first control signal, and the second light source emits a second visible light signal based on the first control signal; the signal acquisition device is arranged in a brain region and is configured to acquire an electroencephalogram signal generated by the human brain in real time based on synchronous stimulation of the first visible light signal and the second visible light signal; the signal processing device is connected with the signal acquisition device, extracts an SSVEP feature based on the electroencephalogram signal, and extracts a P300 feature based on the electroencephalogram signal; generates a preliminary control instruction based on the SSVEP feature, and generates a verification marker based on the P300 feature; verifies the consistency of the preliminary control instruction and the verification marker, discards the preliminary control instruction when the preliminary control instruction and the verification marker are inconsistent, and generates an output control instruction corresponding to the preset region when the preliminary control instruction and the verification marker are consistent.

[0012] For example, in at least one embodiment, the first light source is a green ring LED light source array composed of a plurality of LED light sources; and the second light source is a red LED light source arranged at the center of the green ring LED light source array.

[0013] For example, in at least one embodiment, the microcontroller has a multi-thread architecture and supports multiple independent outputs of the light source control signals; and the multiple light source control signals correspond to the first light sources and the second light sources in multiple preset areas, respectively, and the first light sources in each of the preset areas emit the first visible light signals at different preset frequencies.

[0014] For example, in at least one embodiment, the light emitting device further comprises an external device, and the five preset areas correspond to five output control instructions, respectively, and the five output control instructions control the forward movement, the backward movement, the left turn, the right turn and the stop of the external device, respectively.

[0015] For example, in at least one embodiment, the signal acquisition device comprises an acquisition electrode, a reference electrode and a grounding electrode; the acquisition electrode is arranged at the midline of the forehead, the midline of the center, the midline of the top, the left top occiput, the right top occiput and the midline of the occiput of the human brain; and the reference electrode is arranged at the left mastoid of the human body. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, and are not a limitation on the present disclosure.

[0017] Figure 1 is a flow chart of a visual brain-computer interface control method according to an embodiment of the present disclosure

[0018] Figure 2 is a schematic diagram of a first light emitting device according to an embodiment of the present disclosure

[0019] Figure 3 is a schematic diagram of a second light emitting device according to an embodiment of the present disclosure

[0020] Figure 4 is a schematic diagram of a visual brain-computer interface control device according to an embodiment of the present disclosure DETAILED DESCRIPTION

[0021] In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure clearer, the following will be combined with the drawings of the embodiments of the present disclosure to make a clear and complete description of the technical solutions of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by a person of ordinary skill in the art without any inventive effort fall within the protection scope of the present disclosure.

[0022] Unless otherwise defined, technical terms or scientific terms used in the present disclosure shall have the ordinary meaning as understood by a person of ordinary skill in the art to which the present disclosure belongs. The terms "first", "second" and similar terms used in the present disclosure do not denote any order, quantity or importance, but are used to distinguish different components. The terms "comprise", "contain" and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connect" or "connected" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "up", "down", "left", "right" and the like are only used to indicate relative positional relationships, and when the absolute positions of the described objects are changed, the relative positional relationships may also be changed accordingly. In the present disclosure, "a plurality of" means two or more.

[0023] According to the embodiments of the present disclosure, Figure 1 A control method of a visual brain-computer interface is disclosed, comprising the following steps:

[0024] S101: simultaneously output a first control signal corresponding to a preset frequency and a second control signal corresponding to a fixed frequency. The first control signal of the preset frequency is used to control visible light for stimulating to generate an SSVEP signal. The SSVEP signal generated by the human brain has a strong correlation with the frequency of the stimulating light source, so that by setting the preset frequency, the corresponding SSVEP signal can be captured. By setting different preset frequencies for different stimulating light sources, multiple preset light sources can be effectively distinguished. The second control signal of the fixed frequency is used to control visible light for stimulating to generate a P300 signal. The P300 signal generated by the human brain does not have an obvious correlation with the frequency of the stimulating light source, so that the frequency of the second control signal is fixed. The first control signal and the second control signal are simultaneously output, the light sources are controlled to emit light at the same time, and the SSVEP signal and the P300 signal obtained by stimulating the two light sources at the same time are processed, so that the control information at the time point can be obtained, and the control information is verified.

[0025] Preferably, in at least one embodiment, the first control signal and the second control signal are implemented by a Teensy 3.2 microcontroller with an ARM Cortex-M4 processor (72MHz clock speed). The system architecture can use multi-threading technology to generate precise timing of parallel control signals to achieve the technical effect of setting up multiple preset areas capable of emitting the first visible light signal and the second visible light signal. In the case of setting up multiple preset areas capable of emitting the first visible light signal and the second visible light signal, the timing resolution of the light source frequency generation is in the nanosecond level, which can ensure the precise phase control and time stability required for SSVEP induction. The multi-threading design ensures deterministic timing by running independent threads, maintaining the phase relationship and frequency accuracy required for neural response frequency discrimination.

[0026] S1021: Emitting a first visible light signal of a preset frequency based on the first control signal.

[0027] Preferably, in at least one embodiment, the preset frequency can be selected and set in the frequency range of 7-30Hz.

[0028] Biophysical research results show that neurons have resonance characteristics, and the complex connections of cerebral cortex neurons form different oscillation networks. The spectral characteristics of SSVEP are that there are power peaks at its fundamental frequency (same as the external stimulus frequency) and its harmonic frequency (the fundamental power is the largest), which shows that the equivalent network corresponding to the brain visual cortex fundamental and its harmonic and the corresponding frequency component of the stimulus signal produce synchronous oscillation. For the same subject, under the same other test conditions, the fundamental power of SSVEP changes with the change of the stimulus frequency, and the stimulus frequency that induces the maximum fundamental power is called the "resonance frequency". The resonance frequency of SSVEP is the stimulus frequency corresponding to the maximum SSVEP response, so selecting the frequency range to set the first visible light signal can obtain better technical effects.

[0029] Experiments show that the frequency of 7-30 Hz can better cover the resonance frequency of SSVEP: when the stimulation frequency is within 5-60 Hz, the resonance frequency of SSVEP is 15 Hz; when the stimulation frequency is within 5-16 Hz, the SSVEP response is larger, and the resonance frequency is 12 Hz; using a sampling sine coding method, a rectangular visual stimulus is generated on a computer display, the flicker frequency of the pattern is within 4-45 Hz (step 1 Hz), and the resonance frequency of SSVEP is detected to be 10 Hz; under the screen flicker pattern stimulation of frequency of 6-40 Hz and time modulation mode of sine wave and square wave, the resonance frequency of SSVEP is 10 Hz when the square wave is flickered, and the resonance frequency of SSVEP is 14 Hz when the sine wave is flickered. The common visible light signal in the prior art has a frequency division limit of 60 Hz, and it can only have excellent light emission performance when the frequency is a multiple or a divisor of 60 Hz. This characteristic greatly limits the application of visual brain-computer signals. The present disclosure solves this problem and can set a more free emission preset frequency in a more suitable frequency range.

[0030] Preferably, in at least one embodiment, the first visible light signal is green light, which has high photoreceptor sensitivity and superior cortical response characteristics, so it is selected as the main stimulation source.

[0031] S1022: Emitting a second visible light signal with a fixed frequency based on the second control signal. Preferably, the second visible light signal is realized by randomly flickering in a certain period according to a pseudo-random sequence, and each random flickering event is time-labeled by transmitting a specific character mark to the signal processing device through serial communication. Thus, in the P300 feature extraction, the accurate time synchronization between the second visible light signal and the electroencephalogram data acquisition is realized. After time synchronization, by comparing the random flickering waveform emitted by the second visible light signal based on the corresponding random flickering waveform in the P300 feature, it can be judged whether the P300 feature is generated based on the stimulation of the second visible light signal. In the case of multiple second visible light signals, the specific character mark can be used to determine the second visible light signal sequence number corresponding to the P300 feature and all preset regions.

[0032] The first visible light signal emitted in step S1021 and the second visible light signal emitted in step S1022 are located in the same preset region. The visual pathway of the human body has singularity, and the subject can usually only gaze at the same region, which makes it challenging to output effective control signals simultaneously in the visual brain-computer interface, and also causes the problem of signal delay in the brain-computer interface device of the double visual signal mixed paradigm. Placing different control signals in the same preset region can solve the above two problems, greatly improving the efficiency of executing operations after receiving signals.

[0033] Preferably, the first light source is a green LED light source, and the plurality of green LED light sources are arranged in a ring. This light source can effectively reduce the occurrence of visual fatigue, and after five consecutive tests, the accuracy rate only decreases by 2.5%, which is significantly improved compared to the case where the accuracy rate decreases by 15-20% for other light sources. The second light source is a red LED light source, and the second visible light signal is realized by randomly flashing in a certain period according to a pseudo-random sequence. Each random flashing event is time-labeled by transmitting a specific character mark to the signal processing device through serial communication. Thus, in the P300 feature extraction, accurate time synchronization between the start of stimulation and the electroencephalogram data acquisition is achieved. After time synchronization, by comparing the random flashing waveform of the second visible light signal with the corresponding random flashing waveform in the P300 feature, it can be determined whether the P300 feature is generated based on the stimulation of the second visible light signal. In the case of multiple second visible light signals, the specific character mark can be used to determine the second visible light signal number corresponding to the P300 feature and all the preset regions.

[0034] Preferably, in at least one embodiment, as shown in Figure 2 The light-emitting device 3 comprises a first light source 11 capable of emitting a first visible light signal and a second light source 21 emitting a second visible light signal. The first light source 11 surrounds the second light source 21, and the area within the outer edge of the first light source 11 is the preset area. Such a setting makes it difficult for the subject to only gaze at one of the first light source 11 and the second light source 21 while ignoring the other, thereby ensuring the identity of the first visible light signal and the second visible light signal stimulation. The first light source 11 is composed of a ring of green LED bulbs. This light source can effectively reduce the occurrence of visual fatigue, and after five consecutive tests, the accuracy rate only decreases by 2.5%, which is significantly improved compared to the case where the accuracy rate decreases by 15-20% for other light sources. The second light source 21 is a red LED bulb, and the second light source 21 is arranged to randomly flash the second visible light signal once in a millisecond-level period according to a pseudo-random sequence. Each random flashing event is time-labeled by transmitting a specific character mark to the signal processing device through serial communication. Thus, in the P300 feature extraction, accurate time synchronization between the start of stimulation and the electroencephalogram data acquisition is achieved. The LED light source can adapt to a microsecond-level timing resolution, which can be as low as 13.89 ns, and the frequency deviation can be controlled to be below 0.2%. Compared with the common light-emitting devices in the prior art, it has a significant advantage.

[0035] Preferably, in at least one embodiment, as shown in Figure 3As shown, the light-emitting device 3 includes 5 preset areas capable of emitting the first visible light signal and the second visible light signal, including 5 first light sources 11-15 capable of emitting the first visible light signal and 5 second light sources 21-25 emitting the second visible light signal. The annular arrangement of the 5 preset areas is an optimized geometric array structure that can maximize the amplitude of the visual evoked potential and improve signal quality. The first light sources 11-15 and the second light sources 21-25 correspond one-to-one, and the second light sources 21-25 are arranged at the annular center of the first light sources 11-15, and the outer edge of the first light sources 11-15 forms 5 preset areas. The first visible light signal is a light signal generated by an LED light source, for example, the first light sources 11-15 are composed of an annular arrangement of green LED bulbs. The preset frequencies of the first light sources 11-15 in each preset area are different, and the preset frequencies of the first light sources 11-15 are set to 11 Hz, 12 Hz, 13 Hz, 14 Hz, and 15 Hz in turn, and the corresponding preliminary control instructions are "front", "back", "left", "right", and "stop". The second visible light signal is a light signal generated by an LED light source, and the second light sources 21-25 are red LED light sources of the same frequency, which are programmed using a pseudo-random stimulation presentation scheme. Each LED flashes once at a random interval within a 2000 millisecond cycle. Each LED flash event is time-stamped by transmitting a specific character marker "F", "B", "L", "R", "S" to the signal processing device 5 through serial communication. The corresponding specific character markers of the second light sources 21-25 are "F", "B", "L", "R", "S" in turn, and the corresponding verification markers are "front", "back", "left", "right", and "stop". Thus, in the P300 feature extraction, precise time synchronization between the stimulus initiation and the electroencephalogram data acquisition is achieved.

[0036] S103: Collecting human brain electrical signals generated based on real-time synchronization of the first visible light signal and the second visible light signal.

[0037] In at least one embodiment, the brain electrical signals are collected using a wireless amplifier system. Six electrodes are arranged according to the international 10-20 system at Fz (midline of the forehead), Cz (midline of the center), Pz (midline of the top), PO7 (left occipital), PO8 (right occipital), and Oz (midline of the occipital), and the conductive paste keeps the impedance below 5kΩ, the reference electrode is the left mastoid, and the grounding electrode is AFz.

[0038] S1041: Extracting SSVEP features based on the brain electrical signals, and generating preliminary control instructions based on the SSVEP features. The raw brain electrical data is sequentially processed by 50Hz notch filtering, 6.5-30Hz fourth-order Butterworth SSVEP signal low-pass filtering, and 15Hz cutoff fourth-order Butterworth P300 signal low-pass filtering to process noise in the signal and retain and enhance the effective signal.

[0039] The SSVEP feature extraction method of the electroencephalogram signal can use canonical correlation analysis method, filter bank canonical correlation analysis method, task-related component extraction method, etc., and can also use other analysis methods. The feature extraction angle is not limited to the frequency domain, and can also be extracted from the time domain, spatial domain, or a combination of the three.

[0040] Preferably, in at least one embodiment, the SSVEP feature extraction can be realized based on the FFT (Fast Fourier Transform) frequency domain analysis method. The FFT (Fast Fourier Transform) frequency domain analysis method can convert the waveform formed by the discrete signal sampling points into a corresponding frequency spectrum, which is also discrete. The amplitude of each point in the frequency spectrum is the amplitude at that frequency, and the larger the amplitude, the stronger the frequency component.

[0041] In existing research, since the signal characteristics of the SSVEP signal in the frequency domain are more significant, the SSVEP time domain signal is usually converted to the frequency domain for feature extraction, and then combined with the powerful automatic feature extraction capability of the neural network to further improve the performance of SSVEP signal classification. Combining Fourier transform with neural network can more efficiently convert SSVEP signal from time domain to frequency domain.

[0042] Preferably, in at least one embodiment, the SSVEP feature extraction uses Welch power spectral density estimation, Hanning window, 50% overlap, and amplitude peak value of the target frequency is collected.

[0043] Preferably, in at least one embodiment, the steps of extracting SSVEP features and extracting P300 features are processed in parallel. The FFT frequency domain analysis method is used to extract SSVEP features, and the time range peak detection method is used to extract P300 features, and the delay of the signal is <50ms, which is much better than the prior art (>200ms).

[0044] S1042: Extract P300 features based on the electroencephalogram signal, and generate a verification label based on the P300 features.

[0045] The P300 feature extraction method of the electroencephalogram signal can use common space model, adaptive regression model, spectral analysis method, waveform parameter calculation method, wavelet transform method, and other analysis methods.

[0046] Preferably, in at least one embodiment, the P300 feature is extracted by using the time range peak detection method. The electroencephalogram data is aligned by the microcontroller event marker, and the peak value of the visual evoked P300 component is detected in the 290-500 millisecond window after stimulation.

[0047] Preferably, in at least one embodiment, as Figure 1The visual brain-computer interface control method shown, the brain electrical signal processing process includes the following steps:

[0048] S105: Verify the consistency of the preliminary control instruction and the verification mark. Among them, the preliminary control instruction obtained based on the SSVEP frequency detection is the main classification basis, and the verification mark obtained based on the P300 event is the auxiliary classification basis. That is, the final control instruction is made based on the preliminary control instruction obtained based on the SSVEP frequency detection, and the verification mark obtained based on the P300 event is used for further verification. This method of main detection supplemented by verification can significantly improve the classification reliability and reduce the misjudgment rate.

[0049] S1061: When the preliminary control instruction is inconsistent with the verification mark, abandon the preliminary control instruction. In most scenarios of brain-computer interface applications, for example, medical treatment, control of external devices, and the like, safety is the primary consideration. Under the premise of first ensuring safety rather than efficiency, it is a reasonable strategy to abandon part of the preliminary control instruction. When the preliminary control instruction obtained based on the SSVEP frequency detection is the main classification basis, and the verification mark obtained based on the P300 event is the auxiliary classification basis, inconsistent results are obtained. The control instruction obviously has the possibility of error. At this time, the control instruction is executed recklessly, which may bring unacceptable serious consequences, therefore, the preliminary control instruction must be abandoned.

[0050] S1062: When the preliminary control instruction is consistent with the verification mark, generate an output control instruction corresponding to the preset area. When the preliminary control instruction obtained based on the SSVEP frequency detection is the main classification basis, and the verification mark obtained based on the P300 event is the auxiliary classification basis, consistent results are obtained. The possibility of error of the control instruction is very low at this time, and the correct control instruction can be output.

[0051] S107: Control the external device based on the generated output control instruction.

[0052] As Figure 3In the shown embodiment, the first light sources 11-15 are green LED light sources, preset frequencies are set to 11 Hz, 12 Hz, 13 Hz, 14 Hz, 15 Hz respectively, and the corresponding preliminary control instructions are "forward", "backward", "left", "right", "stop". The second light sources 21-25 are red LED light sources with the same frequency, and the five red LEDs are programmed using a pseudo-random stimulation presentation scheme. Each LED flashes once at a random interval within a 2000 millisecond cycle. Each LED flash event is time-stamped by transmitting a specific character mark "F", "B", "L", "R", "S" to the signal processing device 5 through serial communication. The corresponding specific character marks of the second light sources 21-25 are "F", "B", "L", "R", "S" in turn, and the corresponding verification marks are "forward", "backward", "left", "right", "stop". Thus, in the P300 feature extraction, accurate time synchronization between the start of the stimulus and the electroencephalogram data collection is achieved. For example, when the subject gazes at the first light source 11 and the second light source 21 in Figure 3 , since the first light source 11 surrounds the second light source 21, the subject must simultaneously gaze at the visible light signals emitted by the two light sources. Since the first light source 11 emits a 11 Hz green LED light signal, the SSVEP feature corresponding to the 11 Hz preset frequency can be extracted, and based on the extracted SSVEP feature, the preliminary control instruction "forward" can be generated. At the same time, the second light source 21 emits a red LED light signal, and the subject observes the red LED light source, and the corresponding P300 feature can be extracted. The microcontroller controls the second light source 21 to emit the red LED light signal at the same time, and transmits a specific character mark "F" to the signal processing device 5 through serial communication for time stamping. In the P300 feature extraction, based on the time of issuing the instruction given by the controller and the time of collecting the electroencephalogram signal data, the signal processing device 5 can achieve accurate time synchronization between the start of the stimulus and the electroencephalogram data collection. After time synchronization, the corresponding random flashing waveform in the P300 feature is compared with the random flashing waveform emitted by the second light sources 21-25, for example, the P300 feature collected within a period of time matches the flashing event of the light source with the specific character mark "F", and based on the extracted P300 feature, the verification mark "forward" can be generated. When the preliminary control instruction and the verification mark are consistent, both are "forward", the "forward" control instruction is output, and the external device is controlled to move forward based on the "forward" control instruction. Other control instructions are obtained according to the same method, for example, when the subject gazes at the first light source 13 and the second light source 23 in Figure 3 , the preliminary control instruction and the verification mark are consistent, both are "left", and at this time the "left turn" control instruction is output.

[0053] In many cases, due to external interference or the subject's psychology and other factors, the wrong SSVEP feature may be extracted, and the wrong preliminary control instruction is obtained. At this time, because the corresponding P300 feature is not collected, no verification mark is generated, so the wrong control instruction is not output.

[0054] Preferably, before step 101, a training step of the system is further included. The subject gazes at the preset area emitting the first visible light signal and the second visible light signal for multiple times, and the electroencephalogram of the subject is collected. Thus, the SSVEP feature corresponding to different first visible light signal frequencies and the P300 feature corresponding to the second visible light signal are obtained.

[0055] Preferably, in at least one embodiment, the control instruction can be transmitted to the external device through Bluetooth; wired transmission modes including but not limited to, for example, RS485 bus, MODBUS, HART, PROFIBUS, etc., and wireless communication modes such as LoRa, NB-IoT, 4G, etc. can also be used.

[0056] Preferably, in at least one embodiment, the external device is one or more of, for example, a wheelchair, an external limb, a robot, a human-computer interaction interface, etc.

[0057] According to the embodiments of the present disclosure, Figure 4 A visual brain-computer interface control device 1 is disclosed, comprising a microcontroller 2, a light emitting device 3, a signal collecting device 4 and a signal processing device 5. The microcontroller 2 is configured to output a light source control signal, the light source control signal comprising a first control signal corresponding to a preset frequency and a second control signal corresponding to a fixed frequency. The first control signal of the preset frequency is used to control the visible light that stimulates the generation of the SSVEP signal. The SSVEP signal generated by the human brain has a strong correlation with the frequency of the stimulating light source, so by setting the preset frequency, the corresponding SSVEP signal can be captured. By setting different preset frequencies for different stimulating light sources, multiple preset light sources can be effectively distinguished. The second control signal of the fixed frequency is used to control the visible light that stimulates the generation of the P300 signal. The P300 signal generated by the human brain has no obvious correlation with the frequency of the stimulating light source, so the frequency of the second control signal is fixed. The first control signal and the second control signal are output at the same time, and the light source is controlled to emit light at the same time. The SSVEP signal and the P300 signal obtained by stimulating the two light sources at the same time are processed, and the control information at the time point can be obtained, and the control information is verified.

[0058] The light emitting device 3 is connected with the microcontroller 2, and includes five preset areas capable of emitting the first visible light signal and the second visible light signal, wherein the five preset areas include five first light sources 11-15 capable of emitting the first visible light signal and five second light sources 21-25 capable of emitting the second visible light signal. The first light sources 11-15 and the second light sources 21-25 are in one-to-one correspondence, and the second light sources 21-25 are respectively arranged at the annular centers of the first light sources 11-15, and the outer edges of the first light sources 11-15 form the five preset areas. The first visible light signal is a light signal generated by an LED light source, for example, the first light sources 11-15 are composed of annularly arranged green LED bulbs. The preset frequencies of emitting the first visible light signal in the preset areas are different from each other, and the preset frequencies of the first light sources 11-15 are respectively set as 11 Hz, 12 Hz, 13 Hz, 14 Hz and 15 Hz, and the corresponding preliminary control instructions are “forward”, “backward”, “left”, “right” and “stop”. The second visible light signal is a light signal generated by an LED light source, and the second light sources 21-25 are red LED light sources of the same frequency, which are programmed by using a pseudo-random stimulation presentation scheme. Each LED generates a flash at a random interval within a 2000 millisecond cycle. Each LED flash event is time-stamped by transmitting a specific character mark “F”, “B”, “L”, “R” and “S” to the signal processing device 5 through serial communication. The corresponding specific character marks of the second light sources 21-25 are “F”, “B”, “L”, “R” and “S” in turn, and the corresponding verification marks are “forward”, “backward”, “left”, “right” and “stop”. Thus, in the P300 feature extraction, the precise time synchronization between the stimulation initiation and the electroencephalogram data acquisition is realized.

[0059] The signal acquisition device 4 is arranged in the brain region of the subject, and is configured to acquire the electroencephalogram generated by the human brain based on the real-time synchronous stimulation of the first visible light signal and the second visible light signal. The signal acquisition device 4 includes a wireless amplifier system, and six electrodes are arranged on Fz (midline of forehead), Cz (midline of central), Pz (midline of top), PO7 (left top occipital), PO8 (right top occipital) and Oz (midline of occipital) according to the international 10-20 system, and the conductive paste keeps the impedance below 5kΩ, the reference electrode is the left mastoid, and the grounding electrode is AFz.

[0060] The signal processing device 5 is connected with the signal acquisition device 4, and the original electroencephalogram data is sequentially processed by 50Hz notch filtering, 6.5-30Hz fourth-order Butterworth SSVEP signal low-pass filtering and 15Hz cutoff fourth-order Butterworth P300 signal low-pass filtering, to process the noise in the signal, retain and enhance the effective signal.

[0061] The SSVEP feature is extracted based on the electroencephalogram signal, and the P300 feature is extracted based on the electroencephalogram signal. The SSVEP feature extraction can be realized based on the FFT (Fast Fourier Transform) frequency domain analysis method. The FFT (Fast Fourier Transform) frequency domain analysis method can convert the waveform formed by the discrete signal sampling points into a corresponding frequency spectrum, which is also discrete. The amplitude value corresponding to each point in the frequency spectrum is the amplitude value at that frequency, and the greater the amplitude value means that the frequency component is stronger. The P300 feature is extracted by using the peak value detection method in the time range. The electroencephalogram data is aligned by the microcontroller event marker, and the peak value of the visual evoked P300 component is detected in the 290-500 millisecond window after the stimulation.

[0062] The preliminary control instruction is generated based on the SSVEP feature, and the verification marker is generated based on the P300 feature. For example, when the subject gazes at the first light source 11 and the second light source 21 in Figure 3 , since the first light source 11 surrounds the second light source 21, the subject must gaze at the visible light signals emitted by the two light sources at the same time. Since the first light source 11 emits a 11Hz green LED light signal, the SSVEP feature corresponding to the 11Hz preset frequency can be extracted, and based on the extracted SSVEP feature, the preliminary control instruction can be generated as "forward". At the same time, the second light source 21 emits a red LED light signal, and the subject observes the red LED light source, and the corresponding P300 feature can be extracted. The microcontroller 2 controls the second light source 21 to emit a red LED light signal at the same time, and transmits a specific character marker "F" to the signal processing device through serial communication for time marking. Based on the control instruction time given by the controller and the electroencephalogram signal data acquisition time, the signal processing device can realize accurate time synchronization between the stimulation start and the electroencephalogram data acquisition in the P300 feature extraction. After time synchronization, the corresponding random flicker waveform in the P300 feature is compared with the random flicker waveform emitted by the second light source 21-25, for example, the P300 feature collected in a period of time matches the flicker event of the light source with the specific character marker "F", and based on the extracted P300 feature, the verification marker can be generated as "forward". The preliminary control instruction and the verification marker are consistent, both are "forward", and the "forward" control instruction is output at this time. Other control instructions are obtained according to the same method, for example, when the subject gazes at the first light source 13 and the second light source 23 in Figure 3 , the preliminary control instruction and the verification marker are consistent, both are "left", and the "left turn" control instruction is output at this time.

[0063] The consistency of the preliminary control instruction and the verification mark is verified, when the preliminary control instruction is inconsistent with the verification mark, the preliminary control instruction is abandoned, and when the preliminary control instruction is consistent with the verification mark, the output control instruction corresponding to the preset area is generated. For example, when the preliminary control instruction and the verification mark are consistent, both are "front", the control instruction of "forward" is output, and the external device is controlled to move forward based on the control instruction of "forward". In many cases, because of external interference or the subject's psychology and many other factors, the wrong SSVEP feature may be extracted, and the wrong preliminary control instruction is obtained. At this time, because the corresponding P300 feature is not collected, the verification mark will not be generated, so the wrong control instruction will not be output.

[0064] Preferably, in at least one embodiment, the microcontroller 2 has a multi-thread architecture, supporting multiple independent frequency outputs.

[0065] Preferably, in at least one embodiment, the data transmission between the microcontroller 2 and the signal processing device 5 is completed through the FTD FT232R USB-UART interface, so as to ensure that the communication delay of the event mark is lower than 1 millisecond, meeting the synchronization requirement of the accurate extraction of the P300 event.

[0066] Preferably, in at least one embodiment, the visual brain-computer interface control device can be used to control the electric lamp switch. For example Figure 2 The light emitting device shown includes a preset area, when the subject looks at the area, the control instruction is output according to the method of the embodiment of the present disclosure. The subject does not look at the area, that is, there is no control instruction. The control instruction is output to the electric lamp switch remote control system through a wireless signal, and the electric lamp can be turned on or off according to the control instruction. If the electric lamp receives a control instruction, it will perform an on or off operation. The visual brain-computer interface control device can control the external equipment of the switch, which includes not only the electric lamp, but also the curtain, electric rice cooker, television, etc.

[0067] Preferably, in at least one embodiment, the external device is a robot. For example Figure 3 As shown, when the light emitting device includes 5 preset areas, the control instructions corresponding to the 5 preset areas are "forward", "backward", "left", "right" and "stop". The control instruction is output to the robot through a wireless signal, and the robot can move or stop according to the direction indicated by the signal. The appearance of the robot is not limited, and it can also be a robot dog or a robot car. The control instruction can be set according to the demand, and the motion behavior pointed to is not limited to forward, backward, left and right, but also can be jumping, rolling, raising hands, squatting, picking up, cleaning, lighting and playing music.

[0068] Preferably, in at least one embodiment, the signal acquisition device 4 is integrated in a head-mounted device, such as a portable electroencephalogram cap.

[0069] The following points need to be explained:

[0070] (1) The drawings of the embodiments of the present disclosure only relate to the structures involved in the embodiments of the present disclosure, and other structures can be referred to the general design.

[0071] (2) For the sake of clarity, the thickness of a device, layer or region in the drawings used to describe the embodiments of the present disclosure is exaggerated or reduced, that is, the drawings are not drawn according to the actual proportion. It can be understood that when an element such as a layer, film, region or substrate is referred to as being located "on" or "under" another element, the element can be "directly" located "on" or "under" another element or there can be an intermediate element.

[0072] (3) In the case of no conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0073] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A visual brain-computer interface control method, comprising: simultaneously outputting a first control signal corresponding to a preset frequency and a second control signal corresponding to a fixed frequency; emitting a first visible light signal of the preset frequency based on the first control signal and a second visible light signal of the fixed frequency based on the second control signal, the first visible light signal and the second visible light signal being located in a same preset area; collecting a human brain electrical signal generated by real-time synchronous stimulation based on the first visible light signal and the second visible light signal; extracting an SSVEP feature based on the brain electrical signal and generating a preliminary control instruction based on the SSVEP feature; extracting a P300 feature based on the brain electrical signal and generating a verification mark based on the P300 feature; verifying consistency of the preliminary control instruction and the verification mark, discarding the preliminary control instruction when the preliminary control instruction and the verification mark are inconsistent, and generating an output control instruction corresponding to the preset area when the preliminary control instruction and the verification mark are consistent.

2. The visual brain-computer interface control method of claim 1, wherein, a plurality of preset areas capable of emitting the first visible light signal and the second visible light signal; the first visible light signal and the second visible light signal are both light signals generated by LED light sources; the preset frequencies at which the first visible light signal is emitted in each of the preset areas are different from each other; the preset frequency range is 7-30 HZ.

3. The visual brain-computer interface control method of claim 1, wherein, the SSVEP feature is extracted using an FFT frequency domain analysis method, and the P300 feature is extracted using a peak value detection method in a time range, and the steps of extracting the SSVEP feature and extracting the P300 feature are processed in parallel.

4. The visual brain-computer interface control method of claim 3, wherein, the second control signal includes random flickering in a certain period according to a pseudo-random sequence, and each random flickering event is time-labeled by transmitting a specific character mark through serial communication; in the P300 feature extraction, time synchronization between the second control signal and the brain electrical signal is achieved based on the time label.

5. The visual brain-computer interface control method of claim 1, wherein, further comprising a step of controlling an external device based on the generated output control instruction.

6. A visual brain-computer interface control device comprising: a microcontroller, a light emitting device, a signal collecting device, and a signal processing device, wherein the microcontroller is configured to output a light source control signal, the light source control signal including a first control signal corresponding to a preset frequency and a second control signal corresponding to a fixed frequency; the light emitting device is connected with the microcontroller and includes a first light source and a second light source arranged in a same preset area, the first light source emitting a first visible light signal based on the first control signal, and the second light source emitting a second visible light signal based on the first control signal; the signal collecting device is arranged in a human brain area and is configured to collect a brain electrical signal generated by real-time synchronous stimulation of the human brain based on the first visible light signal and the second visible light signal; the signal processing device is connected with the signal collecting device, extracts an SSVEP feature based on the brain electrical signal, extracts a P300 feature based on the brain electrical signal, generates a preliminary control instruction based on the SSVEP feature, and generates a verification mark based on the P300 feature; The consistency of the preliminary control instruction and the verification mark is verified, when the preliminary control instruction is inconsistent with the verification mark, the preliminary control instruction is abandoned, and when the preliminary control instruction is consistent with the verification mark, the output control instruction corresponding to the preset area is generated.

7. The visual brain-computer interface control device of claim 6, wherein, The first light source is a green annular LED light source array composed of a plurality of LED light sources. The second light source is a red LED light source arranged at the center of the green annular LED light source array.

8. The visual brain-computer interface control apparatus according to claim 7, wherein, The microcontroller has a multi-thread architecture and supports multiple independent outputs of the plurality of light source control signals. The plurality of light source control signals correspond to the first light source and the second light source of a plurality of preset areas respectively, and the first light source in each of the preset areas emits the preset frequency of the first visible light signal which is different from each other.

9. The visual brain-computer interface control device of claim 8, wherein, Further comprising an external device, the light emitting device includes 5 preset areas, 5 output control instructions corresponding to 5 preset areas respectively, and 5 output control instructions control the forward, backward, left turn, right turn and stop of the external device respectively.

10. The visual brain-computer interface control apparatus according to claim 6, wherein, The signal acquisition device includes an acquisition electrode, a reference electrode and a grounding electrode. The acquisition electrode is arranged at the midline of the forehead, the midline of the center, the midline of the top, the left top occiput, the right top occiput and the midline of the occiput of the human brain. The reference electrode is arranged at the left mastoid position of the human body.

Images