Mixed brain-computer interface method combined with visual and auditory weak implicit stimulation

Abstract

The invention relates to a mixed brain-computer interface method combined with visual and auditory weak implicit stimulation, which is technically characterized by comprising the following steps of: determining a video stimulation interface, the number of visual stimulation targets and visual stimulation attributes, and realizing a visual weak implicit stimulation function in a high-frequency flickering mode of a peripheral visual field; the left and right ears apply binaural frequency division low sound pressure level auditory stimulation signals to realize an auditory weak implicit stimulation function; simultaneously performing visual ambiguity stimulation and auditory ambiguity stimulation, and selectively paying attention to the auditory stimulation of the left ear or the right ear to complete a mixed paradigm task; and extracting steady-state visual evoked potential features and auditory steady-state response features of the electroencephalogram signals of the user and carrying out identification and classification while visual weak implicit stimulation and auditory weak implicit stimulation are carried out, and generating a coding instruction. According to the method, a visual weak-implicit stimulation mode and an auditory weak-implicit stimulation mode are combined, two electroencephalogram signals are used for coding at the same time, the naturalness and friendly degree of man-machine interaction are improved, and meanwhile more targets can be coded with less frequency.

Description

Technical Field

[0001] The present invention belongs to the field of neural engineering technology and relates to a brain-computer interface stimulation method, in particular to a hybrid brain-computer interface method combining visual and auditory weak implicit stimulation. Background Art

[0002] Brain-computer interface technology establishes a new output channel between the brain and external devices that does not rely on peripheral nerves and muscles, and can realize the direct interaction between the brain and the outside world. Brain-computer interfaces include invasive and non-invasive types. Among non-invasive brain-computer interface signal acquisition methods, scalp EEG has a dominant position due to its low cost and portable acquisition equipment.

[0003] For the brain-computer interface paradigm, the SSVEP (steady-state visual evoked potential, SSVEP) brain-computer interface paradigm utilizes the consistency of visual cortical response and specific frequency flicker in the frequency domain. It has the advantages of less system calibration time, less training required, and high information transmission rate. It has become one of the three mainstream paradigms of brain-computer interface. At present, the mainstream visual stimulation is mainly presented in the central visual field, and strong flicker stimulation with medium and low frequencies is used to induce SSVEP. The following patent documents: a hybrid brain-computer interface method for visual and auditory combination (patent application number CN202110837782.4), a brain-computer interface system and a performance test method for a brain-computer interface system (patent application number CN202311235750.2), and a brain-computer interface auditory spatial attention detection system driven by a similar auditory system (patent application number CN 202311530029.6) all use strong stimulation methods. This strong stimulation encoding method will lead to unnatural human-computer interaction and low comfort level. When users pay attention to a strongly flickering target for a long time, they are prone to visual fatigue and even risk photosensitive epilepsy. The user's fatigue state will also affect the performance of the SSVEP brain-computer interface, resulting in a poor human-computer interaction experience. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a hybrid brain-computer interface method that combines visual and auditory weak implicit stimulation, which can significantly improve the comfort of using the brain-computer interface.

[0005] The present invention solves the existing technical problems by adopting the following technical solutions:

[0006] A hybrid brain-computer interface method combining visual and auditory weak implicit stimulation, comprising the following steps:

[0007] Step 1, determine the video stimulation interface, the number of visual stimulation targets and the visual stimulation attributes, and realize the visual dim stimulation function by high-frequency flickering of the peripheral visual field;

[0008] Step 2, applying binaural frequency division low sound pressure level auditory stimulation signals to the left and right ears to realize the auditory weak stimulation function;

[0009] Step 3: Perform visual and auditory dim stimulation simultaneously. The user fixates on the flashing of a target and selectively pays attention to the auditory stimulation in the left or right ear to complete the mixed paradigm task.

[0010] Step 4: While performing visual dim stimulation and auditory dim stimulation, extract the steady-state visual evoked potential characteristics and auditory steady-state response characteristics of the user's EEG signal, identify and classify them, and generate coding instructions.

[0011] Furthermore, the visual stimulation interface is a left area and a right area presented on the display screen, the left area and the right area correspond to left ear stimulation and right ear stimulation respectively, and the visual stimulation attributes of the targets in the two areas are the same.

[0012] Furthermore, the number of stimulation targets n=2k, each visual stimulation target is presented in the form of peripheral visual field and high-frequency flicker; the video stimulation attributes include visual stimulation frequency F i and Phase The inner ring viewing angle α and the outer ring viewing angle β of the peripheral field of view, the inner ring radius r1 and the outer ring radius r2 of the target ring and the position distribution, where i=1,…,k, k≥2.

[0013] Furthermore, the visual stimulation frequency F i ≥30Hz, the inner ring viewing angle α>2°.

[0014] Furthermore, the auditory stimulation signal is a sinusoidal amplitude modulated low sound pressure level signal in a binaural frequency division form, the stimulation frequency of the auditory stimulation signal includes the modulation frequency of the left and right ear signals and the modulation frequency of the signal carrier frequency, and the sound pressure level SPL of the auditory stimulation signal is ≤45dB.

[0015] Furthermore, in step 3, when the user is looking at the flashing of a certain target, he / she pays attention to the flashing of a certain target according to the prompt. If the target is located on the left side of the stimulation interface, the user needs to pay attention to the auditory stimulation of the left ear; if the target is located on the right side of the stimulation interface, the user needs to pay attention to the auditory stimulation of the right ear.

[0016] Furthermore, the step 4 uses a filter bank canonical correlation analysis algorithm and a filter bank common space pattern algorithm to extract steady-state visual evoked potential features and auditory steady-state response features of the user's EEG signal and perform identification and classification.

[0017] The advantages and positive effects of the present invention are:

[0018] 1. The present invention combines visual weak implicit stimulation mode with auditory weak implicit stimulation mode, and uses two EEG signals for encoding at the same time, wherein an auditory stimulation of one frequency and a visual stimulation correspond to a unique target, through the high-frequency flickering visual stimulation of the peripheral visual field and the auditory stimulation of the low sound pressure level sinusoidal amplitude modulated signal, which fully considers the sensory stimulation properties, significantly improves the comfort of using the brain-computer interface, enhances the naturalness and friendliness of human-computer interaction, and can encode more targets with fewer frequencies, and has the potential to expand the number of brain-computer interface instructions.

[0019] 2. The present invention uses high-frequency, imperceptible flickering to induce SSVEP, and can reduce the sense of flicker and improve comfort. It vacates the central visual field and uses the peripheral visual field for stimulation encoding. It can induce SSVEP while vacating the visual pathway, reduce the user's visual fatigue, and be closer to natural application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a coding schematic diagram of the present invention;

[0021] Figure 2 is a schematic diagram of the stimulation interface of the present invention;

[0022] Figure 3 Flow chart of each trial of the online experiment of the present invention. DETAILED DESCRIPTION

[0023] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0024] The design concept of the present invention is:

[0025] Since ASSR (auditory steady-state response) is a steady-state response of the brain caused by periodic sound signals, it has good frequency characteristics. When the user is subjected to auditory stimulation of a specific frequency, the corresponding frequency components can be extracted from the scalp electroencephalogram. The spectrum of ASSR has obvious peaks at the stimulation frequency and its harmonics. When the left and right ears are stimulated at the same time, the brain area with the strongest response is the frontal lobe-central area. The spatial attention mechanism plays a decisive role in multiple competing sound sources and also plays an important role in detecting and analyzing spatial sound movement. Based on the spatial attention mechanism, ASSR exhibits certain spatial domain characteristics, and spatial filtering technology can be used to effectively extract the characteristics of ASSR signals to improve the discriminability of ASSR. At the same time, considering that in the process of obtaining external information, the visual and auditory senses account for the highest proportion of information, which is the main source channel for the brain to obtain information, and the audio-visual interaction combination mode is better than a single sensory mode, mobilizing multiple senses is more conducive to brain-computer interaction. Therefore, the present invention uses two high-frequency flicker frequencies to induce SSVEP in the peripheral visual field, and uses two low sound pressure level sinusoidal amplitude modulation signals to induce ASSR, and the two paradigms encode targets in parallel. The user selectively pays attention to the sound stimulation on one side and looks at the target at the same time to complete the mixed paradigm task. The corresponding algorithms are used to extract features and classify the ASSR and SSVEP signals respectively, and finally the coding instructions are output in real time.

[0026] Based on the above design concept, the present invention proposes a hybrid brain-computer interface method combining visual and auditory weak implicit stimulation, including the following steps:

[0027] Step 1: Determine the video stimulation interface, the number of stimulation targets and the visual stimulation attributes, and realize the visual dim stimulation function by means of high-frequency flickering of the peripheral visual field.

[0028] According to the actual application scenario and requirements, the present invention divides the stimulation interface into two areas, the left area and the right area, which correspond to the left ear stimulation and the right ear stimulation respectively, and the visual stimulation attributes of the targets in the two areas are the same. The number of stimulation targets n = 2k (k ≥ 2, k∈N * ).

[0029] The present invention presents the visual stimulation target through the peripheral visual field and high-frequency flickering, that is, the visual dim stimulation function is realized through the high-frequency flickering of the peripheral visual field. When performing dim stimulation, the following parameters need to be determined: the distance between the user's head (eyes) and the display is fixed by a headrest, the video stimulation attributes include the visual stimulation frequency F i (i=1,…,k,F i ≥30Hz) and phase The inner ring viewing angle α (α>2°) and the outer ring viewing angle β of the peripheral field of vision, the inner ring radius r1 and the outer ring radius r2 of the target ring, and the position distribution.

[0030] like Figure 1 and Figure 2 As shown, the following takes the stimulation target number n=4 (k=2) as an example to explain this step in detail:

[0031] The stimulation interface was presented using a display with a screen resolution of 1920×1080 pixels and a screen refresh rate of R=120Hz. The display was 54.1cm high and 30.3cm wide. A headrest was used to fix the user's eyes to the center of the display, 50cm away from the display. The inner ring viewing angle α=2.1° and the outer ring viewing angle β=4.1° were set. The inner ring radius r1=65 pixels and the outer ring radius r2=127 pixels were calculated and rounded. Taking the upper left corner of the stimulation interface as the coordinate origin, the center coordinates of the four targets were (640,315), (640,465), (1280,315), and (1280,465). The background RGB value of the stimulation interface was [102102102]. For the targets in the left and right areas, two different flicker frequencies were assigned, namely F1=34Hz and F2=35Hz, and the flicker phase was 0. The sine sampling coding method is used to adjust the brightness of each frame target so that it changes according to the law of the sine wave and achieves a flicker of a specified frequency. For each visual stimulus target, its brightness can be expressed as:

[0032]

[0033] Among them, s is the brightness of the visual stimulus target in each frame, f is the frame index of the stimulus sequence, and R is the screen refresh rate.

[0034] Step 2: Apply binaural frequency-divided low sound pressure level auditory stimulation signals to the left and right ears to realize the auditory weak stimulation function.

[0035] The auditory stimulation signal is a sinusoidal amplitude modulated low sound pressure level signal in the form of binaural frequency division. The stimulation frequency of the auditory stimulation signal includes the modulation frequency f of the left and right ear signals. mi (i=1,2), signal carrier frequency modulation frequency f ci (i=1,2) and auditory stimulation sound pressure level SPL≤45dB.

[0036] The auditory stimulation signal of this embodiment can be expressed as:

[0037] c(t)=Asin(πf mi t)sin(2πf ci t),i=1,2 (2)

[0038] Where t is time and A is the amplitude modulation coefficient. In this embodiment, binaural frequency division is used to perform auditory stimulation simultaneously, and the modulation frequency f of the left ear stimulation signal is m1=34Hz, carrier frequency f c1 =2500Hz, the modulation frequency of the right ear stimulation signal f m2 =35Hz, carrier frequency f c2 =1000Hz, the signal sampling rate is 44100Hz, and the sound card parameters are adjusted to make the auditory signal sound pressure level 40dB.

[0039] Step 3: Perform visual dim stimulation and auditory dim stimulation simultaneously according to the methods of steps 1 and 2. The user fixates on the flashing of a target and selectively pays attention to the auditory stimulation in the left or right ear to complete the mixed paradigm task.

[0040] In this step, the ASSR task and the SSVEP task are performed simultaneously: the user determines the area and position of the target to be paid attention to based on the randomly appearing prompts in the prompt stage, and completes the ASSR task by selectively paying attention to the left ear stimulation signal or the right ear stimulation signal in the next task stage, that is, paying attention to the flickering of a certain target according to the prompt. If the target is located on the left side of the stimulation interface, the user needs to pay attention to the auditory stimulation of the left ear; if the target is located on the right side of the stimulation interface, the user needs to pay attention to the auditory stimulation of the right ear. At the same time, the SSVEP task is completed by paying attention to the flickering visual stimulation target. In the task stage, the user looks at the cross in the center of the target for visual fixation, and the visual pathway of the central visual field is vacated, so there is no need to look directly at the flickering of the target.

[0041] like Figure 3 As shown in the figure, the specific process of the online experiment in this step is as follows: the user needs to complete a total of 10 blocks of online experimental tasks. Each block of the online experiment contains 20 trials, and each trial contains a blank time stage, a prompt stage, and a task stage. At the beginning of each trial, a 0.5s blank time stage is set to remind the user that the task is about to start. In the blank time stage, the stimulus interface presents all targets, but does not present visual flickering stimulation and auditory stimulation. The RGB value of the target is [178178178]. The user only needs to adjust the state to prepare for the next task; the prompt stage lasts for 0.5s, and the stimulus interface randomly presents a red box with a side length of 300 pixels to select a target, prompting the user to pay attention to the target in the task stage; the task stage lasts for 5s, and the stimulus interface presents the flickering of all visual stimulation targets, and the modulated auditory stimulation signal is played simultaneously in both ears. The subject completes the SSVEP task by focusing on the flickering of the target prompted in the prompt stage. At the same time, if the target is located on the left side of the stimulation interface, the user needs to pay attention to the auditory stimulation of the left ear; if the target is located on the right side of the stimulation interface, the user needs to pay attention to the auditory stimulation of the right ear.

[0042] Step 4: While performing visual dim stimulation and auditory dim stimulation, extract the steady-state visual evoked potential characteristics and auditory steady-state response characteristics of the user's EEG signal, identify and classify them, and generate coding instructions.

[0043] In this step, existing technologies are used to extract features of user EEG signals and classify and identify them. For example, the filterbank canonical correlation analysis (FBCCA) algorithm and the filterbank common spatial pattern algorithm can be used to extract SSVEP and ASSR features and classify them, respectively, to achieve real-time feedback function.

[0044] To decode this encoding method, corresponding algorithms need to be used to extract features and classify ASSR signals and SSVEP signals respectively, and finally the classification results are output to form a brain-computer interface system with real-time feedback.

[0045] It should be emphasized that the embodiments described in the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation manner. Any other implementation manners derived by those skilled in the art based on the technical solution of the present invention also fall within the scope of protection of the present invention.

Claims

1. A hybrid brain-computer interface method combining visual and auditory weak implicit stimulation, characterized in that: The following steps are involved: Step 1, determine the video stimulation interface, the number of visual stimulation targets and the visual stimulation attributes, and realize the visual dim stimulation function by high-frequency flickering of the peripheral visual field; Step 2, applying binaural frequency division low sound pressure level auditory stimulation signals to the left and right ears to realize the auditory weak stimulation function; Step 3: Perform visual and auditory dim stimulation simultaneously. The user fixates on the flashing of a target and selectively pays attention to the auditory stimulation in the left or right ear to complete the mixed paradigm task. Step 4: While performing visual dim stimulation and auditory dim stimulation, extract the steady-state visual evoked potential characteristics and auditory steady-state response characteristics of the user's EEG signal, identify and classify them, and generate coding instructions.

2. The hybrid brain-computer interface method of combined visual and auditory weak implicit stimulation according to claim 1 is characterized in that: The visual stimulation interface is a left area and a right area presented on the display screen. The left area and the right area correspond to left ear stimulation and right ear stimulation respectively, and the visual stimulation attributes of the targets in the two areas are the same.

3. The hybrid brain-computer interface method of combined visual and auditory weak implicit stimulation according to claim 2 is characterized in that: The number of stimulation targets n=2k, each visual stimulation target is presented in the form of peripheral visual field and high-frequency flicker; the video stimulation attributes include visual stimulation frequency F i and Phase The inner ring viewing angle α and the outer ring viewing angle β of the peripheral field of view, the inner ring radius r1 and the outer ring radius r2 of the target ring and the position distribution, where i=1,…,k, k≥2.

4. The hybrid brain-computer interface method of combined visual and auditory weak implicit stimulation according to claim 3 is characterized in that: The visual stimulus frequency F i ≥30Hz, the inner ring viewing angle α>2°.

5. The hybrid brain-computer interface method for combined visual and auditory weak implicit stimulation according to any one of claims 1 to 4, characterized in that: The auditory stimulation signal is a sinusoidal amplitude modulated low sound pressure level signal in binaural frequency division form. The stimulation frequency of the auditory stimulation signal includes the modulation frequency of the left and right ear signals and the modulation frequency of the signal carrier frequency. The sound pressure level SPL of the auditory stimulation signal is ≤45dB.

6. The hybrid brain-computer interface method for combined visual and auditory weak implicit stimulation according to any one of claims 1 to 4, characterized in that: In step 3, when the user is looking at the flashing of a certain target, he / she pays attention to the flashing of a certain target according to the prompt. If the target is located on the left side of the stimulation interface, the user needs to pay attention to the auditory stimulation of the left ear; if the target is located on the right side of the stimulation interface, the user needs to pay attention to the auditory stimulation of the right ear.

7. The hybrid brain-computer interface method for combined visual and auditory weak implicit stimulation according to any one of claims 1 to 4, characterized in that: The step 4 uses the filter bank canonical correlation analysis algorithm and the filter bank common space pattern algorithm to extract the steady-state visual evoked potential characteristics and auditory steady-state response characteristics of the user's EEG signal and perform identification and classification.

Images