Hybrid brain-computer interface method combining visual and auditory weak and implicit stimulation

Abstract

The present invention relates to a hybrid brain-computer interface method that combines visual and auditory weak stimulation. Its technical characteristics are: determining the video stimulation interface, the number of visual stimulation targets and the visual stimulation attributes, and realizing the visual weak stimulation function by means of high-frequency flickering of the peripheral visual field; realizing the auditory weak stimulation function by applying binaural frequency-divided low-sound-pressure-level auditory stimulation signals to the left and right ears; performing visual weak stimulation and auditory weak stimulation at the same time, and selectively paying attention to the auditory stimulation of the left or right ear to complete the task of the hybrid paradigm; while performing visual weak stimulation and auditory weak stimulation, extracting the steady-state visual evoked potential characteristics and auditory steady-state response characteristics of the user's electroencephalogram signal and performing identification and classification to generate coding instructions. The present invention combines the visual weak stimulation method with the auditory weak stimulation method, and uses two electroencephalogram signals for encoding at the same time, thereby improving the naturalness and friendliness of human-computer interaction, and can encode more targets using fewer frequencies.

Description

Technical Field

[0001] The present invention belongs to the field of neural engineering technology and relates to a brain-computer interface stimulation method, in particular to a hybrid brain-computer interface method combining visual and auditory weak implicit stimulation. Background Art

[0002] Brain-computer interface (BCI) technology establishes a novel output channel between the brain and external devices that is independent of peripheral nerves and muscles, enabling direct interaction between the brain and the outside world. BCIs come in two types: invasive and non-invasive. Among non-invasive BCI signal acquisition methods, scalp EEG is dominant due to its low cost and portable equipment.

[0003] The SSVEP (steady-state visual evoked potential) BCI paradigm utilizes the frequency domain consistency between visual cortical responses and flicker at a specific frequency. It has the advantages of requiring less system calibration time, less training, and a high information transmission rate, making it one of the three mainstream BCI paradigms. Currently, mainstream visual stimulation is presented primarily in the central visual field, and SSVEP is induced using strong flicker at medium and low frequencies. The following patent documents: A hybrid BCI method combining visual and auditory perception (patent application number CN202110837782.4), A BCI system and performance testing method for a BCI system (patent application number CN202311235750.2), and A BCI auditory spatial attention detection system driven by an auditory system (patent application number CN202311530029.6) all use strong stimulation. This strong stimulation encoding method can lead to unnatural human-computer interaction and low user comfort. When users focus on a strongly flickering target for a long time, they are prone to visual fatigue and even risk photosensitive epilepsy. The user's fatigue can also affect the performance of the SSVEP brain-computer interface, resulting in a poor human-computer interaction experience. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a hybrid brain-computer interface method that combines visual and auditory weak implicit stimulation, which can significantly improve the comfort of using the brain-computer interface.

[0005] The present invention solves the existing technical problems by adopting the following technical solutions:

[0006] A hybrid brain-computer interface method combining visual and auditory weak and implicit stimulation includes the following steps:

[0007] Step 1: Determine the video stimulation interface, the number of visual stimulation targets, and the visual stimulation attributes, and implement the visual dim stimulation function by means of high-frequency flickering of the peripheral visual field;

[0008] Step 2: applying binaural frequency-divided low-sound-pressure-level auditory stimulation signals to the left and right ears to realize the auditory dim stimulation function;

[0009] Step 3: Perform visual and auditory dim stimulation simultaneously. The user fixates on a flashing target and selectively pays attention to the auditory stimulation in the left or right ear to complete the mixed paradigm task.

[0010] Step 4: While performing visual dim stimulation and auditory dim stimulation, extract the steady-state visual evoked potential characteristics and auditory steady-state response characteristics of the user's EEG signal and perform identification and classification to generate coding instructions.

[0011] Furthermore, the visual stimulation interface is a left area and a right area presented on the display screen, the left area and the right area correspond to left ear stimulation and right ear stimulation respectively, and the visual stimulation attributes of the targets in the two areas are the same.

[0012] Furthermore, the number of the stimulation targets n=2k, each visual stimulation target is presented in the form of peripheral visual field and high-frequency flicker; the video stimulation attributes include visual stimulation frequency F i and phase The inner ring viewing angle α and the outer ring viewing angle β of the peripheral field of view, the inner ring radius r1 and the outer ring radius r2 of the target ring and the position distribution, where i=1,…,k, k≥2.

[0013] Furthermore, the visual stimulation frequency F i ≥30Hz, the inner ring viewing angle α>2°.

[0014] Furthermore, the auditory stimulation signal is a sinusoidal amplitude modulated low sound pressure level signal in the form of binaural frequency division. The stimulation frequency of the auditory stimulation signal includes the modulation frequency of the left and right ear signals and the modulation frequency of the signal carrier frequency. The sound pressure level SPL of the auditory stimulation signal is ≤45dB.

[0015] Furthermore, in step 3, when the user is looking at the flashing of a certain target, he or she pays attention to the flashing of a certain target according to the prompt. If the target is located on the left side of the stimulation interface, the user needs to pay attention to the auditory stimulation of the left ear; if the target is located on the right side of the stimulation interface, the user needs to pay attention to the auditory stimulation of the right ear.

[0016] Furthermore, the step 4 uses a filter bank canonical correlation analysis algorithm and a filter bank co-spatial pattern algorithm to extract steady-state visual evoked potential features and auditory steady-state response features of the user's EEG signal and perform identification and classification.

[0017] The advantages and positive effects of the present invention are:

[0018] 1. The present invention combines visual weak implicit stimulation mode with auditory weak implicit stimulation mode, and uses two EEG signals for encoding at the same time, wherein one frequency of auditory stimulation and one visual stimulation correspond to a unique target. Through the high-frequency flickering visual stimulation of the peripheral visual field and the auditory stimulation of the low sound pressure level sinusoidal amplitude modulated signal, it fully considers the sensory stimulation properties, significantly improves the comfort of using the brain-computer interface, and enhances the naturalness and friendliness of human-computer interaction. At the same time, it can use fewer frequencies to encode more targets, and has the potential to expand the number of brain-computer interface instructions.

[0019] 2. The present invention uses high-frequency, imperceptible flickering to induce SSVEP, and can reduce the sense of flicker and improve comfort. It vacates the central visual field and uses the peripheral visual field for stimulation encoding, which can induce SSVEP while vacating the visual pathway, reducing the user's visual fatigue and being closer to natural application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a coding schematic diagram of the present invention;

[0021] Figure 2 Schematic diagram of the stimulation interface of the present invention;

[0022] Figure 3 Flowchart of each trial of the online experiment of the present invention. DETAILED DESCRIPTION

[0023] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0024] The design concept of the present invention is:

[0025] Because the ASSR (auditory steady-state response) is a steady-state brain response induced by periodic sound signals and exhibits excellent frequency characteristics, the corresponding frequency components can be extracted from the scalp EEG when a user is exposed to auditory stimulation of a specific frequency. The ASSR spectrum exhibits distinct peaks at the stimulus frequency and its harmonics. When stimulation is applied simultaneously to both ears, the brain region with the strongest response is the frontal-central region. Spatial attention mechanisms play a crucial role in the context of multiple competing sound sources and are also important in detecting and analyzing spatial sound motion. Based on this spatial attention mechanism, the ASSR exhibits certain spatial characteristics, allowing spatial filtering techniques to effectively extract the features of the ASSR signal and improve its discriminability. Furthermore, considering that visual and auditory perception account for the highest proportion of information acquisition in the process of acquiring external information, making them the primary source of information for the brain, and that a combined audiovisual interaction model is superior to a single sensory modality, mobilizing multiple senses is more conducive to brain-computer interaction. Therefore, the present invention uses two high-frequency flicker frequencies to induce SSVEPs in the peripheral visual field and two low-sound-pressure-level sinusoidal amplitude-modulated signals to induce ASSRs. These two paradigms encode the target in parallel. The user selectively attends to the sound stimulus on one side while simultaneously fixating on the target to complete the hybrid paradigm task. The corresponding algorithms are used to extract and classify the ASSR and SSVEP signals, respectively, and finally output the encoded instructions in real time.

[0026] Based on the above design concept, the present invention proposes a hybrid brain-computer interface method combining visual and auditory weak and implicit stimulation, including the following steps:

[0027] Step 1: Determine the video stimulation interface, the number of stimulation targets, and the visual stimulation attributes, and realize the visual dim stimulation function by high-frequency flickering of the peripheral visual field.

[0028] According to the actual application scenario and requirements, the present invention divides the stimulation interface into two areas, the left area and the right area, which correspond to the left ear stimulation and the right ear stimulation respectively. The visual stimulation attributes of the targets in the two areas are the same. The number of stimulation targets n = 2k (k ≥ 2, k∈N * ).

[0029] The present invention presents the visual stimulation target through the peripheral visual field and high-frequency flickering, that is, the visual dim stimulation function is achieved through the high-frequency flickering of the peripheral visual field. When performing dim stimulation, the following parameters need to be determined: the distance between the user's head (eyes) and the display using a headrest to fix the user, the video stimulation attributes including the visual stimulation frequency F i (i=1,…,k,F i ≥30Hz) and phase The inner ring viewing angle α (α>2°) and the outer ring viewing angle β of the peripheral field of view, the inner ring radius r1 and the outer ring radius r2 of the target ring, and the position distribution.

[0030] like Figure 1 and Figure 2 As shown, the following takes the number of stimulation targets n=4 (k=2) as an example to explain this step in detail:

[0031] The stimulation interface was presented using a monitor with a screen resolution of 1920 × 1080 pixels and a refresh rate of R = 120 Hz. The monitor was 54.1 cm tall and 30.3 cm wide. A headrest was used to position the user's eyes facing the center of the monitor, 50 cm away. The inner ring viewing angle α was set to 2.1°, and the outer ring viewing angle β was set to 4.1°. The inner ring radius r1 was calculated and rounded to 65 pixels, and the outer ring radius r2 was calculated to 127 pixels. With the upper left corner of the stimulation interface as the coordinate origin, the center coordinates of the four targets were (640, 315), (640, 465), (1280, 315), and (1280, 465), respectively. The background RGB value of the stimulation interface was [102102102]. Targets in the left and right regions were assigned two different flicker frequencies: F1 = 34 Hz ​​and F2 = 35 Hz, respectively, with a flicker phase of 0. The sine sampling encoding method is used to adjust the brightness of each frame target so that it changes according to the law of the sine wave to achieve a flicker at a specified frequency. For each visual stimulus target, its brightness can be expressed as:

[0032]

[0033] Where s is the brightness of the visual stimulus target in each frame, f is the frame index of the stimulus sequence, and R is the screen refresh rate.

[0034] Step 2: Apply binaural frequency-divided low-sound-pressure-level auditory stimulation signals to the left and right ears to achieve a weak auditory stimulation function.

[0035] The auditory stimulation signal is a sinusoidal amplitude modulated low sound pressure level signal in the form of binaural frequency division. The stimulation frequency of the auditory stimulation signal includes the modulation frequency f of the left and right ear signals. mi (i=1,2), signal carrier frequency modulation frequency f ci (i=1,2) and auditory stimulation sound pressure level SPL≤45dB.

[0036] The auditory stimulation signal of this embodiment can be expressed as:

[0037] c(t)=Asin(πf mi t)sin(2πf ci t),i=1,2 (2)

[0038] Where t is time and A is the amplitude modulation coefficient. In this embodiment, binaural frequency division is used to perform auditory stimulation simultaneously, and the modulation frequency f of the left ear stimulation signal is m1=34Hz, carrier frequency f c1 =2500Hz, the modulation frequency of the right ear stimulation signal f m2 =35Hz, carrier frequency f c2 =1000Hz, the signal sampling rate is 44100Hz, and the sound card parameters are adjusted to make the auditory signal sound pressure level 40dB.

[0039] Step 3: Perform visual dim stimulation and auditory dim stimulation simultaneously according to the methods of steps 1 and 2. The user fixates on the flashing of a target and selectively pays attention to the auditory stimulation in the left or right ear to complete the mixed paradigm task.

[0040] In this step, the ASSR and SSVEP tasks are performed simultaneously: The user determines the target area and location based on a randomly displayed cue during the prompt phase. In the subsequent task phase, the user completes the ASSR task by selectively attending to either the left or right ear stimulation signal. Specifically, the user attends to the flickering target according to the cue. If the target is on the left side of the stimulation interface, the user attends to the auditory stimulus in the left ear; if the target is on the right side of the stimulation interface, the user attends to the auditory stimulus in the right ear. Simultaneously, the user completes the SSVEP task by attending to the flickering visual stimulus target. During the task phase, the user fixates on a cross in the center of the target, freeing up the visual pathways in the central visual field and eliminating the need to directly observe the flickering target.

[0041] like Figure 3 As shown in Figure 2, the specific process of the online experiment for this step is as follows: users are required to complete 10 blocks of online experimental tasks. Each block of the online experiment contains 20 trials, each of which consists of a blank period, a cue period, and a task period. At the beginning of each trial, a 0.5-second blank period is set to remind the user that the task is about to begin. During this blank period, the stimulus interface displays all targets, but no visual flickering or auditory stimuli are presented. The target's RGB value is [178, 178, 178]. The user only needs to adjust their state and prepare for the next task. The cue period lasts for 0.5 seconds. The stimulus interface randomly displays a 300-pixel red box around a target, prompting the user to pay attention to the target during the task phase. The task phase lasts for 5 seconds. The stimulus interface flashes all visual stimuli, while modulated auditory stimuli are played simultaneously in both ears. The subject completes the SSVEP task by focusing on the target indicated in the cue period. At the same time, if the target is located on the left side of the stimulation interface, the user needs to pay attention to the auditory stimulation of the left ear; if the target is located on the right side of the stimulation interface, the user needs to pay attention to the auditory stimulation of the right ear.

[0042] Step 4: While performing visual dim stimulation and auditory dim stimulation, extract the steady-state visual evoked potential characteristics and auditory steady-state response characteristics of the user's EEG signal and perform identification and classification to generate coding instructions.

[0043] In this step, existing technologies are used to extract features of the user's EEG signals and to classify and identify them. For example, the filterbank canonical correlation analysis (FBCCA) algorithm and the filterbank common spatial pattern (FCP) algorithm can be used to extract SSVEP and ASSR features and classify them, respectively, to achieve real-time feedback.

[0044] To decode this encoding method, corresponding algorithms need to be used to extract features and classify ASSR signals and SSVEP signals respectively, and finally output the classification results to form a brain-computer interface system with real-time feedback.

[0045] It should be emphasized that the embodiments described in the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solutions of the present invention also fall within the scope of protection of the present invention.

Claims

1. A hybrid brain-computer interface method combining visual and auditory weak and implicit stimulation, characterized by: The following steps are involved: Step 1: Determine the video stimulation interface, the number of visual stimulation targets, and the visual stimulation attributes, and implement the visual dim stimulation function by means of high-frequency flickering of the peripheral visual field; Step 2: applying binaural frequency-divided low-sound-pressure-level auditory stimulation signals to the left and right ears to realize the auditory dim stimulation function; Step 3: Perform visual and auditory dim stimulation simultaneously. The user fixates on a flashing target and selectively pays attention to the auditory stimulation in the left or right ear to complete the mixed paradigm task. Step 4: while performing visual dim stimulation and auditory dim stimulation, extract the steady-state visual evoked potential characteristics and auditory steady-state response characteristics of the user's EEG signal, perform identification and classification, and generate coding instructions; The visual stimulation interface is a left area and a right area presented on the display screen, the left area and the right area correspond to left ear stimulation and right ear stimulation respectively, and the visual stimulation attributes of the targets in the two areas are the same; In step 3, when the user is looking at the flashing of a certain target, he / she pays attention to the flashing of a certain target according to the prompt. If the target is located on the left side of the stimulation interface, the user needs to pay attention to the auditory stimulation of the left ear; if the target is located on the right side of the stimulation interface, the user needs to pay attention to the auditory stimulation of the right ear.

2. The hybrid brain-computer interface method for combined visual and auditory weak and implicit stimulation according to claim 1 is characterized by: The number of the stimulation targets n=2k, each visual stimulation target is presented in the form of peripheral visual field and high-frequency flicker; the video stimulation attributes include visual stimulation frequency F i and phase The inner ring viewing angle α and the outer ring viewing angle β of the peripheral field of view, the inner ring radius r1 and the outer ring radius r2 of the target ring and the position distribution, where i=1,…,k, k≥2.

3. The hybrid brain-computer interface method for combined visual and auditory weak and implicit stimulation according to claim 2 is characterized by: The visual stimulation frequency F i ≥30Hz, the inner ring viewing angle α>2°.

4. The hybrid brain-computer interface method for combined visual and auditory weak and implicit stimulation according to any one of claims 1 to 3, characterized in that: The auditory stimulation signal is a sinusoidal amplitude modulated low sound pressure level signal in the form of binaural frequency division. The stimulation frequency of the auditory stimulation signal includes the modulation frequency of the left and right ear signals and the modulation frequency of the signal carrier frequency. The sound pressure level SPL of the auditory stimulation signal is ≤45dB.

5. The hybrid brain-computer interface method for combined visual and auditory weak and implicit stimulation according to any one of claims 1 to 3, characterized in that: The step 4 uses the filter bank canonical correlation analysis algorithm and the filter bank common space pattern algorithm to extract the steady-state visual evoked potential features and auditory steady-state response features of the user's EEG signal and perform identification and classification.

Images