An asynchronous hybrid brain-computer interface system based on SSVEP and EOG

By combining SSVEP and EOG signals in the asynchronous SSVEP brain-computer interface, the fusion strategy of CCA and Bayesian probability is used to identify eye saccade activities, which solves the problem of error recognition when users switch gaze positions, and improves the system's recognition speed and accuracy , promotes the practical application of brain-computer interfaces.

CN114911349BActive Publication Date: 2024-11-08SHANGHAI JIAOTONG UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202210587868.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-26
Publication Date
2024-11-08
Estimated Expiration
2042-05-26

AI Technical Summary

Technical Problem

The existing asynchronous brain-computer interface based on SSVEP will have a long-term error recognition state when the user switches the gaze position, which limits the development of the system's practical application.

Method used

Using an asynchronous hybrid brain-computer interface system based on SSVEP and EOG, combined with a fusion optimization strategy of typical correlation analysis (CCA) and Bayesian probability, the user can be quickly and accurately judged by identifying the dynamic threshold of the user's saccade activity and the SSVEP recognition results. gaze position.

Benefits of technology

When the user switches the gaze position, the time for the wrong recognition state is significantly reduced, the system's recognition speed and accuracy are improved, and the practicality of the brain-computer interface is promoted.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114911349B_ABST
    Figure CN114911349B_ABST
Patent Text Reader

Abstract

The invention discloses an asynchronous hybrid brain-computer interface system based on SSVEP and EOG, which relates to the field of human-computer interaction technology, including a stimulation module, an acquisition module, an SSVEP recognition module, an eye saccade activity recognition module, and a fusion module, wherein the stimulation module is used to display SSVEP stimulation; the acquisition module is used to obtain the electroencephalogram and electrooculogram induced by the stimulation module stroboscopic stimulation; the SSVEP recognition module uses a decoder of typical correlation analysis for recognition, and obtains the current SSVEP recognition gaze position of the user; the eye saccade activity recognition module recognizes eye saccade activity by a dynamic threshold method based on a primary differential signal of EOG; and the fusion module uses a fusion optimization strategy based on Bayesian probability to judge the current gaze position of the user. When eye saccade occurs, the eye saccade activity recognized by the electrooculogram optimizes the probability of the gaze position, and the decoding results of SSVEP and eye saccade decoding are fused to obtain a more accurate probability distribution, thereby more accurately estimating the current gaze position of the user.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to an asynchronous hybrid brain-computer interface system based on SSVEP and EOG. Background Art

[0002] In recent years, brain-computer interface (BCI) has attracted much attention due to its ability to bypass traditional peripheral nerve pathways, directly obtain the user's intention through the brain's electrophysiological signals, and then communicate with the outside world. At present, the widely used brain-computer interface signal modes include motor imagery, P300, steady-state evoked visual potential (SSVEP), etc. Among them, the brain-computer interface based on SSVEP is favored by researchers because of its high recognition rate, short training time, and large number of instructions. In the brain-computer interface based on SSVEP, the user looks at the visual stimuli that flicker at different frequencies and phases within the field of vision, and the decoder determines the user's current gaze position based on the specific SSVEP response induced by the user.

[0003] At present, synchronous brain-computer interfaces based on SSVEP have achieved a fairly high accuracy and information transmission rate, but users of the system must follow the time frame set by the system and output commands at a specific time, which greatly limits the practical application development of brain-computer interfaces based on SSVEP. On the other hand, asynchronous brain-computer interfaces based on SSVEP enable users to output commands at any time by looking at visual stimuli, but there will be a long period of misrecognition when switching gaze positions (gaze shifting). The reason is that when the user switches his gaze to a new target (new visual stimulus), the SSVEP induced by the new target begins to accumulate in the detection window; however, this SSVEP needs to accumulate to a certain proportion in the detection window before it can be recognized by the system. Before that, the system recognition result will be in an erroneous state.

[0004] To address this problem, researchers used the "dwell time strategy" method to screen the results of decoder recognition and output the results only when N identical results appear consecutively, thereby achieving control over the external system. This method can improve the stability of the system, but it will extend the system's reaction time and further prolong the duration of the erroneous recognition state. Therefore, solving the problem of erroneous recognition state caused by switching gaze position is an important part of promoting the practical application of asynchronous SSVEP brain-computer interface.

[0005] In addition to the BCI system using a single signal mode, hybrid BCIs have also been gradually developed in recent years. Hybrid BCIs integrate two or more physiological signals to improve the recognition ability and stability of the system. Among the many hybrid BCIs, hybrid BCIs based on SSVEP and electrooculography (EOG) have attracted widespread attention from researchers. In this type of BCI, in addition to looking at visual stimuli, users also need to perform eye activities synchronously or sequentially, such as blinking of both eyes or blinking of one eye. Therefore, most of the existing hybrid BCIs based on SSVEP and EOG increase the burden on users while improving the effectiveness of the system. However, in asynchronous BCIs based on SSVEP, when users switch their gaze positions, they will naturally be accompanied by saccadic activities, that is, eye movement activities when quickly switching from one gaze position to another. If the asynchronous SSVEP BCI can be combined with the characteristics of saccadic activities, it is expected to solve the above-mentioned problem of "incorrect recognition state caused by switching gaze positions".

[0006] Therefore, technicians in this field are committed to developing an asynchronous hybrid brain-computer interface system framework based on SSVEP and EOG, so as to improve the system's recognition speed and accuracy when the user switches gaze state without increasing the user's usage burden, thereby promoting the practicality of the system. Summary of the invention

[0007] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is that in the existing asynchronous brain-computer interface based on SSVEP, when the user switches the gaze position, it will inevitably cause a long period of erroneous recognition state.

[0008] To achieve the above-mentioned purpose, the present invention provides an asynchronous hybrid brain-computer interface system based on SSVEP and EOG, comprising a stimulation module, an acquisition module, an SSVEP recognition module, an eye saccade activity recognition module, and a fusion module, wherein the stimulation module is used to display SSVEP stimulation; the acquisition module is used to obtain the electroencephalogram (EEG) and electrooculogram (EOG) induced by the stimulation stroboscopic stimulation of the stimulation module; the SSVEP recognition module uses a canonical correlation analysis (CCA) decoder to recognize the EEG and obtain the user's current SSVEP recognition gaze position; the eye saccade activity recognition module recognizes eye saccade activities through a dynamic threshold method based on an EOG primary differential signal; the fusion module uses a fusion optimization strategy based on Bayesian probability to determine the user's current gaze position according to the recognition results of the SSVEP recognition module and the eye saccade activity recognition module.

[0009] Furthermore, the SSVEP recognition module calculates the correlation coefficient between the EEG data and the template signal corresponding to the frequency of each visual stimulus through CCA, wherein the frequency of the template signal with the largest correlation coefficient corresponds to the visual stimulus that the current user is looking at, thereby determining the user's current SSVEP recognition gaze position.

[0010] Furthermore, the decoder of the CCA is specifically as follows: for two high-dimensional variables X, Y and their linear combination x=X T W X ,y=Y T W Y The optimal weight vector W can be obtained by the following optimization problem X , W Y Make the correlation coefficient between x and y reach the maximum value:

[0011]

[0012] Where ρ is the correlation coefficient;

[0013] For SSVEP identification, we assume that X∈R N×M is a multi-channel EEG data with N channels and M data samples, Y∈R Q×M is the reference template signal:

[0014]

[0015] Where f represents the frequency of visual stimulation, N h is the harmonic order, Q = 2N h .

[0016] Furthermore, the SSVEP recognition module adopts an asynchronous sliding window method.

[0017] Furthermore, the dynamic threshold method of the saccade activity recognition module is: to determine whether the primary differential value of the current EOG signal exceeds the dynamic threshold; if the primary differential value of the EOG signal remains within the dynamic threshold range, it indicates that the user is stably gazing at the same position; if the primary differential value of the EOG signal exceeds the dynamic threshold range, it indicates that saccade activity has occurred, and then it is necessary to determine whether the time exceeding the dynamic threshold exceeds the specified time; if it exceeds the specified time, it is determined to be a valid saccade activity, otherwise the dynamic threshold is recalculated.

[0018] Furthermore, if valid saccade activity is identified, the current dynamic threshold range will be maintained at 200ms to ensure that a rapid and accurate response can be made to subsequent saccades.

[0019] Furthermore, the dynamic threshold is obtained by calculating the standard deviation of the differential signal from 200 data sample points before the current moment and then multiplying it by a multiple of N.

[0020] Furthermore, the eye-saccade activity recognition module adopts an asynchronous sliding window method.

[0021] Furthermore, the fusion optimization strategy is:

[0022] Assume that the SSVEP recognition gaze position at the last moment is S pre , the correlation coefficient ρ of each visual stimulus Si frequency calculated by CCA at the current moment i Mapped to the probability P(Si) that the user is looking at each visual stimulus:

[0023]

[0024] If no saccadic activity is recognized at the current moment, the visual stimulus corresponding to the maximum value of P(Si) is the user's current gaze position, which is consistent with the gaze position recognized by SSVEP; if saccadic activity is recognized at the current moment, the probability distribution of the user's current gaze visual stimulus is estimated based on the Bayesian posterior probability:

[0025]

[0026] In the formula, P(S i |E) is the user from S pre Switch to S i The probability distribution of saccadic activity occurring at , which is estimated by the recognition performance of the saccadic activity decoder;

[0027] Finally, for cases where saccade activity is identified:

[0028] k=argmax{P(S i |E),i=1,2,...,n}

[0029] S k That is, the visual stimulus that the user is currently looking at.

[0030] Furthermore, the recognition performance of the eye-saccade activity decoder includes a correct rate and a false trigger rate.

[0031] Compared with the prior art, the present invention has the following obvious outstanding substantive features and significant advantages:

[0032] The present invention can more quickly and accurately judge the current gaze state of the user when the user switches the gaze position, thereby reducing the time of the wrong recognition state occurring in the traditional method.

[0033] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the system structure of a preferred embodiment of the present invention;

[0035] Figure 2 It is a schematic diagram of an EEG acquisition channel of a preferred embodiment of the present invention;

[0036] Figure 3 It is a schematic diagram of an electrooculogram acquisition channel of a preferred embodiment of the present invention;

[0037] Figure 4 is a display schematic diagram of a stimulation module of a preferred embodiment of the present invention;

[0038] Figure 5 is a flow chart of a dynamic threshold method of a preferred embodiment of the present invention;

[0039] Figure 6 It is a flow chart of the fusion optimization strategy of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0041] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is appropriately exaggerated in some places in the drawings.

[0042] Figure 1 It is a schematic diagram of the system structure of a preferred embodiment of the present invention.

[0043] A 24.5-inch LCD monitor with a resolution of 1920x1080 and a screen refresh rate of 240Hz is placed on the desktop. The user sits in front of the monitor with the eyes about 70cm away from the screen, and the center of the eyes is aligned with the center of the screen in the direction perpendicular to the screen.

[0044] The user's electroencephalogram (EEG) and electrooculogram (EOG) are acquired through the EEG cap and muscle electrical signal electrodes developed by Biosemi, respectively. The signals are amplified by the Biosemi Active Two amplifier with a sampling rate of 1024Hz. The electrode channels CMS and DRL of the EEG cap are used as reference electrodes and ground electrodes. Conductive paste is injected into each electrode channel through a syringe to keep the voltage bias of the electrode channel within plus or minus 40mV. In addition, a 50Hz notch filter is used to remove power frequency noise in the EEG and EOG. Figure 2 The PZ, POZ, PO3, PO4, PO7, PO8, OZ, O1, and O2 channels shown will be used to identify SSVEP; Figure 3 The A, B, C, and D channels shown will be used to identify saccadic activity.

[0045] like Figure 4 As shown, in order to induce SSVEP by enabling users to look at different directions of the screen, square visual stimuli flickering at 7.5 Hz, 9.0 Hz, 10.5 Hz, and 12.0 Hz were displayed on the top, right, bottom, and left of the display, respectively, using a sinusoidal sampling stimulation method.

[0046] 1. Identification of visual steady-state evoked potentials

[0047] For SSVEP, a decoder based on canonical correlation analysis (CCA) with high accuracy, low computational complexity and ease of use is used for recognition. For two high-dimensional variables X, Y and their linear combination x = X T W X ,y=Y T W Y , the optimal weight vector W can be obtained through the following optimization problem X , W Y Make the correlation coefficient between x and y reach the maximum value:

[0048]

[0049] Where ρ is the correlation coefficient. For SSVEP identification, it can be considered that X∈R N×M is a multi-channel EEG data with N channels and M data samples, Y∈R Q×M is the reference template signal:

[0050]

[0051] Where f represents the frequency of visual stimulation, N h is the harmonic order, Q = 2N hThrough CCA, the correlation coefficient between the EEG data and the template signal corresponding to the frequency of each visual stimulus can be calculated, where the frequency of the template signal with the largest correlation coefficient corresponds to the visual stimulus that the current user is looking at, thereby determining the user's current gaze direction. For the recognition of SSVEP, an asynchronous sliding window method is adopted. In this embodiment, the step size of the sliding window is set to 0.03125s and the window length is set to 2s.

[0052] 2. Recognition of saccadic activity

[0053] like Figure 3 As shown in the figure, the recognition process of saccadic activity. When the user's gaze position switches from one place to another, saccadic activity will naturally occur. Saccadic activity can be identified by the dynamic threshold method based on the primary differential signal of EOG. The recognition mode still adopts the asynchronous sliding window method, and the step size is set to 0.03125s. Every 0.03125s, the standard deviation of the differential signal is calculated by taking the 200 data sample points before the current moment, and the dynamic threshold NSigma is obtained after multiplying it by a certain multiple (N=3). When the user is stably gazing at the same position, the primary differential value of the signal remains within the dynamic threshold range [-NSigma, +NSigma], and when saccadic activity occurs, it will exceed the threshold range. If the primary differential value of the signal not only exceeds the dynamic threshold range, but also exceeds the time for a certain period of time (30ms), it is judged as a valid saccadic activity, otherwise the dynamic threshold is recalculated and the above process is repeated. If a valid saccadic activity is identified, the current threshold range will be maintained at 200ms to ensure that a rapid and accurate response can be made to subsequent saccadic activity.

[0054] 3. Fusion optimization strategy based on Bayesian probability

[0055] like Figure 6 As shown in the figure, the fusion optimization strategy flow based on Bayesian probability is shown. When the user switches the gaze position, there will be a non-steady-state dynamic response process in the EEG signal, and the use of traditional SSVEP recognition algorithms (such as CCA) will produce an erroneous recognition state. Therefore, by fusing the eye movement and using the Bayesian probability model, the erroneous recognition state time can be greatly reduced. Assume that the gaze position identified by CCA at the last moment is S pre , the correlation coefficient ρ of each visual stimulus Si frequency calculated by CCA at the current moment i It can be mapped to the probability P(Si) that the user is looking at each visual stimulus:

[0056]

[0057] If no saccadic activity is recognized at the current moment, the visual stimulus corresponding to the maximum value of P(Si) is the position where the user is currently looking, which is consistent with the recognition of CCA; if saccadic activity is recognized at the current moment, the probability distribution of the visual stimulus that the user is currently looking at can be better estimated based on the Bayesian posterior probability:

[0058]

[0059] Among them, P(S i |E) is the user from S pre Switch to S i The probability distribution of eye saccade activity when , this probability distribution can be estimated by the recognition performance (correct rate, false trigger rate) of the eye saccade decoder. Finally, for the case where eye saccade activity is recognized:

[0060] k=argmax{P(S i |E),i=1,2,...,n}

[0061] S k That is, the visual stimulus that the user is currently looking at.

[0062] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. An asynchronous hybrid brain-computer interface system based on SSVEP and EOG, characterized in that: It includes a stimulation module, an acquisition module, an SSVEP recognition module, an eye saccade recognition module, and a fusion module, wherein the stimulation module is used to display SSVEP stimulation; the acquisition module is used to obtain the electroencephalogram (EEG) and electrooculogram (EOG) induced by the stimulation of the stimulation module; the SSVEP recognition module uses a canonical correlation analysis (CCA) decoder to recognize the EEG and obtain the user's current SSVEP recognition gaze position; the eye saccade recognition module recognizes eye saccades by a dynamic threshold method based on a primary differential signal of EOG; the fusion module uses a fusion optimization strategy based on Bayesian probability to determine the user's current gaze position according to the recognition results of the SSVEP recognition module and the eye saccade recognition module; The fusion optimization strategy is: Assume that the SSVEP recognition gaze position at the last moment is S pre , the correlation coefficient ρ of each visual stimulus Si frequency calculated by CCA at the current moment i Mapped to the probability P(Si) that the user is looking at each visual stimulus: If no saccadic activity is detected at the current moment, then P(S i )The visual stimulus corresponding to the maximum value is the user's current gaze position, which is consistent with the SSVEP recognition gaze position; if the eye movement is recognized at the current moment, the probability distribution of the user's current gaze visual stimulus is estimated based on the Bayesian posterior probability: In the formula, P(S i |E) is the user from S pre Switch to S i The probability distribution of saccadic activity occurring at , which is estimated by the recognition performance of the saccadic activity decoder; Finally, for cases where saccade activity is identified: k=argmax{P(S i |E),i=1,2,...,n} S k That is, the visual stimulus that the user is currently looking at.

2. The asynchronous hybrid brain-computer interface system based on SSVEP and EOG as claimed in claim 1, characterized in that: The SSVEP recognition module calculates the correlation coefficient between the EEG data and the template signal corresponding to the frequency of each visual stimulus through CCA, wherein the frequency of the template signal with the largest correlation coefficient corresponds to the visual stimulus that the current user is looking at, thereby determining the user's current SSVEP recognition gaze position.

3. The asynchronous hybrid brain-computer interface system based on SSVEP and EOG as claimed in claim 2, characterized in that: The CCA decoder is specifically as follows: for two high-dimensional variables X, Y and their linear combination x=X T W X ,y=Y T W Y The optimal weight vector W can be obtained by the following optimization problem X , W Y Make the correlation coefficient between x and y reach the maximum value: Where ρ is the correlation coefficient; For SSVEP identification, we assume that X∈R N×M is a multi-channel EEG data with N channels and M data samples, Y∈R Q×M is the reference template signal: Where f represents the frequency of visual stimulation, N h is the harmonic order, Q = 2N h .

4. The asynchronous hybrid brain-computer interface system based on SSVEP and EOG as claimed in claim 3, characterized in that: The SSVEP recognition module adopts an asynchronous sliding window method.

5. The asynchronous hybrid brain-computer interface system based on SSVEP and EOG as claimed in claim 4, characterized in that: The dynamic threshold method of the eye saccade activity recognition module is as follows: determine whether a primary differential value of the current EOG signal exceeds a dynamic threshold; if the primary differential value of the EOG signal remains within the dynamic threshold range, it indicates that the user is stably gazing at the same position; if the primary differential value of the EOG signal exceeds the dynamic threshold range, it indicates that eye saccade activity has occurred, and then it is necessary to determine whether the time exceeding the dynamic threshold exceeds a specified time; if it exceeds the specified time, it is determined to be a valid eye saccade activity; otherwise, the dynamic threshold is recalculated.

6. The asynchronous hybrid brain-computer interface system based on SSVEP and EOG as claimed in claim 5, characterized in that: If valid saccade activity is identified, the current dynamic threshold range will be maintained at 200ms to ensure a rapid and accurate response to subsequent saccades.

7. The asynchronous hybrid brain-computer interface system based on SSVEP and EOG as claimed in claim 6, characterized in that: The dynamic threshold is obtained by calculating the standard deviation of the differential signal from 200 data sample points before the current moment and then multiplying it by a multiple of N.

8. The asynchronous hybrid brain-computer interface system based on SSVEP and EOG as claimed in claim 7, characterized in that: The eye-saccade activity recognition module adopts an asynchronous sliding window method.

9. The asynchronous hybrid brain-computer interface system based on SSVEP and EOG as claimed in claim 1, characterized in that: The recognition performance of the eye-saccade activity decoder includes the accuracy rate and the false trigger rate.

Citation Information

Patent Citations

  • Man-machine interaction method supported by multi-modal non-implanted brain-computer interface technology

    CN104360730A

  • Brain emotion identification method based on multi-classifier fusion model constructed via hierarchical mechanism

    CN106886792A

  • Fatigue detection and regulation method based on electroencephalogram-eye movement bimodal signal

    CN109009173A