Device and method for detecting visual stimulation based on steady-state visual evoked potential

Abstract

Devices and methods for detecting visual stimulation based on steady-state visual evoked potentials are provided. The apparatus for detecting visual stimulation based on steady state visual evoked potential (SSVEP) may include: a visual stimulation signal receiver configured to receive a visual stimulation signal extracted by electroencephalogram (EEG) analysis of a user with respect to visual stimulation at a specific frequency; a visual guidance unit configured to arrange a visual guidance having a specific form on the visual stimulus; a visual stimulation signal processor configured to classify the visual stimulation based on the received visual stimulation signal and generate visual feedback; and a visual feedback reflector configured to reflect the generated visual feedback to the visual guidance. The visual guidance unit may be configured to change the shape of the visual guidance based on the reflected visual feedback.

Description

[0001] Related Applications

[0002] This application claims priority to and the benefit of Korean Patent Application No. 10-2023-0156298 filed in the Korean Intellectual Property Office on November 13, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to an apparatus and method for detecting visual stimulation based on steady-state visual evoked potential (SSVEP). More specifically, the present disclosure relates to an apparatus and method for detecting visual stimulation based on SSVEP with visual guidance providing real-time feedback. Background Art

[0004] Steady-state visual evoked potentials (SSVEPs) are electroencephalographic potentials generated when fixating on a visual stimulus that flickers at a specific frequency and can be extracted by analysis of the electroencephalogram (EEG) measured near the occipital lobe.

[0005] Since the specific frequency of the visual stimulus being gazed at can be detected from the EEG signal, the visual stimulus being gazed at by the user can be identified through EEG analysis. Therefore, it can be used to develop various brain-computer interfaces (BCI).

[0006] Although there has been some progress in SSVEP-based BCI algorithms, there is still the problem that BCI performance is greatly affected by user fatigue, concentration, and attitude.

[0007] Typically, in order to detect SSVEP signals with constant performance, a fixed SSVEP stimulation time (fixed time window) is used through previous offline analysis, and the results of the stimulation that the user fixates on after the end of the stimulation may be known.

[0008] With regard to the SSVEP stimulation, there are differences in the way each person views the stimulation, and as a result, there is a problem that it is difficult to view the stimulation in the way they view it. Summary of the invention

[0009] The present disclosure seeks to provide an apparatus and method for detecting visual stimuli based on SSVEP, which may include a visual guide configured to provide real-time feedback based on saliency probability.

[0010] The present disclosure attempts to provide an apparatus and method for detecting visual stimuli based on SSVEP, which can intuitively identify which stimulus the feedback is directed to by responding to the stimulus that the user is gazing at in real time, and has visual guidance, which improves the detection performance relative to the visual stimulus by strengthening the user's gaze concentration.

[0011] An apparatus for detecting visual stimulation based on SSVEP may include: a visual stimulation signal receiver configured to receive a visual stimulation signal extracted by electroencephalogram (EEG) analysis of a user with respect to visual stimulation of a specific frequency of gaze; a visual guide unit configured to arrange a visual guide having a specific form on the visual stimulation; a visual stimulation signal processor configured to classify the visual stimulation based on the received visual stimulation signal and generate visual feedback, and a visual feedback reflector configured to reflect the generated visual feedback to the visual guide. The visual guide unit may be configured to change the shape of the visual guide based on the reflected visual feedback.

[0012] The visual guide may have the same frequency as the specific frequency of the visual stimulus.

[0013] The visual guide unit may be configured to change the shape of the visual guide when the visual stimulus flickers according to a specific frequency.

[0014] The visual stimulation may include a checkerboard-based visual stimulation that is reversed according to a specific frequency, and the visual guide may appear on the checkerboard at a reverse time point of the checkerboard relative to the checkerboard-based visual stimulation.

[0015] The visual stimulation signal processor may include: a visual stimulation classifier configured to extract features of the visual stimulation signal and classify the visual stimulation based on the extracted features; and a visual feedback generator configured to generate visual feedback containing feedback information related to the classification of the visual stimulation.

[0016] The visual stimulus signal processor may be configured to calculate the slope of a regression line of each visual stimulus by performing an analysis of covariance (ANCOVA) relative to a plurality of visual stimuli including the visual stimulus, and the visual feedback generator may be configured to generate a significance probability relative to a difference between a maximum first slope corresponding to the visual stimulus and a second slope corresponding to another visual stimulus other than the visual stimulus, as the visual feedback, and the significance probability may be inversely proportional to the size of the difference between the slopes.

[0017] The significance probability may have a value varying in a range above 0 and below 1 relative to the visual stimulus, and the visual feedback generator may be configured to generate positive first feedback relative to the classification as the significance probability is closer to 0, and to generate negative second feedback relative to the classification as the significance probability is closer to 1.

[0018] The visual guide unit may be configured to change the shape of the visual guide to a first shape in response to the first feedback, and to change the shape of the visual guide to a second shape in response to the second feedback.

[0019] The visual feedback reflector can be configured to reflect the visual feedback including the changing significance probability to the visual guide in real time, and the visual guide unit can be configured to change the shape of the visual guide in real time in response to the real-time visual feedback to have a continuous shape that dynamically changes between a first shape and a second shape.

[0020] When the significance probability satisfies the significance level, the visual stimulus classifier may be configured to classify a stimulus corresponding to a maximum Canonical Correlation Analysis (CCA) coefficient among CCA coefficients used in the covariance analysis as a visual stimulus intended by the user.

[0021] The visual guide unit may be configured to arrange the visual guide at the center of the visual stimulation.

[0022] A visual stimulus detection method may include: receiving a visual stimulus signal extracted by EEG analysis of a user who is fixated on a visual stimulus of a specific frequency by a device for detecting visual stimulus based on SSVEP, arranging a visual guide having a specific form on the visual stimulus by the device for detecting visual stimulus based on steady-state visual evoked potential (SSVEP), classifying the visual stimulus based on the received visual stimulus signal by the device for detecting visual stimulus based on SSVEP and generating visual feedback, and reflecting the visual feedback to the visual guide in real time by the device for detecting visual stimulus based on SSVEP. Arranging the visual guide may include changing the shape of the visual guide in real time based on the visual feedback.

[0023] The visual guide may have the same frequency as the specific frequency of the visual stimulus.

[0024] Arranging the visual guide may further include changing a shape of the visual guide when the visual stimulus flickers according to a specific frequency.

[0025] Classifying the visual stimulus based on the received visual stimulus signal and generating visual feedback may include extracting features of the visual stimulus signal and classifying the visual stimulus based on the extracted features, and generating visual feedback including feedback information related to the classification of the visual stimulus.

[0026] Generating visual feedback may include: calculating the slope of a regression line of each visual stimulus by covariance analysis relative to a plurality of visual stimuli comprising the visual stimulus; and calculating a significance probability relative to a difference between a maximum first slope corresponding to the visual stimulus and a second slope corresponding to another visual stimulus other than the visual stimulus, as visual feedback, wherein the significance probability may be inversely proportional to the size of the difference between the slopes.

[0027] The significance probability may have a value varying in a range above 0 and below 1 relative to the visual stimulus, and generating visual feedback may further include: generating positive first feedback relative to the category as the significance probability is closer to 0; and generating negative second feedback relative to the category as the significance probability is closer to 1.

[0028] Arranging the visual guide may further include changing the shape of the visual guide to a first shape in response to the first feedback, and changing the shape of the visual guide to a second shape in response to the second feedback.

[0029] Generating visual feedback may further include setting all significance probabilities relative to slope differences between other visual stimuli to 1, excluding significance probabilities relative to the difference between the first slope and the second slope, and the magnitude of the second slope may be second largest, second only to the magnitude of the first slope.

[0030] Arranging the visual guide may further include arranging the visual guide at the center of the visual stimulation.

[0031] According to the apparatus and method for detecting visual stimuli based on SSVEP according to the embodiments, it includes providing visual guidance for real-time feedback related to the classification of visual stimuli, so that the user can intuitively know which stimulus the feedback is for, and the user's SSVEP visual detection performance can be improved by enhancing gaze concentration.

[0032] According to the apparatus and method for detecting visual stimulation based on SSVEP according to the embodiment, usability is high because it can be directly added to visual stimulation appropriately designed to induce SSVEP.

[0033] According to the apparatus and method for detecting visual stimulation based on SSVEP according to the embodiments, the visual guidance is synchronized to the same frequency as the SSVEP, thereby strengthening the frequency stimulation and improving the detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a diagram showing an example of a BCI system of a device for detecting visual stimulation based on SSVEP according to an embodiment.

[0035] Figure 2 is a block diagram showing an apparatus for detecting visual stimulation based on SSVEP according to an embodiment.

[0036] Figure 3 is a diagram showing a visual stimulus processing procedure of an apparatus for detecting visual stimulus based on SSVEP according to an embodiment.

[0037] Figure 4is a flow chart illustrating classification of visual stimuli and generation of visual feedback by a visual stimulus signal processor according to an embodiment.

[0038] Figure 5 and Figure 6 is a diagram illustrating a visual guide according to an embodiment.

[0039] Figure 7 is an exemplary diagram illustrating the appearance of a visual guide according to a user's gaze according to an embodiment.

[0040] Figure 8 is a flow chart illustrating a method of visual stimulus detection based on SSVEP according to an embodiment.

[0041] Fig. 9 is a diagram showing a computing device according to an embodiment. DETAILED DESCRIPTION

[0042] Hereinafter, the embodiments of the present disclosure will be described more fully with reference to the accompanying drawings so that those skilled in the art can easily implement the embodiments. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the present disclosure. In order to clarify the present disclosure, components not related to the description will be omitted, and the same elements or equivalents are represented by the same reference numerals throughout the specification.

[0043] In addition, unless explicitly described to the contrary, the word "comprise" and variations such as "comprises" or "comprising" will be understood to imply the inclusion of the stated elements but not the exclusion of any other elements. Terms including common numbers such as first and second are used to describe various constituent elements, but the constituent elements are not limited by the terms. These terms are only used to distinguish one component from other components.

[0044] In addition, the terms "unit", "part" or "section", "device" and "module" in the specification refer to a unit that processes at least one function or operation, which can be implemented by hardware, software, or a combination of hardware and software.

[0045] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0046] Figure 1 is a diagram of an example of a brain-computer interface (BCI) system to which an apparatus for detecting visual stimulation based on SSVEP according to an embodiment is applied.

[0047] See also Figure 1A BCI system to which the apparatus for detecting visual stimulation based on SSVEP according to an embodiment is applied may include an apparatus for detecting visual stimulation based on SSVEP 100 (hereinafter referred to as visual stimulation detection apparatus 100 ), a steady-state visual evoked potential (SSVEP) generator 200 , and an external device 300 .

[0048] The apparatus 100 for detecting visual stimulation based on SSVEP is the core point of the present disclosure, and can determine which visual stimulation the user has gazed at based on steady-state visual evoked potential (SSVEP).

[0049] Here, the steady-state visual evoked potential (SSVEP) is an electroencephalogram potential generated when gazing at a visual stimulus flickering at a specific frequency, and can be extracted by electroencephalogram (EEG) analysis measured near the occipital lobe. Since the specific frequency of the visual stimulus being gazed at can be detected from the EEG signal, the steady-state visual evoked potential (SSVEP) regarding the visual stimulus being gazed at by the user can be identified by EEG analysis.

[0050] Therefore, steady-state visual evoked potential (SSVEP) can be used to develop various brain-computer interfaces (BCI). Steady-state visual evoked potential (SSVEP) can be called a visual stimulation signal.

[0051] The apparatus 100 for detecting visual stimuli based on SSVEP according to an embodiment may include: a visual guide that reacts in real time to the visual stimuli that the user is gazing at, so as to provide real-time feedback to the user, thereby being able to detect the visual stimuli that the user is gazing at more quickly and accurately.

[0052] The SSVEP generator 200 may provide the user with a visual stimulus corresponding to a control command regarding the external device 300 , and may induce the user to generate an electroencephalogram (EEG) signal including an EEG corresponding to the visual stimulus.

[0053] For example, when the user looks at an arrow in the forward direction, the electroencephalogram corresponding to the forward direction is included in the user's EEG signal. Therefore, the apparatus 100 for detecting visual stimuli based on SSVEP can detect the electroencephalogram corresponding to the arrow in the forward direction according to the user's EEG signal. That is, the SSVEP generator 200 can transmit the EEG signal to the apparatus 100 for detecting visual stimuli based on SSVEP.

[0054] The external device 300 may be connected to the apparatus 100 for detecting visual stimulation based on SSVEP through a network. The external device 300 may communicate with the apparatus 100 for detecting visual stimulation based on SSVEP and may be controlled according to a command received from the apparatus 100 for detecting visual stimulation based on SSVEP.

[0055] The external device 300 may include a personal mobile body, and may include a wheelchair, an exoskeleton, and the like.

[0056] Figure 2 is a block diagram of an apparatus for detecting visual stimulation based on SSVEP according to an embodiment. Figure 3 is a diagram showing a visual stimulus processing procedure of an apparatus for detecting visual stimulus based on SSVEP according to an embodiment.

[0057] See also Figure 2 and Figure 3 The apparatus 100 for detecting visual stimulation based on SSVEP may include a visual stimulation signal receiver 110 , a visual guidance unit 120 , a visual stimulation signal processor 130 , and a visual feedback reflector 140 .

[0058] The visual stimulation signal receiver 110 may receive a visual stimulation signal (SSVEP) extracted through EEG analysis with respect to a user who gazes at a visual stimulation STI of a specific frequency.

[0059] The visual stimulation STI may be an image that flickers according to a specific frequency. The visual stimulation STI may be a checkerboard image that is repeatedly inverted according to a specific frequency.

[0060] For example, the visual stimulation signal receiver 110 may receive a visual stimulation signal extracted from an electroencephalogram signal of a user who is looking at a visual stimulation STI including an image flashing 10 times within 1 second at a frequency of 10 Hz.

[0061] The visual guide unit 120 may arrange the visual guide VG having a specific form on the visual stimulation STI. The visual guide unit 120 may arrange the visual guide VG at the center of the visual stimulation STI.

[0062] The visual guide VG can improve the user's gaze concentration. The shape, pattern and color of the form of the visual guide VG are not limited to Figure 3 , and can be set freely.

[0063] In an embodiment, the visual guide VG may have the same frequency as the specific frequency of the visual stimulus STI that the user is looking at. That is, the visual guide VG may be synchronized with the frequency of the visual stimulus STI that the user is looking at, and strengthen the frequency stimulus delivered to the user. For example, the visual guide VG may flash at the same frequency as the visual stimulus STI.

[0064] The visual stimulus signal processor 130 may classify the visual stimulus STI based on the received visual stimulus signal and may generate visual feedback. The visual stimulus signal processor 130 may classify the visual stimulus that the user is looking at from other visual stimuli based on the visual stimulus signal. Figure 3 , the visual stimulation signal processor 130 may include a visual stimulation classifier 131 and a visual feedback generator 132 .

[0065] The visual stimulation classifier 131 may extract features of the visual stimulation signal received from the visual stimulation signal receiver 110 , and may classify the visual stimulation STI based on the extracted features.

[0066] The visual stimulus classifier 131 may extract features of the visual stimulus signal. Feature extraction is the process of extracting important information from the EEG signal. The most important feature in the visual stimulus signal (SSVEP) may be the frequency of the EEG.

[0067] For example, the visual stimulus classifier 131 may extract frequency components from the electroencephalogram signal by using Fourier transform or wavelet transform, etc. The visual stimulus classifier 131 may identify which visual stimulus the user has reacted to based on the power magnitude of each frequency component.

[0068] The visual stimulus classifier 131 may classify the visual stimulus STI based on the extracted features. Classification is the process of assigning the EEG signal to a specific category or type using the extracted features.

[0069] That is, when the user focuses on a specific visual stimulus, an EEG response corresponding to the frequency of the visual stimulus may be generated, and the visual stimulus classifier 131 may determine which stimulus the EEG signal responds to by classification. In an embodiment, the classification algorithm may be based on machine learning technology. For example, the visual stimulus classifier 131 may classify the visual stimulus by using a support vector machine (SVM), K-nearest neighbor (K-NN), linear deterministic analysis (LDA), etc.

[0070] In an embodiment, the visual stimulation classifier 131 may periodically calculate correlation coefficients (or CCA coefficients) between multiple reference signals with different stimulation frequencies and a user's multi-channel EEG signal for a reference time, perform covariance analysis (ANCOVA) on the multiple correlation coefficients for each calculated stimulation (stimulation frequency) to calculate the slope of the regression line for each stimulation, and classify the stimulation intended by the user by multiple comparisons between the slopes of the regression lines for each calculated stimulation.

[0071] Here, multiple comparison refers to a process of referring to the maximum slope among the slopes calculated for the regression lines of the respective stimuli and calculating the significance probability P value for the difference with each of the remaining slopes.

[0072] For example, when the slope decreases in the order of the first visual stimulus, the second visual stimulus, and the third visual stimulus, the visual stimulus classifier 131 can be configured to calculate a first significance probability between the slope of the first visual stimulus and the slope of the second visual stimulus, and calculate a second significance probability between the slope of the first visual stimulus and the slope of the third visual stimulus.

[0073] When all the significance probabilities calculated through multiple comparisons meet the significance level (significance probability <significance level), the visual stimulus classifier 131 can determine that there is a statistically significant difference, and can detect the stimulus corresponding to the maximum correlation coefficient among multiple correlation coefficients as the stimulus intended by the user.

[0074] Here, the significance level may be set to one of, for example, 0.05, 0.01, and 0.001. Figure 4 Provide a detailed description.

[0075] The visual feedback generator 132 may generate visual feedback including feedback information related to the classification of the visual stimulus.

[0076] The visual feedback generator 132 can generate feedback relative to the visual stimulus that the user is looking at as visual feedback. When the user looks at a specific visual stimulus, the visual feedback can include statistically based information extracted relative to the visual stimulus. In addition, the visual feedback can include the user's focus relative to the specific visual stimulus.

[0077] The visual feedback generator 132 may generate the visual feedback by comparing the slopes between regression lines calculated for each visual stimulus through an analysis of covariance (ANCOVA) with respect to the plurality of visual stimuli.

[0078] The visual feedback generator 132 may generate a significance probability P value as visual feedback with respect to a difference between a maximum first slope occurring corresponding to a visual stimulus STI gazed by the user and a second slope corresponding to another visual stimulus not gazed by the user.

[0079] Unlike the visual stimulus classifier 131, the visual feedback generator 132 only needs the significance probability P value, and therefore does not need to compare the slopes of all visual stimuli. That is, the visual feedback generator 132 can generate visual feedback based on the significance probability P value relative to the slope difference between the first visual stimulus (which is the visual stimulus STI with the largest first slope) and the second visual stimulus with the second largest second slope.

[0080] That is, the visual feedback may reflect in real time the significance probability P value with respect to the difference between the first slope of the first visual stimulus and the second slope of the second visual stimulus.

[0081] The significance probability P value may have a value that varies within a range of more than 0 and less than 1. The significance probability P value may be inversely proportional to the size of the difference between the slopes. For example, as the difference between the first slope and the second slope is larger, the significance probability P value is closer to 0. As the difference between the first slope and the second slope is smaller, the significance probability P value is closer to 1.

[0082] In an embodiment, the visual feedback generator 132 may preset the significance probability P value to 1 for visual stimuli other than the first visual stimulus having the largest slope.

[0083] The visual feedback generator 132 may generate first feedback when the significance probability P value is closer to 0, and may generate second feedback when the significance probability P value is closer to 1. The first feedback may be positive feedback relative to the classification of the visual stimulus STI, and the second feedback may be negative feedback relative to the classification of the visual stimulus STI.

[0084] That is, as the significance probability P value is closer to 0, the visual feedback may appear as the first feedback relative to the visual stimulation STI, and as the significance probability P value is closer to 1, the visual feedback may appear as the second feedback relative to the visual stimulation STI.

[0085] The first feedback may indicate that the gaze concentration relative to the visual stimulus STI gazed by the user is at a high level, and the second feedback may indicate that the gaze concentration of the user relative to the visual stimulus STI is at a low level. The visual feedback generator 132 may generate visual feedback indicating the gaze concentration of the user.

[0086] The visual feedback reflector 140 may reflect the generated visual feedback to the visual guide VG. That is, the visual feedback reflector 140 may reflect the real-time change of the significance probability P value between the first visual stimulus and the second visual stimulus to the visual guide VG.

[0087] For example, the visual feedback reflector 140 may directly change the shape of the visual guide VG based on the visual feedback. Alternatively, the visual feedback reflector 140 may transmit the visual feedback to the visual guide unit 120, and it may be reflected to the visual guide VG in various methods by the visual guide unit 120.

[0088] In an embodiment, the visual feedback reflector 140 may reflect the user's gaze concentration relative to the visual stimulus STI to the visual guide VG through visual feedback.

[0089] The visual guide unit 120 may change the shape of the visual guide VG based on the visual feedback reflected from the visual feedback reflector 140. That is, the visual guide VG may have a shape that is dynamically changed based on the visual feedback.

[0090] For example, the visual guide unit 120 may reflect the real-time change of the significance probability P value between the first visual stimulus and the second visual stimulus, and may change the shape of the visual guide VG arranged on the visual stimulus STI in real time.

[0091] For example, the visual guide unit 120 may change the shape of the visual guide VG to a first shape VG1 in response to the first feedback, and may change the shape of the visual guide VG to a second shape VG2 in response to the second feedback. The first shape VG1 and the second shape VG2 are different shapes, but are not particularly limited to Figure 3 .

[0092] In response to the real-time visual feedback, the visual guide unit 120 may change the shape of the visual guide VG in real time to have a continuous shape that dynamically changes between the first shape VG1 and the second shape VG2. Figure 3 In the example, the visual guide VG may be a graphic element of a specific shape.

[0093] For example, the visual guide VG may be configured in three parts. The three parts of the visual guide VG may be separated from each other in the second shape VG2. The three parts of the visual guide VG may be connected to each other in the first shape VG1.

[0094] The three parts of the visual guide VG can move away from each other or become closer to each other, thereby reflecting real-time visual feedback. As the significance probability P value is closer to 0, the three parts are closer to each other, and as the significance probability P value is closer to 1, the three parts can become farther away from each other.

[0095] In an embodiment, the visual guide unit 120 may represent the user's gaze concentration reflected in the visual feedback through the visual guide VG. That is, the longer the shape of the visual guide VG that dynamically changes by reflecting the significance probability P value is maintained as the first shape VG1 (that is, the closer the significance probability is maintained to 0), the higher the user's gaze concentration may appear. At this time, there is no need to set the significance level. Therefore, the shape of the visual guide VG can change according to the significance probability regardless of the significance level.

[0096] In another embodiment, the visual guide unit 120 can communicate with the visual feedback reflector 140 to improve the user's gaze concentration, and can receive changes in the significance probability P value caused by the shape of the visual guide VG. The visual guide unit 120 can dynamically change the shape of the visual guide VG in a direction in which the significance probability P value is maintained close to 0.

[0097] Figure 4is a flow chart illustrating the classification of visual stimuli and the generation of visual feedback by a visual stimulus signal processor according to an embodiment. Figure 4 The process of calculating the significance probability P value for visual feedback is shown.

[0098] First, in step S501, the visual stimulation signal receiver 110 may obtain an EEG signal within a preset time (eg, 0.175 seconds), wherein the preset time may refer to the time required to obtain the EEG signal that is minimally required to perform canonical correlation analysis (CCA).

[0099] Thereafter, in step S502 , the visual stimulation signal processor 130 may perform CCA on a plurality of visual stimulation signals having visual stimulations of different frequencies and a multi-channel EEG signal of the user to calculate a CCA coefficient for each visual stimulation.

[0100] Thereafter, in step S503, the visual stimulus signal processor 130 may perform softmax on the CCA coefficients for each visual stimulus in order to normalize them. In an embodiment, the visual stimulus signal processor 130 may perform normalization by using an L1 norm instead of softmax in order to generate visual feedback.

[0101] Thereafter, in step S504, the visual stimulus signal processor 130 may accumulate the normalized CCA coefficient for each visual stimulus. For example, the visual stimulus signal processor 130 may accumulate data for 0.5 seconds.

[0102] Thereafter, in step S505 , the visual stimulation signal processor 130 may determine whether a threshold time (eg, 4 seconds) has passed after the logic is initiated.

[0103] As a result of the determination of step S505 , when the threshold time is not exceeded, the process may proceed to step S506 , and when the threshold time is exceeded, the process may proceed to step S510 .

[0104] Thereafter, at step S506, the visual stimulus signal processor 130 may determine whether a reference time (minimum window length or detection time) has passed after the logic is activated. For example, the reference time may be 0.5 seconds.

[0105] When the reference time has not been exceeded, the process may proceed to step S501. When the reference time has been exceeded, in step S507, the visual stimulus signal processor 130 may perform covariance analysis based on the normalized CCA coefficient for each stimulus to calculate the slope of the regression line for each visual stimulus relative to the plurality of visual stimuli.

[0106] Thereafter, in step S508 , the visual stimulation signal processor 130 may calculate a significance probability P value.

[0107] In an embodiment, in terms of generating visual feedback, the visual stimulus signal processor 130 may only calculate the significance probability P value relative to the slope difference between the largest slope and the second largest slope. This is because the visual feedback may only need to be relative to the actual visual stimulus STI (reference Figure 3 The visual stimulus signal processor 130 may set the significance probability P value relative to other visual stimuli to 1.

[0108] At step S508 - 1 , the visual stimulation signal processor 130 may generate visual feedback using the calculated significance probability P value.

[0109] The time point at which the visual guide VG reacts after generating visual feedback is from the reference time (minimum window length), which is the time point at which the significance probability P value is calculated. For example, when 0.5 seconds elapses after receiving the visual stimulus signal, visual feedback can be generated based on the significance probability P value.

[0110] In an embodiment, the visual stimulus signal processor 130 may perform multiple comparisons to classify the visual stimulus to calculate multiple significance probability P values. That is, the visual stimulus signal processor 130 may refer to the maximum slope among the slopes calculated for the regression lines of the respective visual stimuli, and calculate the significance probability P value with respect to the difference with each remaining slope, respectively. The visual stimulus signal processor 130 may determine whether the significance level is met after the significance probability P value calculated in this way.

[0111] At step S509 , the visual stimulation signal processor 130 may determine whether all significance probability P values ​​satisfy the significance level.

[0112] As a determination result S509, when any significance probability P value does not satisfy the significance level, the process proceeds to step S501, and when all significance probability P values ​​satisfy the significance level, in step S510, the visual stimulus signal processor 130 can detect the visual stimulus corresponding to the maximum CCA coefficient among the normalized CCA coefficients as the visual stimulus intended by the user.

[0113] In an embodiment, the visual stimulus signal processor 130 derives a significance probability P value based on the slope of the regression line change for each visual stimulus, and thus, the initial slope setting from 0 seconds to a reference time (e.g., 0.5 seconds) for obtaining the regression line can be offset as a slope relative to the number of entire visual stimuli.

[0114] In the above, although the CCA method that does not require training is used as an example, a subject-independent filter bank canonical correlation analysis (FBCCA) method, a subject-dependent task-related component analysis (TRCA) method, etc. may also be used.

[0115] At this time, as another embodiment, when the FBCCA method is used, step S502 can be replaced by "In step S502, the visual stimulation signal processor 130 can perform FBCCA on multiple reference signals with different stimulation frequencies and the user's multi-channel EEG signal to calculate the FBCCA coefficient of each stimulus."

[0116] In addition, as another embodiment of the present disclosure, when the TRCA method is used, step S502 can be replaced by "In step S502, the visual stimulation signal processor 130 can perform TRCA on multiple reference signals with different stimulation frequencies and the user's multi-channel EEG signal to calculate the TRCA coefficient of each stimulus."

[0117] Figure 5 and Figure 6 is a diagram illustrating a visual guide according to an embodiment.

[0118] Figure 5 The flashing of the visual stimulus STI with a frequency of 6 Hz and the changes of the visual guide VG are shown. Figure 5 (a) in the figure shows the visual guide VG flickering at the same frequency whenever the visual stimulus STI with a frequency of 6 Hz flickers.

[0119] Figure 5 (b) in FIG. 5 shows a visual guide VG whose shape changes every time the visual stimulus STI flickers at a frequency of 6 Hz.

[0120] exist Figure 5 In the embodiment of (a), the visual guide VG appears when the visual stimulus STI is white, and disappears when the visual stimulus STI is black. That is, the visual guide VG may flicker at the same frequency as the visual stimulus STI.

[0121] In accordance with Figure 5 In the implementation of (b), as long as the visual stimulus STI flickers according to a specific frequency, the visual guide unit 120 may change the shape of the visual guide VG.

[0122] When the visual stimulus STI is black, the vision guide unit 120 may have a first shape F1, and when the visual stimulus STI is white, the vision guide unit 120 may have a second shape F2. Figure 5The first shape F1 and the second shape F2 are merely embodiments, and the shape change of the vision guide VG is not limited thereto. That is, the vision guide VG may be continuously exposed while changing its shape and / or pattern.

[0123] Figure 6 Shown are changes in the checkerboard that is reversed at 1-second intervals and the visual guide VG corresponding thereto with respect to the visual stimulus STI having a frequency of 6 Hz.

[0124] In an embodiment, the visual stimulation STI may include a checkerboard-based visual stimulation that is reversed according to a specific frequency, and the visual guide VG may appear on the checkerboard at the reversal time point of the checkerboard relative to the checkerboard-based visual stimulation. That is, the checkerboard-based visual stimulation STI only reverses but does not disappear, and the visual guide VG may repeatedly disappear after appearing at the reversal time point of the checkerboard and appear again at the next reversal time point.

[0125] Figure 7 is an exemplary diagram illustrating the appearance of a visual guide according to a user's gaze according to an embodiment.

[0126] exist Figure 7 , the visual guide arranged on the first visual stimulus that the user is looking at may have the first form VG1. However, it can be seen that the visual guides of the second visual stimulus and the third visual stimulus that are not looked at by the user may have the second form VG2. If the user then looks at the second visual stimulus, the visual guide of the first visual stimulus may change to the second form VG2, and the visual guide of the second visual stimulus may change to the first form VG1.

[0127] Figure 8 is a flow chart of a method for detecting visual stimuli based on SSVEP according to an embodiment. The method for detecting visual stimuli based on SSVEP can be performed by a device 100 for detecting visual stimuli based on SSVEP (see Figure 1 ) to execute.

[0128] exist Figure 8 In step S100 , the apparatus 100 for detecting visual stimulation based on SSVEP (hereinafter referred to as a visual stimulation detection apparatus) may receive a visual stimulation signal extracted through EEG analysis of a user who is fixating on a visual stimulation of a specific frequency.

[0129] At step S200, the visual stimulation detection device 100 can arrange a visual guide having a specific form on the visual stimulation. The visual guide and the visual stimulation can have the same frequency. The visual guide can flicker at the same frequency as the visual stimulation. As long as the visual stimulation flickers, the shape and pattern of the visual guide can be changed in various ways. The visual guide can be arranged at the center of the visual stimulation.

[0130] At step S300, the visual stimulation detection device 100 can classify the visual stimulation based on the received visual stimulation signal, and generate visual feedback to reflect it to the visual guide in real time. In an embodiment, the visual stimulation detection device 100 can extract the features of the visual stimulation signal and classify the visual stimulation based on the extracted features. The features can be calculated as the slope of the regression line for each visual stimulation by covariance analysis (ANCOVA).

[0131] Visual feedback may be generated based on a significance probability P value of the difference between the slopes of the regression lines for each visual stimulus. The significance probability P value may have a value between 0 and 1 and may be inversely proportional to the magnitude of the slope difference. That is, the visual stimulus STI (referenced to Figure 3 ) and the other visual stimulus (the regression line with the next largest slope), the greater the slope difference, the closer the significance probability can be to 0. The significance probability relative to the slope difference between the other visual stimuli can be set to 1.

[0132] The slope difference can represent the degree of classification relative to the visual stimulus STI. That is, the closer the significance probability is to 0, the greater the degree of classification of the visual stimulus STI with other visual stimuli, and the visual stimulus detection device 100 can detect the visual stimulus STI. The classification of each stimulus and the generation of visual feedback can be referred to in reference Figure 4 Description made.

[0133] In step S400, the visual stimulus detection apparatus 100 may change the shape of the visual guide in real time based on the visual feedback. As the saliency probability gets closer to 0, the visual guide VG (see Figure 3 ) can be changed to a first shape by reflecting the generated first feedback, and can be changed to a second shape by reflecting the generated second feedback as the significance probability is closer to 1. The visual guide VG can have a continuous shape that dynamically changes between the first shape and the second shape.

[0134] Fig. 9 is a diagram for explaining a computing device according to an embodiment.

[0135] See also Fig. 9 By using the computing device 900, the apparatus and method for detecting visual stimulation based on SSVEP according to the embodiment may be implemented.

[0136] The computing device 900 may include at least one of a processor 910, a memory 930, a user interface input device 940, a user interface output device 950, and a storage device 960 that communicate via a bus 920. The computing device 900 may also include a network interface 970 that is electrically connected to the network 90. ​​The network interface 970 may send or receive signals with other entities via the network 90.

[0137] The processor 910 may be implemented in various types, such as a microcontroller unit (MCU), an application processor (AP), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), etc., and may be any type of semiconductor device capable of executing instructions stored in the memory 930 or the storage device 960. The processor 910 may be configured to implement the above-mentioned Figures 1 to 8 Describes the functions and methods. For example, see Figures 1 to 8 The different elements described may be implemented by the processor 910 as a whole or individually, so that the processor 910 may be configured to implement the functions and methods described above.

[0138] The memory 930 and the storage device 960 may include various types of volatile or non-volatile storage media. For example, the memory may include a read-only memory (ROM) 931 and a random access memory (RAM) 932. In this embodiment, the memory 930 may be located inside or outside the processor 910, and the memory 930 may be connected to the processor 910 by various known means.

[0139] In some embodiments, at least some configurations or functions of the apparatus and method for detecting visual stimulation based on SSVEP according to the embodiments may be implemented as a program or software executable by the computing device 900, and the program or software may be stored in a computer-readable medium.

[0140] In some embodiments, at least some configurations or functions of the apparatus and method for detecting visual stimuli based on SSVEP according to the embodiments may be implemented by using hardware or circuits of the computing device 900, or may also be implemented as separate hardware or circuits that may be electrically connected to the computing device 900.

[0141] While the disclosure has been described in conjunction with what are presently considered to be practical embodiments, it is to be understood that the disclosure is not limited to the disclosed embodiments, but on the contrary, the disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A device for detecting visual stimulation based on steady-state visual evoked potential (SSVEP), comprising: a visual stimulation signal receiver configured to receive a visual stimulation signal extracted by electroencephalogram (EEG) analysis of a user with respect to fixating on a visual stimulation of a specific frequency; A visual guide unit configured to arrange a visual guide having a specific form on the visual stimulus; a visual stimulus signal processor configured to classify the visual stimulus and generate visual feedback based on the received visual stimulus signal; as well as a visual feedback reflector configured to reflect the generated visual feedback to the visual guide, Wherein, the visual guide unit is configured to change the shape of the visual guide based on the reflected visual feedback.

2. The device according to claim 1, wherein: The visual guide has the same frequency as the specific frequency of the visual stimulus.

3. The device according to claim 2, wherein: The visual guide unit is configured to change a shape of the visual guide when the visual stimulus flickers according to the specific frequency.

4. The device according to claim 1, wherein: The visual stimulus comprises a checkerboard-based visual stimulus that is inverted according to the specific frequency; and The visual guide appears on the checkerboard at a reversal time point of the checkerboard relative to the checkerboard-based visual stimulus.

5. The device according to claim 1, wherein: The visual stimulus signal processor comprises: a visual stimulus classifier configured to extract features of the visual stimulus signal and classify the visual stimulus based on the extracted features; and A visual feedback generator is configured to generate the visual feedback including feedback information related to the classification of the visual stimulus.

6. The device according to claim 5, wherein: The visual stimulus signal processor is configured to calculate a slope of a regression line for each visual stimulus by an analysis of covariance (ANCOVA) with respect to a plurality of visual stimuli including the visual stimulus; The visual feedback generator is configured to generate a significance probability with respect to a difference between a maximum first slope occurring corresponding to the visual stimulus and a second slope corresponding to another visual stimulus other than the visual stimulus as the visual feedback; and The significance probability is inversely proportional to the size of the difference between the slopes.

7. The device according to claim 6, wherein: The significance probability has a value varying within a range of not less than 0 and not more than 1 with respect to the visual stimulus; and The visual feedback generator is configured to generate positive first feedback relative to the classification as the significance probability is closer to 0, and to generate negative second feedback relative to the classification as the significance probability is closer to 1.

8. The device according to claim 7, wherein: The visual guide unit is configured to change the shape of the visual guide to a first shape in response to the first feedback, and to change the shape of the visual guide to a second shape in response to the second feedback.

9. The device according to claim 8, wherein: The visual feedback reflector is configured to reflect the visual feedback including the changed significance probability to the visual guide in real time; and The vision guide unit is configured to change the shape of the vision guide in real time to have a continuous shape that dynamically changes between the first shape and the second shape in response to the real-time visual feedback.

10. The device according to claim 6, wherein: When the significance probability satisfies a significance level, the visual stimulus classifier is configured to classify a stimulus corresponding to a maximum Canonical Correlation Analysis (CCA) coefficient among CCA coefficients used in the analysis of covariance as the visual stimulus intended by the user.

11. The device according to claim 1, wherein: The visual guide unit is configured to arrange the visual guide at the center of the visual stimulus.

12. A method for detecting visual stimulation, comprising: receiving, by a device for detecting visual stimulation based on steady-state visual evoked potential (SSVEP), a visual stimulation signal extracted by electroencephalogram (EEG) analysis of a user with respect to fixating on a visual stimulation of a specific frequency; Arranging a visual guide having a specific form on the visual stimulus by the device for detecting the visual stimulus based on the SSVEP; The device for detecting visual stimuli based on SSVEP classifies the visual stimuli based on the received visual stimulus signals and generates visual feedback; as well as The device for detecting visual stimulation based on SSVEP reflects the visual feedback to the visual guide in real time, Wherein arranging the visual guide comprises changing a shape of the visual guide in real time based on the visual feedback.

13. The visual stimulation detection method according to claim 12, wherein: The visual guide has the same frequency as the specific frequency of the visual stimulus.

14. The visual stimulation detection method according to claim 13, wherein: Arranging the visual guide further includes changing a shape of the visual guide when the visual stimulus flickers according to the specific frequency.

15. The visual stimulation detection method according to claim 12, wherein: The classifying the visual stimulus based on the received visual stimulus signal and generating visual feedback comprises: extracting features of the visual stimulus signal and classifying the visual stimulus based on the extracted features; and The visual feedback is generated including feedback information related to the classification of the visual stimulus.

16. The visual stimulation detection method according to claim 15, wherein: Generating the visual feedback includes calculating the slope of the regression line for each visual stimulus, the calculation being performed by covariance analysis with respect to a plurality of visual stimuli including the visual stimulus; and calculating the significance probability with respect to the difference between a maximum first slope appearing corresponding to the visual stimulus and a second slope corresponding to another visual stimulus other than the visual stimulus, as the visual feedback, The significance probability is inversely proportional to the size of the difference between the slopes.

17. The visual stimulation detection method according to claim 16, wherein: The significance probability has a value varying within a range of greater than 0 and less than 1 with respect to the visual stimulus; and Generating the visual feedback further includes: generating positive first feedback relative to the classification as the significance probability is closer to 0; and generating negative second feedback relative to the classification as the significance probability is closer to 1.

18. The visual stimulation detection method according to claim 17, wherein: Arranging the visual guide further includes changing the shape of the visual guide to a first shape in response to the first feedback, and changing the shape of the visual guide to a second shape in response to the second feedback.

19. The visual stimulation detection method according to claim 16, wherein: Generating visual feedback further comprises: excluding the significance probability with respect to the difference between the first slope and the second slope, and setting all significance probabilities with respect to the slope differences between other visual stimuli to 1; and The magnitude of the second slope is the next largest, second only to the magnitude of the first slope.

20. The visual stimulation detection method according to claim 12, wherein: Arranging the visual guide further comprises arranging the visual guide at a center of the visual stimulus.

Images