Method and apparatus for presenting visual feedback

Abstract

The invention relates to a method and a device for presenting visual feedback. The method includes receiving a steady-state visual evoked potential signal extracted by an electroencephalogram analysis of a user gazing a visual stimulus at a particular frequency. The method further includes classifying the visual stimulus based on the SSVEP signal and generating a classification result. In addition, the method includes setting visual feedback on the visual stimulus having the same frequency as the visual stimulus. The method also includes reflecting the classification result in the visual feedback in real time.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and benefit to Korean Patent Application No. 10-2024-0062759, filed on May 13, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to methods and apparatus for presenting visual feedback. Background Technology

[0004] Steady-state visual evoked potentials (SSVEPs) are electroencephalographic potentials generated when a visual stimulus flashes at a specific frequency. SSVEPs can be extracted through analysis of electroencephalograms (EEGs) measured near the occipital lobe.

[0005] Because specific frequencies of the visual stimulus being gazed upon can be detected from EEG signals, EEG analysis can be used to identify the visual stimulus being gazed upon by the user. Therefore, EEG analysis can be used to develop various brain-computer interfaces (BCIs).

[0006] Despite the development of SSVEP-based BCI algorithms, the performance of BCI remains significantly affected by user fatigue, concentration, and attitude. Furthermore, to detect SSVEP signals with consistent performance, a fixed SSVEP stimulation time (fixed time window) is used in previous offline analysis, and the outcome of the user's gaze can be determined after the stimulation ends.

[0007] For SSVEP stimuli, there is a difficulty in observing how each person views the stimulus because of the variability in how they view it. Summary of the Invention

[0008] The embodiments of this disclosure provide a technique to make users more immersed in the BCI system and correspondingly improve performance by allowing users to check real-time EEG responses while presenting SSVEP-induced visual stimuli.

[0009] Embodiments of this disclosure provide a method and apparatus for presenting visual feedback. The method and apparatus intuitively perceive the visual stimulus being gazed upon by presenting visual feedback in real time in response to the SSVEP signal generated when the user gazes at the visual stimulus, and improve detection performance by enhancing the user's gaze concentration.

[0010] According to an embodiment, a method for presenting visual feedback is provided. The method includes receiving steady-state visual evoked potential (SSVEP) signals extracted from an electroencephalogram (EEG) analysis of a user who has gazed at a visual stimulus of a specific frequency. The method further includes classifying the visual stimulus based on the SSVEP signals to generate a classification result. Additionally, the method includes applying visual feedback with the same frequency as the visual stimulus to the visual stimulus. The method also includes reflecting the classification result in the visual feedback in real time.

[0011] Reflecting the classification results in real time in visual feedback may include: changing the shape of the first visual feedback set in the first visual stimulus classified from multiple visual stimuli according to the classification results to a first shape; setting visual feedback in each of the multiple visual stimuli; and changing the shape of the second visual feedback set in the second unclassified visual stimulus to a second shape.

[0012] Reflecting classification results in visual feedback in real time can include: periodically collecting classification results at each preset time point within the maximum window length; calculating a count value for the visual stimulus based on the classification results collected at each time point; and changing the shape of the visual feedback in real time based on the calculated count value.

[0013] The collection of classification results may include: when the length from the time point when the visual stimulus first flashes to the current time point is greater than the maximum window length, collecting as many classification results as the maximum window length based on the current time point.

[0014] The count value can have a maximum and a minimum value, and calculating the count value for a visual stimulus can include setting the count value at the initial time point when the visual stimulus is presented to an arbitrary maximum value.

[0015] Calculating the count value for a visual stimulus may also include: initializing the count value of each visual stimulus to the maximum value at the initial time point when the visual stimulus is first presented.

[0016] Calculating the count value for a visual stimulus may include: when comparing the classification results of consecutive first and second time points and finding that they are the same as the first visual stimulus, for the classified first visual stimulus, determining the count value at the second time point as the value obtained by subtracting a specific value from the count value at the first time point, and for the unclassified second visual stimulus, determining the count value at the second time point as the value obtained by adding the specific value to the count value at the first time point.

[0017] Calculating the count value for a visual stimulus may further include: when a first visual stimulus is classified at a first time point and a second visual stimulus is classified at a second time point, maintaining the count value at the second time point as the same as the count value at the first time point for the second visual stimulus, and determining the count value at the second time point for the first visual stimulus as a value obtained by adding a specific value to the count value at the first time point.

[0018] The value obtained by adding a specific value to the count value can be less than or equal to the maximum value of the count value, and the value obtained by subtracting a specific value from the count value can be greater than or equal to the minimum value of the count value.

[0019] The shape of the visual feedback has a first shape when the count value is at its maximum value and a second shape when the count value can be at its minimum value. Changing the shape of the visual feedback in real time can include: gradually changing the shape of the visual feedback between the first shape and the second shape at each time point in response to the calculated count value.

[0020] According to another embodiment, a device for presenting visual feedback is provided. The device includes a signal receiver configured to receive steady-state visual evoked potential (SSVEP) signals extracted from an electroencephalogram (EEG) analysis of a user fixating on a visual stimulus of a specific frequency. The device also includes a signal processor configured to classify the visual stimulus based on the SSVEP signals and generate classification results. The device further includes a visual feedback reflector configured to apply visual feedback with the same frequency as the visual stimulus to the visual stimulus and to reflect the classification results in the visual feedback in real time.

[0021] The visual feedback reflector can be configured to change the shape of a first visual feedback set in a first visual stimulus classified from multiple visual stimuli according to classification results to a first shape, set the visual feedback in each of the multiple visual stimuli, and change the shape of a second visual feedback set in a second unclassified visual stimulus to a second shape.

[0022] The visual feedback reflector can be configured to periodically collect classification results at each preset time point within the maximum window length, calculate a count value for the visual stimulus based on the classification results collected at each time point, and change the shape of the visual feedback in real time based on the calculated count value.

[0023] When the distance from the first flash of the visual stimulus to the current time point is greater than the maximum window length, the visual feedback reflector can be configured to collect as many classification results as the maximum window length based on the current time point.

[0024] The count value can have a maximum and a minimum value, and the visual feedback reflector can be configured to set the count value at the initial time point when the visual stimulus is presented to an arbitrary maximum value.

[0025] When visual stimuli are presented, the visual feedback reflector can be configured to initialize the count value of each visual stimulus at the initial time point to the maximum value.

[0026] When the classification results of consecutive first and second time points are compared and found to be the same as the first visual stimulus, the visual feedback reflector can be configured to: determine the count value at the second time point as the value obtained by subtracting a specific value from the count value at the first time point for the classified first visual stimulus; and determine the count value at the second time point as the value obtained by adding the specific value to the count value at the first time point for the unclassified second visual stimulus.

[0027] When the classification result at the first time point is the first visual stimulus and the classification result at the second time point is the second visual stimulus, a difference is generated. The visual feedback reflector can be configured to: maintain the count value at the second time point at the same value as the count value at the first time point with respect to the second visual stimulus, and determine the count value at the second time point as the value obtained by adding a specific value to the count value at the first time point with respect to the first visual stimulus.

[0028] The value obtained by adding a specific value to the count value can be less than or equal to the maximum value of the count value, and the value obtained by subtracting a specific value from the count value can be greater than or equal to the minimum value of the count value.

[0029] The shape of the visual feedback can have a first shape when the count value is at its maximum and a second shape when the count value is at its minimum, and the visual feedback reflector can be configured to gradually change the shape of the visual feedback between the first shape and the second shape at each time point in response to the calculated count value.

[0030] The method and apparatus for presenting visual feedback according to the embodiments can intuitively perceive the visual stimulus being gazed at by presenting visual feedback in real time in response to the SSVEP signal generated when the user gazes at the visual stimulus, and improve detection performance by enhancing the user's gaze concentration. Attached Figure Description

[0031] Figure 1 An SSVEP-based BCI system according to an embodiment is illustrated schematically.

[0032] Figure 2 This is a block diagram of a device for presenting visual feedback according to an embodiment.

[0033] Figure 3 This is a flowchart of a method for presenting visual feedback according to an implementation method.

[0034] Figure 4 This is a flowchart of a method for presenting visual feedback according to an implementation method.

[0035] Figure 5 and Figure 6 This is an illustration illustrating a method for presenting visual feedback according to an implementation method.

[0036] Figure 7 This is a diagram illustrating the computing device according to an embodiment. Detailed Implementation

[0037] Embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings to enable those skilled in the art to readily practice them. As will be appreciated by those skilled in the art, the described embodiments can be modified in various ways without departing from the spirit or scope of this disclosure. For the purpose of clarifying this disclosure, parts irrelevant to the description have been omitted, and the same elements or equivalents are indicated by the same reference numerals throughout the specification.

[0038] Furthermore, unless explicitly stated otherwise, words such as “comprising” or “including” and variations such as “having,” “containing,” “covering,” or “encompassing” should be understood as implying inclusion of the stated element, but not excluding any other element. Terms including common numbers such as first and second are used to describe various constituent elements, but constituent elements are not limited by the terms. These terms are used only to distinguish one component from others.

[0039] In addition, the terms “unit,” “component,” “part,” “device,” and “module” in the specification refer to a unit that performs at least one function or operation, which can be implemented by hardware, software, or a combination of hardware and software.

[0040] When a component, apparatus, element, etc., of this disclosure is described as having a purpose or performing an operation or function, the component, apparatus, or element shall be considered herein as being "configured" to satisfy that purpose or perform that operation or function.

[0041] In the following description, embodiments of the present disclosure are described with reference to the accompanying drawings.

[0042] Figure 1 A brain-computer interface (BCI) system based on SSVEP according to an embodiment is illustrated schematically.

[0043] refer to Figure 1The BCI system according to the embodiments may include a device 100 for presenting visual feedback, a steady-state visual evoked potential (SSVEP) generator 10, and an external device 20.

[0044] According to embodiments of the present disclosure, the device 100 for presenting visual feedback can classify the visual stimuli that the user is looking at based on steady-state visual evoked potentials (SSVEPs), and can reflect the classification results in real time to the visual feedback set in the classified visual stimuli, so that the visual stimuli that the user is looking at can be identified immediately.

[0045] Here, steady-state visual evoked potentials (SSVEPs) are electroencephalographic potentials generated when a visual stimulus flashes at a specific frequency. SSVEPs can be extracted through analysis of electroencephalograms (EEGs) measured near the occipital lobe.

[0046] Since specific frequencies of the visual stimulus being gazed upon can be detected from EEG signals, steady-state visual evoked potentials (SSVEPs) can be identified through EEG analysis of the visual stimulus gazed upon by the user.

[0047] Therefore, SSVEP can be used to develop various brain-computer interfaces (BCIs). SSVEP can be referred to as the SSVEP signal.

[0048] The device 100 for presenting visual stimuli according to an embodiment may include visual feedback set on the visual stimulus being gazed upon by a user. The device 100 may respond to the SSVEP signal in real time to provide real-time feedback to the user so that the visual stimulus being gazed upon by the user can be detected more quickly and accurately.

[0049] The SSVEP generator 10 can provide the user with visual stimuli corresponding to control commands about the external device 20, and can induce the user to generate electroencephalogram (EEG) signals including an electroencephalogram corresponding to the visual stimuli.

[0050] For example, when a user gazes at an arrow pointing forward, the corresponding EEG signal is included in the user's EEG signal. Therefore, device 100 can detect the EEG signal corresponding to the forward-pointing arrow from the user's EEG signal. SSVEP generator 10 can transmit the EEG signal to device 100 for presenting visual feedback.

[0051] External device 20 can be connected to device 100 via a network. External device 20 can communicate with device 100 and be controlled according to commands received from device 100 for presenting visual feedback.

[0052] For example, external device 20 may include personal mobility devices such as wheelchairs, exoskeletons, etc.

[0053] Figure 2 This is a block diagram of a device for presenting visual feedback according to an embodiment.

[0054] refer to Figure 2 The device 100 for presenting visual feedback may include a signal receiver 110, a signal processor 120, and a visual feedback reflector 130.

[0055] The signal receiver 110 can receive steady-state visual evoked potential (SSVEP) signals extracted from the electroencephalogram (EEG) analysis of a user who gazes at visual stimuli of a specific frequency.

[0056] Visual stimulus VS can be an image that flashes at a specific frequency. Visual stimulus VS can also be a chessboard image that is repeatedly reversed at a specific frequency.

[0057] For example, signal receiver 110 may receive an SSVEP signal or an SSVEP signal extracted from the electroencephalogram signal of a user fixating on visual stimulation VS, which includes an image that flashes 10 times per second (i.e., at a frequency of 10 Hz).

[0058] The signal processor 120 can classify visual stimuli based on SSVEP signals and generate classification results.

[0059] The signal processor 120 can classify visual stimuli VS based on the received SSVEP signal. The signal processor 120 can also classify visual stimuli gazed at by the user from another visual stimulus based on the SSVEP signal.

[0060] The signal processor 120 can extract features from the SSVEP signal received from the signal receiver 110 and can classify the visual stimulus VS based on the extracted features.

[0061] The signal processor 120 can extract features from the SSVEP signal. Feature extraction is the process of extracting important information from an electroencephalogram (EEG) signal. The most important feature in the SSVEP signal can be the frequency of the EEG.

[0062] For example, the signal processor 120 can extract frequency components from the electroencephalogram (EEG) signal by using Fourier transform or wavelet transform.

[0063] The signal processor 120 can identify which visual stimulus the user has responded to based on the amplitude of the power of each frequency component.

[0064] The signal processor 120 can classify visual stimuli (VS) based on extracted features. Classification is the process of assigning EEG signals to specific categories or classes using extracted features.

[0065] For example, when a user focuses on a specific visual stimulus, an EEG response corresponding to the frequency of the visual stimulus can be generated, and the signal processor 120 can determine which stimulus the EEG signal responds to by classification.

[0066] The signal processor 120 can classify visual stimuli VS using a classification model. For example, the signal processor 120 can use filter bank canonical correlation analysis (FBCCA) and / or various other classification models used for SSVEP classification.

[0067] In implementation, the classification algorithm can be based on machine learning techniques. For example, the signal processor 120 can classify visual stimuli using support vector machines (SVM), k-nearest neighbors (K-NN), linear deterministic analysis (LDA), etc.

[0068] The visual feedback reflector 130 can set a visual feedback VF with the same frequency as the visual stimulus VS on the visual stimulus VS.

[0069] The visual feedback reflector 130 can reflect the classification results in the visual feedback VF in real time.

[0070] The visual feedback reflector 130 can position a visual feedback VF with a specific shape on the visual stimulus VS. The visual feedback reflector 130 can position the visual feedback VF at the center of the visual stimulus VS.

[0071] Visual feedback (VF) can guide a user's gaze and improve their concentration. The shape, pattern, and color of VF are not particularly limited and can be freely customized.

[0072] In this implementation, the visual feedback VF may have the same frequency as the specific frequency of the visual stimulus VS that the user is looking at. Therefore, the visual feedback VF may flash at the same frequency as the visual stimulus VS. Thus, the visual feedback VF can be synchronized with the frequency of the visual stimulus VS that the user is looking at, and can enhance the frequency stimulation delivered to the user.

[0073] The visual feedback reflector 130 can change the shape of a first visual feedback set in a first visual stimulus classified according to a classification result from multiple visual stimuli VS to a first shape, wherein a visual feedback VF is set in each of the multiple visual stimuli VS. The visual feedback reflector 130 can also change the shape of a second visual feedback set in a second unclassified visual stimulus to a second shape.

[0074] The visual feedback reflector 130 can periodically collect classification results at preset time points within the maximum window length.

[0075] When the distance from the time of the first flash of the visual stimulus VS to the current time is greater than the maximum window length, the visual feedback reflector 130 can collect as many classification results as the maximum window length based on the current time.

[0076] The visual feedback reflector 130 can calculate a count value for the visual stimulus VS based on the classification results collected at each time point.

[0077] The count value can have a maximum and a minimum value. The visual feedback reflector 130 can set the count value at the initial time point when the visual stimulus is presented to an arbitrary maximum value.

[0078] When visual stimuli are presented, the visual feedback reflector 130 can initialize the count of each visual stimulus at the initial time point to the maximum value.

[0079] The visual feedback reflector 130 can change the shape of the visual feedback VF in real time based on the calculated count value.

[0080] When the classification results of consecutive first and second time points are compared and found to be the same as the first visual stimulus, the visual feedback reflector 130 can determine the value obtained by subtracting a specific value from the count value at the first time point as the count value of the first visual stimulus at the second time point.

[0081] When the classification result is the same as the first visual stimulus, for the unclassified second visual stimulus, the visual feedback reflector 130 can determine the value obtained by adding the specific value to the count value at the first time point as the count value of the second visual stimulus at the second time point.

[0082] When the classification result at the first time point is the first visual stimulus and the classification result at the second time point is the second visual stimulus, which is different from the classification result at the first time point, the visual feedback reflector 130 can maintain the count value at the second time point as the same as the count value at the first time point for the second visual stimulus.

[0083] When the classification result at the first time point is the first visual stimulus and the classification result at the second time point is the second visual stimulus, which is different from the classification result at the first time point, the visual feedback reflector 130 can determine the count value at the second time point as the value obtained by adding a specific value to the count value at the first time point, regarding the first visual stimulus.

[0084] Here, the value obtained by adding a specific value to the count value can be less than or equal to the maximum value of the count value, and the value obtained by subtracting a specific value from the count value can be greater than or equal to the minimum value of the count value. Therefore, the count value can vary between the maximum and minimum values.

[0085] In an implementation, the visual feedback (VF) may have a first shape when the count value is at its maximum and a second shape when the count value is at its minimum.

[0086] The visual feedback reflector 130 can respond to the calculated count value and gradually change the shape of the visual feedback VF between the first shape and the second shape at each time point.

[0087] Figure 3 This is a flowchart of a method for presenting visual feedback according to an implementation method. Figure 3 The method for presenting visual feedback can be executed by the device 100 for presenting visual feedback.

[0088] exist Figure 3 In step or operation S100, device 100 may receive steady-state visual evoked potential (SSVEP) signals extracted from electroencephalogram (EEG) analysis of a user who gazes at visual stimuli of a specific frequency.

[0089] Device 100 can receive SSVEP signals from an EEG analysis of a user who gazes at visual stimuli, as well as visual feedback set on the visual stimuli.

[0090] In step or operation S200, device 100 can classify visual stimuli based on the SSVEP signal using a classification model and generate classification results.

[0091] Device 100 can generate classification results by using various classification models for SSVEP classification, including filter bank canonical correlation analysis (FBCCA).

[0092] In step or operation S300, device 100 can set visual feedback with the same frequency as the visual stimulus on the visual stimulus, and can reflect the classification result in real time in the visual feedback.

[0093] Device 100 can change the shape of a first visual feedback set in a first visual stimulus classified according to a classification result from multiple visual stimuli to a first shape, and set the visual feedback in each of the multiple visual stimuli. Device 100 can also change the shape of a second visual feedback set in a second unclassified visual stimulus to a second shape.

[0094] Device 100 can periodically collect classification results at preset time points within the maximum window length.

[0095] Device 100 can calculate counts about visual stimuli based on classification results collected at each time point. The counts can have a maximum and a minimum value. The maximum and minimum values ​​can be any preset values.

[0096] Device 100 can change the shape of the visual feedback in real time based on the calculated count value.

[0097] For example, device 100 can change the shape of the visual feedback to a first shape when the count value is at its maximum value, and can change the shape of the visual feedback to a second shape when the count value is at its minimum value.

[0098] At each time point, device 100 can increase the count value with respect to the visual stimulus classified according to the classification result, and can gradually change the shape of the visual feedback set on the classified visual stimulus to the first shape.

[0099] At each time point, device 100 can decrease the count value for unclassified visual stimuli based on the classification results, and can gradually change the shape of the visual feedback set on the unclassified visual stimuli to a second shape.

[0100] Therefore, the device 100 can respond to the calculated count value and gradually change the shape of the visual feedback between the first shape and the second shape at each time point.

[0101] Figure 4 This is a flowchart of a method for presenting visual feedback according to an implementation method.

[0102] Figure 4 It shows the basis in more detail. Figure 3 A flowchart of an implementation method for presenting visual feedback. Figure 4 The method for presenting visual feedback can be executed by the device 100 for presenting visual feedback.

[0103] refer to Figure 4 In step or operation S410, at t0 when the visual stimulus is first presented, device 100 can initialize the count value C for each visual stimulus to the maximum value Cmax. The maximum value Cmax and the minimum value Cmin can be set arbitrarily.

[0104] For example, the maximum value can be set to 11, and the minimum value Cmin can be set to 1. As the maximum value Cmax of the count value C increases, the shape of the visual feedback may change more slowly.

[0105] In step or operation S420, device 100 may check whether the time period from the time point t0 when the visual stimulus first flashes to the current time point t exceeds a predetermined maximum window length. Here, the window may refer to the length of data used to analyze the visual stimulus.

[0106] In step or operation S431, when the time period from the first flash of the visual stimulus to the current time point t does not exceed the maximum window length, the device 100 can extract the accumulated EEG data and classification results from the first flash to the current time point t, and can put them into the classification algorithm or classification model CM as input data.

[0107] In step or operation S432, when the time period from the first flashing point t0 to the current time point t exceeds the maximum window length, device 100 can extract as much EEG data and classification results as the maximum window length from the time point (current time point t - maximum window length) to the current time point t and can send them to the classification model CM.

[0108] For example, when the current time t is 5 seconds and the maximum window length is set to 3 seconds, device 100 can use only the data from 2 seconds to 5 seconds.

[0109] In step or operation S440, device 100 can export the classification result y at the current time point derived from classification model CM. t The classification results y at the next adjacent previous time point （t-1） Comparisons can be made. The classification model CM can use filter bank canonical correlation analysis (FBCCA) and / or other models used for SSVEP classification.

[0110] At step or operation S451, the classification result y at the current time point t The classification results y at the next adjacent previous time point （t-1） At the same time, device 100 can count C(y) from previous time points of the first visual stimulus classified as a target. （t-1） Subtract a specific value (e.g., 1) from the original value.

[0111] Therefore, the device 100 can determine the count value at the second time point as a value obtained by subtracting a specific value from the count value at the first time point for the same first visual stimulus that is classified by comparing the classification results at consecutive first and second time points.

[0112] Furthermore, at step or operation S451, device 100 can assign the same specific value (e.g., 1) to the count value C(-y) of a second visual stimulus that is not classified as a target. （t-1） Add them together.

[0113] Therefore, regarding an unclassified second visual stimulus, the device 100 for presenting visual feedback can determine the count value at the second time point as a value obtained by adding a specific value to the count value at the first time point.

[0114] In the example, the count value C cannot exceed the maximum value Cmax and cannot be less than the minimum value Cmin. Therefore, when the visual stimulus is classified as a target, the count value C(y) t If the value is already 1, it can be maintained thereafter.

[0115] At step or operation S452, the classification result y at the current time point t The classification results y at the next adjacent previous time point （t-1） At the same time, device 100 can maintain the previous time point count C(y) of visual stimuli classified as targets. （t-1） ), and can only be a specific value 1 with the count C(-y) of visual stimuli that are not classified as targets. （t-1） Add them together. This is to avoid misclassification due to floating-point values.

[0116] In other words, when the classification result at the first time point is the first visual stimulus and the classification result at the second time point is the second visual stimulus which is different from the first visual stimulus, the device 100 can maintain the count value at the second time point as the same as the count value at the first time point regarding the second visual stimulus.

[0117] Regarding the first visual stimulus that is not classified at the second time point, the device 100 can determine the count value at the second time point as a value obtained by adding a specific value to the count value at the first time point.

[0118] At step or operation S460, device 100 can reflect the count value C in, for example... Figure 2 The visual feedback VF is shown in the diagram. The shape of the visual feedback VF can change in real time according to the count value C.

[0119] For example, when the count value is the maximum value Cmax, the shape of the visual feedback can have a second shape S2 that is widely extended, and when it is classified as a target and the count value continues to decrease to the minimum value Cmin, the shape of the visual feedback can have a first shape S1 that is a highly contracted triangle.

[0120] The device 100 can gradually change the shape of the visual feedback VF between the first shape S1 and the second shape S2 based on the count value C at each time point.

[0121] For example, visual feedback (VF) can change between a first shape S1, in which the three parts are combined, and a second shape S2, in which the three parts are dispersed, based on a count value.

[0122] As the count value decreases, the shape of the visual feedback VF can change in a direction that is closer to the first shape S1 than the second shape S2. For example, as the count value decreases, the three parts dispersed in the shape of the visual feedback VF can become closer to each other.

[0123] As the count value increases, the shape of the visual feedback VF can change in a direction that is closer to the second shape S2 than the first shape S1. For example, as the count value increases, the three parts can become farther apart from each other.

[0124] When the count value is at its maximum, the three parts can be furthest apart, and when the count value is at its minimum, the three parts can be closest together to form a triangle.

[0125] Figure 4 In and the following description Figure 5 and Figure 6 The shapes of the visual feedback shown are for illustrative purposes only, and the shapes of the visual feedback are not limited to those shown in the figure.

[0126] Figure 5 and Figure 6 This is an illustration illustrating a method for presenting visual feedback according to an implementation method. Figure 5 and Figure 6 It is shown that, in order to explain the basis Figure 4 Examples of different scenarios for implementing methods of presenting visual feedback are provided below. Reference is also made to these examples in the following text. Figure 4 Describe it.

[0127] Figure 5 This is a diagram used to explain a method for presenting visual feedback based on a scenario where the user is looking at a second visual stimulus VS2 that flashes for 5 seconds.

[0128] exist Figure 5 In this process, the first visual stimulus VS1, the second visual stimulus VS2, and the third visual stimulus VS3 can flash at different frequencies.

[0129] The first visual feedback VF1 can be set in the first visual stimulus VS1, the second visual feedback VF2 can be set in the second visual stimulus VS2, and the third visual feedback VF3 can be set in the third visual stimulus VS3.

[0130] The first visual stimulus VS1 and the first visual feedback VF1 can have the same frequency, the second visual stimulus VS2 and the second visual feedback VF2 can have the same frequency, and the third visual stimulus VS3 and the third visual feedback VF3 can have the same frequency.

[0131] When the first visual stimulus VS1, the second visual stimulus VS2, and the third visual stimulus VS3 start at 0s, the initial count value C can be set to 11, i.e., the maximum value Cmax.

[0132] Then, device 100 can receive classification results for the first visual stimulus VS1, the second visual stimulus VS2, and the third visual stimulus VS3 by using an FBCCA classification model based on EEG data collected every 0.04 s.

[0133] Device 100 can calculate a count for each of the first visual stimulus VS1, the second visual stimulus VS2, and the third visual stimulus VS3 based on the classification results.

[0134] exist Figure 5 In the case where the user continuously gazes at the second visual stimulus VS2 for 5 seconds, the classification results from the stimulus start point 0 s to the current time point 5 s are the same as those for the second visual stimulus VS2.

[0135] Therefore, device 100 can subtract 1 from the count value for the second visual stimulus VS2 every 0.04 s.

[0136] The device 100 may subtract 1 from the count value for the classified visual stimulus only if the previous classification result is the same as the current classification result every 0.04 s and the previous classification result is the same as the current classification result.

[0137] For example, since the classification results at time point 0.04 s and time point 0.08 s are the same as those of the second visual stimulus VS2, the count value of the second visual stimulus VS2 at time point 0.08 s can be determined as the value obtained by subtracting 1 from the count value at time point 0.04 s.

[0138] However, since there is no classification result at the initial time point 0 s when the visual stimulus is presented, at time point 0.04 s, the value 1 can be subtracted from the initial count value 11 of the second visual stimulus VS2 classified with respect to the corresponding time point.

[0139] When the previous classification result is compared with the current classification result every 0.04 seconds and the previous classification result is the same as the current classification result, the device 100 for presenting visual feedback can increment the count value of the unclassified visual stimulus by 1.

[0140] However, since the count value cannot exceed the maximum value, the first visual stimulus VS1 and the third visual stimulus VS3, which were initially set to the maximum value, are not incremented by 1.

[0141] Therefore, for the unclassified first visual stimulus VS1 and third visual stimulus VS3, the initial count value of 11 can be maintained until time point 5 s.

[0142] In addition, since the count value cannot decrease below the minimum value, once the count value for the second visual stimulus reaches the minimum value, it can remain at the minimum value thereafter.

[0143] As the user gazes at the second visual stimulus VS2 from time 0.04 s to time 0.44 s, the device 100 for presenting visual feedback can subtract 1 from the count value for the second visual stimulus VS2 at each time point.

[0144] exist Figure 5 In the study, the count value of the second visual stimulus VS2 reached 1 at time point 0.44 s, which is the minimum value.

[0145] When the user's gaze is held on the second visual stimulus VS2 from time 0.44 s to 5 s, the device 100 can maintain the count value of the second visual stimulus VS2 at 1 for up to 5 seconds after time 0.44 s.

[0146] Device 100 can predefine the maximum window length.

[0147] For example, device 100 can predefine the maximum window length to 3 seconds. When the maximum window length is 3 seconds, device 100 can use only the EEG data from time point 2 s to time point 5 s, and can omit the EEG data from time point 0 s to time point 1.96 s.

[0148] Therefore, device 100 can reflect the classification results of visual stimuli based on EEG data from time point 2 s to time point 5 s in the visual feedback.

[0149] Device 100 can reflect the count values ​​calculated about visual stimuli to visual feedback in real time. The visual feedback can change shape based on the count values.

[0150] The shape of the first visual feedback VF1 set in the first visual stimulus VS1 can have the same shape from time point 0 s to time point 5 s when the visual stimulus is presented. That is, since the count value of the unclassified first visual stimulus VS1 remains at the initial value of 11, the shape of the first visual feedback VF1 can remain, for example, as shown in the figure. Figure 4 The second shape S2 is shown in the figure.

[0151] Even in the case of third visual feedback VF3 to third visual stimulus VS3, the shape of third visual feedback VF3 can maintain the second shape S2 because the count value remains at the initial maximum value of 11 until the current time point 5 s.

[0152] The shape of the second visual feedback VF2, set within the second visual stimulus VS2 categorized according to a 5-second classification result, can change based on the count value. As the count value decreases from a maximum value of 11, the shape of the second visual feedback VF2 can change from a second shape S2 to a first shape S1.

[0153] Therefore, based on the maximum count value 11, the second visual feedback VF2 at the initial time point 0 s can have a second shape S2, and based on the minimum count value, the second visual feedback VF2-1 at the time point 0.44 s can have a first shape S1.

[0154] The second visual feedback VF2-1, from time 0.44 s to time 5 s, can maintain the first shape S1 by reflecting the maintained count value.

[0155] Figure 6 This is a diagram used to explain a method for presenting visual feedback based on a scenario where a user changes from looking at a second visual stimulus VS2 to looking at a first visual stimulus VS1 within 5 seconds.

[0156] exist Figure 6 In the experiment, when the user fixates on the second visual stimulus VS2 at the initial time point 0 s, the user can start fixating on the first visual stimulus VS1 at time point 0.48 s and continue fixating on the first visual stimulus VS1 until time point 5 s.

[0157] The classification results from the initial time point 0 s to time point 0.44 s can be the same as those for the second visual stimulus VS2. Therefore, the count value for the second visual stimulus VS2 can decrease by 1 at each time point. The count value for the second visual stimulus VS2 can have a minimum value of 1 at time point 0.44 s.

[0158] Therefore, the second visual feedback VF2-1 at time 0.44 s can have a first shape S1.

[0159] The classification result at time point 0.48 s is the first visual stimulus VS1, which is different from the second visual stimulus VS2. The second visual stimulus VS2 is the classification result at the previous time point 0.44 s.

[0160] Therefore, device 100 can determine the count value of the second visual stimulus VS2 at time point 0.48 s as 2 by adding 1 to 1, where 1 is the count value of the second visual stimulus VS2 at time point 0.44 s.

[0161] In addition, device 100 can maintain the count value of the first visual stimulus VS1 at time point 0.44 s as 11, and can also set the count value of the first visual stimulus VS1 at time point 0.48 s as 11.

[0162] Although the classification result at time 0.48 s is the first visual stimulus VS1, the count value of the first visual stimulus VS1 is not immediately decremented by 1 to ensure the stability of the classification result. To prevent misclassification due to fluctuations, the device 100 for presenting visual feedback can immediately compare the classification result with the previous classification result and decrement by 1 only if the classification results are the same.

[0163] For example, the classification result of the immediate preceding time point 0.44 s at time point 0.48 s is the second visual stimulus VS2, and the device 100 may not subtract 1 from the count value of the first visual stimulus VS1 at time point 0.48 s.

[0164] Subsequently, as the user continuously fixates on the first visual stimulus VS1 from 0.52 s to 5 s, device 100 continuously subtracts 1 from the count value for the first visual stimulus VS1 at each time point until a minimum value is reached. Device 100 may also continuously increment the count value for the second visual stimulus VS2 by 1 at each time point until a maximum value is reached.

[0165] From the initial time point 0 s to the current time point 5 s, the maximum count value of 11, i.e. the initial value, can also be maintained for the unclassified third visual stimulus VS3.

[0166] Device 100 can reflect the count values ​​of the first visual stimulus VS1, the second visual stimulus VS2, and the third visual stimulus VS3 to the first visual feedback VF1, the second visual feedback VF2, and the third visual feedback VF3, respectively.

[0167] exist Figure 6 In the initial time point and up to time point 0.48 s, the first visual feedback VF1 can have the same second shape S2.

[0168] The first visual feedback VF1-1 at time 5 s, which is the current time point, can have a first shape S1. The shape of the first visual feedback VF1 can gradually change at each time point starting from time point 0.48 s. The shape of the first visual feedback VF1 can change until the count value of the first visual stimulus VS1 reaches a minimum value.

[0169] The second visual feedback VF2-3, at the current time point of 5 s, can have a second shape S2. The shape of the second visual feedback VF2 can be the second shape S2 at the initial time point of 0 s, and can gradually change to the first shape S1 until the time point of 0.44 s.

[0170] Subsequently, the shape of the second visual feedback VF2-2 at time point 0.48 s can be closer to the shape of the second shape S2 than the first shape S1 of the second visual feedback VF2-1 at time point 0.44 s.

[0171] Starting from time 0.48 s, the shape of the second visual feedback VF2 can change at each time point until the count value reaches its maximum value.

[0172] The shape of the third visual feedback VF3 can maintain the second shape S2 from the initial time point 0 s to the current time point 5 s.

[0173] Figure 7 This is a diagram illustrating the computing device according to an embodiment.

[0174] refer to Figure 7 The method and apparatus for presenting visual feedback according to the embodiment can be implemented using the computing device 900.

[0175] The computing device 900 may include at least one of a processor 910, a memory 930, a user interface input device 940, a user interface output device 950, and a storage device 960, which communicate via a bus 920. The computing device 900 may also include a network interface 970 electrically connected to a network 90. ​​The network interface 970 can send or receive signals with other entities via the network 90.

[0176] Processor 910 can be implemented in various types, such as a microcontroller unit (MCU), application processor (AP), central processing unit (CPU), graphics processing unit (GPU), neural processing unit (NPU), etc., and can be any type of semiconductor device capable of executing instructions stored in memory 930 or storage device 960. Processor 910 can be configured to implement the above-mentioned... Figures 1 to 6 The functions and methods described.

[0177] The memory 930 and storage device 960 may include various types of volatile or non-volatile storage media. For example, the memory may include read-only memory (ROM) 931 and random access memory (RAM) 932. In this embodiment, the memory 930 may be located inside or outside the processor 910, and the memory 930 may be connected to the processor 910 by various known means.

[0178] In some embodiments, at least some configurations or functions of the apparatus and method for presenting visual feedback according to the embodiments may be implemented as programs or software executable by the computing device 900, and the programs or software may be stored in a computer-readable medium.

[0179] In some embodiments, at least some configurations or functions of the device and method for presenting visual feedback according to the embodiments may be implemented using the hardware or circuitry of the computing device 900, or may be implemented as separate hardware or circuitry that can be electrically connected to the computing device 900.

[0180] Although this disclosure has been described in conjunction with embodiments now regarded as actual practices, it should be understood that this disclosure is not limited to the disclosed embodiments, but rather is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

[0181] <Description of Symbols>

[0182] 100: Devices used to present visual feedback

[0183] 110: Signal Receiver

[0184] 120: Signal Processor

[0185] 130: Visual Feedback Reflector

[0186] VS: Visual stimulation

[0187] VF: Visual feedback.

Claims

1. A method for presenting visual feedback, the method comprising: Steady-state visual evoked potential signals are extracted from the electroencephalogram analysis of users who fixate on visual stimuli of a specific frequency. Based on the steady-state visual evoked potential signal, the visual stimulus is classified to generate a classification result; A visual feedback with the same frequency as the visual stimulus is provided on the visual stimulus; and The classification results are reflected in the visual feedback in real time.

2. The method according to claim 1, wherein, Reflecting the classification results in real time in the visual feedback includes: The shape of the first visual feedback set in the first visual stimulus classified from multiple visual stimuli according to the classification result is changed to a first shape, and the visual feedback is set in each of the multiple visual stimuli; and The shape of the second visual feedback set in the unclassified second visual stimulus is changed to a second shape.

3. The method according to claim 1, wherein, Reflecting the classification results in real time in the visual feedback includes: The classification results are collected periodically at preset time points within the maximum window length; Based on the classification results collected at each of the preset time points, a count value for the visual stimulus is calculated; and Based on the count value, the shape of the visual feedback is changed in real time.

4. The method according to claim 3, wherein, Collecting the classification results includes: when the length from the time point when the visual stimulus first flashes to the current time point is greater than the maximum window length, collecting as many classification results as the maximum window length based on the current time point.

5. The method according to claim 3, wherein: The count value has a maximum value and a minimum value; and Calculating the count value for the visual stimulus includes setting the count value at the initial time point when the visual stimulus is presented to an arbitrary maximum value.

6. The method according to claim 5, wherein, Calculating the count value for the visual stimulus further includes: when the visual stimulus is first presented, initializing the count value of the visual stimulus at the initial time point to the maximum value.

7. The method according to claim 3, wherein, Calculating the count value for the visual stimulus includes: When the classification results at consecutive first and second time points are compared and found to be identical to the first visual stimulus, Regarding the first visual stimulus being categorized, the count value at the second time point is determined as the value obtained by subtracting a specific value from the count value at the first time point, and Regarding the unclassified second visual stimulus, the count value at the second time point is determined as a value obtained by adding the specific value to the count value at the first time point.

8. The method according to claim 7, wherein, Calculating the count value for the visual stimulus also includes: When the first visual stimulus is classified at the first time point, and the second visual stimulus is classified at the second time point, Regarding the second visual stimulus, the count value at the second time point is maintained at the same value as the count value at the first time point, and Regarding the first visual stimulus, the count value at the second time point is determined as a value obtained by adding the specific value to the count value at the first time point.

9. The method according to claim 8, wherein: The value obtained by adding the specific value to the count value is less than or equal to the maximum value of the count value; and The value obtained by subtracting the specific value from the count value is greater than or equal to the minimum value of the count value.

10. The method according to claim 5, wherein: The shape of the visual feedback has a first shape when the count value is the maximum value, and a second shape when the count value is the minimum value; and Changing the shape of the visual feedback in real time includes: in response to the count value, gradually changing the shape of the visual feedback between the first shape and the second shape at each of the preset time points.

11. A device for presenting visual feedback, the device comprising: The signal receiver is configured to receive steady-state visual evoked potential signals extracted from electroencephalogram analysis of a user who fixates on visual stimuli of a specific frequency. A signal processor is configured to classify the visual stimuli and generate classification results based on the steady-state visual evoked potential signals; as well as The visual feedback reflector is configured as follows: A visual feedback with the same frequency as the visual stimulus is provided on the visual stimulus, and The classification results are reflected in the visual feedback in real time.

12. The device according to claim 11, wherein, The visual feedback reflector is configured as follows: The shape of the first visual feedback set in the first visual stimulus classified from multiple visual stimuli according to the classification result is changed to a first shape, and the visual feedback is set in each of the multiple visual stimuli; and The shape of the second visual feedback set in the unclassified second visual stimulus is changed to a second shape.

13. The device according to claim 11, wherein, The visual feedback reflector is configured as follows: The classification results are collected periodically at preset time points within the maximum window length; Based on the classification results collected at each of the preset time points, a count value for the visual stimulus is calculated; and Based on the count value, the shape of the visual feedback is changed in real time.

14. The device according to claim 13, wherein, The visual feedback reflector is configured to collect as many classification results as the maximum window length based on the current time point when the length from the time point of the first flash of the visual stimulus to the current time point is greater than the maximum window length.

15. The device according to claim 13, wherein: The count value has a maximum value and a minimum value; and The visual feedback reflector is configured to set the count value at the initial time point when the visual stimulus is presented to an arbitrary maximum value.

16. The device according to claim 15, wherein, The visual feedback reflector is configured to initialize the count value of the visual stimulus at the initial time point to the maximum value when the visual stimulus is first presented.

17. The device according to claim 13, wherein, The visual feedback reflector is configured as follows: When the classification results at consecutive first and second time points are compared and found to be identical to the first visual stimulus, Regarding the first visual stimulus being categorized, the count value at the second time point is determined as the value obtained by subtracting a specific value from the count value at the first time point, and Regarding the unclassified second visual stimulus, the count value at the second time point is determined as a value obtained by adding the specific value to the count value at the first time point.

18. The device according to claim 17, wherein, The visual feedback reflector is configured as follows: A difference arises when the classification result at the first time point is the first visual stimulus and the classification result at the second time point is the second visual stimulus. Regarding the second visual stimulus, the count value at the second time point is maintained at the same value as the count value at the first time point, and Regarding the first visual stimulus, the count value at the second time point is determined as a value obtained by adding the specific value to the count value at the first time point.

19. The apparatus according to claim 18, wherein: The value obtained by adding the specific value to the count value is less than or equal to the maximum value of the count value; and The value obtained by subtracting the specific value from the count value is greater than or equal to the minimum value of the count value.

20. The apparatus according to claim 15, wherein: The shape of the visual feedback has a first shape when the count value is the maximum value, and a second shape when the count value is the minimum value; and The visual feedback reflector is configured to, in response to a count value, gradually change the shape of the visual feedback between the first shape and the second shape at each of the preset time points.

Images