Brain computer interface

Abstract

The invention relates to a brain computer interface. A system and method involving a brain computer interface in which a visual stimulus overlying one or more subjects is provided, at least a portion of the visual stimulus having a characteristic modulation. A brain computer interface measures a neural response to a subject viewed by a user. Neural response to visual stimulation is related to modulation, and when attention is focused on visual stimulation, correlation is stronger. The visual stimulus includes a feedback element that varies according to a measurement of attention of the or each overlying object.

Description

[0001] This application is a divisional application of a Chinese patent application filed on June 20, 2022, with application number 202080088687.2 and invention title "Brain-Computer Interface". The international filing date of the parent application is November 6, 2020, and the international application number is PCT / EP2020 / 081338.

[0002] Cross-reference to related applications

[0003] This application claims the benefit of priority of U.S. Provisional Patent Application No. 62 / 949,803, entitled "Brain-Computer Interface", filed on December 18, 2019, the entire contents of which are incorporated herein by reference. Technical field

[0004] Embodiments of the present disclosure relate to the operation of a brain-computer interface including visual sensing, and particularly to feedback interaction through such an interface. Background art

[0005] In a visual brain-computer interface (BCI), neural responses to a target stimulus among multiple generated visual stimuli presented to a user are typically used to infer (or "decode") which stimulus is substantially the object of attention at any given time. Then, the object of attention can be associated with an action that the user can select or control.

[0006] Neural responses can be obtained using a variety of known techniques. A convenient method relies on surface electroencephalography (EEG), which is non-invasive, has fine-grained temporal resolution, and is based on a well-known empirical basis. Surface EEG enables real-time measurement of changes in the spread potential on the surface of the subject's skull (i.e., the scalp). These potential changes are commonly referred to as electroencephalogram signals or EEG signals.

[0007] In a typical BCI, visual stimuli are presented in a display generated by a display device. Examples of suitable display devices (some of which are Figure 9shown) include display screens such as a television screen and a computer monitor 902, a projector 910, a virtual reality headset 906, an interactive whiteboard, and a tablet 904, a smart phone, smart glasses 908, etc. Visual stimuli 911, 911', 912, 912', 914, 914' can form part of a generated graphical user interface (GUI), or they can be presented as augmented reality (AR) or mixed reality graphical objects 916, 918 overlaid on a base image: the base image can simply be the user's actual field of view (as in the case of a mixed reality display function projected onto a transparent display in a pair of smart glasses in other cases) or a digital image corresponding to the user's field of view but captured in real time by an optical capture device (the optical capture device can in turn capture images corresponding to the user's field of view in other possible views).

[0008] It is difficult to infer which of multiple visual stimuli (if any) is the object of attention at any given time. For example, when a user is faced with multiple stimuli (e.g., numbers on a keyboard displayed on a screen), it has been shown to be nearly impossible to directly infer which stimulus is being attended to at a given time from brain activity. The user perceives the number being attended to (say the number 5), so the brain must contain information that distinguishes that number from the others, but current methods are unable to extract that information. That is, current methods can more difficultly infer that a stimulus has been perceived, but these methods cannot use only brain activity to determine which specific stimulus is being attended to.

[0009] To overcome this problem and to provide sufficient contrast between stimuli and the background (and between stimuli), it is known to configure the stimuli used by a visual BCI to blink or pulse (e.g., a large surface of pixels switches from black to white and from white to black), such that each stimulus has a distinguishable characteristic distribution over time. Blinking stimuli elicit a measurable electrical response. Specific techniques monitor different electrical responses, such as steady-state visual evoked potentials (SSVEP) and P-300 event-related potentials. In a typical implementation, the stimuli blink at a rate of more than 6 Hz. Thus, such a visual BCI relies on a method that includes discretely rather than continuously, and typically at different time points, displaying various stimuli. Brain activity associated with attention to a given stimulus is found to correspond (i.e., be correlated) with one or more aspects of the temporal distribution of that stimulus (e.g., the frequency at which the stimulus blinks and / or the duty cycle at which the stimulus alternates between a blinking state and a stationary state).

[0010] Thus, the decoding of neural signals relies on the fact that when a stimulus is turned on, the stimulus will trigger a characteristic pattern of neural responses in the brain, which can be determined from the electrical signals, namely SSVEP or P-300 potentials, picked up by the electrodes of an EEG device (such as the electrodes of an EEG headset). This pattern of neural data may be very similar or even identical for different digits, but it is time-locked to the perceived digit: only one digit can generate a pulse at any given time, such that the correlation between the time of digit-generated pulse and the pulsed neural response can be determined as an indication that the digit is the object of interest.

[0011] By presenting each digit at different time points, turning the digit on and off at different rates, applying different duty cycles, and / or simply applying stimuli at different time points, the BCI algorithm can establish which stimulus is most likely to trigger a given neural response when turned on, enabling the system to determine the object of interest.

[0012] In recent years, visual BCIs have improved significantly, making the real-time and accurate decoding of a user's attention increasingly practical. However, the continuous flickering of stimuli, sometimes flickering across the entire screen when there are many stimuli, is an inherent limitation to the large-scale use of this technology. In fact, this can cause discomfort and mental fatigue, and if continued, can also cause physiological responses such as headaches. Additionally, the flickering effect can hinder the user's ability to focus on a specific target, as well as the system's ability to quickly and accurately determine the object of interest.

[0013] For example, when a user of the on-screen keyboard discussed above attempts to focus on the digit 5, the other (i.e., peripheral) digits act as distractors, and the presence of these digits and the fact that these digits exhibit a flickering effect temporarily attract the user's attention. The display of peripheral digits interferes with the user's visual system. This interference in turn can hinder the performance of the BCI.

[0014] Therefore, there is a need for improved methods for differentiating screen targets and their displayed stimuli with speed and accuracy to determine which one the user is focusing on, and for discriminating the object of interest (target) from the objects (distractors) peripheral to the target with speed and accuracy.

[0015] Accordingly, it is desirable to provide a brain-computer interface that addresses the above challenges. SUMMARY OF THE INVENTION

[0016] The present disclosure relates to a brain-computer interface in which visual stimuli are presented on a graphical interface such that the visual stimuli are neurally decodable and provide an improved user experience.

[0017] The present disclosure also relates to a brain-computer interface (BCI) in which a visual stimulus overlaying one or more objects includes a corresponding feedback element that varies according to a measure of attention to the or each object. The visual stimulus is generated by a stimulus generator and is typically presented on a screen or other display device.

[0018] At least a portion of the visual stimulus has characteristic modulation. The neural response to the object in the user's field of view is captured by a neural signal capture device in the BCI. The user's neural response to the viewed object can then be measured and decoded to determine which object of interest is the focus of the user's attention and the user's current level of attention to the object, with the neural response being stronger when the attention is focused on the visual stimulus.

[0019] The change of the feedback element is set to be associated with the intensity of the neural response. Therefore, the feedback element of the visual stimulus can change the visual form (so that the user sees the effect on the feedback element corresponding to its attention level). In addition, the user is provided with a target for this attention, and the user can adjust the user's behavior to enhance the visual effect (effectively learning how to operate the BCI more efficiently). In other words, the user sees the effect on the feedback element that is being caused by the BCI, and can learn to use the BCI to concentrate by seeking to observe the effect change. In addition, the appearance of the visual effect associated with the concentration of attention is used to verify the selection of the underlying object.

[0020] In some embodiments, the feedback element represents the degree of attention from lack of attention to a level of focused attention as a gradual step-wise or continuous change between a disordered (e.g., pseudo-random) distribution of visual elements to a fully ordered distribution (e.g., to a recognizable shape, character, or symbol, such as a reticle, target mark, or crosshairs).

[0021] In some embodiments, the entire visual stimulus is a feedback element. In other embodiments, the visual stimulus includes a background element in addition to the feedback element. In some embodiments, the background element has a characteristic temporal modulation (i.e., a decodable modulation) of the visual stimulus, while the feedback element is not modulated. In some embodiments, the feedback element has a characteristic modulation of the visual stimulus, while the background element is not modulated.

[0022] In some embodiments, a characteristic modulation of the visual stimulus is applied to both the background element and the feedback element. The amplitude of the modulation in the background element and the feedback element may be different.

[0023] In some aspects, the present disclosure describes a system and method for improving the accuracy and speed of determining an object of interest in a field of objects or as a specific region within a single large object. Image data of all objects is processed to extract a version of each object that consists only of high spatial frequency (HSF) components.

[0024] The present disclosure relates to techniques for: obtaining (potential) objects of interest within a user's field of view (generally, but not always on a display presented to the user); extracting components related to the visual properties of these objects (such as their edges); and applying modulation to the high spatial frequency components of these visual properties. Thus, a flickering visual stimulus for eliciting a neural response such as a visual evoked potential (VEP) can be conveyed only through the HSF version of the object. The modulation causes the object to flicker or otherwise change visually, thereby modulating to act as a stimulus for the relevant neural response. The neural response can then be measured and decoded to determine which object of interest is the focus of the user's attention.

[0025] In some aspects, the image data can be further processed to extract another version of the object that consists only of low spatial frequency (LSF) components. In the case where the LSF version is extracted, the modulated HSF version can be superimposed on the LSF version (which does not flicker).

[0026] In one aspect, the present disclosure includes a closed-loop feedback system where the user gazes at the screen and its objects, a neural activity is captured as a signal using an electrode helmet, and the HSF ratio detected from the neural activity and associated with each object will change as the user's object of interest changes. This is somewhat equivalent to flickering the objects at different rates and duty cycles, but due to filtering such that the flickering displayed objects are substantially the objects that elicit an HSF response (such as the HSF version), much less interference is presented. If the object is peripheral, the flickering of its HSF version will naturally be suppressed by human visual behavior. However, the object of interest whose HSF version flickers will elicit a neural response that is easily recognizable. Thus, the interference is significantly eliminated, making the experience more comfortable and the recognition of the object of interest more accurate and timely.

[0027] In each of the above embodiments, the modulation can be preferentially or exclusively applied to the high spatial frequency components of the projected overlay image (i.e., background elements and / or feedback elements).

[0028] According to another aspect, the present disclosure relates to a brain-computer interface system, the brain-computer interface system comprising: a display unit for displaying image data including at least one object, the display unit further outputting a corresponding visual stimulus corresponding to one or more of the objects; a stimulus generator for generating the visual stimulus or each visual stimulus with a corresponding characteristic modulation; a neural signal capture device configured to capture neural signals associated with a user; and an interface device operably coupled to the neural signal capture device and the stimulus generator, the interface device being configured to: receive neural signals from the neural signal capture device; determine the intensity of a component of the neural signals having a property associated with the corresponding characteristic modulation of the visual stimulus or each visual stimulus; determine, based on the neural signals, which of the at least one visual stimulus is associated with an object of interest of the user, the object of interest being inferred from the presence and / or relative intensity of a component of the neural signals having a property associated with the characteristic modulation of the visual stimulus; and cause the stimulus generator to generate a visual stimulus for the object of interest having a feedback element, the feedback element being displayed with an effect that varies according to the determined intensity of the component of the neural signals having a property associated with the characteristic modulation of the visual stimulus for the object of interest.

[0029] According to another aspect, the present disclosure relates to a method of operating a brain-computer interface system, the brain-computer interface system comprising: a display unit, a stimulus generator, and a neural signal capture device, the display unit displaying image data including at least one object and outputting a visual stimulus corresponding to one or more of the objects, the visual stimulus having a characteristic modulation, wherein the method comprises, in a hardware interface device operably coupled to the neural signal capture device and the stimulus generator: receiving neural signals from the neural signal capture device; determining the intensity of a component of the neural signals having a property associated with the corresponding characteristic modulation of the visual stimulus or each visual stimulus; determining, based on the neural signals, which of the at least one visual stimulus is associated with an object of interest of the user, the object of interest being inferred from the presence and / or relative intensity of a component of the neural signals having a property associated with the characteristic modulation of the visual stimulus; and causing the stimulus generator to generate a visual stimulus for the object of interest having a feedback element, the feedback element being displayed with an effect that varies according to the determined intensity of the component of the neural signals having a property associated with the characteristic modulation of the visual stimulus for the object of interest. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] To facilitate identification of the discussion of any particular element or action, one or more of the most significant digits in the reference numerals refer to the figure number in which the element was first introduced.

[0031] Figure 1 An electronic architecture for receiving and processing EEG signals according to the present disclosure is shown;

[0032] Figure 2 illustrates a system incorporating a brain - computer interface (BCI) according to the present disclosure;

[0033] Figures 3A to 3C illustrates a display of a target object having time characteristics with respective different variations according to the present disclosure;

[0034] Figure 4 illustrates an exemplary embodiment in which a BCI system of the present disclosure presents visual feedback;

[0035] Figure 5 illustrates an exemplary embodiment in which a BCI system of the present disclosure presents dynamic visual feedback;

[0036] Figure 6 illustrates another exemplary embodiment in which a BCI system of the present disclosure presents dynamic visual feedback; and

[0037] Figure 7 illustrates main functional blocks in a method of operation of a BCI according to the present disclosure.

[0038] Figure 8A and Figure 8B illustrates an exemplary arrangement in which an HSF version of a screen object or an overlay object is modulated;

[0039] Figure 9 illustrates various examples of display devices suitable for use with a BCI system of the present disclosure;

[0040] Figure 10 is a block diagram showing a software architecture in which the present disclosure may be implemented according to some example embodiments;

[0041] Figure 11 is a graphical representation of a machine in the form of a computer system according to some example embodiments, in which a set of instructions may be executed to cause the machine to perform any one or more of the methods discussed. Detailed Description

[0042] The following description includes systems, methods, techniques, instruction sequences, and computer - program products for implementing illustrative embodiments of the present disclosure. In the following description, for purposes of explanation, numerous specific details are set forth to provide an understanding of the various embodiments of the inventive subject matter. However, it will be apparent to those skilled in the art that embodiments of the inventive subject matter may be practiced without these specific details. In general, well - known instruction instances, protocols, structures, and techniques are not shown in detail.

[0043] Figure 1An example of an electronic architecture for receiving and processing EEG signals by means of an EEG device 100 according to the present disclosure is shown.

[0044] To measure the spread potential on the surface of the skull of a subject 110, the EEG device 100 includes a portable device 102 (i.e., a cap or headset), an analog-to-digital conversion (ADC) circuitry 104, and a microcontroller 106. Figure 1 The portable device 102 includes one or more electrodes 108, typically between 1 and 128 electrodes, advantageously between 2 and 64 electrodes, advantageously between 4 and 16 electrodes.

[0045] Each electrode 108 may include a sensor for detecting an electrical signal generated by the neuronal activity of the subject and an electronic circuit for preprocessing (e.g., filtering and / or amplifying) the detected signal before analog-to-digital conversion: such an electrode is referred to as "active". Figure 1 An active electrode 108 in use is shown, where the sensor is in physical contact with the subject's scalp. The electrodes may be suitable for use with a conductive gel or other conductive liquid (referred to as "wet" electrodes) or without such a liquid (i.e., "dry" electrodes).

[0046] Each ADC circuit 104 is configured to convert the signals of a given number (e.g., between 1 and 128) of active electrodes 108.

[0047] The ADC circuit 104 is controlled by the microcontroller 106 and communicates with the microcontroller 106, for example, through the protocol SPI ("Serial Peripheral Interface"). The microcontroller 106 packs the received data for transmission to an external processing unit (not shown), for example, via Bluetooth, Wi-Fi ("Wireless Fidelity") or Li-Fi ("Light Fidelity"), which external processing unit is, for example, a computer, a mobile phone, a virtual reality headset, an automotive or an aviation computer system such as an automotive computer or a computer system, an aircraft.

[0048] In some embodiments, each active electrode 108 is powered by a battery ( Figure 1 not shown in ). The battery is conveniently arranged in the housing of the portable device 102.

[0049] In some embodiments, each active electrode 108 measures a corresponding potential value, from which the potential measured by a reference electrode is subtracted (Ei = Vi - Vref), and the difference is digitized by means of the ADC circuit 104 and then sent by the microcontroller 106.

[0050] In some embodiments, the methods of the present disclosure introduce target objects for display in a graphical user interface of a display device. The target objects include controls, and the controls are in turn associated with user-selectable actions.

[0051] Figure 2 Shown is a system incorporating a brain-computer interface (BCI) in accordance with the present disclosure. The system incorporates, for example, Figure 1 the neural response device 206 of the EEG device 100 as shown in. In the system, an image is displayed on the display of the display device 202. The subject 204 views the image on the display and focuses on the target object 210.

[0052] In an embodiment, the display device 202 displays at least the target object 210 as a graphical object having a varying temporal characteristic that is different from the temporal characteristics of the background and / or other display objects in the display. The varying temporal characteristic can be, for example, a persistent or time-locked blinking effect that changes the appearance of the target object at a rate greater than 6 Hz. In another embodiment, the varying temporal characteristic can use a pseudo-random time code to generate a blinking effect that changes the appearance of the target object several times per second on average, for example, at an average rate of 3 Hz. In the case where more than one graphical object is a potential target object (i.e., in a situation where a choice of the target object to focus on is provided to the viewing subject), each object is associated with a discrete spatial and / or temporal code. Figures 3A to 3C Shown is the display of target objects having respective different varying temporal characteristics.

[0053] The neural response device 206 detects the neural response associated with the attention focused on the target object (i.e., the tiny electrical potential indicating brain activity in the visual cortex); thus, the visual perception of the varying temporal characteristic of the target object serves as a stimulus in the subject's brain, generating a specific brain response consistent with the code associated with the target object of interest. The detected neural response (e.g., electrical potential) is then converted into a digital signal and transmitted to the processing device 208 for decoding. Examples of neural responses include visual evoked potentials (VEPs), which are commonly used in neuroscience research. The term VEP encompasses: the traditional SSVEP as mentioned above, where the stimulus oscillates at a specific frequency; and other methods such as coded-modulated VEPs, where the stimulus undergoes variable or pseudo-random time coding. The sympathetic response where the brain appears to "oscillate" or respond synchronously with the blinking temporal characteristic is referred to herein as "neural synchronization".

[0054] The processing device 208 executes instructions to interpret the received neural signals to determine in real time feedback indicating a target object having a current (visual) attention focus. Decoding the information in the neural response signals relies on a correspondence between the information and one or more aspects of the temporal distribution of the target object (i.e., the stimulus). In some embodiments, the processing device 208 and the neural response device 206 may be provided in a single device such that a decoding algorithm is directly executed on the detected neural response. Thus, a BCI utilizing visually associated neural signals can be used to determine which objects on the screen the user is attending to.

[0055] In some embodiments, the processing device can conveniently generate image data for presentation on the display device 202 that includes a target object that varies over time.

[0056] In some embodiments, the display device 202 displays an overlay object as a graphical object that has varying temporal characteristics different from the background and / or other display objects in the display, and then displays the overlay object as a graphical layer over at least the identified target object.

[0057] Figure 4 A user experience of the display device 410 is shown, where the display device 410 displays an overlay object 406 having different varying temporal characteristics over a target object 404 (which has been determined to be the object having the current attention focus). This can provide retrospective feedback to the user to validate the user's selection. As Figure 4 shown, visual feedback can be conveniently presented to the user on the display device 410 such that the user is aware that the target object 404 has been determined to be the current attention focus. For example, the display device can display an icon, cursor, or other graphical object or effect (in Figure 4 a crosshair 411) very close to the target object 404, highlighting (e.g., overlaying) the object that appears to be the focus of the current visual attention. This provides a positive feedback loop (where the apparent target object is confirmed (i.e., validated) as the intended target object due to extended magnified attention).

[0058] In some embodiments, the visual feedback is "prospective" in that the visual feedback is actively driven to change its appearance. For Figure 4 retrospective feedback, the neural response to the objects in the user's field of view in the case of prospective feedback is captured by the neural signal capture device in the BCI. The neural response of the user to the viewed object can then be measured and decoded to determine which object of interest is the focus of the user's attention and the user's current level of attention to that object, with a stronger neural response when attention is focused on a visual stimulus.

[0059] Figure 5Shows the stages of appearance changes of the visual feedback effect. In this exemplary embodiment, the target object 504 itself is displayed as a graphical object that has changing temporal characteristics different from the temporal characteristics of the background and / or other display objects 502, 506, 508 in the display. As previously discussed, the putative target object is determined based on the closest correlation between the visual stimulus and the decoded neural response. A measure of this correlation can be referred to as a "decoding score". Candidate target objects are presented in the display screen with dynamic visual feedback elements (e.g., icons, cursors, crosshairs, or other graphical objects): the dynamic visual feedback elements change according to the decoding score (e.g., move or change color, shape, size, or other visual appearance). Thus, as can be seen in Figure 5 view (a), dynamic feedback elements are presented for more than one candidate target object (or indeed for all objects).

[0060] The change of the feedback element is set to be associated with the intensity of the response. Thus, the feedback element of the visual stimulus can change its visual form (so that the user can see the effect on the feedback element corresponding to their attention level). Watching the changing appearance of the feedback element encourages the user to pay further attention. In addition, the target of this attention is provided to the user, and the user's behavior can be adjusted to enhance the visual effect (effectively learning how to operate the BCI more efficiently). In other words, the user sees the effect on the feedback element that they are causing via the BCI, and can learn to focus their attention by seeking to observe the change of the effect. For candidate target objects that are not the focus of attention, the displayed dynamic feedback elements will continue to appear in a substantially unchanged visual form.

[0061] In some embodiments, the feedback element represents the degree of attention from a lack of attention to a focused attention level as a progressive step - by - step or continuous change between a disordered (e.g., pseudo - random) distribution of visual elements to a fully ordered distribution (e.g., to a recognizable shape, character, or symbol, such as a fiducial, a target marker, or a crosshair). The feedback elements of candidate target objects that are not the focus of attention remain disordered.

[0062] In some embodiments, for example, in Figure 5In the exemplary embodiment showing prospective feedback, each of a plurality of objects in the user's field of view is set to exhibit a corresponding small feedback stimulus (e.g., three separate thin lines moving pseudo-randomly). When the user specifically focuses on one of the objects (e.g., the target object 504 such as the "i" icon), the three lines superimposed on the icon move towards each other according to the decoding score until they form a triangle. In this exemplary embodiment, completely pseudo-random lines (as shown at view (a)) mean that no decoding is performed at all (for any of the objects); as shown at view (b), partial decoding is performed for the target object; and the complete triangle as shown at view (c) means 100% decoding at the target object. It is observed that this whole process is fast, and an experienced user can develop their attention from a state of little or no decoding to a state of complete decoding in less than a second (e.g., from (a) to (b), and then to (c)). Clearly, such a visual display of feedback has a reflexive cognitive effect on the perception of the target object, amplifying the brain response.

[0063] In embodiments such as Figure 6 In certain alternative embodiments of the prospective feedback showing prospective feedback as shown, the visual feedback 606 represents the decoding score by reducing the transparency of the overlaid object, changing the line width of the overlaid object, etc. Thus, in view (a), the visual feedback 606 is substantially invisible; in view (b), the visual feedback 606 is represented by intersecting dashed lines, indicating partial decoding; and in view (c), the visual feedback 606 represented by thicker, intersecting solid lines indicates substantially complete decoding.

[0064] Thus, active feedback contrasts with known feedback systems where an effective selection event requires the user to focus on a particular object for more than a predetermined time period and the level of the neural response to exceed a predetermined threshold. Using active feedback (and feedback stimuli), neural feedback can be calculated and provided on a shorter time scale (e.g., a time scale on the order of seconds), providing information on intermediate steps with a certainty of the match between the neural response and the selected object ranging from 0% to 100%.

[0065] In some embodiments, the relationship between the feedback stimulus and the decoding performance is linear. In other embodiments, the relationship is not linear: examples of alternative, non-linear relationships utilize functions such as sigmoid, hyperbolic tangent, rectified linear unit (ReLU), etc. In some embodiments, the use of a non-linear relationship appears to improve the feedback relationship, especially in cases where the certainty level is low, otherwise the feedback would reflect random / arbitrary fluctuations in the EEG signal.

[0066] In some embodiments, the entire visual stimulus is a feedback element. In other embodiments, in addition to the feedback element, the visual stimulus further includes a background element. In some embodiments, the background element has a characteristic modulation of the visual stimulus while the feedback element is not modulated. In some embodiments, the feedback element has a characteristic modulation of the visual stimulus while the background element is not modulated.

[0067] In some embodiments, the characteristic modulation of the visual stimulus is applied to both the background element and the feedback element. The modulation amplitudes in the background element and the feedback element may be different.

[0068] In some embodiments, the operation of the BCI may include brief initialization and calibration phases. Since users may vary significantly in their baseline neural responses to the same stimuli (especially those users with visual cortex disorders or impairments), the calibration phase can be used to generate a user-specific stimulus reconstruction model. Such a phase may take less than a minute to construct (typically, about 30 seconds).

[0069] It has been found that objects of interest in the foveal visual region are associated with a high degree of HSF signal components. Similarly, it has been found that objects in the peripheral visual region are associated with a high degree of LSF signal components.

[0070] By increasing those differences between the HSF signal components and the LSF signal components via various filtering methods, the accuracy and speed of the BCI can be improved.

[0071] In another aspect of the present disclosure, the modulation may be preferentially or exclusively applied to the high spatial frequency components of the projected overlay image (i.e., the background element and / or the feedback element). Then, the method outlined above can be followed to determine the object of interest of the user.

[0072] Figure 3A The effect of peripheral vision is shown. The subject 305 is shown as viewing a display screen 302 that displays a plurality of digits 310, 312 in a keyboard. When the subject attempts to focus on the digit "5" 310 in the keyboard on the screen discussed above, the other (i.e., peripheral) digits (e.g., "3", 312) act as distractors, temporarily attracting the user's attention and interfering with the user's visual system. This interference in turn can hinder the performance of the BCI. Therefore, improved methods are needed for differentiating screen targets and their display stimuli with speed and accuracy to determine which one the user is focusing on and for discriminating the object of interest (target) from the objects (distractors) outside the target with speed and accuracy.

[0073] Typically, visual stimuli will occupy a large portion of the screen surface, filled with high-energy uniform light (bright white shapes) or a rough checkerboard. These large surfaces will remain dedicated to the visual BCI system and cannot be used for any other purpose other than visual stimulation. These large stimulation surfaces are inconsistent with the fine and discrete integration of the visual BCI system and limit the design freedom of the display device, such as Figure 9 the user interface in the display device shown.

[0074] Figure 3B shows the discrimination of multiple target objects using a neural response device, such as Figure 1 and Figure 2 the neural response device in. In Figure 3B , the neural response device worn by the user (i.e., the viewer) 305 is an electrode helmet for an EEG device. Here, the user wearing the helmet views a screen 302 that displays multiple target objects (numbers in a keyboard on the screen), and these target objects flash at significantly different times, frequencies, and duty cycles. The electrode helmet can transmit signals obtained from neural activity. Here, the user is focusing on the number 5, 310, where at time t1, the number 3, 312 flashes; at time t2, the number 4, 314 flashes; at time t3, the number 5, 310’ flashes; and at time t4, the number 6, 316 flashes. The neural activity transmitted by the helmet signal is significantly different at t3 from other time points. This is because the user is focusing on the number 5, 310, which flashes as 310’ at t3. However, in order to distinguish the signal that appears at t3 from the signals at other times, all the objects on the screen must flash at significantly different times. As a result, the screen becomes active with the flashing objects, leading to an uncomfortable viewing experience.

[0075] Figure 3B The system in can use a display signal pattern, such as Figure 3C the exemplary pattern shown, where the screen objects will flash at different time points with different frequencies and duty cycles.

[0076] One method for the challenge of determining a target from objects (distractors) peripheral to the object of interest (target) with speed and accuracy relies on the characteristics of the human visual system.

[0077] Studies of the way the human visual sensing operates have shown that when gazing at a screen with multiple objects and focusing on one of these objects, the human visual system will be receptive to both high spatial frequency (HSF) and low spatial frequency (LSF). Evidence shows that the human visual system is mainly sensitive to the HSF component of the specific display area being attended to (e.g., the object the user is gazing at): this corresponds to the central area of the subject's retina filled with cone cells, called the fovea. This can be in Figure 3AAs seen in the right view, the foveal region, which has the clearest vision among the displays 318, is contrasted with the peripheral region 304.

[0078] Conversely, for peripheral objects, the human visual system is mainly sensitive to their LSF components.

[0079] In existing neural capture systems, due to the operation of the human visual system, the picked-up neural signals will be substantially affected by both the HSF components from the target of interest and the LSF components from peripheral targets. However, since all objects induce a certain proportion of both HSF and LSF, processing the neural signals to determine the object of interest may be hampered by the LSF noise contributed by peripheral objects. This tends to make the identification of the object of interest less accurate and timely.

[0080] The basic science for this method is related to the differences in how the human eye-brain system processes stimuli from the object of interest and peripheral objects. This separation between foveal (center of the visual field) and peripheral vision has been described in the literature in terms of specific frequency channels from the retina to the visual cortex, where foveal vision is mainly driven by the HSF channels that convey visual details, while peripheral vision is mainly driven by the LSF channels that convey rough visual information such as the global shape of objects without details. These two types of information are associated with separate neural pathways, different functions, and different effects on unconscious and conscious perception.

[0081] Spatial frequency is usually calculated in cycles per degree. Spatial frequency mainly depends on three parameters: the dots per inch (dpi), also known as pixels per inch (ppi); the distance between the user's eyes and the monitor; and the cut-off frequency of the spatial filter. Spatial frequency filters can be used such that the stimulus signal retains only HSF characteristics, or conversely only LSF characteristics. The spatial frequency filters used in the context of visual BCI can conveniently perform high-pass filtering on values exceeding 7 cycles per degree, while performing low-pass filtering on values below 3 cycles per degree. In some cases, the lower threshold of the low-pass filter may cause a uniformly flat hue in the output (this "low-pass filter" is still a valid filter). In contrast, the maximum value of the high-pass filter threshold is limited by the resolution of the display system and ultimately by the visual physiological capabilities of the subject. In any case, regardless of the further thresholds of the low-pass and high-pass filters, the present disclosure operates independently of the specific values of the frequency filters and / or transforms. The main principle is to separate the spatial frequency components to optimize visual BCI.

[0082] The human visual system is tuned to process multiple stimuli at different locations in the visual field in parallel, often unconsciously or subconsciously. Thus, peripheral object stimuli will continue to trigger neural responses in the user's brain even when they appear in the periphery of the visual field. This, in turn, creates competition between multiple stimuli and makes the specific neural decoding of the attended object (the target) more difficult.

[0083] Consider again Figure 3B the on-screen keyboard of. The flickering peripheral signals 312, 314, and 316 will elicit LSF neural activity in the viewer, and these signals will be captured and processed in parallel with the signals that elicit HSF neural activity in the viewer stimulated by the flickering digit 5, 310. Thus, these peripheral objects can be considered distractors, and the LSF signals they elicit can be considered noise. One result of this noise is that the system takes longer to accurately determine the attended object.

[0084] In one method of the present disclosure, multiple objects are displayed in such a way that each object is divided into a version consisting only of the LSF components of the object and a version consisting only of the HSF components of the object. In one example, the flickering visual stimuli used to elicit a decodable neural response (e.g., SSVEP) are transmitted only through the HSF version of the object. This flickering HSF version is superimposed on the LSF version (which does not flicker). This method will be discussed in more depth below with respect to Figure 8A and Figure 8B be discussed in more depth.

[0085] In each of the feedback overlay arrangements above, the modulation can be applied preferentially or exclusively to the high spatial frequency components of the projected overlay image (i.e., the background elements and / or the feedback elements). Preferential modulation of the HSF components of the overlay object, target object, and / or visual feedback elements can be used to improve the accuracy of determining the attended object (and reduce the distracting effect).

[0086] The BCI described above can be used in conjunction with real-world objects to make the objects controllable or otherwise interactive. In certain embodiments, the generation of the stimuli is handled by one or more light sources (e.g., light-emitting diodes, LEDs) provided associated with (or even on the surface of) the controllable object.

[0087] In certain embodiments, the generation of the stimuli is handled by a projector or a scanning laser device such that the visual stimuli are projected onto the controllable object, and the controllable object outputs visual stimuli by reflecting the projected stimuli.

[0088] As in the case of a BCI using a display screen with which a user interacts with an object on the screen, controllable objects in the present disclosure can be made to exhibit visual stimuli (e.g., flickering stimuli) with characteristic modulation such that the neural response to the presence of these stimuli becomes apparent and can be decoded from neural signals captured by a neural signal capture device (e.g., an EEG device).

[0089] In certain embodiments, determination of the focus of attention on a visual display of a controllable device is used to send a command to the controllable object. The controllable object can then effect an action based on the command: for example, the controllable object can emit an audible sound, unlock a door, turn on or off, change an operating state, etc. The action can also provide visual or other feedback associated with the controllable object: this can be used in conjunction with the positive feedback loop discussed above, but can also provide a real-time indication of the available options for the operation associated with the controllable object.

[0090] Figure 7 Illustrated are the main functional blocks in a method of operation of a BCI system (e.g., the BCI system shown in Figure 2 The brain-computer interface system includes a display unit, a stimulus generator, and a neural signal capture device. The display unit displays image data including at least one object and outputs a visual stimulus corresponding to one or more of the objects, the visual stimulus having characteristic modulation.

[0091] In block 702, a hardware interface device, such as an interface device, operably coupled to the neural signal capture device and the stimulus generator, receives neural signals from the neural signal capture device.

[0092] In block 704, the interface device determines the intensity of the component of the neural signal having a property associated with the corresponding characteristic modulation of the visual stimulus or each visual stimulus.

[0093] In block 706, the interface device determines, based on the neural signals, which visual stimulus among at least one visual stimulus is associated with the user's object of attention, the object of attention being inferred from the presence and / or relative intensity of the component of the neural signal having a property associated with the characteristic modulation of the visual stimulus.

[0094] In block 708, the interface device causes the stimulus generator to generate a visual stimulus for the object of attention having a feedback element that is displayed with an effect that varies according to the intensity of the determined component of the neural signal having a property associated with the characteristic modulation of the visual stimulus for the object of attention.

[0095] The active feedback of the present disclosure is associated with several benefits in terms of user experience (UX) and neural decoding. The feedback stimulus presents a convenient guidance for the user's attention (i.e., "attention catcher") at a specific location on the display screen, helping the user to maintain focus on the object. For viewers with certain attention-related conditions and mild visual impairments, the presence of this feature has been observed to help the user stay focused.

[0096] In addition, the user is given a task (i.e., to bring the feedback stimulus closer to the fully decoded state, indicating "verification" of the selection). This also helps the user to focus on a specific object while suppressing peripheral distractors.

[0097] As the user becomes more focused using this feedback stimulus, it is observed that the user-specific stimulus reconstruction model established during the initial or calibration phase of BCI operation becomes more accurate and is constructed faster.

[0098] In subsequent operation phases, using the feedback stimulus as described above results in improved accuracy and increased speed for real-time BCI applications.

[0099] Figure 8A is an exemplary illustration of an embodiment of the disclosed subject matter. Here, all target objects 801 are filtered to create their own HSF versions and LSF versions.

[0100] In Figure 8A 's embodiment, the target object 801 is different graphic elements within a graphical interface presented on the display of the display device. Examples of target objects include glyphs, captured images, and user interface elements. The display device may include a processing unit (not shown), a modulator subsystem 805, and a display driver subsystem 806.

[0101] For each target object 801, the processing unit of the display device operates to apply a spatial frequency filter 802 (or spatial frequency transformation) to generate the HSF version and the LSF version of the target object 801 (denoted as 803 and 804 respectively). In other embodiments, the target object 801 may be filtered to create only the HSF version of each object. Thus, only the HSF version is generated.

[0102] The modulator subsystem 805 processes the HSF version 803 and the LSF version 804 of each object and transmits them to the display driver subsystem 806. The HSF version 803 and the LSF version 804 of each object are processed differently: the LSF version signal 804 is encoded to produce a static (e.g., non - flickering) display, while the HSF version signal 803 is time - modulated (e.g., made to flicker at optionally different frequencies and / or duty - cycle characteristics at different times). The time - modulation effect is typically a periodic change in the visual characteristics of the object such as brightness, hue, color components, etc., and may be abrupt (switching between on and off states) or may include smoother transitions. Examples of time - modulation include a flickering effect where the brightness characteristics of a group of pixels in a target object switch between two distinguishable brightness levels. Conveniently, the modulation effect encodes the optical characteristics in a detectable and decodable manner for the user's brain to respond to it (when the object is viewed). This processing may also include increasing the contrast level in the HSF version signal and / or decreasing the contrast level in the LSF version signal.

[0103] In some embodiments, the modulation effect can be random or pseudo - random in nature: changes in the visual characteristics of an object such as flickering are performed at random times, e.g., following a pseudo - random time pattern. Conveniently, a pseudo - random time pattern is used instead of a strictly random pattern to reduce the temporal overlap between different time patterns associated with different objects.

[0104] The display driver subsystem 806 receives the processed HSF version and LSF version of each object from the modulator subsystem 805 (either as separate signals or as a superimposed signal combining the modulated HSF version and LSF version). Thus, the signals driving the display include screen objects where the LSF - related signal produces a constant display and the HSF - related signal produces a flickering effect (or other time - modulation effect).

[0105] In some embodiments, as described above, the modulator subsystem 805 can process only the HSF version of each object and transmit it to the display driver subsystem 806: such that only the HSF version is received at the display driver subsystem 806. In this case, the modulator subsystem 805 applies a high-pass filter to generate only the HSF version of each object. In some other embodiments, the modulator subsystem can apply one or more spatial frequency filters (or spatial frequency transforms) to generate both the HSF version and the LSF version of the target object, but transmit only the HSF version of each object to the display driver subsystem 806. In each case, the HSF version is temporally modulated as described above, such that the display driver subsystem drives the display with a signal including screen objects having temporally modulated HSF components. In another embodiment, in addition to the (entire) target object 801, the display driver subsystem 806 also displays the generated HSF version. In this case, the contrast in the modulated HSF version can be increased to enhance the visibility of the visual stimuli over the attended object.

[0106] Then, the user's attention to the displayed (HSF-modulated) object can be detected (by capture 822 and decoding 807 of the neural response). Accordingly, the temporal modulation effect of the peripherally viewed objects is significantly reduced. When a user (i.e., the subject) wearing the type of neural response device 820 described in the above Figure 1 and Figure 2 discussion views the screen, only the attended object will blink vigorously, while the peripherally viewed objects that are also temporally modulated will contribute less noise (since their modulation is manifested only in the HSF components that are off-field with respect to the user's intent regarding the attended object). This enables the system to quickly and accurately determine which object the viewer is currently attending to. The modulated HSF components are nearly invisible when not the attended object, making the stimuli nearly invisible to an external viewer (i.e., a viewer other than the subject), especially when viewed from a certain distance (e.g., a distance greater than twice the distance between a typical active user and the display device). This allows for discreet (e.g., more private) interaction between the user and the attended object. This private effect is particularly pronounced when the feedback stimuli consist only of HSF signals.

[0107] In some alternative embodiments, the target object (or “screen object”) is a different graphical element presented on the display of the display device, and an overlay object is generated to correspond to one or more of the screen objects. It is the overlay object, rather than the screen object itself, that is now filtered to produce the HSF version and the LSF version.

[0108] Figure 8B is an exemplary illustration of another embodiment of the present disclosure for implementing such a scheme. As in Figure 8AIn [the figure], the display device may include a processing unit (not shown), a modulator subsystem 805, and a display driver subsystem 806: Figure 8B The display device is further provided with an overlay subsystem 814.

[0109] Here, the screen object 801 will have a display signal transmitted to the overlay subsystem 814.

[0110] In some embodiments, each screen object will have an associated overlay object. Alternatively, as Figure 8B shown, only some screen objects have an associated overlay object 811. For example, a graphical overlay object may be a geometry surrounding the corresponding screen object.

[0111] In some embodiments, such as in the Figure 8B embodiment shown, filtering is now performed on the overlay object 811 rather than on the screen object itself to generate the HSF version and the LSF version (again denoted as 803 and 804 respectively). For each overlay object 811, the processing unit of the display device operates to apply a spatial frequency filter 812 (or a spatial frequency transform) to generate the HSF version 803 and the LSF version 804 of the overlay object 811.

[0112] The modulator subsystem 805 modulates the HSF version 803 and the LSF version 804 of each overlay object respectively. Then, the modulator subsystem 805 transmits the modulated HSF version and LSF version of each overlay object to the overlay subsystem 814. The modulator subsystem 805 may optionally transmit the modulated HSF version and LSF version of each overlay object as a single superimposed overlay object or as separate versions.

[0113] The overlay subsystem 814 that receives the processed graphical overlay object processes the screen object 801 and the modulated overlay object to generate a superimposed display signal from the screen object and the overlay object.

[0114] The overlay subsystem 814 then transmits the processed superimposed display signal to the display driver subsystem 806 to drive the display.

[0115] Separate modulation may reflect the Figure 8ADifferent processing of the target object version in []. The modulator subsystem 805 is configured to process the overlay object 811 such that the LSF version 804 is static (e.g., constantly displayed), while the HSF version 803 is modulated to produce a unique blinking pattern for differentiating the overlay object. When the subject focuses on the screen object 801, the subject clearly sees the blinking overlay object 811, while the peripheral screen objects and their blinking HSF overlays are naturally suppressed by the human visual system. Again, the overlay object 811 associated with the focused target screen object 801 is easily distinguishable from other overlay objects, enabling the system to quickly and accurately determine which screen object the user is focusing on.

[0116] When the stimulus provided by the HSF component of the target object / overlay object is insufficient, the above-described schemes for processing the target object itself to apply modulation to the HSF version of the object or processing the overlay object visually superimposed on the target object to apply modulation to the HSF version of the overlay object become less effective. For example, the target object may be visually smooth such that there are not enough sharp edges or high-contrast patches to generate a large HSF component. In such a case, the amount of modulation based on the HSF version of the object is not visually apparent to the viewing subject.

[0117] Figure 10 is a block diagram showing an example software architecture 1006, which can be used in combination with various hardware architectures described herein. Figure 10 is a non-limiting example of a software architecture, and it will be understood that many other architectures can be implemented to facilitate the functions described herein. The software architecture 1006 can be executed on hardware such as Figure 11 a machine 1110 including a processor 1114, a memory 1116, and input / output (I / O) components 1118, etc. A representative hardware layer 1052 is shown and the representative hardware layer 1052 can represent, for example, Figure 11 the machine 1110. The representative hardware layer 1052 includes a processing unit 1054 with associated executable instructions 1004. The executable instructions 1004 represent the executable instructions of the software architecture 1006, including the implementations of the methods, modules, etc. described herein. The hardware layer 1052 also includes a memory and / or storage module shown as a memory / storage device 1056, which also has executable instructions 1004. The hardware layer 1052 may also include other hardware 1058, such as dedicated hardware for docking with EEG electrodes and / or for docking with a display device.

[0118] In Figure 10In the example architecture, the software architecture 1006 can be conceptualized as a stack of layers, where each layer provides a specific function. For example, the software architecture 1006 can include layers such as an operating system 1002, libraries 1020, frameworks or middleware 1018, applications 1016, and a presentation layer 1014. Operationally, the applications 1016 and / or other components within the layer can make application programming interface (API) calls 1008 through the software stack and receive responses as messages 1010. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile operating systems or dedicated operating systems may not provide frameworks / middleware 1018, while other operating systems may provide such layers. Other software architectures can include additional layers or different layers.

[0119] The operating system 1002 can manage hardware resources and provide common services. The operating system 1002 can include, for example, a kernel 1022, services 1024, and drivers 1026. The kernel 1022 can act as an abstraction layer between the hardware and other software layers. For example, the kernel 1022 can be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. The services 1024 can provide other common services to other software layers. The drivers 1026 can be responsible for controlling or interfacing with the underlying hardware. For example, depending on the hardware configuration, the drivers 1026 can include a display driver, an EEG device driver, a camera device driver, a driver, a flash drive, a serial communication driver (e.g., a universal serial bus (USB) driver), a driver, an audio driver, a power management driver, etc.

[0120] Library 1020 can provide a common infrastructure that can be used by application 1016 and / or other components and / or layers. Compared with directly interfacing with underlying operating system 1002 functions (e.g., kernel 1022, services 1024, and / or drivers 1026), library 1020 typically provides functions that enable other software modules to perform tasks in an easier way. Library 1020 can include system library 1044 (e.g., C standard library), which can provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. Additionally, library 1020 can include API library 1046, such as media libraries (e.g., libraries that support the presentation and manipulation of various media formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG), graphics libraries (e.g., OpenGL framework that can be used to render 2D and 3D graphic content on a display), database libraries (e.g., SQLite that can provide various relational database functions), web libraries (e.g., WebKit that can provide web browsing functions), etc. Library 1020 can also include a variety of other libraries 1048 to provide many other APIs to application 1016 and other software components / modules.

[0121] Framework 1018 (sometimes also referred to as middleware) provides a higher-level common infrastructure that can be used by application 1016 and / or other software components / modules. For example, framework / middleware 1018 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. Framework / middleware 1018 can provide a wide range of other APIs that can be used by application 1016 and / or other software components / modules, some of which can be specific to a particular operating system or platform.

[0122] Application 1016 includes built-in applications 1038 and / or third-party applications 1040.

[0123] Application 1016 can use built-in operating system functions (e.g., kernel 1022, services 1024, and / or drivers 1026), library 1020, or framework / middleware 1018 to create a user interface to interact with the user of the system. Alternatively or additionally, in some systems, the interaction with the user can occur through a presentation layer such as presentation layer 1014. In these systems, the application / module "logic" can be separated from aspects of the application / module that interact with the user.

[0124] Figure 11 is a block diagram showing the components of a machine 1110 according to some example embodiments. Machine 1100 is capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and executing any one or more of the methods discussed herein. Specifically, Figure 11A diagrammatic representation of a machine 1110 is shown in the example form of a computer system, in which instructions 1111 (e.g., software, program, application, applet, app, or other executable code) can be executed to cause the machine 1110 to perform any one or more of the methods discussed herein. Similarly, the instructions 1111 can be used to implement the modules or components described herein. The instructions 1111 transform a general purpose, unprogrammed machine 1110 into a particular machine programmed to perform the described and illustrated functions in the described manner. In an alternative embodiment, the machine 1110 operates as a stand-alone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1110 can operate in a server-client network environment as a server machine or a client machine, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1110 can include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smartphone, a mobile device, a wearable device (e.g., a smartwatch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing the instructions 1111 specifying the actions to be taken by the machine 1110. Further, although only a single machine 1110 is shown, the term "machine" shall also be taken to include a collection of machines that individually or jointly execute the instructions 1111 to perform any one or more of the methods discussed herein.

[0125] The machine 1110 can include a processor 1114, a memory 1116, and input / output (I / O) components 1118 that can be configured to communicate with each other, for example, via a bus 1112. In an example embodiment, the processor 1114 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), other processors, or any suitable combination thereof) can include, for example, a processor 1108 and a processor 1112 that can execute the instructions 1111. The term "processor" is intended to include a multi-core processor that can include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Figure 11 multiple processors are shown, the machine 1110 can include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0126] The memory 1116 may include a memory 1114 such as a main memory, a static memory, or other memory storage devices, and a storage unit 716, both of which are accessible by a processor 1114, for example, via a bus 1112. The storage unit 716 and the memory 1114 store instructions 1111 that implement any one or more of the methods or functions described herein. The instructions 1111 may also reside, completely or partially, within the memory 1114, within the storage unit 716, within at least one of the processors 1114 (e.g., within a cache memory of the processor), or in any suitable combination thereof during execution by the machine 1110. Accordingly, the memory 1114, the storage unit 716, and the memory of the processor 1114 are examples of machine-readable media.

[0127] As used herein, "machine-readable medium" refers to a device that can store instructions and data temporarily or permanently, and may include, but is not limited to: random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EEPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be regarded as including a single medium or multiple media that can store the instructions 1111 (e.g., a centralized or distributed database or associated cache memory and servers). The term "machine-readable medium" should also be regarded as including any medium or combination of multiple media that can store instructions (e.g., instructions 1111) for execution by a machine (e.g., machine 1110) such that the instructions, when executed by one or more processors (e.g., processor 1114) of the machine 1110, cause the machine 1110 to perform any one or more of the methods described herein. Accordingly, "machine-readable medium" refers to a single storage device or apparatus, as well as a "cloud"-based storage system or storage network that includes multiple storage devices or apparatuses. The term "machine-readable medium" does not include the signal itself.

[0128] Input / output (I / O) components 1118 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific input / output (I / O) components 1118 included in a particular machine will depend on the type of the machine. For example, a portable machine such as a mobile phone or a user interface machine may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It will be understood that the input / output (I / O) components 1118 may include Figure 11 many other components not shown.

[0129] For purposes of simplifying the following discussion, the input / output (I / O) components 1118 are grouped according to function, and this grouping is in no way limiting. In various example embodiments, the input / output (I / O) components 1118 may include output components 1126 and input components 1128. The output components 1126 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), auditory components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, and the like. The input components 1128 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens that provide touch gestures or the location and / or force of a touch, or other haptic input components), audio input components (e.g., microphones), and the like.

[0130] In yet another example embodiment, the input / output (I / O) components 1118 may include various other components such as biometric components 1130, motion components 1134, environmental components 1136, or positioning components 1138. For example, the biometric components 1130 may include components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measuring biometric signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves such as the output from an EEG device), identifying people (e.g., voice recognition, retina recognition, facial recognition, fingerprint recognition, or EEG-based recognition), and the like. The motion components 1134 may include acceleration sensor components (e.g., accelerometers), gravity sensor components, rotational sensor components (e.g., gyroscopes), and the like. The environmental components 1136 may include, for example, lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors that detect the concentration of hazardous gases for safety or measure pollutants in the atmosphere), or other components that may provide an indication, measurement, or signal corresponding to the surrounding physical environment. The positioning components 1138 may include position sensor components (e.g., global positioning system (GPS) receiver components), altitude sensor components (e.g., altimeters or barometers that detect the air pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), and the like.

[0131] Various techniques can be used to implement communication. The input / output (I / O) component 1118 can include a communication component 1140 that is operable to couple the machine 1110 to the network 1132 or the device 1120 via the couplings 1124 and 1122, respectively. For example, the communication component 1140 can include a network interface component or other suitable device to interface with the network 1132. In other examples, the communication component 1140 can include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, components (e.g., low power), components, and other communication components that provide communication via other modalities. The device 1120 can be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via a universal serial bus (USB)). In the case where an EEG device or a display device is not integrated with the machine 1110, the device 1120 can be an EEG device and / or a display device.

[0132] Although described through multiple detailed exemplary embodiments, the portable device for acquiring electroencephalogram signals according to the present disclosure includes various variations, modifications, and improvements that will be apparent to those skilled in the art. It should be understood that these various variations, modifications, and improvements fall within the scope of the subject matter of the present disclosure defined by the appended claims.

[0133] Although the overview of the inventive subject matter has been described with reference to specific example embodiments, various modifications and changes can be made to these embodiments without departing from the broader scope of the embodiments of the present disclosure. Such embodiments of the inventive subject matter may be referred to herein individually or collectively by the term "invention" merely for convenience and are not intended to voluntarily limit the scope of the present application to any single disclosure or inventive concept in the event that more than one disclosure or inventive concept is actually disclosed.

[0134] The embodiments shown herein have been described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments can be used and obtained therefrom, such that structural and logical substitutions and changes can be made without departing from the scope of the present disclosure. Accordingly, the specific embodiments should not be considered limiting, and the scope of the various embodiments is defined only by the appended claims and the full scope of equivalents to such claims.

[0135] As used herein, the term "or" may be interpreted in an inclusive or exclusive sense. In addition, multiple instances may be provided for resources, operations, or structures that are described herein as a single instance. Further, the boundaries between various resources, operations, modules, engines, and data stores are somewhat arbitrary, and particular operations are illustrated in the context of a particular illustrative configuration. Other allocations of functionality are envisioned, and other allocations of functionality may fall within the scope of various embodiments of the present disclosure. Generally, structures and functions that are presented as separate resources in an example configuration may be implemented as a combined structure or resource. Similarly, structures and functions that are presented as a single resource may be implemented as separate resources. These and other variations, modifications, additions, and improvements fall within the scope of the embodiments of the present disclosure as expressed by the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

[0136] Accordingly, the present disclosure describes systems and methods for improving the accuracy, speed performance, and visual comfort of a BCI.

[0137] In accordance with one aspect of the present disclosure, the system is a closed-loop system and includes: a display subsystem that is operative to display one or more object images; a display driver subsystem that is operative to transmit a display signal to the display subsystem; an HSF / LSF discrimination, filtering, and processing subsystem that is operative to substantially separate the object image into an HSF version and an LSF version that respectively substantially elicit an HSF neural response and substantially elicit an LSF neural response, the HSF / LSF discrimination, filtering, and processing subsystem being operative to process the HSF version to turn on and off the flicker and to process the LSF version to not flicker; and an electrode helmet that is operative to detect neural brain activity, generate an electrical signal representative of the neural brain activity, and transmit the electrical signal to the HSF / LSF discrimination, filtering, and processing subsystem, wherein the electrical signal is compared with a concurrent display signal in the HSF / LSF discrimination, filtering, and processing subsystem in order to correlate the electrical signal with the corresponding concurrent display signal.

[0138] The system may further include means for modulating the display signal of the screen object.

[0139] Alternatively or additionally, the system may further include means for modulating the display signal covered by the screen object.

[0140] According to another aspect of the present disclosure, a method for improving the accuracy, speed performance, and visual comfort of BCI is provided. The method includes: detecting neural signals from the helmet electrodes when the helmet electrodes are worn on the user's head and when the user gazes at an object on the display screen; associating the detected neural signals with a display signal for modulating the object; comparing the neural signals with the display signal; and identifying the object of interest, wherein the neural signals are associated with the display signal.

[0141] The method may further include identifying the object of interest, wherein the object of interest is a screen object.

[0142] Alternatively or additionally, the method may further include identifying the object of interest, wherein the object of interest is a screen object overlay.

[0143] Example

[0144] To better illustrate the systems and methods disclosed herein, a non-limiting list of examples is provided here:

[0145] 1. A method, comprising:

[0146] Filtering the graphic data of one or more screen objects to generate a high spatial frequency (HSF) version of the screen object or each screen object;

[0147] Applying feature modulation to each HSF version of the screen object;

[0148] Generating a display signal that includes a visual stimulus corresponding to the modulated version of the screen object;

[0149] Displaying the display signal on a display screen;

[0150] Receiving the neural signals of the user from a neural signal capture device when the user gazes at the display screen;

[0151] For each visual stimulus, determining whether the neural signals received when the user gazes at the display screen include a neural signature of the visual stimulus; and

[0152] When it is determined that the neural signals include the neural signature of the visual stimulus, identifying the object of interest of the user in the display signal displayed on the display screen, the object of interest being a display object on the display screen that is consistent with the visual stimulus.

[0153] 2. The method according to Example 1, wherein the neural signals correspond to neural oscillations measured in the visual cortex of the user's brain.

[0154] 3. The method according to Example 1 or 2, wherein the neural signature includes information associated with the characteristic modulation of the visual stimulus.

[0155] 4. The method according to any one of Examples 1, 2, or 3, further comprising: filtering the graphic data of one or more screen objects to generate a low spatial frequency (LSF) version of the screen object or each screen object; and for each LSF version of the screen object, encoding a static display signal.

[0156] 5. The method according to any one of Examples 1 to 4, wherein the graphic data of the one or more screen objects is filtered using at least one of a spatial frequency filter or a spatial frequency transform.

[0157] 6. The method according to any one of Examples 1 to 5, wherein the characteristic modulation is characteristic time modulation.

[0158] 7. The method according to any one of Examples 1 to 6, wherein determining whether the received neural signal includes the digital signature of the visual stimulus comprises:

[0159] performing a spectral analysis on the received neural signal and determining whether the spectral characteristics of the received neural signal correspond to the spectrum associated with the characteristic modulation of the visual stimulus.

[0160] 8. The method according to any one of Examples 1 to 7, wherein the neural signal capture device includes an EEG helmet containing electrodes, the EEG helmet being configured to be worn on the user's head.

[0161] 9. The method according to any one of Examples 1 to 8, wherein the object of interest is the screen object itself.

[0162] 10. The method according to any one of Examples 1 to 9, wherein the object of interest is a display object displayed on the display screen, and the visual stimulus is an overlay object that is different from and displayed above the display object.

[0163] 11. A brain-computer interface system, comprising:

[0164] a display subsystem configured to present a display screen to the user;

[0165] a neural signal capture device configured to capture neural signals associated with the user;

[0166] an interface device operably coupled to the display subsystem and the neural signal capture device, the interface device comprising:

[0167] A memory; and

[0168] A processor operably coupled to the memory and configured to:

[0169] Filter graphical data of one or more screen objects to generate the

[0170] High spatial frequency (HSF) version of the screen object or each screen object;

[0171] For each HSF version of the screen object, apply characteristic modulation;

[0172] Generate a display signal including visual stimuli corresponding to the modulated versions of the screen objects;

[0173] Transmit the display signal to the display subsystem for display on the display screen;

[0174] When the user is gazing at the display screen, receive the user's neural signals from the neural signal capture device;

[0175] On the display screen;

[0176] For each visual stimulus, determine whether the neural signals received when the user is gazing at the display screen include a neural signature of the visual stimulus; and

[0177] When it is determined that the neural signals include the neural signature of the visual stimulus,

[0178] Identify the user's object of attention in the display signal displayed on the display screen, where the object of attention is a display object on the display screen that is consistent with the visual stimulus.

[0179] When it is determined that the neural signals include the neural signature of the visual stimulus,

[0180] Identify the user's object of attention in the display signal displayed on the display screen, where the object of attention is a display object on the display screen that is consistent with the visual stimulus.

[0181] 12. The brain-computer interface system according to Example 11, wherein the processor is further configured to:

[0182] Associate the object of attention with at least one control item in a set of control items;

[0183] Based on the object of attention and the at least one control item, determine the action expected by the user; and

[0184] Implement the action expected by the user.

[0185] 13. The brain-computer interface system according to Example 11 or Example 12, wherein the processor further includes:

[0186] A display driver subsystem that operates to transmit a display signal to the display subsystem.

[0187] 14. The brain-computer interface system according to any one of Examples 11 to 13, wherein the neural signal capture device includes an electrode helmet that operates to detect neural brain activity, generate an electrical signal representing the neural brain activity, and transmit the electrical signal to the interface device.

[0188] 15. The brain-computer interface system according to any one of Examples 11 to 14, wherein the object of interest is the screen object itself.

[0189] 16. The brain-computer interface system according to any one of Examples 11 to 14, wherein the object of interest is a display object displayed on the display screen, and the visual stimulus is an overlay object that is different from and displayed on top of the display object.

[0190] 17. The brain-computer interface system according to any one of Examples 11 to 16, wherein the processor is further configured to: filter the graphic data of one or more screen objects to generate a low spatial frequency (LSF) version of the screen object or each screen object; and for each LSF version of the screen object, encode a static display signal.

[0191] 18. A computer-readable storage medium carrying instructions that, when executed by a computer, cause the computer to perform operations including:

[0192] Filter the graphic data of one or more screen objects to generate a high spatial frequency (HSF) version of the screen object or each screen object;

[0193] For each HSF version of the screen object, apply feature modulation;

[0194] Generate a display signal that includes a visual stimulus corresponding to the modulated version of the screen object;

[0195] Display the display signal on a display screen;

[0196] When the user gazes at the display screen, receive the user's neural signals from a neural signal capture device;

[0197] For each visual stimulus, determine whether the neural signals received when the user gazes at the display screen include a neural signature of the visual stimulus; and

[0198] When it is determined that the neural signal includes the neural signature of the visual stimulus, identify the object of interest of the user in the display signal displayed on the display screen, where the object of interest is a display object on the display screen that is consistent with the visual stimulus.

[0199] 19. The computer-readable storage medium according to Example 18, wherein the neural signal corresponds to neural oscillations measured in the visual cortex of the user's brain.

[0200] 20. The computer-readable storage medium according to Example 18 or Example 19, wherein the neural signature includes information associated with the characteristic modulation of the visual stimulus.

[0201] 21. The computer-readable storage medium according to any one of Examples 18, 19, or 20, wherein the instructions further cause the computer to perform operations, the operations including: filtering the graphic data of one or more screen objects to generate a low spatial frequency (LSF) version of the screen object or each screen object; and for each LSF version of the screen object, encoding a static display signal.

[0202] 22. The computer-readable storage medium according to any one of Examples 18 to 21, wherein the graphic data of the one or more screen objects is filtered using at least one of a spatial frequency filter or a spatial frequency transform.

[0203] 23. The computer-readable storage medium according to any one of Examples 18 to 22, wherein the characteristic modulation is characteristic time modulation.

[0204] 24. The computer-readable storage medium according to any one of Examples 18 to 23, wherein determining whether the received neural signal includes the digital signature of the visual stimulus includes: performing a spectral analysis on the received neural signal and determining whether the spectral characteristics of the received neural signal correspond to a spectrum associated with the characteristic modulation of the visual stimulus.

[0205] 25. A brain-computer interface system, comprising:

[0206] A display unit for displaying image data, the image data including at least one object, and the display unit further outputs a corresponding visual stimulus corresponding to one or more of the objects;

[0207] A stimulus generator for generating the visual stimulus or each visual stimulus with a corresponding characteristic modulation;

[0208] A neural signal capture device configured to capture neural signals associated with a user; and

[0209] An interface device operably coupled to the nerve signal capture device and the stimulation generator, the interface device being configured to:

[0210] Receive the nerve signal from the nerve signal capture device;

[0211] Determine the intensity of a component of the nerve signal having a property associated with modulation of the corresponding property of the visual stimulus or each visual stimulus;

[0212] Based on the nerve signal, determine which visual stimulus in the at least one visual stimulus is associated with the user's object of attention, the object of attention being inferred from the presence and / or relative intensity of a component of the nerve signal having a property associated with modulation of the property of the visual stimulus; and

[0213] Cause the stimulation generator to generate a visual stimulus for the object of attention having a feedback element, the feedback element being displayed with an effect that varies according to the intensity of the determined component having a property associated with modulation of the property of the visual stimulus for the object of attention.

[0214] 26. The system according to example 25, wherein the display effect of the feedback element varies as a linear function of the response intensity.

[0215] 27. The system according to example 25, wherein the display effect of the feedback element varies as a non-linear function of the response intensity.

[0216] 28. The system according to example 27, wherein the non-linear function is selected from a sigmoid function, a rectified linear unit (RELU) function, or a hyperbolic tangent function.

[0217] 29. The system according to example 25, wherein the modulation is selectively applied to the high spatial frequency (HSF) component of the visual stimulus.

[0218] 30. The system according to example 25, wherein the display effect includes a stepwise or continuous change from an initial visual state to a final visual state.

[0219] 31. The system according to example 25, wherein the visual stimulus is the feedback element.

[0220] 32. The system according to example 25, wherein in addition to the feedback element, the visual stimulus further includes a background element.

[0221] 33. The system according to example 32, wherein the background element has the property modulation of the visual stimulus while the feedback element is not modulated.

[0222] 34. The system according to Example 32, wherein the feedback element has a characteristic modulation of the visual stimulus, while the background element is not modulated.

[0223] 35. The system according to Example 32, wherein the characteristic modulation of the visual stimulus is applied to both the background element and the feedback element.

[0224] 36. The system according to Example 35, wherein the modulation amplitudes in the background element and the feedback element are different.

[0225] 37. A method of operating a brain-computer interface system, the brain-computer interface system including a display unit, a stimulus generator, and a neural signal capture device, the display unit displaying image data including at least one object and outputting a visual stimulus corresponding to one or more of the objects, the visual stimulus having a characteristic modulation,

[0226] wherein the method includes, in a hardware interface device operably coupled to the neural signal capture device and the stimulus generator:

[0227] Receiving neural signals from the neural signal capture device;

[0228] Determining the intensity of a component of the neural signals having a property associated with the characteristic modulation of the visual stimulus or each visual stimulus;

[0229] Determining, based on the neural signals, which visual stimulus among the at least one visual stimulus is associated with the user's object of attention, the object of attention being inferred from the presence and / or relative intensity of a component of the neural signals having a property associated with the characteristic modulation of the visual stimulus; and

[0230] Causing the stimulus generator to generate a visual stimulus for the object of attention having a feedback element, the feedback element being displayed with an effect that varies according to the intensity of the determined component having a property associated with the characteristic modulation of the visual stimulus for the object of attention.

[0231] 38. The method according to Example 37, wherein the display effect of the feedback element varies as a linear function of the response intensity.

[0232] 39. The method according to Example 37, wherein the display effect of the feedback element varies as a non-linear function of the response intensity.

[0233] 40. The method according to Example 37, wherein the modulation is selectively applied to the high spatial frequency (HSF) components of the visual stimulus.

[0234] 41. A computer-readable storage medium carrying instructions which, when executed by a computer, cause the computer to perform the method according to any one of Examples 37 to 40.

[0235] Regarding embodiments including the above embodiments, the following technical solutions are also disclosed:

[0236] Solution 1. A brain-computer interface system, comprising:

[0237] A display unit for displaying image data, the image data including at least one object, and the display unit further outputting a corresponding visual stimulus corresponding to one or more of the objects;

[0238] A stimulus generator for generating the visual stimulus or each visual stimulus with corresponding characteristic modulation;

[0239] A neural signal capture device configured to capture neural signals associated with a user; and

[0240] An interface device operably coupled to the neural signal capture device and the stimulus generator, the interface device being configured to:

[0241] Receive the neural signals from the neural signal capture device;

[0242] Determine the intensity of the component of the neural signals having a property associated with the corresponding characteristic modulation of the visual stimulus or each visual stimulus;

[0243] Based on the neural signals, determine which visual stimulus among the at least one visual stimulus is associated with the user's object of attention, the object of attention being inferred from the presence and / or relative intensity of the component of the neural signals having a property associated with the characteristic modulation of the visual stimulus; and

[0244] Cause the stimulus generator to generate a visual stimulus for the object of attention having a feedback element, the feedback element being displayed with an effect that varies according to the determined intensity of the component having a property associated with the characteristic modulation of the visual stimulus for the object of attention.

[0245] Solution 2. The brain-computer interface system according to Solution 1, wherein the display effect of the feedback element varies as a linear function of the response intensity.

[0246] Solution 3. The brain-computer interface system according to Solution 1, wherein the display effect of the feedback element varies as a non-linear function of the response intensity.

[0247] Solution 4. The brain-computer interface system according to Solution 3, wherein the non-linear function is selected from a sigmoid function, a rectified linear unit (RELU) function, or a hyperbolic tangent function.

[0248] Solution 5. The brain-computer interface system according to any one of Solutions 1 to 4, wherein the modulation is selectively applied to the high spatial frequency (HSF) component of the visual stimulus.

[0249] Solution 6. The brain-computer interface system according to any one of Solutions 1 to 5, wherein the display effect includes a gradual change or a continuous change from an initial visual state to a final visual state.

[0250] Solution 7. The brain-computer interface system according to any one of Solutions 1 to 6, wherein the visual stimulus is the feedback element.

[0251] Solution 8. The brain-computer interface system according to any one of Solutions 1 to 7, wherein in addition to the feedback element, the visual stimulus further includes a background element.

[0252] Solution 9. The brain-computer interface system according to Solution 8, wherein the background element has a characteristic modulation of the visual stimulus, while the feedback element is not modulated.

[0253] Solution 10. The brain-computer interface system according to Solution 8, wherein the feedback element has a characteristic modulation of the visual stimulus, while the background element is not modulated.

[0254] Solution 11. The brain-computer interface system according to Solution 8, wherein the characteristic modulation of the visual stimulus is applied to both the background element and the feedback element.

[0255] Solution 12. The brain-computer interface system according to Solution 11, wherein the modulation amplitudes in the background element and the feedback element are different.

[0256] Solution 13. A method of operating a brain-computer interface system, the brain-computer interface system including a display unit, a stimulus generator, and a neural signal capture device, the display unit displaying image data including at least one object and outputting a visual stimulus corresponding to one or more of the objects, the visual stimulus having a characteristic modulation,

[0257] wherein the method includes, in a hardware interface device operably coupled to the neural signal capture device and the stimulus generator:

[0258] Receiving neural signals from the neural signal capture device;

[0259] Determine the intensity of the component of the nerve signal having a property associated with the modulation of the corresponding property of the visual stimulus or each visual stimulus;

[0260] Based on the nerve signal, determine which visual stimulus in the at least one visual stimulus is associated with the user's object of interest, the object of interest being inferred from the presence and / or relative intensity of the component of the nerve signal having a property associated with the modulation of the property of the visual stimulus; and

[0261] Cause the stimulus generator to generate a visual stimulus for the object of interest having a feedback element, the feedback element being displayed with an effect that varies according to the intensity of the determined component having a property associated with the modulation of the property of the visual stimulus for the object of interest.

[0262] Scheme 14. The method according to Scheme 13, wherein the display effect of the feedback element varies as a linear function of the response intensity.

[0263] Scheme 15. The method according to Scheme 13, wherein the display effect of the feedback element varies as a non-linear function of the response intensity.

[0264] Scheme 16. The method according to any one of Schemes 13 to 15, wherein the modulation is selectively applied to the high spatial frequency (HSF) component of the visual stimulus.

[0265] Scheme 17. A computer-readable storage medium storing instructions that, when executed by a computer, cause the computer to perform the method according to any one of Schemes 13 to 16.

Claims

1. A method of operating a brain-computer interface system, comprising: displaying, by a display unit, image data including a plurality of objects; generating, by a stimulus generator, a visual stimulus having a characteristic modulation corresponding to each of the plurality of objects; receiving, from a neural signal capture device, neural signals; determining, from the plurality of objects, a target object based on detecting a correlation between the neural signals and the characteristic modulation of the visual stimulus corresponding to the target object; and displaying a feedback element of the target object, wherein the feedback element changes from a disordered distribution of visual elements to an ordered distribution forming a recognizable shape based on the strength of the correlation between the neural signals and the characteristic modulation.

2. The method according to claim 1, wherein The feedback element varies as a linear function of the strength of the correlation.

3. The method according to claim 1, wherein The feedback element varies as a non-linear function of the strength of the correlation.

4. The method according to claim 1, wherein The recognizable shape is selected from a scale line, a target marker, or a crosshair.

5. The method according to claim 1, wherein The disordered distribution includes visually elements in a pseudo-random distribution.

6. The method according to claim 1, wherein The transition from the disordered distribution to the ordered distribution occurs in a progressive step-by-step manner.

7. The method according to claim 1, wherein The transition from the disordered distribution to the ordered distribution occurs continuously.

8. The method according to claim 1, wherein, The feedback elements of objects that are not the target object remain in a disordered distribution.

9. A machine, comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the machine to perform operations including: displaying, by a display unit, image data including a plurality of objects; generating, by a stimulus generator, a visual stimulus having a characteristic modulation corresponding to each of the plurality of objects; receiving, from a neural signal capture device, neural signals; determining, from the plurality of objects, a target object based on detecting a correlation between the neural signals and the characteristic modulation of the visual stimulus corresponding to the target object; and displaying a feedback element of the target object, wherein the feedback element changes from a disordered distribution of visual elements to an ordered distribution forming a recognizable shape based on the strength of the correlation between the neural signals and the characteristic modulation.

10. A machine-readable medium comprising instructions that, when executed by a machine, cause the machine to perform operations including: displaying, by a display unit, image data including a plurality of objects; generating, by a stimulus generator, a visual stimulus having a characteristic modulation corresponding to each of the plurality of objects; receiving, from a neural signal capture device, neural signals; determining, from the plurality of objects, a target object based on detecting a correlation between the neural signals and the characteristic modulation of the visual stimulus corresponding to the target object; and displaying a feedback element of the target object, wherein the feedback element changes from a disordered distribution of visual elements to an ordered distribution forming a recognizable shape based on the strength of the correlation between the neural signals and the characteristic modulation.

Images