Brain Computer Interface

Abstract

A system and method involving a brain-computer interface in which a visual stimulus is provided overlying one or more objects, at least a portion of the visual stimulus having a characteristic modulation. The brain-computer interface measures a neural response to the object viewed by a user. The neural response to the visual stimulus is correlated with the modulation, and the correlation is stronger when attention is focused on the visual stimulus. The visual stimulus includes a feedback element that varies according to a measure of attention to the or each overlaid object.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 62 / 949,803, filed on December 18, 2019, and entitled “Brain-Computer Interface,” which is incorporated herein by reference in its entirety. Technical Field

[0003] Embodiments of the present disclosure relate to the operation of brain-computer interfaces that include visual sensing, and in particular to feedback interaction through such interfaces. Background Art

[0004] In visual brain-computer interfaces (BCIs), neural responses to target stimuli among multiple generated visual stimuli presented to a user are typically used to infer (or "decode") which stimulus is essentially the object of attention at any given time. The object of attention can then be associated with a user-selectable or user-controllable action.

[0005] The neural responses can be obtained using a variety of known techniques. One convenient method relies on surface electroencephalography (EEG), which is non-invasive, has a fine-grained temporal resolution, and is based on a well-known empirical basis. Surface EEG allows for real-time measurement of changes in diffusion potentials on the surface of the subject's skull (i.e., the scalp). These changes in potentials are commonly referred to as electroencephalographic signals or EEG signals.

[0006] In a typical BCI, visual stimuli are presented in a display generated by a display device. Examples of suitable display devices (some of which are listed in Fig. 9 906, a virtual reality headset 908, and the like. The visual stimuli 911, 911', 912, 912', 914, 914' may form part of a generated graphical user interface (GUI), or they may be presented as augmented reality (AR) or mixed reality graphical objects 916, 918 overlaying a base image: the base image may simply be the user's actual field of view (as in the case of a mixed reality display function projected onto an otherwise transparent display of a set of smart glasses) or a digital image corresponding to the user's field of view but captured in real time by an optical capture device (which may in turn capture an image corresponding to the user's field of view in other possible views).

[0007] Inferring which of multiple visual stimuli (if any) is the object of attention at any given time is fraught with difficulty. For example, when a user is presented with multiple stimuli (e.g., such as numbers on a keyboard displayed on a screen), it has proven nearly impossible to infer directly from brain activity which stimulus is being attended to at a given time. The user perceives the number being attended to (let's say the number 5), so the brain must contain information to distinguish this number from other numbers, but current methods are unable to extract this information. That is, current methods can infer with difficulty that a stimulus has been perceived, but these methods cannot determine which specific stimulus is being attended to using only brain activity.

[0008] In order to overcome this problem and to provide sufficient contrast between the stimulus and the background (and between the stimuli), it is known to configure the stimuli used by the visual BCI to flicker or pulse (e.g., a large surface of pixels switches from black to white and from white to black) so that each stimulus has a distinguishable characteristic distribution over time. The flickering stimulus causes a measurable electrical response. Specific techniques monitor different electrical responses, such as steady-state visual evoked potentials (SSVEPs) and P-300 event-related potentials. In a typical implementation, the stimulus flickers at a rate of more than 6 Hz. Therefore, such a visual BCI relies on the following method: the method includes displaying various stimuli discretely rather than continuously, and usually at different time points. Brain activity associated with attention to a given stimulus is found to correspond to (i.e., correlate with) one or more aspects of the temporal distribution of the stimulus (e.g., the frequency of stimulus flickering and / or the duty cycle of the stimulus alternating between a flickering state and a static state).

[0009] The decoding of the neural signal therefore relies on the fact that when the stimulation is turned on, it will trigger a characteristic pattern of neural responses in the brain that can be determined from the electrical signals, i.e. the SSVEP or P-300 potentials, picked up by the electrodes of the EEG device (e.g. the electrodes of an EEG helmet). This neural data pattern can be very similar or even identical for different digits, but it is time-locked to the perceived digit: only one digit can produce a pulse at any one time, so that the correlation of the time at which a digit produces a pulse with the pulsed neural response can be determined as an indication that the digit is the object of interest.

[0010] By displaying each digit at a different point in time, turning the digit on and off at different rates, applying different duty cycles, and / or simply applying stimulation at different points in time, the BCI algorithm can establish which stimulation is most likely to trigger a given neural response when turned on, enabling the system to determine the target of interest.

[0011] In recent years, visual BCIs have improved significantly, making real-time and accurate decoding of a user's attention increasingly practical. However, the constant flickering of stimuli, sometimes across the entire screen when there are many stimuli, is an inherent limitation to the large-scale use of this technology. In fact, this can cause discomfort and mental fatigue, and if it persists, can also cause physiological responses such as headaches. In addition, the flickering effect can hinder the user's ability to focus on a specific target, as well as the system's ability to quickly and accurately determine the object of attention.

[0012] For example, when a user of the on-screen keyboard discussed above attempts to focus on the number 5, the other (i.e., peripheral) numbers act as distractors, the presence of these numbers and the fact that these numbers appear to flicker temporarily attract the user's attention. The display of peripheral numbers creates interference with the user's visual system. This interference, in turn, can hamper the performance of the BCI.

[0013] Therefore, there is a need for improved methods for quickly and accurately distinguishing a screen target and its displayed stimulus to determine which one the user is focusing on, and for quickly and accurately discriminating an object of focus (target) from objects peripheral to the target (distractors).

[0014] It would therefore be desirable to provide a brain-computer interface that addresses the above challenges. Summary of the invention

[0015] The present disclosure relates to a brain-computer interface in which visual stimuli are presented on a graphical interface such that the visual stimuli are neurally decodable and provide an improved user experience.

[0016] The present disclosure also relates to a brain-computer interface (BCI) in which a visual stimulus overlaying one or more objects includes a corresponding feedback element that varies according to a measure of attention to the or each object. The visual stimulus is generated by a stimulus generator and is typically presented on a screen or other display device.

[0017] At least a portion of the visual stimulus has characteristic modulation. The neural response to the object in the user's field of view is captured by a neural signal capture device in the BCI. The user's neural response to the viewed object can then be measured and decoded to determine which object of interest is the focus of the user's attention and the user's current level of attention to the object, with the neural response being stronger when the attention is focused on the visual stimulus.

[0018] The change of the feedback element is set to be associated with the intensity of the neural response. Therefore, the feedback element of the visual stimulus can change the visual form (so that the user sees the effect on the feedback element corresponding to its attention level). In addition, the user is provided with a target for this attention, and the user can adjust the user's behavior to enhance the visual effect (effectively learning how to operate the BCI more efficiently). In other words, the user sees the effect on the feedback element that is being caused by the BCI, and can learn to use the BCI to concentrate by seeking to observe the effect change. In addition, the appearance of the visual effect associated with the concentration of attention is used to verify the selection of the underlying object.

[0019] In some embodiments, the feedback element represents the degree of attention from lack of attention to a level of focused attention as a gradual step-wise or continuous change between a disordered (e.g., pseudo-random) distribution of visual elements to a fully ordered distribution (e.g., to a recognizable shape, character, or symbol, such as a reticle, target mark, or crosshairs).

[0020] In some embodiments, the entire visual stimulus is a feedback element. In other embodiments, the visual stimulus includes a background element in addition to the feedback element. In some embodiments, the background element has a characteristic temporal modulation (i.e., a decodable modulation) of the visual stimulus, while the feedback element is not modulated. In some embodiments, the feedback element has a characteristic modulation of the visual stimulus, while the background element is not modulated.

[0021] In some embodiments, a characteristic modulation of the visual stimulus is applied to both the background element and the feedback element. The amplitude of the modulation in the background element and the feedback element may be different.

[0022] In certain aspects, the present disclosure describes a system and method for improving the accuracy and speed of determining objects of interest in a field of objects or as specific regions within a single large target. Image data for all objects is processed to extract a version of each object consisting only of high spatial frequency (HSF) components.

[0023] The present disclosure relates to techniques for acquiring (potential) objects of interest within a user's field of view (generally, but not always on a display presented to the user); extracting components related to the visual properties of these objects (e.g., their edges); and applying modulation to the high spatial frequency components of these visual properties. Thus, flickering visual stimuli used to elicit neural responses such as visual evoked potentials (VEPs) can be delivered only through the HSF version of the object. The modulation causes the object to flicker or otherwise change visually, so that the modulation acts as a stimulus for the relevant neural response. The neural response can then be measured and decoded to determine which object of interest is the focus of the user's attention.

[0024] In certain aspects, the image data may be further processed to extract another version of the object consisting of only low spatial frequency (LSF) components. In the case of extracting the LSF version, the modulated HSF version may be superimposed on the LSF version (which does not flicker).

[0025] In one aspect, the present disclosure includes a closed-loop feedback system in which a user stares at a screen and its objects, neural activity is captured as a signal using an electrode helmet, and the HSF ratio detected from the neural activity and associated with each object will change as the user's object of attention changes. This is somewhat equivalent to flickering objects at different rates and duty cycles, but because the filtering makes the flickering display objects the objects that actually cause an HSF response (e.g., an HSF version), it presents much less interference. If the object is peripheral, the flickering of its HSF version will naturally be suppressed by human visual behavior. However, the object of attention whose HSF version flickers will cause an easily recognizable neural response. Therefore, interference is significantly eliminated, making the experience more comfortable, and the identification of the object of attention is also more accurate and timely.

[0026] In each of the above embodiments, modulation may be applied preferentially or exclusively to high spatial frequency components of the projected overlay image (ie, background elements and / or feedback elements).

[0027] According to another aspect, the present disclosure relates to a brain-computer interface system, which includes: a display unit for displaying image data, the image data including at least one object, and the display unit also outputs corresponding visual stimuli to correspond to one or more of the objects; a stimulus generator, which is used to generate the visual stimulus or each visual stimulus with a corresponding characteristic modulation; a neural signal capture device, which is configured to capture neural signals associated with a user; and an interface device, which is operably coupled to the neural signal capture device and the stimulus generator, the interface device being configured to: receive neural signals from the neural signal capture device; determine the intensity of a component of the neural signal having a property associated with the corresponding characteristic modulation of the visual stimulus or each visual stimulus; determine which of the at least one visual stimulus is associated with the user's object of interest based on the neural signal, the object of interest being inferred from the presence and / or relative intensity of the component of the neural signal having a property associated with the characteristic modulation of the visual stimulus; and cause the stimulus generator to generate a visual stimulus for the object of interest having a feedback element, the feedback element being displayed with an effect that varies according to the intensity of the determined component having a property associated with the characteristic modulation of the visual stimulus for the object of interest.

[0028] According to another aspect, the present disclosure relates to a method for operating a brain-computer interface system, the brain-computer interface system comprising: a display unit, a stimulus generator and a neural signal capture device, the display unit displaying image data including at least one object and outputting visual stimulation to correspond to one or more of the objects, the visual stimulation having a characteristic modulation, wherein the method comprises, in a hardware interface device operably coupled to the neural signal capture device and the stimulus generator: receiving a neural signal from the neural signal capture device; determining the intensity of a component of the neural signal having a property associated with the corresponding characteristic modulation of the visual stimulus or each visual stimulus; determining which of the at least one visual stimulation is associated with an object of interest of the user based on the neural signal, the object of interest being inferred from the presence and / or relative intensity of the component of the neural signal having a property associated with the characteristic modulation of the visual stimulation; and causing the stimulus generator to generate a visual stimulation for the object of interest having a feedback element, the feedback element being displayed with an effect that varies according to the intensity of the determined component having a property associated with the characteristic modulation of the visual stimulation for the object of interest. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] To easily identify the discussion of any particular element or act, the highest digit(s) in a reference number refers to the figure number in which the element is first introduced.

[0030] Figure 1 An electronic architecture for receiving and processing EEG signals according to the present disclosure is shown;

[0031] Figure 2 A system incorporating a brain computer interface (BCI) according to the present disclosure is shown;

[0032] FIG. 3A to FIG. 3C shows a display of target objects having respective different varying temporal characteristics according to the present disclosure;

[0033] Figure 4 An exemplary implementation of presenting visual feedback by a BCI system of the present disclosure is shown;

[0034] Figure 5 An exemplary embodiment of presenting dynamic visual feedback by a BCI system of the present disclosure is shown;

[0035] Figure 6 Another exemplary embodiment of presenting dynamic visual feedback by a BCI system of the present disclosure is shown; and

[0036] Figure 7 The main functional blocks in the operation method of the BCI according to the present disclosure are shown.

[0037] Fig. 8A and Figure 8B An exemplary arrangement is shown in which an HSF version of a screen object or an overlay object is modulated;

[0038] Fig. 9 Various examples of display devices suitable for use with the BCI system of the present disclosure are shown;

[0039] Fig.10 is a block diagram illustrating a software architecture in which the present disclosure may be implemented according to some example embodiments;

[0040] Fig.11 is a diagrammatic representation of a machine in the form of a computer system according to some example embodiments, within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed. DETAILED DESCRIPTION

[0041] The following description includes systems, methods, techniques, instruction sequences and computing machine program products that implement the illustrative embodiments of the present disclosure. In the following description, for the purpose of illustration, many specific details are set forth to provide an understanding of the various embodiments of the inventive subject. However, it will be apparent to those skilled in the art that the embodiments of the inventive subject can be practiced without these specific details. Generally, known instruction instances, protocols, structures and techniques do not need to be shown in detail.

[0042] Figure 1 An example of an electronic architecture for receiving and processing EEG signals by means of the EEG device 100 according to the present disclosure is shown.

[0043] To measure diffuse potentials on the surface of a subject's 110 skull, EEG device 100 includes a portable device 102 (ie, a cap or headgear), analog-to-digital conversion (ADC) circuitry 104 , and a microcontroller 106 . Figure 1 The portable device 102 comprises one or more electrodes 108, typically between 1 and 128 electrodes, advantageously between 2 and 64 electrodes, advantageously between 4 and 16 electrodes.

[0044] Each electrode 108 may include a sensor for detecting electrical signals generated by the subject's neuronal activity and electronic circuitry for pre-processing (eg, filtering and / or amplifying) the detected signals prior to analog-to-digital conversion: such electrodes are referred to as "active". Figure 1 , where the sensor is in physical contact with the subject's scalp. The electrodes may be suitable for use with a conductive gel or other conductive liquid (referred to as a "wet" electrode) or without such a liquid (ie, a "dry" electrode).

[0045] Each ADC circuit 104 is configured to convert signals of a given number (eg, between 1 and 128) of active electrodes 108 .

[0046] The ADC circuit 104 is controlled by the microcontroller 106 and communicates with the microcontroller 106, for example, via the protocol SPI (“Serial Peripheral Interface”). The microcontroller 106 packages the received data for transmission, for example, via Bluetooth, Wi-Fi (“Wireless Fidelity”) or Li-Fi (“Light Fidelity”) to an external processing unit (not shown), such as a computer, a mobile phone, a virtual reality headset, an automotive or aeronautical computer system, such as a car computer or a computer system, an aircraft.

[0047] In some embodiments, each active electrode 108 is powered by a battery ( Figure 1 The battery is conveniently provided in the housing of the portable device 102.

[0048] In certain embodiments, each active electrode 108 measures a corresponding potential value, from which the potential measured by the reference electrode (Ei=Vi−Vref) is subtracted, and the difference is digitized with the aid of the ADC circuit 104 and then sent by the microcontroller 106 .

[0049] In certain embodiments, the method of the present disclosure introduces a target object for display in a graphical user interface of a display device. The target object includes a control item, which in turn is associated with an action selectable by a user.

[0050] Figure 2 A system incorporating a brain-computer interface (BCI) according to the present disclosure is shown. The system incorporates, for example, Figure 1 206 of the EEG device 100 shown in FIG. 20. In the system, an image is displayed on a display of a display device 202. A subject 204 watches the image on the display, focusing on a target object 210.

[0051] In an embodiment, the display device 202 displays at least the target object 210 as a graphical object having a changing temporal characteristic that is different from the temporal characteristics of the background and / or other displayed objects in the display. The changing temporal characteristic can be, for example, a continuous or time-locked flickering effect that changes the appearance of the target object at a rate greater than 6 Hz. In another embodiment, the changing temporal characteristic can use a pseudo-random time code to generate a flickering effect that changes the appearance of the target object several times per second on average, for example, at an average rate of 3 Hz. In the case where more than one graphical object is a potential target object (i.e., in the case where a selection of target objects to focus attention on is provided to the viewing subject), each object is associated with a discrete spatial and / or temporal code. FIG. 3A to FIG. 3C Display of target objects having respective different changing time characteristics is shown.

[0052] The neural response device 206 detects neural responses associated with attention focused on the target object (i.e., tiny electrical potentials indicative of brain activity in the visual cortex); thus, the visual perception of the changing temporal characteristics of the target object acts as a stimulus in the subject's brain, generating a specific brain response consistent with the code associated with the target object of attention. The detected neural response (e.g., electrical potential) is then converted into a digital signal and transmitted to the processing device 208 for decoding. Examples of neural responses include visual evoked potentials (VEPs), which are commonly used in neuroscience research. The term VEP encompasses: traditional SSVEPs, as mentioned above, in which the stimulus oscillates at a specific frequency; and other methods such as code-modulated VEPs, in which the stimulus is subjected to variable or pseudo-random temporal coding. Sympathetic neural responses in which the brain appears to "oscillate" or respond in synchronization with the temporal characteristics of flickering are referred to herein as "neural synchronization."

[0053] Processing device 208 executes instructions to interpret the received neural signals to determine, in real time, feedback indicating a target object having the current (visual) focus of attention. Decoding the information in the neural response signal relies on a correspondence between that information and one or more aspects of the temporal distribution of the target object (i.e., stimulus). In some embodiments, processing device 208 and neural response device 206 may be provided in a single device so that a decoding algorithm is performed directly on the detected neural responses. Thus, a BCI utilizing visually associated neural signals may be used to determine which objects on the screen the user is focusing on.

[0054] In some embodiments, the processing device may conveniently generate image data for presentation on the display device 202 that includes a target object that changes over time.

[0055] In some embodiments, the display device 202 displays the overlay object as a graphic object having changing temporal characteristics that are different from the temporal characteristics of the background and / or other displayed objects in the display, and then displays the overlay object as a graphic layer on at least the identified target object.

[0056] Figure 4 4 shows a user experience of a display device 410 that displays an overlay object 406 with different changing temporal characteristics on top of a target object 404 (which has been determined to have the current focus of attention). This can provide retrospective feedback to the user, validating the user's selection. Figure 4 As shown, visual feedback can be conveniently presented to the user on the display device 410 so that the user is aware that the target object 404 is determined as the current focus of attention. For example, the display device can display an icon, cursor, or other graphical object or effect very close to the target object 404 (in Figure 4 In the example of FIG. 4 , crosshairs 411 are used to highlight (e.g., cover) an object that appears to be the focus of current visual attention. This provides a positive feedback loop (wherein a seemingly target object is confirmed (i.e., verified) as the intended target object due to prolonged, amplified attention).

[0057] In some embodiments, the visual feedback is "proactive" in that it is actively driven to change appearance. Figure 4 In the case of retrospective feedback, the neural response to the object in the user's field of view is captured by the neural signal capture device in the BCI. The user's neural response to the viewed object can then be measured and decoded to determine which object of interest is the focus of the user's attention and the user's current level of attention to the object. When the attention is focused on the visual stimulus, the neural response is stronger.

[0058] Figure 5 Stages of changing appearance of a visual feedback effect are shown. In this exemplary embodiment, the target object 504 itself is displayed as a graphical object having changing temporal characteristics that are different from the temporal characteristics of the background and / or other displayed objects 502, 506, 508 in the display. As previously discussed, the apparent target object is determined based on the closest correlation between the visual stimulus and the decoded neural response. A measure of this correlation can be referred to as a "decoding score." Candidate target objects are presented in the display screen with a dynamic visual feedback element (e.g., an icon, cursor, crosshair, or other graphical object): the dynamic visual feedback element changes (e.g., moves or changes color, shape, size, or other visual appearance) based on the decoding score. Thus, as can be seen in FIG. Figure 5As seen in view (a) of , dynamic feedback elements are presented for more than one candidate target object (or indeed for all objects).

[0059] The change of feedback elements is set to be associated with the intensity of the response. Therefore, the feedback elements of visual stimulation can change the visual form (so that the user sees the effect on the feedback element corresponding to its attention level). Watch the changing appearance of the feedback elements to encourage the user to pay further attention. In addition, the user is provided with a target for this attention, and the user's behavior can be adjusted to enhance the visual effect (effectively learning how to operate BCI more efficiently). In other words, the user sees the effect on the feedback element caused by BCI, and can learn to use BCI to concentrate attention by seeking to observe the effect changes. For candidate target objects that are not the focus of attention, the displayed dynamic feedback elements will continue to appear as a substantially unchanged visual form.

[0060] In some embodiments, the feedback element represents the degree of attention from lack of attention to the level of focused attention as a gradual stepwise change or continuous change between a disordered (e.g., pseudo-random) distribution of visual elements to a fully ordered distribution (e.g., to a recognizable shape, character, or symbol, such as a reticle, target mark, or crosshair). Feedback elements for candidate target objects that are not the focus of attention remain in a disordered state.

[0061] In certain embodiments, for example, Figure 5 In the illustrated exemplary embodiment showing prospective feedback, each of a plurality of objects in the user's field of view is configured to present a corresponding small feedback stimulus (e.g., three separate thin lines moving pseudo-randomly). When the user pays particular attention to one of the objects (e.g., the target object 504 such as the "i" icon), the three lines superimposed on the icon move toward each other according to the decoding score until they form a triangle. In this exemplary embodiment, completely pseudo-random lines (as shown at view (a)) mean that no decoding is performed at all (for any of the objects); partial decoding is performed for the target object as shown at view (b); and a complete triangle as shown at view (c) means 100% decoding is performed at the target object. It is observed that this entire process is very fast, and it takes less than a second for an experienced user to develop attention from a state of little or no decoding to a state of full decoding (e.g., from (a) to (b), and then to (c)). Clearly, such a visual display of feedback has a reflexive cognitive effect on the perception of the target object, amplifying the brain response.

[0062] In performance, for example Figure 6In certain alternative embodiments of the forward-looking feedback shown, the visual feedback 606 indicates the decoding score by reducing the transparency of the overlay object, changing the line width of the overlay object, etc. Thus, in view (a), the visual feedback 606 is substantially invisible; in view (b), the visual feedback 606 is represented by intersecting dashed lines, indicating partial decoding; and in view (c), the visual feedback 606 represented by thicker, intersecting solid lines indicates substantially complete decoding.

[0063] Thus, active feedback is in contrast to known feedback systems in which a valid selection event requires the user to focus on a particular object for more than a predetermined period of time, and for the level of the neural response to exceed a predetermined threshold. Using active feedback (and feedback stimulation), neural feedback can be calculated and provided at shorter time scales (e.g., time scales on the order of seconds), providing information about intermediate steps of certainty of the match between the neural response and the selected object, from 0% to 100%.

[0064] In some embodiments, the relationship between feedback stimulus and decoding performance is linear. In other embodiments, the relationship is not linear: examples of alternative, non-linear relationships utilize functions such as sigmoid, hyperbolic tangent, rectified linear unit (ReLU), etc. In some embodiments, the use of a non-linear relationship appears to improve the feedback relationship, particularly in cases where the level of certainty is low, otherwise the feedback would reflect random / arbitrary fluctuations in the EEG signal.

[0065] In some embodiments, the entire visual stimulus is a feedback element. In other embodiments, the visual stimulus includes a background element in addition to the feedback element. In some embodiments, the background element has a characteristic modulation of the visual stimulus, while the feedback element is not modulated. In some embodiments, the feedback element has a characteristic modulation of the visual stimulus, while the background element is not modulated.

[0066] In some embodiments, a characteristic modulation of the visual stimulus is applied to both the background element and the feedback element. The amplitude of the modulation in the background element and the feedback element may be different.

[0067] In some embodiments, the operation of the BCI may include a brief initialization and calibration phase. Since users may differ significantly in their baseline neural responses to the same stimulus (particularly those with visual cortical disorders or damage), the calibration phase may be used to generate a user-specific stimulus reconstruction model. Such a phase may take less than a minute to construct (typically, about 30 seconds).

[0068] It has been found that objects of interest in the foveal visual area are associated with a high degree of HSF signal components. Similarly, it has been found that objects in the peripheral visual area are associated with a high degree of LSF signal components.

[0069] By increasing those differences between HSF and LSF signal components through various filtering methods, the accuracy and speed of BCI can be improved.

[0070] In another aspect of the disclosure, modulation may be applied preferentially or exclusively to high spatial frequency components of the projected overlay image (ie, background elements and / or feedback elements).The method outlined above may then be followed to determine the user's object of interest.

[0071] Figure 3A The effect of peripheral vision is shown. Subject 305 is shown as watching display screen 302, which displays multiple numbers 310, 312 in a keyboard. When the subject attempts to focus on the number "5" 310 in the keyboard on the screen discussed above, other (i.e. peripheral) numbers (e.g., "3", 312) act as distractors, temporarily attracting the user's attention and causing interference to the user's visual system. This interference then hinders the performance of BCI. Therefore, there is a need for an improved method for distinguishing screen targets and their displayed stimuli with speed and accuracy to determine which one the user is focusing on and for distinguishing an object of interest (target) from an object (distractor) peripheral to the target with speed and accuracy.

[0072] Typically, visual stimulation will occupy a large amount of the screen surface, filled with high energy uniform light (bright white shapes) or a rough checkerboard. These large surfaces will remain dedicated to the visual BCI system and cannot be used for any other purpose besides visual stimulation. These large stimulation surfaces are inconsistent with the fine and discrete integration of visual BCI systems and limit the display device, such as Fig. 9 The design freedom of the user interface in the display device shown.

[0073] Figure 3B The use of a neural responsive device, such as Figure 1 and Figure 2 The neural response device in the discriminates multiple target objects. Figure 3BIn FIG. 3 , the neural response device worn by the user (i.e., viewer) 305 is an electrode helmet for an EEG device. Here, the user wearing the helmet views a screen 302 showing multiple target objects (numbers in an on-screen keyboard) that flash at significantly different times, frequencies, and duty cycles. The electrode helmet can transmit signals derived from neural activity. Here, the user is focusing on the number 5, 310, where at time t1, the number 3, 312 flashes; at time t2, the number 4, 314 flashes; at time t3, the number 5, 310' flashes; and at time t4, the number 6, 316 flashes. The neural activity transmitted by the helmet signal is significantly different at t3 than at other time points. This is because the user is focusing on the number 5, 310, which flashes as 310' at t3. However, in order to distinguish the signal appearing at t3 from the signals at other times, all objects on the screen must flash at significantly different times. Therefore, the screen becomes active with the flashing objects, resulting in an uncomfortable viewing experience.

[0074] Figure 3B The system in can use display signal mode, such as Figure 3C An example pattern is shown, in which screen objects will flash at different frequencies and duty cycles at different points in time.

[0075] One approach to the challenge of identifying an object from objects (distractors) that are peripheral to an object of interest (target) with speed and accuracy relies on the characteristics of the human visual system.

[0076] Studies of the way human visual perception operates have shown that when staring at a screen with multiple objects and focusing on one of these objects, the human visual system is good at picking up both high spatial frequencies (HSF) and low spatial frequencies (LSF). Evidence suggests that the human visual system is primarily sensitive to the HSF component of the particular display region of interest (e.g., the object the user is looking at): this corresponds to the central region of the subject's retina filled with cones, known as the fovea. This can be seen in the Figure 3A As seen in the right view of FIG. 3 , a foveal region of the display 318 where vision is clearest contrasts with the peripheral region 304 .

[0077] In contrast, for peripheral objects, the human visual system is mainly sensitive to their LSF components.

[0078] In prior art neural capture systems, due to the operation of the human visual system, the picked-up neural signal will be substantially affected by both the HSF component from the object of interest and the LSF component from peripheral objects. However, since all objects cause a certain proportion of both HSF and LSF, processing the neural signal to determine the object of interest may be hampered by the LSF noise contributed by the peripheral objects. This tends to make the identification of the object of interest less accurate and timely.

[0079] The basic science for this approach is related to differences in how the human eye-brain system processes stimuli from objects of attention and peripheral objects. This separation between foveal (center of the visual field) and peripheral vision is described in the literature in terms of special frequency channels from the retina to the visual cortex, where foveal vision is driven primarily by HSF channels that convey visual detail, while peripheral vision is driven primarily by LSF channels that convey coarse visual information such as the global shape of objects without detail. These two types of information are associated with separate neural pathways, different functions, and different influences on unconscious and conscious perception.

[0080] Spatial frequency is usually calculated in cycles per degree. Spatial frequency depends mainly on three parameters: density pixels per inch (dpi), also known as pixels per inch (ppi); the distance between the user's eyes and the monitor; and the cutoff frequency of the spatial filter. A spatial frequency filter can be used so that the stimulus signal retains only the HSF characteristics, or conversely only the LSF characteristics. The spatial frequency filter used in the context of visual BCI can conveniently perform high-pass filtering on values ​​exceeding 7 cycles per degree, and perform low-pass filtering on values ​​below 3 cycles per degree. In some cases, the lower threshold of the low-pass filter may cause the output to have a uniform flat tone in some cases (such a "low-pass filter" is still an effective filter). In contrast, the maximum value of the high-pass filter threshold is limited by the resolution of the display system and ultimately by the visual physiological ability of the subject. In any case, regardless of the further thresholds of the low-pass and high-pass, the present disclosure operates regardless of the specific values ​​of the frequency filter and / or the transform. The main principle is to separate the spatial frequency components to optimize the visual BCI.

[0081] The human visual system is tuned to process multiple stimuli at different locations of the field of view in parallel, usually unconsciously or subconsciously. Therefore, peripheral object stimuli will continue to trigger neural responses in the user's brain even if they appear at the periphery of the field of view. This therefore causes competition between multiple stimuli and makes specific neural decoding of the object of attention (target) more difficult.

[0082] Consider Again Figure 3B The on-screen keyboard, flickering peripheral signals 312, 314, and 316 will induce LSF neural activity in the viewer, which will be captured and processed in parallel with the signals that induce HSF neural activity in the viewer stimulated by the flickering number 5, 310. Therefore, these peripheral objects can be considered as distractors, and the LSF signals they induce can be considered as noise. One result of this noise is that the system takes longer to accurately determine the object of interest.

[0083] In one method of the present disclosure, multiple objects are displayed in such a way that each object is divided into a version consisting of only the LSF component of the object and a version consisting of only the HSF component. In one example, a flickering visual stimulus used to elicit a decodable neural response (e.g., SSVEP) is delivered only via the HSF version of the object. The flickering HSF version is superimposed on the LSF version (which does not flicker). This method is described below with respect to Fig. 8A and Figure 8B Have a more in-depth discussion.

[0084] In each of the above feedback overlay arrangements, modulation can be applied preferentially or exclusively to high spatial frequency components of the projected overlay image (i.e., background elements and / or feedback elements). Preferential modulation of HSF components of overlay objects, target objects, and / or visual feedback elements can be used to improve the accuracy of determining the object of interest (and reduce distraction effects).

[0085] The above-mentioned BCI can be used in conjunction with real-world objects to make the objects controllable or otherwise interactive. In some embodiments, the generation of stimuli is handled by one or more light sources (e.g., light emitting diodes, LEDs) provided in association with the controllable object (or even on the surface of the controllable object).

[0086] In certain embodiments, the generation of the stimulus is handled by a projector or a scanning laser device, such that the visual stimulus is projected onto the controllable object, and the controllable object outputs the visual stimulus by reflecting the projected stimulus.

[0087] As in the case of a BCI using a display screen through which a user interacts with objects on the screen, the controllable objects in the present disclosure can be presented with visual stimuli having characteristic modulations (e.g., flickering stimuli) so that the neural responses to the presence of these stimuli become apparent and can be decoded from neural signals captured by a neural signal capture device (e.g., an EEG device).

[0088] In some embodiments, the determination of the focus of attention on the visual display of the controllable device is used to send a command to the controllable object. The controllable object can then implement an action based on the command: for example, the controllable object can emit an audible sound, unlock a door, turn on or off, change operating state, etc. The action can also provide visual or other feedback associated with the controllable object to the user: this can be used in conjunction with the positive feedback loop discussed above, but can also provide real-time indication of a valid selection of an operation associated with the controllable object.

[0089] Figure 7 A BCI system (eg, Figure 2The main functional blocks in the method of operation of the BCI system shown in FIG. 1 . The brain-computer interface system includes a display unit, a stimulus generator, and a neural signal capture device. The display unit displays image data including at least one object, and outputs a visual stimulus to correspond to one or more of the objects, the visual stimulus having a characteristic modulation.

[0090] In block 702 , a hardware interface device, such as an interface device, operably coupled to a neural signal capture device and a stimulation generator, receives neural signals from the neural signal capture device.

[0091] In block 704, the interface device determines the strength of the component of the neural signal having a property associated with the modulation of the respective characteristic of the or each visual stimulus.

[0092] In block 706, the interface device determines which of the at least one visual stimulus is associated with an object of interest of the user based on the neural signal, the object of interest being inferred from the presence and / or relative strength of a component of the neural signal having a property associated with a characteristic modulation of the visual stimulus.

[0093] In block 708, the interface device causes the stimulus generator to generate a visual stimulus for the object of interest having a feedback element displayed with an effect that varies according to the intensity of the determined component having a property associated with the characteristic modulation of the visual stimulus for the object of interest.

[0094] The active feedback of the present disclosure is associated with several benefits in terms of user experience (UX) and neural decoding. The feedback stimulus presents the user with a convenient guide to the user's attention (i.e., an "attention grabber") at a specific location in the display screen, helping the user to maintain focus. For viewers with certain attention-related conditions and mild visual impairments, the presence of this feature has been observed to help users maintain focus.

[0095] In addition, the user is given a task (i.e., bringing the feedback stimulus close to a fully decoded state, indicating “validation” of the selection). This also helps the user focus on a specific object while suppressing peripheral distractors.

[0096] As the user becomes more focused using this feedback stimulation, it is observed that the user-specific stimulus reconstruction model built during the initial or calibration phase of BCI operation becomes more accurate and is built faster.

[0097] In subsequent operation phases, the use of feedback stimulation as described above results in improved accuracy and speed for real-time BCI applications.

[0098] Fig. 8Ais an exemplary illustration of one implementation of the disclosed subject matter. Here, all target objects 801 are filtered to create their own HSF and LSF versions.

[0099] exist Fig. 8A In the embodiment of the present invention, the target object 801 is a different graphical element in the graphical interface presented on the display of the display device. Examples of target objects include glyphs, captured images, and user interface elements. The display device may include a processing unit (not shown), a modulator subsystem 805, and a display driver subsystem 806.

[0100] For each target object 801, the processing unit of the display device operates to apply a spatial frequency filter 802 (or a spatial frequency transform) to generate an HSF version and an LSF version (represented as 803 and 804, respectively) of the target object 801. In other embodiments, the target objects 801 may be filtered to create only an HSF version of each object. Thus, only an HSF version is generated.

[0101] The modulator subsystem 805 processes the HSF version 803 and LSF version 804 of each object and transmits them to the display driver subsystem 806. The HSF version 803 and LSF version 804 of each object are processed differently: the LSF version signal 804 is encoded to produce a static (e.g., non-flickering) display, while the HSF version signal 803 is temporally modulated (e.g., caused to flicker at different times with optionally different frequencies and / or duty cycle characteristics). Temporal modulation effects are typically periodic changes in the visual characteristics of an object, such as brightness, hue, color components, etc., and may be abrupt (switching between an on state and an off state) or may include a more gradual transition. Examples of temporal modulation include a flickering effect, in which the brightness characteristics of a group of pixels in a target object switch between two distinguishable brightness levels. Conveniently, the modulation effect encodes the optical characteristics to which the user's brain responds (when the object is viewed) in a detectable and decodable manner. The processing may also include increasing the contrast level in the HSF version signal and / or decreasing the contrast level in the LSF version signal.

[0102] In some embodiments, the modulation effect can be random or pseudo-random in nature: the change of the visual characteristics of the object, such as the flickering, is performed at random times, for example following a pseudo-random temporal pattern. Conveniently, a pseudo-random temporal pattern is used rather than a strictly random pattern to reduce the temporal overlap between different temporal patterns associated with different objects.

[0103] The display driver subsystem 806 receives the processed HSF and LSF versions of each object (either as separate signals or as a combined modulated HSF and LSF version superimposed signal) from the modulator subsystem 805. Thus, the signals driving the display include screen objects where the LSF-related signals produce a constant display and the HSF-related signals produce a flickering effect (or other temporal modulation effect).

[0104] In some embodiments, as described above, the modulator subsystem 805 may process only the HSF version of each object and transmit it to the display driver subsystem 806: so that only the HSF version is received at the display driver subsystem 806. In this case, the modulator subsystem 805 applies a high-pass filter to generate only the HSF version of each object. In some other embodiments, the modulator subsystem may apply one or more spatial frequency filters (or spatial frequency transforms) to generate both the HSF version and the LSF version of the target object, but only transmit the HSF version of each object to the display driver subsystem 806. In each case, the HSF version is temporally modulated as described above, so that the display driver subsystem drives the display with a signal including a screen object having a temporally modulated HSF component. In another embodiment, the display driver subsystem 806 also displays the generated HSF version in addition to the (entire) target object 801. In this case, the contrast in the modulated HSF version can be increased to improve the visibility of the visual stimulus above the object of interest.

[0105] The user's attention to the displayed (HSF modulated) object can then be detected (via capture 822 and decoding 807 of the neural response). Thus, the temporal modulation effect of peripherally viewed objects is significantly reduced. Figure 1 and Figure 2 When a user (i.e., subject) of a neuro-responsive device 820 of the type described in the discussion above views the screen, only the object of interest will flicker vibrantly, while peripheral objects that are also temporally modulated will contribute less noise (because their modulation only appears in HSF components that are out-of-field with respect to the user's intent regarding the object of interest). This enables the system to quickly and accurately determine which object the viewer is currently focusing on. The modulated HSF components are barely visible when it is not the object of interest, making the stimulus nearly invisible to external viewers (i.e., viewers other than the subject), particularly when viewed from a distance (e.g., a distance greater than twice the distance between a typical active user and the display device). This allows for discreet (e.g., more private) interaction between the user and the object of interest. This privacy effect is particularly pronounced when the feedback stimulus consists solely of the HSF signal.

[0106] In some alternative embodiments, the target objects (or "screen objects") are distinct graphical elements presented on a display of a display device, and overlay objects are generated to correspond to one or more of the screen objects. It is the overlay objects, not the screen objects themselves, that are now filtered to produce the HSF and LSF versions.

[0107] Figure 8B is an exemplary illustration of another embodiment of the disclosed subject matter to implement such a scheme. Fig. 8A In the embodiment, the display device may include a processing unit (not shown), a modulator subsystem 805 and a display driver subsystem 806: Figure 8B The display device is also provided with a covering subsystem 814 .

[0108] Here, the screen object 801 will have a display signal passed to the overlay subsystem 814 .

[0109] In some embodiments, each screen object will have an overlay object associated with it. Figure 8B As shown, only some of the screen objects have associated overlay objects 811. For example, a graphical overlay object may be a geometric shape surrounding a corresponding screen object.

[0110] In certain embodiments, such as Figure 8B In the illustrated embodiment, the overlay object 811 is now filtered rather than the screen object itself to produce an HSF version and an LSF version (again represented as 803 and 804, respectively). For each overlay object 811, the processing unit of the display device operates to apply a spatial frequency filter 812 (or spatial frequency transform) to generate an HSF version 803 and an LSF version 804 of the overlay object 811.

[0111] The modulator subsystem 805 modulates the HSF version 803 and the LSF version 804 of each overlay object, respectively. The modulator subsystem 805 then transmits the modulated HSF version and LSF version of each overlay object to the overlay subsystem 814. The modulator subsystem 805 may optionally transmit the modulated HSF version and LSF version of each overlay object as a single superimposed overlay object or as separate versions.

[0112] Overlay subsystem 814, which receives the processed graphics overlay object, processes screen object 801 and the modulated overlay object to generate an overlay display signal from the screen object and the overlay object.

[0113] The overlay subsystem 814 then transmits the processed overlay display signal to the display driver subsystem 806 to drive the display.

[0114] Individual modulations can reflect Fig. 8AThe modulator subsystem 805 is configured to process the overlay object 811 so that the LSF version 804 is static (e.g., constantly displayed) while the HSF version 803 is modulated to produce a unique flickering pattern for distinguishing the overlay object. When the subject focuses on the screen object 801, the subject sees the flickering overlay object 811 very clearly, while the peripheral screen objects and their flickering HSF overlay objects are naturally suppressed by the human visual system. Again, the overlay object 811 associated with the focused target screen object 801 is easily distinguished from other overlay objects, so that the system can quickly and accurately determine which screen object the user is focusing on.

[0115] The above schemes for processing the target object itself to apply modulation to the HSF version of the object or processing the overlay object visually superimposed on the target object to apply modulation to the HSF version of the overlay object become less effective when the HSF component of the target object / overlay object provides insufficient stimulus impact. For example, the target object may be visually smooth so that there are not enough sharp edges or high-contrast patches to generate a large HSF component. In this case, the amount of modulation based on the HSF version of the object is not visually noticeable to the viewing subject.

[0116] Fig.10 is a block diagram illustrating an example software architecture 1006 that may be used in conjunction with the various hardware architectures described herein. Fig.10 is a non-limiting example of a software architecture, and it will be appreciated that many other architectures may be implemented to facilitate the functionality described herein. Software architecture 1006 may be implemented in a variety of environments such as Fig.11 110, which includes a processor 1114, a memory 1116, and an input / output (I / O) component 1118, among others. A representative hardware layer 1052 is shown and may represent, for example, Fig.11 The representative hardware layer 1052 includes a processing unit 1054 having associated executable instructions 1004. The executable instructions 1004 represent executable instructions of the software architecture 1006, including implementations of the methods, modules, etc. described herein. The hardware layer 1052 also includes a memory and / or storage module, shown as a memory / storage device 1056, which also has executable instructions 1004. The hardware layer 1052 may also include other hardware 1058, such as specialized hardware for interfacing with EEG electrodes and / or for interfacing with a display device.

[0117] exist Fig.10In the example architecture of , software architecture 1006 can be conceptualized as a stack of layers in which each layer provides specific functionality. For example, software architecture 1006 may include layers such as operating system 1002, library 1020, framework or middleware 1018, application 1016, and presentation layer 1014. Operationally, application 1016 and / or other components within a layer may call application programming interface (API) call 1008 through the software stack and receive a response as message 1010. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile operating systems or dedicated operating systems may not provide framework / middleware 1018, while other operating systems may provide such a layer. Other software architectures may include additional layers or different layers.

[0118] The operating system 1002 can manage hardware resources and provide public services. The operating system 1002 can include, for example, a kernel 1022, services 1024, and drivers 1026. The kernel 1022 can act as an abstraction layer between the hardware and other software layers. For example, the kernel 1022 can be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. The services 1024 can provide other public services to other software layers. The drivers 1026 can be responsible for controlling or interfacing with the underlying hardware. For example, depending on the hardware configuration, the drivers 1026 can include a display driver, an EEG device driver, a camera device driver, drives, flash drives, serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0119] The library 1020 may provide a common infrastructure that may be used by the application 1016 and / or other components and / or layers. The library 1020 typically provides functions that enable other software modules to perform tasks in an easier manner than by directly interfacing with the underlying operating system 1002 functions (e.g., kernel 1022, service 1024, and / or driver 1026). The library 1020 may include a system library 1044 (e.g., a C standard library) that may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. In addition, the library 1020 may include an API library 1046, such as a media library (e.g., a library that supports the presentation and manipulation of various media formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG), a graphics library (e.g., an OpenGL framework that may be used to render 2D and 3D graphics content on a display), a database library (e.g., SQLite that may provide various relational database functions), a web library (e.g., WebKit that may provide web browsing functions), and the like. The library 1020 may also include a variety of other libraries 1048 to provide many other APIs to the applications 1016 and other software components / modules.

[0120] Framework 1018 (sometimes also referred to as middleware) provides a higher level common infrastructure that can be used by applications 1016 and / or other software components / modules. For example, framework / middleware 1018 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. Framework / middleware 1018 can provide a wide range of other APIs that can be used by applications 1016 and / or other software components / modules, some of which can be specific to a particular operating system or platform.

[0121] The applications 1016 include built-in applications 1038 and / or third-party applications 1040 .

[0122] Applications 1016 may create user interfaces to interact with users of the system using built-in operating system functionality (e.g., kernel 1022, services 1024, and / or drivers 1026), libraries 1020, or framework / middleware 1018. Alternatively or additionally, in some systems, interaction with the user may occur through a presentation layer, such as presentation layer 1014. In these systems, the application / module "logic" may be separate from aspects of the application / module that interact with the user.

[0123] Fig.11 1 is a block diagram illustrating components of a machine 1100 according to some example embodiments, the machine 1100 being capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methodologies discussed herein. Specifically, Fig.11A diagrammatic representation of a machine 1110 is shown in the form of an example of a computer system in which instructions 1111 (e.g., software, programs, applications, applet, app, or other executable code) for causing the machine 1110 to perform any one or more of the methods discussed herein may be executed. Similarly, instructions 1111 may be used to implement the modules or components described herein. Instructions 1111 transform a general unprogrammed machine 1110 into a specific machine that is programmed to perform the functions described and shown in the described manner. In alternative embodiments, the machine 1110 operates as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1110 may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1110 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, web appliances, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 1111 specifying actions to be taken by the machine 1110. In addition, while only a single machine 1110 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions 1111 to perform any one or more of the methodologies discussed herein.

[0124] Machine 1110 may include processor 1114, memory 1116, and input / output (I / O) components 1118 that may be configured to communicate with each other, for example, via bus 1112. In an example embodiment, processor 1114 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), other processors, or any suitable combination thereof) may include, for example, processor 1108 and processor 1112 that may execute instructions 1111. The term "processor" is intended to include multi-core processors that may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Fig.11 Multiple processors are shown, but the machine 1110 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0125] The memory 1116 may include a memory 1114 such as a main memory, static memory, or other memory storage device, and a storage unit 716, both of which are accessible by the processor 1114, for example, via the bus 1112. The storage unit 716 and the memory 1114 store instructions 1111 that implement any one or more of the methods or functions described herein. The instructions 1111 may also reside, completely or partially, within the memory 1114, within the storage unit 716, within at least one of the processors 1114 (e.g., within a cache memory of the processor), or any suitable combination thereof during execution thereof by the machine 1110. Thus, the memory 1114, the storage unit 716, and the memory of the processor 1114 are examples of machine-readable media.

[0126] As used herein, "machine-readable medium" refers to a device capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EEPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be deemed to include a single medium or multiple media (e.g., a centralized or distributed database or associated cache memory and server) capable of storing instructions 1111. The term "machine-readable medium" should also be deemed to include any medium or combination of multiple media capable of storing instructions (e.g., instructions 1111) for execution by a machine (e.g., machine 1110), such that the instructions, when executed by one or more processors (e.g., processor 1114) of machine 1110, cause machine 1110 to perform any one or more of the methods described herein. Therefore, "machine-readable medium" refers to a single storage device or device, as well as a "cloud-based" storage system or storage network including multiple storage devices or devices. The term "machine-readable medium" does not include the signal itself.

[0127] Input / output (I / O) components 1118 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific input / output (I / O) components 1118 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone or a user interface machine will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It will be appreciated that the input / output (I / O) components 1118 may include Fig.11 Many other components not shown.

[0128] The input / output (I / O) components 1118 are grouped according to function only for the purpose of simplifying the following discussion, and the grouping is by no means limiting. In various example embodiments, the input / output (I / O) components 1118 may include output components 1126 and input components 1128. The output components 1126 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), auditory components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 1128 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), touch-sensitive input components (e.g., physical buttons, touch screens that provide touch gestures or the location and / or force of touch, or other touch-sensitive input components), audio input components (e.g., microphones), etc.

[0129] In yet another example embodiment, the input / output (I / O) component 1118 may include various other components such as a biometric component 1130, a motion component 1134, an environmental component 1136, or a positioning component 1138. For example, the biometric component 1130 may include a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves such as output from an EEG device), identifying a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1134 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environmental component 1136 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The positioning component 1138 may include a position sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), etc.

[0130] Various technologies may be used to implement communications. Input / output (I / O) components 1118 may include communications components 1140 operable to couple machine 1110 to network 1132 or device 1120 via coupling 1124 and coupling 1122, respectively. For example, communications components 1140 may include a network interface component or other suitable device to interface with network 1132. In other examples, communications components 1140 may include wired communications components, wireless communications components, cellular communications components, near field communications (NFC) components, Parts (e.g. Low power consumption), Components and other communication components that provide communication via other modalities. Device 1120 can be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via a universal serial bus (USB)). In the case where the EEG device or display device is not integrated with machine 1110, device 1120 can be an EEG device and / or a display device.

[0131] Although described through a number of detailed exemplary embodiments, the portable device for acquiring electroencephalographic signals according to the present disclosure includes various variations, modifications and improvements that will be obvious to those skilled in the art, and it should be understood that these various variations, modifications and improvements fall within the scope of the subject matter of the present disclosure as defined by the appended claims.

[0132] Although an overview of the inventive subject matter has been described with reference to specific example embodiments, various modifications and changes may be made to these embodiments without departing from the broader scope of the embodiments of the present disclosure. Such embodiments of the inventive subject matter may be referred to herein individually or collectively by the term "invention", which is merely for convenience and is not intended to autonomously limit the scope of the present application to any single disclosure or inventive concept when more than one disclosure or inventive concept is in fact disclosed.

[0133] The embodiments shown herein are described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Therefore, the specific embodiments should not be considered to have a limiting meaning, and the scope of the various embodiments is limited only by the appended claims and the full scope of equivalents to which such claims are assigned.

[0134] As used herein, the term "or" may be interpreted as an inclusive meaning or an exclusive meaning. In addition, multiple instances may be provided for resources, operations, or structures described herein as a single instance. In addition, the boundaries between various resources, operations, modules, engines, and data stores are arbitrary to some extent, and specific operations are shown in the context of specific illustrative configurations. Other allocations of functions may be conceivable, and other allocations of functions may fall within the scope of various embodiments of the present disclosure. Typically, the structures and functions presented as separate resources in the example configuration may be implemented as a combined structure or resource. Similarly, the structures and functions presented as a single resource may be implemented as separate resources. These and other variations, modifications, additions, and improvements fall within the scope of the embodiments of the present disclosure as represented by the appended claims. Therefore, the specification and the accompanying drawings are considered to be illustrative rather than restrictive.

[0135] Thus, the present disclosure describes systems and methods for improving the accuracy, speed performance, and visual comfort of BCIs.

[0136] According to one aspect of the present disclosure, the system is a closed-loop system, comprising: a display subsystem, which operates to display one or more object images; a display driver subsystem, which operates to transmit a display signal to the display subsystem; an HSF / LSF discrimination, filtering and processing subsystem, which operates to substantially separate the object image into an HSF version and an LSF version that substantially cause an HSF neural response and substantially cause an LSF neural response, respectively, the HSF / LSF discrimination, filtering and processing subsystem operates to process the HSF version to turn flicker on and off, and to process the LSF version to not flicker; and an electrode helmet, which operates to detect neural brain activity, generate an electrical signal representing the neural brain activity, and transmit the electrical signal to the HSF / LSF discrimination, filtering and processing subsystem, wherein the electrical signal is compared with a concurrent display signal in the HSF / LSF discrimination, filtering and processing subsystem so as to associate the electrical signal with the corresponding concurrent display signal.

[0137] The system may further comprise means for modulating said display signal of a screen object.

[0138] Alternatively or additionally, the system may further comprise means for modulating said display signal of a screen object overlay.

[0139] According to another aspect of the present disclosure, a method for improving the accuracy, speed performance, and visual comfort of a BCI is provided, the method comprising: detecting a neural signal from the helmet electrode when the helmet electrode is worn on the user's head and when the user is looking at an object on a display screen; associating the detected neural signal with a display signal for modulating the object; comparing the neural signal with the display signal; and identifying an object of interest, wherein the neural signal is associated with the display signal.

[0140] The method may further include identifying the object of interest, wherein the object of interest is a screen object.

[0141] Alternatively or additionally, the method may further include identifying the object of interest, wherein the object of interest is a screen object overlay.

[0142] Example

[0143] To better illustrate the systems and methods disclosed herein, a non-limiting list of examples is provided here:

[0144] 1. A method comprising:

[0145] filtering graphics data for one or more screen objects to generate a high spatial frequency (HSF) version of the or each screen object;

[0146] For each HSF version of the screen object, applying characteristic modulation;

[0147] generating a display signal comprising a visual stimulus corresponding to the modulated version of the screen object;

[0148] Displaying the display signal on a display screen;

[0149] When the user looks at the display screen, receiving a neural signal of the user from a neural signal capturing device;

[0150] for each visual stimulus, determining whether the neural signal received when the user is looking at the display screen includes a neural signature of the visual stimulus; and

[0151] When it is determined that the neural signal includes the neural signature of the visual stimulus, an object of interest of the user is identified in the display signal displayed on the display screen, the object of interest being a display object on the display screen that is consistent with the visual stimulus.

[0152] 2. A method according to Example 1, wherein the neural signal corresponds to neural oscillations measured in the visual cortex of the user's brain.

[0153] 3. A method according to Example 1 or 2, wherein the neural signature includes information associated with a characteristic modulation of the visual stimulus.

[0154] 4. The method according to any one of Examples 1, 2 or 3 further includes: filtering the graphic data of one or more screen objects to generate a low spatial frequency (LSF) version of the screen object or each screen object; and encoding a static display signal for each LSF version of the screen object.

[0155] 5. A method according to any one of Examples 1 to 4, wherein the graphic data of the one or more screen objects is filtered using at least one of a spatial frequency filter or a spatial frequency transform.

[0156] 6. A method according to any one of Examples 1 to 5, wherein the characteristic modulation is a characteristic time modulation.

[0157] 7. The method of any one of Examples 1 to 6, wherein determining whether the received neural signal includes the digital signature of the visual stimulus comprises:

[0158] A spectral analysis is performed on the received neural signal, and a determination is made as to whether a spectral characteristic of the received neural signal corresponds to a spectrum associated with a characteristic modulation of the visual stimulus.

[0159] 8. A method according to any one of Examples 1 to 7, wherein the neural signal capture device includes an EEG helmet comprising electrodes, and the EEG helmet is configured to be worn on the head of a user.

[0160] 9. A method according to any one of Examples 1 to 8, wherein the object of focus is the screen object itself.

[0161] 10. A method according to any one of Examples 1 to 9, wherein the object of interest is a display object displayed on the display screen, and the visual stimulus is an overlay object that is different from the display object and displayed above the display object.

[0162] 11. A brain-computer interface system comprising:

[0163] a display subsystem configured to present a display screen to a user;

[0164] a neural signal capturing device configured to capture neural signals associated with the user;

[0165] An interface device, which is operably coupled to the display subsystem and the neural signal capturing device, the interface device comprising:

[0166] Memory; and

[0167] a processor operably coupled to the memory and configured to:

[0168] Filtering the graphics data of one or more screen objects to generate the

[0169] a high spatial frequency (HSF) version of the or each screen object;

[0170] For each HSF version of the screen object, applying characteristic modulation;

[0171] Generate a display signal, the display signal includes a

[0172] a modulated version of the visual stimulus;

[0173] The display signal is transmitted to the display subsystem to be displayed on the display screen

[0174] On display;

[0175] When the user looks at the display screen, the neural signal capturing device

[0176] receiving a neural signal from the user;

[0177] for each visual stimulus, determining whether the neural signal received when the user is looking at the display screen includes a neural signature of the visual stimulus; and

[0178] When it is determined that the neural signal includes the neural signature of the visual stimulus,

[0179] An object of interest of the user is identified in the display signal displayed on the display screen, the object of interest being a display object on the display screen that is consistent with the visual stimulus.

[0180] 12. The brain-computer interface system of example 11, wherein the processor is further configured to:

[0181] Associating the object of interest with at least one control item from a set of control items;

[0182] determining the user's intended action based on the object of interest and the at least one control item; and

[0183] Implement the action expected by the user.

[0184] 13. The brain-computer interface system of example 11 or example 12, wherein the processor further comprises:

[0185] A display driver subsystem operates to transmit display signals to the display subsystem.

[0186] 14. A brain-computer interface system according to any one of Examples 11 to 13, wherein the neural signal capture device includes an electrode helmet, which is operated to detect neural brain activity, generate electrical signals representing the neural brain activity, and transmit the electrical signals to the interface device.

[0187] 15. A brain-computer interface system according to any one of Examples 11 to 14, wherein the object of interest is the screen object itself.

[0188] 16. A brain-computer interface system according to any one of Examples 11 to 14, wherein the object of interest is a display object displayed on the display screen, and the visual stimulus is an overlay object that is different from the display object and displayed above the display object.

[0189] 17. A brain-computer interface system according to any one of Examples 11 to 16, wherein the processor is further configured to: filter the graphic data of one or more screen objects to generate a low spatial frequency (LSF) version of the screen object or each screen object; and encode a static display signal for each LSF version of the screen object.

[0190] 18. A computer-readable storage medium carrying instructions, which, when executed by a computer, cause the computer to perform operations, the operations comprising:

[0191] filtering graphics data for one or more screen objects to generate a high spatial frequency (HSF) version of the or each screen object;

[0192] For each HSF version of the screen object, applying characteristic modulation;

[0193] generating a display signal comprising a visual stimulus corresponding to the modulated version of the screen object;

[0194] Displaying the display signal on a display screen;

[0195] When the user looks at the display screen, receiving a neural signal of the user from a neural signal capturing device;

[0196] for each visual stimulus, determining whether the neural signal received when the user is looking at the display screen includes a neural signature of the visual stimulus; and

[0197] When it is determined that the neural signal includes the neural signature of the visual stimulus, an object of interest of the user is identified in the display signal displayed on the display screen, the object of interest being a display object on the display screen that is consistent with the visual stimulus.

[0198] 19. A computer-readable storage medium according to Example 18, wherein the neural signal corresponds to neural oscillations measured in a visual cortex of the user's brain.

[0199] 20. A computer-readable storage medium according to Example 18 or Example 19, wherein the neural signature includes information associated with a characteristic modulation of the visual stimulus.

[0200] 21. A computer-readable storage medium according to any one of Examples 18, 19 or 20, wherein the instructions further cause the computer to perform operations comprising: filtering graphic data of one or more screen objects to generate a low spatial frequency (LSF) version of the screen object or each screen object; and encoding a static display signal for each LSF version of the screen object.

[0201] 22. A computer-readable storage medium as described in any of Examples 18 to 21, wherein the graphic data of the one or more screen objects is filtered using at least one of a spatial frequency filter or a spatial frequency transform.

[0202] 23. A computer-readable storage medium according to any one of Examples 18 to 22, wherein the characteristic modulation is a characteristic time modulation.

[0203] 24. A computer-readable storage medium according to any one of Examples 18 to 23, wherein determining whether the received neural signal includes the digital signature of the visual stimulus includes: performing spectral analysis on the received neural signal, and determining whether the spectral characteristics of the received neural signal correspond to the spectrum associated with the characteristic modulation of the visual stimulus.

[0204] 25. A brain-computer interface system comprising:

[0205] a display unit for displaying image data, the image data including at least one object, the display unit further outputting corresponding visual stimuli to correspond to one or more of the objects;

[0206] a stimulus generator for generating the or each visual stimulus having a corresponding characteristic modulation;

[0207] a neural signal capturing device configured to capture neural signals associated with a user; and

[0208] An interface device operably coupled to the neural signal capturing device and the stimulation generator, the interface device being configured to:

[0209] receiving the neural signal from the neural signal capturing device;

[0210] determining the intensity of a component of the neural signal having a property associated with a modulation of a respective characteristic of the or each visual stimulus;

[0211] determining, based on the neural signal, which of the at least one visual stimulus is associated with an object of interest of the user, the object of interest being inferred from the presence and / or relative strength of a component of the neural signal having a property associated with a characteristic modulation of the visual stimulus; and

[0212] The stimulus generator is caused to generate a visual stimulus for the object of interest having a feedback element displayed with an effect that varies according to the intensity of the determined component having a property associated with characteristic modulation of the visual stimulus for the object of interest.

[0213] 26. A system according to Example 25, wherein the display effect of the feedback element varies as a linear function of the response strength.

[0214] 27. A system according to Example 25, wherein the display effect of the feedback element varies as a non-linear function of the response strength.

[0215] 28. A system according to Example 27, wherein the nonlinear function is selected from a sigmoid function, a rectified linear unit (RELU) function, or a hyperbolic tangent function.

[0216] 29. A system according to Example 25, wherein the modulation is selectively applied to a high spatial frequency (HSF) component of the visual stimulus.

[0217] 30. A system according to Example 25, wherein the display effect includes a gradual change or a continuous change from an initial visual state to a final visual state.

[0218] 31. The system of Example 25, wherein the visual stimulus is the feedback element.

[0219] 32. A system according to Example 25, wherein the visual stimulation also includes a background element in addition to the feedback element.

[0220] 33. A system according to Example 32, wherein the background element has a characteristic modulation of the visual stimulus and the feedback element is not modulated.

[0221] 34. A system according to Example 32, wherein the feedback element has a characteristic modulation of the visual stimulus and the background element is not modulated.

[0222] 35. A system according to Example 32, wherein the characteristic modulation of the visual stimulus is applied to both background elements and feedback elements.

[0223] 36. A system according to Example 35, wherein the modulation amplitudes in the background element and the feedback element are different.

[0224] 37. A method of operating a brain-computer interface system, the brain-computer interface system comprising a display unit, a stimulus generator, and a neural signal capture device, the display unit displaying image data including at least one object and outputting visual stimuli to correspond to one or more of the objects, the visual stimuli having characteristic modulation,

[0225] The method comprises, in a hardware interface device operably coupled to the neural signal capture device and the stimulation generator:

[0226] receiving a neural signal from the neural signal capturing device;

[0227] determining the intensity of a component of the neural signal having a property associated with a modulation of a respective characteristic of the or each visual stimulus;

[0228] determining, based on the neural signal, which of the at least one visual stimulus is associated with an object of interest of the user, the object of interest being inferred from the presence and / or relative strength of a component of the neural signal having a property associated with a characteristic modulation of the visual stimulus; and

[0229] The stimulus generator is caused to generate a visual stimulus for the object of interest having a feedback element displayed with an effect that varies according to the intensity of the determined component having a property associated with characteristic modulation of the visual stimulus for the object of interest.

[0230] 38. A method according to Example 37, wherein the display effect of the feedback element varies as a linear function of the response intensity.

[0231] 39. A method according to Example 37, wherein the display effect of the feedback element varies as a non-linear function of the response intensity.

[0232] 40. A method according to Example 37, wherein the modulation is selectively applied to a high spatial frequency (HSF) component of the visual stimulus.

[0233] 41. A computer-readable storage medium carrying instructions which, when executed by a computer, cause the computer to perform a method according to any one of Examples 37 to 40.

Claims

1. A brain-computer interface system, comprising: a display unit for displaying image data, the image data including objects, the display unit also outputting corresponding visual stimuli to correspond to one or more of the objects; a stimulus generator for generating said visual stimulus with corresponding characteristic modulation; a neural signal capturing device configured to capture neural signals associated with a user; as well as An interface device operably coupled to the neural signal capturing device and the stimulation generator, the interface device being configured to: receiving the neural signal from the neural signal capturing device; determining a strength of a component of the neural signal having a property associated with a corresponding characteristic modulation of the visual stimulus; determining the object as an object of interest of the user based on the neural signal, the object of interest being inferred from the presence of a component of the neural signal having a property associated with a characteristic modulation of the visual stimulus; as well as The stimulus generator is caused to generate a visual stimulus for the object of interest having a feedback element displayed with an effect that varies according to an intensity of a component having a property associated with a characteristic modulation of the visual stimulus for the object of interest.

2. The brain-computer interface system according to claim 1, wherein: The displayed effect of the feedback element varies as a linear function of the response strength.

3. The brain-computer interface system according to claim 1, wherein: The displayed effect of the feedback element varies as a non-linear function of the response strength.

4. The brain-computer interface system according to claim 3, wherein: The nonlinear function is selected from a sigmoid function, a rectified linear unit (RELU) function or a hyperbolic tangent function.

5. The brain-computer interface system according to claim 1, wherein: The modulation is selectively applied to a high spatial frequency (HSF) component of the visual stimulus.

6. The brain-computer interface system according to claim 1, wherein: The effect displayed includes a gradual change or a continuous change from an initial visual state to a final visual state.

7. The brain-computer interface system according to claim 1, wherein: The visual stimulus is the feedback element.

8. The brain-computer interface system according to claim 1, wherein: In addition to the feedback element, the visual stimulus also includes a background element.

9. The brain-computer interface system according to claim 8, wherein: The background element has a characteristic modulation of the visual stimulus, while the feedback element is not modulated.

10. The brain-computer interface system according to claim 8, wherein: The feedback element has a characteristic modulation of the visual stimulus, while the background element is not modulated.

11. The brain-computer interface system according to claim 8, wherein: The characteristic modulation of the visual stimulus is applied to both the background element and the feedback element.

12. The brain-computer interface system according to claim 11, wherein: The modulation amplitudes in the background element and the feedback element are different.

13. A method of operating a brain-computer interface system, the brain-computer interface system comprising a display unit, a stimulus generator, and a neural signal capture device, the display unit displaying image data including an object to a user and outputting visual stimulus to correspond to the object, the visual stimulus having a characteristic modulation, in, The method includes performing the following operations in a hardware interface device operably coupled to the neural signal capture device and the stimulation generator: receiving a neural signal from the neural signal capturing device; determining a strength of a component of the neural signal having a property associated with a corresponding characteristic modulation of the visual stimulus; determining the object as an object of interest of the user based on the neural signal, the object of interest being inferred from the presence of a component of the neural signal having a property associated with a characteristic modulation of the visual stimulus; as well as The stimulus generator is caused to generate a visual stimulus for the object of interest having a feedback element displayed with an effect that varies according to an intensity of a component having a property associated with a characteristic modulation of the visual stimulus for the object of interest.

14. The method according to claim 13, wherein: The displayed effect of the feedback element varies as a linear function of the response strength.

15. The method according to claim 13, wherein: The displayed effect of the feedback element varies as a non-linear function of the response strength.

16. The method according to claim 13, wherein: The modulation is selectively applied to a high spatial frequency (HSF) component of the visual stimulus.

17. A computer-readable storage medium storing instructions, which, when executed by a computer, cause the computer to perform operations including: displaying image data including an object and a visual stimulus corresponding to the object to a user, the visual stimulus having a characteristic modulation; receiving a neural signal of the user from a neural signal capturing device; determining a strength of a component of the neural signal having a property associated with a characteristic modulation of the visual stimulus; determining the object as an object of interest of the user based on the neural signal, the object of interest being inferred from the presence of a component of the neural signal having a property associated with a characteristic modulation of the visual stimulus; as well as A visual stimulus for the object of interest is generated having a feedback element displayed with an effect that varies according to the intensity of the determined component having a property associated with the characteristic modulation of the visual stimulus for the object of interest.

18. The computer-readable storage medium of claim 17, wherein: The displayed effect of the feedback element varies as a linear function of the response strength.

19. The computer-readable storage medium of claim 17, wherein: The displayed effect of the feedback element varies as a non-linear function of the response strength.

20. The computer-readable storage medium of claim 17, wherein: The modulation is selectively applied to a high spatial frequency (HSF) component of the visual stimulus.

Images