Method and machine related to brain-computer interface

Abstract

The invention relates to a method and a machine related to a brain-computer interface. In one embodiment, a method is provided comprising: generating a composite visual stimulus having two or more distinguishable portions; displaying the composite visual stimulus to the user; determining, using a first decoding model, a first decoded neural signal captured from the user in response to a first portion of the composite visual stimulus, the first portion having a first visual characteristic; determining, using a second decoding model, a second decoded neural signal captured from the user in response to a second portion of the composite visual stimulus, the second portion having a second visual characteristic distinguishable from the first visual characteristic; comparing the first decoded neural signal with the second decoded neural signal to determine attention of the user; and determining an attention of the user when defocused based on a balanced response of the first and second portions of the composite visual stimulus.

Description

[0001] This application is a divisional application of the invention patent application with the application date of January 11, 2021, application number 202180009809.9 (international stage application number PCT / EP2021 / 050393), and invention name “Brain-Computer Interface”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 62 / 958,072, filed on January 7, 2020, entitled “BRAIN-COMPUTER INTERFACE,” which is incorporated herein by reference in its entirety. Technical Field

[0004] Embodiments of the present disclosure relate to the distinction between exploration and selection in brain-computer interfaces involving visual sensing. Background Art

[0005] In a visual brain-computer interface (BCI), neural responses to a target stimulus are used to infer (or "decode") which stimulus is the primary focus object at any given time, typically among multiple generated visual stimuli presented to a user. The focus object can then be associated with a user-selectable or user-controllable action.

[0006] Various known techniques can be used to obtain neural responses. A convenient method relies on surface electroencephalography (EEG), which is non-invasive, has fine-grained temporal resolution, and is based on well-known empirical foundations. Surface EEG allows real-time measurement of diffusion potential changes on the surface of the subject's skull (i.e., scalp). These potential changes are generally referred to as electroencephalographic signals or EEG signals.

[0007] In a typical BCI, visual stimuli are presented in a display generated by a display device. Examples of suitable display devices (some of which are listed in Figure 3300 ) includes: a television screen and a computer monitor 302, a projector 310, a virtual reality headset 306, an interactive whiteboard, and display screens of tablets 304, smartphones, smart glasses 308, etc. Visual stimuli 311, 311', 312, 312', 314, 314', 316 may form part of a generated graphical user interface (GUI), or they may be presented as augmented reality (AR) or mixed reality graphical objects 316 overlaid on a base image: the base image may be simply the user's actual field of view (as in the case of a mixed reality display function projected onto an otherwise transparent display of a set of smart glasses) or a digital image corresponding to the user's field of view but captured in real time by an optical capture device (which in turn may capture an image corresponding to the user's field of view among other possible views).

[0008] Inferring which of multiple visual stimuli (if any) is the object of focus at any given time is fraught with difficulty. For example, when a user is presented with multiple stimuli such as, for example, numbers displayed on an on-screen keyboard (see Figure 4 ), it has proven difficult to infer directly from brain activity which one is in focus at a given time. The user perceives a number in focus (say, the number 5), so the brain must contain information to distinguish that number from the others, but current methods cannot extract that information. That is, current methods can infer that a stimulus has been perceived, but they cannot determine which specific stimulus is in focus using brain activity alone.

[0009] In order to overcome this problem, and provide sufficient contrast between the stimulus and the background (and between the stimuli), it is known to configure the stimuli used by visual BCI to flicker or pulse (e.g., a large surface of pixels switches from black to white and vice versa) so that each stimulus has a distinguishable characteristic distribution over time. The flickering stimulus causes a measurable electrical response. Specific techniques monitor different electrical responses, such as steady-state visual evoked potentials (SSVEPs) and P-300 event-related potentials. In a typical implementation, the stimulus flickers at a frequency of more than 6 Hz. Therefore, such a visual BCI relies on the following method, which includes displaying various stimuli in a display device discretely rather than continuously, and usually at different time points. It is found that brain activity associated with attention focused on a given stimulus corresponds to (i.e., correlates to) one or more aspects of the temporal distribution of the stimulus (e.g., the frequency of stimulus flickering and / or the duty cycle of the stimulus alternating between a flickering state and a static state).

[0010] The decoding of neural signals therefore relies on the fact that when a stimulus is turned on (i.e., presented), it will trigger a characteristic pattern of neural responses in the brain that can be determined from the electrical signals, i.e., SSVEP or P-300 potentials, picked up by the electrodes of an EEG device (e.g., an EEG helmet). Figure 4 In the keyboard example shown, this neural data pattern may be very similar, or even identical, for different digits, but it is time-locked to the digit being perceived: at any one time, only one digit can pulse, such that the correlation with the pulsing neural response and the timing of that digit pulse can be determined as an indication that that digit is the object in focus. By displaying each digit at a different point in time, turning the digit on and off at different rates, applying different duty cycles, and / or simply applying stimulation at different points in time, the BCI algorithm can determine which stimulus, when turned on, is most likely to trigger a given neural response, thereby enabling the system to determine the object in focus.

[0011] Even after a target is determined to be in focus, visual computer interfaces such as the BCI described above face further challenges. One major challenge in the field of visual computer interfaces is the so-called "Midas Touch" problem, in which a user inadvertently generates an output action when simply viewing a target stimulus (the stimulus may be located anywhere in the user's field of view 404 including the focus area 418) without ever intending to trigger the associated action. In fact, it has proven difficult to accurately estimate whether a viewed target is merely "exploratory" or whether the user also intended to select the target (i.e., generate an output action).

[0012] In the field of eye tracking, the Midas touch problem reflects the difficulty of estimating whether a user is fixing their gaze on a particular target for exploration or intentionally fixing their gaze on a particular target (for the purpose of "selecting" that target and / or generating an output action on the interface). This estimation is typically done by measuring dwell time: a timer is started when the (tracked) gaze enters the target area, and the timer is verified when the timer elapses (without obvious gaze divergence). Dwell time can be very frustrating to work with because it relies on the user's observation to infer interaction. While eye tracking information has been reliably used to reveal the user's gaze location, it has proven difficult to provide intentional control to the user due to the inability to distinguish between the user's mere observation of the (fixated) target (i.e., exploration) and an intentional gaze intended to express the user's intention to trigger an action associated with the target.

[0013] While the Midas touch problem is a major challenge in the field of eye-tracking-based user interfaces, it also arises in visual BCIs. A user may wish to investigate (i.e., attend to) a displayed object without intending to control the object or trigger an associated action. Furthermore, there are situations in which a user of a BCI enables their gaze to rest on a screen object that displays a (decodable) visual stimulus without associated attention—e.g., in a blank (or empty) gaze. It is desirable to distinguish such situations from situations intended to control or trigger an action.

[0014] Therefore, there is a need for improved methods for operating a BCI to distinguish between a target that a user is focusing on with the intent to trigger an action and a screen object that is merely being viewed (either unintentionally or with intent to investigate). It would therefore be desirable to provide a brain-computer interface that addresses the above challenges. Summary of the invention

[0015] The present disclosure relates to a brain-computer interface (BCI) in which a display object or an overlay object overlaying one or more display objects presents a composite visual stimulus. The composite visual stimulus is generated by a stimulus generator and is typically presented on a screen or other display device. The composite visual stimulus includes at least a first portion and a second portion. In some embodiments, the first portion and the second portion of the stimulus may be presented overlapping with the same display location (e.g., Fig. 7A (b) where the first stimulus part consists of a "hook" and the second part consists of a "cross").

[0016] A first portion of the composite visual stimulus has a first characteristic modulation, and a second portion of the composite visual stimulus has a second characteristic modulation, the second characteristic modulation being different from the first characteristic modulation. Neural responses to objects in the user's field of view are captured by a neural signal capture device in the BCI. The user's neural responses to the viewed objects (and their associated characteristic modulations) can in turn be measured and decoded to determine which object of interest is the focus of the user's attention.

[0017] In the case of viewing only the displayed object or the overlay object, the neural response is evenly distributed between the first and second parts of the composite visual stimulus. The user is able to consciously distinguish the parts by virtue of the visual characteristics of the parts. For example, the first part can be presented in a first color (e.g., green) and the second part in a different second color (e.g., red). When the user intentionally focuses attention on one of the first and second parts (e.g., the part displayed in green), the neural response becomes stronger for the part with the visual characteristics of attention, and the BCI can infer that the intentional attention is associated with the viewed object.

[0018] According to a first aspect, the present disclosure relates to a brain-computer interface system, comprising: a display unit for displaying image data, the image data including at least one object, the display unit also outputting corresponding visual stimuli corresponding to one or more of the objects; a stimulus generator for generating a visual stimulus or each visual stimulus having a corresponding characteristic modulation; a neural signal capture device configured to capture neural signals associated with a user; and an interface device operatively coupled to the neural signal capture device and the stimulus generator, the interface device being configured to: receive neural signals from the neural signal capture device; determine the intensity of the following components of the neural signal, the components having characteristics associated with the corresponding characteristic modulation of the visual stimulus or each visual stimulus; determine which of the at least one visual stimulus is associated with the user's focused object based on the neural signal, the focused object being inferred from the presence and / or relative intensity of the components of the neural signal having characteristics associated with the characteristic modulation of the visual stimulus; and cause the stimulus generator to generate a visual stimulus as a composite visual stimulus, the composite visual stimulus comprising at least a first portion having a first characteristic modulation and a first visual characteristic and a second portion having a second characteristic modulation and a second visual characteristic, the second visual characteristic being different from the first visual characteristic. In some embodiments, the second visual characteristic is orthogonal to the first visual characteristic, and each characteristic is perceived and decoded independently of the other characteristic.

[0019] According to a second aspect, the present disclosure relates to a method for operating a brain-computer interface system, the brain-computer interface system comprising: a display unit, a stimulus generator and a neural signal capture device, the display unit displays image data including at least one object, and outputs visual stimuli corresponding to one or more of the objects, the visual stimuli having a characteristic modulation, wherein the method comprises, in a hardware interface device operatively coupled to the neural signal capture device and the stimulus generator: receiving a neural signal from the neural signal capture device; determining the intensity of the following components of the neural signal, the components having characteristics associated with the corresponding characteristic modulation of the visual stimulus or each visual stimulus; determining which of the at least one visual stimulus is associated with a focused object of the user based on the neural signal, the focused object being inferred from the presence and / or relative intensity of the components of the neural signal having characteristics associated with the characteristic modulation of the visual stimulus; and causing the stimulus generator to generate a visual stimulus as a composite visual stimulus, the composite visual stimulus comprising at least a first portion having a first characteristic modulation and a first visual characteristic and a second portion having a second characteristic modulation and a second visual characteristic, the second visual characteristic being different from the first visual characteristic. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] To easily identify the discussion of any particular element or action, the most significant digit or digits in a reference number refer to the figure number in which the element is first introduced.

[0021] Figure 1 An electronic architecture for receiving and processing EEG signals according to the present disclosure is shown;

[0022] Figure 2 A system including a brain-computer interface (BCI) according to the present disclosure is shown;

[0023] Figure 3 Various examples of display devices suitable for use with the BCI system of the present disclosure are shown;

[0024] Figure 4 A keyboard use case of a BCI system of the present disclosure is shown;

[0025] Figure 5A and Figure 5B shows corresponding exemplary embodiments of composite visual stimuli according to the present disclosure;

[0026] Figure 6 shows different types of feature modulation applied to corresponding visual stimuli according to the present disclosure;

[0027] Fig. 7A and Figure 7B shows a progression from exploration to validation selection in a method of operation of a BCI according to the present disclosure;

[0028] Figure 8 Another exemplary embodiment of a composite visual stimulus according to the present disclosure is shown;

[0029] Fig. 9 illustrates the operation of eye tracking technology in conjunction with a BCI in accordance with the present disclosure;

[0030] Fig.10 shows the main functional blocks in the method of operation of the BCI according to the present disclosure;

[0031] Fig.11 is a block diagram illustrating a software architecture in which the present disclosure may be implemented according to some example embodiments; and

[0032] Fig.12 is a diagrammatic representation of a machine in the form of a computer system within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein according to some example embodiments. DETAILED DESCRIPTION

[0033] The following description includes systems, methods, techniques, instruction sequences and computing machine program products that embody the illustrative embodiments of the present disclosure. In the following description, for the purpose of illustration, many specific details are set forth to provide an understanding of the various embodiments of the subject of the present invention. However, it is obvious to those skilled in the art that the embodiments of the subject of the present invention can be practiced without these specific details. Usually, known instruction instances, protocols, structures and techniques are not necessarily shown in detail.

[0034] Figure 1 An example of an electronic architecture for receiving and processing EEG signals with the EEG device 100 according to the present disclosure is shown.

[0035] To measure diffuse potentials on the surface of a subject's 110 skull, EEG device 100 includes a portable device 102 (ie, a cap or headgear), analog-to-digital conversion (ADC) circuitry 104 , and a microcontroller 106 . Figure 1 The portable device 102 comprises one or more electrodes 108, typically between 1 and 128 electrodes, advantageously between 2 and 64 electrodes, advantageously between 4 and 16 electrodes.

[0036] Each electrode 108 may include a sensor for detecting electrical signals generated by the subject's neuronal activity, and electronic circuitry for pre-processing (e.g., filtering and / or amplifying) the detected signals prior to analog-to-digital conversion: such electrodes are referred to as "active". Figure 1 , where the sensor is in physical contact with the subject's scalp. The electrodes may be adapted for use with a conductive gel or other conductive liquid (referred to as a "wet" electrode), or without such a liquid (ie, a "dry" electrode).

[0037] Each ADC circuit 104 is configured to convert signals from a given number (eg, between 1 and 128) of active electrodes 108 .

[0038] The ADC circuit 104 is controlled by the microcontroller 106 and communicates with the microcontroller 106, for example, via the protocol SPI (“Serial Peripheral Interface”). The microcontroller 106 packages the received data for transmission, for example, via Bluetooth, Wi-Fi (“Wireless Fidelity”) or Li-Fi (“Light Fidelity”) to an external processing unit (not shown), such as a computer, a mobile phone, a virtual reality headset, an automotive or aerospace computer system such as a car computer or a computer system, an aircraft.

[0039] In some embodiments, each active electrode 108 is powered by a battery ( Figure 1 The battery is conveniently disposed in the housing of the portable device 102 .

[0040] In certain embodiments, each active electrode 108 measures a respective potential value, from which the potential measured by the reference electrode (Ei=Vi−Vref) is subtracted, and the difference is digitized by the ADC circuit 104 and then transmitted by the microcontroller 106 .

[0041] In certain embodiments, the method of the present disclosure introduces a target object for display in a graphical user interface of a display device. The target object includes a control item, and the control item is in turn associated with an action selectable by a user.

[0042] Figure 2 A system including a brain-computer interface (BCI) according to the present disclosure is shown. The system includes a neural response device 206, such as Figure 1 The EEG device 100 shown in FIG. 2 is a schematic diagram of an EEG apparatus 100 shown in FIG. 2 . In the system, an image is displayed on a display of a display device 202 . A subject 204 views the image on the display, focusing on a target object 210 .

[0043] In an embodiment, the display device 202 displays at least the target object 210 as a graphical object having a changing temporal characteristic that is different from the temporal characteristics of the background and / or other displayed objects in the display. For example, the changing temporal characteristic can be a constant or time-locked flickering effect that changes the appearance of the target object at a rate greater than 6 Hz. In another embodiment, the changing temporal characteristic can use a pseudo-random time code to generate a flickering effect that changes the appearance of the target object several times per second on average, for example, at an average rate of 3 Hz. In the case where more than one graphical object is a potential target object (i.e., in the case where the viewing subject is provided with a choice of target objects to focus attention on), each object is associated with a discrete spatial and / or temporal code.

[0044] The neural response device 206 detects neural responses (i.e., tiny electrical potentials indicative of brain activity in the visual cortex) associated with attention focused on the target object; thus, the visual perception of the changing temporal features of the target object acts as a stimulus in the subject's brain, generating a specific brain response consistent with the code associated with the target object of attention. The detected neural response (e.g., electrical potential) is then converted into a digital signal and transmitted to the processing device 208 for decoding. Examples of neural responses include visual evoked potentials (VEPs), which are commonly used in neuroscience research. The term VEP encompasses: conventional SSVEPs, as mentioned above, in which the stimulus oscillates at a specific frequency; and other methods such as code-modulated VEPs, in which the stimulus is subjected to a variable or pseudo-random time code. Neural responses in which the brain appears to "oscillate" or respond in sync with the temporal features of the flicker are referred to herein as "neural synchronization."

[0045] Processing device 208 executes instructions to interpret the received neural signals to determine, in real time, feedback indicative of a target object having a current focus of (visual) attention. Decoding the information in the neural response signals relies on a correspondence between that information and one or more aspects of the temporal distribution of the target object (i.e., stimulus). In some embodiments, processing device 208 and neural response device 206 may be provided in a single device such that a decoding algorithm is performed directly on the detected neural responses.

[0046] In some embodiments, the processing device may conveniently generate image data for presentation on the display device 202 that includes a target object that changes over time.

[0047] In some embodiments, the display device 202 displays the overlay object as a graphic object having varying temporal characteristics that are different from temporal characteristics of a background and / or other displayed objects in the display, and the overlay object is then displayed as a graphic layer over at least the identified target object.

[0048] Although neural activity in response to external stimuli can be modulated by spontaneous top-down attention, stimulus reconstruction is also largely driven by bottom-up factors that will automatically influence the neural oscillations of the observed target regardless of attention, making it difficult to estimate whether an action is intentional by the user.

[0049] To address this difficulty, embodiments of the present disclosure present a composite visual stimulus for each target: instead of a single visual stimulus, the composite visual stimulus includes at least two stimulus portions with different neural signatures. In certain embodiments, the two stimulus portions are overlaid at substantially the same location, each with a different temporal modulation pattern. One portion is used for verification (i.e., enabling the BCI to infer intentional attention), and the other portion is used to balance the bottom-up influence of the verification stimulus: intentionally paying attention to one stimulus portion (at the expense of another stimulus portion) for the purpose of triggering an action has a neural synchronization pattern different from simply viewing the target without distinguishing the portions. The principle behind this approach is that users who pay attention to a given display object (e.g., the letter 'A') simply for exploration will not select it unless they pay special attention to a given overlay portion, such as the green line ("verification stimulus") rather than the red line ("balanced stimulus").

[0050] Research in cognitive neuroscience on cortical modulation of feature-based attention has shown that it is possible to distinguish neural signatures between two sets of flickering dots displayed at the same location, such as one displayed as green at a particular frequency (e.g., 8 Hz) and another displayed as red at another frequency (e.g., 9 Hz). That is, peaks at two corresponding frequencies can be observed in the spectral representation of the neural data. Furthermore, if a subject is asked to preferentially attend to a particular stimulus such as a green dot and ignore an alternative red stimulus, the neural response at the frequency of the attended stimulus (8 Hz) will increase, while the neural response at the frequency of the ignored stimulus (9 Hz) will decrease compared to baseline.

[0051] In certain embodiments, this observation is applied to BCI. Thus, feature-based attention can be used to provide real-time volitional control over the selection of a target for triggering a corresponding output action. By overlaying two stimulation portions with different neural signatures at the same location on a given target (i.e., a display object), such as, for example, a green dot at 8 Hz and a red dot at 9 Hz on the letter A, the user is provided with the possibility to explore the letter A without making a selection, and then select the letter for the action by preferentially attending to the green dot on the letter. In some cases, the display object is a composite visual stimulus of two juxtaposed stimulation portions.

[0052] Consider the case where the green dots are part of the validation stimulus and the red dots are part of the balancing stimulus. In such an example, observing only the target (e.g., the letter 'A') will result in bottom-up (automatic) neural amplification of both frequencies associated with the validation stimulus and the balancing stimulus in a manner that can be interpreted as a balanced, unbiased neural response that reflects exploration of the target alone. This is because the balancing stimulus serves to counteract the bottom-up (automatic) neural influence of the validation stimulus.

[0053] However, when the user intentionally pays attention to the validating (eg, green) stimulus portion, neural amplification at the corresponding frequency can be interpreted as triggering volitional control of the action.

[0054] In other examples, the general approach of the present disclosure can be extended to any situation in which a validation stimulus is used in conjunction with one or more counterbalancing stimuli having a distinguishing neural signature, whether it be based on temporal features, spectral features, spatial features, or any other features of the neural data. Thus, the distinction between stimuli by which a user can indicate intent need not be color: for example, an alternative visible characteristic would be the rate or direction of movement of the visual element.

[0055] In another alternative, an overlay of green lines and an overlay of red lines can be presented superimposed on the letter A, both oscillating at a common frequency (e.g., 8 Hz), but in opposite phases. The same principle applies to the letter B at 9 Hz. Analysis of the neural responses will reveal whether the user is exploring the letter A (stronger neural oscillations at 8 Hz) or the letter B (stronger neural oscillations at 9 Hz). This frequency analysis can be combined with analysis of the phase of the neural responses to determine whether the user is just exploring (i.e., investigating) one of the letters or intending to verify it by focusing on the green overlay.

[0056] Note that in the above examples, the overlaid stimulus portions have characteristic modulations that differ only in phase and not in frequency.

[0057] An important aspect to consider is the fact that neural responses (as measured by EEG devices) are prone to nonlinearity, so when the user is visually exploring displayed objects rather than actively seeking to select one (i.e., when fixating on the letter A without demonstrating a preference for the validation stimulus over the counterbalance stimulus), the effects of the two stimulus components are not necessarily additive. This is true whether stimulus reconstruction or spectral analysis is used to decode the neural responses.

[0058] For spectrum analysis, there is a well-known phenomenon called intermodulation. The term intermodulation refers to any sum of non-zero integer multiples of the fundamental frequency (i.e., n*f1+m*f2, where n, m=-1,-2,-3,...). Intermodulation components occur as a result of nonlinear interactions between fundamental frequencies. For example, there may be a situation where the position between two closely placed display objects (e.g., modulated at frequencies f1 of 8 Hz and f2 of 9 Hz, respectively) is noted, with the letter "A" on the left side of the position and the letter "B" on the right side at the same time (so-called "double hidden attention"). In such a case, spectrum analysis will reveal power increases at both 8 Hz and 9 Hz, but also for their intermodulation components such as (f2-f1:1 Hz) and the sum (f1+f2:17 Hz). The size or presence or absence of intermodulation components can provide information about whether the user receives information from the two stimulation parts or pays priority attention to one of the two stimulation parts.

[0059] Successful identification of the attended stimulus may rely on the assumption that, by reconstructing the verification stimulus and the counterbalance stimulus and comparing them, the reconstruction is stronger for the verification stimulus than for the counterbalance stimulus. However, to address non-additive neural responses, certain embodiments are based on the use of more than one decoding model. In one such embodiment, two models are built, corresponding to attending to two stimuli versus one stimulus, respectively; 'attending to two stimuli' may mean that the user is 'defocused' or focused on the underlying object, but not on the distinguishing elements of either stimulus. In another such embodiment, two models are trained separately for each stimulus and for each stimulus. These embodiments can help distinguish between situations where the user attends to two stimuli versus no stimuli (e.g., during 'fixation') or attends to one stimulus versus another (e.g., during 'exploration verification' or 'selection verification'). Since different stimuli can have different effects on neural responses, both qualitatively (they may generate different VEP waveforms) and quantitatively (a particular stimulus may produce a weaker response amplitude in the brain than others), it may be a challenge to fully balance the verification and counterbalance stimuli when using only a single decoding model. By employing more than one model, it becomes possible to account for (and compensate for) neural response specificities and response biases.

[0060] As described above, the present disclosure is not limited to color-differentiated stimulus portions, but can be extended to any distinguishing visual features. Instead of using green and red overlapping stimulus portions, different movement directions (e.g., the constituent elements of each stimulus portion move coherently (i.e., "drift") in respective different directions, such as upward and downward, while occupying the same area of ​​the display), different spatial frequencies, contrasts, etc., can be used. Additionally or alternatively, visual distinction can be provided by using constituent elements of corresponding stimulus portions having different orientations (not just the direction of movement such as horizontal bars to vertical bars) and / or shapes (e.g., one or more small circles to one or more small triangles overlap at the center or on an object). Different distinguishing visual characteristics can be used in combination naturally. In addition, in some embodiments, corresponding stimulus portions can adopt visual distinction modes that are different from each other: therefore, a visual characteristic in a first stimulus portion can be said to be "orthogonal" to another visual characteristic in a second stimulus portion, because each characteristic is perceived and decoded independently of another characteristic. So that the two stimulus portions may look completely different. For example, consider the case where one stimulus component employs coherently colored lenticular components, while the other stimulus component employs none of these difference patterns and appears as a patch of visual noise.

[0061] Figure 5AAn exemplary embodiment of a composite visual stimulus according to the present disclosure is shown. In (a), various parts 502, 504 of the composite visual stimulus are shown, each part having a characteristic modulation of a different variation. These stimulus parts can be generated as different visual stimuli, but the BCI causes them to be displayed at least partially overlapping each other. Although the user can visually distinguish the corresponding parts by color, the corresponding parts can be distinguished in addition or alternatively by the shape and orientation of the stimulus component elements (as described above). For illustration, Figure 5A The composite visual stimulus shown in includes a portion having a 'check mark' symbol as a constituent element, and another portion having a 'cross mark' symbol. According to the present disclosure, the first portion, such as the 'check mark' portion 502, can be considered a 'verification stimulus', and the second portion, such as the 'cross mark' portion 504, can be considered a 'balance stimulus'. A user who focuses on the check mark of the verification stimulus 502 will produce a neural response that is conducive to verifying the selection of the displayed object under the composite visual stimulus.

[0062] Figure 5B Another exemplary embodiment of a composite visual stimulus according to the present disclosure is shown. In (a), various parts 506, 508 of the composite visual stimulus are shown, each with a characteristic modulation of a different variation. In this example, the shading of the "hook" part 506 is lighter than the shading of the "cross" part 508, illustrating the combination of visual characteristics (color / shading, and distinct constituent visual elements).

[0063] The verification stimulus portion and the balance stimulus portion of the composite visual stimulus do not need to be strongly modulated, but should elicit equivalent response amplitudes so as to balance each other when one of them is not preferentially attended by the user. If the two categories are biased one way or the other regardless of attention (e.g., when the verification stimulus has greater intensity than the other stimulus due to saliency, preference, familiarity, training, etc.), a normalization process can be used. In fact, as discussed below, it is noted that selectively applying feature modulation to the high spatial frequency (HSF) components of the visual stimulus portion can reduce discomfort and mental fatigue without significantly impairing the intensity of the neural response.

[0064] Figure 6Three different types of characteristic modulation (HSF components for corresponding multiple displayed visual stimuli) are shown. At (a), each of the first multiple different visual stimuli has a corresponding sinusoidal characteristic modulation of different frequency (but normalized peak amplitude). At (b), the characteristic modulation is again of different frequency and with normalized peak amplitude, but the modulation is a square wave. At (c), the characteristic modulation is instead a minimum correlated signal with normalized peak amplitude. In each case, the corresponding different stimuli can be applied to different display objects and / or (in pairs) to the same display object to provide verification stimulation and balance stimulation for that display object.

[0065] The method of presenting stimulus portions of a composite visual stimulus at the same location described in the present disclosure enables the BCI to capture attentional selection of one stimulus among two or more stimuli displayed at the same location, and enables the BCI to provide any desired form of feedback or action based on the stimulus.

[0066] The feedback may be a modification of the stimulus itself (e.g., visual highlighting or enlarging the displayed object itself, etc. or a visual stimulus overlaying the displayed object), or it may be an action associated symbolically or arbitrarily with the selected stimulus. For example, with an overlay of two arrows pointing in opposite directions (one arrow is green and the other arrow is red), attentional selection of one arrow will result in a corresponding movement in that direction in the virtual reality environment. This feedback provides a positive feedback loop (a "neural feedback" loop) that occurs when the user is presented with a visual representation of his real-time relevance for the target being attended to. The user's heightened attention to, for example, a conscious intention to "select the target" enhances the signal-to-noise ratio of the response to the stimulus, resulting in a positive feedback loop.

[0067] In co-pending International Patent Application No. PCT / EP2020 / 081338 (File No. 5380.005WO1), filed on November 6, 2020, the entire specification of which is incorporated herein by reference, the authors describe a method of feedback interaction in which a user is presented with a visual feedback element that exhibits an effect corresponding to their level of attention. Such feedback interaction can be used in conjunction with the currently disclosed operations. Thus, for example, the user will receive feedback and even a first verification (i.e., confirmation or "exploration verification") that the BCI has decoded (i.e., determined) that they are gazing at (or looking at) the letter A, but a second verification (i.e., "selection verification") only occurs when the green overlay is noticed in preference to the red overlay. Feedback can be continuous until verification of the selection is achieved.

[0068] Fig. 7AFIG. 7 shows a user experience of a display device 710 displaying a display object 704. In (a), a composite visual stimulus 706 (such as a Figure 5A ), and a composite visual stimulus 706 is overlaid on the display object 704. Each portion exhibits a respective, varying characteristic modulation that, when viewed by a human user, elicits a neural response encoding the modulation.

[0069] As shown in (b), when the user only looks at the display location of the composite visual stimulus 706 (i.e., the location of the display object 704), the evoked neural responses from each part are balanced (illustrated by showing both parts at the same opacity level). Although not shown here, the exploration currently focused on the display object 704 (i.e., the first verification) can optionally be shown by displaying a feedback element such as a "crosshair" symbol at or near the display object 704. In some embodiments, the visual characteristics of the feedback element (e.g., scale) can be mapped to the real-time attention level estimate of the neural decoder to close the neural feedback loop.

[0070] Figure 7B Continuing with the above description of the user experience, in (c), the user is now intentionally focusing on the checkmark portion 502 (here the validation stimulus) - depicted by showing only the checkmark portion 502, with the cross portion (the "balancing stimulus") visually de-emphasized.

[0071] Optionally, another feedback element may be displayed to confirm valid selection of displayed object 704 - here, represented by the emphasized appearance of that object in (d).

[0072] In each of the above embodiments, modulation can be applied preferentially or exclusively to high spatial frequency components of the projected overlay image (i.e., background and / or feedback elements). Preferential modulation of HSF components of overlay objects, target objects, and / or visual feedback elements can be used to improve the accuracy of the determination of the focused object (and reduce distraction effects).

[0073] Using the above method, real-time and accurate decoding of a user's focus becomes more practical. Performance still has obstacles: in particular, the possibility of interference between respective stimuli presented in the same display. A stimulus at one display location may fall within the peripheral vision of a user viewing a second stimulus at an adjacent display location.

[0074] Back to Figure 4 The effect of peripheral vision on the BCI described above can also be observed in the keyboard use case in FIG. A typical human visual system can be approximately considered to have a peripheral field of view 404 surrounding a narrower area 418 where the user's eyes focus. Figure 4, a display screen 402 showing a plurality of numbers 410, 412 in a keyboard is shown for subject 416. When the subject attempts to focus on the number "5" 410, other (i.e., peripheral) numbers (such as "3", 412) act as distractors, temporarily attracting the user's attention and causing interference in the user's visual system. This interference, in turn, hinders the performance of the BCI. For example, when the user attempts to focus on the number 5, other (i.e., peripheral) numbers act as distractors, temporarily attracting the user's attention and causing interference in the user's visual system. This interference, in turn, hinders the performance of the BCI.

[0075] Co-pending International Patent Application No. PCT / EP2020 / 081348 (File No. 5380.002WO1), filed on November 6, 2020, the entire specification of which is incorporated herein by reference, describes a method for addressing the challenge of determining a focused object (target) from objects in the periphery of the target (distractors) with speed and accuracy. This method relies on characteristics of the human visual system.

[0076] Studies of the way human visual perception operates have shown that when looking at a screen with multiple objects and focusing on one of these objects, the human visual system will accept both high spatial frequencies (HSF) and low spatial frequencies (LSF). Evidence shows that the human visual system is primarily sensitive to the HSF component of the specific display area that is focused (e.g., the object the user is staring at). In contrast, for peripheral objects, the human visual system is primarily sensitive to their LSF components. In other words, the picked-up neural signals will be substantially affected by both the HSF component from the focused target and the LSF component from the peripheral targets. However, since all objects cause a certain proportion of both HSF and LSF, processing neural signals to determine the focused object may be hindered by the LSF noise contributed by the peripheral objects. This tends to make the identification of the focused object less accurate and less timely.

[0077] As the human visual system is tuned to process multiple stimuli at different locations in the visual field in parallel, usually unconsciously, peripheral object stimuli will continue to trigger neural responses in the user's brain even if they appear at the periphery of the visual field. This will therefore create competition between multiple stimuli and make specific neural decoding of the focused object (target) more difficult.

[0078] In PCT / EP2020 / 081348, multiple objects are displayed in such a way that each object is divided into a version consisting only of the LSF component of the object and a version consisting only of the HSF component. The flickering visual stimulus used to elicit a decodable neural response is delivered only through the HSF version of the object. The flickering HSF version is superimposed on the LSF version (which does not flicker).

[0079] In some embodiments, the BCI described above can be used in conjunction with a real-world object to make the object controllable or otherwise interactive. In some embodiments, the generation of stimuli is handled by one or more light sources (e.g., light emitting diodes LEDs) disposed in association with (or on the surface of) a controllable object.

[0080] In certain embodiments, the generation of the stimulus is handled by a projector or a scanning laser device such that the visual stimulus is projected onto the controllable object and the controllable object outputs the visual stimulus by reflecting the projected stimulus.

[0081] As in the case of a BCI using a display screen through which a user interacts with on-screen objects, controllable objects in the present disclosure can be caused to present visual stimuli (e.g., flickering stimuli) with characteristic modulations so that neural responses to the presence of these stimuli become apparent and decodable from neural signals captured by a neural signal capture device (e.g., an EEG device).

[0082] In some embodiments, the determination of the focus of attention on the visual display of the controllable device is used to send a command to the controllable object. The controllable object can then implement an action based on the command: for example, the controllable object can emit an audible sound, unlock a door, turn on or off, change operating state, etc. The action can also provide visual feedback or other feedback associated with the controllable object to the user: this can be used in conjunction with the positive feedback loop discussed above, but can also provide real-time indication of the effective selection of an operation associated with the controllable object.

[0083] The co-pending international patent application No. PCT / EP2020 / 083088 (file No. 5380.007WO1), filed on November 23, 2020, the entire specification of which is incorporated herein by reference, describes how eye tracking technology can be used in conjunction with a brain-computer interface.

[0084] Figure 8 An exemplary embodiment of a composite visual stimulus according to the present disclosure is shown. Figure 5A Similar to the embodiment in (a), corresponding parts 806, 808 of the composite visual stimulus are shown, each part having a different characteristic modulation of the variation. Similarly, the user can visually distinguish the corresponding parts by the color, movement, shape and / or orientation of the stimulus component elements. Figure 8 In the diagram, these parts are shown as circular blocks or oval blocks instead of Figure 5A and Figure 5B Rectangular blocks. Figure 8'Check' and 'cross' symbols are used in the corresponding parts of the present invention, but this should not be construed as limiting as the reader will understand that any suitable visual differentiation pattern may be employed interchangeably.

[0085] Fig. 9 A system 900 combining eye tracking technology with a BCI in accordance with the present disclosure is shown.

[0086] In the illustrated optical tracking technique, the outputs of the respective eye tracking cameras 932, 934 are processed in an eye tracking unit 930. The eye tracking unit 930 outputs the eye tracking information 904 including the fixation information to a processing device 908. The processing device 908 (which may be Figure 2 The processing device 208) also receives the information at the neural response device 920 (e.g., Figure 2 The neural signal detected at the neural response device 206. Figure 2 As shown, the processing device 908 executes instructions for interpreting the received neural signals to determine feedback indicating the target object with the current (visual) attention focus in real time. The processing device 908 also generates image data presented on the display device 910, the image data including at least one visual stimulus overlaying the object displayed on the display device 910, the visual stimulus being a composite visual stimulus according to the present disclosure.

[0087] In some embodiments, the composite visual stimulation is applied only to objects in the field of view 906. Eye tracking information 904 can then be used to determine which objects are in the field of view, and the composite visual stimulation (such as Figure 8 Stimuli shown) can be used in accordance with the present disclosure to determine whether a user has verified intent to select one of the targets (and if so, to verify selection of that object).

[0088] Optionally, another feedback element may be displayed to confirm the valid selection of the displayed object - in Fig. 9 The field of view 906 is represented by a crosshair above the object.

[0089] Eye tracking information typically indicates the angle of the nominal gaze point relative to a fixed direction, such as a reference direction of the head. Conventional eye tracking techniques (even those that use cameras for each of the user's eyes) generate information that is essentially two-dimensional - points on a virtual sphere around the user's head can be distinguished, but depth is difficult to capture with any accuracy. Binocular eye tracking techniques are being developed that attempt to resolve the problem of different depths (i.e., distances from the user) using tracking information for more than one eye. Such techniques require extensive calibration for a particular user before they can be used with any reliability.

[0090] Fig.10A BCI system (eg, Figure 2 The main functional blocks in the method of operation of the BCI system shown in FIG. 1 are shown in FIG. 2. The brain-computer interface system includes a display unit, a stimulus generator, and a neural signal capture device. The display unit displays image data including at least one object and outputs a visual stimulus corresponding to one or more of the objects, the visual stimulus having a characteristic modulation.

[0091] In block 1002 , a hardware interface device (operably coupled to a neural signal capture device and a stimulation generator) such as interface device 208 receives neural signals from a neural signal capture device.

[0092] In block 1004, the interface device determines the strength of a component of the neural signal having a characteristic associated with a respective characteristic modulation of the or each visual stimulus.

[0093] In block 1006, the interface device determines, based on the neural signal, which of the at least one visual stimulus is associated with a focus object of the user, the focus object inferred from the presence and / or relative strength of a component of the neural signal having characteristics associated with a characteristic modulation of the visual stimulus.

[0094] In block 1008, the interface device causes the stimulus generator to generate the visual stimulus as a composite visual stimulus including at least a first portion having a first characteristic modulation and a first visual property and a second portion having a second characteristic modulation and a second visual property, the second visual property being different from the first visual property.

[0095] The active feedback (or "neural feedback") loop of certain embodiments of the present disclosure is associated with several benefits in terms of user experience (UX) and neural decoding. The feedback stimulus presents the user with a convenient guide to the user's attention (i.e., an "attention grabber") at a specific location in the display screen, helping the user to maintain focus on the object. For viewers with certain attention-related conditions and mild visual impairments, the presence of such a feature has been observed to help the user maintain focus.

[0096] In addition, the user is given a task (i.e., bringing the feedback stimulus close to a fully decoded state, indicating “validation” of the selection). This also helps the user focus on a specific object while suppressing peripheral distractors.

[0097] As the user becomes more focused using such feedback stimulation, it is observed that the user-specific stimulation reconstruction model established during the initial or calibration phase of BCI operation becomes more accurate and is built faster.

[0098] In subsequent operation phases, the use of feedback stimulation as described above results in improved accuracy and increased speed for real-time BCI applications.

[0099] Fig.11 is a block diagram illustrating an example software architecture 1106 that may be used in conjunction with the various hardware architectures described herein. Fig.11 is a non-limiting example of a software architecture, and it should be understood that many other architectures may be implemented to facilitate the functionality described herein. The software architecture 1106 may be implemented in hardware such as Fig.12 1200, which includes a processor 1204, a memory 1206, and an input / output (I / O) component 1218. A representative hardware layer 1152 is shown and may represent, for example, Fig.12 The machine 1200 of the embodiment of the present invention. The representative hardware layer 1152 includes a processing unit 1154 having associated executable instructions 1104. The executable instructions 1104 represent executable instructions of the software architecture 1106, including implementations of the methods, modules, etc. described herein. The hardware layer 1152 also includes a memory and / or storage module, shown as a memory / storage device 1156, which also has executable instructions 1104. The hardware layer 1152 may also include other hardware 1158, such as specialized hardware for interfacing with EEG electrodes, for interfacing with an eye tracking unit, and / or for interfacing with a display device.

[0100] exist Fig.11 In the example architecture of , software architecture 1106 can be conceptualized as a stack of layers, each of which provides specific functionality. For example, software architecture 1106 can include layers such as operating system 1102, library 1120, framework or middleware 1118, application 1116, and presentation layer 1114. Operationally, application 1116 and / or other components within a layer can call application programming interface (API) call 1108 through the software stack and receive a response as message 1110. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile operating systems or dedicated operating systems may not provide framework / middleware 1118, while other operating systems may provide such a layer. Other software architectures may include additional layers or different layers.

[0101] The operating system 1102 can manage hardware resources and provide public services. The operating system 1102 may include, for example, a kernel 1122, services 1124, and drivers 1126. The kernel 1122 may be used as an abstraction layer between the hardware and other software layers. For example, the kernel 1122 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services 1124 may provide other public services to other software layers. Drivers 1126 may be responsible for controlling or interfacing with the underlying hardware. For example, drivers 1126 may include display drivers, EEG device drivers, camera drivers, drives, flash drives, serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0102] The library 1120 may provide a common infrastructure that may be used by the application 1116 and / or other components and / or layers. The library 1120 generally provides functions that enable other software modules to perform tasks in an easier manner than by directly interfacing with the underlying operating system 1102 functions (e.g., kernel 1122, service 1124, and / or driver 1126). The library 1120 may include a system library 1144 (e.g., a C standard library) that may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. In addition, the library 1120 may include an API library 1146, such as a media library (e.g., a library to support the presentation and manipulation of various media formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG), a graphics library (e.g., an OpenGL framework that may be used to render 2D and 3D graphics content on a display), a database library (e.g., SQLite that may provide various relational database functions), a network library (e.g., WebKit that may provide network browsing functions), and the like. The library 1120 may also include various other libraries 1148 to provide numerous other APIs to the applications 1116 and other software components / modules.

[0103] Framework 1118 (also sometimes referred to as middleware) provides a higher level common infrastructure that can be used by applications 1116 and / or other software components / modules. For example, framework / middleware 1118 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. Framework / middleware 1118 can provide other broad spectrum APIs that can be used by applications 1116 and / or other software components / modules, some of which can be specific to a particular operating system or platform.

[0104] Applications 1116 include built-in applications 1138 and / or third-party applications 1140 .

[0105] Applications 1116 may create user interfaces to interact with users of the system using built-in operating system functionality (e.g., kernel 1122, services 1124, and / or drivers 1126), libraries 1120, or framework / middleware 1118. Alternatively or additionally, in some systems, interaction with the user may occur through a presentation layer such as presentation layer 1114. In these systems, the application / module "logic" may be separated from aspects of the application / module that interact with the user.

[0106] Fig.12 1 is a block diagram illustrating components of a machine 1200 according to some example embodiments, the machine 1200 being capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methods discussed herein. Specifically, Fig.12 A diagrammatic representation of a machine 1200 is shown in the form of an example of a computer system, within which instructions 1210 (e.g., software, programs, applications, applets, applications, or other executable code) for causing the machine 1200 to perform any one or more of the methods discussed herein may be executed. In this way, the instructions 1210 may be used to implement the modules or components described herein. The instructions 1210 convert a general-purpose, unprogrammed machine 1200 into a specific machine that is programmed to perform the functions described and shown in the described manner. In alternative embodiments, the machine 1200 operates as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1200 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1200 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a network device, a network router, a network switch, a network bridge, or any machine capable of executing, sequentially or otherwise, the instructions 1210 specifying actions to be taken by the machine 1200. In addition, while only a single machine 1200 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute the instructions 1210 to perform any one or more of the methodologies discussed herein.

[0107] The machine 1200 may include a processor 1204, a memory 1206, and an input / output (I / O) component 1218 that may be configured to communicate with each other, for example, via a bus 1202. In an example embodiment, the processor 1204 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1208 that may execute instructions 1210 and a processor 1212. The term "processor" is intended to include a multi-core processor that may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Fig.12 Multiple processors are shown, but the machine 1200 may include: a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0108] The memory 1206 may include a memory 1214 such as a main memory, static memory, or other memory storage device, and a storage unit 1216, both of which are accessible to the processor 1204, for example, via the bus 1202. The storage unit 1216 and the memory 1214 store instructions 1210 that embody any one or more of the methods or functions described herein. During execution of the instructions 1210 by the machine 1200, the instructions 1210 may also reside, completely or partially, within the memory 1214, within the storage unit 1216, within at least one of the processors 1204 (e.g., within a cache memory of a processor), or within any suitable combination thereof. Thus, the memory 1214, the storage unit 1216, and the memory of the processor 1204 are examples of machine-readable media.

[0109] As used herein, "machine-readable medium" means a device capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EEPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be deemed to include a single medium or multiple media (e.g., a centralized or distributed database or associated cache memory and server) capable of storing instructions 1210. The term "machine-readable medium" should also be deemed to include any medium or combination of multiple media that can store instructions (e.g., instructions 1210) executed by a machine (e.g., machine 1200) such that the instructions, when executed by one or more processors (e.g., processor 1204) of machine 1200, cause machine 1200 to perform any one or more of the methods described herein. Therefore, "machine-readable medium" refers to a single storage device or device as well as a "cloud-based" storage system or storage network comprising multiple storage devices or devices. The term "machine-readable medium" does not include the signal itself.

[0110] Input / output (I / O) components 1218 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific input / output (I / O) components 1218 included in a particular machine will depend on the type of machine. For example, user interface machines and portable machines such as mobile phones are likely to include a touch input device or other such input mechanism, while headless server machines are unlikely to include such a touch input device. It will be appreciated that input / output (I / O) components 1218 may include Fig.12 Many other components not shown.

[0111] The input / output (I / O) components 1218 are grouped according to function only for the purpose of simplifying the following discussion, and the grouping is by no means limiting. In various example embodiments, the input / output (I / O) components 1218 may include output components 1226 and input components 1228. The output components 1226 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), auditory components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 1228 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide location and / or force of touch or touch gestures), audio input components (e.g., microphones), etc.

[0112] In other example embodiments, the input / output (I / O) component 1218 may include various other components such as a biometric component 1230, a motion component 1234, an environment component 1236, or a position component 1238. For example, the biometric component 1230 may include a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves, such as output from an EEG device), identifying a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1234 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 1236 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The location component 1238 may include a positioning sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), etc.

[0113] A variety of technologies may be used to implement communications. Input / output (I / O) components 1218 may include communications components 1240 operable to couple machine 1200 to network 1232 or device 1220 via coupler 1224 and coupler 1222, respectively. For example, communications components 1240 may include a network interface component or other suitable device for interfacing with network 1232. In other examples, communications components 1240 may include wired communications components, wireless communications components, cellular communications components, near field communications (NFC) components, Parts (e.g. Low energy consumption), Components and other communication components that provide communication via other modalities. Device 1220 can be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via a universal serial bus (USB)). In the case where the EEG device, eye tracking unit, or display device is not integrated with machine 1200, device 1220 can be an EEG device, an eye tracking unit, and / or a display device.

[0114] Although described through some detailed exemplary embodiments, the portable device for acquiring electroencephalographic signals according to the present disclosure includes various variations, modifications and improvements that are obvious to those skilled in the art, and it should be understood that these various variations, modifications and improvements fall within the scope of the subject matter of the present disclosure as defined by the appended claims.

[0115] Although an overview of the inventive subject matter has been described with reference to specific example embodiments, various modifications and changes may be made to these embodiments without departing from the broader scope of the embodiments of the present disclosure. These embodiments of the inventive subject matter may be referred to herein individually or collectively by the term "invention", which is merely for convenience and is not intended to voluntarily limit the scope of the present application to any single disclosure or inventive concept if, in fact, more than one disclosure or inventive concept is disclosed.

[0116] The embodiments shown herein are described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, so that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Therefore, the specific embodiments should not be considered to have a limiting meaning, and the scope of the various embodiments is limited only by the appended claims and the full scope of equivalents to which such claims are assigned.

[0117] As used herein, the term "or" may be interpreted as inclusive or exclusive meaning. In addition, multiple instances may be provided for resources, operations, or structures described herein as single instances. In addition, the boundaries between various resources, operations, modules, engines, and data stores are somewhat arbitrary, and specific operations are shown in the context of specific illustrative configurations. Other allocations of functions are contemplated and other allocations of functions may fall within the scope of various embodiments of the present disclosure. Typically, structures and functions presented as independent resources in the example configuration may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as independent resources. These and other variations, modifications, additions, and improvements fall within the scope of the embodiments of the present disclosure as represented by the appended claims. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

[0118] Thus, the present disclosure describes systems and methods for improving the accuracy, speed performance, and visual comfort of BCIs.

[0119] Example

[0120] To better illustrate the systems and methods disclosed herein, a non-limiting list of examples is provided here:

[0121] 1. A brain-computer interface system, comprising:

[0122] a display unit for displaying image data, the image data comprising at least one object, the display unit further outputting corresponding visual stimuli corresponding to one or more of the objects,

[0123] a stimulus generator for generating the or each visual stimulus having a corresponding characteristic modulation;

[0124] a neural signal capturing device configured to capture neural signals associated with a user; and

[0125] an interface device operatively coupled to the neural signal capturing device and the stimulation generator, the interface device being configured to:

[0126] receiving a neural signal from the neural signal capturing device;

[0127] determining a strength of a component of the neural signal having a characteristic associated with a respective characteristic modulation of the or each visual stimulus;

[0128] determining, based on the neural signal, which of at least one of the visual stimuli is associated with a focus object of the user, the focus object being inferred from the presence and / or relative strength of a component of the neural signal having a characteristic associated with a characteristic modulation of the visual stimulus; and

[0129] The stimulus generator is caused to generate the visual stimulus as a composite visual stimulus comprising at least a first portion having a first characteristic modulation and a first visual property and a second portion having a second characteristic modulation and a second visual property, the second visual property being different from the first visual property.

[0130] 2. A system according to Example 1, wherein modulation is selectively applied to a high spatial frequency (HSF) component of the visual stimulus.

[0131] 3. A method of operating a brain-computer interface system, the brain-computer interface system comprising: a display unit, a stimulus generator, and a neural signal capture device, the display unit displays image data including at least one object, and outputs visual stimuli corresponding to one or more of the objects, the visual stimuli having a characteristic modulation,

[0132] The method comprises, in a hardware interface device operatively coupled to the neural signal capture device and the stimulation generator:

[0133] receiving a neural signal from the neural signal capturing device;

[0134] determining a strength of a component of the neural signal having a characteristic associated with a respective characteristic modulation of the or each visual stimulus;

[0135] determining, based on the neural signal, which of at least one of the visual stimuli is associated with a focus object of the user, the focus object being inferred from the presence and / or relative strength of a component of the neural signal having a characteristic associated with a characteristic modulation of the visual stimulus; and

[0136] The stimulus generator is caused to generate the visual stimulus as a composite visual stimulus comprising at least a first portion having a first characteristic modulation and a first visual property and a second portion having a second characteristic modulation and a second visual property, the second visual property being different from the first visual property.

[0137] 4. A method according to Example 3, wherein modulation is selectively applied to a high spatial frequency (HSF) component of the visual stimulus.

[0138] 5. A computer-readable storage medium for a brain-computer interface system, the brain-computer interface system comprising: a display unit, a stimulus generator, and a neural signal capture device, the display unit displays image data including at least one object, and outputs visual stimuli corresponding to one or more of the objects, the visual stimuli having a characteristic modulation,

[0139] The computer-readable storage medium carries instructions, which, when executed by a computer operatively coupled to the neural signal capture device and the stimulation generator, cause the computer to perform operations including:

[0140] receiving a neural signal from the neural signal capturing device;

[0141] determining a strength of a component of the neural signal having a characteristic associated with a respective characteristic modulation of the or each visual stimulus;

[0142] determining, based on the neural signal, which of at least one of the visual stimuli is associated with a focus object of the user, the focus object being inferred from the presence and / or relative strength of a component of the neural signal having a characteristic associated with a characteristic modulation of the visual stimulus; and

[0143] The stimulus generator is caused to generate the visual stimulus as a composite visual stimulus comprising at least a first portion having a first characteristic modulation and a first visual property and a second portion having a second characteristic modulation and a second visual property, the second visual property being different from the first visual property.

[0144] In addition, the present disclosure also provides the following configuration.

[0145] 1. A brain-computer interface system, comprising:

[0146] a display unit for displaying image data, the image data comprising at least one object, the display unit further outputting corresponding visual stimuli corresponding to one or more of the objects,

[0147] a stimulus generator for generating the or each visual stimulus having a corresponding characteristic modulation;

[0148] a neural signal capturing device configured to capture neural signals associated with a user; and

[0149] an interface device operatively coupled to the neural signal capturing device and the stimulation generator, the interface device being configured to:

[0150] receiving a neural signal from the neural signal capturing device;

[0151] determining a strength of a component of the neural signal having a characteristic associated with a respective characteristic modulation of the or each visual stimulus;

[0152] determining, based on the neural signal, which of at least one of the visual stimuli is associated with a focus object of the user, the focus object being inferred from the presence and / or relative strength of a component of the neural signal having a characteristic associated with a characteristic modulation of the visual stimulus; and

[0153] The stimulus generator is caused to generate the visual stimulus as a composite visual stimulus comprising at least a first portion having a first characteristic modulation and a first visual property and a second portion having a second characteristic modulation and a second visual property, the second visual property being different from the first visual property.

[0154] 2. The system according to configuration 1 further includes an eye tracking unit, which receives images captured by at least one eye tracking camera and determines eye tracking information based on the captured images, wherein the interface device is operatively coupled to the eye tracking unit; and wherein the interface device is further configured to: determine which of at least one of the visual stimuli is associated with the user's focused object based at least in part on the eye tracking information.

[0155] 3. A system according to configuration 2, wherein the eye tracking information includes at least one of gaze direction and / or focus depth.

[0156] 4. A system according to any one of configurations 1 to 3, wherein the interface device is further configured to: verify the selection of the focus object based on the composite visual stimulus.

[0157] 5. A system according to configuration 4, wherein the interface device verifies the selection of the focused object by determining which of the first part and the second part of at least one of the composite visual stimulation is the focused part, and verifying the selection when the determined part corresponds to a predetermined reference part, wherein the focused part is inferred based on the presence and / or relative strength of the following component of the neural signal, the component having characteristics associated with the first characteristic modulation or the second characteristic modulation and a second visual characteristic.

[0158] 6. A system according to any one of configurations 1 to 4, wherein modulation is selectively applied to a high spatial frequency (HSF) component of the visual stimulus.

[0159] 7. A method of operating a brain-computer interface system, the brain-computer interface system comprising: a display unit, a stimulus generator, and a neural signal capture device, the display unit displays image data including at least one object, and outputs visual stimuli corresponding to one or more of the objects, the visual stimuli having a characteristic modulation,

[0160] The method comprises, in a hardware interface device operatively coupled to the neural signal capture device and the stimulation generator:

[0161] receiving a neural signal from the neural signal capturing device;

[0162] determining a strength of a component of the neural signal having a characteristic associated with a respective characteristic modulation of the or each visual stimulus;

[0163] determining, based on the neural signal, which of at least one of the visual stimuli is associated with a focus object of the user, the focus object being inferred from the presence and / or relative strength of a component of the neural signal having a characteristic associated with a characteristic modulation of the visual stimulus; and

[0164] The stimulus generator is caused to generate the visual stimulus as a composite visual stimulus comprising at least a first portion having a first characteristic modulation and a first visual property and a second portion having a second characteristic modulation and a second visual property, the second visual property being different from the first visual property.

[0165] 8. A method according to configuration 7, wherein the interface device is operably coupled to an eye tracking unit, the eye tracking unit receives images captured by at least one eye tracking camera and determines eye tracking information based on the captured images, and wherein the method further comprises: determining which of at least one of the visual stimuli is associated with the user's focused object based at least in part on the eye tracking information.

[0166] 9. The method of configuration 8, wherein the eye tracking information comprises at least one of gaze direction and / or focus depth.

[0167] 10. The method according to any one of configurations 7 to 9 further includes: verifying the selection of the focus object based on the composite visual stimulus.

[0168] 11. A method according to configuration 10, wherein verifying the selection of the focused object includes: determining which of the first part and the second part of at least one of the composite visual stimulus is a focused part, wherein the focused part is inferred based on the presence and / or relative strength of the following component of the neural signal, wherein the component of the neural signal has characteristics associated with the first characteristic modulation or the second characteristic modulation and a second visual characteristic; and verifying the selection when the determined part corresponds to a predetermined reference part.

[0169] 12. A method according to any one of configurations 7 to 11, wherein modulation is selectively applied to a high spatial frequency (HSF) component of the visual stimulus.

[0170] 13. A computer-readable storage medium for a brain-computer interface system, the brain-computer interface system comprising: a display unit, a stimulus generator, and a neural signal capture device, the display unit displays image data including at least one object, and outputs visual stimuli corresponding to one or more of the objects, the visual stimuli having a characteristic modulation,

[0171] The computer-readable storage medium carries instructions, which, when executed by a computer operatively coupled to the neural signal capture device and the stimulation generator, cause the computer to perform operations including:

[0172] receiving a neural signal from the neural signal capturing device;

[0173] determining a strength of a component of the neural signal having a characteristic associated with a respective characteristic modulation of the or each visual stimulus;

[0174] determining, based on the neural signal, which of at least one of the visual stimuli is associated with a focus object of the user, the focus object being inferred from the presence and / or relative strength of a component of the neural signal having a characteristic associated with a characteristic modulation of the visual stimulus; and

[0175] The stimulus generator is caused to generate the visual stimulus as a composite visual stimulus comprising at least a first portion having a first characteristic modulation and a first visual property and a second portion having a second characteristic modulation and a second visual property, the second visual property being different from the first visual property.

[0176] 14. A computer-readable storage medium according to configuration 13, wherein the computer is operatively coupled to an eye tracking unit, the eye tracking unit receives images captured by at least one eye tracking camera device and determines eye tracking information based on the captured images, and wherein the instructions cause the computer to perform operations further comprising: determining which of at least one of the visual stimuli is associated with the user's focused object based at least in part on the eye tracking information.

[0177] 15. The computer-readable storage medium of configuration 14, wherein the eye tracking information comprises at least one of gaze direction and / or focus depth.

[0178] 16. The computer-readable storage medium of any one of configurations 13 to 15, wherein the instructions cause the computer to perform operations further comprising: verifying the selection of the focus object based on the composite visual stimulus.

[0179] 17. A computer-readable storage medium according to configuration 16, wherein verifying the selection of the focused object includes: determining which of the first part and the second part of at least one of the composite visual stimulus is a focused part, wherein the focused part is inferred based on the presence and / or relative strength of the following component of the neural signal, wherein the component of the neural signal has characteristics associated with the first characteristic modulation or the second characteristic modulation and a second visual characteristic; and verifying the selection when the determined part corresponds to a predetermined reference part.

Claims

1. A method comprising: generating composite visual stimuli having two or more distinguishable parts; displaying the composite visual stimulus to a user; determining, using a first decoding model, a first decoded neural signal captured from the user in response to a first portion of the composite visual stimulus, the first portion having a first visual characteristic; decoding, using a second decoding model, a second decoded neural signal captured from the user in response to a second portion of the composite visual stimulus, the second portion having a second visual characteristic that is distinguishable from the first visual characteristic; comparing the first decoded neural signal to the second decoded neural signal to determine the user's attention; as well as The attention of the user while defocused based on a balanced response of the first portion and the second portion of the composite visual stimulus is determined.

2. The method according to claim 1, wherein: The first portion and the second portion of the composite visual stimulus are displayed overlapping at a same display location.

3. The method according to claim 1, wherein: The first visual characteristic and the second visual characteristic differ in at least one of color, shape, orientation, or movement.

4. The method according to claim 1, in, The first visual characteristic comprises a first characteristic modulation and the second visual characteristic comprises a second characteristic modulation, the first characteristic modulation differing from the second characteristic modulation in at least one of frequency, phase or time code.

5. The method according to claim 1, further comprising: inferring that the user is exploring when the neural response is evenly distributed between the first part and the second part of the composite visual stimulus; as well as When the neural response is stronger for one of the first portion and the second portion of the composite visual stimulus, it is inferred that the user is intentionally paying attention.

6. A machine comprising: at least one processor; as well as at least one memory storing instructions that, when executed by the at least one processor, cause the machine to perform operations comprising: generating composite visual stimuli having two or more distinguishable parts; displaying the composite visual stimulus to a user; determining, using a first decoding model, a first decoded neural signal captured from the user in response to a first portion of the composite visual stimulus, the first portion having a first visual characteristic; decoding, using a second decoding model, a second decoded neural signal captured from the user in response to a second portion of the composite visual stimulus, the second portion having a second visual characteristic that is distinguishable from the first visual characteristic; comparing the first decoded neural signal to the second decoded neural signal to determine the user's attention; and The attention of the user while defocused based on a balanced response of the first portion and the second portion of the composite visual stimulus is determined.

7. The machine according to claim 6, wherein: The first portion and the second portion of the composite visual stimulus are displayed overlapping at a same display location.

8. The machine according to claim 6, wherein: The first visual characteristic and the second visual characteristic differ in at least one of color, shape, orientation, or movement.

9. The machine according to claim 6, in, The first visual characteristic comprises a first characteristic modulation and the second visual characteristic comprises a second characteristic modulation, the first characteristic modulation differing from the second characteristic modulation in at least one of frequency, phase or time code.

10. The machine according to claim 6, wherein: The operations also include: inferring that the user is exploring when the neural response is evenly distributed between the first portion and the second portion of the composite visual stimulus; and When the neural response is stronger for one of the first portion and the second portion of the composite visual stimulus, it is inferred that the user is intentionally paying attention.

Images