Brain-computer interface

Abstract

An adaptive calibration method in a brain-computer interface is disclosed. The method is used to reliably associate neural signals with objects that caused the neural signals by the attention of a user. Visual stimuli covering one or more objects are provided, at least a portion of the visual stimuli having a characteristic modulation. The brain-computer interface measures a neural response to the objects viewed by the user. The neural response to the visual stimuli is correlated with the modulation, the correlation being stronger when attention is focused on the visual stimuli. Based on the determined correlation, a weight is applied to a model of the resulting neural response of the user. Both the neural signal model weighting and the displayed object display modulation are adjusted in order to improve the certainty of the association of the neural signals with the objects that caused the signals.

Description

[0001] This application is a divisional application of Chinese Patent Application No. 202080069110.7, “Brain-Computer Interface” (Filing Date: 31 July 2020). TECHNICAL FIELD

[0002] The present invention relates to the operation of brain-computer interfaces. In particular, the present invention relates to the calibration of systems using brain-computer interfaces involving visual sensing. BACKGROUND

[0003] Certain brain-computer interface (BCI) based systems utilize a plurality of electrodes attached to the head that can detect changes in electrical properties caused by brain activity. While there are similarities in the neural signals produced by individuals due to these changes in electrical properties as individuals perform the same functions, there are still significant differences in the neural signal parameters between individuals. Therefore, calibration is required for systems that require reasonably accurate BCI results so that the neural signals can be reliably associated with the controls and tasks required by the system. In order to maintain accurate associations throughout the duration of BCI use, recalibration of the BCI can be required due to changes in electrode position and other factors that can affect the efficacy of the neural signal model. However, the calibration step can be both time consuming and disruptive. Therefore, a calibration method that can perform calibration adaptation to correct for changes in neural signals unobtrusively and quickly can provide multifaceted benefits by at least maintaining BCI accuracy and improving the user BCI experience.

[0004] Accordingly, it would be desirable to provide a brain-computer interface that addresses the challenges described above. SUMMARY

[0005] The present disclosure relates to a brain-computer interface system in which a computer controls or monitors a sensory stimulus that is perceptible to an individual when measuring brain activity of the individual. A model relating the sensory stimulus to brain activity (i.e. neural response) is then constructed and used to decode the neural signals of each individual. The model and sensory stimulus are updated (i.e. calibrated) to ensure a more reliable association between the stimulus and neural signals, thereby providing an improved user experience in the performance of graphical interface tasks. In certain embodiments, the BCI system is a visual BCI system and the sensory stimulus is a visual stimulus.

[0006] According to a first aspect, the present disclosure relates to a computer-implemented method, in at least one processor, the method comprising: in an initial phase, receiving, from a neural signal acquisition device, a first set of neural signals of a user, the user perceiving sensory information in a training sequence, the training sequence comprising at least one sensory stimulus, each sensory stimulus having at least one predetermined corresponding characteristic; determining, from the first set of neural signals and the training sequence, neural response data associated with each of the one or more sensory stimuli, the neural response data being combined to generate a model of the neural response of the user to the sensory stimuli, the model comprising weights applied to features of the neural signals; in a calibration phase, receiving, from the neural signal acquisition device, a second set of neural signals of the user, the user further perceiving the sensory information in a validation sequence, the validation sequence comprising at least one of the sensory stimuli; using the model, estimating which of the sensory stimuli is the object of attention of the user; determining whether the identification of the estimated object of attention corresponds to the sensory stimulus in the validation sequence; if it is determined that it corresponds, modifying the weights; and if it is determined that it does not correspond, modifying the sensory stimulus in the validation sequence.

[0007] According to a second aspect, the present disclosure relates to a brain-computer interface system comprising: a sensory information generation unit configured to output a signal for reproduction by a reproduction device, the reproduced signal being perceived by a user as sensory information; a neural signal acquisition device configured to acquire neural signals associated with the user; and a signal processing unit operably coupled to the sensory information generation unit and the neural signal acquisition device, the signal processing unit being configured to: in an initial phase, receive, from the neural signal acquisition device, a first set of neural signals of the user, the user perceiving sensory information in a training sequence, the training sequence comprising at least one sensory stimulus, each sensory stimulus having at least one predetermined corresponding characteristic; determine, from the first set of neural signals and the training sequence, neural response data associated with each of the one or more sensory stimuli, the neural response data being combined to generate a model of the neural response of the user to the sensory stimuli, the model comprising weights applied to features of the neural signals; in a calibration phase, receive, from the neural signal acquisition device, a second set of neural signals of the user, the user further perceiving the sensory information in a validation sequence, the validation sequence comprising at least one of the sensory stimuli; using the model, estimate which of the sensory stimuli is the object of attention of the user; determine whether the identification of the estimated object of attention corresponds to the sensory stimulus in the validation sequence; if it is determined that it corresponds, modify the weights; and if it is determined that it does not correspond, modify the sensory stimulus in the validation sequence. BRIEF DESCRIPTION OF DRAWINGS

[0008] For ease of reference to the discussion of any particular element or action, one or more most significant digits of a reference number refer to the figure number in which the element is first introduced.

[0009] Figure 1 An electronic architecture of a BCI system for receiving and processing EEG signals according to the present disclosure is depicted;

[0010] Figure 2 A system how it acquires neural signals that can be associated with a display object of interest is shown; Figure 1

[0011] Figure 3 A system how it acquires neural signals that are different from each other in time is shown when each display object has its own modulated display characteristic; Figure 1

[0012] Figure 4 A flowchart showing the flow of certain functional blocks in a calibration method according to the present disclosure is shown;

[0013] Figure 5 A flowchart showing the flow of certain functional blocks in another calibration method according to the present disclosure is shown;

[0014] Figure 6 An exemplary technique for building a decoding model according to the present disclosure is shown;

[0015] Figure 7 A flowchart showing the flow of certain main functional blocks in an operating method of a BCI after calibration has been performed according to the present disclosure is shown;

[0016] Figure 8 Various examples of display devices suitable for use with the BCI system of the present disclosure are shown;

[0017] Figure 9 is a block diagram showing a software architecture in which the present disclosure can be implemented according to some example embodiments; and

[0018] Figure 10 is a diagrammatic representation of a machine in the form of a computer system within which a set of instructions can be executed for causing the machine to perform any one or more of the methodologies discussed herein, according to some example embodiments. DETAILED DESCRIPTION

[0019] Brain-computer interfaces (BCIs) attempt to interpret measured brain activity in an individual to allow determination (i.e., inference) of the individual's focus of attention. The inferred focus of attention can be used to perform input or control tasks in an interface with a computer, possibly a component of the BCI. The computer controls or monitors sensory stimuli that are perceptible to the individual, such that the sensory stimuli can be related to the measured brain activity.

[0020] ​​In certain embodiments, the BCI system is a visual BCI system and the sensory stimuli are visual stimuli. In a visual BCI, neural responses to a target stimulus are typically used to infer (or "decode") which stimulus is the object of (visual attention) focus at any given time among a plurality of generated visual stimuli presented to the user. The object of focus can then be associated with an action that the user can select or control.

[0021] Similar provisions can be made for non-visual BCI systems, where the target stimuli used to infer focus can include auditory and tactile / touch stimuli.

[0022] Neural responses can be obtained using a variety of known techniques. One convenient approach relies on surface electroencephalography (EEG), which is non-invasive, has fine-grained temporal resolution, and is based on a well-understood empirical foundation. Surface electroencephalography (sEEG) can measure diffuse potential changes on the surface of a subject's skull (i.e., scalp) in real time. These potential changes are commonly referred to as electroencephalogram signals or EEG signals. Other techniques are of course available and can be used instead of (or in conjunction with) surface EEG - examples include intracranial EEG (iEEG) (also known as electrocorticography (ECoG)), magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI), functional near-infrared spectroscopy (fNIRS), etc.

[0023] In a typical visual BCI, a computer controls the display of visual stimuli in a graphical interface (i.e., display) generated and presented by a display device. The computer also generates a model of prior neural responses to the visual stimuli for decoding each individual's current neural signals.

[0024] Examples of suitable display devices (some of which are shown in Figure 8 include television screens and computer monitors 802, projectors 810, virtual reality headsets 806, display screens of interactive whiteboards and tablet computers 804, smart phones, smart glasses 808, etc. The visual stimuli 811, 811', 812, 812', 814, 814', 816 can form part of a generated graphical user interface (GUI), or they can be presented as augmented reality (AR) or mixed reality graphical objects 816 overlaid on a base image: this base image can simply be the actual field of view of the user (as in the case of a mixed reality display function projected onto a transparent display of a set of smart glasses) or a digital image corresponding to the user's field of view but captured in real time by an optical acquisition device that can in turn capture images corresponding to the user's field of view in other possible views.

[0025] It is fraught with difficulty to infer which of a plurality of visual stimuli, if any, is the object of attention at any given time. For example, when a user is faced with multiple stimuli, such as, for example, numbers displayed on a screen keyboard or icons displayed in a graphical user interface, it proves almost impossible to infer directly from the brain activity at a particular time which one is under attention. The user perceives a number under attention (say the number 5), and therefore the brain must contain information that distinguishes that number from the others, but current methods are unable to extract that information from the brain activity alone. That is, current methods can infer that a stimulus has been perceived, but they are unable to determine which particular stimulus is under attention using brain activity alone.

[0026] To overcome this problem and provide sufficient contrast between stimuli and background (and between stimuli), it is known to configure the stimuli used by the visual BCI to blink or pulse (i.e. so that each stimulus has a distinguishable characteristic profile over time). The blinking stimuli elicit a measurable electrical response. Particular techniques monitor different electrical responses, such as the steady-state visual evoked potential (SSVEP) and the P-300 event-related potential. In typical implementations, the stimuli are blinked at a rate in excess of 6 Hz. Thus, the visual BCI relies on a method that includes displaying the various stimuli discretely rather than continuously in the display device, and typically at different points in time. The brain activity associated with attention focused on a given stimulus is found to correspond (i.e. correlate) to one or more aspects of the temporal profile of that stimulus, such as the frequency of the stimulus blink and / or the duty cycle at which the stimulus alternates between a blinking state and a quiescent state.

[0027] Thus, the decoding of the neural signals relies on the fact that when a stimulus is activated, it will trigger a characteristic pattern of neural response in the brain that can be determined (e.g. picked up by the electrodes of the EEG device) from the electrical signals (i.e. the SSVEP or P-300 potential). This neural data pattern can be very similar or even identical for the various numbers or icons, but over time, for the number / icon that is perceived, the neural data pattern time-locks (i.e. synchronises) with the characteristic profile: only one number / icon can pulse at any time, such that the correlation of the pulsed neural response with the time (at which the number / icon is pulsed) can be determined to indicate that number / icon as the object of attention. By displaying each number / icon at different points in time, turning the number / icon on and off at different rates, applying different duty cycles, and / or simply applying the stimuli at different points in time, the BCI algorithm can determine which stimulus, when activated, is most likely to trigger a given neural response, thereby allowing the system to determine the object of attention. A timestamp of the data displayed on the display can be shared with the visual BCI and used to synchronise the visual stimuli with the corresponding neural responses. This is discussed further below.

[0028] More generally, modulation applied to the sensory stimulus is combined with a model of the neural response using previous neural responses to allow interpretation of the current neural response. The model provides a confidence value (i.e., model weight) with which signal features (in the neural response elicited by the sensory stimulus) can be associated with the object of focus.

[0029] The model of previous neural responses to visual stimuli is used to decode the current neural signal of each individual in an operation called "stimulus reconstruction".

[0030] Co-pending U.S. Patent Application 62 / 843,651 (Docket No. 3901.00.0001), filed May 6, 2019, describes one approach to the challenge of quickly and accurately determining an object of interest (target) from objects (interferers) in the target's periphery, the entire specification of which is incorporated herein by reference. This approach relies on properties of the human visual system.

[0031] Systems requiring reasonably accurate BCI results also require calibration of each individual's neural signal so that the neural signal can be reliably associated with the controls and tasks required by the system. Furthermore, due to variations in electrode position and other factors that can affect the efficacy of the neural signal model to maintain accurate associations throughout the duration of BCI use, the BCI can need to be recalibrated. However, the calibration step can be both time consuming and disruptive.

[0032] Accordingly, there is a need for a calibration method that can perform calibration adaptation that corrects for changes in neural signals unobtrusively and quickly, which can provide multifaceted benefits by at least maintaining BCI accuracy and improving the user BCI experience.

[0033] There are essentially two approaches to BCI calibration today: operant conditioning and "machine learning." Operant conditioning keeps the details of the hidden interface from the user and enables the user to slowly "learn" how to move screen objects and invoke tasks using brain activity through trial and error. Obviously, this is both frustrating and time consuming, so it is not the first choice for today's calibration needs.

[0034] With machine learning, the user interacts with one or more display targets on a display screen, focusing on (i.e., attending to) one by selecting it. In the case of multiple display targets, each display target can have a display feature unique to each object, such that attention to one particular target can elicit a response that "encodes" the unique display feature of that target, as described above.

[0035] For example, in the case of causing objects to flicker at different flicker rates, the object that the user is focusing on will stimulate a neural response and a signal that can be distinguished by signal characteristics that reflect the flicker rate of that object. A neural signal model can be constructed that is associated with that modulation and the signal characteristics resulting from that modulation, and then used to associate that modulation with neural signals stimulated thereby.

[0036] However, as previously mentioned, electrode positions can change during or between successive recording sessions, and external interference can add electromagnetic interference, noise, and other artifacts to the neural signals. In either case, the detected signals can no longer match the previously constructed model, and so the BCI can no longer link neural signals to displayed stimulus modulations with sufficient certainty. It is necessary to recalibrate the model of neural response to sensory stimuli.

[0037] Prior art BCIs related to visual BCIs use neural feedback or model weighting to attempt and adapt the BCI to changing neural signal conditions.

[0038] Neural feedback is a type of biofeedback in which neural activity is measured (e.g., by sEEG, MEG, iEEG / ECoG, fMRI, or fNIRS) and a sensory representation of that activity is presented to the user in order to self-regulate their mental state and / or adjust their behavior accordingly. In a neural feedback approach, information is presented to the user that they can use to train their interaction with the BCI, e.g., "learn" how to move screen objects and / or invoke tasks using brain activity. The feedback can be highlighting the object that is currently determined to be the object of focus by changing its color, size, display position, or other aspect of its appearance.

[0039] Model weighting in the context of a visual BCI refers to the training of a machine learning system. The weights are associated with predictions of user intent. In a traditional calibration procedure, signal features in the neural signal that most closely match neural signal patterns of known visual stimuli are given greater weight than those that are less reliable.

[0040] There are several methods that can extract a set of signal features from a single channel or a region of interest (i.e., a collection of channels whose electrodes originate from that region) from a neural signal. In the case of MEG or EEG, some examples of features originating from neural time series include: event-related potentials (ERPs), evoked potentials (which can be visual evoked potentials, auditory evoked potentials, sensory evoked potentials, motor evoked potentials), oscillatory signals (signal power in specific frequency bands), slow cortical potentials, brain state-related signals, etc.

[0041] In a visual BCI embodiment, the present disclosure builds on techniques for discriminating which (potentially) object of interest within a user's field of view (typically, but not always, on a display presented to the user) is the focus of visual attention. As described above, modulation of one or more of these objects causes the object to flicker or otherwise change visually, such that the modulation acts as a stimulus for a related neural response. In turn, the neural response can be measured and decoded to determine which object of interest is the focus of the user's attention. The object determined to be the focus of attention is then visually modified, such that the user understands the result of the decoding, and is encouraged to continue exhibiting the activity that prompted the determination, confirming (i.e., validating) the decoding (or to cease from that activity, marking an undesired decoding result). Either way, the strength of the association between the visual stimulus and the related neural response can be adjusted according to the accuracy of the decoding determined from the user's response to the modified focus of attention.

[0042] Certain embodiments of the method disclosed herein incorporate both techniques - changing the object modulation to obtain a more definitive association, while also changing the model weighting to obtain a better match. The model weighting and the modulation of the visual stimulus are updated (i.e., calibrated) in real-time to ensure a more reliable association between the (reconstructed) stimulus and the neural signal, thereby providing an improved user experience in the performance of the graphical interface task. This results in a "closed loop" or adaptive neural feedback that effectively changes the experimental task in real-time based on neural activity.

[0043] In addition, the disclosed method can employ neural signal filtering to remove artifacts that can confound the association model. Thus, the method can be selective in that it determines whether the neural signal is informative by first filtering out any artifacts (i.e., interference due to motion or background electromagnetic conditions, noise, etc.) before attempting to associate the signal with the modulation.

[0044] In the case where the display employs multiple display targets (with associated modulated stimuli), the adaptive method repeats the process of refinement and modulation association with each new display target of interest. Thus, the model is essentially refined by these new trials.

[0045] In certain embodiments of the calibration method disclosed herein, both the object modulation and the model weighting are refined. As a result, the calibration method is faster and more accurate than traditional calibration methods that keep one factor (such as the display target stimulus modulation) fixed. In addition, the described calibration method is suitable for a less obtrusive process, whereby the calibration can be performed in the background as the user employs the BCI for various applications, such as a game or a hands-free productivity tool.

[0046] In essence, the method creates a closed loop process in which real-time neurofeedback enhances the user's attention and focus, which in turn enhances the EEG signal caused by the user's attention and focus, which improves the accuracy of BCI decoding, which can increase the user's immersion.

[0047] In certain embodiments, an adaptive calibration method can be used with a system employing a BCI to enable hands-free system control. To detect brain activity, two of the most prominent methods involve electroencephalography (EEG), whereby electrodes placed on the scalp of a user (“surface EEG”) or directly inserted into the brain (i.e., “intracranial EEG,” iEEG, or electrocorticography, ECoG) will detect the electrical field changes resulting from neuronal synaptic activity. Every person's brain, while generally similar, is different. Two people exposed to the same stimulus will have different specific EEG results, but will generally show similarities in temporal and spatial response. Thus, any system, depending on decoding neural signals and associating them with specific system controls, needs to calibrate its operation to the individual's specific EEG neural signals.

[0048] The accuracy of system control is directly affected by the accuracy and certainty of the association of neural signals with intended system controls.

[0049] As noted above, in the early stages of BCI technology, experimenters often relied on operant conditioning, whereby users used trial and error to attempt to adjust the mental approach to system response. This often required long training times to achieve any success. As the technology developed, machine learning techniques were applied to map neural signal responses to specific end results. Thus, for example, if a user is shown a display screen with one object or one of a plurality of objects, and instructed to focus on one object. If a consistent neural signal is picked up when the user “focuses” on the object, then the signal can begin to be associated with that object, and so on.

[0050] However, for screens with multiple objects representing data or control calls, it is often difficult to associate neural signals with those objects with a high degree of certainty. As noted above, one approach to solving this difficulty is to cause the objects (referred to as “display targets” or stimuli associated with the target) to blink at different rates, with different light intensities, increased contrast changes, color shifts, geometric distortions, rotations, oscillations, displacement along a path, etc. As an example, the time difference between the on and off states of an object at different blink rates will be reflected in the detected neural signals. Thus, by looking at the times of various signal changes and associating them with blink rates, the signals can be associated with objects with a higher degree of certainty. Thus, modulating the display pattern to impart a distinction to each object display, while risking annoying the user, increases the certainty of associating signals with objects.

[0051] Those skilled in the art of BCI know of a variety of methods for representing neural signals and processing neural signals to produce reliable models. They also know of a variety of methods for modulating the displayed objects to create easily distinguishable differences in the neural signal patterns they elicit. Thus, when the term "modulate" or any variant of that term is used, it should be interpreted broadly to include any known way of modulating the display characteristics of the displayed objects.

[0052] Likewise, when the term "signal processing" or any variant of that term is used, it should be interpreted broadly to include any known way of processing neural signals. When the term "filter" or any variant of that term is used, it should be interpreted broadly to include any known method of filtering neural signals to mitigate or eliminate artifacts such as neural signals or brain rhythms unrelated to the object of interest.

[0053] Furthermore, system implementations that support brain-computer interaction are also well known in the art. The most common involve headsets with distributed electrodes that allow for the parallel detection of electric field changes as the user engages in an activity.

[0054] In certain embodiments according to the present disclosure, the operation of the BCI includes a brief, initialization, and calibration phase. Since users can vary significantly in their baseline neural responses to the same stimulus (particularly people with impaired or damaged visual cortex), the calibration phase can be used to generate a user-specific stimulus reconstruction model. This phase can take less than a minute to construct (typically, about 30 seconds).

[0055] Embodiments according to the present disclosure implement a method for "adaptive calibration" of a system employing a BCI. The method allows for the need to recalibrate due to distracting user focus, changes in headset / electrode position, "noisy" signal conditions, etc., that can introduce artifacts into the signal. Thus, the method is adaptive and is designed to allow for the unobtrusive repetition of calibration with minimal user distraction when necessary, rather than initiating a single calibration event and risking loss of certainty due to changes. The method also utilizes changes in model weighting and object display modulation to refine and improve neural signal modeling for improved operational reliability.

[0056] Figure 1An example of an electronic architecture for receiving and processing neural signals by means of a BCI device 100 according to the present disclosure is shown. A user of a head-mounted electrode headset 101 gazes at a display 107 having one or more objects and notices one object 103. The headset detects the electric field's substantial time-varying changes and each electrode provides an output signal that varies over time. In total, the combination of electrodes provides a set of time-parallel electrode signals. The individual electrodes are fed to an EEG acquisition unit (EAU, 104) that transmits its results to a signal processing unit (SPU, 105) whose processing results are used to control a display generation unit (DGU, 106). The DGU 106 in turn controls the presentation of image data on the display 107. The DGU can also provide timestamp information 102 of the image data. It is essentially a closed-loop system, as any changes made by the DGU 106 will affect the display 107 and be fed back through the user's neural response.

[0057] To measure the diffuse potentials on the surface of the subject's skull, a surface EEG device comprises a portable device (i.e. a cap or a headset). In Figure 1 the portable device is exemplified as an electrode headset 101. Figure 1 The portable device 101 comprises one or more electrodes 108, typically between 1 and 128 electrodes, advantageously between 2 and 64, advantageously between 4 and 16.

[0058] Each electrode 108 can comprise a sensor for detecting electrical signals generated by the subject's neuronal activity and an electric circuit for pre-processing (e.g. filtering and / or differential amplification) the detected signals before analog-to-digital conversion: the electrode is said to be "active". In Figure 1 an active electrode 108 is shown in use, with the sensor in physical contact with the subject's scalp. The electrode can be adapted for use with a conductive gel or other conductive liquid (referred to as a "wet" electrode) or without such liquid (i.e. a "dry" electrode).

[0059] The EAU 104 can comprise analog-to-digital conversion (ADC) circuits and a microcontroller. Each ADC circuit is configured to convert the signals of a given number of active electrodes 108, for example between 1 and 128.

[0060] The ADC circuits are controlled by the microcontroller and communicate with it, for example through the protocol SPI ("Serial Peripheral Interface"). The microcontroller packs the received data for transmission to an external processing unit (the SPU 105 in Figure 1 ). The SPU can for example be a computer, a mobile phone, a virtual reality headset, a game console, an automotive or an aeronautical computer system, etc.

[0061] In certain embodiments, each active electrode 108 is powered by a battery (not shown). The battery is conveniently disposed in the housing of the portable device 101. Figure 1

[0062] In certain embodiments, each active electrode 108 measures a respective potential value from which the potential measured by the reference electrode is subtracted (Ei = Vi-Vref) and the difference is digitized by means of an ADC circuit and then transmitted by the microcontroller.

[0063] In certain embodiments, the DGU 106 uses the results of the processing from the SPU 105 to change a target object for display in a graphical user interface of a display device, such as the display 107. The target object can include a control item, and the control item in turn is associated with a user-selectable action.

[0064] In the system 100 of Figure 1 , an image is displayed on the display of the display device 107. The subject views the image on the display 107, focusing on the target object 103.

[0065] In an embodiment, the display device 107 displays the target object 103 as a graphical object with a varying temporal characteristic that is different from the temporal characteristics of other displayed objects and / or the background in the display. The varying temporal characteristic can for example be a constant or time-locked flashing effect that changes the appearance of the target object at a rate greater than 6 Hz. In another embodiment, the varying temporal characteristic can use a pseudo-random temporal code so that an average (e.g. at an average rate of 3 Hz) of several changes per second of the appearance of the target object is generated. In the case of more than one graphical object being a potential target object (i.e. providing the viewing subject with a choice of target objects to focus on), each object is associated with a discrete spatial and / or temporal code.

[0066] In the system 100 of Figure 2 , a representation of the parallel occurrence signals 201 detected by the electrodes in the electrode headgear 101 as acquired at the EAU 104 is shown. The respective signal traces for each electrode response are shown vertically spaced apart but sharing the same time axis: so that time extends in the horizontal direction, and the height of a trace above or below the horizontal line represents the intensity of the response of that electrode at that time. The response to the most relevant signal features can be refined using any of several well-known signal processing methods.

[0067] In the system 100 of Figure 3 ​In contrast to the signal pattern 201 associated with the user's object of interest 103 and the signal pattern 301 associated with the user who is instead focusing on a second object 103', and the flicker rates of the objects 103 and 103' such that 103 flickers at time Tl and 103' flickers at time T2, the change can be distinguished by comparing the times at which the corresponding stimulus responses begin. This is just one way of modulating the graphical properties of the objects in order to uniquely associate a given EEG signal / segment to the object that is being focused on.

[0068] The EAU 104 detects neural responses (i.e., tiny electric potentials indicative of brain activity in the visual cortex) associated with the attention of the user to the object of interest; thus, the visual perception of the changing temporal characteristics of the object of interest acts as a stimulus in the subject's brain, generating a specific brain response corresponding to the code associated with the object of interest that is being focused on. The detected neural responses (e.g., electric potentials) are then converted to digital signals and transmitted to the SPU 105 for decoding. The sympathetic nervous response in which the brain appears to "oscillate" or respond in synchrony with the flashing temporal characteristics is referred to herein as "neural synchrony."

[0069] The SPU 105 executes instructions that interpret the received neural signals to determine feedback in real time that indicates which object of interest is the current (visual) focus of attention. Decoding the information in the neural response signals relies on a correspondence between that information and one or more aspects of the temporal profile of the object of interest (i.e., the stimulus). In the example of a simple flickering object, the information in the neural response signal is the timing of the response to the stimulus (i.e., the object of interest). In the example of a simple flickering object, the information in the neural response signal is the timing of the response to the stimulus (i.e., the object of interest). Figure 3 In the example, the signal patterns 201 and 301 represent simple models of the signal patterns associated with the objects of interest 103 and 103', respectively: the received neural signals (here, according to timing) can be compared to determine which pattern is the closest match to the received neural signals).

[0070] In certain embodiments, the SPU 105 and the EAU 104 can be provided in a single device, such that the decoding algorithm is performed directly on the detected neural responses.

[0071] In certain embodiments, the DGU 106 can facilitate the generation of image data that presents the object of interest as changing over time on the display device 107. In certain embodiments, the SPU 105 and the DGU 106 can be provided in a single device, such that information about the determined focus of visual attention can be incorporated into the generation (and modulation) of the visual stimulus in the display.

[0072] In certain embodiments, the display device 107 displays the overlay object as a graphical object with changing temporal characteristics that are different from the temporal characteristics of other displayed objects and / or the background in the display, and then displays the overlay object as a graphical layer over at least one identified object of interest.

[0073] The visual stimulus (i.e. the time-varying target object or overlay object) can provide retrospective feedback to the user, thereby validating their selection. The visual feedback can conveniently be presented to the user on the display screen 107, so that they know that the target object 103 was determined to be the current focus of attention. For example, the display device can display an icon, cursor or other graphical object or effect near the target object 103, highlighting (e.g. overlaying) the object that appears to be the current focus of visual attention. This provides a positive feedback loop, in which the apparent target object is confirmed (i.e. validated) as the intended target object by prolonged attention, and the association determined by the user's neural response model is reinforced.

[0074] Figure 4 is a flowchart illustrating a flow of certain functional blocks in accordance with embodiments of the present disclosure. As shown, the flow starts with an initial sequence (operations 401-407).

[0075] At operation 401, the objects A, B and C are displayed using the DGU 106 of the BCI 100, e.g.: Figure 1 These can be, for example, three shapes shown on the display unit in Figures 1-3 The user then focuses on object A (operation 402). Substantially simultaneously, the neural signal is acquired (operation 403). The acquisition of the neural signal can be performed by the EAU 104.

[0076] The SPU 105 optionally makes a decision about whether the signal is informative, e.g. whether the signal quality is good enough for predicting the user's focus of attention (operation 404). The presence of artefacts (e.g. interference or noise) in the signal can cause it to lack informativeness, as can a lack of sufficiently stable attention / gaze to a single object.

[0077] Although not shown in Figure 4 , a determination that the signal is not informative (operation 404, no) can trigger a reduction in the number of objects presented to the user (simplifying the task of classifying the neural signal) and / or a change in the appearance of the objects (e.g. increasing the size of the stimulus to ensure that the user sees it). In such a case, the flow returns to the display of the objects of the initial sequence (operation 401).

[0078] The user can be provided with feedback about the accuracy / confidence with which the signal can be used to predict the user's focus of attention (operation 405). For example, the feedback to the user can take the form of a graphical, textual, haptic or audible message. Thus, for example, if the signal is not informative (no), the user is informed, e.g. instructed to reduce blinking or stop saccading away from the objects. Optionally, the user can be informed that the signal does appear to be informative (yes), e.g. providing positive feedback to confirm that the user is making progress.

[0079] The test as to whether the signal is informative can be a comparison to a threshold (e.g., a threshold signal-to-noise ratio, etc.). However, it is equally conceivable that the test can be implemented by one or more algorithms specifically designed to predict whether a signal is informative (and label accordingly). For example, the algorithm can use an adaptive Bayesian classifier or a linear discriminant analysis (LDA) classifier. In fact, some classifiers such as LDA or Bayesian classifiers are completely determined by the mean and variance of the BCI data from each class (e.g., "informative," "non-informative") and the number of samples of each class, which is updated incrementally and robustly with new input data without the need to know the class labels. This is a pseudo-supervised approach, where the posteriori labels estimated by the Bayesian / LDA classifier are used during the adaptation process. When the probability that the signal is clean is not high enough, the wrongly estimated labels corrupt the parameter adaptation, similar to noise or outliers in supervised learning.

[0080] Once the signal is deemed informative (operation 404, YES), the model weights can be set (operation 406). The model weights are discussed below, one particular example being the parameters of the stimulus reconstruction algorithm described in the context of Figure 6 The signal is associated with object A (operation 407), meaning that the user's focus of attention is determined to be on object A with the current signal.

[0081] By confining the user to a single stimulus (object A) at a time, the model can ensure that the association between the signal and the object is correct. This process can be repeated, asking the user to focus on objects B and C in turn. In effect, the user is given fewer choices by displaying only one object at a time in the initial sequence. The resulting multiple weights form a decoding model for decoding the focus of attention.

[0082] Again, the user can be presented with one or more objects at a time, while using another method of targeting to determine the true association between the focus of attention and the signal (e.g., using a short supervised training session that tracks the user's eye movements).

[0083] In these cases, the model weights are set using reliable knowledge of the association. This embodiment is suitable for cases where no prior information is available, and can be characterized as a "cold start" or "from scratch" embodiment.

[0084] The calibration phase can also be initiated by using predetermined weights, and then adjusting these weights according to their accuracy for the current user / electrode positioning, etc. These weights can be obtained, for example, from previous sessions of that user, or from an "average" model obtained by training an algorithm on a database of previous EEG recordings. Although it is not expected that a model based on predetermined weights will perfectly fit the neural signals of the current user, it is found that these models provide a valid approximation on which the calibration method of the present disclosure can build. Such embodiments can be characterized as "warm start" embodiments. In certain embodiments, the use of predetermined weights can allow the initial sequence to be replaced with calibration (operations 401-407) by a step of retrieving from storage a model with predetermined (i.e. default values or average values using old user's weighted data and / or weights), effectively making the initial sequence optional.

[0085] The confirmation sequence is then executed (operations 408-417), verifying the accuracy of the decoding model.

[0086] At operation 408, the modulation of object A is modified. The display (e.g. by DGU 106) is caused to reflect this modification (operation 409). As the user continues to focus on object A (operation 410), a confirmation neural signal is acquired (operation 411). The quality of the confirmation neural signal is optionally determined (operation 412) using the same optional procedure as was used to determine the "informative" status of the neural signal at operations 404 and 405.

[0087] The user can again be provided with feedback regarding the accuracy / confidence with which the signal can be used to predict the user's focus of attention (operation 413, similar to operation 405). Thus, if the confirmation neural signal is determined to be "non-informative" (NO), the user can be informed that the confirmation sequence will continue. The acquisition of the confirmation neural signal continues until a decision is made (by SPU 105, for example) that the confirmation neural signal is informative (operation 412, YES). This positive outcome can also be provided to the user (operation 415).

[0088] The model weights can be adjusted (operation 416) to reflect the neural signal that produced the positive outcome, and the association between the confirmation signal and object A is now confirmed (operation 417), so that the model can be reliably used to identify the focus of attention in subsequent operations of the BCI.

[0089] One key aspect of this calibration method is to have real-time feedback to improve the focus of attention. It is found that receiving real-time feedback can motivate the user to pay more attention, thereby improving accuracy, thereby significantly improving user satisfaction. The confirmation sequence can be applied at intervals to ensure that the calibration remains reliable.

[0090] In Figure 5 , a method of calibrating a BCI is shown. Figure 4The same procedure embodiment, except that in the operation of acquiring the neural signals, the neural signals are now also filtered in both the initial sequence (operation 501) and the confirmation sequence (operation 502).

[0091] Figure 6 One technique for constructing the decoding model is shown, in which the model weights can be set and adjusted (as done in operation 406). The decoding model is configured to decode the focus of attention of the user whose neural responses are modeled. The decoding model is trained to "reconstruct" one or more time-varying features of the attended sensory stimulus (e.g. the target object), hereinafter referred to as the "modulation signal". The time-varying features can be a change in luminance, contrast, color, frequency, size, position, or any stimulus feature whose influence can be observed directly or indirectly on the recorded neural signals.

[0092] As mentioned above, an initial phase (operation 602) is performed in which a set of neural signals Xi is obtained for each of N target objects (where i is a member of [1,...,N]), and a time-varying stimulus feature SM i is known.

[0093] In operation 604, the neural signals X i are synchronized with the corresponding time-varying stimulus feature SM i The time-varying stimulus feature SM i conveniently provides a timestamp or synchronization pattern.

[0094] During a pre-processing step (operation 606), the raw neural signals Xi are optionally denoised to optimize the signal-to-noise ratio (SNR). In the case of EEG, the data can be heavily contaminated by a variety of noise sources: external / exogenous noise can originate from electronic artifacts (e.g. 50 Hz or 60 Hz line noise), while biological / endogenous noise can come from muscle artifacts, blinks, electrocardiogram, task-unrelated brain activity, etc.

[0095] In one or more embodiments, the neural signals X1, X2,..., X N may be denoised prior to reconstructing the modulation signal of the attended stimulus. For example, this denoising step can include a simple high-pass filter around 40 Hz to remove high-frequency activity including line noise from the signals X1, X2,..., X N

[0096] Multivariate methods such as Principal Component Analysis (PCA), Independent Component Analysis (ICA), Canonical Component Analysis (CCA), or any variant thereof can also be used, allowing to separate "useful" neural signal components (i.e. originating from task-related brain activity) from less relevant components.

[0097] ​During a subsequent step (operation 608), the reconstructed model parameters are estimated. This estimation can be done in a way that minimizes the reconstruction error. The reconstruction method can take the form of a plurality of parametric combinations of the neural signals. These combination parameters are determined from a mathematical equation analytical method to estimate the optimal parameters of the combination, i.e. for a given stimulus, the plurality of neural signals X i,j The values a of the reconstruction model are applied j are recorded, yielding a reconstructed modulating signal MSR that optimally corresponds to the modulating signal of the stimulus of interest, i.e. the value with the minimum reconstruction error. In this context, the term "weights" refers to a linear or non-linear combination of the EEG channel data that maximizes the similarity with the stimulus of interest.

[0098] In one or several embodiments, the values a j may be fixed (i.e. independent of time).

[0099] In other embodiments, these values (model weights) can be adjusted in real time in order to take into account possible adaptation of the user's neural activity, or possible variations of the SNR during the recording session.

[0100] In certain embodiments, the reconstruction model is a linear model that yields the modulating signal MSR by a linear combination of the neural signals X i,j In this case, the combination parameters are the linear combination coefficients a and the mathematical equation is a linear equation, and the reconstructed modulating signal MSR is computed from the linear combination of the neural signals X1, X2,..., X N using the following formula:

[0101] MSR = a1X1+ a2X2+... + anXn

[0102] In alternative embodiments, more complex models can be used. One class of such models consists in using neural networks, where the modulating signal MSR is obtained by a non-linear mathematical operation on a concatenation of the neural signals X1, X2,..., X N .

[0103] After the stimulus has been "reconstructed", it can be compared to all the different stimuli presented to the user. The stimulus of interest (target) corresponds to the stimulus whose temporal variation features SM are the most similar to the reconstructed MSR.

[0104] For example, a convolutional network can be trained to match any input neural signal X to a modulation signal MSR, such that for two neural signals X1 and X2 recorded at different points in time, two modulation signals MSR1 and MSR2 are produced, which are similar when the user's attention is focused on the same target, and different when the user's attention is focused on different targets. Several mathematical definitions of similarity can be used (for example, it can be a simple Pearson correlation coefficient, or the inverse of the Euclidean distance, mutual information, etc.). The reconstructed modulation signal is a multi-dimensional signal R generated by the neural network from newly acquired EEG data.

[0105] The above-described BCI can be used in conjunction with a real-world object, making the object controllable or otherwise interactable. In certain embodiments, the generation of the stimulus is handled by one or more light sources (such as light-emitting diodes, LEDs) provided in association with (or even on the surface of) the controllable object.

[0106] In certain embodiments, the generation of the stimulus is handled by a projector or a scanning laser device, such that the visual stimulus is projected onto the controllable object and the controllable object outputs the visual stimulus by reflecting the projected stimulus.

[0107] As in the case of the BCI using a display screen, through which the user interacts with an object on the screen (for example, in Figure 1 The controllable object in the present disclosure can be made to present visual stimuli (for example, flickering stimuli) with characteristic modulations, such that the neural responses to the presence of these stimuli become apparent and decodable from the neural signals acquired by a neural signal acquisition device, such as an EEG device.

[0108] In certain embodiments, determining the focus of attention on the visual display of the controllable device is used to send a command to the controllable object. The controllable object can then perform an action based on the command: for example, the controllable object can emit a sound, unlock a door, turn on or off, change operating state, etc. The action can also provide visual or other feedback associated with the controllable object to the user: this can be used in conjunction with the positive feedback loop discussed above, but can also provide a real-time indication of the effective selection of an operation associated with the controllable object.

[0109] As the user becomes more focused using the feedback stimuli, it is observed that the user-specific stimulus reconstruction model built in the initial or confirmation phase of the operation of the BCI system is more accurate, while being built faster.

[0110] In the subsequent operational phase, the use of the feedback stimuli as described above leads to an increase in the accuracy and speed of the real-time BCI application. When the initial or confirmation calibration step is not performed, the visual BCI system can be used according to Figure 7The function box shown is for operation.

[0111] In box 702, it is operatively coupled to a neural signal acquisition device and a stimulation generator (such as...) Figure 1 Hardware interface devices (such as EAU and DGU) Figure 1 The SPU receives neural signals from the neural signal acquisition device.

[0112] In block 704, the interface device determines the intensity of a component of a neural signal having characteristics associated with the modulation of the visual stimulus or a corresponding characteristic of each visual stimulus.

[0113] In box 706, the interface device determines, based on neural signals, which of at least one visual stimuli is associated with the user’s object of interest by comparing the modulation of the corresponding characteristics of the at least one visual stimulus with the modulation of the reconstructed sensory stimulus, and reconstructs the reconstructed sensory stimulus using the received neural signals, wherein the object of interest is the visual stimulus whose temporal variation characteristics are most similar to the reconstructed sensory stimulus.

[0114] Figure 9 This is a block diagram illustrating example software architecture 906, which can be used in conjunction with various hardware architectures described herein. Figure 9 This is a non-limiting example of software architecture, and it should be understood that many other architectures can be implemented to facilitate the functionality described herein. Software Architecture 906 can be applied in various ways, such as... Figure 10 The machine 1000 executes on hardware including a processor 1004, a memory 1006, and input / output (I / O) components 1018, etc. A representative hardware layer 952 is shown and can represent, for example... Figure 10 The machine 1000. A representative hardware layer 952 includes a processing unit 954 having associated executable instructions 904. The executable instructions 904 represent executable instructions of the software architecture 906, including implementations of the methods, modules, etc., described herein. Hardware layer 952 also includes a memory / storage module, shown as a memory and / or storage device 956, which also has executable instructions 904. Hardware layer 952 may also include other hardware 958, such as dedicated hardware for interfacing with EEG electrodes and / or for interfacing with display devices.

[0115] exist Figure 9In the example architecture of FIG. 9, the software architecture 906 can be conceptualized as a stack of layers where each layer provides particular functionality. For example, the software architecture 906 can include layers such as an operating system 902, libraries 920, frameworks or middleware 918, applications 916, and a presentation layer 914. Operationally, the applications 916 and / or other components within the layers can invoke application programming interface (API) calls 908 through the software stack, and receive a response proximate as a message 910. The layers illustrated are representative, and not all software architectures have all layers. For example, some mobile or special purpose operating systems can not provide a framework / middleware 918, while others can provide such a layer. Other software architectures can include additional or different layers.

[0116] The operating system 902 can manage hardware resources and provide common services. The operating system 902 can include, for example, a kernel 922, services 924 and drivers 926. The kernel 922 can act as an abstraction layer between the hardware and the other software layers. For example, the kernel 922 can be responsible for memory management, processor management (for example, scheduling), component management, networking, security settings, and so on. The services 924 can provide other common services for the other software layers. The drivers 926 can be responsible for controlling or interfacing with the underlying hardware. For instance, the drivers 926 can include display drivers, EEG device drivers, camera drivers, Bluetooth® drivers, flash memory drivers, serial communication drivers (for example, Universal Serial Bus (USB) drivers), Wi-Fi® drivers, audio drivers, power management drivers, and so on, depending on the hardware configuration.

[0117] The libraries 920 can provide a common infrastructure that can be used by the applications 916 and / or other components and / or layers. The libraries 920 can typically provide functionality that allows other software modules to perform tasks without having to perform the same functions themselves. The libraries 920 can include system libraries 944 (for example, C standard library) that can provide functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. In addition, the libraries 920 can include API libraries 946 such as media libraries (for example, libraries to support presentation and manipulation of various media formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG), graphics libraries (for example, an OpenGL framework that can be used to render 2D and 3D graphics on a display), database libraries (for example, SQLite that can provide various relational database functions), web libraries (for example, WebKit that can provide web browsing functionality), and the like. The libraries 920 can also include a wide variety of other libraries 948 to provide many other APIs to the applications 916 and other software components / modules.

[0118] The frameworks 918 (also sometimes referred to as middleware) provide a higher-level common infrastructure that can be used by the applications 916 and / or other software components / modules. For example, the frameworks 918 can provide various graphic user interface (GUI) functions, high-level resource management, high-level location services, and so forth. The frameworks 918 can provide a broad spectrum of other APIs that can be used by the applications 916 and / or other software components / modules, some of which can be specific to a particular operating system or platform.

[0119] The applications 916 include built-in applications 938 and / or third-party applications 940.

[0120] The applications 916 can use built-in operating system functions (e.g., kernel 922, services 924 and / or drivers 926), libraries 920, or frameworks / middleware 918 to create user interfaces to interact with users of the system. Alternatively, or additionally, in some systems, interaction with a user can occur through a presentation layer, such as presentation layer 914. In these systems, the application / module "logic" can be separated from the aspects of the applications / modules that interact with users.

[0121] Figure 10 FIG. 10 is a block diagram illustrating components of a machine 1000, according to some example embodiments, able to read instructions from a machine-readable medium (e.g., a machine-readable storage medium) and perform any one or more of the methodologies discussed herein. Specifically, FIG. 10 shows a diagrammatic representation of the machine 1000 in the example form of a computer system, within which instructions 1018 (e.g., software, a program, an application, an applet, an app, or machine code) for causing the machine 1000 to perform any one or more of the methodologies discussed herein can be executed. The machine 1000 can be a server computer, a client computer, a mobile device, a user device, a personal computer (PC), a laptop computer, a tablet, a personal digital assistant (PDA), a cellular telephone, a smartphone, or other Figure 1 The SPU, EAU, and DGU of FIG. 10 can each be implemented as a machine with some or all of the components of the machine 1000. Specifically, Figure 10FIG. 10 illustrates a diagrammatic representation of a machine in the example form of a computing system within which instructions 1010 (e.g., software, programs, applications, applets, application software, or other executable code) for causing the machine 1000 to perform any one or more of the methodologies discussed herein can be executed. As such, the instructions 1010 can be used to implement modules or components described herein. The instructions 1010 transform the general, non-programmed machine into a particular machine 1000 programmed to carry out the described and illustrated functions in the manner described. In alternative embodiments, the machine 1000 operates as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1000 can operate in the capacity of a server machine or a client machine in server-client network environments, or as a peer machine in peer-to-peer (or distributed) network environments. The machine 1000 can comprise, but not be limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular telephone, a smartphone, a mobile device, a wearable device (e.g., a smartwatch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing the instructions 1010, sequentially or otherwise, that specify actions to be taken by the machine 1000. Further, while only a single machine 1000 is illustrated, the term “machine” shall also be taken to include a collection of machines 1000 that individually or jointly execute the instructions 1010 to perform any one or more of the methodologies discussed herein.

[0122] The machine 1000 can include processors 1004, memory 1006, and input / output (I / O) components 1018, which can be configured to communicate with one another via a bus 1002. In an example embodiment, the processors 1004 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), a specific application integrated circuit (ASIC), a radio-frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) can include, for example, a processor 1008 and a processor 1012 that can execute the instructions 1010. The term “processor” is intended to include a multi-core processor having two or more independent processors (sometimes referred to as “cores”) that can execute instructions contemporaneously. Although FIG. 10 shows the machine 1000 as a particular configuration of hardware, other configurations that are different than the one shown are also Figure 10 Although shown with multiple processors, the machine 1000 can include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0123] Memory 1006 can include memory 1014, such as a main memory, a static memory or other memory storage devices, and storage unit 1016, both accessible to the processor 1004, such as via the bus 1002. The storage unit 1016 and memory 1014 store the instructions 1010 embodying any one or more of the methodologies of this disclosure or functions described herein. The instructions 1010 can also reside, completely or

[0124] As used in this description, the term "machine-readable medium" means a device that can store instructions and data temporarily or permanently and can include, in whole or in part, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage (e.g., Erasable Programmable Read-Only Memory (EEPROM)) and / or any suitable combination thereof. The term "machine-readable medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) able to store instructions 1010. The term "machine-readable medium" shall also be taken to include any medium or combination of media that is capable of storing instructions (e.g., instructions 1010) for execution by a machine (e.g., machine 1000), such that the instructions, when executed by one or more processors of the machine (e.g., processors 1004), cause the machine 1000 to perform any one or more of the methodologies described herein. Accordingly, a "machine-readable medium" is a single storage device or article of manufacture or a "cloud" based storage system or storage network that includes multiple storage devices or articles of manufacture. The term "machine-readable medium" excludes signals per se.

[0125] Input / output (I / O) components 1018 can include a wide variety of components to receive input, provide output, produce output, transmit information, exchange information, capture measurements, and so on. Specific input / output (I / O) components 1018 included in particular machine will vary depending on the type of machine. For example, a user interface machine such as a mobile phone or a portable computer can include a touch input device or other such input mechanisms, while a headless server machine will not include such a touch input device. It will be appreciated that the input / output (I / O) components 1018 can include many other components that are not shown in FIG. 10. These and other input / output (I / O) components 1018 can provide inputs to and receive outputs from the application 1012 and / or the interface element 1014. Figure 10 Many of the other components shown in FIG. 10 can also be used in a wide variety of machines and with equally advantageous results. In this regard, the described features can be employed in input / output (I / O) components 1018, the interface element 1014, and / or the memory 1014.

[0126] For simplicity of the following discussion, the input / output (I / O) components 1018 are grouped into input components 1028 and output components 1026, which is by no means limiting. In various example embodiments, the input / output (I / O) components 1018 can include output components 1026 and input components 1028. The output components 1026 can include visual components (e.g., a display such as a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., a vibratory motor, an electro- mechanical actuator), other signal generators, and the like. The input components 1028 can include alphanumeric input components (e.g., a keyboard, a touchscreen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instruments), tactile input components (e.g., physical buttons, a touchscreen that provides locations and / or forces of touches or touch gestures, or other tactile input components), audio input components (e.g., microphones), and the like.

[0127] In further example embodiments, the input / output (I / O) components 1018 can include biometric components 1030, motion components 1034, environmental components 1036, or position components 1038, among a wide array of other components. For example, the biometric components 1030 can include components to detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves, such as output from EEG devices), identify a person (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or electroencephalogram-based identification), and the like. The motion components 1034 can include acceleration sensor components (e.g., accelerometer), gravitation sensor components, rotation sensor components (e.g., gyroscope), and so forth. The environmental components 1036 can include, for example, illumination sensor components (e.g., photometer), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometer), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors components (e.g., gas detection sensors to detect concentrations of hazardous gases, or to measure pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to a surrounding physical environment. The position components 1038 can include location sensor components (e.g., a Global Position System (GPS) receiver component), altitude sensor components (e.g., altimeters or barometers that detect air pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), and the like.

[0128] Communication can be implemented using a wide variety of technologies. The input / output (I / O) components 1018 can include communication component 1040 operable to couple the machine 1000, via the coupling 1024 and the coupling 1022, with a network 1032 or the device 1020, respectively. For example, the communication component 1040 can include a network interface component or other suitable components to interface with the network 1032. In further examples, the communication component 1040 can include wired

[0129] Although the subject matter has been described in reference to the particular examples, it is to be understood that various modifications can be made without departing from the scope of the embodiments disclosed herein. The disclosed subject matter is to be considered merely illustrative, and the scope of the embodiments is to be accorded the broadest scope permissible under the applicable law. The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various embodiments, and other uses of the embodiments disclosed and of variations thereof.

[0130] The embodiments illustrated herein are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed. Other embodiments can be utilized and derived therefrom, without departing from the scope of the disclosure, and the detailed description is, therefore, not to be regarded as limiting the scope of the disclosure. The detailed description contains specific information for public knowledge and does not limit the scope of the disclosure.

[0131] As used herein, the term "or" can be construed in either an inclusive or exclusive sense. Furthermore, multiple instances of a resource, operation, or structure described herein can be provided as a single instance. Moreover, the boundaries between various resources, operations, modules, engines, and data stores are somewhat arbitrary, and particular operations can be performed by a combination of software components and / or hardware components, including one or more processors operating under the control of machine-readable instructions designed and constructed to carry out the desired processes. Other features, objects, and / or operations can be performed by or in association with the disclosed embodiments, and this disclosure is to be accorded the widest scope of protection consistent with the principles and novel features disclosed herein. In particular, the scope of the disclosure encompasses various acts and / or operational combinations of the disclosed resources, operations, and structures.

[0132] Examples

[0133] To better illustrate the methods and systems disclosed herein, a non-limiting list of examples is provided:

[0134] Example 1 : A method for adaptive calibration for a brain-computer interface, comprising:

[0135] a. displaying one or more objects on a display screen;

[0136] b. modulating a display characteristic of the one or more objects such that each object has a unique time-varying display characteristic;

[0137] c. a user viewing a first displayed object of the one or more objects on the display screen;

[0138] d. detecting and recording, by a system, a first set of current neural signals detected from the user;

[0139] e. associating the first set of current neural signals with the first displayed object on the display screen;

[0140] f. changing the modulation of the display characteristic of the first displayed object;

[0141] g. detecting and recording, by the system, a second set of current neural signals detected from the user;

[0142] h. associating the second set of neural signals with the first displayed object to confirm that the first displayed object is being attended to;

[0143] i. if confirmed, modeling the recorded neural signals associated with the first displayed object;

[0144] j. using the model of the recorded neural signals to identify when the user is attending to the first displayed object;

[0145] k. repeating steps a through j for each of the one or more objects as the user sequentially attends to each object;

[0146] l. repeating steps a through k while modifying the modeling weighting and the modulated display characteristic to optimize the accuracy of the determination of the object being attended to.

[0147] Example 2: The method as recited in example 1, further comprising:

[0148] determining whether the neural signals contain artifacts; and

[0149] if the neural signals contain artifacts, determining whether the neural signals are informative.

[0150] Example 3: The method as recited in example 1 : further comprising:

[0151] filtering the neural signal to mitigate noise;

[0152] determining whether the filtered neural signal contains artifacts; and

[0153] if the filtered neural signal contains artifacts, determining whether the filtered neural signal is informative.

[0154] Example 4: The method as recited in example 2, further comprising:

[0155] if the neural signal is usable, feeding back to the user an indication that the neural signal is usable;

[0156] if the neural signal is not usable, feeding back to the user an indication that the neural signal is not usable; and

[0157] indicating to the user to take one or more actions to correct the non-usable neural signal.

[0158] Example 5: The method as recited in example 3, further comprising:

[0159] if the filtered neural signal is usable, feeding back to the user an indication that the filtered neural signal is usable;

[0160] if the filtered neural signal is not usable, feeding back to the user an indication that the filtered neural signal is not usable; and

[0161] indicating to the user to take the one or more actions to correct the non-usable filtered neural signal.

[0162] Example 6: A computer-implemented method, in at least one processor, the method comprising:

[0163] in an initial phase, receiving a first set of neural signals of a user from a neural signal acquisition device, the user perceiving sensory information in a training sequence, the training sequence comprising at least one sensory stimulus, each sensory stimulus having at least one predetermined corresponding characteristic;

[0164] determining, from the first set of neural signals and the training sequence, neural response data associated with each of the one or more sensory stimuli, the neural response data being combined to generate a model of neural response of the user to the sensory stimuli, the model comprising weights applied to features of the neural signals;

[0165] in a calibration phase, receiving a second set of neural signals of the user from the neural signal acquisition device, the user further perceiving the sensory information in a confirmation sequence, the confirmation sequence comprising at least one of the sensory stimuli;

[0166] using the model, estimating which of the sensory stimuli is the object of attention of the user;

[0167] determining whether the identified object of attention estimated corresponds to a sensory stimulus in the confirmation sequence;

[0168] if it is determined that it corresponds, modifying the weight; and

[0169] if it is determined that it does not correspond, modifying the sensory stimulus in the confirmation sequence.

[0170] Example 7: The method of example 6, wherein modifying the sensory stimulus in the confirmation sequence comprises modifying at least one of: a corresponding property of the at least one sensory stimulus; reducing a number of sensory stimuli in the confirmation sequence; and changing an appearance of the sensory stimulus in the confirmation sequence.

[0171] Example 8: The method of example 6 or example 7, wherein the at least one sensory stimulus is a visual stimulus, wherein the training sequence comprises displayed training image data viewed by the user, and wherein the training image data comprises the visual stimulus.

[0172] Example 9: The method of example 8, wherein the or each visual stimulus is displayed in a known order.

[0173] Example 10: The method of example 8 or example 9, wherein the training image data comprises the or each visual stimulus displayed at a known display location.

[0174] Example 11: The method of any one of examples 8 to 10, wherein the predetermined corresponding property modulation is a change in at least one of: display location, brightness, contrast, flicker frequency, colour and / or scale.

[0175] Example 12: The method of any one of examples 5 to 11, wherein the at least one sensory stimulus is an audio stimulus, the training sequence comprises training audio data heard by the user, and wherein the training audio data comprises the audio stimulus.

[0176] Example 13: The method of any one of examples 5 to 12, wherein the at least one sensory stimulus is a haptic stimulus, the training sequence comprises training haptic data felt by the user, and wherein the training haptic data comprises the haptic stimulus.

[0177] Example 14: The method of any one of examples 5 to 13, wherein estimating which of the sensory stimuli is the object of attention of the user comprises:

[0178] comparing the different stimuli in the training sequence to the reconstructed sensory stimuli reconstructed using the second set of neural signals; and

[0179] determining the object of interest is the stimulus whose time-varying characteristics are most similar to the reconstructed sensory stimulus.

[0180] Example 15: The method of any one of examples 5-14, further comprising repeating the operations of the calibration phase.

[0181] Example 16: The method of any one of examples 5-15, wherein the sensory stimulus corresponds to a control item, the method further comprising performing a control task with respect to the control item associated with the inferred focus of attention.

[0182] Example 17: The method of any one of examples 5-16, further comprising filtering the received neural signals to mitigate noise prior to determining the neural response data used to generate the model.

[0183] Example 18: The method of any one of examples 5-17, further comprising:

[0184] determining whether the received neural signals include artifacts; and

[0185] wherein the neural signals include artifacts, rejecting the received neural signals.

[0186] Example 19: The method of example 18, further comprising:

[0187] in the event that the neural signals do not include artifacts, feeding back to the user an indication that the neural signals are available;

[0188] in the event that the neural signals include artifacts, feeding back to the user an indication that the neural signals are not available if the neural signals are not available; and

[0189] indicating to the user to take one or more actions to correct the unavailable neural signals.

[0190] Example 20: A brain-computer interface system, comprising:

[0191] a sensory information generation unit configured to output a signal for reproduction by a reproduction device, the reproduced signal being perceived by a user as sensory information;

[0192] a neural signal acquisition device configured to acquire neural signals associated with the user; and

[0193] a signal processing unit operably coupled to the sensory information generation unit and the neural signal acquisition device, the signal processing unit being configured to:

[0194] in an initial phase, receive a first set of neural signals of the user from the neural signal acquisition device, the user perceiving sensory information in a training sequence, the training sequence including at least one sensory stimulus, each sensory stimulus having at least one predetermined corresponding characteristic;

[0195] determining, from the first set of neural signals and the training sequence, neural response data associated with each of the one or more sensory stimuli, the neural response data being combined to generate a model of neural responses of the user to the sensory stimuli, the model including weights applied to features of the neural signals;

[0196] in a calibration phase, receiving a second set of neural signals of the user from the neural signal acquisition device, the user further perceiving sensory information in a confirmation sequence, the confirmation sequence including at least one of the sensory stimuli;

[0197] using the model, estimating which of the sensory stimuli is the object of attention of the user;

[0198] determining whether the identified object of attention as estimated corresponds to a sensory stimulus in the confirmation sequence;

[0199] if it is determined that it corresponds, modifying the weights; and

[0200] if it is determined that it does not correspond, modifying the sensory stimuli in the confirmation sequence.

[0201] Example 21 : The brain-computer interface system of example 20,

[0202] wherein the at least one sensory stimulus is a visual stimulus,

[0203] wherein the sensory information in the training sequence includes displayed training image data viewed by the user, the training image data including the visual stimulus,

[0204] wherein the further sensory information in the confirmation sequence includes displayed confirmation image data viewed by the user, the confirmation image data including the visual stimulus, and

[0205] wherein the sensory information generation unit includes a display generation unit (DGU) configured to cause a display to reproduce the training image data and the confirmation image data.

[0206] Example 22: A non-transitory computer-readable storage medium storing instructions that, when executed by a computer system, cause the computer system to perform operations comprising:

[0207] in an initial phase, receiving a first set of neural signals of the user from the neural signal acquisition device, the user perceiving sensory information in a training sequence, the training sequence including at least one sensory stimulus, each sensory stimulus having at least one predetermined corresponding characteristic;

[0208] determining, from the first set of neural signals and the training sequence, neural response data associated with each of the one or more sensory stimuli, the neural response data being combined to generate a model of a neural response of the user to the sensory stimuli, the model comprising weights applied to features of the neural signals;

[0209] in a calibration phase, receiving a second set of neural signals of the user from the neural signal acquisition device, the user further perceiving the sensory information in a confirmation sequence, the confirmation sequence comprising at least one of the sensory stimuli;

[0210] using the model, estimating which of the sensory stimuli is the object of attention of the user;

[0211] determining whether the identified object of attention as estimated corresponds to a sensory stimulus in the confirmation sequence;

[0212] if it is determined that it corresponds, modifying the weights; and

[0213] if it is determined that it does not correspond, modifying the sensory stimuli in the confirmation sequence.

[0214] Example 23: A computer-readable storage medium carrying instructions which, when executed by a computer, cause the computer to perform the method of any one of examples 5 to 19.

[0215] The figures are exemplary and should not be understood to limit the scope or the steps to only those. The same approach applies in the case where the system displays only a single object. The same approach applies in the case where the system displays more than three objects. Although the displayed objects have different two-dimensional shapes, this is for clarity. The objects can have any shape, any color, and can be located anywhere on the display screen. The sequence of successive object checks and calibrations can be done in any order, not necessarily one in the sequence of adjacent objects.

[0216] Although described through a number of detailed exemplary embodiments, the portable device for acquiring electroencephalogram signals according to the present disclosure includes various variants, modifications and improvements apparent to those skilled in the art, which should be understood to fall within the scope of the subject matter of the present disclosure, as defined by the appended claims.

Claims

1. A computer-implemented method comprising: receiving neural signals from a user while the user is exposed to sensory stimuli during use of a brain-computer interface (BCI), the sensory stimuli including display of a target object as a first graphical object, the target object having a varying first temporal characteristic that is different from a second temporal characteristic of a second graphical object of the sensory stimuli; decoding the neural signals using a neural response model to identify a focus of attention of the user on the target object; modifying the sensory stimuli in real-time based on the focus of attention to provide real-time feedback to the user; refining the neural response model using a response of the user to the real-time feedback; and continually repeating the receiving, decoding, modifying, and refining operations to continually calibrate the neural response model during use of the BCI by the user. the sensory stimuli include visual stimuli and the real-time feedback includes modifying a display characteristic of the visual stimuli.

2. The method of claim 1, wherein, the sensory stimuli are part of a validation sequence that includes at least one of the sensory stimuli from a training sequence.

3. The method of claim 1, wherein, the sensory stimuli are modified by changing an appearance of the sensory stimuli in the validation sequence.

4. The method of claim 3, wherein, the neural response model includes weights that are applied to features of the neural signals, and refining the neural response model includes modifying the weights.

5. The method of claim 1, wherein, 6. A machine comprising: one or more processors; and memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising: receiving neural signals from a user while the user is exposed to sensory stimuli during use of a brain-computer interface (BCI), the sensory stimuli including display of a target object as a first graphical object, the target object having a varying first temporal characteristic that is different from a second temporal characteristic of a second graphical object of the sensory stimuli; decoding the neural signals using a neural response model to identify a focus of attention of the user on the target object; modifying the sensory stimuli in real-time based on the focus of attention to provide real-time feedback to the user; refining the neural response model using a response of the user to the real-time feedback; and continually repeating the receiving, decoding, modifying, and refining operations to continually calibrate the neural response model during use of the BCI by the user. the sensory stimuli include visual stimuli and the real-time feedback includes modifying a display characteristic of the visual stimuli.

7. The machine of claim 6, wherein, the sensory stimuli are part of a validation sequence that includes at least one of the sensory stimuli from a training sequence.

8. The machine of claim 6, wherein, the sensory stimuli are modified by changing an appearance of the sensory stimuli in the validation sequence.

9. The machine of claim 8, wherein, 10. A machine-storage medium storing machine-executable instructions that, when executed by a machine, cause the machine to perform operations comprising: ​ While a user is exposed to sensory stimuli during use of a brain-computer interface BCI, receiving neural signals from the user, the sensory stimuli including display of a target object as a first graphical object, the target object having a varying first temporal characteristic that is different from a second temporal characteristic of a second graphical object of the sensory stimuli; decoding the neural signals using a neural response model to identify a focus of attention of the user on the target object; real-time modifying the sensory stimuli based on the focus of attention to provide real-time feedback to the user; refining the neural response model using a response of the user to the real-time feedback; and continually repeating the receiving, decoding, modifying, and refining operations to continually calibrate the neural response model while the user uses the BCI.

Images