Brain-computer interface

Abstract

A method for adaptive calibration in a brain-computer interface is disclosed. The method is used to reliably associate neural signals with objects that were elicited by a user's attention. A visual stimulus is provided that overlays one or more objects, at least a portion of which has a characteristic modulation. The brain-computer interface measures neural responses to the objects viewed by the user. The neural responses to the visual stimulus are correlated with the modulation, with the correlation being stronger when attention is focused on the visual stimulus. Based on the determined correlations, weights are applied to a resulting model of the user's neural responses. Both the neural signal model weights and the displayed object display modulation are adjusted to increase the certainty of the association of the neural signals with the objects that elicited the signals.

Description

Technical Field

[0001] The present invention relates to the operation of brain-computer interfaces. In particular, the present invention relates to the calibration of systems using brain-computer interfaces involving visual sensing. Background Art

[0002] Some brain-computer interface (BCI)-based systems utilize multiple electrodes attached to the head that can detect changes in electrical properties caused by brain activity. Although there are similarities in these neural signals generated by individuals due to these changes in electrical properties when individuals perform the same function, there are still significant differences in the neural signal parameters between individuals. Therefore, systems that require fairly accurate BCI results need to be calibrated so that the neural signals can be reliably associated with the controls and tasks required by the system. In order to maintain accurate association throughout the use of the BCI, the BCI may need to be recalibrated due to changes in electrode position and other factors that may affect the efficacy of the neural signal model. However, the calibration step can be time-consuming and destructive. Therefore, ideally, a calibration method that can unobtrusively and quickly perform calibration adaptation to correct for neural signal changes can provide multiple benefits by at least maintaining BCI accuracy and improving the user's BCI experience.

[0003] Therefore, it is desirable to provide brain-computer interfaces that address the aforementioned challenges. Summary of the Invention

[0004] The present disclosure relates to a brain-computer interface system in which a computer controls or monitors sensory stimuli perceptible to an individual when measuring the brain activity of the individual. A model is then constructed that associates the sensory stimuli with brain activity (i.e., neural responses) and is used to decode the neural signals of each individual. The model and sensory stimuli are updated (i.e., calibrated) to ensure a more reliable association between the stimuli and the neural signals, thereby providing an improved user experience in the execution of graphical interface tasks. In certain embodiments, the BCI system is a visual BCI system and the sensory stimuli are visual stimuli.

[0005] According to a first aspect, the present disclosure relates to a computer-implemented method, in at least one processor, the method comprising: in an initial stage, receiving a first set of neural signals of a user from a neural signal acquisition device, the user perceiving sensory information in a training sequence, the training sequence comprising at least one sensory stimulus, each sensory stimulus having at least one predetermined corresponding characteristic; determining neural response data associated with each of one or more sensory stimuli from the first set of neural signals and the training sequence, the neural response data being combined to generate a model of the user's neural response to the sensory stimuli, the model comprising weights applied to features of the neural signals; in a calibration stage, receiving a second set of neural signals of the user from the neural signal acquisition device, the user further perceiving sensory information in a confirmation sequence, the confirmation sequence comprising at least one sensory stimulus in the sensory stimuli; using the model, estimating which of the sensory stimuli is the user's object of attention; determining whether the identification of the estimated object of attention corresponds to the sensory stimulus in the confirmation sequence; modifying the weight if a correspondence is determined; and modifying the sensory stimulus in the confirmation sequence if a non-correspondence is determined.

[0006] According to a second aspect, the present disclosure relates to a brain-computer interface system, comprising: a sensory information generation unit, which is configured to output a signal for reproduction by a reproduction device, the reproduced signal being perceived by a user as sensory information; a neural signal acquisition device, which is configured to acquire neural signals associated with the user; and a signal processing unit, which is operably coupled to the sensory information generation unit and the neural signal acquisition device, the signal processing unit being configured to: in an initial stage, receive a first set of neural signals of a user from the neural signal acquisition device, the user perceiving sensory information in a training sequence, the training sequence comprising at least one sensory stimulus, each sensory stimulus having at least one predetermined corresponding characteristic; The signal and training sequence determine neural response data associated with each of one or more sensory stimuli, and the neural response data are combined to generate a model of the user's neural response to the sensory stimuli, the model including weights applied to features of the neural signals; in a calibration phase, a second set of neural signals of the user is received from the neural signal acquisition device, and the user further perceives sensory information in a confirmation sequence, which confirmation sequence includes at least one sensory stimulus among the sensory stimuli; using the model, estimating which of the sensory stimuli is the user's object of attention; determining whether the estimated identification of the object of attention corresponds to the sensory stimulus in the confirmation sequence; if it is determined to correspond, modifying the weight; and if it is determined not to correspond, modifying the sensory stimulus in the confirmation sequence. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] To easily identify the discussion of any particular element or act, the most significant digit or digits in a reference number refer to the figure number in which the element is first introduced.

[0008] Figure 1 depicts the electronic architecture of a BCI system for receiving and processing EEG signals according to the present disclosure;

[0009] Figure 2 Show Figure 1 How the system acquires neural signals that can be associated with attention to displayed objects;

[0010] Figure 3 Shows that when each display object has its own modulation display characteristics Figure 1 How does the system acquire neural signals that differ from each other in time?

[0011] Figure 4 A flow chart showing the flow of certain functional blocks in the calibration method according to the present disclosure;

[0012] Figure 5 A flow chart showing the flow of certain functional blocks in another calibration method according to the present disclosure;

[0013] Figure 6 illustrates an exemplary technique for constructing a decoding model according to the present disclosure;

[0014] Figure 7 shows the flow of certain major functional blocks in the method of operation of the BCI after calibration has been performed according to the present disclosure;

[0015] Figure 8 Various examples of display devices suitable for use with the BCI system of the present disclosure are shown;

[0016] Figure 9 is a block diagram illustrating a software architecture in which the present disclosure may be implemented according to some example embodiments; and

[0017] Figure 10 is a diagrammatic representation of a machine in the form of a computer system according to some example embodiments, within which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed. DETAILED DESCRIPTION

[0018] Brain-computer interfaces (BCIs) attempt to interpret measured brain activity in an individual to allow determination (i.e., inference) of the individual's focus of attention. The inferred focus of attention can be used to perform input or control tasks in an interface with a computer (which may be a component of the BCI). The computer controls or monitors sensory stimuli perceivable by the individual so that the sensory stimuli can be correlated with the measured brain activity.

[0019] In some embodiments, the BCI system is a visual BCI system and the sensory stimuli are visual stimuli. In a visual BCI, typically among multiple generated visual stimuli presented to a user, the neural response to a target stimulus is used to infer (or "decode") which stimulus is essentially the object of interest (of visual attention) at any given time. The object of interest can then be associated with an action that is selectable or controllable by the user.

[0020] Similar provisions can be made for non-visual BCI systems, where target stimuli used to infer attention can include auditory and tactile / touch stimuli.

[0021] A variety of known techniques can be used to obtain neural responses. A convenient method relies on surface electroencephalography (EEG), which is non-invasive, has a fine-grained temporal resolution, and is based on a well-known empirical basis. Surface electroencephalography (sEEG) can measure the diffusion potential changes on the surface of the subject's skull (i.e., scalp) in real time. These potential changes are commonly referred to as electroencephalographic signals or EEG signals. Other techniques are of course also available and can be used instead of (or in combination with) surface EEG - examples include intercranial EEG (iEEG) (also known as electrocorticography (ECoG)), magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI), functional near-infrared spectroscopy (fNIRS), etc.

[0022] In a typical visual BCI, a computer controls the display of visual stimuli in a graphical interface (i.e., a display) generated and presented by a display device. The computer also generates a model of previous neural responses to visual stimuli, which is used to decode each individual's current neural signals.

[0023] Examples of suitable display devices (some of which are in Figure 8 802, a projector 810, a virtual reality headset 806, a display screen of an interactive whiteboard and tablet computer 804, a smartphone, smart glasses 808, etc. The visual stimuli 811, 811', 812, 812', 814, 814', 816 may form part of a generated graphical user interface (GUI), or they may be presented as augmented reality (AR) or mixed reality graphical objects 816 overlaying an underlying image: the underlying image may be simply the user's actual field of view (as in the case of a mixed reality display function projected onto a transparent display of a set of smart glasses) or a digital image corresponding to the user's field of view but captured in real time by an optical capture device (which may in turn capture images corresponding to the user's field of view in other possible views).

[0024] Inferring which of multiple visual stimuli (if any) is the object of attention at any given time is fraught with difficulty. For example, when a user is presented with multiple stimuli (such as, for example, numbers displayed on an on-screen keyboard or icons displayed in a graphical user interface), it has proven almost impossible to infer which one is being attended to directly from brain activity at a particular time. The user perceives the number under attention (such as the number 5), so the brain must contain information to distinguish that number from the others, but current methods cannot extract this information from brain activity alone. That is, current methods can infer that a stimulus has been perceived, but they cannot determine which specific stimulus is under attention using brain activity alone.

[0025] To overcome this problem and provide sufficient contrast between the stimulus and the background (and between stimuli), it is known to configure the stimuli used by visual BCIs to blink or pulse (i.e., so that each stimulus has a distinguishable distribution of characteristics over time). The flashed stimuli cause a measurable electrical response. Specific techniques monitor different electrical responses, such as steady-state visual evoked potentials (SSVEPs) and P-300 event-related potentials. In a typical implementation, the stimuli are flashed at a rate exceeding 6 Hz. Therefore, the visual BCI relies on a method that involves displaying various stimuli discretely rather than continuously in a display device, and typically at different points in time. It was found that brain activity associated with attention focused on a given stimulus corresponds to (i.e., correlates with) one or more aspects of the temporal distribution of the stimulus, such as the frequency with which the stimulus flashes and / or the duty cycle with which the stimulus alternates between a flashing state and a static state.

[0026] Therefore, decoding of neural signals relies on the fact that, when a stimulus is activated, it triggers a characteristic pattern of neural responses in the brain that can be determined from electrical signals (i.e., SSVEP or P-300 potentials) (e.g., picked up by electrodes of an EEG device). This neural data pattern may be very similar or even identical for various digits or icons, but over time, it becomes time-locked (i.e., synchronized) with the characteristic distribution for the perceived digit / icon: only one digit / icon can pulse at any time, so that the correlation of the pulsed neural response with the time (at which the digit / icon pulsed) can be determined to indicate that the digit / icon is the object of attention. By displaying each digit / icon at a different time point, turning the digit / icon on and off at different rates, applying different duty cycles, and / or simply applying the stimulus at different time points, the BCI algorithm can determine which stimulus, when activated, is most likely to trigger a given neural response, thereby allowing the system to determine the object of attention. The timestamps of the data displayed on the display can be shared with the visual BCI and used to synchronize the visual stimulus with the corresponding neural response. This will be discussed further below.

[0027] More generally, the modulation applied to a sensory stimulus is combined with a neural response model that uses previous neural responses to allow interpretation of the current neural response. The model provides confidence values ​​(i.e., model weights) with which signal features (in the neural response evoked by the sensory stimulus) can be associated with the object of attention focus.

[0028] Models of previous neural responses to visual stimuli are used to decode each individual's current neural signal in an operation called "stimulus reconstruction."

[0029] Co-pending U.S. patent application 62 / 843,651 (docket number 3901.00.0001), filed May 6, 2019, and incorporated herein by reference in its entirety, describes a method for quickly and accurately determining an object of interest (target) from objects in its periphery (distractors). The method relies on the characteristics of the human visual system.

[0030] Systems that require reasonably accurate BCI results also require calibration of each individual's neural signals so that they can be reliably associated with the controls and tasks required by the system. Furthermore, recalibration of the BCI may be necessary due to changes in electrode placement and other factors that may affect the ability of the neural signal model to maintain accurate associations throughout BCI use. However, this calibration step can be both time-consuming and disruptive.

[0031] Therefore, there is a need for calibration methods that can unobtrusively and quickly perform calibration adaptation to correct for neural signal variations, which can provide multiple benefits by at least maintaining BCI accuracy and improving the user BCI experience.

[0032] Today, there are essentially two approaches to BCI calibration: operant conditioning and "machine learning." Operant conditioning hides the details of the interface from the user and enables them to slowly "learn" how to move on-screen objects and invoke tasks using brain activity through trial and error. Obviously, this can be frustrating and time-consuming, so it's not the first choice for today's calibration needs.

[0033] Using machine learning, a user interacts with one or more display objects on a display screen by selecting one to attend to (i.e., focus on). Where multiple display objects are displayed, each display object may have display characteristics unique to each object, such that focusing on a particular object may elicit a response that "encodes" the unique display characteristics of that object, as described above.

[0034] For example, if an object is flashed at different rates, the object the user is focusing on will stimulate a neural response and signal that can be distinguished by the signal characteristics that reflect the object's flash rate. A neural signal model associated with this modulation and the signal characteristics caused by this modulation can be constructed, and then used to associate this modulation with the neural signal it stimulates.

[0035] However, as mentioned previously, electrode positions may change during or between consecutive recording sessions, and external interference may add electromagnetic interference, noise, and other artifacts to the neural signal. In either case, the detected signal may no longer match the previously constructed model, so the BCI can no longer associate the neural signal with the displayed stimulus modulation with sufficient certainty. It is necessary to recalibrate the model of the neural response to sensory stimulation.

[0036] Prior art BCIs related to vision BCIs use either neurofeedback or model weighting to try and adapt the BCI to changing neural signal conditions.

[0037] Neurofeedback is a type of biofeedback in which neural activity is measured (e.g., by sEEG, MEG, iEEG / ECoG, fMRI, or fNIRS) and a sensory representation of that activity is presented to the user in order to self-regulate their mental state and / or adjust their behavior accordingly. In neurofeedback methods, the user is presented with information that they can use to train their interaction with the BCI, such as "learning" how to move screen objects and / or invoke tasks using brain activity. This feedback can be in the form of highlighting an object currently identified as the object of interest by changing the object's color, size, display position, or other aspects of its appearance.

[0038] Model weighting in the context of visual BCIs refers to the training of machine learning systems. Weights are associated with predictions of user intent. In traditional calibration procedures, neural signal features that best match the neural signal patterns of known visual stimuli are given greater weight than less reliable signal features.

[0039] There are several methods that can extract a set of signal features from a neural signal, either from a single channel or from a region of interest (i.e., a collection of channels originating from electrodes in that region). In the case of MEG or EEG, some examples of features derived from neural time series include: event-related potentials (ERPs), evoked potentials (which can be visual evoked potentials, auditory evoked potentials, sensory evoked potentials, motor evoked potentials), oscillatory signals (signal power in a specific frequency band), slow cortical potentials, brain state-related signals, etc.

[0040] In a visual BCI embodiment, the present disclosure is based on techniques for discerning which (potential) objects of interest are the focus of visual attention within the user's field of view (typically, but not always, on a display presented to the user). As described above, modulation of one or more of these objects causes the object to flicker or otherwise change visually, such that the modulation acts as a stimulus for the associated neural response. In turn, the neural response can be measured and decoded to determine which object of interest is the focus of the user's attention. The object determined to be the focus of attention is then visually modified so that the user understands the results of the decoding and is encouraged to continue exhibiting the activity that triggered the determination, confirming (i.e., verifying) the decoding (or stopping from the activity, marking an undesirable decoding result). In either case, the strength of the association between the visual stimulus and the associated neural response can be adjusted based on the decoding accuracy determined by the user's response to the modified focus of attention.

[0041] Certain embodiments of the methods disclosed herein combine these two techniques—changing the object modulation to obtain a more explicit association while also changing the model weighting to obtain a better match. The model weighting and the modulation of the visual stimulus are updated (i.e., calibrated) in real time to ensure a more reliable association between the (reconstructed) stimulus and the neural signal, thereby providing an improved user experience in the performance of the graphical interface task. This results in "closed-loop" or adaptive neurofeedback that effectively changes the experimental task in real time based on neural activity.

[0042] Furthermore, the disclosed method can employ neural signal filtering to remove artifacts that could confound the association model. Thus, the method can be selective in that it determines whether a neural signal is informative by ensuring that any artifacts (i.e., interference due to motion or background electromagnetic conditions, noise, etc.) are first filtered out before attempting to associate the signal with the modulation.

[0043] In the case of a display employing multiple display objects (with associated modulated stimuli), the adaptive method repeats the process of refinement and modulation association with each new display object of interest. Thus, the model is essentially refined through these new trials.

[0044] In certain embodiments of the calibration methods disclosed herein, both object modulation and model weighting are refined. As a result, the calibration method is faster and more accurate than conventional calibration methods that keep one factor (such as display target stimulus modulation) fixed. Furthermore, the described calibration method allows for a less obtrusive process, whereby calibration can occur in the background as the user employs the BCI for various applications, such as gaming or hands-free productivity tools.

[0045] Essentially, the approach creates a closed-loop process in which real-time neurofeedback enhances the user's attention and focus, which in turn enhances the EEG signals induced by the user's attention and focus, which improves the accuracy of BCI decoding, which can increase the user's sense of immersion.

[0046] In certain embodiments, adaptive calibration methods can be used with systems employing BCIs to enable hands-free system control. To detect brain activity, the two most prominent methods involve electroencephalography (EEG), whereby electrodes placed on the user's scalp ("surface EEG") or inserted directly into the brain (i.e., "intercranial EEG," iEEG, or electrocorticography, ECoG) detect changes in the electric field resulting from the synaptic activity of neurons. Each person's brain, while generally similar, is different. Two people exposed to the same stimulus will have different specific EEG results, but will generally show temporal and spatial response similarities. Therefore, any system that depends on decoding neural signals and associating them with specific system controls needs to calibrate its operation to the individual's specific EEG neural signals.

[0047] The accuracy of system control is directly affected by the accuracy and certainty with which the neural signals are associated with the intended system control.

[0048] As mentioned above, in the early days of BCI technology, experimenters often relied on operant conditioning, whereby users used trial and error to mentally adjust their responses to the system. This often required lengthy training sessions before achieving even a modicum of success. As the technology evolved, machine learning techniques were applied to map neural signal responses to specific end results. So, for example, if a user is shown a display with one object or one of multiple objects and is instructed to focus on that object, if a consistent neural signal is collected when the user "focuses" on the object, then it's possible to begin to associate that signal with that object, and so on.

[0049] However, for screens with multiple objects representing data or control calls, it is often difficult to associate neural signals with those objects with a high degree of certainty. As mentioned above, one way to address this difficulty is to make the object (called a "display target" or a stimulus associated with that target) flash at different rates, with different light intensities, increased contrast changes, color transitions, geometric deformations, rotations, oscillations, displacements along a path, etc. As an example, with different flicker rates, the time differences between the on and off states of the object will be reflected in the detected neural signals. Therefore, by looking at the times of the various signal changes and associating them with the flicker rate, these signals can be associated with objects with a higher degree of certainty. Therefore, the display mode is modulated to give each object a display distinction, while increasing the certainty of associating the signal with the object at the risk of annoying the user.

[0050] Those skilled in the art of BCI know a variety of methods for representing neural signals and processing them to produce reliable models. They also know a variety of methods for modulating displayed objects to create easily identifiable differences in the neural signal patterns they evoke. Therefore, when the term "modulation" or any variation of the term is used, it should be interpreted broadly to encompass any known way of modulating the display characteristics of a displayed object.

[0051] Likewise, when the term "signal processing" or any variation of that term is used, it should be broadly interpreted to include any known way of processing neural signals. When the term "filtering" or any variation of that term is used, it should be broadly interpreted to include any known method of filtering neural signals to mitigate or eliminate artifacts, such as neural signals or brain rhythms that are not relevant to determining the object of interest.

[0052] Furthermore, system implementations supporting brain-computer interaction are well known in the art, the most common of which involve head-mounted helmets with distributed electrodes that allow for parallel detection of electric field changes while the user is engaging in an activity.

[0053] In certain embodiments according to the present disclosure, the operation of the BCI includes a briefing, initialization, and calibration phase. Because users may have significant differences in their baseline neural responses to the same stimulus (particularly those with damaged or injured visual cortex), the calibration phase can be used to generate a user-specific stimulus reconstruction model. This phase can take less than a minute to build (typically, about 30 seconds).

[0054] Embodiments according to the present disclosure implement a method for "adaptive calibration" of a system employing a BCI. The method allows for situations where recalibration is needed due to distracted user attention, changes in helmet / electrode position, "noisy" signal conditions, etc., which may introduce artifacts to the signal. The method is thus adaptive and is intended to allow unobtrusive calibration to be repeated as necessary with minimal user distraction, rather than initiating a single calibration event and risking loss of certainty due to variations. The method also utilizes changes in model weighting and object display modulation to refine and improve neural signal modeling to improve operational reliability.

[0055] Figure 1An example of an electronic architecture for receiving and processing neural signals using a BCI device 100 according to the present disclosure is shown. A user wearing an electrode helmet 101 looks at a display 107 with one or more objects and pays attention to an object 103. The helmet detects essentially simultaneous changes in the electric field, and each electrode provides an output signal that varies over time. Overall, the combination of electrodes provides a set of temporally parallel electrode signals. Each electrode is fed into an EEG acquisition unit (EAU, 104), which transmits its results to a signal processing unit (SPU, 105). The processing results of the signal processing unit (SPU, 105) are used to control a display generation unit (DGU, 106). The DGU 106, in turn, controls the presentation of image data on the display 107. The DGU can also provide timestamp information 102 for the image data. It is essentially a closed-loop system, as any changes made by the DGU 106 affect the display 107 and are reflected back through the user's neural response.

[0056] Surface EEG equipment consists of a portable device (i.e., a cap or head-mounted device) designed to measure diffuse electrical potentials on the surface of a subject's skull. Figure 1 , the portable device is illustrated as an electrode helmet 101. Figure 1 The portable device 101 comprises one or more electrodes 108 , typically between 1 and 128 electrodes, advantageously between 2 and 64, advantageously between 4 and 16 electrodes.

[0057] Each electrode 108 may include a sensor for detecting electrical signals generated by the subject's neuronal activity and circuitry for pre-processing (e.g., filtering and / or differential amplification) the detected signals prior to analog-to-digital conversion: this electrode is referred to as "active." Figure 1 , wherein the sensor is in physical contact with the subject's scalp. The electrodes may be adapted for use with a conductive gel or other conductive liquid (referred to as "wet" electrodes) or without such liquid (ie, "dry" electrodes).

[0058] The EAU 104 may include analog-to-digital conversion (ADC) circuitry and a microcontroller. Each ADC circuit is configured to convert signals from a given number of active electrodes 108, such as between 1 and 128.

[0059] The ADC circuit is controlled by a microcontroller and communicates with the microcontroller, for example, via the protocol SPI ("Serial Peripheral Interface"). The microcontroller packages the received data for transmission to an external processing unit ( Figure 1 The SPU 105 in FIG. 10 is a schematic diagram of an SPU 105 in FIG. 10. The SPU may be, for example, a computer, a mobile phone, a virtual reality headset, a game console, an automotive or aircraft computer system, or the like.

[0060] In some embodiments, each active electrode 108 is powered by a battery ( Figure 1 The battery is conveniently provided in the housing of the portable device 101.

[0061] In certain embodiments, each active electrode 108 measures a corresponding potential value, subtracts the potential measured by the reference electrode (Ei=Vi-Vref) from the potential value, and digitizes the difference with the aid of an ADC circuit and then transmits it by the microcontroller.

[0062] In some embodiments, DGU 106 uses the processing results from SPU 105 to change a target object for display in a graphical user interface of a display device, such as display 107. The target object may include a control, and the control may in turn be associated with a user-selectable action.

[0063] exist Figure 1 In the system 100 , an image is displayed on a display of a display device 107 . The subject watches the image on the display 107 and focuses on the target object 103 .

[0064] In an embodiment, the display device 107 displays at least the target object 103 as a graphical object having a varying temporal characteristic that is different from the temporal characteristics of other displayed objects and / or the background in the display. The varying temporal characteristic can, for example, be a constant or time-locked flash effect that changes the appearance of the target object at a rate greater than 6 Hz. In another embodiment, the varying temporal characteristic can use a pseudo-random time code so as to generate a flash effect that changes the appearance of the target object several times per second on average (e.g., at an average rate of 3 Hz). In the case where more than one graphical object is a potential target object (i.e., a selection of target objects is provided to the viewing subject to focus attention), each object is associated with a discrete spatial and / or temporal code.

[0065] exist Figure 2 , a representation of the parallel generation signals 201 detected by the electrodes in the electrode helmet 101 as acquired at the EAU 104 is shown. The corresponding signal traces for each electrode response are shown vertically spaced apart but sharing the same time axis: time runs in the horizontal direction, and the height of the trace vertically above or below the horizontal line represents the strength of the response of that electrode at that moment. The responses to the most relevant signal features can be refined using any of several well-known signal processing methods.

[0066] exist Figure 3In the example above, the signal pattern 201 associated with the user focusing on object 103 is compared to the signal pattern 301 associated with the user focusing on a second object 103' instead. The blinking rates of objects 103 and 103' are such that 103 blinks at time T1 and 103' blinks at time T2. This change can be distinguished by comparing the times at which the corresponding stimulus responses begin. This is just one way to modulate the properties of the object graph in order to uniquely associate a given EEG signal / segment to the object being focused on.

[0067] The EAU 104 detects neural responses associated with attention focused on a target object (i.e., tiny electrical potentials indicative of brain activity in the visual cortex); thus, the visual perception of the changing temporal characteristics of the target object acts as a stimulus in the subject's brain, generating a specific brain response corresponding to the code associated with the target object of attention. The detected neural response (e.g., electrical potential) is then converted into a digital signal and transmitted to the SPU 105 for decoding. The sympathetic neural response in which the brain appears to "oscillate" or respond in sync with the temporal characteristics of the flash is referred to herein as "neural synchronization."

[0068] The SPU 105 executes instructions to interpret the received neural signals to determine feedback indicating in real time that the target object is the current focus of (visual) attention. Decoding the information in the neural response signal relies on the correspondence between that information and one or more aspects of the temporal distribution of the target object (i.e., stimulus). Figure 3 In the example of , signal patterns 201 and 301 represent simple models of signal patterns associated with objects of interest 103 and 103', respectively: the received neural signals can be compared (here in terms of timing) to determine which pattern is the closest match to the received neural signals).

[0069] In some embodiments, the SPU 105 and EAU 104 may be provided in a single device so that the decoding algorithm is performed directly on the detected neural responses.

[0070] In some embodiments, DGU 106 may conveniently generate image data including a time-varying target object for presentation on display device 107. In some embodiments, SPU 105 and DGU 106 may be provided in a single device so that information regarding the determined focus of visual attention may be incorporated into the generation (and modulation) of visual stimuli in a display.

[0071] In some embodiments, the display device 107 displays the overlay object as a graphical object having varying temporal characteristics that differ from temporal characteristics of other display objects and / or the background in the display, and then displays the overlay object as a graphical layer on at least one identified target object.

[0072] Visual stimuli (i.e., time-varying target objects or overlay objects) can provide retrospective feedback to the user, thereby validating their selection. Visual feedback can be conveniently presented to the user on the display screen 107 so that they know that the target object 103 is determined to be the current focus of attention. For example, the display device can display an icon, cursor, or other graphical object or effect near the target object 103, highlighting (e.g., overlaying) the object that appears to be the current focus of visual attention. This provides a positive feedback loop in which the apparent target object is confirmed (i.e., verified) to be the intended target object by prolonged attention, and the association determined by the user's neural response model is reinforced.

[0073] Figure 4 4 is a flow chart showing the flow of certain functional blocks in an embodiment according to the present disclosure. As shown in the figure, the flow starts with an initial sequence (operations 401-407).

[0074] At operation 401, use Figure 1 The DGU 106 of the BCI 100 displays objects A, B and C, for example: This could be e.g. Figure 1-3 The user then focuses on object A (operation 402). Substantially simultaneously, neural signals are collected (operation 403). The collection of neural signals may be performed by the EAU 104.

[0075] The SPU 105 optionally makes a determination as to whether the signal is informative, e.g., whether the signal quality is good enough to be used to predict the user's focus of attention (operation 404). The presence of artifacts (e.g., interference or noise) in the signal may cause it to be uninformative, as may the lack of sufficiently stable attention / fixation on a single object.

[0076] Although not in Figure 4 , but determining that the signal is not informational (operation 404, no) may trigger a reduction in the number of objects presented to the user (simplifying the task of classifying the neural signal) and / or a change in the appearance of the object (e.g., increasing the size of the stimulus to ensure that the user sees it). In such a case, the process returns to the display of the objects of the initial sequence (operation 401).

[0077] Feedback may be provided to the user regarding the accuracy / confidence with which the signal can be used to predict the user's focus of attention (operation 405). For example, the feedback to the user may take the form of a graphical, textual, tactile, or auditory message. Thus, for example, if the signal is not informative (no), the user is notified, for example, by instructing the user to reduce blinking or stop glancing away from the object. Alternatively, the user may be notified that the signal does appear to be informative (yes), for example, providing positive feedback to confirm that the user is making progress.

[0078] The test of whether a signal is informative can be a comparison with a threshold (e.g., a threshold signal-to-noise ratio, etc.). However, it is also conceivable that the test can be implemented by one or more algorithms specifically designed to predict whether a signal is informative (and label it accordingly). For example, the algorithm can use an adaptive Bayesian classifier or a linear discriminant analysis (LDA) classifier. In fact, some classifiers (such as LDA or Bayesian classifiers) are entirely determined by the mean and variance of the BCI data from each class (e.g., "informative", "non-informative") and the number of samples in each class (which are incrementally and robustly updated with new input data without knowing the class labels). This is a pseudo-supervised approach in which a posteriori labels estimated by the Bayesian / LDA classifier are used in the adaptation process. When the probability that the signal is clean is not high enough, the incorrectly estimated labels will disrupt the parameter adaptation, similar to noise or outliers in supervised learning.

[0079] Once the signal is deemed informative (operation 404, yes), the model weights may be set (operation 406). Model weights are discussed below, and a specific example is Figure 6 The signal is associated with object A (operation 407), which means that the user's focus is determined on object A using the current signal.

[0080] By limiting the user to a single stimulus (object A) at a time, the model can ensure that the association between the signal and the object is correct. This process can be repeated, asking the user to attend to objects B and C in sequence. In effect, by showing only one object at a time in the initial sequence, the user is given fewer choices. The resulting multiple weights form a decoding model used to decode attentional focus.

[0081] Likewise, one or more objects can be presented to the user while another targeted approach is used to determine the true association between attention focus and signal (e.g., using a short supervised training session that tracks the user's eye movements).

[0082] In these cases, the model weights are set using reliable knowledge of the association.This embodiment is suitable for situations where no prior information is available, and can be characterized as a "cold start" or "from scratch" embodiment.

[0083] The calibration phase can also be started by using predetermined weights, and then adjusting these weights based on their accuracy for the current user / electrode positioning, etc. These weights can be obtained, for example, from a previous session from the user, or from an "average" model obtained by training the algorithm on a database of previous EEG recordings. Although it is not expected that models based on predetermined weights will fit the neural signals of the current user perfectly, it has been found that these models provide an effective approximation on which the calibration methods of the present disclosure can be built. Such embodiments can be characterized as "hot start" embodiments. In certain embodiments, the use of predetermined weights can allow the initial sequence (operations 401-407) to be replaced with a calibration by retrieving a model from storage with a predetermined (i.e., using the old user's weighting data and / or default values ​​or average values ​​of the weights), thereby effectively making the initial sequence optional.

[0084] Then, a validation sequence is performed (operations 408-417) to verify the accuracy of the decoding model.

[0085] At operation 408, the modulation of object A is modified. The display (e.g., by DGU 106) is caused to reflect the modification (operation 409). As the user continues to focus on object A (operation 410), a confirmation neural signal is acquired (operation 411). Using the same optional process as for determining the "informative" status of the neural signal at operations 404 and 405, the quality of the confirmation neural signal is optionally determined (operation 412).

[0086] Feedback can again be provided to the user regarding the accuracy / confidence with which the signal can be used to predict the user's focus of attention (operation 413, similar to operation 405). Thus, if the confirmation neural signal is determined to be "non-informative" (No), the user can be informed that the confirmation sequence will continue. Confirmation neural signals continue to be collected until a decision is made (by the SPU 105, for example) that the confirmation neural signal is informative (operation 412, Yes). This positive result can also be provided to the user (operation 415).

[0087] The model weights may be adjusted (operation 416) to reflect the neural signal that produced the positive result, and the association between the signal and object A is now confirmed (operation 417) so that the model can be reliably used to identify the focus of attention in subsequent operation of the BCI.

[0088] A key aspect of this calibration method is providing real-time feedback to improve focus. Receiving real-time feedback has been found to motivate users to focus more, leading to improved accuracy and significantly increasing user satisfaction. Confirmation sequences can be applied at intervals to ensure calibration remains reliable.

[0089] exist Figure 5 In the figure, it is shown that Figure 4The same process embodiment, except that in the operation of acquiring the neural signal, the neural signal is now also filtered in both the initial sequence (operation 501) and the confirmation sequence (operation 502).

[0090] Figure 6 A technique for constructing a decoding model is shown, wherein model weights can be set and adjusted (as done in operation 406). The decoding model is configured to decode the attention focus of the user (whose neural response is being modeled). The decoding model is trained to "reconstruct" one or more time-varying features of the sensory stimulus being attended to (e.g., a target object), hereinafter referred to as "modulation signals." The time-varying features can be changes in brightness, contrast, color, frequency, size, position, or any stimulus feature whose effects can be observed directly or indirectly on the recorded neural signals.

[0091] As described above, an initial phase (operation 602) is performed in which a set of neural signals Xi is obtained for each of N target subjects (where i is a member of [1, ..., N]) and the time-varying stimulus features SM i is known.

[0092] In operation 604, the neural signal X i The corresponding time-varying stimulus feature SM i Synchrony. Temporally varying stimulus featuresSM i Conveniently provide timestamps or synchronization patterns.

[0093] During the pre-processing step (operation 606), the raw neural signal Xi is optionally denoised to optimize the signal-to-noise ratio (SNR). Taking EEG as an example, the data may be heavily contaminated by multiple noise sources: external / exogenous noise may originate from electronic artifacts (e.g., 50Hz or 60Hz line noise), while biological / endogenous noise may come from muscle artifacts, eye blinks, electrocardiograms, brain activity unrelated to the task, etc.

[0094] In one or more embodiments, the neural signals X1, X2, ..., X N The modulation signal of the attended stimulus may be denoised before reconstruction. For example, this denoising step may comprise a simple high-pass filter of about 40 Hz to extract the signal X1, X2, ..., X N Removes high-frequency activity including line noise.

[0095] Multivariate methods such as principal component analysis (PCA), independent component analysis (ICA), canonical component analysis (CCA) or any of their variants can also be used, allowing the separation of "useful" neural signal components (i.e., originating from task-related brain activity) from less relevant components.

[0096] During a subsequent step (operation 608), the reconstruction model parameters are estimated. This estimation can be done in a way that minimizes the reconstruction error. The reconstruction method can take the form of a combination of multiple parameters of the neural signal. These combined parameters are determined analytically from mathematical equations to estimate the optimal parameters of the combination, that is, for a given stimulus, for multiple neural signals X i,j The value α to apply to the reconstructed model j is recorded, generating a reconstructed modulation signal MSR that optimally corresponds to the modulation signal of the stimulus of interest, i.e., the value with the minimum reconstruction error. In this context, the term "weight" refers to a linear or nonlinear combination of EEG channel data that maximizes the similarity to the stimulus of interest.

[0097] In one or more embodiments, the value α j Can be fixed (ie, independent of time).

[0098] In other embodiments, these values ​​(model weights) may be adjusted in real time in order to account for possible adaptations of the user's neural activity, or possible changes in the SNR during a recording session.

[0099] In some embodiments, the reconstruction model is a linear model that is constructed by the neural signal X i,j The linear combination of produces the modulation signal MSR. In this case, the combination parameter is the linear combination coefficient αj, and the mathematical equation is a linear equation, and the reconstructed modulation signal MSR is obtained by the following formula from the neural signals X1, X2, ..., X N The linear combination of :

[0100] MSR=∑ j α j X j

[0101] In alternative embodiments, more complex models can be used. One class of such models allows us to use a neural network, where the modulation signal MSR is obtained by applying the neural signals X1, X2, ..., X N It is obtained by cascading nonlinear mathematical operations.

[0102] After a stimulus has been "reconstructed", it can be compared with all the different stimuli presented to the user. The stimulus of interest (target) corresponds to the stimulus whose temporal variation characteristics SM are most similar to the reconstructed MSR.

[0103] For example, a convolutional network can be trained to match any input neural signal X with a modulation signal MSR, such that for two neural signals X1 and X2 recorded at different time points, two modulation signals MSR1 and MSR2 are generated, and when the user's attention is focused on the same target, the two modulation signals MSR1 and MSR2 are similar, and when the user's attention is focused on different targets, the two modulation signals MSR1 and MSR2 are different. Several mathematical definitions of similarity can be used (for example, a simple Pearson correlation coefficient, or the inverse of the Euclidean distance, mutual information, etc.). The reconstructed modulation signal is the multidimensional signal R generated by the neural network from the newly acquired EEG data.

[0104] The BCI described above can be used in conjunction with real-world objects to enable the objects to be controlled or otherwise interacted with. In some embodiments, the generation of stimuli is handled by one or more light sources (such as light emitting diodes (LEDs)) provided in association with (or even on the surface of) the controllable object.

[0105] In certain embodiments, the generation of the stimuli is handled by a projector or a scanning laser device such that the visual stimuli are projected onto the controllable object and the controllable object outputs the visual stimuli by reflecting the projected stimuli.

[0106] As in the case of BCIs using a display screen, the user interacts with objects on the screen (e.g. Figure 1 By interacting with a controllable object in the present disclosure, the controllable object can be presented with visual stimuli with characteristic modulation (e.g., flickering stimuli), so that the neural response to the presence of these stimuli becomes obvious and can be decoded from the neural signals collected by the neural signal collection device (such as an EEG device).

[0107] In some embodiments, determining the focus of attention on a visual display of a controllable device is used to send a command to the controllable object. The controllable object can then perform an action based on the command: for example, the controllable object can emit a sound, unlock a door, turn a device on or off, change its operating state, etc. The action can also provide the user with visual or other feedback associated with the controllable object: this can be used in conjunction with the positive feedback loop discussed above, but can also provide real-time indication of valid selections of actions associated with the controllable object.

[0108] As the user becomes more attentive using this feedback stimulus, the user-specific stimulus reconstruction model built in the initial or confirmation phase of operation of the BCI system is observed to be more accurate while being built faster.

[0109] In subsequent operational phases, the use of feedback stimulation as described above leads to increased accuracy and speed in real-time BCI applications. When no initial or confirmatory calibration steps are performed, the visual BCI system can be operated according to Figure 7The function box operation shown in .

[0110] In block 702, a device operatively coupled to a neural signal acquisition device and a stimulation generator (such as Figure 1 Hardware interface devices (such as EAU and DGU) Figure 1 The SPU) receives neural signals from the neural signal acquisition device.

[0111] In block 704 , the interface device determines the strength of a component of a neural signal having a characteristic associated with a modulation of a respective characteristic of the or each visual stimulus.

[0112] In box 706, the interface device determines which of the at least one visual stimulus is associated with the user's object of interest based on the neural signal, infers the object of interest by comparing the corresponding characteristic modulation of the at least one visual stimulus with the modulation of the reconstructed sensory stimulus, and reconstructs the reconstructed sensory stimulus using the received neural signal, wherein the object of interest is the visual stimulus whose time-varying characteristics are most similar to the reconstructed sensory stimulus.

[0113] Figure 9 is a block diagram illustrating an example software architecture 906 that may be used in conjunction with the various hardware architectures described herein. Figure 9 is a non-limiting example of a software architecture, and it should be understood that many other architectures can be implemented to facilitate the functionality described herein. The software architecture 906 can be implemented in a system such as Figure 10 1000, which includes, among other things, a processor 1004, a memory 1006, and input / output (I / O) components 1018. Representative hardware layers 952 are shown and may represent, for example, Figure 10 The machine 1000 is shown as a computer program product. A representative hardware layer 952 includes a processing unit 954 having associated executable instructions 904. Executable instructions 904 represent executable instructions of a software architecture 906, including implementations of the methods, modules, and the like described herein. The hardware layer 952 also includes a memory / storage module, shown as a memory and / or storage device 956, which also has executable instructions 904. The hardware layer 952 may also include other hardware 958, such as specialized hardware for interfacing with EEG electrodes and / or for interfacing with a display device.

[0114] exist Figure 9In the example architecture of , software architecture 906 can be conceptualized as a stack of layers, wherein each layer provides specific functionality. For example, software architecture 906 may include layers such as operating system 902, library 920, framework or middleware 918, application 916, and presentation layer 914. In operation, applications 916 and / or other components within a layer may call application program interface (API) calls 908 through the software stack and receive responses as messages 910. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile or dedicated operating systems may not provide framework / middleware 918, while other operating systems may provide such layers. Other software architectures may include additional layers or different layers.

[0115] The operating system 902 can manage hardware resources and provide common services. The operating system 902 can include, for example, a kernel 922, services 924, and drivers 926. The kernel 922 can act as an abstraction layer between the hardware and other software layers. For example, the kernel 922 can be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services 924 can provide other common services to other software layers. Drivers 926 can be responsible for controlling the hardware or interfacing with the underlying hardware. For example, depending on the hardware configuration, drivers 926 can include display drivers, EEG device drivers, camera drivers, drives, flash drives, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), drivers, audio drivers, power management drivers, and more.

[0116] The libraries 920 may provide a common infrastructure that can be used by the applications 916 and / or other components and / or layers. The libraries 920 generally provide functionality that allows other software modules to perform tasks more easily than by directly interfacing with the underlying operating system 902 functions (e.g., kernel 922, services 924, and / or drivers 926). The libraries 920 may include system libraries 944 (e.g., C standard libraries), which may provide functionality such as memory allocation functions, string manipulation functions, mathematical functions, and the like. In addition, the libraries 920 may include API libraries 946, such as media libraries (e.g., libraries for supporting the presentation and manipulation of various media formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG), graphics libraries (e.g., the OpenGL framework for presenting 2D and 3D graphics content on a display), database libraries (e.g., SQLite, which may provide various relational database functions), web libraries (e.g., WebKit, which may provide web browsing functionality), and the like. The library 920 may also include various other libraries 948 to provide many other APIs to the applications 916 and other software components / modules.

[0117] The framework 918 (sometimes also referred to as middleware) provides a higher-level common infrastructure that can be used by applications 916 and / or other software components / modules. For example, the framework / middleware 1118 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. The framework / middleware 918 can provide a wide range of other APIs that can be used by applications 916 and / or other software components / modules, some of which may be specific to a particular operating system or platform.

[0118] Applications 916 include built-in applications 938 and / or third-party applications 940 .

[0119] Applications 916 may create user interfaces to interact with users of the system using built-in operating system functionality (e.g., kernel 922, services 924, and / or drivers 926), libraries 920, or framework / middleware 918. Alternatively or additionally, in some systems, interaction with the user may occur through a presentation layer, such as presentation layer 914. In these systems, application / module "logic" may be separated from aspects of the application / module that interact with the user.

[0120] Figure 10 is a block diagram illustrating components of a machine 1000 capable of reading instructions from a machine-readable medium (eg, a machine-readable storage medium) and performing any one or more of the methodologies discussed herein, according to some example embodiments. Figure 1 The SPU, EAU, and DGU of the machine 1000 may each be implemented as a machine having some or all of the components of the machine 1000. Specifically, Figure 10A diagrammatic representation of a machine 1000 in the example form of a computer system is shown within which instructions 1010 (e.g., software, programs, applications, applet, application software, or other executable code) may be executed for causing the machine 1000 to perform any one or more of the methodologies discussed herein. Thus, the instructions 1010 may be used to implement the modules or components described herein. The instructions 1010 transform a general-purpose, unprogrammed machine into a specialized machine 1000 that is programmed to perform the functions described and illustrated in the manner described. In alternative embodiments, the machine 1000 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1000 may operate in the capacity of a server or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1000 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smartphone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing, sequentially or otherwise, the instructions 1010 specifying actions to be taken by the machine 1000. Furthermore, while only a single machine 1000 is shown, the term "machine" should also be construed to include a collection of machines that individually or jointly execute the instructions 1010 to perform any one or more of the methodologies discussed herein.

[0121] The machine 1000 may include a processor 1004, a memory 1006, and input / output (I / O) components 1018, which may be configured to communicate with each other, for example, via a bus 1002. In an example embodiment, the processor 1004 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1008 and a processor 1012 that may execute instructions 1010. The term "processor" is intended to include a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Figure 10 Multiple processors are shown, but machine 1000 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0122] The memory 1006 may include a memory 1014 (such as a main memory, static memory, or other memory storage device) and a storage unit 1016, both of which are accessible by the processor 1004, such as via the bus 1002. The storage unit 1016 and the memory 1014 store instructions 1010 that embody any one or more of the methods or functions described herein. During execution of the instructions 1010 by the machine 1000, the instructions 1010 may also reside, completely or partially, within the memory 1014, within the storage unit 1016, within at least one of the processors 1004 (e.g., within a cache of the processor), or any suitable combination thereof. Thus, the memory 1014, the storage unit 1016, and the memory of the processor 1004 are examples of machine-readable media.

[0123] As used herein, a “machine-readable medium” refers to a device capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of memory (e.g., erasable programmable read-only memory (EEPROM)), and / or any suitable combination thereof. The term “machine-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) that can store instructions 1010. The term “machine-readable medium” should also be taken to include any medium or combination of media that can store instructions (e.g., instructions 1010) for execution by a machine (e.g., machine 1000), such that the instructions, when executed by one or more processors (e.g., processor 1004) of machine 1000, cause machine 1000 to perform any one or more of the methods described herein. Thus, a “machine-readable medium” refers to a single storage device or device, as well as a “cloud-based” storage system or storage network comprising multiple storage devices or devices. The term “machine-readable medium” does not include the signal itself.

[0124] The input / output (I / O) components 1018 may include a variety of components to receive input, provide output, generate output, send information, exchange information, collect measurements, etc. The specific input / output (I / O) components 1018 included in a particular machine will depend on the type of machine. For example, user interface machines and portable machines such as mobile phones may include touch input devices or other such input mechanisms, while headless server machines may not include such touch input devices. It should be understood that the input / output (I / O) components 1018 may include Figure 10 Many other components not shown.

[0125] The input / output (I / O) components 1018 are grouped according to their functionality merely to simplify the following discussion and are by no means limiting. In various exemplary embodiments, the input / output (I / O) components 1018 may include output components 1026 and input components 1028. The output components 1026 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistive mechanisms), other signal generators, and the like. The input components 1028 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide the location and / or force of touches or touch gestures, or other tactile input components), audio input components (e.g., microphones), and the like.

[0126] In further example embodiments, the input / output (I / O) component 1018 may include, among a wide range of other components, a biometric component 1030, a motion component 1034, an environmental component 1036, or a positioning component 1038. For example, the biometric component 1030 may include components that detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves, such as output from an EEG device), identify people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1034 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environmental component 1036 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor component (e.g., a gas detection sensor for detecting hazardous gas concentrations for safety purposes or measuring pollutants in the atmosphere), or other components that may provide indications, measurements, or signals corresponding to the surrounding physical environment. The positioning component 1038 may include a positioning sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter or a barometer that detects air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), etc.

[0127] Various technologies may be used to implement communications. Input / output (I / O) components 1018 may include a communications component 1040 operable to couple machine 1000 to network 1032 or device 1020 via coupler 1024 and coupler 1022, respectively. For example, communications component 1040 may include a network interface component or other suitable device to interface with network 1032. In further examples, communications component 1040 may include a wired communications component, a wireless communications component, a cellular communications component, a near field communications (NFC) component, Components (e.g. Low power consumption) Components and other communication components to provide communication via other modes. Device 1020 can be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via a universal serial bus (USB)). In the case where the EEG device or display device is not integrated with the machine 1000, the device 1020 can be an EEG device and / or a display device.

[0128] Although an overview of the inventive subject matter has been described with reference to specific example embodiments, various modifications and changes may be made to these embodiments without departing from the broader scope of the embodiments of the present disclosure. These embodiments of the inventive subject matter may be referred to herein, individually or collectively, by the term "invention," which is merely for convenience and is not intended to voluntarily limit the scope of this application to any single disclosure or inventive concept, if in fact there is more than one disclosure.

[0129] The embodiments shown herein are described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Therefore, the detailed description should not be construed in a limiting sense, and the scope of the various embodiments is defined solely by the appended claims and the full scope of equivalents to which such claims are entitled.

[0130] As used herein, the term "or" may be interpreted in an inclusive or exclusive sense. In addition, multiple instances may be provided for the resources, operations, or structures described herein as a single instance. Furthermore, the boundaries between the various resources, operations, modules, engines, and data stores are somewhat arbitrary, and particular operations are described in the context of particular illustrative configurations. Other functional allocations are conceivable and may fall within the scope of the various embodiments of the present disclosure. In general, structures and functions presented as separate resources in the example configurations may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as separate resources. These and other variations, modifications, additions, and improvements fall within the scope of the embodiments of the present disclosure as represented by the appended claims. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive.

[0131] Example

[0132] To better illustrate the methods and systems disclosed herein, a non-limiting list of examples is provided here:

[0133] Example 1: A method for adaptive calibration of a brain-computer interface, comprising:

[0134] a. Displaying one or more objects on a display screen;

[0135] b. modulating the display characteristics of the one or more objects so that each object has a unique time-varying display characteristic;

[0136] c. The user views the first display object of the one or more objects on the display screen;

[0137] d. Detecting and recording, by the system, a first set of current neural signals detected from the user;

[0138] e. associating the first set of current neural signals with the first display object on the display screen;

[0139] f. changing the modulation of the display characteristics of the first display object;

[0140] g. detecting and recording by the system a second set of current neural signals detected from the user;

[0141] h. associating the second set of neural signals with the first display object to confirm that the first display object is being focused on;

[0142] i. If confirmed, modeling the recorded neural signal associated with the first displayed object;

[0143] j. using a model of the recorded neural signals to identify when the user is focusing on the first displayed object;

[0144] k. When the user focuses on each of the one or more objects in turn, repeat steps a to j for the one or more objects;

[0145] 1. Repeat steps a through k while modifying the modeling weights and the modulation display characteristics to optimize the accuracy of the determination of the object of interest.

[0146] Example 2: The method as described in Example 1 further includes:

[0147] determining whether the neural signal contains artifacts; and

[0148] If the neural signal contains artifacts, it is determined whether the neural signal is informative.

[0149] Example 3: The method of Example 1 further comprising:

[0150] Filtering neural signals to reduce noise;

[0151] determining whether the filtered neural signal contains artifacts; and

[0152] If the filtered neural signal contains artifacts, it is determined whether the filtered neural signal is informative.

[0153] Example 4: The method as described in Example 2 further includes:

[0154] If the neural signal is available, feeding back to the user an indication that the neural signal is available;

[0155] If the neural signal is unavailable, providing feedback to the user indicating that the neural signal is unavailable; and

[0156] The user is instructed to take one or more actions to correct the unusable neural signal.

[0157] Example 5: The method as described in Example 3 further includes:

[0158] If the filtered neural signal is available, feeding back to the user an indication that the filtered neural signal is available;

[0159] If the filtered neural signal is unavailable, providing feedback to the user indicating that the filtered neural signal is unavailable; and

[0160] The user is instructed to take the one or more actions to correct the unusable filtered neural signal.

[0161] Example 6: A computer-implemented method, in at least one processor, comprising:

[0162] In an initial stage, a first set of neural signals of a user is received from a neural signal acquisition device, and the user perceives sensory information in a training sequence, the training sequence including at least one sensory stimulus, each sensory stimulus having at least one predetermined corresponding characteristic;

[0163] determining neural response data associated with each of the one or more sensory stimuli from the first set of neural signals and the training sequence, the neural response data being combined to generate a model of the user's neural responses to the sensory stimuli, the model comprising weights applied to features of the neural signals;

[0164] In a calibration phase, a second set of neural signals of the user is received from the neural signal acquisition device, and the user further perceives sensory information in a confirmation sequence, the confirmation sequence including at least one sensory stimulus among the sensory stimuli;

[0165] Using the model, estimating which of the sensory stimuli is the user's object of attention;

[0166] determining whether the estimated identification of the object of attention corresponds to the sensory stimulus in the confirmation sequence;

[0167] If a correspondence is determined, modifying the weight; and

[0168] If a mismatch is determined, the sensory stimulation in the confirmation sequence is modified.

[0169] Example 7: The method of Example 6, wherein modifying the sensory stimuli in the confirmation sequence includes modifying at least one of: a corresponding characteristic of at least one sensory stimulus; reducing the number of sensory stimuli in the confirmation sequence; and changing the appearance of the sensory stimuli in the confirmation sequence.

[0170] Example 8: The method of Example 6 or Example 7, wherein at least one sensory stimulus is a visual stimulus, wherein the training sequence includes training image data displayed for viewing by a user, and wherein the training image data includes the visual stimulus.

[0171] Example 9: The method of Example 8, wherein the or each visual stimulus is displayed in a known order.

[0172] Example 10: The method of Example 8 or Example 9, wherein the training image data comprises the or each visual stimulus displayed at a known display position.

[0173] Example 11: A method according to any one of Examples 8 to 10, wherein the predetermined corresponding characteristic modulation is a change in at least one of display position, brightness, contrast, flicker frequency, color and / or scale.

[0174] Example 12: The method of any of Examples 5 to 11, wherein at least one sensory stimulus is an audio stimulus, the training sequence comprises training audio data heard by the user, and wherein the training audio data comprises the audio stimulus.

[0175] Example 13: The method of any of Examples 5 to 12, wherein at least one sensory stimulus is a tactile stimulus, the training sequence includes training tactile data sensed by the user, and wherein the training tactile data includes the tactile stimulus.

[0176] Example 14: The method of any one of Examples 5 to 13, wherein estimating which of the sensory stimuli is the object of attention of the user comprises:

[0177] comparing different stimuli in the training sequence with reconstructed sensory stimuli reconstructed using the second set of neural signals; and

[0178] The object of attention is determined to be the stimulus whose time-varying characteristics are most similar to the reconstructed sensory stimulus.

[0179] Example 15: The method of any one of Examples 5 to 14, further comprising repeating the operations of the calibration phase.

[0180] Example 16: The method of any of Examples 5 to 15, wherein the sensory stimulus corresponds to a control item, the method further comprising performing a control task with respect to the control item associated with the inferred attentional focus.

[0181] Example 17: The method of any of Examples 5 to 16, further comprising filtering the received neural signals to mitigate noise before determining the neural response data for generating the model.

[0182] Example 18: The method of any one of Examples 5 to 17, further comprising:

[0183] determining whether the received neural signal includes an artifact; and

[0184] Among them, the neural signal contains artifacts and the received neural signal is rejected.

[0185] Example 19: The method of Example 18 further comprises:

[0186] In the case where the neural signal does not include artifacts, providing feedback to the user indicating that the neural signal is available;

[0187] In the case where the neural signal includes an artifact, if the neural signal is not available, providing feedback to the user indicating that the neural signal is not available; and

[0188] The user is instructed to take one or more actions to correct the unusable neural signals.

[0189] Example 20: A brain-computer interface system, comprising:

[0190] a sensation information generating unit configured to output a signal for reproduction by a reproduction device, the reproduced signal being perceived by a user as sensation information;

[0191] a neural signal acquisition device configured to acquire neural signals associated with a user; and

[0192] A signal processing unit operatively coupled to the sensory information generating unit and the neural signal collecting device, the signal processing unit being configured to:

[0193] In an initial stage, a first set of neural signals of a user is received from a neural signal acquisition device, and the user perceives sensory information in a training sequence, where the training sequence includes at least one sensory stimulus, each sensory stimulus having at least one predetermined corresponding characteristic;

[0194] determining neural response data associated with each of the one or more sensory stimuli from the first set of neural signals and the training sequence, the neural response data being combined to generate a model of the user's neural responses to the sensory stimuli, the model comprising weights applied to features of the neural signals;

[0195] In a calibration phase, a second set of neural signals of the user is received from the neural signal acquisition device, and the user further perceives sensory information in a confirmation sequence, the confirmation sequence including at least one sensory stimulus among the sensory stimuli;

[0196] Using the model, estimating which of the sensory stimuli is the user's object of attention;

[0197] determining whether the estimated identification of the object of interest corresponds to a sensory stimulus in the confirmation sequence;

[0198] If a correspondence is determined, modifying the weight; and

[0199] If a mismatch is determined, the sensory stimulation in the confirmation sequence is modified.

[0200] Example 21: The brain-computer interface system described in Example 20,

[0201] wherein at least one sensory stimulus is a visual stimulus,

[0202] The sensory information in the training sequence includes training image data displayed and viewed by the user, the training image data including visual stimulation,

[0203] wherein another sensory information in the confirmation sequence includes confirmation image data displayed for viewing by the user, the confirmation image data including a visual stimulus, and

[0204] The sensory information generating unit includes a display generating unit (DGU) configured to cause a display to reproduce the training image data and the confirmation image data.

[0205] Example 22: A non-transitory computer-readable storage medium storing instructions that, when executed by a computer system, cause the computer system to perform operations comprising:

[0206] In an initial stage, a first set of neural signals of a user is received from a neural signal acquisition device, and the user perceives sensory information in a training sequence, the training sequence including at least one sensory stimulus, each sensory stimulus having at least one predetermined corresponding characteristic;

[0207] determining neural response data associated with each of the one or more sensory stimuli from the first set of neural signals and the training sequence, the neural response data being combined to generate a model of the user's neural responses to the sensory stimuli, the model comprising weights applied to features of the neural signals;

[0208] In a calibration phase, a second set of neural signals of the user is received from the neural signal acquisition device, and the user further perceives sensory information in a confirmation sequence, the confirmation sequence including at least one sensory stimulus among the sensory stimuli;

[0209] Using the model, estimating which of the sensory stimuli is the user's object of attention;

[0210] determining whether the estimated identification of the object of interest corresponds to a sensory stimulus in the confirmation sequence;

[0211] If a correspondence is determined, modifying the weight; and

[0212] If a mismatch is determined, the sensory stimulation in the confirmation sequence is modified.

[0213] Example 23: A computer-readable storage medium carrying instructions that, when executed by a computer, cause the computer to perform the method of any one of Examples 5 to 19.

[0214] These figures are exemplary and should not be construed as limiting the scope or steps to only those. The same method applies if the system displays only a single object. The same method applies if the system displays three or more objects. Although the objects displayed have different two-dimensional shapes, this is for clarity. The objects can have any shape, any color, and can be located anywhere on the display. The sequence of consecutive object inspections and calibrations can be completed in any order, not necessarily in a sequence of adjacent objects.

[0215] Although described through a number of detailed exemplary embodiments, the portable device for collecting electroencephalographic signals according to the present disclosure includes various variations, modifications and improvements that are obvious to those skilled in the art, and it should be understood that these various variations, modifications and improvements fall within the scope of the subject matter of the present disclosure as defined in the appended claims.

Claims

1. A computer-implemented method for calibration of a brain-computer interface, comprising, in at least one processor: In an initial stage, a first set of neural signals of a user is received from a neural signal acquisition device, wherein the user perceives sensory information in a training sequence, wherein the training sequence includes a plurality of sensory stimuli, each of the plurality of sensory stimuli having a corresponding characteristic modulation, wherein The plurality of sensory stimuli include a first sensory stimulus; determining neural response data associated with each of the plurality of sensory stimuli from the first set of neural signals and the training sequence, the neural response data being combined to generate a model of the user's neural responses to the plurality of sensory stimuli, the model comprising weights applied to features of the neural signals; In a calibration phase, a second set of neural signals of the user is received from the neural signal acquisition device, and the user further perceives sensory information in a confirmation sequence, the confirmation sequence including a modified sensory stimulus obtained by modifying the characteristic modulation of the first sensory stimulus and at least one other sensory stimulus from the plurality of sensory stimuli; estimating which sensory stimulus in the confirmation sequence is the object of attention of the user using the model and the second set of neural signals of the user; as well as The weights are modified according to the estimated object of interest corresponding to the modified sensory stimulus in the confirmed sequence.

2. The method according to claim 1, wherein At least one sensory stimulus of the plurality of sensory stimuli is a visual stimulus, wherein the training sequence comprises training image data displayed for viewing by the user, and wherein the training image data comprises the visual stimulus.

3. The method according to claim 2, wherein: Each visual stimulus was presented in a known order.

4. The method according to claim 2, wherein: The training image data includes each visual stimulus displayed at a known display position.

5. The method according to claim 2, wherein: The corresponding characteristic modulation is a change in at least one of display position, brightness, contrast, flicker frequency, color or scale.

6. The method according to claim 1, wherein At least one sensory stimulus of the plurality of sensory stimuli is an audio stimulus, the training sequence comprises training audio data heard by the user, and wherein the training audio data comprises the audio stimulus.

7. The method according to claim 1, wherein At least one sensory stimulus of the plurality of sensory stimuli is a tactile stimulus, the training sequence comprises training tactile data to be felt by the user, and wherein the training tactile data comprises the tactile stimulus.

8. The method according to claim 1, wherein Estimating which sensory stimulus in the confirmation sequence is the object of interest of the user includes: comparing the different sensory stimuli in the validation sequence with reconstructed sensory stimuli reconstructed using the second set of neural signals; and The object of interest is determined to be a sensory stimulus whose temporal variation characteristics are most similar to the reconstructed sensory stimulus.

9. The method of claim 1, further comprising repeating the calibration phase.

10. The method according to claim 1, wherein The sensory stimulus corresponds to a control item, and the method further includes performing a control task with respect to the control item associated with the object of interest.

11. The method according to claim 1 , further comprising: The received first set of neural signals is filtered to mitigate noise before determining the neural response data for generating the model.

12. The method according to claim 1, further comprising: determining whether the received first set of neural signals or the second set of neural signals includes an artifact; and In a case where the received first set of neural signals or the second set of neural signals contains artifacts, the first set of neural signals or the second set of neural signals are rejected.

13. The method according to claim 12, further comprising: In a case where the received first set of neural signals or the second set of neural signals do not include artifacts, feeding back to the user an indication that the received first set of neural signals or the second set of neural signals are usable; In a case where the received first set of neural signals or the second set of neural signals include artifacts, if the received first set of neural signals or the second set of neural signals is unusable, feeding back to the user an indication that the received first set of neural signals or the second set of neural signals is unusable; and The user is instructed to take one or more actions to correct the unusable neural signals.

14. A brain-computer interface system, comprising: a sensation information generating unit configured to output a signal for reproduction by a reproduction device, the reproduced signal being perceived by a user as sensation information; a neural signal acquisition device configured to acquire neural signals associated with a user; as well as a signal processing unit operatively coupled to the sensory information generating unit and the neural signal collecting device, the signal processing unit being configured to: In an initial stage, a first set of neural signals of the user is received from the neural signal acquisition device, and the user perceives the sensory information in a training sequence, the training sequence including a plurality of sensory stimuli, each of the plurality of sensory stimuli having a corresponding characteristic modulation, wherein the plurality of sensory stimuli include a first sensory stimulus; determining neural response data associated with each of the plurality of sensory stimuli from the first set of neural signals and the training sequence, the neural response data being combined to generate a model of the user's neural responses to the plurality of sensory stimuli, the model comprising weights applied to features of the neural signals; In a calibration phase, a second set of neural signals of the user is received from the neural signal acquisition device, and the user further perceives sensory information in a confirmation sequence, the confirmation sequence including a modified sensory stimulus obtained by modifying the characteristic modulation of the first sensory stimulus and at least one other sensory stimulus from the plurality of sensory stimuli; estimating which sensory stimulus in the confirmation sequence is the object of attention of the user using the model and the second set of neural signals of the user; and The weights are modified according to the estimated object of interest corresponding to the modified sensory stimulus in the confirmed sequence.

15. The brain-computer interface system according to claim 14, in, At least one sensory stimulus among the plurality of sensory stimuli is a visual stimulus, wherein the sensory information in the training sequence includes training image data displayed and viewed by the user, the training image data including the visual stimulus, wherein the sensory information in the confirmation sequence includes confirmation image data displayed for viewing by the user, the confirmation image data including the visual stimulus, and The sensory information generating unit includes a display generation unit (DGU) configured to cause a display to reproduce the training image data and the confirmation image data.

16. A computer-readable storage medium carrying instructions that, when executed by a computer, cause the computer to perform operations comprising: In an initial stage, a first set of neural signals of a user is received from a neural signal acquisition device, wherein the user perceives sensory information in a training sequence, wherein the training sequence includes a plurality of sensory stimuli, each of the plurality of sensory stimuli having a corresponding characteristic modulation, wherein The plurality of sensory stimuli include a first sensory stimulus; determining neural response data associated with each of the plurality of sensory stimuli from the first set of neural signals and the training sequence, the neural response data being combined to generate a model of the user's neural responses to the plurality of sensory stimuli, the model comprising weights applied to features of the neural signals; In a calibration phase, a second set of neural signals of the user is received from the neural signal acquisition device, and the user further perceives sensory information in a confirmation sequence, the confirmation sequence including a modified sensory stimulus obtained by modifying the characteristic modulation of the first sensory stimulus and at least one other sensory stimulus from the plurality of sensory stimuli; estimating which sensory stimulus in the confirmation sequence is the object of attention of the user using the model and the second set of neural signals of the user; as well as The weights are modified according to the estimated object of interest corresponding to the modified sensory stimulus in the confirmed sequence.

Images