Brain-computer interface with high-speed target tracking feature

The hybrid BCI system combines eye tracking and neural activity monitoring to enhance BCIs, addressing speed and accuracy issues, enabling real-world applications and broader accessibility.

JP2024167203AInactive Publication Date: 2024-12-03NEURABLE INC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2024124668
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2017-08-23
Filing Date
2024-07-31
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing brain-computer interfaces (BCIs) suffer from low speed and accuracy, requiring extensive training and being limited to laboratory settings, making them unsuitable for real-world applications and general public use.

Method used

A hybrid BCI system integrating real-time eye movement tracking with neural activity monitoring, using a video-based eye tracker and neural recording headset to enhance pointing and action control functions, allowing for high-speed and precise human-machine interaction.

Benefits of technology

Enables fast and accurate human-machine interaction without the need for traditional input devices, suitable for real-world tasks and accessible to a broader audience, reducing cognitive load and improving user interface complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024167203000001_ABST
    Figure 2024167203000001_ABST
Patent Text Reader

Abstract

To provide a system, apparatus and method, as described in embodiments of the present description, for use in implementing a brain-computer interface that integrates real-time eye movement tracking with brain activity tracking to present and update a user interface strategically designed to achieve high speed and accuracy of human-machine interaction.SOLUTION: The embodiments described in the description also relate to implementing a hardware-independent brain-computer interface for mediating user interaction with a machine using real-time target tracking and online analysis of neural signals.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and benefit of U.S. Provisional Patent Application No. 62 / 549,253, filed August 23, 2017, and entitled "Brain-Computer Interface with High-Speed ​​Eye Tracking Features," the entire disclosure of which is incorporated herein by reference. [Background technology]

[0002] background

[0002] The embodiments described herein relate to systems, devices, and methods for use in implementing brain-computer interfaces that integrate real-time eye movement tracking with brain activity tracking to present and update user interfaces that are strategically designed for high speed and high accuracy of human-machine interaction. The embodiments described herein also relate to implementing hardware-independent brain-computer interfaces that mediate user operation of a machine using real-time eye tracking and online analysis of neural activity.

[0003]

[0003] A brain-computer interface (BCI) is a hardware and software communication system that allows a computer or external device to be controlled by brain activity alone, using a direct communication channel between a wired brain and an external device. BCIs are primarily designed as assistive technologies to provide direct access to operating machines and applications by interpreting brain signals. One of the main goals of BCI development is to provide communication capabilities to severely disabled people who are completely paralyzed or "immobilized" by neurological neuromuscular diseases such as amyotrophic lateral sclerosis, brain stem stroke, or spinal cord injury, where effective communication with others can be extremely difficult.

[0004]

[0004] Some known implementations of brain-computer interfaces include the Spella, such as that designed by Farwell and Donchin, in which the 26 letters of the alphabet, along with several other symbols and commands, are displayed on a screen in a 6x6 matrix with randomly flashing rows and columns. The user focuses their attention on the screen and continuously focuses on the letter they wish to write, while the brain's neural responses are monitored for signature signals. Once detected, the signature signal allows the system to identify the desired symbol. The Farwell-Donchin Spella allows normal subjects to spell at a rate of approximately 2 letters per minute.

[0005]

[0005] However, BCIs can be designed to assist even physically capable people in operating computers or other data processing machines and / or software applications without the need for traditional input / output interfaces such as a mouse and keyboard. BCIs can also provide an interface for more intuitive and natural interaction with computers than traditional input controls. In addition, BCIs can also be developed to provide many other functions, including augmenting, repairing, and mapping and exploring the cognitive and / or sensory-motor systems and their functions of humans and animals. Some applications of BCIs include word processors, adaptive web browsers, brain-based control of wheelchairs or neuroprostheses, and games, among others. However, most applications are designed exclusively for training or demonstration purposes. Summary of the Invention [Means for solving the problem]

[0006] overview

[0006] Described herein are systems, devices, and methods for various embodiments of a hardware-independent integrated oculomotor hybrid brain-computer interface (BCI) platform for tracking eye movements and brain activity to mediate real-time positioning of a user's gaze or attention and selection / activation of a desired action. The present disclosure presents an integrated hybrid BCI system to address the demand for brain-computer interfaces that operate at high speed and high accuracy. [Brief description of the drawings]

[0007] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1] 1 is a schematic diagram of a hybrid brain-computer interface system, according to one embodiment. [Figure 2A]

[0008] FIG. 1 is an illustration of a video-based eye tracker. [Figure 2B]

[0009] 1A-1D show front and rear views of a video-based eye-tracking device. [Figure 3A]

[0010] FIG. 1 is an illustration of an exemplary neurorecording headset. [Figure 3B]

[0011] 1A-1C show side (left) and top (right) views of example electrode placement locations on a user's scalp for recording neuro-brain signals. [Figure 3C]

[0012] An example of a response recorded from the brain indicating the type of neural activity is shown, the difference in event-related potentials (ERPs) elicited by a desired target stimulus (black) as opposed to a non-target stimulus (magenta) presented at the time indicated by the vertical dashed line. [Figure 4]

[0013] FIG. 1 is a schematic diagram of an example hybrid BCI system that uses example stimulus driver neural activity (P300 signal) to distinguish stimuli and select or deselect stimuli based on a user's intent. [Figure 5A]

[0014] FIG. 1A is an explanatory diagram of an example of a user interface before a user selection, and FIG. 1B is an explanatory diagram of an example of a user interface after a user selection;

[0015] (c) shows a user interacting with the (a) and (b) interfaces via a video-based eye tracker and neural recording headset. [Figure 5B]

[0016] FIG. 5C shows an example of signals generated and recorded by a video-based eye tracker and a neural recording headset monitoring brain activity. [Figure 5C]

[0016] FIG. 5C shows an example of signals generated and recorded by a video-based eye tracker and a neurorecording headset for monitoring brain activity. [Figure 5D]

[0017] We demonstrate the operation of a high-speed eye movement tracking system that can be implemented to initiate and carry out a desired action even before a subject consciously perceives their decision. [Figure 6]

[0018] FIG. 1 is a schematic diagram of a hybrid brain-computer interface device, according to one embodiment. [Figure 7]

[0019] 1 illustrates an example process of operation of a hybrid brain-computer interface device, according to one embodiment. [Figure 8A]

[0020] 1 shows an example of a set of brain signals recorded over time from a set of neural recording electrodes using a hybrid BCI system, according to one embodiment. [Figure 8B]

[0021] 1 illustrates an example of a set of neural brain signals recorded over time using a hybrid BCI system, according to another embodiment. [Figure 9A]

[0022] 1 illustrates an example of a user interface before (left) and after (right) a user selection, according to one embodiment. [Figure 9B]

[0023] 13 illustrates an example of another user interface example that follows different layouts before and after a user selection according to another embodiment. [Figure 10]

[0024] 1 shows several instances of an example of a user interface illustrating the operation of dynamic tagging of moving objects over time. [Figure 11]

[0025] FIG. 1 shows an illustration of the usage space of BCI systems based on user capabilities and types of BCI systems. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Detailed Description

[0026] The embodiments described herein relate to systems, devices, and methods for use in implementing hybrid brain-computer interfaces (BCIs) that integrate real-time eye movement tracking with brain activity tracking to present and update user interfaces that are strategically designed for high speed and high accuracy of human-machine interaction. The embodiments described herein also relate to the implementation of hardware-independent brain-computer interfaces that mediate user operation of a machine using real-time eye tracking and online analysis of neural brain signals.

[0009]

[0027] To make BCI technology more suitable for patients, useful to the general public, and used in controlling real-world tasks, the information transfer rate must be improved to match the pace of natural interaction, the error rate must be reduced, and the complexity of the interaction interface must be minimized compared to current implementations. In addition, BCI applications require high cognitive load from the user, and therefore the user interface needs to be improved to move away from the quiet laboratory environment to the real world. To make BCI devices and applications easier and more intuitive, improved devices and techniques in the implementation of brain-machine interfaces are required that operate at high speed and high accuracy to enable user-mediated action selection through natural intuitive processes.

[0010] Hybrid BCI system

[0028] As described herein, a BCI is a hardware and software communication system that allows brain activity alone to control a computer or external device. A hybrid BCI system includes hardware devices for displaying stimuli through an interface, positioning the user's attention on the interface, devices for recording and processing brain activity, and devices for controlling the interface, which may result in controlling the user's environment. These standard functions can be characterized as (1) pointing control functions, (2) action control functions, and (3) user interface functions. Pointing control functions can be analogized to a traditional pointing device such as a mouse pointer that allows the user to narrow down to a small number of manipulators to control. Action control functions can be analogized to a selection device such as a mouse click or a keyboard stroke that allows the user to effect change through an action. An action can be one of many things, such as activation or deactivation, a continuous change in the user interface (e.g., scrolling) or a discontinuous start and stop (e.g., highlighting, hovering, etc.), an abrupt change in the user interface with pinching, zooming, tilting, rotating, swiping, among others. Some other examples of action control through the user interface may include virtual keyboard control, menu navigation, actions to place and unplace objects or items, actions to move objects or items, actions to zoom in and / or zoom out of objects, movement or navigation of a first person observer or player, changing the observer's viewpoint, and actions such as grabbing, picking, or hovering. Some of these aspects of action control are disclosed below. User interface functionality in a hybrid BCI system can be analogized to an operating system that creates and maintains an environment that implements pointing and action control functionality in addition to other functions such as providing menus of selections, navigation controls, etc.

[0011]

[0029] In some embodiments of the hybrid BCI system described herein, the pointing control functions and methods for identifying the user's attention may include an eye-tracking device. In some embodiments, the action control functions and methods for identifying the user's intent may include any suitable form of monitoring neural signals in the brain. Such forms may include brain imaging, for example, by electrical or optical methods. For example, in some embodiments, the hybrid BCI system may use electrodes that record neural signals of brain activity that are routed through amplifiers and processors that convert the user's brain signals into BCI commands. In some embodiments, the hybrid BCI system may implement sophisticated user interfaces that implement machine control based on brain activity. As described in more detail below, specific adaptations to one or more of these features may be implemented to achieve high speed and high accuracy of human interaction with the hybrid BCI system.

[0012]

[0030] FIG. 1 is a schematic diagram of a hybrid brain-computer interface system 100 according to one embodiment. One example of this hybrid brain-computer interface system 100 (also referred to herein as a "hybrid BCI system" or "BCI system" or "system") is an integrated oculomotor hybrid BCI system including a video-based eye tracker 102 and a neuro-recording headset 104 for recording one or more control signals of a user's brain. The control signals can be any form of neural activity recorded by any suitable technique, such as electroencephalography (EEG), electrocorticography (ECoG), or magnetoencephalography (MEG). Examples of forms of neural activity include event-related potentials (ERPs), motor imagery, visual evoked potentials (VEPs), brain state-dependent signals, etc. This example of the hybrid BCI system 100 also includes a brain-computer interface device 110, and optionally an audiovisual display 106.

[0013]

[0031] In some embodiments of the hybrid BCI system 100, the user's attention can be determined from a video-based eye tracker 102 (i.e., pointing control functionality) and a neural recording headset 104 can be used to collect neural brain activity data (i.e., action control functionality). The collected neural and eye-tracking data can be communicated to a brain-computer interface device 110 that processes the signals as an ensemble along with data regarding which stimuli were presented. The combined information enables the brain-computer interface device 110 to detect relevant signal features based on statistical models to predict the user's intent. This predicted intent can be communicated to the user and acted upon, for example, by a user interface presented by the display 106.

[0014] Video-based eye tracking - pointing control function

[0032] In some embodiments, the video-based eye tracker 102 can be used to determine where in the user's visual field the user is looking by quickly tracing the user's eye movements in two-dimensional or three-dimensional space. Provided that the user has control over his or her eye movements, the video-based eye tracker 102 can be used to determine which subspace in the user's visual field the user's eyes are "pointing" at. In other words, the video-based eye tracker 102 can use the trajectory of the user's eye movements as a pointing control function, revealing important information about the subject's intentions and behavior. In some embodiments, aspects of where the user is looking in visual space, which stimuli they are looking at, or which stimuli they are responding to, can be effectively used in the BCI system 100.

[0015]

[0033] In some embodiments, the video-based eye tracker 102 relies on tracking the user's pupil and the surface corneal reflection (CR) of the illumination source by using a head-mounted eye-tracking video camera to image the user's eyes. The position difference between these two features can be used to determine the viewer's gaze direction. Some examples of head-mounted eye-tracking devices that can be used as the video-based eye tracker 102 are shown in Figures 2A and 2B and are available from SenseMotoric Instruments, Tobii Eye Tracking, and Pupil-labs, among other commercial vendors. For example, the illustration in Figure 2B shows a front view and a back view of a video-based eye tracker with left and right eye cameras shown. These cameras can be connected to each other by a wired or wireless connection. The video-based eye tracker can also include an additional scene camera that captures the field of view of use.

[0016]

[0034] In some embodiments, the video-based eye tracker 102 simultaneously records an image of the pupil and the corneal reflection of the illumination source. The video-based eye tracker 102 can use a near-infrared (IR) illumination source, which is optimally reflected by the iris and is invisible to humans, and therefore does not impede or distract the user. Strong IR reflectance results in high-contrast images that are particularly beneficial for pupil detection. In some embodiments, the video-based eye tracker 102 can use a collimated long-distance light source, where parallel light rays are emitted from a distant illumination source and collimated by optics. In some embodiments, the video-based eye tracker 102 can use an uncollimated near-light source to illuminate the eye, whereby the illumination source is mounted a finite distance from the eye (typically 50 mm or less) and there are no optics between the light source and the eye to collimate the light rays.

[0017]

[0035] As described herein, the video-based eye tracker 102 utilizes light reflected from the eye, which is detected by a video camera or any other suitable optical sensor specifically designed for this application. The detected light is then analyzed to extract eye rotation from changes in reflection. In some embodiments, the video-based eye tracker 102 can use the corneal reflection (i.e., the first Purkinje image) and the center of the pupil as features to track over time. In some embodiments, the video-based eye tracker 102 can use the reflection from the front of the cornea (i.e., the first Purkinje image) and the reflection from the back of the lens (i.e., the fourth Purkinje image) as features to track eye movements in a more sensitive manner. In some embodiments, the video-based eye tracker 102 can use a more sensitive tracking method by imaging features within the eye, such as retinal blood vessels, and following the movement of these features as the eye rotates.

[0018]

[0036] In some embodiments, the video-based eye tracker 102 may include an integrated display, as described below. The video-based eye tracker 102 integrated into the display 106 may be a system configured to view a virtual reality space. In some embodiments, the video-based eye tracker 102 integrated into the display 106 may be configured to view an augmented reality space.

[0019] Neural recording of brain signals - action control functions

[0037] The cornerstone of any BCI system are brain signals that can be used as control signals to make the brain signals action-controlling functions. Electrophysiological activity, one of the main types of brain activity monitored, is generated by electrochemical transmitters that exchange information between neurons. Neurons generate ionic currents that flow within and across neuronal populations. The wide variety of current pathways can be simplified as dipoles that carry current from source to sink via dendritic trunks. These intracellular currents are known as primary currents. Conservation of charge dictates that the primary currents are enclosed by extracellular currents known as secondary currents.

[0020]

[0038] As described herein, the neuro-recording headset 104 gathers information about the user's intent through a recording phase that uses brain signals to measure brain activity and converts the information into manageable electrical signals that can be translated into commands. In some embodiments, the neuro-recording headset 104 can be configured to record electrophysiological activity through electroencephalography (EEG), which has high time resolution, low setup and maintenance costs, is highly portable, and is non-invasive to the user. The neuro-recording headset 104 includes a set of electrodes with sensors that obtain electroencephalography signals from different brain regions. These sensors measure electrical neural activity caused by current flow during synaptic excitation of dendrites in neurons and are highly sensitive to the effects of secondary currents. The neural signals are recorded by electrodes that are appropriately positioned within the neuro-recording headset 104 and placed on the user's scalp. An example of an illustration of an embodiment of the neuro-recording headset 104 is shown in FIG. 3A, and similar neuro-recording headsets are available from Biosemi, Wearable Sensing, and G.Tec, among other commercial vendors.

[0021]

[0039] In some embodiments, the neuro-recording headset 104 may include electrodes, amplifiers, A / D converters, and recording devices. In some embodiments, electrodes on the neuro-recording headset 104 may acquire signals from the scalp, and an amplifier may boost the analog signal to increase the amplitude of the neuro-recording signal. In some embodiments, one or more electrodes disposed in the neuro-recording headset 104 may be directly connected to an amplifier, an A / D converter, and one or more recording devices for storing the signal from each electrode. In some embodiments, these components may be housed within the neuro-recording headset 104. In some embodiments, only immediate signal amplification may be performed within the neuro-recording headset 104, and other processes such as A / D conversion and recording may be performed after transferring the signal to the BC integrating device 110.

[0022]

[0040] In some embodiments, the electrodes on the neuro-recording headset 104 may be configured to be placed on the scalp according to the International 10-20 system, which is commonly adopted and standardized by the American Electroencephalographic Society. FIG. 3B shows an example of an illustration of the placement of electrodes according to this system. The 10-20 system uses two reference points on the head to define the location of the electrodes. One of these reference points is the nasion point, which is located at the top of the nose at the same level as the eyes. The other reference point is the external occipital protuberance point, which is found on the bony mass at the base of the skull. A transverse plane and a median plane separate the skull from these two points. The electrode locations are determined by marking these planes at 10% and 20% intervals. The letters at each location correspond to a particular brain region, such as A for earlobe, C for central region, Pg for nasopharynx, P for parietal, F for frontal, Fp for frontal pole, and O for occipital. FIG. 3C shows an example of a set of brain responses after stimulation, commonly referred to as stimulus-driven "neural activity", recorded using the neurorecording headset 104. The neural signal is measured over time as a potential difference between an active electrode (also called a signal electrode) and a reference electrode. In some embodiments, a third electrode known as a ground electrode can be used to measure the differential voltage between the active and reference electrodes. In some embodiments, the neurorecording headset 104 can include one or more active electrodes, one or more reference electrodes, and one ground electrode. In some embodiments, the neurorecording headset 104 can include as few as seven active electrodes. In some embodiments, the neurorecording headset can include up to 128 or 256 active electrodes. The electrodes can be made of silver chloride (AgCl) or any other suitable material. The electrodes can be configured such that the electrode-scalp contact impedance can be appropriately adjusted to record accurate signals. The electrode-tissue interface can be resistive as well as capacitive, thus acting as a low-pass filter. The impedance depends on several factors such as the interface layer, the electrode surface area, and the temperature.

[0023]

[0041] Neural signals recorded non-invasively across a user's scalp must cross the scalp, skull, and many other layers, which can weaken the neural signals and make them difficult to acquire. Neural signals can also be affected by background noise generated within the brain or outside the scalp, which can affect the ability to extract meaningful information from the recorded signals. An embodiment of the system 100, including the neural recording headset 104, can incorporate several adaptations to improve the acquisition of neural signals. For example, gel (i.e., conductive gel) can be used to create a conductive path between the skin and each electrode to reduce impedance. In some embodiments, the neural recording headset 104 can include "dry" electrodes that do not require the use of gel, and such dry electrodes can be made of other materials such as titanium and stainless steel. In some embodiments, the neural recording headset 104 can include dry active electrodes with preamplification circuitry to accommodate very high electrode / skin interface impedances. In some embodiments, the neural recording headset 104 can include dry passive electrodes that do not have any active circuitry but can be linked to a neural recording system configured with a very high input impedance.

[0024]

[0042] The amplitude of electrical biosignals is typically on the order of microvolts. As a result, the signals are highly sensitive to electronic noise. In some embodiments, the BCI system 100 can be designed to reduce the effects of noise through adaptations such as electromagnetic interference shielding or reduction of common mode signals, among others.

[0025] Display and presentation of user interface

[0043] As described herein, the user interface in the hybrid BCI system 100 serves as a communication link between the user (e.g., the user's brain, eyes, etc.) and the BC interface device 110, allowing the user to focus on and indicate particular stimuli via pointing control functions and to select or deselect particular stimuli using action control functions. A strategically designed user interface includes a process of presenting stimuli to the user by any modality, most commonly by visual mode. For example, the user interface may be configured to present visual, auditory, tactile, or vestibular stimuli to the user. In some embodiments, the user interface presenting visual stimuli may be rendered on a display, such as the display 106 shown in FIG. 1.

[0026]

[0044] In some embodiments, the display 106 may be a separate, standalone audiovisual display unit that may be connected and data-communicated with the rest of the hybrid BCI system 100. That is, a standalone display (e.g., liquid crystal display) with an audio system (e.g., speakers or headphones) may bidirectionally communicate with one or more of the other components of the hybrid BCI system 100, such as the BC interface device 110, the video-based eye tracker 102, and the neural recording headset 104. In some embodiments, the display 106 may be integrated with the video-based eye tracker 102 such that it is part of the eyeglasses area. The integrated video-based eye tracker 102 and display 106 may be configured to view the virtual reality space in the form of a user interface presented on the display 106. In some embodiments, the integrated video-based eye tracker 102 and display 106 may be configured such that the display 106 is on a semi-transparent eyeglasses area, allowing the user to view the augmented reality space. That is, the user may view the real world through the semi-transparent eyeglasses area, which is also the integrated display 106 that presents the user with a user interface with which the user can interact.

[0027] Brain-computer interface device

[0045] In some embodiments, the BC interface device 110 may be configured to achieve three main functions. First, the BC interface device 110 may be configured to generate a strategically designed user interface. For example, the strategically designed user interface may be for a training session or a test session. In some embodiments, the user interface may be designed as a virtual reality interface and / or an augmented reality interface. In some embodiments, the user interface may be tailored for specific needs, such as a particular user's history, reaction time, user preferences, etc. Second, in addition to designing and generating the user interface, the BC interface device 110 may be configured to receive pointing control signals (e.g., from the video-based eye tracker 102) and action control signals (e.g., from the neural recording headset 104) and process the signals as an ensemble to determine the user's intent. Finally, the BC interface device 110 may be configured to implement pointing control and action control functions by (1) detecting meaningful features from the neural signals and (2) implementing indicated changes in the stimuli in response to the user's intent. In some embodiments, the BC interface device 110 can also be connected to other peripheral devices that may be part of the hybrid BCI system 100, such as peripheral sensors and actuators that function in modalities other than visual. Such peripheral sensors can include audio microphones, tactile sensors, accelerometers, angle meters, etc., and peripheral actuators can include audio speakers, tactile stimulation providers, etc.

[0028]

[0046] In some embodiments, the BC interface device 110 may include an input / output unit 140 for sending and receiving signals to and from the video-based eye tracker 102, the neuro-recording headset 104, and the optional audiovisual display 106 via one or more data communication ports. The transfer of signals may be by wired connection or wirelessly by any suitable communication channel such as Bluetooth. The functions of the input / output unit 140 may include several procedures such as signal acquisition, signal pre-processing, and / or signal enhancement. The acquired and / or pre-processed signals may be sent to a processor 120 in the BC interface device 110. In some embodiments, the processor 120 and its sub-elements (not shown) may be configured to process input data, transfer data to and from the memory 160, and perform functions to build and maintain a user interface that can be rendered on the display 106 or on a display integrated in the video-based eye tracker 102. In some embodiments, the processor 120 and its sub-elements may be configured to perform functions necessary to enable user-specific interpretation of the brain signals and package output signals to the input / output unit 140 to be relayed to an external device. Other functions of the processor 120 and its sub-components may include several procedures such as feature extraction, classification, and operation of a control interface.

[0029] Control Signals in Hybrid BCI Systems

[0047] As described herein, the purpose of the hybrid BCI system 100 is to interpret the user's intention by monitoring brain activity and finding control signals that can be used as action control functions. The formation of the user's intention can be represented as a cognitive task. The integrated hybrid BCI system 100 shown in FIG. 1 can use one of several signature brain signals that are simultaneously induced by or related to the cognitive task performed by the user. Some of these brain signals can be decoded in a way that may allow the person to modulate them at will. Using these signals that are considered as control signals can enable the hybrid BCI system 100 to interpret the user's intention. Thus, the hybrid BCI system 100 can record and use control signals from the brain and use them as action control functions to control the interface between the brain and the computer. Any suitable neural activity can be the control signal used to implement the action control function. Some examples of time-related neural activity include event-related potentials (ERPs), evoked potentials (EPs, e.g., visual evoked potentials (VEPs), auditory evoked potentials, sensory evoked potentials, motor evoked potentials), motor imagery signals, slow cortical potentials, brain state-dependent signals, and other yet to be discovered signature action potentials that underlie various cognitive or sensorimotor tasks.

[0030]

[0048] As an example of a form of neural activity, an event-related potential or ERP can be a signature neural activity related to a time-correlated event or stimulus presentation. ERPs can have unique shapes and characteristics that aid in their detection and identification (such as a P300 signal known to peak approximately 300 ms from a triggering stimulus). ERPs can also vary in magnitude and shape across different brain regions, and how ERPs map between brain regions can indicate specific brain functions and / or user intent. The neural activity data obtained from the neural recording headset can be analyzed for specific ERP signals, and once properly detected and classified, the BCI device 110 can implement any specific action associated with the detected ERP on a desired portion of the user interface.

[0031]

[0049] Another example of the control signal may be in the form of motor imagery signals, which are neural activity signals associated with the user's experience of mental cognitive processes underlying voluntary or involuntary movements. That is, motor imagery signals are brain signals that may be recorded from various brain regions and analyzed by the BCI system 100 while the user is imagining and / or performing an action. In embodiments where the user is expected to perform physical movements or gestures, the BCI system may also use information collected by a set of peripheral sensors coupled to the BCI device 110, such as a goniometer, torsionometer, accelerometer, etc., to help recognize the gestures with high definition during a training session. In some other embodiments, the user may not be required to perform physical movements (e.g., after a training session in which the BCI system is allowed to learn the association between the user's set of movements and the set of motor imagery signals induced by the movements), since imagining a set of movements that induce motor imagery signals may be sufficient.

[0032]

[0050] Further examples of control signals may be in the form of brain state signals or brain state dependent signals. For example, a BCI system may be used to detect patterns of EEG signals or scalp topography associated with a particular attentional or perceptual state of the user's brain. The neural activity usable by the BCI system may also be in the frequency domain. Some examples include specific signal frequency bands such as sensorimotor rhythms, event-related spectral perturbations (ERSPs), delta, theta, alpha, beta, gamma, or mu rhythms, among others. An example P300 evoked potential is described below.

[0033] P300 signal

[0051] The P300 evoked potential is a positive peak in the neural signal due to low frequency auditory, visual, or somatosensory stimuli. The P300 signal is an "intrinsic" brain signal that depends on the processing of the stimulus context and attention and arousal levels. Without intending to be bound by any particular theory, the P300 is widely believed to be a neural signature of the mechanisms required to modify the mental model of the environment in order to perform an appropriate response. Thus, the P300 signal is an important signature of cognitive processes such as attention and working memory. Advantageously, the use of a P300-based BCI does not require training.

[0034]

[0052] The intrinsic P300 response is elicited approximately 300ms (up to 900ms) after responding to a statistically unusual stimulus or what is known as an oddball stimulus among several frequent stimuli. The P300 signal can be located across several brain regions and can indicate several brain processes. Recordings of brain activity associated with subjects performing controlled tasks with visual targets have shown the occurrence of these consistent signature signals associated with specific stimuli. Target-related responses can occur in the parietal cortex and cingulate region of the brain, while novelty-related activation may occur primarily in the inferior parietal and prefrontal cortices. Stimulus modality-specific contributions arise from the inferior and superior parietal cortices for the visual modality and the superior temporal cortex for the auditory modality.

[0035]

[0053] This large (>10 μV) positive wave can occur when participants are uncertain about the nature of the upcoming stimulus. The P300 is commonly associated with "oddball" paradigms, in which participants must detect an occasional relevant ("target") stimulus among a series of frequent unrelated "non-target" or "standard" stimuli. The hybrid BCI system 100 can use any neural activity, such as the P300 signal, to reveal the relevance of a particular stimulus that distinguishes it from others, and to reveal the user's intent when attending to that stimulus.

[0036]

[0054] An example of a P300 signal (black line) evoked by a specific target stimulus compared to the absence of a characteristic signal (magenta) when another non-target stimulus was presented, recorded using an example neural recording headset 104 of an example system 100, is shown in FIG. 3C. The oddball paradigm presented by the example user interface of the system 100 can reliably produce a P300 response with a parietocentral scalp distribution to the target compared to the standard stimulus, regardless of the stimulus modality (visual, auditory, somatosensory) or response modality (button press, count). P300 responses can also be observed when a regular series of stimuli is interrupted by a stimulus absence, highlighting the endogenous nature of this component.

[0037]

[0055] The amplitude of the P300 may be larger with lower probability and higher discrimination of the target stimulus. The latency of the P300 signal relative to the onset of the target stimulus may increase when it is more difficult to distinguish the target stimulus from the standard stimulus, but may not increase when reaction times are longer for other reasons. Thus, the latency of the P300 is an attractive tool for distinguishing between response selection and execution and mental time measures of stimulus evaluation. The occurrence of the P300 signal and its association with a particular stimulus may be exploited by the system 100 as an action control function to select P300-eliciting stimuli in preference to non-eliciting stimuli.

[0038]

[0056] 4A shows a flow diagram of one example of how neural activity can be used as an action control function during operation of the hybrid BCI system 100, according to one embodiment. In some embodiments, the P300 signal can be the neural activity used to determine a user's intent by distinguishing (as a change or difference) a first stimulus from a second stimulus and act on the user's intent to select one stimulus over another. The presence or absence of a P300 signal in the recorded response can be used to make the selection.

[0039]

[0057] In some embodiments, the hybrid BCI system 100 can be used to manipulate the presentation of various stimuli on the display 106 via a user interface and to determine the user's intent based on recorded brain signals across several brain regions following the presentation of the stimuli. Neural activity can vary in shape and extent across various brain regions. For example, a centroparietal P300 or P3b is elicited by attended task-relevant deviant stimuli, deviant stimuli that are unrelated to the task but more salient than targets elicit a slightly earlier positive wave with centrofrontal scalp topography. This novel P300 or P3a component can be elicited by inserting a unique and highly salient ("novel") stimulus within a series of repeated standard and target events (e.g., a visual fractal within a series of simple geometric shapes). However, a similar P3a component can also be elicited by highly dissimilar repeated distractors in paradigms that use difficult target / standard discrimination.

[0040] VEP signal

[0058] Visual evoked potential (VEP) signals are brain activity modulations that occur in the visual cortex after receiving a visual stimulus. These modulations are relatively easy to detect due to the large amplitude of the VEP when the stimulus approaches the central visual field. In the hybrid BCI system 100 disclosed herein, the VEP can be used in combination with eye-tracking input to adopt an integrated approach to enable pointing control functions.

[0041]

[0059] VEPs can be classified as transient VEPs (TVEPs) and steady-state VEPs (SSVEPs). TVEPs occur when the frequency of visual stimuli is less than 6Hz, whereas SSVEPs occur in response to stimuli with higher frequencies. TVEPs can be evoked by any change in the visual field, such as a flashing light, a pattern that is turned on or off, or a pattern whose contrast is suddenly reversed. TVEPs change with the presented stimuli and are rarely used in BCI systems.

[0042]

[0060] SSVEPs, more commonly used in BCI systems, are evoked by visual stimuli that vary at frequencies above 6 Hz. When the stimulus is a flash, the SSVEP exhibits a sinusoidal waveform whose fundamental frequency is the same as the flash frequency of the stimulus. When the stimulus is a pattern, the SSVEP occurs at the reversal rate and its harmonics. In contrast to TVEPs, the constituent discrete frequency components of the SSVEP remain strictly constant in amplitude and phase over long periods of time. SSVEPs are less susceptible than TVEPs to artifacts generated by blinks and eye movements and to noise contamination of electromyograms.

[0043] Hybrid BCI Systems: An Integrated Approach

[0061] As described herein, video-based eye tracking can be used in combination with VEP or ERP-type neural activity or other types of neural activity to enable pointing control functions and to allow neural activity (e.g., ERPs, or sensory-motor signals, motor imagery signals, brain state signals, etc.) to be used as control signals for enabling action control functions. The BC interface device 110 can be configured to process the integrated signals as an ensemble. In some embodiments, the hybrid BCI system 100 can use SSVEP signals from the neurorecording headset 104 in addition to the eye tracking signals from the video-based eye tracker 102 to obtain an integrated hybrid approach to identifying the user's gaze point. That is, in some embodiments, the video-based eye tracker 102 can be used to detect any suitable type of eye movement information, such as saccadic, central fixation, and / or pupil dilation information, as well as central fixation information, by oculomotor data conveying eye muscle movements. Information regarding saccadic eye position can also be obtained indirectly from neural activity obtained from the neurorecording headset 104, such as ERPs induced by visual responses. For example, the hybrid BCI system 100 can be configured to correlate the occurrence of ERPs with the presentation of specific stimuli in time and space to form causal connections. Thus, oculomotor data from the video-based eye tracker 102 can be combined with data from visually evoked neural activity and the strategic presentation of stimuli within a user interface provided by a display.

[0044]

[0062] The integrated hybrid approach to tracking eye movements allows the user to quickly select a target by voluntary movement of his / her gaze direction as a pointing control function, while the signal from the video-based eye tracker 102 and the visually evoked neural activity complement each other in the information provided to determine gaze. The user visually fixates attention on a target, and the BC interface device 110 can identify the target by analyzing the features of the neural activity and the video-based eye tracker signal. In particular, in the hybrid BCI system 100, the signals from the video-based eye tracker 102 and the neural recording headset 104 are analyzed as an ensemble by the BC interface device 110 with appropriate weighting for each signal source.

[0045]

[0063] For example, the hybrid BCI system 100 can utilize visually evoked neural activity evoked by specific stimuli that may be desired by the user, as recorded by the neuro-recording headset 104. The user interface presented on a visual display (e.g., display 106) can include a matrix of symbols or commands in a specific arrangement. For example, the symbols can be arranged in rows and columns, and the rows or columns of this matrix can be randomly flashed while the neural signals are monitored. When the user gazes at a desired symbol, neural activity can be evoked only when the row or column containing the desired symbol flashes. The hybrid BCI system 100 can use data from the video-based eye tracker 102 (i.e., eye movement data) and data from the neuro-recording headset 104 (i.e., stimulus presentation data) to triangulate and quickly identify and determine the target symbol desired by the user.

[0046]

[0064] The hybrid BCI system 100 can use ensemble processing techniques to determine the desired target stimuli. The ensemble processing techniques can simultaneously process signals from one or more available sources, including neural activity and brain signals related to eye position, stimulus presentation, and visual response, by appropriately weighting them. Once the desired target symbol is identified and determined, the hybrid BCI system 100 can use the recorded neural activity (e.g., P300 signal) to implement an action control function.

[0047]

[0065] For example, in some other embodiments of the hybrid BCI system 100, operating under one or more modes, the action control function may be implemented to select stimuli based solely on the stimulus presentation at the time the neural activity is elicited, regardless of the location of the eye position at that time. In this mode, signal weighting may be skewed to favor the stimulus presentation information during ensemble processing.

[0048]

[0066] Alternatively, in some embodiments, the action control function may be implemented in one or more modes, with the hybrid BCI system selecting any feature that the user may be gazing at (taking into account appropriate user reaction times and reaction latencies) at the time neural activity is detected. Thus, any symbol or stimulus that the user's gaze passes over when neural activity is detected is selected, regardless of which stimulus may have elicited the neural activity. This function may be implemented by varying the weighting to favor eye position signals to determine which stimulus to select during ensemble processing. Embodiments of the hybrid BCI system 100 may include modes of operation capable of both scenarios presented above, as well as other intermediate modes that may also operate between the examples of those modes by appropriately varying the weighting given to the various signals used, such as eye position signals, stimulus presentation information (including stimulus characteristics, spatiotemporal aspects of presentation, order of presentation, etc.), and recorded neural activity. The hybrid BCI system 100 may also use other information sources, including a biological model of the user's eye, likelihood estimates based on historical information or cumulative information (collected in real time) of specific neural activity generated by a particular stimulus.

[0049]

[0067] In some embodiments, the placed symbols can be flashed several times in different combinations to sample the user's selection multiple times. The signals from each combination can be averaged to improve the accuracy of detecting the correct symbol. The visual symbols can be accompanied by other modalities of stimulation, such as auditory or tactile stimulation.

[0050]

[0068] In general, repetition of trials can be tedious in BCI systems, slowing down the process of selection using neural activity signals such as P300. However, reducing repetitions can run the risk of reducing the accuracy of selection. In the hybrid BCI system 100 described herein, the accuracy of neural activity-based selection can be improved while keeping the number of repetition trials low by using sophisticated classifiers as described below. By optimizing the characteristics of the stimulus presentation (e.g., the display of visual symbols) and by selecting and modifying appropriate characteristics of the stimuli (e.g., the size or color of the symbols, their placement, etc.), the accuracy of detection of visually evoked neural activity and the overall user experience can also be improved. The experience can also be improved by configuring the user interface to use specific feedback from the user to predict behavior.

[0051]

[0069] One advantage of using the video-based eye tracker 102 in addition to neural activity from the neural recording headset 104 is that eye movements can be analyzed very quickly while maintaining robust signal acquisition. By implementing sophisticated analysis routines, the user's gaze and attention positions can be processed in real time. Furthermore, the video-based eye tracker 102 may not be affected by the same noise sources as the neural activity recorded by the neural recording headset 104. Processing eye movements using both neural activity as well as the video-based eye tracker signal in the integrated approach described herein can provide a more robust signal suitable for desirable high speed and high accuracy interfaces. In this configuration, one channel can compensate for the weaknesses of the other channel. Furthermore, the approach of processing both data sets as an ensemble allows for appropriate weighting of the individual signals according to other parameters such as the user's history and the specific details of the interface being navigated.

[0052]

[0070] In some embodiments, the oculomotor response can be used to change the order of stimulus presentation. Visually evoked neural activity requires repetitive blinking of stimuli while having high accuracy of eye movement detection. Such blinking can have a jarring effect on the user experience. To avoid this, the hybrid BCI system 100 can, for example, estimate the quadrant of the user's attention from the oculomotor information and reduce or limit the number of visual stimuli that need to be blinked in that quadrant of the display. Thus, the blinking of a subsection of the display can be linked to a saccade and the blinking can be triggered only after a saccade to that location.

[0053] User interaction with a hybrid BCI system

[0071] 5A-5D show a user's interaction with the hybrid BCI system 100, according to one embodiment. In this example, the hybrid BCI system 100 is used to spell a word in a two-step process, where in FIG. 5A (a), the display 106 presents several subgroups of characters (e.g., letters, numbers, and symbols commonly found on a keyboard). As shown in FIG. 5A (c), the user wears a video-based eye tracker 102 and a neural recording headset 104. When the user focuses his or her gaze on a subgroup containing a desired character (e.g., the subgroup shown by the highlighted circle in FIG. 5A (a)), the user interface presented in the display 106 changes to the user interface shown in FIG. 5A (b), where the focused subgroup is magnified. The user can perform the action of selecting a character by focusing his or her gaze on a particular character desired within that subgroup. The action control function is then implemented by using the recorded neural activity to select the character to be used in forming a word or sentence.

[0054]

[0072] The pointing control function described above with respect to Fig. 5A(a) and (b) is implemented using a video-based eye tracker 102 shown in Fig. 5A(c). The video-based eye tracker 102 is configured to detect where the user focuses his / her gaze and output a signal, for example, as shown in Fig. 5B. The action control function (i.e., activation) is implemented using a neuro-recording headset 104 shown in Fig. 5A(c). The neuro-recording headset 104 is configured to record neural signals from the user's brain and output a signal, for example, as shown in Fig. 5C. A processor (not shown) can then extract meaningful features from the eye-tracking signals (Fig. 5B) and the neural signals (Fig. 5C) as an ensemble and analyze them by classifying the signals in an unsupervised and / or semi-supervised manner or based on a priori models built by rigorous training with each user. The analyzed data can be used to predict user behavior, such as the user's attention location and / or the selection or activation of symbols that are predicted to be attended to.

[0055]

[0073] FIG. 5D shows an example of a time sequence of events following video-based eye tracking. In some embodiments, as shown in the example of FIG. 5D, saccadic eye movements can be detected by the hybrid BCI system even before the user's conscious perception of self-motion. That is, eye movements can be detected online when they occur immediately after or even before the user's conscious decision. For example, a user can focus on a group of letters shown in FIG. 5A(a) and begin to make an eye movement to another group. Even before the user consciously recognizes their own movement, the hybrid BCI system disclosed herein can detect and use eye movements to adapt the user interface as appropriate, which in this case is to expand the next group of letters.

[0056] An example of a hybrid BCI system

[0074] 6 illustrates a hybrid BCI system 600 according to one embodiment. In some embodiments, the BCI system 600 may be similar in structure and / or function to corresponding portions of the hybrid BCI system 100 described above with respect to FIG. 1. For example, the BCI system 600 includes a video-based eye tracker 606, a neuro-recording headset 604, an optional display 606, and a brain-computer interface device 610, which may be the same as or similar to the video-based eye tracker 106, the neuro-recording headset 104, the optional display 106, and the brain-computer interface device 110 of the hybrid BCI system 100. As such, such similar portions and / or aspects will not be described in further detail herein.

[0057]

[0075] In some embodiments, the processor 620 of the brain-computer interface device 610 may include a synchronization event logger 622 that receives the eye tracking and neural signal data via the input / output unit 640 and timestams the signal data to be synchronized with each other. In some embodiments, the synchronization event logger 622 may be configured to perform high speed eye movement classification.

[0058]

[0076] Gaze signals can be decomposed into saccadic (short duration) and central fixation (long duration) periods. These different components of gaze can be better utilized if the time delays inherent in current standard threshold-based saccade detection algorithms can be overcome. In addition, video-based eye trackers can be associated with non-physiological artifacts related to blinks and camera movement relative to the eyes, among others. To overcome these limitations, among others, in some embodiments, the synchronization event logger 622 can incorporate a probabilistic filter system with custom machine learning algorithms to enable high-speed eye movement detection while minimizing the sampling rate requirements of the eye tracker camera. Specifically, the synchronization event logger 622 can implement a zero-latency filter combined with a machine learning classifier that does not require explicit training and can detect saccades in real time within 2 microseconds of movement onset.

[0059]

[0077] Typically, any smoothing function over the time series (e.g., Butterworth filter, moving average, filter) implemented by currently available BCIs introduces a delay in the smoothed value compared to the actual measurement, preventing a fast real-time gaze classification system. In some embodiments, the brain-computer interface device 610 described herein can use a nonlinear generative model to estimate gaze as a dynamical system, along with a state and measurement model based on gaze kinematics and technical parameters of the video-based eye tracker. In some embodiments, a dual estimation based on a dual Kalman filter can be used to estimate the hidden state of noisy gaze data and approximate the underlying system model. Such an implementation can enable filtering of gaze data without introducing time delays. In addition, when the camera sampling rate is reduced or transient artifacts occur, a theoretical gaze kinematic model can be used to estimate the missing gaze data. Together, this results in a machine learning classification system that is automatically trained without the user's explicit knowledge and detects eye movements in real time during the movement, rather than after the movement.

[0060]

[0078] In some embodiments, the synchronization event logger 622 can be configured to perform ensemble estimation of attention using different physiological signals as described in the example of the integrated approach. In some embodiments, the synchronization event logger 622 can be configured to implement an analysis pipeline that uses (1) a Bayesian linear discriminant system, (2) state-of-the-art pre-processing, (3) spatial filtering, (4) a bagging ensemble classifier algorithm, and (5) a higher-order oracle algorithm that incorporates information from the classification algorithm with program routines during experimental work to improve the selection accuracy. The brain-computer interface device 610 can also be configured to ensure performance on consumer-level personal computers using parallelization and asynchronous processing of separable components in the pipeline.

[0061]

[0079] In some embodiments, the synchronization event logger 622 may be configured for real-time processing of the collected ensemble oculomotor data to detect user intent as described herein. For example, each presented visual stimulus may elicit a signature neural activity (e.g., P300 signal) that is captured by the neural recording headset 604. As described herein, the neural activity may serve as a control signal indicative of the user's attention to the stimulus and intent to act, and the probability of overt attention at a particular visual stimulus or tag may be estimated according to the neural activity classification score for each visual stimulus or tag. Depending on the high fidelity foveal receptive field in the retina, there may be spatial uncertainty based on the degrees in visual space centered on the gaze fixation as well as signal attenuation associated with the stimulus reflecting overt attention. Quantification of neural activity scores and signal attenuation can be used for real-time and accurate oculomotor neural recording-based user intent classification that solves the challenges of machine learning to combine different physiological signals as well as temporal drift in the latency of neural activity signals over time. Furthermore, such quantification and signal attenuation maximizes signal information so that individual visual stimulus and gaze position signals can update the probability of explicit attention for multiple objects.

[0062]

[0080] In some embodiments, the processor 620 may include a trainer 624 configured to present a training environment rendered by the user interface. The training environment may be configured to present a set of predefined controlled stimuli to the user and record the resulting eye movements and / or brain activity. The set of controlled stimuli and the evoked eye and brain activities corresponding to each of the controlled stimuli may be stored in memory 660 and used by the model builder 626 as training data to build a statistical model tailored to the individual user. The model builder 626 may use one or more statistical tools such as dimensionality reduction, feature extraction, machine learning tools, etc. to build a classifier, etc. The model may be built, tested, and cross-validated using the training data provided by the trainer 624 and then used with new data from that particular user to achieve high accuracy and speed of interaction with the user interface.

[0063]

[0081] In some embodiments, the processor 620 can include a predictor 628 that receives new data from a particular user, classifies the data based on the results from the model builder 626, and can generate predictions of the user's behavior using statistical tools such as maximum likelihood estimation, maximum a posteriori estimation, etc. In some embodiments, the processor 620 can also include an architecture developer 632 that receives eye-tracking and brain signal data as well as data from other sub-components of the processor 620 (e.g., the trainer 624, the model builder 626, and the predictor 628). The architecture developer 632 is not intended for real-time use, but can be for offline robust statistical analysis towards prototyping potential BCI algorithm detection architectures.

[0064] Use and implementation of hybrid BCI systems

[0082] FIG. 7 illustrates an example of a process 700 of the operation of an integrated hybrid brain-computer interface device, according to one embodiment. In some embodiments, the process 700 includes an initial step 701 of initiating acquisition and pre-processing of data for a particular user associated with a neural recording headset and eye tracker (and other peripheral sensors and / or actuators) at a certain point in time. This initiation and signal acquisition may be performed, for example, by a synchronization event logger 622. The process 700 also includes a step 703 of applying a training environment implemented by a trainer, such as trainer 624 described above. The presentation of the training environment may include a set of pre-programmed and controlled stimuli displayed in step 705, and the resulting eye movement and brain activity may be recorded in step 707. The presentation of the modified but controlled stimuli and the collection of corresponding eye movement and brain data may be repeated as indicated by step 719 for a series of stimuli. The data may be fed to a model builder (e.g., model builder 626), and the model may be updated with each new set of stimuli in the training set. The process 700 may then include presenting a new environment that includes one or more novel stimuli, but that may be related to one or more of the pre-programmed stimuli that the system was trained on. Using the statistical model constructed by the model builder 626, the predictor 628 may use any suitable statistical tool to generate a prediction of the user's behavior, such as determining a focus point at step 713, and infer the user's intent (i.e., to select or activate a particular displayed symbol) from the brain data at step 715. Based on this prediction, the process 700 includes a step 717 of implementing the user's intent within the user interface. For example, step 717 may include selecting a letter in a speller, or selecting a character in a game, or selecting an on function associated with a TV system that is operable within an augmented reality system.

[0065] Adaptation of Hybrid BCI Systems Operational Mode

[0083] The hybrid BCI system 100 and / or 600 and other embodiments described herein can be configured as an exogenous BCI system that uses brain activity induced by external stimuli, such as VEP, Auditory Evoked Potential (AEP), or P300 signals. An exogenous system does not require extensive training because its control signal from neural activity can be easily and quickly set. Notably, signal control can be realized using fewer neural recording channels and can achieve a high information transfer rate of up to 60 bits / min.

[0066]

[0084] In some embodiments, the hybrid BCI system 100, 600 may be adapted to operate as an endogenous BCI system based on self-regulation of brain rhythms and brain potentials without external stimuli. For example, by neurofeedback training, the user learns to generate specific brain patterns that can be decoded by the BCI, such as modulations in sensory-motor rhythms. One advantage of an endogenous BCI is that the user can freely manipulate the BCI and make or imagine voluntary movements that translate into the movement of a pointing control function, such as a cursor, to any point in two-dimensional space. Other features, such as hand gestures, can also be trained and used as cues to gain operational control of the user interface, reducing any constraints on the user being limited to the options presented.

[0067]

[0085] In some embodiments, the hybrid BCI system 100, 600 can be used in synchronous (cue-paced) or asynchronous (self-paced) modes. The synchronous mode of the BCI can be of a simpler design, focusing on a predefined time window of brain activity caused by a particular stimulus. In the synchronous mode, the hybrid BCI system 100, 600 can only analyze brain signals during a predefined time window and ignore any brain signals outside the window. Thus, the user is only allowed to send commands during a specific period determined by the hybrid BCI system 100, 600. One advantage of the synchronous mode is that the onset of mental activity is known in advance and is associated with a specific cue. The user may also make unintentional blinks and other eye or body movements that can cause artifacts. Brain signals outside the specified time window are not analyzed, thus avoiding the misleading effects of these artifacts.

[0068]

[0086] In some embodiments, the hybrid BCI system 100, 600 can operate in an asynchronous mode, which continuously analyzes brain signals regardless of when the user is acting. Although more complex and therefore computationally demanding, the asynchronous mode can provide a more natural mode of human-machine interaction since the user does not need to wait for any external cues. In addition, richer data is collected because the user interacts with the hybrid BCI system 100, 600 during the normal course of their actions.

[0069] Implementing fast signal detection and classification

[0087] The oculomotor nerve integrated hybrid BCI system described herein intelligently and flexibly exploits the strengths of each of the neural signals, including oculomotor signals, such as signals from the video-based eye tracker 102, and VEPs as selection and activation signals. One important component of the system is the fast classification of eye movements. As described above with respect to the operation of the synchronization event logger 622, the hybrid BCI system 600 can be configured to perform saccade detection in a delay-free manner. For example, the hybrid BCI system 600 can filter gaze data with zero delay using a nonlinear generative model that estimates gaze as a dynamical system, together with a state and measurement model based on gaze kinematics and technical parameters of the video-based eye tracker. A dual estimation based on a dual Kalman filter can be used to estimate the hidden state of the noisy gaze data and approximate the underlying system model. Thus, the filtering of gaze data can be achieved without introducing time delay. In addition, when the sampling rate of the camera is reduced or when transient artifacts occur, the theoretical gaze kinematic model can estimate the missing gaze data. The hybrid BCI system 600 may also include a machine learning classification system that is automatically trained without an explicit training session and allows for real-time detection of eye movements.

[0070]

[0088] In some embodiments, zero-latency saccade detection can be the ability to detect the onset of a saccade while an eye movement is in progress. For example, as shown in FIG. 5D, the detection latency (approximately 2 ms after saccade onset) can be shorter than the human perception of a visual scene change (approximately 50-150 ms). Thus, the hybrid BCI system 100 and / or 600 can use the saccade onset vector as a prediction of where the user will look before the user recognizes a change in visual information, and update the user environment accordingly. For example, during a program or game, the user may think of pausing the game. The saccade onset vector can be used as the user's eyes begin to turn toward the pause menu item to predict the intention to pause before the user changes gaze to focus on the pause menu item and select the pause menu. In some embodiments, the detection latency can be approximately 1.00 ms, 2.00 ms, 3.00 ms, 5.00 ms, 10.00 ms, or 15.00 ms (or any period in between).

[0071] Artifact Treatment

[0089] Artifacts are unwanted signals that contaminate brain activity and are mostly of non-brain origin. Artifacts can degrade the performance of BCI-based systems because the shape of neurological phenomena is affected. Artifacts can be physiological or non-physiological in origin, or technical artifacts.

[0072]

[0090] Physiological artifacts are usually due to muscle, eye, and cardiac activity. Large disturbances in the brain signal can result from electrical activity caused by muscle contractions that occur when the patient is speaking, chewing, or swallowing. Other artifacts can be caused by blinking and other eye movements. Blinking creates a generally high amplitude pattern on the brain signal, in contrast to eye movements that generate low frequency patterns. Artifacts related to cardiac activity give rise to rhythmic signals in the brain activity. Technical artifacts are mainly due to power line noise or changes in electrode impedance and can usually be avoided by proper filtering or shielding. Avoiding physiological artifacts is difficult, and rejection techniques propose to discard epochs that are corrupted by artifacts. Automatic rejection can fail if the amplitude of the artifacts is too small. In addition, the rejection method causes parts of the interaction where the user loses control of the device when discarding signals corrupted by artifacts. Instead of blocking samples, artifact removal techniques attempt to identify and remove artifacts while preserving the neurological phenomenon intact. Common methods for removing neural artifacts are linear filtering, linear combination and regression, BSS and PCA. Instead of removing artifacts from brain signal recordings, the system 100 and / or 600 can process the artifacts to provide a usable communication channel.

[0073]

[0091] In some embodiments, the hybrid BCI system 100, 600 can benefit from artifacts to benefit the user experience. For example, the system 100, 600 can be configured to detect artifact motions such as eye blinking, teeth clenching, and other heart-related changes. Detecting and classifying these artifacts allows the use of the original neural signals for noise-free applications. In some embodiments, the detected artifacts in the oculomotor neural data can also be exploited due to their known causal relevance in the user's behavior. For example, artifacts such as eye blinking, teeth clenching, and facial expressions can be used as control signals to implement action control functions as disclosed below.

[0074]

[0092] An embodiment of the hybrid BCI system 100, 600 operating in asynchronous mode can use an unsupervised classification system that incorporates a warm training set with a clustering model and an examination of Euclidean distance in an n-dimensional feature space. The artifact identification technique that can be performed by uses a sliding window of data in which features are calculated and classified based on a clustering model on an n-dimensional space. The strength of the artifacts described over time can be used to classify them by any suitable cluster-based technique, for example a classification method based on the Euclidean distance of the data points to each cluster centroid.

[0075]

[0093] Traditional BCI systems record data and extract and process the recorded signals at set time intervals to determine the user's intent and the user's state. In some embodiments, the hybrid BCI systems 100 and / or 600 described herein may be configured to process the recorded data with each input sample to determine the presence of an "event" and use the presence or absence of an "event" to trigger a programmatic chain reaction. This may be further analysis of the data to change the processing pipeline in anticipation of a particular behavior or modify the user's environment in some way. This process may be handled in a multi-threaded environment by separating the processing pipeline for each input sample (from any device source) and running the data through a classifier to determine the type of data.

[0076]

[0094] Events include eye blinks, event potentials, movement artifacts, excessive signal noise, and sudden drops in observed activity. Following the analysis phase, the interface can determine whether a classifiable event has occurred, and if so, can trigger a specific set of events. An example of an epoch of recorded activity is shown in FIG. 8A. For example, if an eye blink is observed, the trigger can instruct the system to ignore incoming eye data for the next 5 milliseconds. If an event potential is observed, the trigger can instruct the system to identify a correlation between the environmental change and the sudden presence of the event, and which particular change affected the user's brain activity. Examples of such events are highlighted by the green and red bars in FIG. 8A.

[0077]

[0095] Past approaches to BCI implementation utilize a fixed time delay (i.e., 50 ms) after an event. After the fixed time delay has elapsed, the user environment is changed and data for that period is analyzed. In contrast, the system 100 and / or 600 according to some embodiments may implement an approach that does not use a specific time delay or period, but instead analyzes the input data continuously. Once the system identifies and classifies a particular type of event (defined by the experimenter), the appropriate extracted data may be analyzed for additional information. If an event such as that shown in FIG. 8B is detected, the detection may trigger a series of processes. Thus, brain activity may act as a control signal, and detecting a particular feature may be an actuator for additional events, actions, and / or results. This approach may show similarities to performing an action such as clicking a mouse or pressing a button that directly triggers an immediate action, as opposed to a polling model, where a user may periodically update their mailing address and phone book to replace an old address with a new one.

[0078] User Interface Adaptation

[0096] As described herein, the user interface in the hybrid BCI system 100, 600 can be presented by an audiovisual display (e.g., display 106 in FIG. 1). The user interface allows the user to look at and indicate particular stimuli via pointing controls and to select or deselect particular stimuli using action controls. To do this, the user interface includes a selection set, which is a group of available options the user can make. Examples of selection sets can include letters / numbers on an alphanumeric matrix strategically placed on the control display, or directional arrows, or a group of clearly shaped icons or targets. The selection method describes how commands from the user are interpreted by the BCI, either directly or indirectly. Direct selection allows the user to directly select any item from the selection set, whereas indirect selection requires an intermediate step before the user can make a selection.

[0079]

[0097] FIG. 9A shows two examples of user interface layouts including symbols (letters or numbers). In one example, all of the symbols are presented and flashed together at once. In the second example, only one symbol (X) is shown in the flashing group while the other symbols are hidden in asterisk form. In some embodiments, this second layout can improve processing time while reducing perceptual issues that are known to reduce neural activity based BCI classification accuracy. This layout can also be used to indicate a change in user interface strategy, for example, this layout can indicate entering a "hold-release mode" as described further below.

[0080]

[0098] A user interface can be configured to indicate characteristics of the interface between the user and the BCI. The user interface can include three types of characteristics: (1) spatial, (2) sensory, and (3) activation / deactivation. Spatial characteristics include the size, number, and shape of icons or targets. Sensory characteristics include the stimuli and feedback provided to the user, whether auditory, visual, or somatosensory. Activation / deactivation includes the quality of the human / technology interaction.

[0081]

[0099] Interaction quality can be characterized by effort (how difficult it is to use the BCI), displacement (how much movement is required to respond), flexibility (how many ways the BCI can be used), durability (how reliable the BCI hardware is), maintainability (how easily the BCI can be repaired), and activation or deactivation methods (the ability to make / activate or stop / deactivate a selection, and how that selection is made). Activation and deactivation can have different functions, using only activation as an action control input can be thought of as a trigger or momentary switch. In this case, only activation has an effect, and the duration that activation is held does not change the outcome. Using both activation and deactivation allows for more complex control functions, where the control input can function as a button. For example, one can activate and hold one of the volume keys on a TV remote to keep the volume up. In this case, holding the selection causes a continuous change, whereas deselecting maintains the status quo.

[0082] Neural activity-based retention-release model

[0100] As described herein, the BCI system 100, 600 may incorporate neural activity-based BCI functionality as part of its user interface, where the initial activation and deactivation (hold-release) of targets in the neural activity-based BCI may be separately controlled. Such control may enable the hybrid BCI system 100, 600 to be used in applications requiring indirect selection or rapid changes between states. Furthermore, such control may enable the acceptance-cancellation of a selected target by holding the selection or switching attention to a release target.

[0083]

[0101] In potential real-world applications, targets on the BCI display can have various activation / deactivation characteristics. An example of a hybrid BCI user interface layout for a real-world application is shown in FIG. 8B. Some items can be hold-release compliant to allow fine adjustments, such as reclining the wheelchair shown in item B1 of the layout in FIG. 8B, changing the temperature (items A3 and B3), or turning up the volume on the TV (items C2 and D2). Items critical to safety, such as unlocking / locking the door (items D1 and E1), can require a hold-release acknowledge-cancel step that requires a short hold period before activation. The remaining items perform traditional discrete neural activity-based actions, such as turning on the lights or changing the TV channel (C1, A2, and B2). When a user selects a target with a hold-release response (for adjustment or acknowledgement), the screen changes to a hold-release mode (right panel of FIG. 9B), where only the previously selected target and the release target are in the active state, and the remaining targets on the BCI matrix cannot be selected. If the BCI correctly identifies the desired target, the user holds the selected target, and the BCI performs the action (reclining a wheelchair or changing the volume of a television) until the user wishes to stop the action or for a specified duration to confirm the selection (thereby preventing inadvertent activation of a safety-critical action). The hold-release feature thus extends the utility of neural activity-based BCIs to reflect the multiple modes of control available in existing assistive technologies.

[0084]

[0102] During the retention process, the only information requested by the BCI is when the user changes their selection (e.g., turn the volume up or down or stop reclining the wheelchair). The binary nature of the release decision allows the BCI to make the decision from very few blinks instead of after multiple blink sequences. For the BCI user, this translates to faster response times and continuous control than using traditional neural activity-based BCI approaches.

[0085] Mapping neural activity-based evoked stimuli

[0103] Conventional BCI systems (observing how the user's brain activity changes in response to changes made to the user's environment) generally attempt to elicit all possible behaviors the user may have before analyzing all the data acquired at once, and from there to reveal the user's intent. In some embodiments, the hybrid BCI system 100 and / or 600 described herein can analyze the user's brain activity after each change is made, and use information obtained from this analysis phase to, among other things, inform the next change made to the environment. For example, if the user has 10 possible actions, it can make 10 changes to the environment and observe which change elicits the most ideal response (as determined by the program). Alternatively, in the hybrid BCI system 100 and / or 600, if the same user has the same 10 options, it can use information obtained from the first change to inform the next change made, preemptively eliminating certain actions, thereby reducing the duration it takes to move from intent to action. Thus, complete knowledge of how the user's brain responds to each change made to the environment is not required, as each change informs more than just one potential user intent. This reduces the amount of information that must be recorded, and instead intelligently determines what information is needed to identify the user's intent, quickly retrieves that information, and proceeds to create the intended action.

[0086]

[0104] In some embodiments, the hybrid BCI system 100 and / or 600 may also implement error correction codes to perform feature extraction from acquired neural signals and improve information transmission rates using neural activity-based systems. In addition, the hybrid BCI system may utilize novel techniques to reduce the refractory effect, which is one of the main sources of noise in neural activity-based BCI systems, by superimposing targets on an appropriate background or by using alternative motion-based stimulus types.

[0087] Dynamic Stimulus Detection and Tagging

[0105] In some embodiments, the hybrid BCI system 100 and / or 600 can incorporate dynamic stimuli or tag existing dynamic stimuli in the augmented or virtual reality presented within its user interface. That is, the hybrid BCI system 100, 600 can identify moving stimuli and dynamically tag them to track their movement over time. An example of such a dynamic identification and tagging implementation is shown in FIG. 9.

[0088]

[0106] The hybrid BCI system 100, 600 can detect the presence of a moving object and place a selection tag on the moving object. The tag can flash on the tracking timeline and maintain co-location with the object. The user can use the BCI selection to trigger an action on that object. Alternatively, the hybrid BCI system 100, 600 can be primarily concerned with the beginning or end of tracking. A time window of neural data near the beginning or end of tracking can be examined to see if neural activity is elicited. This indicates that the new object may be of interest to the user and may trigger some action. In some embodiments, eye-tracking information can be used to see if a saccade is triggered by the initial movement of the object to further strengthen the assumption of the object as of interest before triggering an action. Such an action can be the dynamic tag placement described above in addition to any others.

[0089] Feature extraction and model building

[0107] Various thinking activities (cognitive tasks) result in different patterns of brain signals. In some embodiments, the hybrid BCI system 600 (and / or 100) can include a pattern recognition system that classifies each signal pattern into a class according to its features. The pattern recognition system extracts some features from the brain signals that reflect the similarity to a particular class as well as the difference from the remaining classes. The features can be measured or derived from the characteristics of the signals that contain the discriminatory information required to distinguish different types of signals. As explained above, the model builder 626 of can use a number of statistical tools to build a statistical model of the features extracted from the data corresponding to a particular user under a particular user interface coupled to a particular hardware peripheral, etc.

[0090]

[0108] Information of interest in brain signals is typically hidden in high-noise environments, and brain signals contain multiple sources of information that exist simultaneously. Potentially interesting signals may overlap in time and space with multiple signals resulting from different brain tasks. Therefore, in many cases, it is not sufficient to use simple methods such as band-pass filters to extract the desired band power.

[0091]

[0109] Brain signals can also be measured by multiple channels. Not all information provided by the measured channels is generally relevant to understanding the underlying phenomenon of interest. Dimension reduction techniques such as principal component analysis or independent component analysis can be applied to reduce the dimensionality of the original data and remove irrelevant and redundant information, which can reduce computational costs.

[0092]

[0110] Brain signals are also non-stationary in nature, so in some embodiments it may be desirable to obtain time information regarding when certain features occur. Some techniques may be used to split the signal into short segments, and parameters may be estimated from each segment. One or more data transformations may be performed, including methods such as Fast Fourier Transform (FFT), Wavelet Transform, or Adaptive Autoregressive Components, or techniques such as Stationary Subspace Analysis (SSA) may be performed to reveal non-stationary time changes in brain signals. SSA decomposes multivariate time series into stationary and non-stationary components. Multiple features may be extracted from several channels and from several time segments before being concatenated into a single feature vector.

[0093] Signal Classification and Prediction

[0111] The classification step in the hybrid BCI system 100, 600 aims to recognize the user's intent based on the feature vectors characterizing the brain activity provided by the feature step. The classification algorithm can be developed by offline sessions, online sessions, or both types of sessions. The offline sessions include the examination of a dataset, such as a BCI competition dataset collected from an adaptive or closed-loop system. Statistics of the data can be estimated from observations over the entire session, and long-term calculations can be performed. The results can be examined by an analyst to fine-tune the algorithm. In some embodiments of the hybrid BCI system, the data can be processed online in a causal manner that evaluates the real world. The analysis algorithms used can be tested in an environment where the user profile can change over time, for example in response to changes in the user's motivation or engagement.

[0094]

[0112] The classification algorithm can be calibrated by the user through supervised learning using a labeled data set. However, brain signals as well as eye movement signals can be inherently non-stationary with changes in their salient features over time. For example, the patterns observed in an experimental sample during a calibration session may differ from those recorded during an online session. Or even the user's gradual mental training or changes in concentration, attention, or motivation may affect the brain signals. In some embodiments, the hybrid BCI system 600 (and / or 100) described herein can use adaptive algorithms that can adapt to changes in salient features.

[0095]

[0113] In some embodiments, the hybrid BCI system 100, 600 can use semi-supervised learning algorithms to detect and classify signals. A small labeled data set can be used to initially train the semi-supervised learning classifier, which can then be updated using online test data. This approach can reduce the training time required by continuously updating the classifier during the online session.

[0096]

[0114] In some embodiments, the hybrid BCI system 100, 600 can use unsupervised learning or reinforcement learning or a combination of these methods to detect and classify brain and / or eye movement signals. Unsupervised methods can be used to find hidden structures in unlabeled data and classify them. Some embodiments of the hybrid BCI device can use unsupervised methods that rely on techniques for user and machine mutual adaptive learning or covariate shift adaptation. Some embodiments can use reinforcement learning methods based on distinguishing neural signals evoked when a subject realizes an incorrect decision. These feedback signals can be used as learning signals to prevent the error from being repeated in the future.

[0097] conclusion

[0115] In summary, systems and methods are described herein for use in implementing an integrated hybrid brain-computer interface that can be manipulated by a user in real-time. The disclosed system includes an eye movement tracking system for implementing pointing control functions, and a brain activity tracking system for implementing action control functions. Both functions are implemented through the presentation of a user interface that is strategically designed to enable fast and accurate manipulation. In addition, the disclosed system and method are configured to be hardware independent for implementing a real-time hybrid BCI on any suitable platform to mediate user manipulation of a virtual, augmented, or real environment. Figure 11 shows an illustration of the usage space of the hybrid BCI system and method of the present invention.

[0098]

[0116] While various embodiments have been described above, it should be understood that they have been presented by way of example only and not limitation. Although the above method shows certain events occurring in a particular order, the order of certain events may be modified. In addition, certain of the events may be performed simultaneously, where possible, by parallel processes, or may even be performed sequentially as described above.

[0099]

[0117] Although the schematic diagrams and / or embodiments described above show certain components disposed in particular orientations or locations, the arrangement of components may be modified. Although the embodiments have been individually illustrated and described, it will be understood that various changes in form and detail may be made. Any portion of the apparatus and / or methods described herein may be combined in any combination except mutually exclusive combinations. The embodiments described herein may include various combinations and / or subcombinations of the functions, components, and / or features of the various embodiments described.

Claims

1. a display configured to present an interactive environment to a user; an eye tracker configured to record eye movement signals generated by the user; a neurorecording device configured to record neural signals generated by the user; an interface device operably coupled to the display, the eye tracker, and the neurorecording device, comprising: Memory, and operatively coupled to the memory; receiving the eye movement signals from the eye tracker and the neural signals from the neural recording device; generating and presenting stimuli to the user via the interactive environment; processing the eye movement signal and the neural signal to determine a focus of the user based on at least one of the eye movement signal and the neural signal; processing the neural signals received from the neurorecording device to determine an intended action of the user; and performing the action intended by the user and updating the presentation of the interactive user interface; a processor configured to: an interface device including: Including, equipment.

2. the eye tracker is a video-based eye tracker; the neural signals include electroencephalography (EEG) signals, including visual evoked potentials, motor imagery signals, event-related potentials (ERPs), and brain-state dependent signals; The processor is further configured to integrate the eye movement signal and the EEG signal to determine the user's focus.

10. The device of claim 1.

3. the neural signals include electroencephalography (EEG) signals including at least one of visual evoked potentials, auditory evoked potentials, motor imagery signals, event-related potentials (ERPs), and brain-state dependent signals; 10. The device of claim 1.

4. The device of claim 1 , wherein the stimulus comprises at least one of a visual stimulus, an auditory stimulus, and a tactile stimulus.

5. 2. The device of claim 1, wherein the processor is further configured to define a set of portions of the display and to repeatedly present the stimulus within one of the set of portions of the display based on the determining the user's focus.

6. 10. The device of claim 1, wherein the processor is further configured to implement a zero-latency filter and a classifier to process the at least one of the eye movement signal and the neural signal to determine the user's focus.

7. 10. The device of claim 1, wherein the processor is further configured to implement a dual Kalman filter and a classifier to process the at least one of the eye movement signal and the neural signal to determine the user's focus.

8. The processor, generating a kinematic model of a set of eye movements of a simulated user; identifying a set of missing data points in the eye movement signal; calculating a set of replacement data points for replacing the set of missing data points in the eye movement signal based on the kinematic model; and incorporating the set of replacement data points to replace the set of missing data points and generate an updated eye movement signal. The apparatus of claim 1 , further configured to:

9. A non-transitory processor-readable medium storing code representing instructions for execution by a processor, the instructions comprising: generating an interactive user environment that can be manipulated by a user to perform a set of actions; defining a set of stimuli presentable to the user by the interactive user environment; presenting at least one stimulus from the set of stimuli to the user; receiving input related to the user's behavior from an eye tracker and a neuro-recording device; determining an intended action of the user based on the received input and the presented stimuli; and performing said action through said interactive user environment; 23. A non-transitory processor-readable medium comprising code for causing the processor to:

10. the code for receiving an input further comprising: extracting eye movement signals generated by the user from the input received from the eye tracker; extracting neural signals generated by the user from the input received from the neurorecording device, the neural signals including electroencephalography (EEG) signals, the EEG signals including visual evoked potentials, motor imagery signals, event-related potentials (ERPs), and brain state dependent signals; and Processing the eye movement signals and the EEG signals using an integrated approach to determine the user's focus.

10. The non-transitory processor-readable medium of claim 9, comprising code for causing the processor to:

11. the code for receiving an input further comprising: extracting, from the input received from the neurorecording device, electroencephalography (EEG) signals generated by the user, the EEG signals including at least one of visual evoked potentials, auditory evoked potentials, motor imagery signals, event-related potentials (ERPs), and brain state dependent signals; and Processing the EEG signals to determine a location of attention of the user.

10. The non-transitory processor-readable medium of claim 9, comprising code for causing the processor to:

12. the code for causing the processor to determine the action intended by the user, extracting information related to the presented stimuli from the set of stimuli; determining and applying a set of weights to each of the inputs received from the eye tracker, the inputs received from the neurorecording device, and the information related to the presented stimuli to generate a weighted representation of each of the inputs received from the eye tracker, the inputs received from the neurorecording device, and the information related to the presented stimuli; combining the weighted representations to form an ensemble representation; and Processing the ensemble representation to generate a prediction of the user's intended action.

10. The non-transitory processor-readable medium of claim 9, comprising code for causing the processor to:

13. the code for causing the processor to determine the action intended by the user, extracting eye movement signals from the input received from the eye tracker; Implementing a zero-latency filter and classifier to determine the user's focus based on the eye movement signals.

10. The non-transitory processor-readable medium of claim 9, comprising code for causing the processor to:

14. the code for receiving an input further comprising: extracting eye movement signals from the input received from the eye tracker; identifying a set of missing data points in the eye movement signal; generating a kinematic model of the simulated user's eye movements; estimating a set of replacement data points for replacing the set of missing data points in the eye movement signal based on the kinematic model; and incorporating the set of replacement data points to replace the set of missing data points and generate an updated eye movement signal.

10. The non-transitory processor-readable medium of claim 9, comprising code for causing the processor to:

15. the code for receiving an input further comprising: extracting eye movement signals from the input received from the eye tracker; identifying artifact data points within the eye movement signal; generating a kinematic model of the simulated user's eye movements; estimating a set of replacement data points for replacing the artifact data points in the eye movement signal based on the kinematic model; and incorporating the set of replacement data points to replace the artifact data points and generate an updated eye movement signal.

10. The non-transitory processor-readable medium of claim 9, comprising code for causing the processor to:

16. presenting the stimuli to a user via an interactive user interface; receiving input related to the user's behavior from an eye tracker and a neuro-recording device; receiving information related to the presented stimuli; generating a set of ensemble data comprising the information related to the inputs and the presented stimuli associated with the user's behavior; processing the set of ensemble data to determine an intended action of the user; and performing said action via said interactive user interface; A method comprising:

17. the presented stimuli include dynamic visual stimuli; detecting movement of the dynamic visual stimulus; tracking the movement of the dynamic visual stimulus; and Tagging the dynamic visual stimuli The method of claim 16 further comprising:

18. extracting eye movement signals from the input received from the eye tracker; extracting neural signals from the input received from the neurorecording device, the neural signals including electroencephalography (EEG) signals, the EEG signals including visual evoked potentials, motor imagery signals, event-related potentials (ERPs), and brain state dependent signals; and Integrating the eye movement signal and the EEG signal to determine the user's focus. The method of claim 16 further comprising:

19. extracting eye movement signals from the input received from the eye tracker; and Implementing a combined zero-latency filter and classifier based on the eye movement signals to determine the user's focus. The method of claim 16 further comprising:

20. extracting eye movement signals from the input received from the eye tracker; identifying a set of missing data points in the eye movement signal; generating a kinematic model for predicting eye movements of a simulated user; estimating a set of replacement data points for replacing the set of missing data points in the eye movement signal based on the predicted eye movements of the simulated user according to the kinematic model; and incorporating the set of replacement data points to replace the set of missing data points and generate an updated eye movement signal. The method of claim 16 further comprising:

21. said generating a set of ensemble data comprising: extracting eye movement signals from the input received from the eye tracker; extracting a neural signal from the input received from the neural recording device; determining and applying a set of weights to each of the eye movement signal, the neural signal, and the information related to the presented stimulus to generate a weighted representation of the eye movement signal, the neural signal, and the information related to the presented stimulus; and combining the weighted representations of the eye movement signals, the neural signals, and the information related to the stimulus to form the set of ensemble data.

17. The method of claim 16, comprising:

22. extracting eye movement signals from the input received from the eye tracker; extracting technical parameters of the eye tracker from the input received from the eye tracker; generating a non-linear generative kinematics model for predicting eye movements of the user based on the technical parameters of the eye tracker; calculating an updated eye movement signal based on the eye movement signal received from the eye tracker and the eye movement signal predicted by the nonlinear generative kinematic model; and determining a location of attention of the user associated with the updated eye movement signal; The method of claim 16 further comprising:

Citation Information

Patent Citations

  • Brain wave interface system, brain wave interface device, method and computer program

    WO2009093435A1