Brain-computer interfaces adapted for fast, accurate, and intuitive user interaction.

Abstract

To provide systems, devices, and methods for use in the implementation of a brain-computer interface that tracks brain activities, with or without additional sensors providing additional sources of information, while presenting and updating a user interface / user experience that is strategically designed for obtaining high speed and accuracy of human - machine interactions.SOLUTION: Embodiments described herein also relate to the implementation of a hardware agnostic brain-computer interface that uses neural signals to mediate user manipulation of machines and devices.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] Cross-reference of related applications

[0001] This application claims priority and interest in U.S. Provisional Patent Application No. 62 / 618,846, filed on 18 January 2018, entitled “Brain-Computer Interface with Adaptations for High-Speed, Accurate, and Intuitive User Interactions,” which is incorporated herein by reference in its entirety. [Background technology]

[0002] background

[0002] Embodiments described herein relate to systems, apparatus, and methods for use in implementing brain-computer interfaces that integrate real-time eye movement and / or head motion tracking with brain activity tracking to present and update user interfaces (UI) or user experiences (UX) that are strategically designed to achieve high speed and high accuracy in human-machine interaction. Embodiments described herein also relate to hardware-independent brain-computer interface implementations that mediate user interaction of a machine using real-time target tracking and online analysis of neural activity.

[0003]

[0003] A brain-computer interface (BCI) is a hardware and software communication system that enables the control of a computer or external device solely through brain activity using a direct communication channel between the brain and the external device. BCIs are primarily designed as assistive technologies to provide direct access to machines and applications in operation by interpreting brain signals. One of the main objectives of BCI development is to enable communication for severely disabled individuals who are completely paralyzed or "fixed" due to neurological neuromuscular diseases such as amyotrophic lateral sclerosis, brainstem stroke, or spinal cord injury, for whom effective communication with others can be extremely difficult.

[0004]

[0004] Some known implementations of brain-computer interfaces include spellers, such as the one designed by Farwell and Donchin. In this speller, the 26 letters of the alphabet, along with several other symbols and commands, are displayed on the screen in a 6x6 matrix with randomly flashing rows and columns. The user focuses their attention on the screen and continuously focuses their awareness on the letter they intend to write, while the brain's neural response to signature neural signals is monitored. When detected, signature neural signals allow the system to identify the desired symbol. This Farwell-Donchin speller allows a person to spell at a rate of about two letters per minute.

[0005]

[0005] BCI systems can be designed to assist and enhance even physically capable individuals in operating computers or other data processing machines and / or software applications without requiring conventional input / output interfaces such as mice and keyboards. BCI can also provide an interface for more intuitive and natural interaction with computers than conventional input methods. In addition, BCI can be developed to provide many other functions, including enhancing, repairing, mapping, and exploring the cognitive and / or sensorimotor systems and their functions in humans and animals. Some applications of BCI include, among others, word processors, adaptive web browsers, brain-controlled wheelchairs or neuroprostheses, and games. [Overview of the project] [Means for solving the problem]

[0006] overview

[0006] This disclosure describes systems, apparatus, and methods for various embodiments of a hardware-independent integrated oculomotor-neuro-hybrid brain-computer interface (BCI) platform for tracking eye movements and brain activity to mediate real-time positioning of a user's gaze or attention and selection / activation of desired actions. This disclosure presents an integrated BCI system to address the demand for brain-computer interfaces that operate at high speed and with high precision. [Brief explanation of the drawing]

[0007] Brief explanation of the drawing [Figure 1]

[0007] This is a schematic diagram of a hybrid brain-computer interface system according to one embodiment. [Figure 2]

[0008] This is a diagram illustrating a series of steps in an example of an implementation of a pointing control function and an action control function for selecting / deselecting a single stimulus icon using one embodiment of a BCI device. [Figure 3]

[0009] Figure 3A shows the UI / UX during the presentation of options, and Figure 3B shows the UI / UX after the pointing control function has been implemented.

[0010] Figure 3D shows a user wearing an eye tracker and a nerve recording headset according to one embodiment. [Figure 3C]

[0009] The UI / UX after the action control function has been implemented is shown. [Figure 3E]

[0011] Figure 3D shows an example of signals acquired by the eye tracker and neural recording headset. [Figure 3F]

[0011] An example of signals acquired by the eye tracker and neural recording headset shown in Figure 3D is shown. [Figure 3G]

[0012] This is an explanatory diagram of signal analysis using an example of a classifier used in a BCI device according to one embodiment. [Figure 4]

[0013] An example of an operation flow followed by a processor in a brain-computer interface device to clarify a user's intention according to an embodiment is shown. [Figure 5A]

[0014] An example of a BCI system presenting an example of a stimulation group (e.g., tag group blinking) and an example of the UI / UX of a speller is shown. [Figure 5B]

[0015] Examples of neural signals obtained in response to presenting a stimulation (or tag group) including a target tag or stimulation intended by a user and a stimulation (or tag group) not including it using a BCI system according to an embodiment are shown. [Figure 5C]

[0016] Examples of brain signals obtained from various brain regions in response to repeatedly presenting tag group blinking using a BCI system according to an embodiment are shown. [Figure 6A]

[0017] An explanatory diagram of a brain activity signal obtained by an embodiment of a BCI system is shown. [Figure 6B]

[0018] An example of the analysis of a brain activity signal by a classifier in a BCI system according to an embodiment is shown. [Figure 7]

[0019] A flowchart showing an example of a method for determining a target or tag of interest in a BCI system according to an embodiment is shown. [Figure 8A]

[0020] An example of an analysis method used to determine a target or tag of interest to a user used when implementing a BCI system according to an embodiment is shown. [Figure 8B]

[0020] An example of an analysis method used to determine a target or tag of interest to a user used when implementing a BCI system according to an embodiment is shown. [Figure 9]

[0021] A schematic flowchart of an example of a method for determining a tag of interest in a BCI system according to an embodiment is shown. [Figure 10A]

[0022] This example shows a UI / UX with visible tags or visible symbols that, according to one embodiment, demonstrates a relationship that depends on the distance between tags used to determine the target tag. [Figure 10B]

[0023] One embodiment demonstrates a distance-dependent relationship between brain signal activity induced by nearby tags, which is used to determine a target tag. [Figure 10C]

[0023] One embodiment shows a distance-dependent relationship between brain signal activity induced by nearby tags, which is used to determine a target tag. [Figure 11]

[0024] This flowchart shows an example of a method for determining a target or tag in a BCI system according to one embodiment. [Figure 12]

[0025] A schematic flowchart of an example of a scaling method based on the distance of scores associated with a tag when determining the target tag, according to one embodiment, is shown. [Figure 13]

[0026] A schematic flowchart of one embodiment of a method for incorporating score-based distance scaling when determining the target tags is shown. [Figure 14]

[0027] This flowchart shows an example of a method for determining a target or tag in a BCI system according to one embodiment. [Figure 15]

[0028] A schematic flowchart of an example of a procedure for generating a visual score based on eye movement signals, according to one embodiment, is shown. [Figure 16]

[0029] This flowchart illustrates an example of a method for determining a tag to incorporate signals from various sensors, according to one embodiment. [Figure 17]

[0030] A flowchart illustrating an example of a procedure that combines the analysis of signals from various sensors to determine a target tag, according to one embodiment, is shown. [Modes for carrying out the invention]

[0008] Detailed explanation

[0031] Embodiments described herein relate to systems, apparatus, and methods for use in implementing brain-computer interfaces (BCIs) that analyze brain activity recorded while presenting a user interface (UI) or user experience (UX) to a user, which is strategically designed to achieve high speed and high accuracy in human-machine interaction. Embodiments described herein also relate to hardware-independent brain-computer interface implementations that use neurobrain signal analysis to mediate user interaction with interfaces, apparatus, and / or machines.

[0009]

[0032] For BCI technology to become more patient-friendly, useful for the general public, and usable in controlling real-world tasks, the speed of information transfer must be improved to match a more natural pace of interaction compared to current implementations, the error rate must be reduced, and the complexity of the interaction interface must be minimized. In addition, BCI applications require a high cognitive load on the user, and therefore the UI / UX and underlying signal processing must be improved to move away from the quiet laboratory environment and into the real world. To configure BCI devices and applications to be easier and more intuitive, improved devices and techniques in brain-machine interface implementations are needed that operate at high speed and with high accuracy, enabling user-mediated action selection through a natural and intuitive process.

[0010] BCI System

[0033] As described herein, a BCI is a hardware and software communication system that enables the control of a computer or external device by either brain activity alone or in combination with other activities such as oculomotor activity or motor nerve (e.g., EMG) activity. A BCI system includes hardware devices for displaying stimuli via an interface, for positioning the user's point of focus on the interface, for recording and processing brain activity, and for controlling the interface, the control of which may involve controlling the user's environment. These standard functions can be characterized as (1) pointing control functions, (2) action control functions, and (3) user interface / user experience (UI / UX) functions. Pointing control functions can be described by analogy to conventional pointing devices such as a mouse pointer, which allow the user to narrow down to a small set of one or more manipulators for control. Action control functions can be described by analogy to devices that mediate actions (e.g., selection, deselection, etc.), such as mouse clicks or keystrokes on a keyboard, which enable the user to perform actions that result in changes to the UI / UX and, consequently, changes to the connected machine. The UI / UX functionality in a BCI system can be described by analogy to an operating system that creates and maintains an environment for implementing pointing control and action control functions, in addition to other functions such as providing selection menus and navigation control.

[0011]

[0034] The actions performed by the action control function can be one of many and can be adapted to suit various versions of the UI / UX designed to control various devices or machines. To give a few examples, actions can be activation or deactivation, continuous or semi-continuous changes to the UI / UX. In particular, scrolling, hovering, or pinching, zooming, tilting, rotating, and swiping. Actions can also cause abrupt changes to the UI / UX using discontinuous starts and stops, such as highlighting. Some other examples of action control by UI / UX may include virtual keyboard control, checkboxes, radio buttons, dropdown lists, list boxes, toggles, text fields, search fields, breadcrumb navigators, sliders, menu navigation, actions to position and unposition objects or items, actions to move objects or items, actions to zoom in and / or out of objects, movement or navigation of a first-person observer or player, changing the observer's viewpoint, and actions such as grabbing, picking, or hovering. Some of these aspects of action control are disclosed below.

[0012]

[0035] In some embodiments of the BCI system implementation, pointing control functionality and methods for identifying the user's point of focus may be implemented by manipulating the UI / UX and / or by using brain signals that can provide information about the user's point of focus. In some embodiments of the BCI system described herein, pointing control functionality and identification of the user's point of focus may include eye movement tracking devices and / or head motion tracking devices, or other body motion tracking devices or body position tracking devices. In yet other embodiments, a combination of brain signals, target tracking signals, motor nerve signals such as electromyography (EMG) signals, and strategic manipulation of the UI / UX may be used simultaneously (e.g., in the BCI system) or individually to implement pointing control functionality. In addition to the signals described above, hybrid or other BCI systems may also monitor and use other signals from various peripheral sensors (e.g., head position tracking signals). In some embodiments, hybrid or other BCI systems may optionally include an electromyograph (EMG) for recording EMG signals that can be integrated with oculomotor signals or neural activity signals.

[0013]

[0036] In some embodiments, the action control function and the method for identifying user intent may include any suitable form of monitoring neural signals in the brain. Such forms may include, for example, brain imaging by electroimaging, optical imaging, or magnetic imaging. For example, in some embodiments, the BCI system may use electrodes that record neural signals of brain activity transmitted via amplifiers and processors that translate the user's brain signals into BCI commands. In some embodiments, the BCI system may implement a sophisticated UI / UX that performs machine control based on brain activity. As described below, specific adaptations to one or more of these features may be implemented to achieve high speed and high accuracy in human interaction with the BCI system. For example, in some embodiments, the BCI system may be substantially similar to that described in U.S. Patent Application No. 62 / 549253 ("Application 253"), filed on 25 August 2017 and titled "Brain-computer interface with high-speed eye tracking features," which is incorporated herein by reference in its entirety.

[0014]

[0037] The UI / UX can be adapted to take into account the needs that the BCI system will satisfy. For example, a BCI system used by patients to obtain mobility may include a UI / UX that aims for ease of use along with low cognitive load. As another example, a BCI system used by children as a learning tool may include a UI / UX adapted to intuitive interaction by children. Similarly, a BCI system intended for a gaming experience may include a UI / UX designed to provide high speed and high accuracy, etc. For example, in some embodiments, the BCI system and / or user interface / user experience (UI / UX) may be substantially similar to that described in U.S. Patent Application No. 62 / 585209 ("Application 209"), filed November 13, 2017, titled "Brain-computer interface with adaptations for high-speed, accurate, and intuitive user interactions," which is incorporated herein by reference in its entirety.

[0015]

[0038] Figure 1 is a schematic diagram of a brain-computer interface system 100 according to one embodiment. An example of this brain-computer interface system 100 (also referred to herein as the “Hybrid BCI System,” “BCI System,” or “System”) is a BCI system that includes a neural recording headset 104 (e.g., a neural recording device) for recording one or more control signals from the user’s brain. The BCI system 100 may also include an eye tracker 102 (e.g., a target tracking device), which may be a video-based eye tracker. The eye tracker 102 may be an accessory or integrated into the BCI system 100. The eye tracker 102 may be configured to capture, record, and / or transmit the user’s eye movement response indicating the user’s point of focus at any given time (i.e., pointing control function). The neural recording headset 104 may be configured to capture, record, and / or transmit neural control signals from one or more brain regions indicating the user’s cognitive intentions (i.e., action control function). In some embodiments, the neural recording headset 104 may be adapted to indicate the user's point of interest, implementing a pointing control function. The neural control signals may be any form of neural activity recorded by any suitable method, such as electroencephalography (EEG), corticoelectroencephalography (ECoG), magnetoencephalography (MEG), endogenous signal imaging (ISI), etc. Examples of forms of neural activity include event-related potentials (ERPs), motor images, steady-state visual evoked potentials (SSVEPs), transient visual evoked potentials (TVEPs), brain state commands, visual evoked potentials (VEPs), P300 evoked potentials, sensory evoked potentials, motor evoked potentials, sensorimotor rhythms such as Mu-rhythms or Beta-rhythms, event-related desynchronization (ERDs), event-related synchronization (ERSs), slow brain potentials (SCPs), etc. An example of this BCI system 100 may also include a brain-computer interface device 110, one or more optionally selected peripheral sensors 108, and optionally an audiovisual display 106.Some embodiments of the BCI system 100 may also include other peripheral sensors 108 (not shown in Figure 1) and peripheral actuators to collect data on user behavior in other ways, such as sound, touch, and orientation, and to provide a rich and diverse user experience.

[0016]

[0039] In some embodiments of the BCI system 100, neural and oculomotor signals (and other peripheral signals from peripheral sensors 108) collected from the neural recording headset 104 and eye tracker 102, respectively, can be transmitted to a brain-computer interface (BCI) device 110 that processes these signals individually or together as an ensemble. In connection with signal processing, the BCI device 110 can also access and process data about stimuli presented by the UI / UX that evoked the signals being processed. With the combined information, the BCI device 110 can detect relevant signal features based on a statistical model, as will be described in more detail below, and apply appropriate confidence scores to predict the user's intent. This predicted intent can be communicated to the user, for example, by the UI / UX presented by the display 106, and can be used to cause changes within the UI / UX and in any connected controllable machine.

[0017] Target tracking and pointing control function in 2D and 3D space.

[0040] In some embodiments, the eye tracker 102 can be used to determine where a user is looking within their field of vision by rapidly tracing the user's eye movements in two-dimensional or three-dimensional space. For example, provided the user has voluntary accommodation of their eye movements, the video-based eye tracer 102 can be used to determine which subspace within the user's field of vision each of the user's eyes is "pointing" to. In other words, the eye tracker 102 can use the trajectory of the user's eye movements as a pointing control function, revealing important information about the subject's intentions and behavior. In some embodiments, aspects such as where the user is focusing in visual space, which stimulus they are focusing on, or which stimulus they are responding to can be effectively used within the BCI system 100. By simultaneously tracking the movement trajectories of both eyes relative to each other, the eye tracker 102 can also register the depth of the user's focus, thus enabling pointing control in three-dimensional space.

[0018]

[0041] In some embodiments, the eye tracker 102 relies on tracking the user's pupil and the surface corneal reflection (CR) of the illumination source by using a head-mounted target-tracking video camera to image the user's eye. The positional difference between these two features can be used to determine the observer's line of sight direction. Some examples of head-mounted target-tracking devices usable as the eye tracker 102 are available from several commercial vendors, including SenseMotoric Instruments, Tobii Eye Tracking, and Pupil-labs. In some embodiments, the eye tracker 102 may include one or more illumination sources to illuminate the user's eye. The illumination sources can emit light of any suitable wavelength and can be mounted in any suitable position. The illumination sources may be connected by wired or wireless communication for functional control and data transmission, etc.

[0019]

[0042] The eye tracker 102 may include left and right eye cameras, each configured to simultaneously image the pupil and the corneal reflection of one or more light sources from each eye. These cameras can be connected to each other by wired or wireless connections and may be connected to an external device such as a brain-computer interface (BCI) device 110 shown in Figure 1. The eye tracker may also include an additional scene camera to capture the user's field of view. Signals from the scene camera can also be relayed to an external device such as the BCI device 110 by wired or wireless communication.

[0020]

[0043] In some embodiments, the eye tracker 102 may include an integrated display 106 rather than a separate display 106. For example, an eye tracker 102 integrated with the display 106 may be a system configured to view a virtual reality space. In some embodiments, an eye tracker 102 integrated with the display 106 may be configured to view an augmented reality space; that is, it functions as eyeglasses to view the real world with the superimposed UI / UX presented by the display 106 added.

[0021] Brain signal neural recording - action control function

[0044] The purpose of the BCI system 100 is to actively control the associated UI / UX and / or connected external devices and / or machines by revealing the user's intentions by monitoring brain activity, such as predicting the user's intended actions and / or determining the user's intended actions by interpreting signals related to the user's activities. The key to this purpose is brain signals that can indicate the user's intentions, which can then be used as action control functions. The BCI system 100 can use one or more of several signature brain signals that are simultaneously evoked by or related to cognitive tasks performed by the user. Some of these brain signals can be decoded in a way that allows a person to modulate them at will. Using these signals, which can be considered control signals, can enable the BCI system 100 to interpret the user's intentions.

[0022]

[0045] The neural recording headset 104 can be adapted to record neural activity generated by electrochemical transmitters that exchange information between neurons, using any appropriate technique. Neural activity can be directly captured by electrically recording primary ion currents generated by neurons, which flow within and within neural clusters. Neural activity can also be indirectly captured by recording secondary currents or other changes within the nervous system that are related to or result from primary currents. For example, neural activity can also be monitored by other methods such as optical imaging (e.g., functional magnetic resonance imaging, fMRI) by optical changes during recording resulting from primary currents. Other techniques for recording neural activity in the brain include electroencephalography (EEG), electrocortical electroencephalography (ECoG), functional near-infrared (FNIR) imaging, and other similar intrinsic signal imaging (ISI) techniques, magnetoencephalography (MEG), etc.

[0023]

[0046] Various signature brain signals, taking the form of neural activity, can be used as control signals to implement action control functions. Some examples of neural activity related to time include event-related potentials (ERPs), evoked potentials (EPs, e.g., sensory evoked potentials, motor evoked potentials, visual evoked potentials), motor imagery, slow cortical potentials, sensorimotor rhythms, event-related desynchronization (ERDs), event-related synchronicity (ERSs), brain state-dependent signals, and other undiscovered signature action potentials that underlie various cognitive or sensorimotor tasks. Neural activity can also be in the frequency domain. Some of the many examples include sensorimotor rhythms, event-related spectral perturbations (ERSPs), and specific signal frequency bands such as theta, gamma, or mu rhythms.

[0024]

[0047] As described herein, the neuro-recording headset 104 can record neural activity signals to gather information about the user's intentions through a recording phase that measures brain activity, and convert that information into an easily manageable electrical signal that can be converted into commands. In some embodiments, the neuro-recording headset 104 may be configured to record electrophysiological activity by electroencephalography (EEG) that is highly temporally responsive, has low setup and maintenance costs, is highly portable, and is non-invasive to the user. The neuro-recording headset 104 may include a pair of electrodes having sensors that acquire EEG recording signals from different brain regions. These sensors can measure electrical signals triggered by the flow of current during synaptic excitation of dendrites within neurons, thereby relaying the effects of secondary currents. The neural signals can be recorded by electrodes in the neuro-recording headset 104 that are appropriately positioned over desired brain regions when placed on the user's head, scalp, face, ears, neck, and / or other parts. Examples of neuro-recording headsets may be available from commercial vendors such as Biosemi, Wearable Sensing, and G.Tec, among others. For example, in some embodiments, the neural recording headset 104, the operation of the neural recording headset 104 when collecting neural brain activity signals, and the signal transmission from the neural recording headset 104 may be substantially the same as those described in Application No. 253, the entire disclosure of which is incorporated herein by reference above, and / or those described in Application No. 209, the entire disclosure of which is incorporated herein by reference above.

[0025]

[0048] The neural activity recorded and analyzed to decode user intent may be any form of control signal indicating user intent. An example of a control signal may be an event-related potential (e.g., a P300 signal). An event-related potential, or ERP, may be signature neural activity related to a temporally correlated event or stimulus presentation. ERPs may have distinctive shapes and characteristics (such as the P300 signal, which is known to peak approximately 300 ms from the trigger stimulus) that aid in their detection and identification. ERPs may vary in size and shape across different brain regions, and how ERPs map between brain regions may indicate specific brain function and / or user intent. Neural activity data acquired from a neural recording headset can be analyzed for specific ERP signals, and once appropriately detected and classified, the BCI device 110 can perform any specific action related to the detected ERP on a desired part of the UI / UX.

[0026]

[0049] Another example of a control signal may be in the form of a motor image signal, which is a neural activity signal associated with the user experiencing the mental process of movement. That is, a motor image signal is a brain signal that is recorded from various brain regions while the user imagines and / or performs an action and can be analyzed by the BCI system 100. The BCI system may also use information collected by peripheral sensors 108, such as goniometers and torsometers, to help recognize gestures with high resolution during training sessions.

[0027] UI / UX display and presentation

[0050] As described herein, the UI / UX within the BCI system 100 functions as a communication link between the user (e.g., the user's brain, eyes, muscles / motor nerves, etc.) and the BCI device 110, allowing the user to focus on and point to specific stimuli using pointing control functions and to select or deselect specific stimuli using action control functions. As described herein, the UI / UX can be an example of a control interface. The UI / UX may include a series of visually stimulating two-dimensional images presented by a display. The UI / UX may be designed and operated by the BCI device 110 to be presented in a way that is most intuitive to the user and makes it easier and clearer to identify the user's intent. The UI / UX may present one or more stimuli designed to attract the user's attention and / or to convey information about the UI / UX, including information about the availability of methods of user control. The stimuli can be presented in any suitable way. For example, the UI / UX may be designed to present "tags" (e.g., control items) as stimuli. Each stimulus may contain one or more tags. For example, tags can be visual icons that change their appearance in a specific way to attract the user's attention and to indicate their availability for controlling the UI / UX. For example, a group of one or more tags can be made to flash or change their appearance in a specific way. Tags or control items can be associated with actions. For example, a transient change in the appearance of tags, also referred to herein as “tag flashing,” can indicate that those tags are available to perform one or more specific actions. Two or more tags can be flashed at once, and the grouping of tags (also referred to herein as “tag grouping”) can be done in any specific way (e.g., rows, columns, pseudo-random grouping of tags). After tag flashing, the eye tracker 102 can capture a signal indicating that the user has foveated to the location of the tag flashing, and / or the neural recording headset 104 can capture a signal indicating the occurrence of signature brain activity. The BCI device 110 can analyze these signals to reveal the user’s intent, as will be described in more detail herein.Based on this decision, the UI / UX can perform one or more specific actions related to tag blinking.

[0028]

[0051] As explained above, UI / UX can also be a rich mixture of stimuli in several forms that together form what can be called a user experience (UX) that also functions as an interface (UI). As explained above regarding user interfaces, a strategically designed user experience involves the process of presenting stimuli to the user in any form that manipulates the presentation (similar to flashing tags). By analyzing brain activity signals and associated eye movement signals and / or other peripheral signals and decoding the user's intent, the UI / UX can perform one or more specific actions related to the presented stimuli.

[0029]

[0052] Some examples include visual stimuli, auditory stimuli, tactile stimuli, or vestibular stimuli. In some embodiments, a UI / UX can be rendered that presents visual stimuli on a display, such as the display 106 shown in Figure 1. Other forms of stimuli can be delivered by appropriate peripheral actuators (not shown in Figure 1), which are also part of the BCI system 100.

[0030]

[0053] In some embodiments, the display 106 can be a separate, standalone audiovisual display unit that can connect to and communicate data with the rest of the BCI system 100. That is, a standalone display (e.g., a liquid crystal display) equipped with an audio system (e.g., speakers or headphones) can communicate bidirectionally with one or more of the other components of the BCI system 100, such as the BC interface device 110, the eye tracker 102, and the neural recording headset 104. In some embodiments, the display 106 can be integrated with the eye tracker 102 so that it is part of the eyeglass area. The integrated eye tracker 102 and display 106 can be configured to view the augmented reality space in a UI / UX format presented on the display 106. In some embodiments, the integrated eye tracker 102 and display 106 can be configured such that the display 106 rests on a translucent eyeglass area, allowing the user to view the augmented reality space. That is, the user can see the real world through a translucent eyeglass area, which is also an integrated display 106 that presents the user with an interactive UI / UX.

[0031] Peripheral devices that operate in a non-visual manner

[0054] In some embodiments, the BCI system 100 may include several peripheral sensors 108 (shown as optional units indicated by dashed rectangles in Figure 1) and peripheral actuators (not shown in Figure 1). One or more peripheral actuators may be configured to provide a rich and diverse user experience, and one or more peripheral sensors 108 may be configured to capture diverse inputs from the user and the user's environment, respectively. These peripheral actuators 112 and sensors 108 may be appropriately mounted individually or integrated into other devices (such as the eye tracker 102). For example, the BCI system 100 may include earphones for relaying auditory stimuli and microphones for capturing sounds such as user voice commands. The earphones (auditory actuators or auditory output devices) and microphones (auditory sensors or auditory input devices) may be standalone devices connected to the hybrid system 100 by a wired or wireless channel. Alternatively, the earphones and microphones may be mounted and integrated into the eye tracker 102 or the neural recording headset 104. Similarly, peripheral sensors such as accelerometers, goniometers, torsometers, optical sensors such as infrared cameras, depth sensors, and microphones can be included in and / or coupled to the BCI system 100 to register body movements. For example, goniometers can be used to register limb movements that form gestures, and accelerometers can be used to register body movements. Peripheral sensors may also include field-of-view cameras configured to capture the user's real-world field of view. Signals acquired by the field-of-view camera can be analyzed and used to generate and present an augmented reality or mixed reality experience to the user, which has real-world images superimposed by a UI / UX using selectable options, etc. Peripheral actuators connectable to the BCI system 100 may include haptic or kinesthetic devices that can create and apply forces such as touch and vibration to enrich the user experience provided.

[0032] Brain-computer interface device

[0055] In some embodiments, the brain-computer interface device (i.e., BCI device) 110 may be configured to achieve three main functions in particular. First, the BCI device 110 may be configured to generate a strategically designed UI / UX as described herein. For example, the strategically designed user experience may be for training sessions or testing sessions. In some embodiments, the user experience may be designed as a virtual reality environment and / or an augmented reality environment. In some embodiments, the UI / UX may be tailored to specific needs such as a particular user's history, reaction time, user selection, etc. The BCI device 110 may take all of these requirements into consideration when generating and updating the UI / UX. Second, in addition to designing and generating the UI / UX, the BCI device 110 may be configured to receive pointing control signals (e.g. from an eye tracker 102) and action control signals (e.g. from a neural recording headset 104) (and peripheral signals from peripheral sensors 108, if applicable) and process these signals individually or as an ensemble to reveal the user's intent. The BCI device 110 may perform any method suitable for analysis. For example, the BCI device 110 can detect meaningful features from a signal, construct and apply a statistical model to interpret the signal, classify the signal, score the signal and the stimuli that elicit the signal, calculate the probability that any given tag or stimulus is the point intended by the user (e.g., a target tag or target stimulus), and determine the target tag or target stimulus and the associated action desired by the user. Thirdly, the BCI device 110 can be configured to implement pointing control and action control functions by changing the designated target tag or target stimulus according to the user's intention.

[0033]

[0056] In some embodiments, the BCI device 110 may also be connected to other peripheral devices that may be part of the BCI system 100, such as peripheral sensors and actuators that function in ways other than the visual modes described above. Such peripheral sensors may include voice microphones, tactile sensors, accelerometers, goniometers, etc., and peripheral actuators may include voice speakers, tactile stimulus providers, etc.

[0034]

[0057] In some embodiments, the BCI device 110 may include an input / output unit 140 configured to send and receive signals between the BCI device 110 and one or more external devices via a wired or wireless communication channel. For example, the input / output unit 140 may send and receive signals between an eye tracker 102, a nerve recording headset 104, and an optional audiovisual display 106 via one or more data communication ports. The BCI device 110 may also be configured to connect to a remote server (not shown in Figure 1) and access a database or other appropriate information contained within the remote server. The BCI device 110 may include a communicator 180 configured to handle an appropriate communication channel suitable for the type of data being transferred. The communicator 180, along with the rest of the BCI device 110, is connected to the I / O unit 140 and can control the functions of the input / output unit 140. Signal transfer may be performed via a wired connection such as wired Ethernet, serial, FireWire, or USB connection, or wirelessly via any appropriate communication channel such as Bluetooth or short-range wireless communication.

[0035]

[0058] In some embodiments, the functions of the input / output unit 140 within the BCI device 110 may include several procedures such as signal acquisition, signal preprocessing, and / or signal augmentation. The acquired and / or preprocessed signals can be sent to the processor 120 within the BC interface device 110. In some embodiments, the processor 120 and its sub-components (not shown) may be configured to process input data and exchange data with memory 160. The processor 120 may also connect to a communicator 180 to access a remote server (not shown in Figure 1) and utilize information from the remote server.

[0036]

[0059] The processor 120 within the BCI device 110 may be configured to perform functions for building and maintaining a renderable UI / UX on the display 106 or on a display integrated into the eye tracker 102. In some embodiments, the processor 120 and its sub-components may be configured to perform functions necessary to enable user-specific interpretation of brain signals and to package output signals to the input / output unit 140 for relay to an external device. Other functions of the processor 120 and its sub-components may include several procedures such as feature extraction, classification, and control interface operation.

[0037]

[0060] In some embodiments, the BCI device 110 can be configured to optimize the speed so that the execution of action control occurs within 5 seconds, or within 4 seconds, or within 3 seconds, or within 2 seconds, or within 1 second, or within 0.9 seconds, or within 0.8 seconds, or within 0.7 seconds, or within 0.6 seconds, or within 0.5 seconds, in order to improve the user experience. In some embodiments, in order to improve the user experience, the execution of action control speed (in seconds) multiplied by the average accuracy of the system (in percent) can be optimized to 5 (e.g., 10 seconds). * Less than 50% precision, or less than 4, or less than 3, less than 2, or 1.125 (e.g., 1.5s) * Less than 75% precision, or less than 1, or 0.9 (e.g., 1s) *(90% precision) less than 0.8, or less than 0.7, or less than 0.6, or 0.5 (e.g., 0.6s) * The speed should be less than 83.33% accuracy, or less than 0.4, or less than 0.3, or less than 0.2, or even less than 0.1. * The BCI device 110 can be adjusted to reduce or minimize the accuracy % value.

[0038] Action indication and selection - BCI system functionality

[0061] Figure 2 illustrates the functionality of a BCI system (similar to system 100 above) with respect to a specific example in which a user focuses on an example of an input symbol and controls its selection. An example of a series of operational events for illustrating Figure 2 includes presenting a stimulus (e.g., a stimulus containing a set of tags associated with a set of actions), acquiring the resulting neural activity signals and oculomotor signals, and / or peripheral signals, where applicable, analyzing the acquired signals, interpreting them to infer or decode the user's intent, and causing a change in the UI / UX (e.g., by selecting one or more tags associated with one or more actions). One or more actions performed to change the UI / UX may also control one or more external machines connected by the UI / UX.

[0039]

[0062] A concrete example of the functionality of the BCI system shown in Figure 2 begins with step 251, which presents an input stimulus. The input stimulus may be a set of tags or symbols 279, as shown in the example UI / UX 271. All tags 279 within the UI / UX 271 can be made visible, but one or more tags 279 may have their visual appearance transiently altered to indicate whether that tag can be used for selection. The change in appearance may be a change in any appropriate property of the tag (e.g., fill, transparency, brightness, contrast, color, shape, size, orientation, texture, hue, contour, position, depth in a 3D environment, mobility, etc.). For example, one or more tags 279 may flash to indicate a potential selection (otherwise referred to herein as "tag flashing"). Various groups of visible tags 279 may flash together, resulting in several combinations of tag flashing or several tag group flashing, each tag flashing or tag group flashing being a stimulus. While it was noted that examples of stimuli are presented in a visual format and changes are presented in a visual format, it should be noted that any appropriate format can be used to present stimuli and perform similar action selections. For example, auditory sounds can be used as tags. Any appropriate auditory characteristics of auditory tags can be transiently changed to demonstrate the availability of those tags as selections. For example, characteristics such as volume, duration, pitch, chirp, and timbre can be transiently changed so that they can be used as tag blinking in the auditory space of a UI / UX.

[0040]

[0063] The various tags presented, such as the three symbols 279 within UI / UX 271, each represent a separate action during selection, which can act as an intermediary. One of the visible tags may be a target tag or a tag the user wants to select. The goal of the BCI system (such as the BCI system 100 above) is to determine which of the visible tags 279 is the target tag the user wants to select, using an example procedure shown in Figure 2.

[0041]

[0064] The UI / UX 271 can be configured in step 251 to present each visible tag 279 as a stimulus (e.g., by flashing the tag) once or multiple times, and in step 253 the BCI system (e.g., system 100) acquires, if applicable, the resulting brain activity signals 273 and / or eye movement signals 275 and signals from other peripheral sensors (not shown), along with information about the stimulus presentation 277 (e.g., where on the UI / UX 271, at what time, which tag or tag group was presented, etc.). Visible tags 279 can be presented by flashing the tag individually or by flashing a combination of tag groups. Flashing tags in a tag group can reduce the number of flashes required to identify the target tag 285. Stimulus presentation may also include the presentation of invisible stimuli, such as ghost flashes that are not associated with tags and are expected to go unnoticed by the user. Ghost flashes can be used to calibrate the stimulus presentation by the UI / UX 271. For example, ghost flashing can be used to set a detection threshold while analyzing signals indicating user gaze or attention to a specific tag 279.

[0042]

[0065] Step 255 of the procedure shown in Figure 2 involves analyzing the acquired oculomotor signals 275 and / or nerve signals 273 (and other peripheral signals from other sensors), which can be performed individually or as an ensemble in an integrated method, as further disclosed below. The analysis of the signal nerve and oculomotor (and peripheral) signals is performed in the context of stimulus information 277 (e.g., spatiotemporal characteristics of the presented stimulus). The analysis may include one or more steps of several computational methods, such as signal preprocessing, feature detection and extraction, dimensionality reduction, supervised classification, unsupervised classification, or semi-supervised classification, construction or application of one or more pre-built statistical models for interpreting the signals, calculation of confidence scores for each analysis (e.g., confidence scores for classification), calculation of appropriate ways to incorporate and use the stimulus information 277 (e.g., application of one or more scaling functions), calculation of the likelihood that each tag 279 is a target tag 285, decoding and / or decision-making regarding the identity of the target tag 285, etc.

[0043]

[0066] For example, step 257 may include determining the identity of the target tag 285 based on the analysis performed in step 255. The decision or determination in step 257 can be made using any suitable method. For example, one or more threshold crossing algorithms or machine learning tools may be used.

[0044]

[0067] The decision in step 257 may lead to a selection of tag 279 in step 259. The selection in step 259 may then lead to the execution of a related action. For example, if target tag 285 is correctly identified as an octagonal tag, action 2, which is associated with octagons, can be performed. One or more user authentication steps may be included to verify whether the identification of target tag 285 was correct. The user can provide feedback on whether the identification of target tag 285 was correct or incorrect. This user feedback can be used to support or correct various analytical processes and statistical models used to determine target tag 285, and to train the BCI system to better suit specific users or use cases. This feedback can also be used to train the user. For example, if the information for making a decision in 257 is insufficient, for example, due to ambiguity or one or more signals being too weak, the user can be given an indicator to try again under different circumstances (e.g., better concentration).

[0045] User interaction with the BCI system

[0068] Figures 3A to 3G show an example of user interaction with a BCI system 300 and some of the underlying processes within the BCI system 300, which may be the same or similar in structure and / or function as the BCI system 100 disclosed above. For example, the BCI system 300 may include an eye tracker 302, a neural recording headset 304, a BCI device (not shown), and a display 306. In the illustrated example, the BCI system 300 can use both oculomotor signals to perform pointing control and neural signals to perform action control to help the user spell words and / or sentences. For example, the BCI system 300 may include a UI / UX 371 used to spell words in a two-step process, and a display 306 that presents the UI / UX 371. As shown in Figures 3A to 3C, the UI / UX 371 can present stimuli in the form of tag group flashing 379 (e.g., letters, numbers, and symbols commonly found on a keyboard).

[0046]

[0069] The pointing control function described in Figures 3A to 3C can be implemented using data acquired by an eye tracker 302 (an example of which is shown in Figure 3D). The eye tracker 302 may be configured to record oculomotor signals to detect where the user is focusing their gaze and to output signals corresponding to each eye that can be processed by a BCI device (not shown). An example of oculomotor signals is shown in Figure 3E. The action control function (i.e., target tag determination and target tag activation) is implemented using data recorded by a neural recording headset 304 (an example of which is shown in Figure 3F). The neural recording headset 304 may be configured to record neural signals from specific areas in the user's brain, and a BCI device (not shown) can analyze these signals, an example of which is shown in Figure 3F. The BCI device (not shown) can extract meaningful features from the oculomotor signals (Figure 3E) and neural signals (Figure 3F) and analyze these features by classifying the signals in an unsupervised and / or semi-supervised manner or based on a statistical model built by training. The BCI device can incorporate stimulus information in determining and selecting target tags 385.

[0047]

[0070] For example, a user can focus their attention on a group of tags containing a target tag (e.g., the letter Q), as shown in Figure 3A. In this example, the tag group flashing could be in the form of a highlighted circle. After analyzing the eye-movement signal indicating that the user has focused on that particular group of tags, the tag group indicated by the highlighted circle in Figure 3A can be enlarged and the UI / UX 371 can be changed to what is shown in Figure 3B, as shown in Figure 3B. For the user to choose, a set of different tag flashes 379 of the enlarged group in Figures 3B and 3C can be presented in sequence. As shown in Figure 3C, the tag flashing could be enlarging and making the letters bold.

[0048]

[0071] As shown by an example of the projection 381 of the signals used for classification (shown in Figure 3G), the target tag 385 can be selected after appropriately analyzing the oculomotor signals, nerve signals, and / or other relevant signals. By repeating this procedure to select each character to be used, words and / or sentences can be spelled using the BCI system 300 and implementation procedures described above.

[0049] Decoding of nerve signals

[0072] While the process sequence shown in Figure 2, and the implementation examples of pointing and action control shown in Figure 3, can be illustrated for individual stimuli, a similar process with a series of similar steps can be followed while presenting diverse environments that are virtual or augmented by the UI / UX or user experience. An example of process 400 is shown in Figure 4. As shown in Figure 4, process 400 can include a series of substeps (indicated as optional by dashed rectangles) that form a training session, or it can be used to present a new stimulus without any training data.

[0050]

[0073] An example of process 400 shown in Figure 4 describes some of the steps involved in interpreting recorded signals, clarifying user intent, and operating based on user intent. Process 400 includes an initial step 401 initiating the BCI system at a given time for a specific user associated with a system including an eye tracker and / or neural recording headset (and other peripheral sensors and / or actuators) by time-specified data acquisition and UI / UX presentation. This stimulus presentation and signal acquisition initiation may be performed by a component that is part of a BCI system similar to the BCI system 100 or 300 described above.

[0051]

[0074] Process 400 may include a subset of steps for generating and training a statistical model (optionally used in training sessions, shown within the dashed rectangle in Figure 4). After the presentation of training stimuli (which may be relevant to the training environment), a subset of steps for a training session may include step 403, which receives information about the acquired signals and the training stimuli presented to the user. In step 405, the BCI system can analyze the acquired signals by any appropriate analytical procedure, for example, by detecting and extracting features that give specific information within the signals, and / or by constructing / applying one or more statistical models that take into account oculomotor signals and / or neural (and / or peripheral) signals. In step 407, the BCI system can interpret, classify, and / or label the acquired signals using any appropriate method. For example, the BCI system may associate each signal with a classified group and a confidence score that measures the confidence of the classification. Step 407 may also include updating the classification and / or labels using information about the presented stimuli (e.g., distance scaling methods, which are described in more detail below). In step 409, the BCI system may include a cross-validation step to evaluate the analytical tools used to interpret the signals and clarify the user's intent.

[0052]

[0075] After a training session or without a training session, stimuli can be presented to the user by UI / UX or user experience after data acquisition begins in step 401. These new stimuli may elicit oculomotor responses, neural responses, and / or peripheral responses, which are captured as signals by appropriate sensors in the BCI system. As shown in step 411 of process 400, these signals may be received in relation to information about the stimulus that elicited the response. In step 413, the BCI system may generate a new statistical model or use a pre-generated and cross-validated statistical model derived from training. Using the statistical model, the BCI system can analyze and interpret the signals by following analytical procedures similar to those described for steps 405 and 407. For example, the BCI system may classify and / or label the signals based on a scoring system, incorporating stimulus information within the scoring system. Based on the scores associated with each available stimulus and / or response signal, in step 415, the BCI system can reveal the user's intent (e.g., identify target tags of interest to the user). In step 417, the BCI system can perform the selection of the revealed target tag, which may result in one or more actions related to the selection of the target tag. For example, step 417 may include selecting a letter in Spella, or a character in a game, or an ON function related to a TV system that can be operated within the augmented reality system.

[0053] Signal analysis

[0076] As described herein, BCI systems 100 and 300 can process oculomotor activity signals and neural activity signals (and other peripheral signals) as an ensemble or individually to reveal and act upon user intent at high speed and with high accuracy. Appropriate stimuli can be presented using one or more processes, such as process 200 or process 400, to reveal user intent. As described below, the BCI system can employ an analytical pipeline suitable for analyzing signals and revealing user intent.

[0054]

[0077] Some embodiments of a BCI system and / or a process for implementing a BCI system may use a holistic approach to implement pointing control and action control functions, utilizing complementary information sources derived from various signals received and processed (e.g., oculomotor signals, nerve signals, peripheral signals, etc.). Furthermore, the holistic approach for processing signals and implementing the BCI interface may allow for appropriate weighting of individual signals according to other parameters such as usage, user history, and specific details of the navigated UI / UX.

[0055]

[0078] An example of an analysis pipeline for analyzing signals (e.g., neural activity signals, oculomotor signals, etc.) to reveal user intent may include: (1) appropriate preprocessing of one or more signals by one or more filter systems (e.g., a double Kalman filter or any other arbitrary delay-free filter) to improve the accuracy of selection; (2) a Bayesian linear discriminant classifier for classifying events registered within a significant epoch of the signal (e.g., an epoch after or concurrently with a stimulus or tag flashing); (3) spatial filtering across weighted signal packages; (4) a bagging ensemble classifier algorithm; and (5) a higher-order oracle algorithm that incorporates information from the classification algorithm along with program routines during experimental work.

[0056] Stimulus-response relationship

[0079] Signals acquired during and after stimulus presentation, including oculomotor signals, neural signals, or other peripheral signals (e.g., gestures, body position, voice commands, etc.), can be information-rich. However, analysis procedures can extract relevant epochs and / or features from the signals to analyze and determine target tags. For example, a BCI system may include a UI / UX 571 shown in Figure 5A for use when spelling a word. In an example where the user might want to use the letter I to spell a word, the letter I becomes the target tag 585, as shown in Figures 5A-5B. In the example shown in Figure 5A, an example of a stimulus or tag flashing could be a visible tag or a row or column of letters. For example, an example of tag flashing 579A presents the stimulus in the form of a highlighted tag group containing a row of tags G-L. Another example of tag flashing 579B (not currently presented, e.g., letters not highlighted) could include tags A, G, M, S, Y, and 5. Tag flashing can be a row, column, or an arbitrarily selected group of tags presented together by specific changes in appearance (e.g., by highlighting, enlarging, bolding, etc.).

[0057]

[0080] The neural activity signals acquired during the presentation of one or more stimuli may include specific identifiable signature events or responses called control signals. Signature neural responses, or control signals as they may be called, are specific brain activity signals that may be related to the user's cognitive intentions, as described herein. Thus, the occurrence of signature brain activity responses or control signals in one or more brain regions during the presentation of a stimulus or tag flashing may indicate that the tag flashing is informing the user's intentions.

[0058]

[0081] Figure 5B shows two examples of neural activity signals that may be acquired during and after the presentation of two stimuli, with the stimuli presented at time 0 (e.g., the presentation of tag flashes 579A and 579B as described in relation to Figure 5A). For example, signal 573A may be a neural signal evoked and acquired after tag flash 579A, and signal 573B may be a neural signal acquired after tag flash 579B in Figure 5A. As shown in the figure, I is the target tag 585, which is part of tag flash 579A, and the neural signal 573A corresponding to tag flash 579A including the target tag 585 may include a signature neural response (e.g., event-related potential) indicated by a transient amplitude change of the signal, which is shown as a distinctive upward deflection of trace 573A around time points of 200 ms to 300 ms. Depending on the method of acquiring the signal, the control signal may be a change in any appropriate parameter, such as an upward or downward deflection of frequency. On the other hand, signal 573B corresponding to tag flashing 579B that does not include target tag 585 may lack any signature neural response. Neural activity signals may or may not include control signals or signature responses based on the task, the stimulus presented, and the brain region from which the signal is recorded. Figure 5C shows examples of signals recorded from three examples of brain regions while a stimulus is repeatedly presented. As illustrated, the presentation of a stimulus can be repeated once or multiple times, which can help increase the signal-to-noise ratio and improve the accuracy of determining the target tag. Repeated presentation of a stimulus can be used appropriately considering other requirements such as the speed of user interaction.

[0059] Feature extraction

[0082] In some embodiments of BCI systems 100, 300, or in some embodiments of processes 200 or 400 for implementing a BCI system, the signal acquired after stimulus presentation may be fully utilized to gather information about the user's intent. In other embodiments of BCI systems 100, 300, or in some embodiments of processes 200 or 400 for implementing a BCI system, one or more dimensionality reduction methods may be used to optimally utilize the information provided by the acquired signal. For example, an analytical procedure used to reveal the user's intent may include one or more steps of detecting and / or extracting features from the acquired signal as disclosed above. The signal features may include several parameters that represent the signal. In some example conditions, the features may also include components of the signal (e.g., principal components or independent components) or values ​​or vectors obtained using other similar dimensionality reduction methods. Some examples of features may include peak amplitude, duration, frequency bandwidth, mean deviation from a baseline, etc. One or more features may be specific to other particular parameters. For example, the features may include the peak amplitude 200ms to 500ms after stimulus presentation, or the peak amplitude of the frequency response within a specific frequency range.

[0060]

[0083] Figure 6A shows an example of a neural signal, illustrating feature 687A, which is the peak amplitude of the negative deviation of neural signal 673 at 400 ms after stimulus presentation. Another example of a feature could be 687B, which is the peak amplitude of the positive deviation at a point between 100 ms and 200 ms after stimulus presentation. Similarly, one or more features can be defined and used to distinguish between brain activity responses or neural signals induced by stimuli containing a target tag and those induced by stimuli without a target tag. For example, several stimuli or tag flashes can be presented, and simultaneous signals can be acquired. The BCI system can then perform one or more feature extraction routines on the acquired signals to extract one or more specific features, such as features 687A and 687B (feature 1 and feature 2) shown in Figure 6A. The extracted features derived from the signal can be considered as dimensions (e.g., Dim1 and Dim2) and used to evaluate the signal. Figure 6B shows an example of how extracted features 687A and 687B, extracted from signals acquired during the presentation of a series of repetitions of four stimulus or tag flashes (TF1, TF2, TF3, and TF4), are projected as Dim1 and Dim2. For example, tag flash TF2 may contain a target tag and register larger amplitudes for both features 687A and 687B plotted on axes Dim1 and Dim2, while tag flash TF3 may not contain a target tag and register smaller (or zero) amplitudes for both features 687A and 687B plotted on axes Dim1 and Dim2, respectively. As described below, one or more classifiers or interpreters in a BCI system (e.g., systems 100, 300) or a method for implementing a BCI system (e.g., processes 200, 400) can use their features and / or dimensions to appropriately classify and / or label signals and the stimuli (or tag flashes) that elicit signals.

[0061] Identifying target tags

[0084] As explained above, one of the objectives of a BCI system is to present a set of choices as stimuli and decode the user's intention to select a particular stimulus that can mediate a specific action from the neural signals of brain activity. A set of stimuli can be a set of visible tags, and a particular tag of interest to the user can be designated as the target tag. In other words, the objective of a BCI system may be to identify the identity of the target tag from a set of available visible tags with a certain degree of accuracy and confidence. The process of identifying the target tag can incorporate several sources of information, such as the prior likelihood that a particular tag will be presented and the likelihood that a particular tag can elicit a signature brain activity response.

[0062]

[0085] To perform this function, BCI systems 100 and 300 can carry out process 700 as shown in Figure 7. Process 700 may be part of or the same as or substantially similar to process 200 and / or process 400 described above. For example, process 700 may include one or more training sessions and / or test sessions with the presentation of training stimuli and / or new stimuli. Process 700 may also include acquiring and analyzing signals such as neural signals, oculomotor signals and / or peripheral signals. Accordingly, such similar parts and / or aspects will not be described in further detail herein.

[0063]

[0086] As shown in the flowchart of Figure 7, process 700 may include step 701 of presenting a stimulus (e.g., flashing of a tag or control item) and step 703 of acquiring various signals, including neural activity signals simultaneous with and / or after the stimulus presentation in step 701. For example, process 700 may include presenting the stimulus by a control interface. In some embodiments, the processor may be configured to present the stimulus by changing the appearance of a visual representation associated with the stimulus or tag (e.g., a control item), such as the size, color, orientation, brightness, thickness, or mobility of the visual representation. In step 703, process 700 may receive a set of user neural signals associated with the stimulus from a neural recording device after the stimulus presentation. Alternatively, process 700 may also include receiving a set of eye movement signals associated with the stimulus from a target tracking device. In step 705, the acquired signals (e.g., neural signals, eye movement signals) can be appropriately processed (e.g., preprocessing, filtering, feature extraction, etc.), including processing the neural signals as described above, to extract information related to a set of features from, for example, an EEG signal (e.g., amplitude of the response contained in the neural signal, duration of the response, shape of the response, timing of the response to the presentation of a stimulus from a set of stimuli, frequency associated with the neural signal, etc.). In step 707, the processed signals can be interpreted by applying any appropriate statistical model or mathematical structure to the signals. In step 707, the processed signals can also be classified, labeled, or scored based on a confidence scoring system used to calculate the likelihood that the signal contains a signature neural response or the likelihood that the neural signal contains a control signal. In step 709, the processed or labeled / scored signals can be associated with the stimulus or tag flash that evoked the signal. In other words, in step 709, the processed and analyzed results of the neural signal can be associated or linked to the tag flash that elicits a response in the signal.Steps 701-709 can be repeated to present various distinct stimuli or tag flashes, or to repeatedly present a single stimulus or tag flash (the signal and analysis results can be averaged by repeatedly presenting stimuli to increase the signal-to-noise ratio, SNR, as long as the conditions are comparable). In step 711, process 700 may include determining (e.g., calculating) the likelihood or score of each visible stimulus or tag (e.g., control item) being the target tag. Then, in step 713, the likelihood or score of all visible tags is evaluated, the tag with the highest likelihood or score is determined to be the target tag and returned for selection. As described above, identifying the target tag may also include considering the likelihood that a tag is the target tag or the likelihood that a tag will be presented. For example, if UI / UX is the spelling of an English word, the BCI system may include considering the likelihood that a particular letter in the English alphabet is the target tag or the probability that a particular letter will be presented. Since some letters (e.g., vowels) are used far more often in English than others (e.g., z, q, etc.), this consideration can be speedy and accurate in identifying the target tag. Identifying a target tag may also include and / or be related to determining the user's point of focus, which is associated with a stimulus or tag. Based on the tag and / or point of focus, the BCI system can determine and perform the user's intended action (e.g., activating or deactivating a tag or control item).

[0064] Score table

[0087] The UI / UX of BCI systems such as System 100, 300, etc., can present the user with a series or combination of stimuli or tags. Each tag may be associated with one or more actions that give the user control of a machine, device, and / or interface. At any given step, one (or more) of the tags may be a target tag that can bring about an action desired by the user upon selection. As described above, the objective is to identify the target tag among the presented tags or combination of tag groups from neural signals (and other relevant signals such as oculomotor signals or peripheral signals).

[0065]

[0088] Identifying target tags can be achieved using any suitable method for analyzing the neural signals evoked by each presented stimulus or tag (or group of tags). One example of such a method is to calculate the likelihood that each visible tag is a target tag for all possible visible tags. Each visible tag with a calculated likelihood can also be associated with a score according to any suitable scoring scheme. Thus, all visible tags can have scores that form a score table, which can be evaluated so that the visible tag with the highest score is identified as the target tag, as will be described in more detail below.

[0066]

[0089] Figure 8A shows some examples of analytical methods that can be used to calculate the likelihood that a visible tag in the UI / UX of a BCI system is a target tag of interest to the user in a given example. The likelihood may relate to a score that ranks the presented stimulus or tag, such as a confidence score. For example, a user may be presented with P tags known to elicit specific target responses and specific non-target responses when presented as flashing tags in an appropriate combination as a tag group (t1, t2, etc.). The BCI system scores the stimulus-response elicited during the presentation of the flashing tag (or tag group) t (y t A feature vector can be generated that contains one or more features that can be used for the score y. tcan be obtained based on a likelihood metric calculated using prior knowledge of the response to a known stimulus. For example, for a given stimulus response x to a stimulus or tag dot pattern t that includes a control signal or a signature response signal (e.g., ERP or P300) t the likelihood can be calculated by an appropriate analysis method based on whether the tag dot pattern included the target tag. Some examples of methods, such as 891 and 893, are shown in FIG. 8A. As shown in the example of the distribution shown in FIG. 8B, using the scores of the stimulus responses of the target tag and the non-target tag, the mean (μ a ) and variance (σ a 2 ) of the target response, and the mean (μ n ) ) and variance (σ n 2 ) of the non-target response, a distribution can be calculated.

[0067]

[0090] The probability that a particular tag elicits a signature response when presented as a stimulus can be calculated using any suitable method. For example, as shown in FIGS. 8A and 8B, by analyzing the neural signals elicited for each presentation of the tag, with or without comparison to known signature responses from a training dataset, probability metrics can be generated using equations such as 891 and 893 to calculate probability metrics. The probability metrics resulting from the analysis can be used to generate a confidence score 895. The confidence scores corresponding to all available (visible and invisible) tags, including the presented tags and the non-presented tags, can be a distribution of scores. Two examples of score distributions are shown in FIG. 8B. The null distribution centered around the zero score corresponds to known tags that do not elicit a signature response (the stimulus response of non-target tags with mean (μ n = 0) and variance (σ n 2 )). The distribution of samples is that of the scores resulting from potential target tags that may have elicited a signature response, with mean (μ a ) and variance (σ a 2 ). The separation and / or overlap between the distributions can depend on factors related to the stimulus, the user, and / or the characteristics of the UI / UX.

[0068]

[0091] In some embodiments, any suitable method can be used to distinguish whether a particular score (e.g., score 895 with values ​​of 0.3 and 0.9) belongs to a null distribution or a sample distribution. For example, a threshold score value (e.g., score = 0.1) can be used as a criterion 897 to help determine whether a tag is a target tag. In some other embodiments, the scores from all tags do not need to be classified but can be compared to one another, and the tag with the highest score can be selected as the target tag. In some embodiments, neural responses may be fed through an ensemble of classifiers developed to fit specific characteristics of the neural responses, and confidence scores can be generated using the output of the ensemble of classifiers, as shown in the example method in Figure 9 and described below.

[0069]

[0092] In some embodiments, training data can be pre-collected and analyzed to set meaningful thresholds or criteria that, for example, can indicate that a signature response has been induced if a previously acquired signal satisfies them. For example, responses induced by stimuli known to elicit a specific signature response (e.g., a P300 response) can be used to compare responses originating from an unknown or novel stimulus. One or more criteria can be set on various parameters to register a signature stimulus. For example, criteria can be set for parameters such as amplitude, frequency, brain region, onset latency, response duration, and response shape. A distribution can be made up of one or more such parameters derived from known responses induced by known stimuli that elicit a signature or control response. One or more parameters derived from a new unknown response induced by a novel stimulus can be compared to their distribution to determine whether the novel response contains one or more signature responses. For example, the amplitude parameter of an example response to a novel stimulus can be compared to the distribution, mean, and variance of amplitude parameters derived from a known control response, such as a P300 signal. Based on where the parameters fall compared to the distribution, mean, and variance of known P300 signals, a confidence score can be assigned to the response to a new stimulus or tag to determine whether the P300 signal qualifies as a signature or control brain signal. Such confidence scores (e.g., P300 scores) can be calculated for all new responses to new or unknown stimuli and compiled, for example, in a score table.

[0070]

[0093] The confidence score calculated for each neural response can be associated with the corresponding stimulus or tag that elicited the response. In some embodiments, the score can be calculated for each response and averaged over responses elicited by the same stimulus (under appropriately similar conditions) to achieve better signal-to-noise considerations. In some embodiments, the score for individual stimulus-response pairs can be obtained from parameters that are compared alone with a prior or expected distribution of the parameters. In some embodiments, the prior distribution of scores can also be generated to compare the calculated scores with an expected score distribution. By comparing the confidence scores associated with various presented stimuli or tags, which are compiled in a score table, the BCI system can be adapted to find the stimulus or tag that elicited the response with the highest confidence score. Furthermore, tags can be grouped during presentation, and the grouping can be modified to facilitate the detection of individual tags within a tag group that is the target tag that elicited the response with the highest confidence score.

[0071]

[0094] In some embodiments, one or more analytical methods can be used to classify or label signals. The analytical methods can then be evaluated for their merit based on one or more performance parameters. The results of one or more analytical methods can then be combined to generate a score table. For example, neural activity signals can be processed by several classifiers using several statistical models, and each processed classification can be evaluated based on the accuracy of the classification. An ensemble of classifiers can then be selected to be used together to form a composite classification score supplied into the score table.

[0072]

[0095] In some embodiments, a scoring scheme based on various other variables can be employed. For example, the scoring scheme may be based on the number of available visible tags, the number of known stimuli or tags that have elicited a signature response (e.g., known P300 stimuli), the degree of difference between various tags that may elicit a signature response, etc. For example, the scoring scheme may be in a range of -1 to +1 passing through 0, with stimuli or tags with a high likelihood of eliciting a signature response (e.g., P300) having a score close to +1, stimuli with the lowest likelihood of eliciting a signature response having a score close to -1, and intermediate stimuli with ambiguous responses having a score close to 0.

[0073] Ensemble of analytical methods: Example - Melange

[0096] As described above, in some embodiments, two or more analysis methods can be used to generate a score table, and these methods are evaluated based on one or more performance parameters. For example, neural activity signals can be processed by several classifiers, each classifier being evaluated against the others. An ensemble of classifiers can then be used together to form a composite classification score to be fed into the score table. The score table can be updated using various other information sources (e.g., stimulus information, information from other signal sources such as oculomotor signals, etc.). Figure 9 shows an example of a method using three or more different classifiers using various feature vectors and classification schemes. As shown in the example plot in Figure 9, the labeling from each classifier is evaluated. The best N classifiers are selected (N is a predetermined number or at the user's discretion, etc.), and an ensemble classifier or "melange" is generated. By analyzing the neural signal using the melange, a composite score table is generated. This composite score table can be updated using other information sources (e.g., stimulus information, eye movement information, etc.). For example, a composite score table can be supplied to be updated by a distance table, which incorporates the effect of the tag's proximity on responses induced to other nearby tags, as will be explained in more detail below.

[0074]

[0097] In some embodiments, the classifier uses a score table (including, for example, one or more score datasets) and can be configured during a training phase, for example, as shown in Figure 4. For example, the method may include presenting stimuli to the user by a control interface, where the stimuli include tags (e.g., control items) associated with actions. The method may include receiving a set of inputs related to the user's behavior from a target tracking device and a neural recording device. The method may include generating a score table based on a set of inputs and information related to the stimuli, and receiving information related to the actions the user intends to take (e.g., information indicating the actions the user intends to take). A set of classifiers can then be configured using the score table and the information related to the actions the user intends to take, so that the score table is associated with the actions the user intends to take and so that (e.g., in a later period) the user can be a later user to predict or reveal the actions the user intends to take according to, for example, Method 700 shown in Figure 7 or other methods described herein.

[0075]

[0098] In some embodiments, this method may further include modifying a set of classifiers to generate a modified set of classifiers, based on evaluating the accuracy of the action to be determined using a set of classifiers. In addition, or alternatively, this method may include generating a set of weights to be applied to inputs received from one or more target tracking devices or neural recording devices, based on evaluating the accuracy of the action to be determined. The weights may relate to the accuracy of the action to be determined, the user experience, and historical information related to the user. This method then includes presenting stimuli to the user by a control interface in a later period, and receiving a set of inputs from the target tracking devices and neural recording devices relating to the user's behavior in a later period. This method may include generating a score table or score dataset based on a set of inputs and information relating to the stimuli presented in a later period, and optionally applying a set of weights to the scores in the score table. This method may also include using the modified set of classifiers to determine the action the user intends to take in a later period.

[0076] Use of stimulus information: Example - Distance scaling

[0099] In some embodiments of BCI systems 100, 300, and / or processes 200, 400, and / or 700 for implementing the BCI system, the accuracy of target tag identification can be improved by using available information regarding how stimuli were presented. For example, with respect to process 700, along with the likelihood and confidence scores calculated from the neural signals described above, information such as the spatial arrangement of stimuli or tag flashes presented by the UI / UX, the time order or time series of tag flashes, the grouping of tags within a tag group, and the degree of prominence associated with tag flashes can be used.

[0077]

[0100] One example of how stimulus information can be used is described herein in the form of distance scaling for generating what we call a distance table. For example, in some embodiments of BCI systems that present UI / UX through a visual interface, the physical distance between tags presented as stimuli can be used to better estimate the likelihood that a particular tag is a target tag. In other words, the Euclidean distance between presented tags in the display can be used to update the confidence score in a score table calculated for each visible tag based on an analysis of the response evoked by that stimulus.

[0078]

[0101] If a particular tag is a target tag (e.g., the letter A shown in Figure 10A), then when that target tag (e.g., A) is presented alone or within a group of tags (e.g., a presentation of a column or row containing A), it may evoke a signature response or control signal that can be recorded in the neural activity signal. After the step of calculating the likelihood and / or score associated with all tags, this particular tag (A) may be associated with a high confidence score in the score table (indicated by a red circle). In some embodiments in particular, even without the presentation of the target tag (e.g., A), presenting a tag near this target tag (e.g., the letters B, G, H, M, C that are spatially close to A) may also evoke a neural response that may be similar to a control signal that yields a high score. In other words, some visible tags that are not the target tag may also evoke signals similar to control signals that produce a high confidence score due to their proximity to the target tag when presented. Figure 10B shows an example plot of signal amplitudes evoked by the presentation of the tags shown in Figure 10C, depending on their proximity to the target tag A.

[0079]

[0102] Signals induced by tags close to the target tag may not be as prominent as signals induced by the target tag itself, but they may still be significant enough to meet certain criteria for analysis or exceed certain thresholds. However, due to various reasons such as temporal proximity, distraction, and random spurious signals, signals from nearby tags can be utilized in the process of identifying the target tag to distinguish and / or differentiate it from tags that may have relatively high confidence scores within the score table.

[0080]

[0103] For example, in some cases, two tags, such as the letter A and the hyphen, may elicit a signature response signal, e.g., P300, and produce a high score when presented together in a flashing sequence or when presented in close chronological order. One example of a BCI system can correctly identify a target tag as the letter A by using information about the spatial arrangement of all visible tags, along with stimulus-response information of previously presented tags. For example, a BCI system can correctly identify the target tag as A and not the hyphen by comparing responses to the presentation of various tags, including letters close to A (e.g., B, G, H) which may produce high scores due to their proximity to A (indicated by circles around the letters B, G, and H), and letters close to the hyphen (e.g., the letters 3, 4, and 9) which are distal to A but produce high scores (indicated by circles around the numbers 3, 4, and 9) due to their distance from the target tag A. The example in Figure 10A shows how the spatial relationship of tags can be used to address ties in scores derived from two or more tags and / or to eliminate ambiguity in such scores, but the temporal relationship of tags at presentation can be used similarly to address tags with ties in responses and / or to eliminate ambiguity in such tags. Figure 11 shows an example of process 1100 that uses signals induced by proximity tags to update scores in a score table.

[0081]

[0104] Process 1100 may be the same as or similar to Process 700 shown in Figure 7. For example, Process 1100 may include a step 1101 of presenting a stimulus or tag flashing using one or more tags, and a step 1103 of recording brain activity signals. Process 1100 may also include a step 1103 of processing one or more brain activity signals, a step 1105 of classifying and / or scoring the signals, and a step 1107 of calculating likelihood. Process 1100 may also include a step 1109 of associating each visible tag with a score and generating a score table. In addition, after each tag flash, Process 1100 may include a step 1111 of calculating one or more distance scores for each visible score based on the proximity of that visible tag to one or more tags that were flashed in a particular example of the stimulus or tag flashing.

[0082]

[0105] In other words, after each tag flashes, step 1111 includes calculating a score (e.g., distance score) for each of all available tags based on its proximity to each of the flashing tags, and generating a distance table. The distance scores in the distance table can be used in step 1113 to update the confidence scores in the score table. Then in step 1115, the updated score table can be evaluated to obtain the tag associated with the highest score, and that tag can be determined to be the target tag.

[0083]

[0106] Figure 12 shows an example of how to calculate distance scores to generate a distance table. For example, tags 1-5 can be a set of available tags, and the user wants to select tag 1. Therefore, tag 1 is the target tag. In a given instance of UI / UX, tags 1 and 5 can be presented together as a grouped tag flashing. Each tag in a set of available tags can have a prior probability of being presented. For example, if there are 5 tags and the probability of each being presented is equal, each of the 5 tags may have a presentation probability of 1 / 5 (0.2). Similarly, two or more tags in a set of available tags can have a prior probability of being selected together. For example, as shown in Figure 12, tags 1 and 5, which flash together in the tag group (1,5), may each have a prior probability of being presented, which is given by normalizing the individual probabilities of those tags by the sum of their probabilities.

[0084]

[0107] As shown in Figure 12, a distance measure can be calculated for each tag from all other tags. For example, the distance from tag 1 to all other available tags 1-5 is measured d. i1 This can be calculated using (i=1,2,3,4,5). After calculating the prior probability and distance measure, a likelihood measure of the tag being the target can be assigned to each visible tag (1,2,3,4,5), scaled by the distance measure from each tag flash. In other words, a prior likelihood score of being the target tag can be assigned to each visible tag regardless of which tag is flashed, and for each tag flash, the likelihood score of each visible tag can be updated using an appropriate distance scaling from the last tag flash or the tags in the tag group flash. For example, when flashing a tag group (consisting of tags 1 and 5), the likelihood that tag x (e.g., tag 3) is the target tag is d' x,g (For example, d' 3,g The scoring is given by the sum of the presentation probabilities (p'1 and p'5) of the tags in the flashing group (1 and 5), and the corresponding distance (d) from the tag (3) being scored to that tag (1 or 5). 3,1 or d 3,5Each presentation probability (p'1 or p'5) is scaled by ). As shown in the example in Figure 12, a distance-based likelihood score or distance score can thus be assigned to all visible tags from the information regarding the presentation of each tag. The set of all distance scores in a distance table (e.g., a P300 distance table) can be supplied to update a score table (e.g., a P300 score table) generated by analyzing neural activity signals and calculating confidence scores as described above.

[0085]

[0108] Figure 13 shows an example of another set of steps followed by a process similar to process 1110, where an updated score table with scores s1, s2, s3, etc., can be formed by combining confidence scores from the score table and distance scores from the distance table by analyzing neural activity signals (e.g., using a classifier), and the updated score table can be evaluated to identify the tag or stimulus with the highest score as the target tag. As shown in Figure 13, generating the score table with confidence scores, updating the score table with distance scores, and evaluating the updated score table can be done after each stimulus or tag flashes. If a stimulus with a unique tag can be identified as the target tag based on having the highest score and the highest score meeting one or more threshold criteria, the identified target tag can be returned to be selected, the instance of identifying the target tag can be terminated, and the score table can be cleared.

[0086]

[0109] However, if no tag has a score that meets one or more threshold criteria (including the tag with the highest score in the score table), and therefore no tag can be identified as a target tag, the result may indicate insufficient data, which may lead to another series of new tag flashes, and the subsequent calculation of confidence scores, distance scores, and updating of the score table, with the UI / UX moving forward to incorporate data from the latest tag flashes. The updated score table can then be re-evaluated for the highest score meeting the threshold criteria. This series of new tag flashes and the subsequent calculation of confidence and distance scores to update the score table can be repeated until the score table reaches a state where at least one tag has a score that meets the threshold criteria for being identified as a target tag.

[0087] Use of eye movement signals: Example - Visual scoretable

[0110] In some embodiments of a BCI system (e.g., systems 100, 300), or processes for implementing a BCI system (e.g., processes 200, 400, 700, 1100, and / or 1400), the system may be configured to incorporate eye movement information available from one or more eye trackers. That is, information derived from one or more eye-movement signals can be used to update a score table to identify a target tag from a set of available tags.

[0088]

[0111] Figure 14 shows a flowchart outlining process 1400. Process 1400 may be the same as or nearly the same as processes 200, 400, 700, and / or 1100. For example, process 1400 may include step 1401, which presents a stimulus or tag flashing using one or more tags. Process 1400 may include step 1403, which records brain activity signals and eye movement signals. In steps 1405 and 1407, the acquired brain activity signals and eye movement signals can be processed, and appropriate statistical models can be applied, respectively. Then, using one or more appropriate scoring schemes, tags can be assigned to scores or tags can be associated to scores based on the analysis of the brain activity signals. In step 1407, based on one or more scores, a likelihood metric can be calculated in which each tag is a target tag.

[0089]

[0112] In addition, process 1400 may include step 1409 for calculating the oculomotor score associated with each tag. After calculations in steps 1409 and 1411, the score table in step 1407 can be updated with the distance scores from the distance table and / or the oculomotor scores from the visual score table. The updated score table can be evaluated in step 1415 to identify the target tag. The vision table can be generated by calculating the oculomotor score using any method suitable for incorporating eye movement information acquired simultaneously with the presentation of stimuli and the acquisition of neural activity signals. Figure 15 shows an example of a procedure used in some embodiments for calculating the oculomotor score and generating a visual score table.

[0090]

[0113] As shown in Figure 15, the target tracking signals acquired in response to the acquisition of neural activity signals (e.g., EEG signals) can be preprocessed using any suitable analysis method. For example, the target tracking signals can be filtered (e.g., using Kalman filtering techniques) and subjected to saccade detection and / or removal routines. Subsequently, in some embodiments using a 3D UI / UX, the eye movement signals corresponding to the detected saccades can be converted from those mapped in 2D space to those mapped in 3D space. In some embodiments, the target tracking signals can be acquired and analyzed using substantially the same or identical instruments and / or methods as those described in Application No. 253, the full disclosure of which is incorporated herein by reference above, and / or Application No. 209, the full disclosure of which is incorporated herein by reference above.

[0091]

[0114] In some embodiments where the UI / UX is designed to be effectively two-dimensional, eye movement signals corresponding to saccades are preserved within a two-dimensional mapping.

[0092]

[0115] After mapping, one or more gaze vectors can be calculated to generate estimates of the degree and direction of the user's gaze. The calculated gaze vectors may have mean estimates of amplitude and direction, as well as variance of the gaze angle. The BCI system or the process of implementing the BCI system may include calculating the ambiguous boundary of the gaze angle or visual angle around each visible tag of a given set of available tags that the user can see using it. The BCI system can build and / or update a visual acuity model using the kinematics of eye movements and information about the user's eye movements to generate predicted gaze vectors 189 shown in Figure 15 using the UI / UX 1571. The BCI system 1500 in the example of Figure 15 can incorporate the expected visual angle from the visual acuity model along with other available user data (e.g., the user's interpupillary distance, eye tracker manufacturer and model, etc.). A visual score table can be generated using the combination of the analyzed visual acuity model, saccadic eye movement signals, and results derived from the gaze vectors, and each visible tag is assigned an oculomotor score based on its proximity to the calculated gaze vectors 1589. As shown in Figure 15, a visual score table with scores v1, v2, v3, etc., can be used to update the score table calculated from the confidence score (and / or updated using the distance score).

[0093] Using sensory information to update the score table

[0116] In some embodiments of a BCI system (e.g., systems 100, 300), or processes for implementing a BCI system (e.g., processes 200, 400, 700, 1100, 1400, and / or 1600), information can be incorporated from any number of sensors that acquire biological (or non-biological) data. For example, information from one or more physiological signals, behavioral signals, or external signals (such as perturbations or events in the environment in which the user is currently placed) can be used to update a composite score table to identify a target tag from a set of available tags.

[0094]

[0117] Figure 16 shows a flowchart outlining an example of process 1600 for integrating data from various sensors to form a composite score table. Process 1600 may be the same as or nearly the same as processes 200, 400, 700, 1100, and / or 1400. For example, process 1600 may include step 1601 for presenting stimuli or tag flashing using one or more tags. Process 1600 may include step 1603 for recording brain activity signals and co-occurring signals from an array of various sensors. In steps 1605 and 1607, the acquired brain activity signals and signals from various sensors can be processed using appropriate weighting vectors for each signal stream, and appropriate statistical models can be applied separately or together as an ensemble. Using one or more appropriate scoring schemes, likelihood scores can be calculated in step 1609 based on the analysis of brain activity signals and the analysis of signals from each of the various sensors, and assigned or associated with tags in step 1611. In other words, for each tag flash, in addition to the tag score obtained by analyzing the brain activity signal that updates the main score table, all signal streams from each of the sensors (1,2,...X) can be associated with scores (S1,S2,...SX) that are updated into, for example, the sensor (1,2,...X) score table. Process 1600 may include step 1613, which updates the main score table with the scores from the score tables corresponding to each of the sensors (1,2,...X) that generate a composite score table. In step 1617, process 1600 may return the tag with the highest score in the composite score table as the target tag.

[0095] Using the Master Score Table

[0118] Figure 17 shows an example of a method 1700 for generating a master score table that incorporates several sources of information to help identify target tags. For example, as shown in Figure 17, several sources of information can be used to generate score sets, and various score sets can be appropriately combined to update the identification of target tags.

[0096]

[0119] In some embodiments, various scores from various sources can be further analyzed by supplying them through an ensemble of classifiers. These various scores may include, for example, a confidence score obtained by analyzing neural responses (e.g., the P300 score shown in Figure 17), a distance score generated by using information about the spatial relationships between tags, a time score obtained by using information about the time series presenting the tags, a vision score obtained by using information about the position of the user's eyes and / or head (e.g., by analyzing epochs in eye movement tracking data or head movement tracking data), and other sensory scores obtained by using information about other sensory parameters (e.g., voice commands, head movements, gestures, etc.) obtained from sensors worn by the user or placed in the user's environment (e.g., sensor X). As shown in Method 1700 in Figure 17, one or more of these scores from their respective score tables corresponding to each available tag can be combined to form an ensemble score dataset.

[0097]

[0120] In some embodiments, the scores in a score table can be supplied via an ensemble of classifiers, such as the example of an ensemble classifier shown in Figure 9. The ensemble of classifiers can be used together to form a composite classification score supplied into a master score table. As shown in Figure 9, for example, confidence scores are best classified by classifier 1, distance scores by classifier 2, and so on, with each score set having a corresponding pair of the best N classifiers. The best N classifiers are then selected (where N is a predetermined number or at the user's discretion) to generate an ensemble classifier (e.g., a "melange"). The composite ensemble classifier (e.g., a melange) can appropriately use the best N classifiers for a particular dataset containing the scores in each score table corresponding to each information source. In some embodiments, the composite score can be weighted based on its classification.

[0098]

[0121] In some embodiments of a BCI system using a method similar to Method 1700, weighting scores from several score tables (with or without the use of an ensemble classifier) ​​can be based on how much information each information source can provide. As shown in Figure 17, in some embodiments, the composite weighted score set (with or without ensemble classification) can be weighted again according to parameters such as the accuracy of training set selection and user performance during training, in some examples. In some embodiments, the composite score set can also be appropriately weighted according to appropriate information sources such as user history, experience level, UI / UX history, and user statistics. The resulting composite weighted score set of all visible tags can be used to generate a master score table, as shown in Figure 17. This master score table can be evaluated using appropriate techniques such as threshold cross-criteria to identify the tag with the highest score as the target tag. This master score table can be updated using multiple presentations of similar or different tags or tag group flashings, improving the accuracy of estimations or obtaining repeatable target tag selection while evaluating the master score table each time.

[0099] conclusion

[0122] In summary, this specification describes a system and method for use in implementing an integrated brain-computer interface that can be operated in real time by a user. The disclosed system includes an eye-movement tracking system for implementing pointing control functions and a brain activity tracking system for implementing action control functions. Both functions are implemented through the presentation of a UI / UX strategically designed to enable high-speed and accurate operation. In addition, the disclosed system and method are configured to be hardware-independent for implementing a real-time BCI on any suitable platform to mediate user operations in virtual, extended, or real environments. Figure 11 shows an illustrative diagram of the usage space of the BCI system and method of the present invention.

[0100]

[0123] While various embodiments have been described above, it should be understood that these are merely examples and not limitations. The methods described above demonstrate that specific events occur in a specific order, but the order of these events can be modified. In addition, certain events can be executed simultaneously by parallel processes if possible, and furthermore, they can be executed sequentially as described above.

[0101]

[0124] The schematic diagrams and / or embodiments described above show that certain components are arranged in a particular orientation or position, but the arrangement of components can be modified. Although embodiments have been illustrated and described individually, it will be understood that various changes can be made in terms of form and detail. Any part of the apparatus and / or methods described herein can be combined in any combination except mutually exclusive combinations. Embodiments described herein may include various combinations and / or subcombinations of the functions, components, and / or features of the various embodiments described.

Claims

1. A display configured to present a control interface to a user, wherein the control interface includes a plurality of control items, each related to an action, A neural recording device configured to record neural signals related to the user, An interface device operably coupled to the display and the neural recording device, memory, and The memory is operably coupled to the aforementioned memory. The control interface presents a set of stimuli individually, wherein each stimulus in the set of stimuli includes a set of control items from the plurality of control items. After presenting each stimulus of the aforementioned set of stimuli, a set of nerve signals associated with that stimulus is received from the nerve recording device. Based on the set of nerve signals for each of the set of stimuli, a score related to each of the multiple control items is determined. The user's area of ​​interest is determined based on the score associated with each of the plurality of control items, wherein the area of ​​interest is associated with at least one of the plurality of control items, and Based on the aforementioned points of interest, the user's intended action is determined. A processor configured to perform Interface device and Includes, Each control item in the aforementioned set of control items is related to a visual representation, The processor is configured to present each of the presented set of stimuli by changing the appearance of the visual representation associated with each control item in the set of control items included in that stimulus. A device in which changing the appearance includes changing at least one of the size, color, hue, texture, contour, orientation, brightness, thickness, or mobility of the visual representation.

2. The aforementioned nerve signals include an electroencephalogram (EEG) signal that includes at least one of event-related potentials (ERPs), motor image signals, steady-state visual evoked potentials (SSVEPs), transient visual evoked potentials (TVEPs), brain state commands, visual evoked potentials (VEPs), P300 evoked potentials, sensory evoked potentials, motor evoked potentials, sensorimotor rhythms such as Mu-rhythms or Beta-rhythms, event-related desynchronization (ERDs), event-related synchronization (ERSs), slow brain potentials (SCPs), or brain state-dependent signals. The processor is further configured to process the set of nerve signals for each stimulus in the set of stimuli and to extract information related to a set of features from the EEG signal. The processor is configured to determine the score associated with each of the plurality of control items using the information associated with the set of features, The apparatus according to claim 1.

3. The apparatus according to claim 2, wherein the set of features includes at least one of the amplitude of the response included in the nerve signal, the duration of the response, the shape of the response, the timing of the response to presenting the stimulus from the set of stimuli, or the frequency associated with the nerve signal.

4. The device further includes a target tracking device configured to record eye movement signals related to the user, The processor is further configured to receive a set of eye movement signals associated with each stimulus from the target tracking device after presenting each stimulus of the set of stimuli. The processor is configured to determine the score associated with each of the plurality of control items based on the set of nerve signals and the set of eye movement signals associated with each of the set of stimuli. The apparatus according to claim 1.

5. The device according to claim 1, wherein the processor is configured to determine the score associated with each of the plurality of control items by calculating a likelihood estimate for each of the plurality of control items, and the likelihood estimate for each of the plurality of control items indicates the likelihood that the control item is associated with the user's point of interest.

6. The device according to claim 1, wherein the processor is further configured to perform the action intended by the user, the action being at least one of activating or deactivating a control item from the plurality of control items.

7. The aforementioned point of interest is the first point of interest during the first period, the action is the first action, and the processor is Determining a second area of ​​interest for the user during a second period following the first period, wherein the second area of ​​interest is related to at least one control item of the plurality of control items. The determination of a second action intended by the user based on the second point of interest, wherein the second action is different from the first action, and After performing the first action, the user performs the second action as intended. The apparatus according to claim 6, further configured to perform the following:

8. The processor is further configured to classify the set of nerve signals associated with each stimulus of the set of stimuli according to at least one classification scheme using a set of statistical models, The processor is configured to determine the score associated with each of the plurality of control items based on the classification of the set of nerve signals associated with each of the set of stimuli. The apparatus according to claim 1.

9. A non-temporary processor-readable medium for storing code representing instructions executed by a processor, wherein the instructions are: To generate a control interface configured to be operated by a user to perform a set of actions. Presenting a stimulus to the user via the control interface, wherein the stimulus includes a set of control items, and each control item in the set of control items is related to at least one action of the set of actions. After presenting the stimulus to the user, information related to the user is received from the neural recording device. Based on the information received from the neural recording device, determine the score associated with each control item of the set of control items. To form a set of control items that determine the user's area of ​​focus based on the score associated with each control item, Identifying at least one control item from the set of control items related to the user's area of ​​interest. This includes code to cause the processor to perform the following: The stimulus includes a set of visual representations related to the set of control items, and each visual representation of the set of visual representations is related to at least one control item of the set of control items and is positioned within the control interface at a different location from the visual representations of each other control item of the set of control items. The code for causing the processor to determine the score associated with each control item of the set of control items includes code for causing the processor to calculate a set of distance scores associated with the set of control items based on the position of each visual representation of the set of visual representations, A non-temporary processor-readable medium in which the code for causing the processor to determine the user's point of interest includes code for causing the processor to determine the user's point of interest based at least in part on a set of distance scores.

10. The information received from the neural recording device includes neural signals related to the user, and the neural signals include electroencephalogram (EEG) signals that include at least one of event-related potentials (ERPs), motor image signals, steady-state visual evoked potentials (SSVEPs), transient visual evoked potentials (TVEPs), brain state commands, visual evoked potentials (VEPs), P300 evoked potentials, sensory evoked potentials, motor evoked potentials, sensorimotor rhythms such as Mu-rhythms or Beta-rhythms, event-related desynchronization (ERDs), event-related synchronization (ERSs), slow brain potentials (SCPs), or brain state-dependent signals. The instruction further includes code for causing the processor to process the neural signal and extract information related to a set of features from the EEG signal, The code for causing the process to determine the score associated with each control item of the set of control items includes code for causing the processor to determine the score associated with each control item of the set of control items using the information associated with the set of features, The non-temporary processor-readable medium according to claim 9.

11. The code for causing the processor to form the set of control items that determine each control item associated with the score is, The process involves calculating a likelihood estimate for each control item of the set of control items, wherein the likelihood estimate for each control item of the set of control items is such that the likelihood of the control item being related to the user's area of ​​interest is calculated, and For each control item in the set of control items, a set of scores is determined based on the likelihood estimation of that control item. This includes code to cause the processor to perform the following: The code for causing the processor to determine the user's area of ​​interest includes code for causing the processor to determine the user's area of ​​interest based on the set of scores for each of the set of control items. The non-temporary processor-readable medium according to claim 9.

12. A non-temporary processor-readable medium for storing code representing an instruction to be executed by a processor, wherein the instruction is: To generate a control interface configured to be operated by a user to perform a set of actions. Presenting a stimulus to the user via the control interface, wherein the stimulus includes a set of control items, and each control item in the set of control items is related to at least one action of the set of actions. After presenting the stimulus to the user, information related to the user is received from the neural recording device. Based on the information received from the neural recording device, determine the score associated with each control item of the set of control items. To form a set of control items that determine the user's area of ​​focus based on the score associated with each control item, Identifying at least one control item from the set of control items related to the user's area of ​​interest. This includes code to cause the processor to perform the following: The stimulus includes a set of visual representations relating to the set of control items, each visual representation of the set relating to at least one of the set of control items, and is configured to be presented within the control interface at a different time than when the visual representations of each other control item of the set of control items are presented. The code for causing the processor to determine the score associated with each control item of the set of control items includes code for causing the processor to calculate a set of time scores associated with the set of control items based on the time at which each of the set of visual representations is presented. A non-temporary processor-readable medium in which the code for causing the processor to determine the user's point of interest includes code for causing the processor to determine the user's point of interest based at least in part on a set of time scores.

13. The aforementioned instruction, Receiving information indicating the user's eye movement signals from the target tracking device, and Based on the information received from the visual target tracking device, a set of oculomotor scores related to the set of control items is determined. The code further includes a code to cause the processor to perform the following: The code for causing the processor to determine the user's point of interest further includes code for causing the processor to determine the user's point of interest based on a set of eye-movement scores, The non-temporary processor-readable medium according to claim 9.

14. A non-temporary processor-readable medium for storing code representing an instruction to be executed by a processor, wherein the instruction is: To generate a control interface configured to be operated by a user to perform a set of actions. Presenting a stimulus to the user via the control interface, wherein the stimulus includes a set of control items, and each control item in the set of control items is related to at least one action of the set of actions. After presenting the stimulus to the user, information related to the user is received from the neural recording device. Based on the information received from the neural recording device, determine the score associated with each control item of the set of control items. To form a set of control items that determine the user's area of ​​focus based on the score associated with each control item, Identifying at least one control item from the set of control items related to the user's area of ​​interest. This includes code to cause the processor to perform the following: The stimulus includes a set of visual representations related to the set of control items, and each visual representation of the set of visual representations is related to at least one of the set of control items. So that the control interface is positioned at a different location from the visual representation of each other control item of the set of control items, and The control interface is configured to present the visual representation of each of the other control items in the set of control items at a different time than when the visual representation of each of the other control items in the set of control items is presented. The code for causing the processor to determine the score associated with each control item in the set of control items is: Based on the position of each visual representation in the set of visual representations, calculate a set of distance scores related to the set of control items. Based on the time point at which each of the set of visual representations is presented, calculate a set of time scores related to the set of control items, and Calculating a set of kinetic scores related to the set of control items based on information received from the visual target tracking device. This includes code to cause the processor to perform the following: A non-temporary processor-readable medium in which the code for causing the processor to determine the user's point of interest includes code for causing the processor to determine the user's point of interest based at least in part on a weighted average of a set of distance scores, a set of time scores, and a set of eye movement scores.

15. The code for causing the processor to determine the score associated with each control item in the set of control items is: Selecting one set of classifiers from the multiple classifiers based on evaluating a set of performance parameters associated with multiple classifiers, To generate an ensemble classifier using the aforementioned set of classifiers, and Analyzing the information received from the neural recording device using the ensemble classifier to generate a set of scores. This includes code to cause the processor to perform the following: The code for causing the processor to determine the user's area of ​​interest includes code for causing the processor to determine the user's area of ​​interest based at least in part on the set of scores, The non-temporary processor-readable medium according to claim 10.

Images