Brain-Computer Interface

Abstract

A system and method involving a brain-computer interface in which visual stimuli are presented in direct association with real-world objects such that the user's intent regarding objects in the real world can be inferred without the intervention of a screen or other display device.

Description

[0001] Cross - reference to related applications

[0002] This application claims the benefit of priority of U.S. Provisional Patent Application No. 62 / 938,112, entitled "BRAIN - COMPUTER INTERFACE", filed on November 20, 2019, the entire content of which is incorporated herein by reference.

[0003] Technical field of the invention

[0004] The present invention relates to the control of real - world objects via a brain - computer interface involving visual sensing. Background art

[0005] In a visual brain - computer interface (BCI), typically among multiple generated visual stimuli presented to a user, the neural response to a target stimulus is used to infer (or "decode") which stimulus is substantially the object of attention at any given time. Then, the object of attention can be associated with an action that is selectable or controllable by the user.

[0006] Neural responses can be obtained using a variety of known techniques. A convenient method relies on surface electroencephalogram (EEG), which is non - invasive, has fine - grained temporal resolution, and is based on a well - known empirical basis. Surface EEG enables the real - time measurement of changes in the spread potential on the surface of a subject's skull (i.e., the scalp). These potential changes are commonly referred to as electroencephalogram signals or EEG signals.

[0007] In a typical BCI, visual stimuli are presented in a display generated by a display device. Examples of suitable display devices (some of which are shown in Figure 3 include a television screen and a computer monitor 302, a projector 310, a virtual reality headset 306, an interactive whiteboard, and the display screens of a tablet 304, a smart phone, smart glasses 308, etc. The visual stimuli 311, 311’, 312, 312’, 314, 314’, 316 can form part of a generated graphical user interface (GUI), or they can be presented as augmented reality (AR) or mixed reality graphical objects 316 overlaying a base image: the base image can simply be the user's actual field of view (as in the case of a mixed reality display function projected onto the originally transparent display of a set of smart glasses) or a digital image corresponding to the user's field of view but captured in real - time by an optical capture device (the optical capture device can in turn capture images corresponding to the user's field of view in other possible views).

[0008] It is difficult to infer which of multiple visual stimuli (if any) is the object of attention at any given time. For example, when a user is faced with multiple stimuli (e.g., numbers displayed on a screen keyboard), it has been shown to be nearly impossible to directly infer which stimulus is being attended to at a given time from brain activity. The user perceives the number being attended to (say the number 5), so the brain must contain information that distinguishes that number from the others, but current methods are unable to extract that information. That is, current methods can infer that a stimulus has been perceived, but these methods cannot use only brain activity to determine which specific stimulus is being attended to.

[0009] To overcome this problem and to provide sufficient contrast between the stimuli and the background (and between the stimuli), it is known to configure the stimuli used by a visual BCI to flash or pulse (e.g., a large surface of pixels switches from black to white and vice versa), such that each stimulus has a distinguishable characteristic distribution that varies over time. The flashing stimuli evoke a measurable electrical response. Specific techniques monitor different electrical responses, such as steady-state visual evoked potentials (SSVEP) and P-300 event-related potentials. In a typical implementation, the stimuli flash at a rate of more than 6 Hz. As a result, such a visual BCI relies on a method that includes discretely rather than continuously displaying various stimuli in a display device and typically at different time points. Brain activity associated with attention to a given stimulus is found to correspond (i.e., be correlated) to one or more aspects of the temporal distribution of that stimulus (such as the frequency at which the stimulus flashes and / or the duty cycle at which the stimulus alternates between a flashing state and a stationary state).

[0010] Thus, the decoding of neural signals relies on the fact that when a stimulus is turned on, the stimulus will trigger a characteristic pattern of neural responses in the brain, which can be determined from the electrical signals, i.e., SSVEP or P-300 potentials, picked up by the electrodes of an EEG device (such as the electrodes of an EEG cap). This pattern of neural data may be very similar or even identical for different numbers, but it is time-locked to the perception of the number: only one number can pulse at any given time, such that the correlation between the time of the number pulse and the pulsed neural response can be determined as an indication that the number is the object of attention. By displaying each number at different time points, turning the number on and off at different rates, applying different duty cycles, and / or simply applying the stimuli at different time points, a BCI algorithm can establish which stimulus is most likely to trigger a given neural response when it is turned on, thus enabling the system to determine the target being attended to.

[0011] In recent years, visual BCIs have improved significantly, making the real-time and accurate decoding of a user's attention increasingly practical. However, the continuous flickering of stimuli - sometimes flickering across the entire screen when there are many stimuli - is an inherent limitation to the widespread use of this technology. In fact, this can cause discomfort and mental fatigue, and if persistent, can also cause physiological responses such as headaches. Additionally, the flickering effect can impede the user's ability to focus on a specific target, as well as the system's ability to quickly and accurately determine the object of attention. For example, when a user tries to focus on the number 5, other (i.e., peripheral) numbers act as distractors - temporarily attracting the user's attention and interfering with the user's visual system. This interference, in turn, can impede the performance of the BCI. Thus, improved methods are needed for differentiating screen targets and their displayed stimuli to determine which one the user is focusing on.

[0012] The requirements for a display device that presents visual stimuli limit the application of the foregoing technology. In particular, a suitable display may not be available or desirable. In some applications, interacting with an object through a screen may be inconvenient or impractical. Additionally, the user acceptability of EEG devices (and their electrodes) presents aesthetic constraints, as well as constraints in terms of comfort and ease of use. In many cases, these constraints are significant obstacles to the effective adoption of EEG technology. Examples of applications where long-term use comfort and the need for technical assistance impede adoption include applications such as video games, training (e.g., for health and safety or flight simulation), sleep aids, etc.

[0013] Accordingly, it is desirable to provide a brain-computer interface that addresses the challenges above. SUMMARY OF THE INVENTION

[0014] The present disclosure relates to a brain-computer interface in which visual stimuli are presented in direct association with real-world objects such that a user's intent can be extended to objects in the real world without the intervention of a screen or other display device, thereby providing an improved and intuitive user experience.

[0015] The present disclosure relates to techniques for applying visual stimuli to other conventional real-world objects that are presented as potentially interesting objects when within a user's field of view.

[0016] In certain embodiments, the applied visual stimuli can include projecting a time-modulated overlay image onto one or more real-world objects. The modulation causes the object to flicker or otherwise visually change such that the modulation serves as a stimulus for a relevant neural response in the user's brain. The neural response, in turn, can be measured and decoded to determine which interesting object is the focus of the user's attention.

[0017] In other embodiments, the object itself can include one or more light sources capable of emitting light with temporal modulation. Here too, the modulation causes the object to blink or otherwise visually change such that the modulation serves as a stimulus for the relevant neural response in the user's brain. The neural response can in turn be measured and decoded to determine which object of interest is the focus of the user's attention.

[0018] In other embodiments, an electronic badge can be provided that is separate from the real-world controllable objects but logically associated with at least one of them. The electronic badge includes one or more light sources capable of emitting light with temporal modulation. Unlike traditional screens and display devices, electronic badges are typically small in size and can be dedicated to outputting visual stimuli. The modulation in the emitted light causes the display portion of the electronic badge to blink or otherwise visually change such that the modulation serves as a stimulus for the relevant neural response in the user's brain. The neural response can in turn be measured and decoded to determine which electronic badge is the focus of the user's attention, and since the badge is logically associated with the real-world object, it is thereby determined which real-world object is the object of interest.

[0019] In each of the above embodiments, the modulation can be preferentially or exclusively applied to the high spatial frequency components of the projected overlay image.

[0020] According to a first aspect, the present disclosure relates to a brain-computer interface system, comprising: at least one light-emitting unit that outputs a corresponding visual stimulus generated by a stimulus generator, the visual stimulus having a characteristic modulation; at least one controllable object configured to receive a user instruction, each controllable object being associated with at least one visual stimulus; a neural signal capture device configured to capture neural signals associated with the user; an interface device operatively coupled to the neural signal capture device and the controllable object, the interface device comprising: a memory; and a processor operatively coupled to the memory and configured to: receive neural signals from the neural signal capture device; determine, based on the neural signals, which visual stimulus among the at least one visual stimulus is the object of the user's attention, the object of attention being inferred based on the presence in the neural signals of components having a property associated with the characteristic modulation of the visual stimulus; and send a command to the controllable object determined to be associated with the object of attention, wherein the controllable object implements an action based on the command.

[0021] In certain embodiments, implementing the action includes controlling the controllable object to change state from a standby state.

[0022] In certain embodiments, at least one controllable object includes a stimulus generator and a light-emitting unit for outputting a visual stimulus generated by the stimulus generator.

[0023] In some embodiments, the light-emitting unit and the stimulation generator are disposed in an electronic badge that is separate from one or more controllable objects and is logically associated with at least one of the controllable objects. Also, the light-emitting unit outputs a visual stimulation generated by the stimulation generator.

[0024] In some embodiments, the light-emitting unit is a projector or a laser display device that is operatively coupled to the stimulation generator and projects a corresponding visual stimulation onto the controllable object; and wherein the controllable object reflects the projected stimulation.

[0025] In some embodiments, the light-emitting unit or each light-emitting unit includes at least one of the following: a light-emitting diode (LED); an LED array; a liquid crystal display (LCD) device; an organic light-emitting diode (OLED) display; an active matrix organic light-emitting diode (AMOLED) display; or an arc.

[0026] In some embodiments, the system further includes a processing device that includes the stimulation generator, wherein the processing device is communicatively coupled to the interface device and is configured to transmit information indicative of the generated visual stimulation to the interface device.

[0027] In some embodiments, modulation is selectively applied to the high spatial frequency (HSF) components of the display data.

[0028] According to a second aspect, the present disclosure relates to a method of operating a brain-computer interface system that includes a neural signal capture device and at least one light-emitting unit for outputting a visual stimulation generated by a stimulation generator, the visual stimulation having a characteristic modulation. The method includes, in a hardware interface device operatively coupled to the neural signal capture device and a controllable real-world object: forming an association between the controllable real-world object and at least one visual stimulation; receiving neural signals associated with a user captured by the neural signal capture device; determining, based on the neural signals, which of the at least one visual stimulation is the user's object of attention, the object of attention being inferred based on the presence in the neural signals of a component having a property associated with the characteristic modulation of the visual stimulation; and sending a command to the controllable object determined to be associated with the object of attention, thereby controlling the controllable object to perform an action based on the command.

[0029] In some embodiments, the method further includes performing an action, and performing the action includes controlling the controllable object to change state from a standby state.

[0030] In some embodiments, at least one controllable object includes a stimulus generator and a light-emitting unit, and forming an association between a controllable real-world object and at least one visual stimulus includes controlling the light-emitting unit to output the visual stimulus generated by the stimulus generator.

[0031] In some embodiments, the light-emitting unit and the stimulus generator are provided in an electronic badge that is separate from one or more controllable objects, and forming an association between a controllable real-world object and at least one visual stimulus includes logically associating the electronic badge with at least one controllable object and controlling the light-emitting unit of the electronic badge to output the visual stimulus generated by the stimulus generator.

[0032] In some embodiments, the light-emitting unit is a projector that is operatively coupled to the stimulus generator, and forming an association between a controllable real-world object and at least one visual stimulus includes controlling the projector to project a corresponding visual stimulus onto the controllable object such that the controllable object reflects the projected stimulus.

[0033] In some embodiments, the brain-computer interface system further includes a processing device that includes the stimulus generator; the processing device is communicatively coupled to the hardware interface device, and forming an association between a controllable real-world object and at least one visual stimulus further includes causing the processing device to transmit information indicating the generated visual stimulus to the interface device.

[0034] In some embodiments of the method, modulation is selectively applied to the high spatial frequency (HSF) component of the display data.

[0035] According to a third aspect, the present disclosure relates to a computer-readable storage medium that includes instructions which, when executed by a machine, cause the machine to perform the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] To facilitate identification of the discussion of any particular element or act, one or more of the most significant digits in the reference numerals refer to the figure number in which the element is first introduced.

[0037] Figure 1 An electronic architecture for receiving and processing EEG signals according to the present disclosure is shown;

[0038] Figure 2 A system incorporating a brain-computer interface (BCI) according to the present disclosure is shown;

[0039] Figure 3 Various examples of display devices suitable for use with the BCI system of the present disclosure are shown;

[0040] Figure 4Shows a first exemplary embodiment of the BCI system of the present disclosure;

[0041] Figure 5 Shows a second exemplary embodiment of the BCI system of the present disclosure;

[0042] Figure 6A and Figure 6B Shows yet another exemplary embodiment of the BCI system of the present disclosure;

[0043] Figure 7 Shows the main functional blocks in the operation method of BCI according to the present disclosure.

[0044] Figure 8 Is a block diagram showing a software architecture in which the present disclosure can be implemented according to some example embodiments; and

[0045] Figure 9 Is a graphical representation of a machine in the form of a computer system according to some example embodiments, in which a set of instructions can be executed to cause the machine to perform any one or more of the methods discussed herein. Detailed Description

[0046] The following description includes systems, methods, techniques, instruction sequences, and computer program products for implementing illustrative embodiments of the present disclosure. In the following description, for purposes of illustration, numerous specific details are set forth to provide an understanding of the various embodiments of the inventive subject matter. However, it will be apparent to those skilled in the art that embodiments of the inventive subject matter can be practiced without these specific details. Generally, well-known instruction instances, protocols, structures, and techniques need not be shown in detail.

[0047] Figure 1 Shows an example of an electronic architecture for receiving and processing EEG signals by means of the EEG device 100 according to the present disclosure.

[0048] To measure the spread potential on the surface of the skull of the subject 110, the EEG device 100 includes a portable device 102 (i.e., a cap or a headgear), an analog-to-digital conversion (ADC) circuitry 104, and a microcontroller 106. Figure 1 The portable device 102 includes one or more electrodes 108, typically between 1 and 128 electrodes, advantageously between 2 and 64 electrodes, advantageously between 4 and 16 electrodes.

[0049] Each electrode 108 may include a sensor for detecting an electrical signal generated by the neuronal activity of a subject and an electronic circuit for preprocessing (e.g., filtering and / or amplifying) the detected signal before analog-to-digital conversion: such an electrode is referred to as “active”. Figure 1 An active electrode 108 in use is shown in Figure 1 , where the sensor is in physical proximity to the scalp of the subject. The electrode may be suitable for use with a conductive gel or other conductive liquid (referred to as a “wet” electrode) or without such a liquid (i.e., a “dry” electrode).

[0050] Each ADC circuit 104 is configured to convert the signals of a given number (e.g., between 1 and 128) of active electrodes 108.

[0051] The ADC circuit 104 is controlled by the microcontroller 106 and communicates with the microcontroller 106, for example, via the protocol SPI (“Serial Peripheral Interface”). The microcontroller 106 packs the received data for transmission to an external processing unit (not shown), for example, via Bluetooth, Wi-Fi (“Wireless Fidelity”) or Li-Fi (“Light Fidelity”), and the external processing unit is, for example, a computer, a mobile phone, a virtual reality headset, a car or an automotive computer system or computer system of an aircraft, an avionics computer system of an aircraft.

[0052] In some embodiments, each active electrode 108 is powered by a battery ( Figure 1 not shown in Figure 1 ). The battery is conveniently disposed in the housing of the portable device 102.

[0053] In some embodiments, each active electrode 108 measures a corresponding potential value, subtracts the potential measured by a reference electrode (Ei = Vi - Vref) from the potential value, and the difference is digitized by means of the ADC circuit 104 and then sent by the microcontroller 106.

[0054] In some embodiments, the method of the present disclosure introduces a target object for display in a graphical user interface of a display device. The target object includes controls, and the controls are in turn associated with user-selectable actions.

[0055] Figure 2 A system incorporating a brain-computer interface (BCI) according to the present disclosure is shown. The system incorporates a neural response device 206 of the EEG device 100 as shown in, for example, Figure 1 Figure 1 . In the system, an image is displayed on the display of the display device 202. The subject 204 views the image on the display and focuses on the target object 210.

[0056] In an embodiment, the display device 202 displays at least the target object 210 as a graphical object having a changing temporal characteristic that is different from the temporal characteristics of the background and / or other display objects in the display. The changing temporal characteristic can be, for example, a continuous or time-locked blinking effect that changes the appearance of the target object at a rate greater than 6 Hz. In cases where more than one graphical object is a potential target object (i.e., in cases where a choice of the target object to focus on is provided to the viewing subject), each object is associated with a discrete spatial and / or temporal code.

[0057] The neural response device 206 detects a neural response associated with the attention focused on the target object (i.e., the tiny electrical potential indicating brain activity in the visual cortex); thus, the visual perception of the changing temporal characteristic of the target object acts as a stimulus in the subject's brain, generating a specific brain response consistent with the code associated with the attended target object. The detected neural response (e.g., electrical potential) is then converted into a digital signal and transmitted to the processing device 208 for decoding. Examples of neural responses include visual evoked potentials (VEPs), which are commonly used in neuroscience research. The term VEP includes: the traditional SSVEP as mentioned above, where the stimulus oscillates at a specific frequency; and other methods such as coded-modulated VEPs, where the stimulus undergoes variable or pseudo-random time coding.

[0058] The processing device 208 executes instructions to interpret the received neural signals to determine in real time a feedback indicating the target object having the current (visual) attention focus. Decoding the information in the neural response signal relies on the correspondence between the information and one or more aspects of the temporal distribution of the target object (i.e., the stimulus). In some embodiments, the processing device 208 and the neural response device 206 can be provided in a single device such that the decoding algorithm is directly executed on the detected neural response. Thus, a BCI utilizing visually associated neural signals can be used to determine which objects on the screen the user is attending to.

[0059] In some embodiments, the processing device can conveniently generate image data for presentation on the display device 202 that includes the target object changing over time.

[0060] The feedback can be conveniently visually presented on the display screen. For example, the display device can display an icon, cursor, crosshair, or other graphical object or effect very close to the target object, highlighting the object that appears to be the current focus of visual attention. Clearly, such a visual display of the feedback has a reflexive cognitive effect on the perception of the target object, amplifying the brain response. This positive feedback (where the apparent target object is confirmed as the expected target object due to extended amplified attention) is referred to herein as "neural synchronization".

[0061] Studies of the way the human visual sensing operation works have shown that when gazing at a screen with multiple objects and focusing on one of them, the human visual system will be receptive to both high spatial frequency (HSF) and low spatial frequency (LSF). Evidence suggests that the human visual system is mainly sensitive to the HSF component of the specific display area being focused on (e.g., the object the user is gazing at). In contrast, for peripheral objects, the human visual system is mainly sensitive to their LSF components. In other words, the picked-up neural signals will be substantially affected by both the HSF component from the focused target and the LSF component from the peripheral targets. However, since all objects evoke a certain proportion of both HSF and LSF, processing the neural signals to determine the focused object may be hampered by the LSF noise contributed by the peripheral objects. This tends to make the identification of the focused object less accurate and timely.

[0062] When the human visual system is tuned to process multiple stimuli (usually unconsciously) at different locations in the visual field in parallel, peripheral object stimuli will continue to trigger neural responses in the user's brain even if they appear in the periphery of the visual field. As a result, this causes competition between the multiple stimuli and makes the specific neural decoding of the focused object (target) more difficult.

[0063] The co-pending international patent application No. PCT / EP2020 / 081348 (file number 5380.002WO1), filed on November 6, 2020 (the entire specification of which is incorporated herein by reference), describes a method for the following challenge: determining a focused object (target) from objects (distractors) peripheral to the target at a certain speed and accuracy. The method described in this international patent application relies on the characteristics of the human visual system discussed above. Multiple objects are displayed such that each object is divided into a version consisting only of the LSF component of the object and a version consisting only of the HSF component. In one example, a flickering visual stimulus used to evoke a decodable neural response (e.g., SSVEP) is transmitted only through the HSF version of the object. This flickering HSF version is superimposed on the LSF version (which does not flicker).

[0064] Known systems in the medical or related research fields generally include a head-mounted device that has attachment locations for receiving respective sensors / electrodes. Electronic circuits are then connected to the electrodes and the housing of the acquisition chain (i.e., the assembly of connection components for acquiring EEG signals). Thus, an EEG device is usually formed by three different elements that the operator / demonstrator must assemble each time it is used. Also, the nature of the EEG device is such that technical assistance is desirable if not essential.

[0065] The BCI described above can be used in combination with real-world objects to make the objects controllable or otherwise interactive. In some embodiments, the generation of the stimuli is handled by one or more light sources (e.g., light-emitting diodes, LEDs, or small display units) provided associated with (or on the surface of) the controllable object.

[0066] In some embodiments, the generation of the stimuli is handled by a projector or a scanning laser device such that the visual stimuli are projected onto the controllable object and the controllable object outputs visual stimuli by reflecting the projected stimuli.

[0067] As in the case of a BCI using a display screen with which a user interacts with objects on the screen, the controllable objects in the present disclosure can themselves exhibit visual stimuli with feature modulation (e.g., flickering stimuli) such that the neural responses to the presence of these stimuli become apparent and can be decoded from the neural signals captured by a neural signal capture device (e.g., an EEG device).

[0068] In some embodiments, an attention focus is determined on the visual display of the controllable device for sending commands to the controllable object. Then, the controllable object can perform actions based on the commands: for example, the controllable object can emit an audible sound, unlock a door, turn on or off, change an operating state, etc. The actions can also provide visual or other feedback associated with the controllable object to the user: this can be used in the positive feedback loop discussed above, but can also provide real-time feedback indicating the degree of certainty that the object has been recognized as the attention focus to give the user information about his level of concentration.

[0069] Figure 4A first exemplary embodiment is shown, in which at least one surface of a remote control RC car 404 incorporates an LED array 410 capable of generating a visual display. In one embodiment, the RC car includes: a receiving circuit that picks up commands sent from a stimulus generator (which may conveniently be a decoding computer 104 as shown) by means of wireless unidirectional communication (e.g., radio frequency RF; infrared; or near field communication NFC); a processing unit (i.e., a microcontroller) that functions to interpret the received commands and control the operation of the RC car and / or the visual display accordingly; and a display driver circuit that applies light modulation to a display device of the type used to generate a visual display (here, the LED panel 410). The RC car 404 may also be configured to receive commands from a conventional radio controller 414 via a radio communication link 412. Thus, the operation of the RC car can be controlled by commands originating from the decoding computer 104 (as a result of decoded brain activity in the user 402) and commands from the radio controller 414. In some embodiments, the stimulus generator, together with the visual display, is embedded in the RC car 404. Commands from the decoding computer 104 may include signals for synchronizing or configuring the modulation generated by the stimulus generator with the decoding computer. If it is assumed that SSVEP is used to monitor brain activity at a predetermined set frequency, this link between the decoding computer and the subject can be omitted. The microcontroller can be programmed to modulate light at the same frequency in an asynchronous manner such that the modulation does not need to be sent or even synchronized with the decoding computer 104.

[0070] The visual display (i.e., the LED array 410) emits a visual stimulus that includes a time modulation (i.e., a blinking effect) unique to a given RC car 404. By visually attending to the car (and thus the stimulus output by the car), the user generates a neural response that can in turn be captured by a BCI (such as the BCI described above). The neural signals captured from the user's brain encode the time modulation. The BCI then decodes the neural signals to determine which car (if any) is the object of the user's attention (or focus). In some embodiments, the BCI includes a communication unit through which commands can be sent to the remote control car. The command can, for example, be an instruction to enter an active state, following which control signals sent by a conventional remote controller device can be sent to the car. The command can conveniently be a "stay active" command, whereby sustained attention results in repeated transmission of the command, but diverted attention causes the car to deactivate. In the presence of more than one such remote control car (each generating a different stimulus), the user can change the focus of attention (the unattended car reverts to a standby or inactive state) and apply the remote controller to the car forming the new focus of attention.

[0071] Figure 5A second exemplary embodiment is shown, in which a projector 510 projects a stimulus including temporal modulation (i.e., a flickering effect) onto at least one surface of a real-world object 504 associated with a controllable object 514. The stimulus is generated by a stimulus generator 512. When a BCI-equipped user 502 views the light reflected from the object 504, the reflected light generates a decodable neural response in the user's brain (which can be captured by the BCI). Figure 5 A plurality of real-world objects 504, 506, etc. are shown - the projector 510 projects corresponding stimuli onto each object. The BCI then decodes the neural signals to determine which of the objects (if any) is the user's attention object. Each real-world object 504, 506, etc. is associated with an interaction with a corresponding command or a controllable object 514. In some embodiments, the BCI includes a communication unit through which a command can be sent to the controllable object identified as the attention object. The command can be, for example, an instruction to enter an active state. In one embodiment, as shown, the controllable object includes a speaker 514, and the command can be a command to control the speaker to reproduce an audio signal such as an alarm or a tone. In some embodiments, the real-world object onto which the stimulus is projected or each real-world object itself can be a controllable object.

[0072] Figure 6A Another exemplary embodiment is shown, in which three visual displays 610, 610', 610'' are provided remote from the controllable object 604. Each of the visual displays 610, 610', 610'' (e.g., respective LED arrays) generates a corresponding different visual stimulus including temporal modulation (i.e., a flickering effect). The visual stimulus used here exploits, for example, the high decodability of a plurality of tiny drawn HSF elements (by using an opaque physical mask on the LED array, which can be considered a kind of printed Gabor filter). In some embodiments, the mask itself is controllable: a controllable motor can switch a shutter device between an open and closed state. Alternatively, a transparent liquid crystal device can be used to controllably mask or transmit incident light from one or more light sources, providing a backlit LCD display with a controllable time and / or spatially variable mask.

[0073] In the example shown, each of the visual displays 610, 610', 610'' incorporates a sign or icon (here, colors: red R; green G; blue B) indicating selectable options. The sign or icon can be formed by setting an engraved pattern in an otherwise opaque physical mask or by controlling the pattern of light transmitted by a shutter device or LCD. By focusing attention on one of the visual displays 610, 610', 610'', the BCI-equipped user 602 generates a neural response corresponding to the visual stimulus unique to that visual display 610. This is then interpreted as an intention to select the corresponding option (assume red R). Then, the processor 104 of the BCI uses the expected option to generate a command that is sent to the controllable object 604. In the example shown, the controllable object is a color-tunable lamp 604, and the command can be an instruction to control the lamp to emit red light. Just as Figure 4 in the RC car example in Figure 6A the circuitry used at each of the visual displays 610, 610', 610'' in the example includes: a receiving circuit that picks up the command sent from the stimulus generator 612; a processing unit; and a display driver circuit. Although the communication link between the processor 104 (i.e., the decoding computer) of the BCI and the object 604 is shown as a wireless link in Figure 6A it can also be wired.

[0074] In some embodiments, each of the visual displays 610, 610', 610'' can display visual stimuli associated with respective commands to be sent to corresponding different controllable objects. Thus, there can be multiple different lamps, each subject to a command associated with a respective one of the displayed visual stimuli. Instead of simply controlling the color of the light emitted by one tunable lamp, attending to one visual stimulus can activate one or more of the lamps; attending to another visual stimulus (displayed on another display) can, for example, dim one or more of the lamps; and attending to a third visual stimulus (displayed on yet another display) can put multiple different lamps into an economy mode in which the lamps only light up when there is not enough natural light.

[0075] In Figure 6BIn another exemplary embodiment shown, a visual display, referred to as an "electronic badge", takes the form of a physical, movable object, separate from real-world controllable objects but logically associated with at least one of them. In the present disclosure, these electronic badges provide small, movable, programmable screens (e.g., 3 cm × 3 cm) that display one or more visual stimuli associated with other (controllable) objects or specific functions of such objects. The screen can be, for example, a backlit LCD, OLED, or AMOLED display. The screen can be programmed to emit light with a temporal modulation similar to the light sources described above, thereby generating one or more visual stimuli. Different from Figure 6A the visual displays, Figure 6B each of the electronic badges 620, 620', 620" further includes its own stimulus generator 612' (see the inserted "exploded" view of badge 620") in addition to a processing unit 614, a display driver circuit 616, and a screen 618 for outputting visual stimuli.

[0076] The electronic badge can be embedded within a physical object (which can be a controllable object associated with the stimulus or each stimulus generated by the electronic badge): Badge 620 is embedded within the controllable lamp 604'. Alternatively, the badge can be placed on or fixed (temporarily or permanently) to the surface of a physical object, or actually placed on or fixed (temporarily or permanently) to any surface from which it is convenient to interact with or control a controllable physical object, such as a nearby wall, a control panel, or a furniture item. In Figure 6B this case, it is also possible to control the controllable lamp 604' by interacting with the electronic badge 620' in the control panel 606. In other embodiments, the electronic badge 620" can be portable or even wearable. In some embodiments, the electronic badge includes a battery and a wireless communication module. The electronic badge can be attached to a surface in any conventional manner, such as by a clip, an adhesive, a magnet, a screw, a bolt, or any other such fixing means.

[0077] In some cases, as Figure 6B in this case, more than one electronic badge can be used to control different types of interactions with a single physical object: each electronic badge displays a different visual stimulus and a different action is associated with each stimulus. The controllable lamp 604' can be turned on by badge 620 and dimmed by badge 620'. In a further illustration, consider the case where instead of Figure 6AFor a three-mask visual display, three electronic badges can be arranged (assumed on a nearby tabletop) to display visual stimuli that differ in their respective modulation aspects, with each badge having a different associated command for the color-tunable light 604. The corresponding badges can also be controlled to have different visual appearances, such as emitting light in a color representing the desired effect of the color-tunable light.

[0078] In some cases, the visual stimuli displayed on the screen of an electronic badge can be applied to the entire screen or to corresponding portions of the screen according to the programming of the badge. As previously described, visual stimuli are generated to elicit a decodable neural response in the brain of a BCI user who notices the stimulus. The visual stimuli can be presented as a plurality of modulated tiny drawn lines or "beans" (reducing the flicker effect without compromising the ability to evoke a neural response when a given stimulus is the focus of the BCI user's attention). The neural response can in turn be measured and decoded to determine which electronic badge is the focus of the user's attention, and since the badge is logically associated with a real-world object and / or with a control action for a real-world object, it is thereby determined which real-world object is the object of interest and / or control.

[0079] In some cases, each badge or each of multiple parts of a badge can display a corresponding symbol associated with an associated action / command (e.g., on / off, play / pause, forward, etc.) that controls the operation of one or more controllable real-world objects. Different visual stimuli can then be applied to the corresponding symbols. In some embodiments, the displayed symbols can be changed according to the programming of the badge. For example, for instance, when a controlled audio system 604” is in a playback state, a badge 620” that shows a “play” symbol can replace the play symbol with a “pause” symbol. Similarly, in a badge fixed to a controllable door, the stimuli displayed on the badge can be interpreted as an “open” command or a “close” command depending on the current state of the door.

[0080] The controllable real-world objects to which visual stimuli are applied by the light-emitting unit are not limited to the objects mentioned in the above examples. In addition to controllable doors, lights, remotely controlled toys, audio systems / speakers, more generally, the same non-contact control can be permitted for the operation of switches and valves. Users who cannot manipulate a controllable device by hand can still control its operation: from operating a faucet to provide water in a sink to selecting an item to be retrieved by a robot from a shelf in a warehouse. Aspects of the present disclosure can be applied in many different scenarios, including home, healthcare, business (retail, wholesale, and logistics), agriculture, and maritime.

[0081] Figure 7 A BCI system according to the present disclosure is shown (e.g., Figure 2The main functional blocks in the method of operation of the BCI system shown in. In block 702, the interface device 208 forms an association between a controllable real-world object and at least one visual stimulus. In block 704, the interface device 208 receives neural signals associated with the user captured by the neural signal capture device 206. In block 706, the interface device 208 determines which of the at least one visual stimuli is the object of the user's attention based on the neural signals, and the object of attention is inferred from the presence of components in the neural signals having properties associated with the feature modulation of the visual stimuli. In block 708, the interface device 208 sends a command to the controllable object determined to be associated with the object of attention, thereby controlling the controllable object to perform an action based on the command.

[0082] The positive neural synchronization feedback loop described with respect to Figure 2 the BCI in can be employed to confirm the user's intention, which may be, for example, to initiate an action such as an information request, a switch of the control state of a real-world object, or the activation / selection of an object (e.g., for control) in a mixed reality setting.

[0083] Figure 8 is a block diagram showing an example software architecture 806, which can be used in conjunction with various hardware architectures described herein (e.g., Figure 2 the processing device 208). Figure 8 is a non-limiting example of a software architecture, and it will be appreciated that many other architectures can be implemented to facilitate the functions described herein. The software architecture 806 can be executed on hardware such as Figure 2 the processing device 208 or Figure 9 the machine 900, which includes a processor 904, a memory 906, and input / output (I / O) components 918, etc. A representative hardware layer 852 is shown and the representative hardware layer 852 can represent, for example, Figure 9 the machine 900. The representative hardware layer 852 includes a processing unit 854 with associated executable instructions 804. The executable instructions 804 represent the executable instructions of the software architecture 806, including the implementation of the methods, modules, etc. described herein. The hardware layer 852 also includes a memory and / or storage module shown as the memory / storage device 856, which also has executable instructions 804. The hardware layer 852 may also include other hardware 858, such as dedicated hardware for interfacing with EEG electrodes and / or for interfacing with a display device.

[0084] In Figure 8In the example architecture, the software architecture 806 can be conceptualized as a stack of layers, where each layer provides a specific function. For example, the software architecture 806 can include layers such as an operating system 802, libraries 820, frameworks or middleware 818, applications 816, and a presentation layer 814. Operationally, the applications 816 and / or other components within the layer can make application programming interface (API) calls 808 through the software stack and receive responses as messages 810. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile operating systems or specialized operating systems may not provide frameworks / middleware 818, while other operating systems may provide such layers. Other software architectures can include additional layers or different layers.

[0085] The operating system 802 can manage hardware resources and provide common services. The operating system 802 can include, for example, a kernel 822, services 824, and drivers 826. The kernel 822 can act as an abstraction layer between the hardware and other software layers. For example, the kernel 822 can be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. The services 824 can provide other common services for other software layers. The drivers 826 can be responsible for controlling or interfacing with the underlying hardware. For example, depending on the hardware configuration, the drivers 826 can include a display driver, an EEG device driver, a camera device driver, a driver, a flash drive, a serial communication driver (e.g., a universal serial bus (USB) driver), a driver, an audio driver, a power management driver, etc.

[0086] The library 820 can provide a common infrastructure that can be used by the application 816 and / or other components and / or layers. Compared with directly interfacing with the underlying operating system 802 functions (such as the kernel 822, services 824, or drivers 826), the library 820 generally provides functions that enable other software modules to perform tasks in an easier manner. The library 820 can include a system library 844 (such as the C standard library), which can provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. Additionally, the library 820 can include an API library 846, such as a media library (e.g., a library that supports the presentation and manipulation of various media formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG), a graphics library (e.g., the OpenGL framework that can be used to render 2D and 3D graphical content on a display), a database library (e.g., SQLite that can provide various relational database functions), a web library (e.g., WebKit that can provide web browsing functions), etc. The library 820 can also include a variety of other libraries 848 to provide many other APIs to the application 816 and other software components / modules.

[0087] The framework 818 (sometimes also referred to as middleware) provides a higher-level common infrastructure that can be used by the application 816 and / or other software components / modules. For example, the framework / middleware 818 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. The framework / middleware 818 can provide a wide range of other APs that can be used by the application 816 and / or other software components / modules, some of which may be specific to a particular operating system or platform.

[0088] The application 816 includes built-in applications 838 and / or third-party applications 840.

[0089] The application 816 can use the built-in operating system functions (such as the kernel 822, services 824, and / or drivers 826), the library 820, or the framework / middleware 818 to create a user interface to interact with the users of the system. Alternatively or additionally, in some systems, the interaction with the user can occur through a presentation layer such as the presentation layer 814. In these systems, the application / module "logic" can be separated from aspects of the application / module that interact with the user.

[0090] Figure 9 is a block diagram showing components (such as Figure 2 the processing device 208) of a machine 900 according to some example embodiments, the machine 900 being capable of reading instructions from a machine-readable medium (such as a machine-readable storage medium) and performing any one or more of the methods discussed herein. Specifically, Figure 9A graphical representation of a machine 900 is shown in the example form of a computer system, in which instructions 911 (e.g., software, program, application, applet, app, or other executable code) can be executed to cause the machine 900 to perform any one or more of the methods discussed herein. Similarly, the instructions 911 can be used to implement the modules or components described herein. The instructions 911 transform the general unprogrammed machine 900 into a particular machine programmed to perform the described and illustrated functions in the described manner. In an alternative embodiment, the machine 900 operates as a stand-alone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 900 can operate in a server-client network environment as a server machine or a client machine, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 900 can include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smartphone, a mobile device, a wearable device (e.g., a smartwatch or a head-mounted display), a smart home device (e.g., a smart appliance), other smart devices, a network service tool, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing the instructions 911 specifying the actions to be taken by the machine 900. Further, although only a single machine 900 is shown, the term "machine" shall also be taken to include a collection of machines that individually or jointly execute the instructions 911 to perform any one or more of the methods discussed herein.

[0091] The machine 900 can include a processor 904, a memory 906, and input / output (I / O) components 918 that can be configured to communicate with each other, for example, via a bus 902. In an example embodiment, the processor 904 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), other processors, or any suitable combination thereof) can include, for example, a processor 908 and a processor 912 that can execute the instructions 911. The term "processor" is intended to include a multi-core processor that can include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Figure 9 multiple processors are shown, the machine 900 can include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0092] The memory 906 may include a memory 914 such as a main memory, a static memory, or other memory storage devices, and storage units 916, both of which are accessible by the processor 904, such as via a bus 902. The storage units 916 and the memory 914 store instructions 911 that embody any one or more of the methods or functions described herein. The instructions 911 may also reside, completely or partially, within at least one of the memory 914, the storage units 916, the processor 904 (e.g., within a cache of the processor) or any suitable combination thereof during execution by the machine 900. Accordingly, the memory 914, the storage units 916, and the memory of the processor 904 are examples of machine-readable media.

[0093] As used herein, "machine-readable medium" refers to a device capable of storing instructions and data temporarily or permanently, and may include, but is not limited to: random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EEPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be regarded as including a single medium or multiple media capable of storing the instructions 911 (e.g., a centralized or distributed database or associated cache and server). The term "machine-readable medium" should also be regarded as including any medium or combination of multiple media capable of storing instructions (e.g., instructions 911) for execution by a machine (e.g., machine 900), such that the instructions, when executed by one or more processors (e.g., processor 904) of the machine 900, cause the machine 900 to perform any one or more of the methods described herein. Accordingly, "machine-readable medium" refers to a single storage device or apparatus, as well as a "cloud-based" storage system or storage network including multiple storage devices or apparatuses. The term "machine-readable medium" does not include the signal itself.

[0094] The input / output (I / O) component 918 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific input / output (I / O) components 918 included in a particular machine will depend on the type of the machine. For example, a portable machine such as a mobile phone or a user interface machine may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It will be appreciated that the input / output (I / O) component 918 may include Figure 9 many other components not shown.

[0095] For purposes of simplifying the following discussion, the input / output (I / O) component 918 is grouped according to function, and this grouping is in no way limiting. In various example embodiments, the input / output (I / O) component 918 can include an output component 926 and an input component 928. The output component 926 can include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), auditory components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, and the like. The input component 928 can include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touchscreens that provide touch gestures or the location and / or force of a touch, or other tactile input components), audio input components (e.g., microphones), and the like.

[0096] In yet another example embodiment, the input / output (I / O) component 918 can include various other components such as a biometric component 930, a motion component 934, an environmental component 936, or a positioning component 938. For example, the biometric component 930 can include components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biometric signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves such as the output from an EEG device), identifying people (e.g., voice recognition, retina recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), and the like. The motion component 934 can include acceleration sensor components (e.g., accelerometers), gravity sensor components, rotational sensor components (e.g., gyroscopes), and the like. The environmental component 936 can include, for example, lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers that detect the ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors that detect the concentration of hazardous gases for safety or measure pollutants in the atmosphere), or other components that can provide an indication, measurement, or signal corresponding to the surrounding physical environment. The positioning component 938 can include position sensor components (e.g., global positioning system (GPS) receiver components), altitude sensor components (e.g., altimeters or barometers that detect the air pressure from which altitude can be obtained), orientation sensor components (e.g., magnetometers), and the like.

[0097] Various techniques can be used to enable communication. Input / output (I / O) component 918 can include communication component 940, which is operable to couple machine 900 to network 932 or device 920 via couplings 924 and 922, respectively. For example, communication component 940 can include a network interface component or other suitable device to interface with network 932. In another example, communication component 940 can include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, components (e.g., low power), components, and other communication components that provide communication via other modalities. Device 920 can be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via a universal serial bus (USB)). In the case where an EEG device or a display device is not integrated with machine 900, device 920 can be an EEG device (e.g., neural response device 206) and / or a display device (e.g., Figure 2 and Figure 3 display device).

[0098] Although described through multiple detailed exemplary embodiments, a portable device for acquiring electroencephalogram signals according to the present disclosure includes various variations, modifications, and improvements that are apparent to those skilled in the art. It should be understood that these various variations, modifications, and improvements fall within the scope of the subject matter of the present disclosure as defined by the appended claims.

[0099] Although the inventive subject matter has been described with reference to specific exemplary embodiments, various modifications and changes can be made to these embodiments without departing from the broader scope of the embodiments of the present disclosure. Such embodiments of the inventive subject matter may be referred to herein individually or collectively by the term "invention" merely for convenience and are not intended to voluntarily limit the scope of the present application to any single disclosure or inventive concept in the event that more than one disclosure or inventive concept is actually disclosed.

[0100] The embodiments shown herein have been described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments can be used and obtained therefrom, such that structural and logical substitutions and changes can be made without departing from the scope of the present disclosure. Accordingly, the specific embodiments should not be considered limiting, and the scope of the various embodiments is defined only by the appended claims and the full scope of equivalents to such claims.

[0101] As used herein, the term "or" may be construed in an inclusive or exclusive sense. Additionally, multiple instances may be provided for resources, operations, or structures that are described herein as a single instance. Further, the boundaries between various resources, operations, modules, engines, and data stores are somewhat arbitrary, and a particular operation is shown in the context of a particular illustrative configuration. Other allocations of functionality are envisioned, and other allocations of functionality may fall within the scope of various embodiments of the present disclosure. Generally, structures and functions that are presented as separate resources in an example configuration may be implemented as a combined structure or resource. Similarly, structures and functions that are presented as a single resource may be implemented as separate resources. These and other variations, modifications, additions, and improvements fall within the scope of the embodiments of the present disclosure as represented by the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

[0102] Accordingly, the present disclosure describes systems and methods for improving the accuracy, speed performance, and visual comfort of a BCI.

[0103] Examples

[0104] To better illustrate the systems and methods disclosed herein, a non-limiting list of examples is provided here:

[0105] 1. A brain-computer interface system, comprising:

[0106] At least one light-emitting unit that outputs a corresponding visual stimulus generated by a stimulus generator, the visual stimulus having a characteristic modulation;

[0107] At least one controllable object configured to receive a user instruction, each controllable object being associated with at least one visual stimulus;

[0108] A neural signal capture device configured to capture neural signals associated with the user;

[0109] An interface device operatively coupled to the neural signal capture device and the controllable object, the interface device comprising:

[0110] A memory; and

[0111] A processor operatively coupled to the memory and configured to:

[0112] Receive the neural signals from the neural signal capture device;

[0113] Determine which of the at least one visual stimulus is the object of the user's attention based on the neural signal, the object of attention being inferred based on the presence in the neural signal of a component having a property associated with the feature modulation of the visual stimulus; and

[0114] Send a command to the controllable object determined to be associated with the object of attention,

[0115] wherein the controllable object performs an action based on the command.

[0116] 2. The brain-computer interface system according to Example 1, wherein performing the action includes controlling the controllable object to change its state from a standby state.

[0117] 3. The brain-computer interface system according to Example 1 or 2, wherein the at least one controllable object includes the stimulus generator and the light-emitting unit for outputting the visual stimulus generated by the stimulus generator.

[0118] 4. The brain-computer interface system according to Example 1 or Example 2, wherein the stimulus generator and the light-emitting unit are provided in an electronic badge, the electronic badge being separate from the at least one controllable object but logically associated with the at least one controllable object.

[0119] 5. The brain-computer interface system according to Example 1 or Example 2, wherein the light-emitting unit is a projector, the projector being operatively coupled to the stimulus generator and projecting the corresponding visual stimulus onto the controllable object; and

[0120] wherein the controllable object reflects the projected stimulus.

[0121] 6. The brain-computer interface system according to any one of Examples 1 to 5, wherein the light-emitting unit or each light-emitting unit includes at least one of the following: a single light-emitting diode (LED); an LED array; a liquid crystal display (LCD) device; an organic light-emitting diode (OLED) display; or an arc.

[0122] 7. The brain-computer interface system according to any one of Examples 1 to 6, further comprising a processing device, the processing device including the stimulus generator, wherein the processing device is communicatively coupled to the interface device and is configured to transmit information indicating the generated visual stimulus to the interface device.

[0123] 8. The brain-computer interface system according to any one of Examples 1 to 7, wherein the modulation is selectively applied to the high spatial frequency (HSF) component of the display data.

[0124] 9. A method of operating a brain-computer interface system, the brain-computer interface system including a neural signal capture device and at least one light-emitting unit for outputting a visual stimulus generated by a stimulus generator, the visual stimulus having a characteristic modulation,

[0125] wherein the method includes, in a hardware interface device operatively coupled to the neural signal capture device and capable of controlling a real-world object:

[0126] forming an association between the controllable real-world object and the at least one visual stimulus;

[0127] receiving neural signals associated with a user captured by the neural signal capture device;

[0128] determining, based on the neural signals, which of the at least one visual stimulus is the object of the user's attention, the object of attention being inferred based on the presence in the neural signals of a component having a property associated with the characteristic modulation of the visual stimulus; and

[0129] sending a command to the controllable object determined to be associated with the object of attention, thereby controlling the controllable object to perform an action based on the command.

[0130] 10. The method according to example 9, wherein performing the action includes controlling the controllable object to change state from a standby state.

[0131] 11. The method according to example 9 or example 10, wherein the at least one controllable object includes the stimulus generator and the light-emitting unit, and wherein forming the association between the controllable real-world object and the at least one visual stimulus includes controlling the light-emitting unit to output the visual stimulus generated by the stimulus generator.

[0132] 12. The method according to example 9 or example 10, wherein the stimulus generator and the light-emitting unit are provided in an electronic badge, the electronic badge being separate from the at least one controllable object, and wherein forming the association between the controllable real-world object and the at least one visual stimulus includes logically associating the electronic badge with the at least one controllable object, and controlling the light-emitting unit of the electronic badge to output the visual stimulus generated by the stimulus generator.

[0133] 13. The method according to Example 9 or Example 10, wherein the light emitting unit is a projector, the projector is operatively coupled to the stimulus generator, and wherein forming the association between the controllable real-world object and the at least one visual stimulus includes controlling the projector to project the corresponding visual stimulus onto the controllable object such that the controllable object reflects the projected stimulus.

[0134] 14. The method according to any one of Examples 9 to 13, wherein the brain-computer interface system further includes a processing device, the processing device including the stimulus generator; wherein the processing device is communicatively coupled to the hardware interface device, and wherein forming the association between the controllable real-world object and the at least one visual stimulus further includes causing the processing device to transmit information indicative of the generated visual stimulus to the interface device.

[0135] 15. The method according to any one of Examples 9 to 14, wherein the modulation is selectively applied to the high spatial frequency (HSF) component of the display data.

[0136] 16. A computer-readable storage medium carrying instructions that, when executed by a machine, cause the machine to perform the method according to any one of Examples 9 to 15.

[0137] 17. An electronic badge for cooperating with the brain-computer interface system according to any one of Examples 1 to 8, the electronic badge including at least one of the light emitting units and a stimulus generator.

Claims

1. A brain-computer interface system, comprising: At least one light-emitting unit configured to output a corresponding visual stimulus generated by a stimulus generator, the corresponding visual stimulus having a characteristic modulation; At least one controllable object configured to receive a user's instruction, each controllable object among the at least one controllable object being associated with the visual stimulus output by a corresponding one of the at least one light-emitting units; A neural signal capture device configured to capture neural signals associated with the user; An interface device operatively coupled to the neural signal capture device and the controllable object, the interface device comprising: A memory; and A processor operatively coupled to the memory and the processor being configured to: Receive the neural signals from the neural signal capture device; Determine, based on the neural signals, which one of the at least one visual stimuli is the user's object of attention, the object of attention being inferred based on the presence in the neural signals of a component having a property associated with the characteristic modulation of the visual stimulus; and Send a command to the controllable object determined to be associated with the object of attention, Wherein the controllable object implements an action based on the command; Wherein the corresponding visual stimulus output by each of the at least one light-emitting units includes a high-spatial-frequency version consisting only of high-spatial-frequency components and a low-spatial-frequency version consisting only of low-spatial-frequency components, the high-spatial-frequency version being superimposed on the low-spatial-frequency version, wherein the high-spatial-frequency components are the components of the corresponding visual stimulus to which the human visual system is mainly sensitive when the components are at the focus of the visual field, and the low-spatial-frequency components are the components of the corresponding visual stimulus to which the human visual system is mainly sensitive when the components are at the periphery of the visual field; and Wherein the characteristic modulation is applied only to the high-spatial-frequency version of the corresponding visual stimulus.

2. The brain-computer interface system according to claim 1, wherein, Implementing the action includes controlling the controllable object to change state from a standby state.

3. The brain-computer interface system according to claim 1 or claim 2, wherein, The at least one controllable object includes the stimulus generator and the at least one light-emitting unit configured to output the corresponding visual stimulus generated by the stimulus generator.

4. The brain-computer interface system according to claim 1 or claim 2, wherein, The stimulus generator and the at least one light-emitting unit are provided in an electronic badge, the electronic badge being separate from the at least one controllable object but logically associated with the at least one controllable object.

5. The brain-computer interface system according to claim 1 or claim 2, wherein The at least one light-emitting unit is a projector operatively coupled to the stimulus generator and projecting the corresponding visual stimulus onto the at least one controllable object associated with the corresponding visual stimulus; and Wherein the at least one controllable object reflects the projected stimulus.

6. The brain-computer interface system according to claim 1 or claim 2, wherein, Each of the at least one light-emitting units includes at least one of the following: a single light-emitting diode; An array of light-emitting diodes; a liquid crystal display device; Or an organic light-emitting diode display.

7. The brain-computer interface system according to claim 1 or claim 2 further comprises a processing device, the processing device comprising the stimulation generator, wherein, The processing device is communicatively coupled to the interface device, and the processing device is configured to transmit information indicating the generated visual stimulus to the interface device.

8. A method of operating a brain-computer interface system, the brain-computer interface system including a neural signal capture device and at least one light-emitting unit for outputting corresponding visual stimuli generated by a stimulus generator, the corresponding visual stimuli having characteristic modulation, Among them, The method includes, in a hardware interface device operatively coupled to the neural signal capture device and capable of controlling a real-world object: Forming an association between the controllable real-world object and the at least one visual stimulus; Receiving neural signals associated with a user captured by the neural signal capture device; Based on the neural signals, determining which of the at least one visual stimulus is the object of the user's attention, the object of attention being inferred based on the presence in the neural signals of a component having a property associated with the characteristic modulation of the visual stimulus; And Sending a command to the controllable real-world object determined to be associated with the object of attention, so as to control the controllable real-world object to perform an action based on the command; Wherein, the corresponding visual stimulus output by each of the at least one light-emitting unit includes a high-spatial-frequency version composed only of high-spatial-frequency components and a low-spatial-frequency version composed only of low-spatial-frequency components, the high-spatial-frequency version being superimposed on the low-spatial-frequency version, wherein the high-spatial-frequency components are the components of the corresponding visual stimulus to which the human visual system is mainly sensitive when the components are at the focus of the visual field, and the low-spatial-frequency components are the components of the corresponding visual stimulus to which the human visual system is mainly sensitive when the components are at the periphery of the visual field; and Wherein, the characteristic modulation is only applied to the high-spatial-frequency version of the corresponding visual stimulus.

9. The method according to claim 8, wherein Performing the action includes controlling the controllable real-world object to change state from a standby state.

10. The method according to claim 8 or claim 9, wherein, The controllable real-world object includes the stimulus generator and the at least one light-emitting unit, and wherein, forming the association between the controllable real-world object and the at least one visual stimulus includes controlling the at least one light-emitting unit to output the visual stimulus generated by the stimulus generator.

11. The method according to claim 8 or claim 9, wherein The stimulus generator and the at least one light-emitting unit are provided in an electronic badge, the electronic badge being separate from the controllable real-world object, and wherein, forming the association between the controllable real-world object and the at least one visual stimulus includes logically associating the electronic badge with the controllable real-world object, and controlling the light-emitting unit of the electronic badge to output the visual stimulus generated by the stimulus generator.

12. The method according to claim 8 or claim 9, wherein, The at least one light-emitting unit is a projector, the projector being operatively coupled to the stimulus generator, and wherein forming the association between the controllable real-world object and the at least one visual stimulus includes controlling the projector to project the corresponding visual stimulus onto the controllable real-world object such that the controllable real-world object reflects the projected stimulus.

13. The method according to claim 8 or claim 9, wherein The brain-computer interface system further includes a processing device, the processing device including the stimulus generator; wherein the processing device is communicatively coupled to the hardware interface device, and wherein forming the association between the controllable real-world object and the at least one visual stimulus further includes causing the processing device to transmit information indicative of the generated visual stimulus to the hardware interface device.

14. A computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 8 to 13.

15. An electronic badge for cooperating with the brain-computer interface system according to any one of claims 1 to 7, the electronic badge including a badge stimulus generator and at least one badge light-emitting unit.

Images