Brain machine interface device and command determination method

Abstract

A brain machine interface device according to an embodiment generates commands for controlling a machine on the basis of the brain waves of a user. This brain machine interface device comprises a display control unit, a brain wave data acquisition unit, and a command determination unit. The display control unit displays visual symbols to which unique blinking cycles are allocated at different positions in the visual field of the user. The brain wave data acquisition unit acquires brain wave data including at least a steady-state visual evoked potential and an ocular potential from a brain wave sensor worn by a user. The command determination unit identifies a first command from the steady-state visual evoked potential and the blinking cycles, and determines a second command following the first command on the basis of information relating to movement of the gaze of the user as detected from the ocular potential.

Description

Brain-machine interface device and command determination method

[0001] One aspect of the present invention relates to a brain-machine interface device used to operate a machine that is driven and controlled by an actuator, such as an electric wheelchair, using electroencephalograms, and a method for determining a command to be output to the machine.

[0002] Brain-machine interface (BMI) is a technology for controlling machines using electroencephalograms (EEG). This technology is being applied to, for example, controlling electric wheelchairs with EEG to support physically disabled users.

[0003] One method for achieving BMI is the use of steady-state visual evoked potentials (SSVEPs). SSVEPs are a response in which, when a user continues to gaze at a visual stimulus (such as a checkerboard or light stimulus) that flashes at a constant frequency (period), an EEG signal containing the same frequency component (or its harmonic components) as the stimulus is generated from the visual cortex. Applying this technique, multiple target stimuli with different flashing frequencies can be presented, and the SSVEP can be used to identify which target stimulus the user is gazing at.

[0004] In BMIs using SSVEP, the system is designed to assign each target stimulus as a command. Examples of commands are forward / backward operations for wheelchair BMIs, and individual keys for PC keyboard BMIs.

[0005] Kondo, Aota, and Tanaka, Hisaya. "Development of a High-Frequency Stimulation SSVEP-BCI for Reducing Flicker-Induced Stress and Its Challenges." Journal of the Japan Society of Kansei Engineering (2023): TJSKE-D.

[0006] Due to the characteristics of SSVEP, it is necessary to maintain gaze on a stimulus for a certain period of time. This has led to the understanding that it takes time to detect commands, resulting in a long operation response time. For example, according to Table 1 in Non-Patent Document 1, even in keyboard-operated BMIs, which are the most actively developed BMIs using SSVEP, it takes at least two seconds to detect a command, as a rough guide for response time.

[0007] In particular, if BMI is applied to operating a wheelchair, it would be difficult to avoid a crisis if it took more than two seconds to detect a command. This is because immediate responses are required when a crisis is avoided, such as stopping or deceleration operations. In other words, to ensure the safety of the user, wheelchair BMIs require immediate responses when a crisis is avoided, such as deceleration and stopping operations. However, existing technology that uses only SSVEP cannot meet this requirement, so some kind of technological innovation is required.

[0008] The present invention has been made in light of the above circumstances, and aims to provide a technique that enables a reduction in the time required for command detection and an improvement in responsiveness.

[0009] A brain-machine interface device according to an embodiment generates a command for controlling a machine based on a user's electroencephalogram (EEG). The brain-machine interface device includes a display control unit, an electroencephalogram data acquisition unit, and a command determination unit. The display control unit displays visual symbols, each assigned a unique blinking cycle, at different positions in the user's visual field. The electroencephalogram data acquisition unit acquires electroencephalogram data, including at least a steady-state visual evoked potential (SVP) and an electrooculogram (EOG), from an EEG sensor worn by the user. The command determination unit identifies a first command based on the SVP and the blinking cycle, and determines a second command following the first command based on gaze movement information of the user detected from the electrooculogram.

[0010] According to one aspect of the present invention, it is possible to reduce the time required for command detection and improve responsiveness.

[0011] Fig. 1 is a block diagram showing an example of a brain-machine interface device according to an embodiment of the present invention. Fig. 2 is a diagram showing an example of a visual stimulus displayed on the display device 12 of Fig. 1. Fig. 3 is a functional block diagram showing the software configuration of the brain-machine interface device according to an embodiment of the present invention. Fig. 4 is a flowchart showing an example of a processing procedure of the BMI device 100 shown in Fig. 1. Fig. 5 is a diagram showing an example of a data flow in the BMI device 100 shown in Fig. 1.

[0012] Fig. 1 is a block diagram showing an example of a brain-machine interface device according to an embodiment of the present invention. The brain-machine interface device (BMI device) 100 shown in Fig. 1 is mounted on the back of, for example, an electric wheelchair 200, and generates and outputs commands 400 that reflect brain wave patterns according to the user's intentions. The commands 400 are input to an actuator 300 that drives the electric wheelchair 200.

[0013] The BMI device 100 is a computer comprising a processor 1, a program storage unit 2 as a storage unit, and a data storage unit 3. The BMI device 100 further comprises a sensor interface (I / F) unit 4 and an input / output I / F unit 5, which are communicatively connected via an internal bus 6. The sensor I / F unit 4 is connected to an electroencephalogram sensor 11 worn on the user's head and a display device 12. The input / output I / F unit 5 outputs a command 400 to an actuator 300 that drives the electric wheelchair 200.

[0014] The electroencephalogram sensor 11 detects electrical signals (electroencephalograms) derived from the user's brain activity and other bioelectric signals such as electrooculography, and outputs the obtained measurement data to the BMI device 100. The display device 12 is, for example, a head-mounted display or smart glasses, and displays visual stimuli on a target stimulus presentation display (not shown) that covers the user's field of vision.

[0015] 2 is a diagram showing an example of a visual stimulus displayed on the display device 12 of FIG. 1. The auditory stimulus includes a plurality of visual symbols (symbol patterns) each assigned with a unique blinking frequency. The symbol patterns are associated with commands such as turn right, turn left, go straight, and reverse, and blink at different frequencies at different positions within the user's field of view.

[0016] By using SSVEP, it is possible to determine which symbol pattern the user is paying attention to and identify the associated command. However, its responsiveness is not necessarily high (fast), so further technological innovation is needed to determine commands related to crisis avoidance, such as deceleration and stopping. The following disclosure describes a technology that can solve this need.

[0017] 3 is a functional block diagram showing the software configuration of a BMI device 100 according to an embodiment of the present invention. The processor 1 of the BMI device 100 includes, as processing functions according to the embodiment, a display control unit 1a, an electroencephalogram data acquisition unit 1b, and a command determination unit 1c.

[0018] The display control unit 1a displays a visual stimulation pattern as shown in Fig. 2 in the user's field of view on the display device 12. The electroencephalogram data acquisition unit 1b acquires steady-state visual evoked potentials (SSVEPs) from the electroencephalogram sensor 11. Furthermore, in the embodiment, the electroencephalogram data acquisition unit 1b acquires electrooculograms (EOGs) from the electroencephalogram sensor 11. That is, in the embodiment, the electroencephalogram data acquired from the electroencephalogram sensor 11 includes at least the steady-state visual evoked potentials (SSVEPs) and the electrooculograms (EOGs).

[0019] The command determination unit 1c uses the SSVEP to identify a first command, which is a command identified from the steady-state visual evoked potential and the blinking period of the symbol pattern, and is, for example, a command that instructs the actuator 300 to individually turn right, turn left, go straight, and go backward, as shown in FIG.

[0020] Furthermore, the command determination unit 1c detects the user's gaze movement information from the electrooculogram and determines a second command based on this gaze movement information. The second command is a command for crisis avoidance that requires more urgent control, such as deceleration or stopping. Here, the second command is temporally continuous with the first command. In other words, the second command is a command that follows the first command.

[0021] FIG. 4 is a flowchart showing an example of a processing procedure of the processor 1 shown in FIG. 1 . In FIG. 4 , the processor 1 determines whether or not gaze movement features are observed from the EEG channel of the frontal region indicated in the electroencephalogram data (step S1). This determination can be made, for example, by applying a simple threshold determination between channels. Alternatively, the presence or absence of gaze movement features can be determined by combining any method, such as eye movement pattern discrimination using ICA (implemented in EEGLAB, etc.). Gaze movement data of the gaze moving between visual stimuli (symbol patterns) and noise data (blinks and minute gaze movements) can be measured in advance by pre-calibration, and parameters for each method can be determined based on the data.

[0022] If no eye gaze movement feature is observed (No in step S1), the processor 1 performs frequency analysis on the EEG signal (step S2), compares the extracted frequency with the stimulation frequency of the symbol pattern (step S3), and identifies the symbol pattern (attention stimulus) that the user is paying attention to (step S4). Then, it outputs a command corresponding to the identified attention stimulus (step S5). Here, the processing of steps S2 to S5 is based on existing technology using SSVEP, and commands such as go forward, go backward, turn right or left, etc. are output.

[0023] On the other hand, if a gaze movement characteristic is detected in step S1 (Yes), the processor 1 outputs a command requiring quick response, such as a deceleration / stop operation. Note that commands such as stop only, deceleration only, or deceleration and stop may be assigned depending on the operation specifications.

[0024] 5 is a diagram showing an example of the flow of data in the BMI device 100 shown in Fig. 1. The user conveys his or her intention by focusing on one symbol pattern displayed on the target stimulus presentation display 12a of the display device 12. The electroencephalogram sensor 11 measures the user's electroencephalogram data, and this electroencephalogram data (EEG data) is passed to the command determination unit 1c.

[0025] The command determination unit 1c determines the first command based on the SSVEP, and also determines the second command based on information on eye movement detected from the electrooculography (EOG). The second command is a command that requires a quick response and is related to a crisis avoidance operation of the electric wheelchair 200. The command (operation command) generated in this manner is input to the electric wheelchair 200. The electric wheelchair 200 moves in quick response to the user's intention.

[0026] As described above, in one embodiment, in addition to the command determination process using only SSVEP, eye movement information is detected using electrooculography (EOG) detected by an electroencephalogram sensor. When eye movement information is detected, a new process is added in which the information is assigned to a crisis avoidance command such as a deceleration or stopping operation. This provides significant advantages mainly in terms of (1), (2), and (3).

[0027] (1) In terms of detection speed, EOG due to gaze shift occurs instantly, and its waveform is localized and distinctive. Therefore, it can be detected more immediately than when SSVEP is used alone. (2) In terms of danger avoidance, gaze shift occurs reflexively when a user senses a danger, such as a person running out onto the road, allowing for immediate danger detection. (3) In terms of safety and operability, gaze shift always occurs when a user switches commands (switches the stimulus being gazed at). Therefore, danger avoidance commands such as deceleration and stopping can be executed concomitantly. Since SSVEP is continuously detected for several seconds even when switching stimuli, SSVEP for the immediately preceding stimulus continues to be detected, and as a result, the immediately preceding command is not interrupted. In contrast, in the embodiment, danger avoidance commands are executed sequentially at the timing of stimulus switching, which is safe, and danger avoidance commands are automatically executed in conjunction with stimulus switching, reducing the operational burden on the user.

[0028] In the case of SSVEP alone, it is conceivable to treat blinking as a crisis avoidance command. However, from the viewpoints of (2) and (3) above, it can be concluded that eye movement is the best crisis avoidance command. Furthermore, by making the spacing between symbol patterns as wide as possible, the amount of eye movement can be increased, and the accuracy of detecting eye movement information can be improved.

[0029] Furthermore, since EOG can be detected from EEG channels (such as Fp1 and F7 channels) in the frontal region near the eyes, a major advantage of the embodiment is that there is no need to add a new measurement device. Incidentally, JP-A-2021-502659 (reference document) proposes a wheelchair BMI system that combines gaze information and SSVEP. However, the technology of this document requires a separate gaze measurement device. In contrast, the technology of the embodiment allows for easier system implementation.

[0030] For these reasons, according to the embodiment, it is possible to shorten the time required to detect commands and improve responsiveness. As a result, wheelchair danger avoidance operations can be performed immediately, improving safety. Furthermore, it is possible to instantly respond to danger avoidance operations such as deceleration and stopping operations without adding any measuring devices, making it possible to build a wheelchair BMI system that ensures the safety of the user.

[0031] It should be noted that the present invention is not limited to the above-described embodiments, and that the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0032] 1...Processor 1a...Display control unit 1b...Electroencephalogram data acquisition unit 1c...Command determination unit 2...Program memory unit 3...Data memory unit 4...Sensor I / F unit 5...Input / output I / F unit 6...Internal bus 11...Electroencephalogram sensor 12...Display device 12a...Target stimulus presentation display 100...Brain-machine interface device 200...Electric wheelchair 300...Actuator 400...Command.

Claims

1. A brain-machine interface device that generates commands to control a machine based on a user's electroencephalogram (EEG), comprising: a display control unit that displays visual symbols assigned unique blinking periods at different positions in the user's field of view; an EEG data acquisition unit that acquires EEG data including at least a steady-state visual evoked potential and an electrooculogram (EOG) from an EEG sensor worn by the user; and a command determination unit that identifies a first command from the steady-state visual evoked potential and the blinking period, and determines a second command that follows the first command based on gaze movement information of the user detected from the electrooculogram.

2. The brain-machine interface device according to claim 1, wherein the second command requires a more immediate response than the first command.

3. A brain-machine interface device as described in claim 2, wherein the first command and the second command are commands for controlling an actuator that drives a wheelchair, and the second command is a command related to a crisis avoidance operation of the wheelchair.

4. A command determination method by a computer processor that generates commands to control a machine based on a user's electroencephalogram, comprising the steps of: the processor displaying visual symbols assigned unique blinking periods at different positions in the user's visual field; the processor acquiring electroencephalogram data including at least a steady-state visual evoked potential and an electrooculogram from an electroencephalogram sensor worn by the user; the processor identifying a first command from the steady-state visual evoked potential and the blinking period; and the processor determining a second command following the first command based on information about the user's line of sight detected from the electrooculogram.

Images