Brain computer interface training system, training device, training method, and program

Abstract

One aspect of the present invention acquires data representing an action corresponding to an instruction recalled by a user, determines the type of the corresponding instruction on the basis of data representing the acquired action, selects a sensory stimulus previously associated with the determined type of the instruction, and executes processing for applying the selected sensory stimulus to the user. Meanwhile, the one aspect of the present invention acquires a brain wave pattern of the user when the sensory stimulus is applied, generates an instruction estimation model by using a set of the inputted instruction type and the acquired brain wave pattern as learning data, and stores the generated instruction estimation model.

Description

Brain-computer interface training system, training device, training method and program

[0001] One aspect of the present invention relates to a system for training a brain-computer interface used to operate a computer using electroencephalograms, and a training device, training method, and program used in this system.

[0002] There is a known technology that estimates recalled operational commands based on differences in brain waves. This technology is called a brain-computer interface (BCI). This technology makes it possible to operate a computer using brain waves, and is expected to make it easier for people with physical disabilities in particular to operate computers.

[0003] To use a BCI, for example, a method is used in which the type of command that is recalled in a person's brain and the corresponding brain wave pattern are learned as a set, and the type of recalled command is estimated based on the measured brain wave pattern. A typical method for inputting commands is motor imagery. For example, different brain wave patterns are observed when imagining moving the right hand and when imagining moving the left hand. Therefore, if different commands are associated with each motor imagery, commands can be input to the BCI.

[0004] However, the method of inputting commands using motor imagery may or may not be able to be used appropriately depending on the user, so a method is needed to train the BCI so that it can accurately estimate the input commands using the user's motor imagery.

[0005] Therefore, for example, Non-Patent Document 1 proposes a technology for users who are unable to properly input commands into a BCI, in which the user is made to imagine the movement while holding a ball, making it easier for the user to imagine the movement and enabling the BCI to recognize the command with high accuracy.

[0006] Sangin Park et al. "Improving motor imagery-based brain-computer interface performance based on sensory stimulation training: an approach focused on poorly performing users." Frontiers in Neuroscience 15 (2021): 732545.

[0007] However, even when the technology described in Non-Patent Document 1 is applied, there are individual differences in users' adaptability to motor imagery, and therefore, sufficient estimation accuracy may not be achieved compared to people who can properly perform motor imagery. Furthermore, since the estimation accuracy of commands by BCI is determined by differences in EEG patterns due to motor imagery, it is necessary to use body parts with large movements, such as the right and left hands, as body parts used for motor imagery. For this reason, it is difficult to use body parts with small differences in movement, such as moving one of the five fingers on the right hand, for motor imagery.

[0008] This invention has been made with the above-mentioned circumstances in mind, and aims to provide a technology that enables highly accurate command estimation by reducing the decline in accuracy of command estimation due to individual differences in recollection behavior and differences in body parts associated with recollection behavior.

[0009] In order to solve the above problem, one aspect of a brain-computer interface training device or training method according to the present invention acquires data representing an action corresponding to a command recalled by a user, determines a type of the corresponding command based on the acquired data representing the action, selects a sensory stimulus pre-associated with the determined type of command, and executes a process for applying the selected sensory stimulus to the user, acquires an electroencephalogram pattern of the user when the sensory stimulus is applied, generates a command estimation model using a set of the input type of command and the acquired electroencephalogram pattern as learning data, and stores the generated command estimation model.

[0010] According to one aspect of the present invention, conditioning training is performed using sensory stimuli for actions corresponding to commands recalled by the user, and the results are used as training data to generate a command estimation model. Therefore, even if the component of the action corresponding to the recall itself is small, it is possible to obtain EEG patterns that reflect the conditioned reflex caused by sensory stimuli and that differ significantly for each action. This makes it possible to improve the accuracy of command estimation when multiple types of actions are selectively imagined and multiple types of commands are input. As a result, even when the user performs small motor recall, such as moving any finger, the brain-computer interface can accurately estimate the type of input command.

[0011] Furthermore, by performing conditioning training using sensory stimuli in advance, the characteristics of the electroencephalogram patterns corresponding to the commands recalled by the user become more pronounced. As a result, even users who are not good at actions associated with recall, such as motor recall, will be able to generate more pronounced electroencephalogram patterns. This reduces the influence of individual differences in actions associated with recall, allowing any user to input commands accurately through recall.

[0012] In other words, according to one aspect of the present invention, a technology can be provided that enables highly accurate command estimation by reducing the decline in accuracy of command estimation due to individual differences in recollection actions and differences in body parts associated with recollection actions.

[0013] Fig. 1 is a block diagram showing an example of a brain-computer interface training system according to an embodiment of the present invention. Fig. 2 is a block diagram showing the software configuration of a brain-computer interface training device according to an embodiment of the present invention. Fig. 3 is a flowchart showing an example of the processing procedure and processing content of training control executed by the control unit of the brain-computer interface training device shown in Fig. 2.

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] [One embodiment] (Summary) In one embodiment of the present invention, when training a brain-computer interface (hereinafter referred to as BCI) that uses motor imagery as a recall action for inputting commands, conditioning is performed using sensory stimuli to generate a conditioned reflex in the brain, thereby increasing the changes in brain wave patterns corresponding to the motor imagery.

[0016] In other words, during the BCI training period, conditioning training is performed for multiple types of motor imagery corresponding to each command using different sensory stimuli, including visual, auditory, tactile, olfactory, and gustatory stimuli, to the user, thereby making the changes in the brain wave patterns corresponding to each motor imagery more pronounced.

[0017] For example, during a training period, a visual stimulus (e.g., a light stimulus) is presented to the user when imagining moving the right hand, and an auditory stimulus (e.g., a beep) is presented to the user when imagining moving the left hand. Repeating this conditioning training changes the brain's responses, such that the visual cortex responds when the user imagines moving the right hand, and the auditory cortex responds when the user imagines moving the left hand. After the conditioning training, the user's brain's visual cortex responds even without a visual stimulus when imagining moving the right hand, and the auditory cortex responds even without an auditory stimulus when imagining moving the left hand.

[0018] The visual cortex is located in the occipital lobe, and the auditory cortex is located in the temporal lobe. Therefore, after conditioning training, the difference in the brain wave patterns generated when the user imagines moving their right hand and when imagining moving their left hand becomes more pronounced, resulting in improved accuracy in estimating commands by the BCI.

[0019] Similarly, the tactile, olfactory, and gustatory areas are all located in different parts of the brain. Therefore, for example, by performing conditioning training for five types of motor imagery using stimulation of each of the five senses, the differences in the EEG patterns corresponding to the five types of motor imagery can be made more pronounced, thereby improving the accuracy of BCI estimation for command input using the five types of motor imagery.

[0020] Therefore, according to one embodiment of the present invention, by performing the above-mentioned conditioning training for each user and having the BCI learn the results, it is possible to reduce individual differences in users' motor imagery, and regardless of which user inputs a command using motor imagery, the BCI can estimate the command with high accuracy.

[0021] In one embodiment of the present invention, commands based on motor imagery are identified based on differences in EEG patterns due to conditioned reflexes in response to sensory stimuli. Therefore, BCI can estimate commands even when the EEG pattern of the motor imagery itself is small. For example, even when a user imagines a small physical movement, such as moving one finger, it is possible to accurately estimate the type of command.

[0022] For example, a user is trained to receive visual stimuli when imagining moving the thumb, auditory stimuli when imagining moving the index finger, tactile stimuli when imagining moving the middle finger, olfactory stimuli when imagining moving the ring finger, and gustatory stimuli when imagining moving the little finger, and the BCI learns the results. In this way, when the user imagines moving each of the five fingers, the sensory cortex corresponding to each image reacts, generating significantly different electroencephalogram (EEG) patterns. As a result, the BCI can accurately identify commands based on the user's motor imagery using the five fingers based on the differences in these EEG patterns.

[0023] Note that the above example has been described using an example of inputting five types of commands using an image of selectively moving five fingers, but different commands may be input using an image of selectively moving two or more fingers but not more than four, or the movement of the arms may be used in combination with the movement of multiple fingers, or multiple types of commands may be associated with the movement of not only the arms and fingers but also the movements of both legs, neck, eyes, mouth, etc.

[0024] (Configuration Example) (1) System FIG. 1 is a block diagram showing an example of the configuration of a BCI training system according to an embodiment of the present invention.

[0025] A BCI training system according to one embodiment of the present invention comprises a BCI training device CS, such as a personal computer, as its core, to which are connected a data glove GS and an electroencephalogram sensor BS as sensing devices, and further to which are connected a display device DP, a speaker SP, an electrical stimulus generator ES, an aroma diffuser AD, and an electric taste generator ET as devices for generating sensory stimuli.

[0026] The data glove GS is worn on the five fingers of the user to be trained, detects the movements of each of the user's five fingers, and outputs glove sensor data representing the detection results to the BCI training device CS.

[0027] The electroencephalogram sensor BS is worn on the head of the user, measures electroencephalograms generated from different parts of the user's brain, and outputs the measurement data to the BCI training device CS.

[0028] The display device DP is, for example, a display provided in the BCI training device CS, and displays a light emission signal generated in the BCI training device CS, thereby providing the user with a visual stimulus using light.

[0029] The speaker SP is, for example, a speaker provided in the BCI training device CS, and outputs a sound signal generated in the BCI training device CS in an amplified manner to provide the user with an auditory stimulus through sound.

[0030] The electrical stimulation generator ES is worn in contact with the skin, for example, on the user's arm, and provides tactile stimulation to the user by generating electrical stimulation of the skin in accordance with the electrical stimulation generation signal output from the BCI training device CS.

[0031] The aroma diffuser AD is placed, for example, near the face of the user, and stimulates the user's sense of smell by emitting an aroma in accordance with an aroma generation signal output from the BCI training device CS.

[0032] The electric taste generator ET is positioned, for example, in the user's oral cavity so as to contact the tongue, and provides taste stimulation to the user by generating an electric taste in response to an electric taste generation signal output from the BCI training device CS.

[0033] (2) BCI Training Device CS FIG. 2 is a block diagram showing the software configuration of the BCI training device CS according to one embodiment of the present invention.

[0034] As shown in FIGS. 1 and 2, the BCI training device CS includes a control unit 1 that uses a hardware processor such as a central processing unit (CPU), and is connected to this control unit 1 via a bus 6 with a storage unit having a program storage unit 2 and a data storage unit 3, a sensor interface (hereinafter, interface will be abbreviated as I / F) unit 4, and an input / output I / F unit 5.

[0035] The data glove GS and the electroencephalogram sensor BS are connected, for example, via a signal cable, to the sensor I / F unit 4. The sensor I / F unit 4 receives the glove sensor data output from the data glove GS and the electroencephalogram measurement data output from the electroencephalogram sensor BS.

[0036] The display device DP, speaker SP, electrical stimulation generator ES, aroma diffuser AD, and electrical gustatory generator ET are connected, for example, via signal cables, to the input / output I / F unit 5. The input / output I / F unit 5 outputs a light emission signal, a sound signal, an electrical stimulation generation signal, an aroma generation signal, and an electrical gustatory generation signal to the display device DP, speaker SP, electrical stimulation generator ES, aroma diffuser AD, and electrical gustatory generator ET, respectively, under the control of the control unit 1.

[0037] Note that, instead of signal cables, a low-power wireless interface such as Bluetooth (registered trademark) may be used as a connection means between the sensor I / F unit 4 and the EEG sensor BS and data glove GS, and between the input / output I / F unit 5 and the display device DP, speaker SP, electrical stimulation generator ES, aroma diffuser AD, and electrical taste generator ET. Using a wireless interface can reduce the burden on the user when undergoing training.

[0038] The program storage unit 2 is, for example, a combination of a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD) as a storage medium that can be written to and read from at any time, and a non-volatile memory such as a read only memory (ROM), and stores application programs necessary for executing various processes related to one embodiment of the present invention, in addition to middleware such as an operating system (OS).

[0039] The data storage unit 3 is, for example, a combination of a non-volatile memory such as an HDD or SSD as a storage medium that can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory), and its storage area includes a command type determination data storage unit 31, a stimulus control data storage unit 32, an electroencephalogram pattern storage unit 33, and a command estimation model storage unit 34.

[0040] The command type determination data storage unit 31 stores the determination data of the type of finger moved by the user together with data indicating the detection time, in association with the user's identification information (hereinafter referred to as user ID) as information indicating the command type.

[0041] The stimulus control data storage unit 32 stores in advance sensory stimulus control data that indicates the type of stimulus and the content of that stimulus in association with the type of finger moved by the user.

[0042] The electroencephalogram pattern storage unit 33 stores the electroencephalogram patterns detected from the user's electroencephalogram measurement data together with data indicating the time of detection in association with the user ID.

[0043] The command estimation model storage unit 34 stores parameters related to the trained BCI command estimation model. The BCI command estimation model may be a machine learning model using, for example, a neural network, but other machine learning models such as a support vector machine (SVM) may also be used.

[0044] The control unit 1 includes, as a processing function unit according to one embodiment of the present invention, a training control unit 10 that controls the entire training process in an integrated manner, and a glove sensor data acquisition processing unit 11, a command judgment processing unit 12, a sensory stimulus generation processing unit 13, an electroencephalogram measurement data acquisition processing unit 14, and a command estimation model learning processing unit 15, each of which performs processing under the control of the training control unit 10.

[0045] The training control unit 10 and each of the processing units 10 to 15 are realized by causing a hardware processor of the control unit 1 to execute an application program stored in the program storage unit 2. Note that the training control unit 10 and a part or all of each of the processing units 11 to 15 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0046] Every time the user to be trained moves a finger corresponding to a type of command, the glove sensor data acquisition processing unit 11 receives glove sensor data output from the data glove GS via the sensor I / F unit 4. The glove sensor data represents the movement of the finger of the user to be trained.

[0047] Each time the glove sensor data is received, the command determination processing unit 12 determines the type of finger moved by the user based on the glove sensor data, and stores information indicating the determined type of finger together with information indicating the time of detection in the command type determination data storage unit 31, corresponding to the user ID as information indicating the type of training command.

[0048] Each time the command type is determined by the command determination processing unit 12, the sensory stimulus generation processing unit 13 reads the sensory stimulus control data associated with the command type from the stimulus control data storage unit 32. Then, in accordance with the read sensory stimulus control data, the sensory stimulus generation processing unit 13 outputs a signal for generating a stimulus from the input / output I / F unit 5 to any one of the display device DP, speaker SP, electric stimulus generator ES, aroma diffuser AD, and electric taste generator ET.

[0049] Each time a signal for generating the stimulation is output, the electroencephalogram measurement data acquisition processing unit 14 acquires the electroencephalogram measurement data output from the electroencephalogram sensor BS within a certain period from the time of output via the sensor I / F unit 4. Then, the electroencephalogram measurement data acquisition processing unit 14 identifies an electroencephalogram pattern from the acquired electroencephalogram measurement data, and stores data indicating the type of the identified electroencephalogram pattern together with information indicating the time the electroencephalogram was detected in the electroencephalogram pattern storage unit 33 in association with the user ID.

[0050] The training control unit 10 repeatedly executes a series of processes performed by the glove sensor data acquisition processing unit 11 to the command estimation model learning processing unit 15 a predetermined number of times for each type of command, for example, for each of the five fingers. Then, the training control unit 10 stores in the command type determination data storage unit 31 and the electroencephalogram pattern storage unit 33 a number of sets of command types and electroencephalogram patterns necessary and sufficient for learning the command estimation model.

[0051] The command estimation model learning processing unit 15 reads out, for each command type, a plurality of sets of command types and electroencephalogram patterns from the command type determination data storage unit 31 and the electroencephalogram pattern storage unit 33, and uses the read out plurality of sets as learning data to learn a command estimation model. Then, the command estimation model learning processing unit 15 stores parameters representing the learned command estimation model in the command estimation model storage unit 34.

[0052] (Example of Operation) Next, an example of operation of the BCI training device CS configured as described above will be described. Here, an example will be described in which the image of moving each of the five fingers is used as a command, and the command is input during training by slightly moving the five fingers.

[0053] FIG. 3 is a flowchart showing an example of a series of processing steps and processing contents related to training control executed by the control unit 1 of the BCI training device CS.

[0054] (1) Input of training commands Prior to training, the user to be trained is fitted with an electroencephalogram sensor BS, a data glove GS, an electrical stimulation generator ES, and an electrical taste generator ET on various parts of his or her body, and an aroma diffuser AD is placed near the user.

[0055] In this state, suppose that the user of the training subject operates an input device (not shown) to input a training start request to the BCI training device CS. When the control unit 1 of the BCI training device CS detects the training start request in step S10, it instructs the user of the training subject to move his or her fingers as a training command. The instruction may be given by, for example, displaying a message on the display device DP or outputting a voice message from the speaker SP.

[0056] In response to the above instruction, when the user first slightly moves their thumb while imagining moving their thumb in their mind, the movement of the thumb at this time is detected by the data glove GS and input to the BCI training device CS as glove sensor data.

[0057] The control unit 1 of the BCI training device CS receives the glove sensor data in step S11 under the control of the glove sensor data acquisition processing unit 11, and then determines the type of finger moved by the user based on the received glove sensor data in step S12 under the control of the command determination processing unit 12. The command determination processing unit 12 then stores the determined finger type together with information indicating the detection time as command type determination information in the command type determination data storage unit 31 in association with the user ID.

[0058] (2) Providing Sensory Stimuli The control unit 1 of the BCI training device CS then performs processing under the control of the sensory stimulus generation processing unit 13 to provide the user with a predetermined sensory stimulus for the type of command corresponding to the moved finger, in order to condition the image of the finger movement.

[0059] That is, in step S13, the sensory stimulus generation processing unit 13 reads out sensory stimulus control data representing the type of sensory stimulus corresponding to the type of command determined by the command determination processing unit 12 from the stimulus control data storage unit 32. Then, the sensory stimulus generation processing unit 13 outputs the read sensory stimulus control data from the input / output I / F unit 5 to any one of a plurality of devices for generating stimuli.

[0060] For example, suppose a visual stimulus is associated with the type of command corresponding to the image of moving the thumb. In this case, the sensory stimulus generation processing unit 13 outputs a light emission signal for generating the visual stimulus from the input / output I / F unit 5 to the display device DP. As a result, for example, a flash of light is emitted from the display device DP, providing the visual stimulus to the user.

[0061] Furthermore, if an auditory stimulus is associated with a command type corresponding to an image of moving an index finger, for example, the sensory stimulus generation processing unit 13 outputs a sound signal for generating the auditory stimulus from the input / output I / F unit 5 to the speaker SP. As a result, for example, a beep is generated from the speaker SP, providing the auditory stimulus to the user.

[0062] Furthermore, if a tactile stimulus is associated with a command type corresponding to, for example, the image of moving the middle finger, the sensory stimulus generation processing unit 13 outputs an electrical stimulus signal for generating the tactile stimulus from the input / output I / F unit 5 to the electrical stimulus generator ES. As a result, the electrical stimulus generator ES applies an electrical tactile stimulus to the skin of the user's arm.

[0063] Furthermore, if an olfactory stimulus is associated with the type of command corresponding to the image of moving a ring finger, for example, the sensory stimulus generation processing unit 13 outputs an aroma generation signal for generating the olfactory stimulus from the input / output I / F unit 5 to the aroma diffuser AD. As a result, aroma is sprayed from the aroma diffuser AD, providing the user with an olfactory stimulus.

[0064] Similarly, if a gustatory stimulus is associated with a command type corresponding to, for example, the image of moving the little finger, the sensory stimulus generation processing unit 13 outputs an electric gustatory signal for generating the gustatory stimulus from the input / output I / F unit 5 to the electric gustatory generator ET. As a result, the electric gustatory stimulus is given to the user by the electric gustatory sense by the electric gustatory generator ET.

[0065] (3) Detection of EEG Patterns After the sensory stimulation is applied to the user, the control unit 1 of the BCI training device CS subsequently receives, in step S14, the EEG measurement data of the user output from the EEG sensor BS within a certain time period from the application of the sensory stimulation, via the sensor I / F unit 4, under the control of the EEG measurement data acquisition processing unit 14. The EEG measurement data acquisition processing unit 14 then identifies an EEG pattern from the received EEG measurement data, and stores the identified EEG pattern together with information indicating the detection time in the EEG pattern storage unit 33 in association with the user ID.

[0066] (4) Repeated Processing for Each Type of Command The control unit 1 of the BCI training device CS repeats the series of data collection processes in steps S11 to S16 for each type of command multiple times each time the user moves the finger corresponding to that type of command, under the control of the training control unit 10. The number of repetitions is set to the number of times required for learning the command estimation model, and the repetition period is set to, for example, 4 seconds.

[0067] By repeating the data collection process multiple times, multiple data sets of command types and corresponding electroencephalogram patterns for each command are stored in the command type determination data storage unit 31 and the electroencephalogram pattern storage unit 33. The command types and corresponding electroencephalogram patterns are linked by the detection times assigned to both.

[0068] The training control unit 10 determines in step S15 whether the data collection process has been completed multiple times for each command, and if it determines that the data collection process has been completed, instructs the command estimation model learning processing unit 15 to perform learning processing.

[0069] (5) Learning of the Command Estimation Model When a learning process is instructed by the training control unit 10, the command estimation model learning processing unit 15 in step S17 reads, for each command type, a plurality of sets of stored command types and corresponding electroencephalogram patterns from the command type determination data storage unit 31 and the electroencephalogram pattern storage unit 33. Then, the command estimation model learning processing unit 15 uses the read-out plurality of sets to learn the command estimation model.

[0070] Then, when the learning process for all types of commands is completed, the command estimation model learning processing unit 15 stores the parameters of the learned command estimation model in the command estimation model storage unit 34. Note that, for example, deep learning is used as the learning algorithm, but other machine learning algorithms may also be used.

[0071] (6) Learning a Command Estimation Model for Other Users The control unit 1 of the BCI training device CS performs training for other users to be trained by following the same processing procedure, conditioning sensory stimuli for each type of command corresponding to finger movements, and uses the resulting sets of command types and EEG patterns as training data to learn a command estimation model. Then, when an instruction to end training is detected in step S18, the control unit 1 of the BCI training device CS, under the control of the training control unit 10, ends training control and returns to a standby state.

[0072] (7) Estimation of Command Types Using Trained Command Estimation Models When using the BCI, a user first inputs their user ID and selects the command estimation model corresponding to the user from among the command estimation models stored in the BCI training device CS. Then, the user mentally imagines moving their fingers corresponding to the type of command they want to input. Then, the user's brain waves are measured by the brain wave sensor BS and input to the BCI training device CS.

[0073] When the control unit 1 of the BCI training device CS acquires the electroencephalogram measurement data, the control unit 1 identifies an electroencephalogram pattern from the acquired electroencephalogram measurement data using the electroencephalogram measurement data acquisition processing unit 14. Then, the control unit 1 inputs the feature quantities of the identified electroencephalogram pattern as explanatory variables into a corresponding command estimation model, and acquires information indicating the type of command output as a target variable from the command estimation model.

[0074] Here, the measured electroencephalogram reflects the image of moving a finger, but is a signal with large fluctuations that reflects the conditioned reflex caused by the sensory stimulation given in the conditioning training conducted in advance. Therefore, the electroencephalogram pattern identification process can accurately identify the electroencephalogram pattern, and as a result, the command estimation model can accurately estimate the type of input command.

[0075] The information representing the type of command output from the command estimation model is used, for example, for processing a predetermined task in the BCI training device CS. Note that the information representing the type of command may also be transmitted from the BCI training device CS to another personal computer or server computer and used to control a remote robot or an avatar in the Metaverse; the uses of the commands are not limited to the above examples.

[0076] (Effects) As described above, in one embodiment, when associating command types with images of moving different fingers during a training period, the data glove GS detects the movements of the training fingers, determines and saves the command type from the sensor data, provides the user with a sensory stimulus associated with the determined command type, detects the user's brain waves at this time with the brain wave sensor BS, and identifies and saves an brain wave pattern from the measurement data. The above process is then repeated multiple times for each command type to accumulate multiple sets of command types and brain wave patterns, and the accumulated multiple sets are used as training data to train a command estimation model.

[0077] Therefore, conditioning training is carried out using sensory stimuli in response to the image of moving the fingers, and even though the component of the motor imagery using the fingers itself is small, it is possible to obtain EEG patterns that reflect the conditioned reflex caused by sensory stimuli and that differ significantly from finger to finger.This makes it possible to improve the estimation accuracy when inputting multiple types of commands using the image of selectively moving multiple fingers.

[0078] As a result, the BCI can accurately estimate the type of command corresponding to the image of moving a finger, and can accurately estimate the type of command even when the user imagines a small physical movement, such as moving any of their fingers.

[0079] Furthermore, conditioning training using sensory stimuli produces distinctive EEG patterns, which means that even users who are not good at motor imagery using their fingers will produce distinctive EEG patterns. This reduces the influence of individual differences in motor imagery, allowing any user to accurately input commands by imagining moving their fingers.

[0080] Other Embodiments (1) In one embodiment, a case where one type of sensory stimulus corresponds to one type of command has been described as an example. However, this is not limiting, and one type of sensory stimulus may correspond to multiple types of commands.

[0081] For example, visual stimuli are associated with the movement of the thumb on the right or left hand, and auditory stimuli are associated with the movement of the index finger on the right or left hand. Generally, the right and left hands are easily distinguishable in electroencephalogram patterns during motor imagery, so even if the same sensory stimuli are associated with the right and left hands, it is relatively easy to determine whether the thumb on the right or left hand was moved from the electroencephalogram patterns of the motor and visual cortices.

[0082] (2) On the other hand, because the olfactory and gustatory areas are located deep within the brain, they may be difficult to express in electroencephalogram (EEG) patterns. Therefore, conditioning training can be performed by associating olfactory stimuli with the thumb and gustatory stimuli with the little finger. This makes it possible to make the difference in EEG patterns between the motor areas more pronounced, thereby improving the accuracy of command recognition between the thumb and little finger.

[0083] (3) Furthermore, in one embodiment, a case where motor imagery using hands or fingers is used as an example of a command input method has been described. However, this is not limited to this, and commands may also be input by, for example, imagining words, figures, or pictures in one's mind. In this case, too, conditioning training is performed during the training period by associating sensory stimuli with words, figures, or pictures, and the results are used to train a command estimation model, thereby enabling accurate estimation of the type of command.

[0084] (4) In the embodiment, the BCI training device CS is configured as a personal computer. However, the BCI training device CS may be configured as a server computer located on the Web or in the cloud. In this case, the BCI command estimation model after training may be downloaded from the server computer to the personal computer used by each user, so that commands can be input using the BCI thereafter.

[0085] (5) In addition, the functional configuration of the BCI training device, the processing procedures and processing contents of the training control, the configuration of various sensors such as EEG sensors, and the configuration of devices for generating each sensory stimulus can be modified and implemented in various ways within the scope of the gist of this invention.

[0086] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0087] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0088] CS...BCI training device GS...data glove BS...EEG sensor DP...display device SP...speaker ES...electrical stimulation generator AD...aroma diffuser ET...electrical taste generator 1...control unit 2...program storage unit 3...data storage unit 4...sensor I / F unit 5...input / output I / F unit 6...bus 10...training control unit 11...glove sensor data acquisition processing unit 12...command determination processing unit 13...sensory stimulation generation processing unit 14...EEG measurement data acquisition processing unit 15...command estimation model learning processing unit 31...command type determination data storage unit 32...stimulus control data storage unit 33...EEG pattern storage unit 34...command estimation model storage unit

Claims

1. A brain-computer interface training system for training a brain-computer interface that operates a computer using brain waves, comprising: - a brain wave sensor for measuring the brain waves of a user; - an action sensor for detecting the actions of the user associated with commands recalled by the user; - a stimulation device for selectively generating and applying a plurality of types of sensory stimuli to the user; and - a training device connected to the brain wave sensor, the action sensor, and the stimulation device, - wherein the training device includes: - a first processing unit that acquires detection data representing the actions of the user from the action sensor and determines the type of the corresponding command based on the acquired detection data representing the actions; - a second processing unit that selects the sensory stimulus pre-associated with the determined type of command from among the plurality of types of sensory stimuli and controls the stimulation device to apply the selected sensory stimulus to the user; - a third processing unit that acquires measurement data representing the brain waves of the user when the sensory stimulus is applied from the brain wave sensor and determines a brain wave pattern from the acquired measurement data; and - a fourth processing unit that generates a command estimation model using a set consisting of the determined type of command and the brain wave pattern as learning data and stores data representing the generated command estimation model. A brain-computer interface training system.

2. The action sensor detects the movement of at least two fingers out of the five fingers of the user, and the first processing unit determines the finger moved by the user based on the detection data representing the action and determines the type of the command corresponding to the determined finger. The brain-computer interface training system according to claim 1.

3. The stimulation device generates at least two stimuli among visual stimuli, auditory stimuli, tactile stimuli, olfactory stimuli, and gustatory stimuli. The brain-computer interface training system according to claim 1.

4. A brain-computer interface training device for training a brain-computer interface that operates a computer using brain waves, comprising: a first processing unit that acquires data representing an action corresponding to an instruction recalled by a user and determines the type of the corresponding instruction based on the acquired data representing the action; a second processing unit that selects a sensory stimulus pre-associated with the determined type of instruction and executes a process for applying the selected sensory stimulus to the user; a third processing unit that acquires a brain wave pattern of the user when the sensory stimulus is applied; and a fourth processing unit that generates an instruction estimation model using a set consisting of the determined type of instruction and the acquired brain wave pattern as learning data and stores the generated instruction estimation model.

5. The brain-computer interface training device according to claim 4, wherein the first processing unit acquires measurement data representing the movement of at least two fingers among the five fingers of the user as the action corresponding to the instruction recalled by the user, and determines the type of the corresponding instruction based on the acquired measurement data.

6. The brain-computer interface training device according to claim 4, wherein the second processing unit selects the sensory stimulus pre-associated with the type of instruction from among at least two stimuli of visual stimulus, auditory stimulus, tactile stimulus, olfactory stimulus, and gustatory stimulus, and executes a process for applying the selected sensory stimulus to the user.

7. A brain-computer interface training method for training a brain-computer interface that operates a computer using brain waves, using an information processing device, the information processing device obtaining measurement data representing an action corresponding to an instruction recalled by a user, and determining the type of the corresponding instruction based on the obtained measurement data; the information processing device selecting a sensory stimulus previously associated with the determined type of instruction, and executing a process for applying the selected sensory stimulus to the user; the information processing device obtaining a brain wave pattern of the user when the sensory stimulus is applied; and the information processing device generating an instruction estimation model using, as learning data, a set consisting of the determined type of instruction and the obtained brain wave pattern, and storing the generated instruction estimation model.

8. A program that causes a processor included in the brain-computer interface training device to execute at least one of the processes performed by the first processing unit, the second processing unit, the third processing unit, and the fourth processing unit included in the brain-computer interface training device according to any one of claims 4 to 6.

Images