Operation control device, operation control method, and program

Abstract

To enable appropriate brain wave commands to be executed without being affected by irrelevant stimuli.SOLUTION: An operation control device comprises: a brain information acquisition section that acquires brain information of a user; a determination section that determines whether the brain information acquired by the brain information acquisition section corresponds to a first command for actuating an object to be operated, a second command obtained by combining one or more images based on the five senses, and a third command for determining execution of the first command; and an execution section that executes operation corresponding to the first command when the determination section determines that brain information corresponding to the first command, brain information corresponding to the second command, and the third command have all been acquired.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an operation control device, an operation control method, and a program. [Background technology]

[0002] In recent years, advances in technology for measuring brain information have made it possible to realize a brain-machine interface (BMI), which is an interface that directly connects the brain and machines. One such technology is described in Patent Document 1, for example. Patent Document 1 discloses a system that obtains a plurality of steady-state visual evoked response potential signals from a user when the user receives a stimulus, and supplies a system command signal to the user. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-117957 Summary of the Invention [Problem to be solved by the invention]

[0004] With conventional BMIs, when an unrelated stimulus is received when executing an EEG command, that stimulus may appear in the EEG and cause the command to be executed. For example, if you think to a robot "move forward," and the command is to make the robot move forward, just having someone nearby say "move forward" will cause the brain to automatically recall "move forward" and issue the command. Also, when operating a robot while watching its movements, if the image of the robot moving forward is displayed on a monitor, the operator's brain will automatically continue to recall "move forward," causing the command to be issued automatically.

[0005] The present invention has been made in view of the above, and has as its object to make it possible to execute appropriate electroencephalogram commands without being influenced by irrelevant stimuli. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the operation control device of the present invention comprises a brain information acquisition unit that acquires brain information of a user, a determination unit that determines whether the brain information acquired by the brain information acquisition unit is brain information corresponding to a first command based on a language system and a second command based on a system other than a language system, and an execution unit that executes processing corresponding to the first command when the determination unit determines that it has acquired brain information corresponding to the first command and brain information corresponding to the second command.

[0007] The operation control method of the present invention includes the steps of acquiring brain information of a user, determining whether the acquired brain information corresponds to a first command based on a language system and a second command based on a system other than a language system, and executing processing corresponding to the first command when it is determined that brain information corresponding to the first command and brain information corresponding to the second command have been acquired.

[0008] The program of the present invention causes a computer to execute the steps of acquiring brain information of a user, determining whether the acquired brain information corresponds to a first command based on a language system and a second command based on a system other than a language system, and executing processing corresponding to the first command when it is determined that brain information corresponding to the first command and brain information corresponding to the second command have been acquired. [Effects of the Invention]

[0009] According to the present invention, it is possible to effectively execute appropriate electroencephalogram commands without being affected by irrelevant stimuli. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an operation control device according to this embodiment. [Figure 2]FIG. 2 is a flowchart showing the flow of processing in the operation control device according to this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An operation control device, an operation control method, and a program according to the present invention will be described in detail below with reference to the accompanying drawings. However, the present invention is not limited to the following embodiments. [Operation control device] FIG. 1 is a block diagram showing an example of the configuration of an operation control device according to this embodiment.

[0012] 1, the operation control device 10 controls the operation of an operation target object 50 based on the brain information of a user. The operation control device 10 includes an electroencephalogram acquiring unit (brain information acquiring unit) 11, an electroencephalogram decoder 12, an execution command buffer 13, an image command buffer 14, a password buffer 15, a storage unit 16, a processing unit 17, a display control unit 18, a display unit 19, and a communication unit 20.

[0013] The operation object 50 is, for example, a robot, but is not limited to a robot and may be any device that operates through communication. Although not shown, the operation object 50 is provided with a drive unit, a control unit that controls the drive unit, and a communication unit that can receive control signals, etc. When the communication unit of the operation object 50 receives a control signal, the control unit drives the drive unit to operate the robot.

[0014] The electroencephalogram acquiring unit 11 can be worn on the user's head. The electroencephalogram decoder 12, processing unit 17, display control unit 18, display unit 19, and communication unit 20 may be worn by the user, or may be placed in predetermined positions without being worn. Furthermore, the electroencephalogram acquiring unit 11, electroencephalogram decoder 12, processing unit 17, display control unit 18, display unit 19, and communication unit 20 may be provided integrally or separately.

[0015] The brain wave acquisition unit 11 acquires brain waves, which are brain information of the user. The brain wave acquisition unit 11 has, for example, an electric sensor (e.g., an electrode) that detects brain waves generated from a weak current flowing in the neural network of the brain. The brain wave acquisition unit 11 detects the potential (electrical signal) of a weak current when the user receives an external stimulus or based on the user's thoughts, such as their intentions. Note that the brain information acquisition unit is not limited to the brain wave acquisition unit 11. The brain information acquisition unit may also acquire, for example, blood flow due to brain activity, which is brain information of the user, by, for example, near-infrared light measurement. The brain wave acquisition unit 11 is connected to the brain wave decoder 12 and transmits brain waves, which are brain information acquired from the user, to the brain wave decoder 12.

[0016] The electroencephalogram decoder 12 restores the user's electrical electroencephalogram signals acquired by the electroencephalogram acquisition unit 11 into the user's thought information. In this case, the relationship between the user's electrical electroencephalogram signals and the user's thought information is linked in advance. In this case, the relationship between the electroencephalogram signals and the user's thought information is linked using machine learning such as deep learning.

[0017] The execution command buffer 13 temporarily stores the first command determined by the determination unit 21 based on the user's thought information. The first command is a command based on brain information in a language system (character information). The first command is brain information in the left brain and is a command for operating the operation object 50, such as commands such as "forward," "backward," "stop," "accelerate," "decelerate," "turn right," and "turn left." In this case, the relationship between the first command and the operation of the operation object 50 is linked in advance. The execution command buffer 13 is capable of sending and receiving various data to and from the processing unit 17.

[0018] The first command is not limited to a concept indicating an instructed action, but may be one to which a numerical value or the like is added to complement the concept indicating the instructed action. For example, the electroencephalogram decoder 103 may restore the user's thought information based on the electrical signal acquired by the electroencephalogram acquisition unit 11, and at the same time, detect the intensity of the electroencephalogram as a decimal number ranging from 0 to 1, and add the intensity (numerical value) to the user's thought information to set the first command. In this case, the numerical value of the first command is a count of the maximum torque output of the drive device provided in the operation target object 50.

[0019] The image command buffer 14 temporarily stores the second command determined by the determination unit 21 based on the user's thought information. The second command is a command based on brain information based on an image system. The second command is brain information in the right brain, and is a command that indicates the intention to execute the operation content of the operation target object 50, and is a command for determining the execution of the first command. The second command corresponding to the first command is set in advance by the user and stored in the storage unit 16. The second command may be set by the processing unit 17, stored in the storage unit 16, and notified to the user in advance. The image command buffer 14 is capable of sending and receiving various data to and from the processing unit 17.

[0020] The second command is an image of the senses of sight, hearing, touch, smell, etc., which means the conditions to be imagined in the mind, and is a combination of one or more abstract concepts. For example, the second command may be a color image based on sight, a sound image based on hearing, a temperature image based on touch, or an odor image based on smell. Furthermore, the second command may be a single command (image) regardless of the content of the first command, or different commands (images) may be set depending on the type of the first command.

[0021] For example, when the first command is a command based on "starting" an operation, the second command corresponding to the first command is set to a visually-based "blue." On the other hand, when the first command is a command based on "stopping" an operation, the second command corresponding to the first command is set to a visually-based "red." Also, when the first command is a work-based command, the second command corresponding to the first command is set to a visually-based "green." On the other hand, when the first command is a private command, the second command corresponding to the first command is set to a visually-based "yellow." The second command for the first command may be set to a related image, or to a completely unrelated image.

[0022] For example, auditory information acquired through the ears is transmitted from the auditory system to the comprehension system, emotion system, thinking system, and communication system in the left brain, and is stored as language in the memory system. On the other hand, for example, auditory information acquired through the ears is stored as images in the memory system in the right brain. Therefore, the parts of the brain that are activated when a user recalls a linguistic command as a first command differ from the parts that are activated when a user recalls an image-based command as a second command. Therefore, by confirming the user's linguistic intention and image-based intention, it is possible to confirm the operation content of the operation target object 50 and the intention to execute the operation content.

[0023] The password buffer 15 temporarily stores a password, which is a third command determined by the determination unit 21 based on the user's thought information. The password, which is the third command, is a command for further determining whether to execute the first command. The password corresponding to the command is set in advance by the user and stored in the storage unit 16. The password may be set by the processing unit 17, stored in the storage unit 16, and notified to the user in advance. The password is a combination of one or more characters, an image of an object, an abstract concept, etc. A common password may be set for different first commands, or multiple different passwords may be set for different first commands. Setting separate passwords for the first commands "forward," "backward," "stop," "accelerate," "decelerate," "turn right," and "turn left" can prevent the user from issuing the next first command impulsively. The password buffer 15 can send and receive various data to and from the processing unit 17.

[0024] The storage unit 16 stores passwords, which are a plurality of first commands, a plurality of second commands, and a plurality of third commands. The storage unit 16 may also store learning data obtained by machine learning the association between the electrical signals of the user's electroencephalograms and thought information. The second commands and the third commands are associated with the first commands in advance.

[0025] The execution command buffer 13, image command buffer 14, and password buffer 15 are, for example, recording units such as semiconductor memory elements, such as RAM (Random Access Memory) and flash memory, provided in the processing unit 17. The storage unit 16 is configured with a memory card, SSD (Solid State Drive), external storage device, or the like.

[0026] The processing unit 17 is an arithmetic processing device (control device) configured with, for example, a CPU (Central Processing Unit). The processing unit 17 loads a stored program into memory and executes instructions included in the program. The processing unit 17 includes an internal memory (not shown), which is used for temporary storage of data in the processing unit 17, etc. The processing unit 17 has a determination unit 21 and an execution unit 22 as functions. The processing unit 17 also transmits processing results to the display control unit 18 and the communication unit 20.

[0027] The determination unit 21 determines whether the user's thought information acquired by the electroencephalogram acquisition unit 11 and restored by the electroencephalogram decoder 12 is an electroencephalogram corresponding to three different first, second, and third commands (passwords).

[0028] Specifically, the determination unit 21 compares the user's thought information restored by the electroencephalogram decoder 12 with the first command stored in the storage unit 16. If the user's thought information matches the first command, the determination unit 21 determines that the user's thought information is the first command. The determination unit 21 stores the command determined to be the first command in the execution command buffer 13. On the other hand, if the user's thought information does not match the first command, the determination unit 21 determines that the user's thought information is not the first command. The determination unit 21 does not store the command determined to not be the first command in the execution command buffer 13. The determination unit 21 determines whether the user's thought information restored by the electroencephalogram decoder 12 matches the first command even if it is not a perfect match, as long as it can determine that the user's thought information means the first command.

[0029] The determination unit 21 also compares the user's thought information restored by the electroencephalogram decoder 12 with the second command stored in the memory unit 16. At this time, if the user's thought information matches the second command, the determination unit 21 determines that the user's thought information is the second command. If the determination unit 21 determines that the user's thought information is the second command, the determination unit 21 stores the second command in the image command buffer 14. On the other hand, if the user's thought information does not match the second command stored in the memory unit 16, the determination unit 21 determines that the user's thought information is not the second command. If the determination unit 21 determines that the user's thought information is not the second command, the determination unit 21 does not store the second command in the image command buffer 14.

[0030] The determination unit 21 also compares the user's thought information restored by the electroencephalogram decoder 12 with the password (third command) stored in the storage unit 16. At this time, if the user's thought information matches the password, the determination unit 21 determines that the user's thought information is a password. If the determination unit 21 determines that the user's thought information is a password, it stores the password in the password buffer 15. On the other hand, if the user's thought information does not match the password stored in the storage unit 16, the determination unit 21 determines that the user's thought information is not a password. If the determination unit 21 determines that the user's thought information is not a password, it does not store the password in the password buffer 15.

[0031] The determination unit 21 also determines whether the first command stored in the execution command buffer 13, the second command stored in the image command buffer 14, and the password stored in the password buffer 15 are associated with each other.

[0032] When the determination unit 21 determines that it has acquired brain waves corresponding to the first command, brain waves corresponding to the second command, and brain waves corresponding to the password, the execution unit 22 executes the process corresponding to the first command. Specifically, when the determination unit 21 determines that the first acquired user thought information is the first command, the determination unit 21 determines that the next acquired user thought information is the second command, and the determination unit 21 determines that the next acquired user thought information is a password, the execution unit 22 executes the process corresponding to the first command. Note that the order of the first command, the second command, and the password (or third command) that the determination unit 21 acquires as brain waves is not limited to the above-mentioned order and may be any order.

[0033] However, the execution unit 22 cancels the acquired first command when a preset waiting time, for example, three seconds, has elapsed since the determination unit 21 determined that the first command and the second command have been acquired. Furthermore, the execution unit 22 cancels the acquired first command when the processing unit 17 acquires a cancel command as the user's thought after the determination unit 21 determines that the first command and the second command have been acquired. The cancel command is preset, and the relationship between the user's electrical electroencephalogram signal and the user's thought information (cancel command) is linked in advance and stored in the storage unit 16. The cancel command is, for example, a symbol or concept different from the first command, such as "x" or "cancel."

[0034] The display control unit 18 is connected to the display unit 19. The display control unit 18 transmits the processing results transmitted from the processing unit 17 to the display unit 19 and causes it to be displayed. The display control unit 18 causes the display unit 19 to display the processing results of the processing unit 17, such as the operating state of the operation control device 10 and thought instructions for the user.

[0035] The display unit 19 displays the operating status of the operation control device 10 and thought instructions for the user transmitted from the display control unit 18. The display unit 19 presents necessary information to the user. The display unit 19 is a display including, for example, a liquid crystal display (LCD) or an organic electroluminescence (EL) display.

[0036] The communication unit 20 is capable of wireless communication with the operation target 50. The communication unit 20 transmits processing information processed by the processing unit 17 to the operation target 50. Specifically, when the determination unit 21 determines that an electroencephalogram corresponding to the first command and an electroencephalogram corresponding to the password have been acquired, the execution unit 22 generates a control signal corresponding to the processing of the first command and transmits it to the communication unit 20. The communication unit 20 transmits the control signal corresponding to the processing of the first command to the operation target 50. Upon receiving the first command transmitted by the communication unit 20, the operation target 50 operates in response to the processing of the first command. [Operation control method] Here, we will explain the operation control method by the operation control device 10. Fig. 2 is a flowchart showing the flow of processing in the operation control device according to this embodiment.

[0037] 1 and 2, in step S11, the processing unit 17 controls the display control unit 18 to display an instruction to the user to recall a second command (image command) on the display unit 19. The display control unit 18 displays on the display unit 19 a message instructing the user to recall the second command, which indicates the operation content, such as "Please recall the image command."

[0038] In step S12, the determination unit 21 determines whether the user's thought information acquired by the electroencephalogram acquisition unit 11 and restored by the electroencephalogram decoder 12 is an electroencephalogram corresponding to the second command (image command). Here, the determination unit 21 compares the user's thought information restored by the electroencephalogram decoder 12 with the second command stored in the memory unit 16. If the determination unit 21 determines that the user's thought information does not match the second command (No), the determination unit 21 determines that the user's thought information is not the first command and exits this routine. On the other hand, if the determination unit 21 determines that the user's thought information matches the second command stored in the memory unit 16 (Yes), the determination unit 21 determines that the user's thought information is the second command, stores the second command in the image command buffer 14, and proceeds to step S14.

[0039] In step S13, the processing unit 17 controls the display control unit 18 to display an instruction to the user to recall the first command (execution command) on the display unit 19. The display control unit 18 displays on the display unit 19 a message instructing the user to recall the first command, which indicates the operation content, such as "Please recall the execution command."

[0040] In step S14, the determination unit 21 determines whether the user's thought information acquired by the electroencephalogram acquisition unit 11 and restored by the electroencephalogram decoder 12 is an electroencephalogram corresponding to the first command (execution command). Here, the determination unit 21 compares the user's thought information restored by the electroencephalogram decoder 12 with the first command stored in the memory unit 16. If the determination unit 21 determines that the user's thought information does not match the first command (No), it determines that the user's thought information is not the first command and exits this routine. On the other hand, if the determination unit 21 determines that the user's thought information matches the first command stored in the memory unit 16 (Yes), it determines that the user's thought information is the first command, and the determination unit 21 stores the first command in the execution command buffer 13 and proceeds to step S15.

[0041] In step S15, processing unit 17 determines whether a timeout has occurred by the lapse of a preset waiting time since determination unit 21 determined that the first command has been acquired. If processing unit 17 determines that a timeout has occurred (Yes), in step S20, processing unit 17 cancels the acquired first command and second command. Then, in step S21, processing unit 17 controls display control unit 18 to display to the user on display unit 19 that the first command and second command have been canceled. Display control unit 18 displays a timeout message, such as "Timeout," on display unit 19.

[0042] If the processing unit 17 determines that a timeout has not occurred (No), then in step S16 it determines whether or not an electroencephalogram corresponding to a cancel command has been detected. If the processing unit 17 determines that an electroencephalogram corresponding to a cancel command has been detected (Yes), then in step S23 it cancels the acquired first command and second command. Then, in step S24, the processing unit 17 controls the display control unit 18 to display to the user on the display unit 19 that the first command and the second command have been canceled. The display control unit 18 displays on the display unit 19 a message indicating that the first command has been canceled, such as "First command is being canceled."

[0043] On the other hand, if the processing unit 17 determines that an electroencephalogram corresponding to the cancel command has not been detected (No), then in step S17 the judgment unit 21 determines whether or not an electroencephalogram corresponding to the third command (password) has been detected. Here, the judgment unit 21 compares the user's thought information restored by the electroencephalogram decoder 12 with the third command (password) stored in the memory unit 16, and if it determines that the user's thought information does not match the third command (password) (No), the process returns to step S15 and continues processing.

[0044] On the other hand, if the determination unit 21 determines that the user's thought information matches the third command (password) stored in the memory unit 16 (Yes), it stores the password determined as the user's thought information to be the third command (password) in the password buffer 15. Furthermore, in step S18, the determination unit 21 determines whether the second command determined in step S12 and stored in the image command buffer 14, the first command determined in step S14 and stored in the execution command buffer 13, and the password determined in step S17 and stored in the password buffer 15 are the first command, second command, and password that are previously associated with each other. If the determination unit 21 determines that the first command, second command, and password are not associated with each other (No), the process returns to step S15. If the determination unit 21 determines that the first command, second command, and password are associated with each other (Yes), the process proceeds to step S19.

[0045] In step S19, the execution unit 22 executes processing corresponding to the first command. That is, the execution unit 22 transmits a control signal corresponding to the processing of the first command to the operation object 50 via the communication unit 20, and operates the operation object 50. Then, in step S18, the processing unit 17 controls the display control unit 18 to display to the user on the display unit 19 that the first command has been executed. The display control unit 18 displays on the display unit 19 a message indicating that the first command has been executed, such as "Acceleration command has been executed," for example. [effect] In this embodiment, the device includes an electroencephalogram acquisition unit (brain information acquisition unit) 11 that acquires the user's brain information, a determination unit 21 that determines whether the brain information acquired by the electroencephalogram acquisition unit 11 is brain information corresponding to a first command based on a language system and a second command based on a system other than the language system, and an execution unit 22 that executes processing corresponding to the first command when the determination unit 21 determines that it has acquired brain information corresponding to the first command and brain information corresponding to the second command.

[0046] Therefore, when brain information corresponding to the first command and brain information corresponding to the second command are acquired, a process corresponding to the first command is executed to operate the operation target object 50. In other words, even if brain information corresponding to the first command indicating the operation content is acquired, the process corresponding to the first command is not executed unless brain information corresponding to the second command indicating the intention to execute the operation content is acquired. Therefore, the content and execution timing of the first command are activated more safely and accurately, and an appropriate brain wave command can be executed without being influenced by unrelated stimuli.

[0047] In this embodiment, the second command is a command in which brain information is based on the image system. Therefore, by detecting both the first command based on the language system and the second command based on the image system, the user can accurately confirm his or her intention to execute the operation content of the operation target object 50.

[0048] In this embodiment, the first command is brain information in the left brain and is a command indicating the operation content for operating the operation object 50, and the second command is brain information in the right brain and is a command indicating the intention to execute the operation content of the operation object 50. Therefore, by acquiring brain information corresponding to the second command, the user can confirm the intention to execute the operation content of the operation object 50, and an appropriate brain wave command can be executed.

[0049] The operation control device 10 according to the present invention has been described above, but it may be embodied in various different forms other than the above-described embodiment.

[0050] Each component of the illustrated operation control device 10 is a functional concept and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of each device is not limited to that shown in the figure, and all or part of them may be functionally or physically distributed or integrated in any unit depending on the processing load and usage status of each device.

[0051] The configuration of the operation control device 10 is realized, for example, as software, by a program loaded into memory. In the above embodiment, the functional blocks are described as being realized by the cooperation of these hardware and software. In other words, these functional blocks can be realized in various forms, using only hardware, only software, or a combination of both.

[0052] The above-described components include those that can be easily imagined by a person skilled in the art and those that are substantially the same. Furthermore, the above-described configurations can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the configurations are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0053] 10 Operation control device 11. Brain wave acquisition unit (brain information acquisition unit) 12 Brainwave Decoder 13 Execution Command Buffer 14 Image Command Buffer 15 Password Buffer 16 Memory section 17 Processing section 18 Display control unit 19 Display section 20 Communications Department 21 Judgment section 22 Executive Department 50 Operational Object

Claims

1. a brain information acquisition unit that acquires brain information of a user; a determination unit that determines whether or not the brain information acquired by the brain information acquisition unit corresponds to a first command that is a command for operating an object to be operated, a second command that is a command combining one or more images based on the five senses, and a third command ... first command is to be executed; an execution unit that executes an operation corresponding to the first command when the determination unit determines that all of the brain information corresponding to the first command, the brain information corresponding to the second command, and the third command have been acquired; An operation control device comprising:

2. acquiring brain information of a user; a step of determining whether the acquired brain information corresponds to a first command which is a command for operating an object to be operated, a second command which is a command combining one or more images based on the five senses, and a third command for determining whether the acquired brain information corresponds to the first command; executing an operation corresponding to the first command when it is determined that all of the brain information corresponding to the first command, the brain information corresponding to the second command, and the third command have been acquired; An operation control method including:

3. acquiring brain information of a user; a step of determining whether the acquired brain information corresponds to a first command which is a command for operating an object to be operated, a second command which is a command combining one or more images based on the five senses, and a third command for determining whether the acquired brain information corresponds to the first command; executing an operation corresponding to the first command when it is determined that all of the brain information corresponding to the first command, the brain information corresponding to the second command, and the third command have been acquired; A program that causes a computer to execute the following.

Images