Operation control device, operation control method and program

Abstract

To secure safety of execution of brainwave command by externally notifying the state of sending or reception of the command when a user executes the sending or reception of the command.SOLUTION: An operation control device is provided, comprising: a brain information acquisition unit acquiring brain information of a user; a determination unit determining whether or not the brain information acquired by the brain information acquisition unit is brain information which corresponds to execution command signifying operation content for operating operation target; an execution unit executing processing which corresponds to execution command when the determination unit determines that the brain information acquisition unit acquires brain information which corresponds to the execution command; and a notification unit which when transmitting a first processing signal which corresponds to the execution command to an operation target or when receiving a second processing signal from the operation target, notifies to surroundings of at least a user and an operation target that the execution unit is processing the first or second processing signal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an operation control device, an operation control method, and a program. [Background technology]

[0002] In recent years, advances in technology for measuring brain information have made it possible to realize a brain-machine interface (BMI), which is an interface that directly connects the brain and machines. One such technology is described in Patent Document 1, for example. Patent Document 1 discloses a system that obtains multiple steady-state visual evoked response potential signals from a user when the user receives a stimulus and supplies the signals to the system. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-117957 Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional BMIs, when a user commands a robot to "move forward," the robot recalls a command to move forward, and when the command is detected, the robot moves forward. However, people near the robot do not know how the user intends to operate the robot or how the robot will operate. This raises the risk that the robot may operate without warning and come into contact with nearby people. Furthermore, if a user speaks to the robot while recalling a command, the robot may recall the wrong command.

[0005] The present invention has been made in consideration of the above, and aims to ensure the safety of executing electroencephalogram commands by notifying the outside world of the status when a user is sending or receiving a command. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the operation control device of the present invention comprises a brain information acquisition unit that acquires brain information of a user, a determination unit that determines whether the brain information acquired by the brain information acquisition unit is brain information corresponding to an execution command that indicates operation content for operating an operation object, an execution unit that executes processing corresponding to the execution command when the determination unit determines that brain information corresponding to the execution command has been acquired, and a notification unit that, when a first processing signal corresponding to the execution command is being transmitted to the operation object or a second processing signal is being received from the operation object, notifies those around at least one of the user and the operation object that the first processing signal or the second processing signal is being processed.

[0007] The operation control method according to the present invention includes the steps of acquiring brain information of a user, determining whether the acquired brain information corresponds to an execution command indicating operation content for operating an operation object, executing processing corresponding to the execution command when it is determined that brain information corresponding to the execution command has been acquired, and notifying those around at least one of the user and the operation object that the first processing signal or the second processing signal is being processed when a first processing signal corresponding to the execution command is being transmitted to the operation object or a second processing signal is being received from the operation object.

[0008] The program of the present invention causes a computer to execute the following steps: acquiring brain information of a user; determining whether the acquired brain information corresponds to an execution command that indicates operation content for operating an object to be operated; executing processing corresponding to the execution command when it is determined that brain information corresponding to the execution command has been acquired; and, when a first processing signal corresponding to the execution command is being transmitted to the object to be operated or a second processing signal is being received from the object to be operated, notifying those around at least one of the user and the object to be operated that the first processing signal or the second processing signal is being processed. [Effects of the Invention]

[0009] According to the present invention, when a user is transmitting or receiving a command, the state is notified to the outside, thereby ensuring the safety of the execution of electroencephalogram commands. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an operation control device according to the first embodiment. [Figure 2] FIG. 2 is a flowchart showing the flow of the transmission process in the operation control device according to the first embodiment. [Figure 3] FIG. 3 is a flowchart showing the flow of the reception process in the operation control device according to the first embodiment. [Figure 4] FIG. 4 is a graph showing the amount of brain activity versus measurement location in the brain. [Figure 5] FIG. 5 is a graph showing the temporal changes in electroencephalograms at specific measurement locations in the brain. [Figure 6] FIG. 6 is a block diagram showing an example of the configuration of an operation control device according to the second embodiment. [Figure 7] FIG. 7 is a flowchart showing the flow of the transmission process in the operation control device according to the second embodiment. [Figure 8]FIG. 8 is a flowchart showing the flow of the reception process in the operation control device according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An operation control device, an operation control method, and a program according to the present invention will be described in detail below with reference to the accompanying drawings. However, the present invention is not limited to the following embodiments.

[0012] First Embodiment [Operation control device] FIG. 1 is a block diagram showing an example of the configuration of an operation control device according to the first embodiment.

[0013] 1, in the first embodiment, the operation control device 10 controls the operation of an operation target object 50 based on brain information of a user. The operation control device 10 includes an electroencephalogram acquiring unit (brain information acquiring unit) 11, an electroencephalogram decoder 12, a start command buffer 13, an execution command buffer 14, a storage unit 15, a processing unit 16, a display control unit 17, a display unit 18, a communication unit 19, and a first notification unit 31.

[0014] The operation object 50 is, for example, a robot, but is not limited to a robot and may be any device that operates through communication. Although not shown, the operation object 50 is provided with a drive unit, a control unit that controls the drive unit, and a communication unit that can receive control signals, etc. When the communication unit of the operation object 50 receives a control signal, the control unit drives the drive unit to operate the robot.

[0015] The electroencephalogram acquiring unit 11 can be worn on the user's head. The electroencephalogram decoder 12, processing unit 16, display control unit 17, display unit 18, and communication unit 19 may be worn by the user, or may be placed in predetermined positions without being worn. Furthermore, the electroencephalogram acquiring unit 11, electroencephalogram decoder 12, processing unit 16, display control unit 17, display unit 18, and communication unit 19 may be provided integrally or separately.

[0016] The brain wave acquisition unit 11 acquires brain waves, which are brain information of the user. The brain wave acquisition unit 11 has, for example, an electric sensor (e.g., an electrode) that detects brain waves generated from a weak current flowing in the neural network of the brain. The brain wave acquisition unit 11 detects the potential (electrical signal) of a weak current when the user receives an external stimulus or based on the user's thoughts, such as their intentions. Note that the brain information acquisition unit is not limited to the brain wave acquisition unit 11. The brain information acquisition unit may also acquire, for example, blood flow due to brain activity, which is brain information of the user, by, for example, near-infrared light measurement. The brain wave acquisition unit 11 is connected to the brain wave decoder 12 and transmits brain waves, which are brain information acquired from the user, to the brain wave decoder 12.

[0017] The electroencephalogram decoder 12 restores the user's electrical electroencephalogram signals acquired by the electroencephalogram acquisition unit 11 into the user's thought information. In this case, the relationship between the user's electrical electroencephalogram signals and the user's thought information is linked in advance. In this case, the relationship between the electroencephalogram signals and the user's thought information is linked using machine learning such as deep learning.

[0018] The start command buffer 13 temporarily stores the first command determined by the determination unit 21 based on the user's thought information. The first command is a command that prepares to start receiving a second command (execution command) for operating the operation target object 50, such as a command such as "start robot operation" or "start input of execution command." The start command buffer 13 is capable of sending and receiving various data to and from the processing unit 16.

[0019] The execution command buffer 14 temporarily stores the second command determined by the determination unit 21 based on the user's thought information. The second command is an execution command for operating the operation target object 50, such as "forward," "backward," "stop," "accelerate," "decelerate," "turn right," or "turn left." In this case, the relationship between the second command and the operation of the operation target object 50 is linked in advance. The execution command buffer 14 can send and receive various data to and from the processing unit 16.

[0020] The second command is not limited to a concept indicating an instructed action, but may be one to which a numerical value or the like is added to complement the concept indicating the instructed action. For example, the electroencephalogram decoder 103 may restore the user's thought information based on the electrical signal acquired by the electroencephalogram acquiring unit 11, detect the intensity of the electroencephalogram as a decimal number ranging from 0 to 1, and set the second command by adding the intensity (numerical value) to the user's thought information. In this case, the numerical value of the second command is a count of the maximum torque output of the drive device provided on the operation target object 50.

[0021] Furthermore, the first command (start command) and the second command (execution command) may be linked and set. That is, the first command may be set as one common command that is common to different second commands. For example, the first command may be set as "start" or "start operation" for second commands such as "forward" or "backward." Also, a plurality of different individual first commands may be set for different second commands. For example, the first command may be set as "start running" for second commands such as "forward" or "accelerate," and the first command may be set as "start arm operation" for second commands such as "extend" or "grab." Also, by setting individual first commands for the second commands "forward," "backward," "stop," "accelerate," "decelerate," "turn right," and "turn left," it is possible to avoid a situation where the next second command is issued impulsively.

[0022] The storage unit 15 stores a plurality of first commands (start commands) and a plurality of second commands (execution commands). The storage unit 15 may also store learning data obtained by machine learning the association between the electrical signals of the user's electroencephalograms and thought information. The second commands are associated with the first commands in advance.

[0023] The start command buffer 13 and the execution command buffer 14 are, for example, recording units such as semiconductor memory elements, such as RAM (Random Access Memory) and flash memory, included in the processing unit 16. The storage unit 15 is configured with a memory card, SSD (Solid State Drive), external storage device, or the like.

[0024] The processing unit 16 is an arithmetic processing device (control device) configured with, for example, a CPU (Central Processing Unit). The processing unit 16 loads a stored program into memory and executes instructions included in the program. The processing unit 16 includes an internal memory (not shown), which is used for temporary storage of data in the processing unit 16, etc. The processing unit 16 has, as its functions, a determination unit 21, an execution unit 22, and a receiving unit 23. The processing unit 16 also transmits processing results to the display control unit 17 and the communication unit 19.

[0025] The determination unit 21 determines whether the user's thought information acquired by the electroencephalogram acquisition unit 11 and restored by the electroencephalogram decoder 12 is an electroencephalogram corresponding to two different first and second commands.

[0026] Specifically, the determination unit 21 compares the user's thought information restored by the electroencephalogram decoder 12 with the first command (start command) stored in the storage unit 15. When the determination unit 21 determines that the user's thought information matches the first command and that the user's thought information is the first command, it stores the command determined to be the first command in the start command buffer 13. On the other hand, when the determination unit 21 does not match the first command, it determines that the user's thought information is not the first command. When the determination unit 21 determines that the command is not the first command, it does not store the command determined to be not the first command in the start command buffer 13. When the determination unit 21 determines that the user's thought information restored by the electroencephalogram decoder 12 matches the first command, it determines that there is a match even if it is not a perfect match, as long as it can determine that the user's thought information means the first command.

[0027] The determination unit 21 also compares the user's thought information restored by the electroencephalogram decoder 12 with the second command (execution command) stored in the memory unit 15. At this time, if the user's thought information matches the second command, the determination unit 21 determines that the user's thought information is the second command. If the determination unit 21 determines that the user's thought information is the second command, the determination unit 21 stores the second command in the execution command buffer 14. On the other hand, if the user's thought information does not match the second command stored in the memory unit 15, the determination unit 21 determines that the user's thought information is not the second command. If the determination unit 21 determines that the user's thought information is not the second command, the determination unit 21 does not store the second command in the execution command buffer 14.

[0028] At this time, the determination unit 21 determines that the user's thought information is brain information corresponding to the first command when the matching rate between the user's thought information restored by the electroencephalogram decoder 12 and the first command judgment value (start command judgment value) stored in the memory unit 15 is equal to or higher than a predetermined first matching rate (e.g., 80%). On the other hand, the determination unit 21 determines that the user's thought information is brain information corresponding to the first command when the matching rate between the user's thought information restored by the electroencephalogram decoder 12 and the second command judgment value (execution command judgment value) stored in the memory unit 15 is equal to or higher than a predetermined second matching rate (e.g., 90%). Here, the second matching rate is set to a higher matching rate than the first matching rate.

[0029] FIG. 4 is a graph showing the amount of brain activity relative to measurement locations in the brain, and FIG. 5 is a graph showing temporal changes in electroencephalograms at specific measurement locations in the brain.

[0030] Figure 4 shows brain activity levels at multiple different measurement locations in the brain. By comparing the user's brain activity levels at each measurement location with a preset brain activity level determination value, a first match rate between the user's thought information and the first command, or a second match rate between the user's thought information and the second command, is calculated.

[0031] Figure 5 shows the temporal change in EEG levels at a specific measurement location. By comparing the temporal change in EEG at a specific time ta with a preset EEG judgment value (the thick line in Figure 5), a first matching rate between the user's thought information and the first command, or a second matching rate between the user's thought information and the second command, is calculated.

[0032] In addition, by comparing the user's brain activity at each measurement position with the brain activity determination value, and also by comparing the temporal change in brain waves with the brain wave determination value, a first matching rate between the user's thought information and the first command, or a second matching rate between the user's thought information and the first command, may be obtained.

[0033] In addition, when the first command (start command) and the second command (execution command) are linked, the judgment unit 21 judges whether or not the first command (start command) stored in the start command buffer 13 and the second command (execution command) stored in the execution command buffer 14 are associated with each other.

[0034] When the determination unit 21 determines that it has acquired an electroencephalogram corresponding to the first command and an electroencephalogram corresponding to the second command, the execution unit 22 executes a process corresponding to the second command. Specifically, when the determination unit 21 determines that the user's thought information acquired first is the first command and the determination unit 21 determines that the user's thought information acquired next is the second command, the execution unit 22 executes a process corresponding to the second command.

[0035] However, the execution unit 22 cancels the acquired first command when a preset waiting time, for example, three seconds, has elapsed since the determination unit 21 determined that the first command has been acquired. Furthermore, the execution unit 22 cancels the acquired first command when the processing unit 16 acquires a cancel command as the user's thought after the determination unit 21 determines that the first command has been acquired. The cancel command is preset, and the relationship between the user's electrical electroencephalogram signal and the user's thought information (cancel command) is linked in advance and stored in the storage unit 15. The cancel command is, for example, a symbol or concept different from the first command, such as "x" or "cancel."

[0036] The receiving unit 23 is capable of receiving a processing signal from the operation target object 50. That is, the executing unit 22 transmits a first processing signal corresponding to the second command (execution command) to the operation target object 50 via the communication unit 19 in order to execute a process corresponding to the second command. At this time, for example, when the operation target object 50 cannot execute the process corresponding to the second command, the operation target object 50 transmits a second processing signal indicating that the process corresponding to the second command cannot be executed to the processing unit 16 via the communication unit 19. The receiving unit 23 receives the second processing signal from the operation target object 50.

[0037] For example, when the second command is "move forward," if there is a wall in front of the operation object 50, the operation object 50 cannot "move forward" because the wall is in the way. In this case, the operation object 50 transmits a second processing signal to the processing unit 16 indicating that it cannot "move forward," and the signal is received by the receiving unit 23. Note that when there is a removable obstacle in front of the operation object 50, the operation object 50 may transmit a second processing signal to the processing unit 16 indicating that the obstacle should be removed. The same applies when the second command is "move backward," "turn right," "turn left," etc. Furthermore, when the operation object 50 cannot operate due to insufficient battery charge, the operation object 50 may transmit a second processing signal to the processing unit 16 indicating that the battery should be charged.

[0038] The second processed signal received by the receiving unit 23 may be converted into an electrical signal of an electroencephalogram by an electroencephalogram encoder (not shown) and transmitted to the user as thought information via an electroencephalogram reproducing unit (not shown). The method of presenting the various information received by the receiving unit 23 as the second processed signal to the user is not limited to presentation by electroencephalogram, and may be displayed on the display unit 18, for example.

[0039] The display control unit 17 is connected to the display unit 18. The display control unit 17 transmits the processing results transmitted from the processing unit 16 to the display unit 18 and causes it to be displayed. The display control unit 17 causes the display unit 18 to display the processing results of the processing unit 16, such as the operating state of the operation control device 10 and thought instructions for the user.

[0040] The display unit 18 displays the operating status of the operation control device 10 and thought instructions for the user transmitted from the display control unit 17. The display unit 18 presents necessary information to the user. The display unit 18 is a display including, for example, a liquid crystal display (LCD) or an organic electro-luminescence (EL) display.

[0041] The communication unit 19 is capable of wireless communication between the processing unit 16 and the operation target 50. The communication unit 19 transmits processing information processed by the processing unit 16 to the operation target 50. Specifically, when the determination unit 21 determines that an electroencephalogram corresponding to the first command and an electroencephalogram corresponding to the second command have been acquired, the execution unit 22 generates a control signal (first processing signal) corresponding to the processing of the second command and transmits it to the communication unit 19. The communication unit 19 transmits the control signal corresponding to the processing of the second command to the operation target 50. Upon receiving the second command transmitted by the communication unit 19, the operation target 50 operates in response to the processing of the second command.

[0042] Furthermore, the communication unit 19 transmits processing information (second processing signal) indicating the processing status of the operation target 50 to the processing unit 16. Specifically, when the operation target 50 is unable to execute the processing corresponding to the second command, the operation target 50 generates a control signal (second processing signal) corresponding to the fact that the operation target 50 cannot execute the processing of the second command and transmits it to the communication unit 19. The communication unit 19 transmits the control signal corresponding to the fact that the operation target 50 cannot execute the processing of the second command to the processing unit 16. The receiving unit 23 receives the control signal transmitted by the communication unit 19 corresponding to the fact that the operation target 50 cannot execute the processing of the second command.

[0043] The first notification unit 31 notifies the outside that the processing unit 16 is currently processing. Specifically, when the processing unit 16 is transmitting a first processing signal corresponding to a second command (execution command) to the operation target 50, the first notification unit 31 notifies the user and those around the operation target 50 that the first processing signal is currently being processed. Furthermore, when the processing unit 16 is receiving a second processing signal from the operation target 50, the first notification unit 31 notifies the user and those around the operation target 50 that the second processing signal is currently being processed. Furthermore, after the determination unit 21 determines the first command (start command), the first notification unit 31 may notify the user and those around the operation target 50 that each processing signal is currently being processed. Note that it is only necessary for the first notification unit 31 to be able to notify the user and those around the operation target 50 that at least one of the first processing signal and the second processing signal is currently being processed. Furthermore, the first notification unit 31 may notify the surroundings of at least one of the user and the operation target object 50 that the first processing signal or the second processing signal is being processed.

[0044] The first notification unit 31 is, for example, a display including a liquid crystal display or an organic EL display. In this case, the first notification unit 31 may also serve as the display unit 18. In addition to a display, the first notification unit 31 may be a visual presentation device such as lighting, or an auditory audio device such as a speaker. The first notification unit 31 may be attached to the user or the operation target 50, or may be disposed near the user or the operation target 50. In this case, the first notification unit 31 may use different notification methods to notify the user and those around the operation target 50 when the processing unit 16 is transmitting a first processing signal corresponding to the second command to the operation target 50 and when the processing unit 16 is receiving a second processing signal from the operation target 50.

[0045] [Operation control method] Here, a description will be given of an operation control method by the operation control device 10. Fig. 2 is a flowchart showing the flow of transmission processing in the operation control device according to the first embodiment.

[0046] 1 and 2, in step S11, the processing unit 16 controls the display control unit 17 to display an instruction to the user to recall a first command (start command) on the display unit 18. The display control unit 17 displays a message on the display unit 18 instructing the user to recall the first command, which indicates the operation content, such as "Please recall the start command."

[0047] In step S12, the determination unit 21 determines whether the user's thought information acquired by the electroencephalogram acquisition unit 11 and restored by the electroencephalogram decoder 12 is an electroencephalogram corresponding to the first command (start command). Here, the determination unit 21 compares the user's thought information restored by the electroencephalogram decoder 12 with the first command stored in the memory unit 15. If the determination unit 21 determines that the user's thought information does not match the first command, that is, that the match rate between the two is less than the first match rate (e.g., 80%) (No), the determination unit 21 determines that the user's thought information is not the first command and exits this routine. On the other hand, if the determination unit 21 determines that the user's thought information matches the first command stored in the memory unit 15, that is, that the match rate between the two is equal to or greater than the first match rate (e.g., 80%) (Yes), the determination unit 21 determines that the user's thought information is the first command, stores the first command in the start command buffer 13, and proceeds to step S13.

[0048] In step S13, the first notification unit 31 notifies the outside that the processing unit 16 is currently processing. Specifically, when the determination unit 21 determines that the user's thought information is a first command (start command) and that a second command (execution command) is being accepted, the first notification unit 31 notifies various information to the user and those around the operation target 50. Here, the processing unit 16 notifies the user and the operation target 50, for example, by using the first notification unit 31 to indicate that the user is recalling the second command (execution command). The first notification unit 31 notifies, for example, a message indicating the processing content, such as "recalling an execution command." At this time, the first notification unit 31 allows people around the user to determine the user's state, making it easier for the user to recall the execution command.

[0049] In step S14, the processing unit 16 controls the display control unit 17 to display an instruction to the user to recall a second command (execution command) on the display unit 18. The display control unit 17 displays on the display unit 18 a message instructing the user to recall a second command, which indicates the operation content, such as "Please recall the execution command."

[0050] In step S15, processing unit 16 determines whether a timeout has occurred by the lapse of a preset waiting time since determination unit 21 determined that the first command was acquired. If processing unit 16 determines that a timeout has occurred (Yes), in step S22, processing unit 16 cancels the acquired first command. Then, in step S23, processing unit 16 controls display control unit 17 to display to the user on display unit 18 that the first command has been canceled. Display control unit 17 displays a timeout message, such as "Timeout," on display unit 18.

[0051] If the processing unit 16 determines that a timeout has not occurred (No), then in step S16 it determines whether or not an electroencephalogram corresponding to a cancel command has been detected. If the processing unit 16 determines that an electroencephalogram corresponding to a cancel command has been detected (Yes), then in step S24 it cancels the acquired first command. Then, in step S25, the processing unit 16 controls the display control unit 17 to display to the user on the display unit 18 that the first command has been canceled. The display control unit 17 displays a message on the display unit 18 indicating that the first command has been canceled, such as "First command is being canceled."

[0052] On the other hand, if the processing unit 16 determines that an electroencephalogram corresponding to a cancel command has not been detected (No), then in step S17 the determination unit 21 determines whether or not an electroencephalogram corresponding to a second command (execute command) has been detected. Here, the determination unit 21 compares the user's thought information restored by the electroencephalogram decoder 12 with the second command stored in the memory unit 15, and if it determines that the user's thought information does not match the second command, that is, that the match rate between the two is less than the second match rate (e.g., 90%) (No), the process returns to step S15 and continues.

[0053] On the other hand, if the determination unit 21 determines that the user's thought information matches the second command stored in the memory unit 15, that is, that the match rate between the two is equal to or greater than the second match rate (e.g., 90%) (Yes), the determination unit 21 determines that the user's thought information is the second command and stores the determined second command in the execution command buffer 14. Furthermore, in step S18, the determination unit 21 determines whether the first command determined in step S12 and stored in the start command buffer 13 and the second command determined in step S17 and stored in the execution command buffer 14 are the first command and the second command that are previously associated with each other. If the determination unit 21 determines that the first command and the second command are not associated with each other (No), the process returns to step S15. If the determination unit 21 determines that the first command and the second command are associated with each other (Yes), the process proceeds to step S19.

[0054] In step S19, the execution unit 22 executes processing corresponding to the second command. That is, the execution unit 22 transmits a control signal corresponding to the processing of the second command to the operation object 50 via the communication unit 19, thereby operating the operation object 50. Then, in step S20, the processing unit 16 controls the display control unit 17 to display to the user on the display unit 18 that the second command has been executed. The display control unit 17 displays on the display unit 18 a message indicating that the first command has been executed, such as "Acceleration command has been executed." The processing unit 16 also notifies the user or the operation object 50, for example, by the first notification unit 31 that the operation object 50 has executed the first command (execution command). The first notification unit 31 notifies the user or the operation object 50 of the execution of the first command (execution command). For example, the first notification unit 31 notifies a message indicating the processing content, such as "The robot will operate." Then, in step S21, the first notification unit 31 ends the notification to the outside.

[0055] FIG. 3 is a flowchart showing the flow of the reception process in the operation control device according to the first embodiment.

[0056] 1 and 3, in step S31, the processing unit 16 controls the display control unit 17 to display an instruction to the user to recall a first command (start command) on the display unit 18. The display control unit 17 displays a message on the display unit 18 instructing the user to recall the first command, which indicates the operation content, such as "Please recall the start command."

[0057] In step S32, the determination unit 21 determines whether the user's thought information acquired by the electroencephalogram acquisition unit 11 and restored by the electroencephalogram decoder 12 is an electroencephalogram corresponding to the first command (start command). Here, the determination unit 21 compares the user's thought information restored by the electroencephalogram decoder 12 with the first command stored in the memory unit 15. If the determination unit 21 determines that the user's thought information does not match the first command, that is, that the match rate between the two is less than the first match rate (e.g., 80%) (No), the determination unit 21 determines that the user's thought information is not the first command and exits this routine. On the other hand, if the determination unit 21 determines that the user's thought information matches the first command stored in the memory unit 15, that is, that the match rate between the two is equal to or greater than the first match rate (e.g., 80%) (Yes), the determination unit 21 determines that the user's thought information is the first command, stores the first command in the start command buffer 13, and proceeds to step S33.

[0058] In step S33, the determination unit 21 determines whether or not a reception command has been received from the operation target object 50. Here, the determination unit 21, for example, compares the received reception command with the reception command stored in the storage unit 15, and if it determines that they do not match (No), it exits this routine. On the other hand, the determination unit 21 compares the received reception command with the reception command stored in the storage unit 15, and if it determines that they match (Yes), it proceeds to step S34.

[0059] In step S34, the first notification unit 31 notifies the outside that the processing unit 16 is performing processing. Specifically, when the receiving unit 23 receives a reception command from the operation target object 50, the first notification unit 31 notifies the user and those around the operation target object 50 of various information. Here, the processing unit 16 notifies the user and the operation target object 50, for example, by using the first notification unit 31 to notify them that a reception command is being received from the operation target object 50. The first notification unit 31 notifies a message indicating the processing content, such as "Receiving from operation target object 50."

[0060] Then, in step S35, the receiving unit 23 receives the received command from the operation object 50 and presents the received content to the user. In step S36, the receiving unit 23 determines whether or not it has finished receiving the received command from the operation object 50. Here, if the receiving unit 23 determines that it has not finished receiving the received command from the operation object 50 (No), it continues the processing of step S35. On the other hand, if the receiving unit 23 determines that it has finished receiving the received command from the operation object 50 (Yes), in step S37, the first notification unit 31 ends notification to the outside.

[0061] In the above description, the first notification unit 31 issues a notification to the outside when the first command (start command) and the received command are detected, but the processing in step S32 may be omitted. That is, the determination unit 21 may make the first notification unit 31 issue a notification to the outside when it detects the received command, even before detecting the first command (start command).

[0062] Second Embodiment [Operation control device] 6 is a block diagram showing an example of the configuration of an operation control device according to the second embodiment. Note that members having the same functions as those in the first embodiment described above are given the same reference numerals, and detailed descriptions thereof will be omitted.

[0063] 6, the operation control device 10 controls the operation of an operation target object 50 based on brain information of a user. The operation control device 10 includes an electroencephalogram acquiring unit (brain information acquiring unit) 11, an electroencephalogram decoder 12, a start command buffer 13, an execution command buffer 14, a storage unit 15, a processing unit 16, a display control unit 17, a display unit 18, a communication unit 19, a first notification unit 31, and a second notification unit 32.

[0064] The EEG acquisition unit 11, EEG decoder 12, start command buffer 13, execution command buffer 14, memory unit 15, processing unit 16, display control unit 17, display unit 18, communication unit 19, and first notification unit 31 are the same as those in the first embodiment, and their explanations will be omitted.

[0065] The second notification unit 32 notifies the user that the processing unit 16 is currently processing. Specifically, when the processing unit 16 is transmitting a first processing signal corresponding to a second command (execution command) to the operation target 50, the second notification unit 32 notifies the user that the first processing signal is currently being processed. Furthermore, when the processing unit 16 is receiving the second processing signal from the operation target 50, the second notification unit 32 notifies the user that the second processing signal is currently being processed. Furthermore, the second notification unit 32 may notify the user that each processing signal is currently being processed after the determination unit 21 determines the first command (start command). Note that it is sufficient for the second notification unit 32 to be able to notify the user that at least one of the first processing signal and the second processing signal is currently being processed.

[0066] The second notification unit 32 is, for example, a display including a liquid crystal display or an organic electroluminescence (EL) display. In this case, the second notification unit 32 may also serve as the display unit 18. The second notification unit 32 may be a visual display device such as a light, or an auditory sound device such as a speaker, headphones, or earphones, in addition to a display. Furthermore, the second notification unit 32 may be a device that notifies the user through a tactile sensation. The second notification unit 32 may be any device that the user can recognize visually, aurally, or tactilely, and may be worn by the user or located near the user. In this case, the second notification unit 32 may notify the user and those around the operation object 50 in different ways depending on whether the processing unit 16 is transmitting a first processing signal corresponding to the second command to the operation object 50 or the processing unit 16 is receiving a second processing signal from the operation object 50.

[0067] [Operation control method] Here, an operation control method by the operation control device 10 will be described. Fig. 7 is a flowchart showing the flow of transmission processing in the operation control device according to the second embodiment. Note that the action of the first notification unit 31 is the same as in the first embodiment, and therefore a description thereof will be omitted.

[0068] 6 and 7, in step S41, the processing unit 16 controls the display control unit 17 to display an instruction to the user to recall the first command (start command) on the display unit 18. The display control unit 17 displays a message on the display unit 18 instructing the user to recall the first command, which indicates the operation content, such as "Please recall the start command."

[0069] In step S42, the determination unit 21 determines whether the user's thought information acquired by the electroencephalogram acquisition unit 11 and restored by the electroencephalogram decoder 12 is an electroencephalogram corresponding to the first command (start command). Here, the determination unit 21 compares the user's thought information restored by the electroencephalogram decoder 12 with the first command stored in the memory unit 15. If the determination unit 21 determines that the user's thought information does not match the first command, that is, that the match rate between the two is less than the first match rate (e.g., 80%) (No), the determination unit 21 determines that the user's thought information is not the first command and exits this routine. On the other hand, if the determination unit 21 determines that the user's thought information matches the first command stored in the memory unit 15, that is, that the match rate between the two is equal to or greater than the first match rate (e.g., 80%) (Yes), the determination unit 21 determines that the user's thought information is the first command, stores the first command in the start command buffer 13, and proceeds to step S13.

[0070] In step S43, the second notification unit 32 internally notifies the processing unit 16 that processing is in progress. Specifically, when the determination unit 21 determines that the user's thought information is a first command (start command) and that a second command (execution command) is being accepted, the second notification unit 32 notifies the user of various information. Here, the processing unit 16 notifies the user, for example, by informing the user that a recall of the second command (execution command) from the user is being accepted by the second notification unit 32. The second notification unit 32 notifies the user of a message indicating the processing content, such as "Recall of execution command being accepted." At this time, the second notification unit 32 may limit the user's vision by darkening the surroundings or dimming the display. The second notification unit 32 may also limit the user's hearing by making it difficult to hear surrounding sounds or by lowering the volume of the speaker. The second notification unit 32 restricts the user's vision and hearing, making it easier for the user to recall the execution command.

[0071] In step S44, the processing unit 16 controls the display control unit 17 to display an instruction to the user to recall a second command (execution command) on the display unit 18. The display control unit 17 displays on the display unit 18 a message instructing the user to recall a second command, which indicates the operation content, such as "Please recall the execution command."

[0072] In step S45, processing unit 16 determines whether a timeout has occurred by the lapse of a preset waiting time since determination unit 21 determined that the first command was acquired. If processing unit 16 determines that a timeout has occurred (Yes), in step S52, processing unit 16 cancels the acquired first command. Then, in step S53, processing unit 16 controls display control unit 17 to display to the user on display unit 18 that the first command has been canceled. Display control unit 17 displays a timeout message, such as "Timeout," on display unit 18.

[0073] If the processing unit 16 determines that a timeout has not occurred (No), then in step S46 it determines whether or not an electroencephalogram corresponding to a cancel command has been detected. If the processing unit 16 determines that an electroencephalogram corresponding to a cancel command has been detected (Yes), then in step S54 it cancels the acquired first command. Then, in step S55, the processing unit 16 controls the display control unit 17 to display to the user on the display unit 18 that the first command has been canceled. The display control unit 17 displays a message on the display unit 18 indicating that the first command has been canceled, such as "First command is being canceled."

[0074] On the other hand, if the processing unit 16 determines that an electroencephalogram corresponding to a cancel command has not been detected (No), then in step S47 the determination unit 21 determines whether or not an electroencephalogram corresponding to a second command (execute command) has been detected. Here, the determination unit 21 compares the user's thought information restored by the electroencephalogram decoder 12 with the second command stored in the memory unit 15, and if it determines that the user's thought information does not match the second command, that is, that the match rate between the two is less than the second match rate (e.g., 90%) (No), the process returns to step S45 and continues.

[0075] On the other hand, if the determination unit 21 determines that the user's thought information matches the second command stored in the memory unit 15, that is, that the match rate between the two is equal to or greater than the second match rate (e.g., 90%) (Yes), the determination unit 21 determines that the user's thought information is the second command and stores the determined second command in the execution command buffer 14. Furthermore, in step S48, the determination unit 21 determines whether the first command determined in step S42 and stored in the start command buffer 13 and the second command determined in step S47 and stored in the execution command buffer 14 are the first command and the second command that are previously associated with each other. If the determination unit 21 determines that the first command and the second command are not associated with each other (No), the process returns to step S45. If the determination unit 21 determines that the first command and the second command are associated with each other (Yes), the process proceeds to step S49.

[0076] In step S49, the execution unit 22 executes processing corresponding to the second command. That is, the execution unit 22 transmits a control signal corresponding to the processing of the second command to the operation object 50 via the communication unit 19, thereby operating the operation object 50. Then, in step S50, the processing unit 16 controls the display control unit 17 to display to the user on the display unit 18 that the second command has been executed. The display control unit 17 displays on the display unit 18 a message indicating that the first command has been executed, such as "Acceleration command has been executed." The processing unit 16 also notifies the user or the operation object 50, for example, by the second notification unit 32 that the operation object 50 has executed the first command (execution command). The second notification unit 32 notifies the user or the operation object 50 of the execution of the first command (execution command). For example, the second notification unit 32 notifies a message indicating the processing content, such as "The robot is operating." Then, in step S51, the second notification unit 32 ends the notification to the inside.

[0077] FIG. 8 is a flowchart showing the flow of the reception process in the operation control device according to the second embodiment.

[0078] 6 and 8, in step S61, the processing unit 16 controls the display control unit 17 to display an instruction to the user to recall the first command (start command) on the display unit 18. The display control unit 17 displays, on the display unit 18, a message instructing the user to recall the first command, which indicates the operation content, such as "Please recall the start command."

[0079] In step S62, the determination unit 21 determines whether the user's thought information acquired by the electroencephalogram acquisition unit 11 and restored by the electroencephalogram decoder 12 is an electroencephalogram corresponding to the first command (start command). Here, the determination unit 21 compares the user's thought information restored by the electroencephalogram decoder 12 with the first command stored in the memory unit 15. If the determination unit 21 determines that the user's thought information does not match the first command, that is, that the match rate between the two is less than the first match rate (e.g., 80%) (No), the determination unit 21 determines that the user's thought information is not the first command and exits this routine. On the other hand, if the determination unit 21 determines that the user's thought information matches the first command stored in the memory unit 15, that is, that the match rate between the two is equal to or greater than the first match rate (e.g., 80%) (Yes), the determination unit 21 determines that the user's thought information is the first command, stores the first command in the start command buffer 13, and proceeds to step S63.

[0080] In step S63, the determination unit 21 determines whether or not a reception command has been received from the operation target object 50. Here, the determination unit 21, for example, compares the received reception command with the reception command stored in the storage unit 15, and if it determines that they do not match (No), it exits this routine. On the other hand, the determination unit 21 compares the received reception command with the reception command stored in the storage unit 15, and if it determines that they match (Yes), it proceeds to step S64.

[0081] In step S64, the second notification unit 32 internally notifies the processing unit 16 that processing is in progress. Specifically, when the receiving unit 23 receives a reception command from the operation target object 50, the second notification unit 32 notifies the user of various information. Here, the processing unit 16 notifies the user, for example, by the second notification unit 32 that a reception command is being received from the operation target object 50. The second notification unit 32 notifies the user of a message indicating the processing content, such as "Receiving from operation target object 50."

[0082] Then, in step S65, the receiving unit 23 receives the reception command from the operation object 50, and in step S66, the receiving unit 23 determines whether or not the reception of the reception command from the operation object 50 has ended. Here, if the receiving unit 23 determines that the reception of the reception command from the operation object 50 has not ended (No), the processing of step S65 continues. On the other hand, if the receiving unit 23 determines that the reception of the reception command from the operation object 50 has ended (Yes), in step S67, the second notification unit 32 ends the notification to the inside.

[0083] [effect] In this embodiment, the device includes an electroencephalogram acquisition unit (brain information acquisition unit) 11 that acquires brain information of the user, a determination unit 21 that determines whether the brain information acquired by the electroencephalogram acquisition unit 11 is brain information corresponding to an execution command that means the operation content for operating the operation object 50, an execution unit 22 that executes processing corresponding to the execution command when the determination unit 21 determines that brain information corresponding to the execution command has been acquired, and a first notification unit 31 that, when a first processing signal corresponding to the execution command is being transmitted to the operation object 50 or a second processing signal is being received from the operation object 50, notifies the surroundings of at least one of the user and the operation object 50 that the first processing signal or the second processing signal is being processed.

[0084] Therefore, the content of the notification from the first notification unit 31 can notify the user and those around the operation target object 50 of the user's recalled state and the robot's operating state, allowing people in the vicinity to determine the user's state and predict the robot's operation, and preventing people nearby from touching the robot or talking to the user, which could lead to the user recalling an incorrect command. As a result, by notifying the outside world of the state when the user is sending or receiving a command, the safety of the execution of electroencephalogram commands can be ensured.

[0085] In this embodiment, the determination unit 21 determines whether the brain information acquired by the electroencephalogram acquisition unit 11 corresponds to a start command, which means that an execution command can be received, and the first notification unit 31 notifies that processing is in progress when the determination unit 21 determines that brain information corresponding to the start command has been acquired. This ensures that the user has enough time to think between recalling the start command and recalling the execution command, ensuring that the appropriate execution command is recalled without error.

[0086] Furthermore, in this embodiment, the determination unit 21 determines that the brain information corresponds to a start command when the coincidence rate between the brain information acquired by the EEG acquirer 11 and the brain information of the start command determination value is equal to or higher than a preset first coincidence rate, and determines that the brain information corresponds to an execution command when the coincidence rate between the brain information acquired by the EEG acquirer 11 and the brain information of the execution command determination value is equal to or higher than a second coincidence rate that is higher than the first coincidence rate. Therefore, by setting the determination level of the coincidence rate of the start command to a low level, it is possible to eliminate individual differences in the determination of the start command, enable standardization of start commands, and simplify the program.

[0087] In the above-described embodiment, the reception of the execution command is started when the brain wave acquisition unit 11 determines that the acquired brain information corresponds to the start command, but the reception of the execution command may also be terminated when the brain wave acquisition unit 11 determines that the acquired brain information corresponds to the end command.

[0088] The operation control device 10 according to the present invention has been described above, but it may be embodied in various different forms other than the above-described embodiment.

[0089] Each component of the illustrated operation control device 10 is a functional concept and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of each device is not limited to that shown in the figure, and all or part of them may be functionally or physically distributed or integrated in any unit depending on the processing load and usage status of each device.

[0090] The configuration of the operation control device 10 is realized, for example, as software, by a program loaded into memory. In the above embodiment, the functional blocks are described as being realized by the cooperation of these hardware and software. In other words, these functional blocks can be realized in various forms, using only hardware, only software, or a combination of both.

[0091] The above-described components include those that can be easily imagined by a person skilled in the art and those that are substantially the same. Furthermore, the above-described configurations can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the configurations are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0092] 10 Operation control device 11. Brain wave acquisition unit (brain information acquisition unit) 12 Brainwave Decoder 13 Starting Command Buffer 14 Execution Command Buffer 15 Storage section 16 Processing section 17 Display control unit 18 Display 19 Communications Department 21 Judgment section 22 Executive Department 23 Receiving unit 31 First Notification Unit (Notification Unit) 32 Second Notification Department 50 Operational Object

Claims

1. a brain information acquisition unit that acquires brain information of a user; a determination unit that determines whether the brain information acquired by the brain information acquisition unit corresponds to an execution command that indicates an operation content for operating an operation target object; an execution unit that executes a process corresponding to the execution command when the determination unit determines that brain information corresponding to the execution command has been acquired; a notification unit that, when a first processing signal corresponding to the execution command is being transmitted to the operation object or a second processing signal is being received from the operation object, notifies a periphery of at least one of the user and the operation object that the first processing signal or the second processing signal is being processed; Equipped with The determination unit When a matching rate between the brain information acquired by the brain information acquisition unit and a start command determination value for determining that the brain information is a start command meaning that the execution command can be accepted is equal to or greater than a first matching rate, the brain information acquired by the brain information acquisition unit is determined to be brain information corresponding to the start command; When a matching rate between the brain information acquired by the brain information acquisition unit and the brain information of the execution command determination value is equal to or higher than a second matching rate that is higher than the first matching rate, the brain information acquired by the brain information acquisition unit is determined to be the brain information corresponding to the execution command. Operation control device.

2. a brain information acquisition unit that acquires brain information of a user; a determination unit that determines whether the brain information acquired by the brain information acquisition unit corresponds to an execution command that indicates an operation content for operating an operation target object; an execution unit that executes a process corresponding to the execution command when the determination unit determines that brain information corresponding to the execution command has been acquired; a notification unit that notifies that processing is in progress when the determination unit determines that brain information corresponding to a start command indicating that the execution command can be accepted has been acquired; and Equipped with The determination unit When a matching rate between the brain information acquired by the brain information acquisition unit and a start command determination value for determining that the brain information is the start command is equal to or greater than a predetermined first matching rate, the brain information acquired by the brain information acquisition unit is determined to be brain information corresponding to the start command; When a matching rate between the brain information acquired by the brain information acquisition unit and the brain information of the execution command determination value is equal to or higher than a second matching rate that is higher than the first matching rate, the brain information acquired by the brain information acquisition unit is determined to be the brain information corresponding to the execution command. Operation control device.

3. acquiring brain information of a user; a step of determining whether the acquired brain information corresponds to an execution command indicating operation content for operating an operation target object; executing a process corresponding to the execution command when it is determined that brain information corresponding to the execution command has been acquired; a step of notifying a surrounding of at least one of the user and the operation object that the first processing signal or the second processing signal is being processed when the first processing signal corresponding to the execution command is being transmitted to the operation object or when the second processing signal is being received from the operation object; When a matching rate between the acquired brain information and a start command determination value for determining that the brain information is a start command meaning that the execution command can be accepted is equal to or greater than a first matching rate, the acquired brain information is determined to be brain information corresponding to the start command; determining that the acquired brain information is the brain information corresponding to the execution command when a matching rate between the acquired brain information and the brain information of the execution command determination value is equal to or higher than a second matching rate that is higher than the first matching rate; An operation control method including:

4. acquiring brain information of a user; a step of determining whether the acquired brain information corresponds to an execution command indicating operation content for operating an operation target object; executing a process corresponding to the execution command when it is determined that brain information corresponding to the execution command has been acquired; a step of notifying a surrounding of at least one of the user and the operation object that the first processing signal or the second processing signal is being processed when the first processing signal corresponding to the execution command is being transmitted to the operation object or when the second processing signal is being received from the operation object; When a matching rate between the acquired brain information and a start command determination value for determining that the brain information is a start command meaning that the execution command can be accepted is equal to or greater than a first matching rate, the acquired brain information is determined to be brain information corresponding to the start command; determining that the acquired brain information is the brain information corresponding to the execution command when a matching rate between the acquired brain information and the brain information of the execution command determination value is equal to or higher than a second matching rate that is higher than the first matching rate; A program that causes a computer to execute the following.

Images