Information processing system, information processing terminal, and recall operation recognition method

Abstract

To provide an information processing system, an information processing terminal, and a recall operation recognition method which recognize operation recalled by a user and apply the operation to operation of an information processing terminal.SOLUTION: An information processing system includes a brain wave detection device and an information processing terminal. The brain wave detection device detects a brain wave of a user, and converts the detected brain wave into data and transmits the data to the information processing terminal, on the basis of a conversion dictionary for converting the feature of the brain wave into data capable of being processed by the information processing terminal. The information processing terminal executes corresponding processing corresponding to the data, on the basis of the received data. The information processing terminal transmits a data group relating to an instruction input used in software during operation of the information processing terminal, to the brain wave detection device, during or after execution of the corresponding processing. The brain wave detection device stores the received data group in a storage device, and restricts the data to be converted on the basis of the data group.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing terminal, and a method for recognizing a recall operation. [Background technology]

[0002] A technique for changing the state of an application based on detected neurological intent data of a user is disclosed in Patent Document 1. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent No. 9,864,431 Summary of the Invention [Problem to be solved by the invention]

[0004] Meanwhile, in recent years, the opportunities for users to operate information processing terminals such as smartphones, tablet terminals, and laptop computers with touch panels have increased significantly.

[0005] However, Patent Document 1 does not take into consideration at all the case where an operation recalled by a user is recognized and applied to the operation of an information processing terminal. In other words, there is no description at all about a specific method or a feasible method for recognizing an operation recalled by a user and applying it to the operation of an information processing terminal.

[0006] In particular, for users who have difficulty performing operations using their own fingers, there is a very high demand for a technique that recognizes an operation that the user has in mind and applies it to the operation of an information processing terminal.

[0007] In view of these circumstances, there is a need for a technique that can recognize an operation that a user has in mind and apply it to the operation of an information processing terminal. [Means for solving the problem]

[0008] Among the inventions disclosed in this application, a brief summary of representative inventions is as follows.

[0009] An information processing system according to a representative embodiment of the present invention includes an electroencephalogram detection device and an information processing terminal, the electroencephalogram detection device including a detection device for detecting an electroencephalogram of a user, a first storage device for storing a conversion dictionary for converting characteristics of the detected electroencephalogram into data processable by the information processing terminal, a first processing device for converting the electroencephalogram detected by the detection device into data based on the conversion dictionary, and a first communication device for transmitting and receiving data to and from the information processing terminal, the information processing terminal including a second communication device for transmitting and receiving data to and from the electroencephalogram detection device, and a second processing device for executing corresponding processing corresponding to the data received by the second communication device based on the data received by the second processing device, the second processing device transmitting a group of data related to an instruction input used in software running on the information processing terminal to the electroencephalogram detection device via the second communication device during or after execution of a collaborative processing that enables the information processing terminal and the electroencephalogram detection device to operate in collaboration, and the first processing device storing the group of data received via the first communication device in the first storage device and executing a conversion processing using the group of data. Effect of the Invention

[0010] The effects obtained by the representative inventions among those disclosed in this application will be briefly described as follows.

[0011] According to the representative embodiment of the present invention, an operation that a user has in mind can be recognized and applied to the operation of an information processing terminal. [Brief description of the drawings]

[0012] [Figure 1] 1 is an external view of an information processing system according to a first embodiment. [Diagram 2]FIG. 2 is a diagram illustrating an example of a hardware configuration of a headset and a smartphone in the information processing system. [Diagram 3] FIG. 2 is a diagram showing a configuration of functional blocks of a headset and a smartphone in an information processing system. [Figure 4] FIG. 11 is a diagram showing an example of a priority order of operation input methods. [Diagram 5] FIG. 4 is a diagram showing an example of a first dictionary. [Figure 6] FIG. 11 is a diagram showing an example of a second dictionary. [Figure 7] FIG. 13 is a diagram showing an example of a list of operation types by input mode. [Figure 8] FIG. 13 is a diagram showing an example of a list of operation types by input mode. [Figure 9] FIG. 13 is a diagram showing an example of a display of a smartphone in a state where only the operating system is being used. [Figure 10A] FIG. 2 is a diagram showing a processing flow of the information processing system according to the first embodiment. [Figure 10B] FIG. 2 is a diagram showing a processing flow of the information processing system according to the first embodiment. [Figure 10C] FIG. 2 is a diagram showing a processing flow of the information processing system according to the first embodiment. [Figure 10D] FIG. 2 is a diagram showing a processing flow of the information processing system according to the first embodiment. [Figure 11] FIG. 13 is an image diagram of an operation type list for an operating system. [Figure 12] FIG. 13 is a diagram showing an example of a display on a smartphone in a state where a first application is being used. [Figure 13] FIG. 13 is an image diagram of a first application operation type list. [Figure 14] FIG. 13 is a diagram showing an example of a display on a smartphone in a state where a learning mode is selected. [Figure 15] FIG. 13 is a diagram showing an example of a display in which the displayed pointer is changed to another pointer and an operation instruction is output. [Figure 16]FIG. 13 is a diagram showing an example display in which the displayed pointer is further changed to another pointer and an operation instruction is output. [Figure 17] FIG. 13 is a diagram showing an example of a display on the smartphone when the software in use is a second application and the input mode is a pointer input mode. [Figure 18] FIG. 13 is a diagram showing an example of a display in a state where a front window is tapped and expanded. [Figure 19] FIG. 13 is a diagram showing an example of the display of the smartphone when the software in use is a second application and the input mode is a character operation input mode. [Figure 20] FIG. 13 is an image diagram of a second application operation type list. [Figure 21] FIG. 11 is a diagram showing a configuration of functional blocks of a headset and a smartphone in an information processing system according to a second embodiment. [Figure 22] FIG. 11 is a diagram showing an example of a third dictionary. [Figure 23A] FIG. 11 is a process flow diagram of Example 1 of the information processing system according to the second embodiment. [Figure 23B] FIG. 11 is a process flow diagram of Example 1 of the information processing system according to the second embodiment. [Figure 23C] FIG. 11 is a process flow diagram of Example 1 of the information processing system according to the second embodiment. [Figure 23D] FIG. 11 is a process flow diagram of Example 1 of the information processing system according to the second embodiment. [Figure 24A] FIG. 11 is a process flow diagram of Example 2 of the information processing system according to the second embodiment. [Figure 24B] FIG. 11 is a process flow diagram of Example 2 of the information processing system according to the second embodiment. [Figure 24C] FIG. 11 is a process flow diagram of Example 2 of the information processing system according to the second embodiment. [Figure 24D] FIG. 11 is a process flow diagram of Example 2 of the information processing system according to the second embodiment. [Diagram 25]FIG. 11 is a diagram showing a configuration of functional blocks of a headset and a smartphone in an information processing system according to a third embodiment. [Figure 26A] FIG. 11 is a diagram showing a processing flow of an information processing system according to a third embodiment. [Figure 26B] FIG. 11 is a diagram showing a processing flow of an information processing system according to a third embodiment. [Figure 26C] FIG. 11 is a diagram showing a processing flow of an information processing system according to a third embodiment. [Figure 26D] FIG. 11 is a diagram showing a processing flow of an information processing system according to a third embodiment. [Figure 27] FIG. 11 is a diagram showing a configuration of functional blocks of a headset and a smartphone in an information processing system according to a fourth embodiment. [Figure 28A] FIG. 13 is a diagram showing a processing flow of an information processing system according to a fourth embodiment. [Figure 28B] FIG. 13 is a diagram showing a processing flow of an information processing system according to a fourth embodiment. [Figure 28C] FIG. 13 is a diagram showing a processing flow of an information processing system according to a fourth embodiment. [Figure 28D] FIG. 13 is a diagram showing a processing flow of an information processing system according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of the present invention will be described. Note that each embodiment described below is an example for realizing the present invention, and does not limit the technical scope of the present invention.

[0014] In addition, in the following embodiments, components having the same functions are denoted by the same reference numerals, and repeated explanations thereof will be omitted unless particularly necessary.

[0015] (Embodiment 1) An information processing system according to a first embodiment of the present invention will be described.

[0016] Overview of the information processing system The information processing system according to the first embodiment is a system that recognizes and executes an operation that a user has thought up in his / her head in an information processing terminal that the user is using by detecting the user's brain waves. Here, this operation is called an operation by the recall input method. In the information processing system, a dictionary is prepared in advance and stored, which is a table in which the types of operations accepted by the information processing terminal (hereinafter also referred to as operation types) and the types of the user's brain waves (hereinafter also referred to as brain wave types) that occur when the user recalls the operation types are associated with each operation type. In addition, for each software and input mode used in the information processing terminal, an operation type list that lists limited operation types that are considered necessary under the conditions is also prepared and stored in advance. When the software and input mode being used are detected, an operation type list corresponding to the conditions is selected and applied to the dictionary, and the operation types to be recognized are narrowed down to those necessary at that time. With such a function, the burden of processing to distinguish the types of brain waves is reduced, the accuracy of operation recognition is improved, and the practicality of operations by the recall input method is enhanced.

[0017] Appearance and hardware configuration The appearance and hardware configuration of the information processing system according to the first embodiment will be described.

[0018] Fig. 1 is an external view of an information processing system according to embodiment 1. As shown in Fig. 1, the information processing system 1 according to embodiment 1 includes a headset (brain wave detecting device) 2 and a smartphone (information processing terminal) 3. The headset 2 is worn on the head of a user 9. The smartphone 3 is an information processing terminal operated by the user 9.

[0019] The headset 2 and the smartphone 3 are configured to enable two-way communication via a wireless connection using radio waves 4a and 4b. That is, the headset 2 and the smartphone 3 are capable of operating in cooperation with each other. Note that the headset 2 and the smartphone 3 may also be configured to enable two-way communication via a wired connection using a connector cable 5.

[0020] FIG. 2 is a diagram illustrating an example of the hardware configuration of a headset and a smartphone in the information processing system.

[0021] 2, the headset 2 includes an operating device 21, an antenna 22, a connector 23, an electrode device for detecting brain waves 24, an interface 25, a processor 26, a memory 27, a storage 28, and a battery 29. These are connected to each other via a bus 20, and are capable of transmitting and receiving electrical signals or data. Note that the headset 2 may receive power from a commercial power source via an AC adapter or the like, instead of the battery 29.

[0022] The smartphone 3 includes an operation device 31, a touch panel image display device 32, an antenna 33, a connector 34, a microphone 35, a speaker 36, a camera 37, an interface 38, a processor 39, a memory 40, a storage 41, and a battery 42. These are connected to each other via a bus 30, and are capable of transmitting and receiving electrical signals or data.

[0023] The connectors 23 and 34 are used when connecting the headset 2 and the smartphone 3 by wire. The antennas 22 and 33 are used when connecting the headset 2 and the smartphone 3 by wireless. The brain wave detection electrode device 24 receives a signal representing the user's brain waves used to recognize an operation imagined by the user. The interface 25 converts electrical signals from the operation device 21, the antenna 22, the connector 23, the brain wave detection electrode device 24, etc. into data and transmits them to the processor 26, etc., or converts data from the processor 26, the memory 27, the storage 28, etc. into electrical signals and transmits them to the connector 23 or the antenna 22. The interface 38 converts electrical signals from the operation device 31, the touch panel image display device 32, the antenna 33, the connector 34, the microphone 35, the camera 37, etc. into data and transmits them to the processor 39, etc., or converts data from the processor 39, the memory 40, the storage 41, etc. into electrical signals and transmits them to the connector 34, the speaker 36, or the antenna 33.

[0024] The processors 26 and 39 are devices that execute arithmetic processing or data processing, and are configured, for example, by a central processing unit (CPU) or a micro processing unit (MPU). The memories 27 and 40 are devices that temporarily store data, and are configured, for example, by a semiconductor storage device such as a random access memory (RAM). The storages 28 and 41 are devices that store data or various types of data, and are configured, for example, by a non-volatile storage device such as a solid state drive (SSD), a hard disk drive (HDD), or a flash memory.

[0025] <Functional configuration> The functional configuration of the information processing system 1 according to the first embodiment will be described.

[0026] 3 is a diagram showing a configuration of functional blocks of a headset and a smartphone in an information processing system. Each functional block of the headset 2 is realized by executing a predetermined program stored in storage 28 using processor 26, memory 27, etc., and cooperating with hardware. Similarly, each functional block of the smartphone 3 is realized by executing a predetermined program stored in storage 41 using processor 39, memory 40, etc., and cooperating with hardware.

[0027] 3, the headset 2 includes, as functional blocks, a communication device (first communication device) 200, a detection device (detection device) 201, an operation type list application device 202, an electroencephalogram type discrimination device 203, a recall operation recognition device 204, a learning device (learning device) 205, a first control device 206, and a first storage device (first storage device) 210. The operation type list application device 202, the electroencephalogram type discrimination device 203, the recall operation recognition device 204, and the first control device 206 are examples of the first processing device in the present invention.

[0028] 3, the smartphone 3 includes, as functional blocks, a communication device (second communication device) 300, an operation type list selection device 301, a used software detection device 302, an input mode detection device 303, a recall operation reception device 304, a touch operation reception device 305, a voice operation reception device 306, an operation reception control device 307, a second control device 308, and a second storage device (second storage device, third storage device) 310. The operation type list selection device 301, the used software detection device 302, the input mode detection device 303, the recall operation reception device 304, the touch operation reception device 305, the voice operation reception device 306, the operation reception control device 307, and the second control device 308 are examples of the second processing device in the present invention.

[0029] Headset function blocks Each functional block of the headset 2 will be described.

[0030] The first storage device 210 stores data necessary for proper operation of the headset 2. In this embodiment, the first storage device 210 stores a first dictionary (conversion dictionary, first table) 211. The first dictionary 211 is configured with a table in which operation types accepted by the smartphone 3 and electroencephalogram types corresponding to the operation types are associated with each operation type.

[0031] Communication device 200 is connected wirelessly or via a wire to communication device 300 of smartphone 3, and performs two-way communication. The communicated signals are data input and output in each functional block constituting headset 2. As a wireless communication standard, for example, Bluetooth (registered trademark) can be used.

[0032] The detection device 201 detects the brain waves 9w of the user 9 based on electrical signals received by an electrode device 24 for detecting brain waves placed on the user's head.

[0033] The operation type list application device 202 limits the operation type to be recognized by the reminded operation recognition device 204 from among the operation types included in the first dictionary 211, according to the type of software or input mode being used in the smartphone 3. More specifically, the operation type list application device 202 receives information representing an operation type list (data group) from the smartphone 3, and applies the operation type list as an operation type to be limitedly recognized. In this embodiment, the operation type list is prepared in advance as a software-specific operation type list created for each software to be used. In addition, the software-specific operation type list includes one or more input mode-specific operation type lists created for each input mode to be used. Based on information representing an input mode used in the smartphone 3, the operation type list application device 202 applies an input mode-specific operation type list corresponding to the input mode as an operation type to be limitedly recognized.

[0034] The electroencephalogram type discrimination device 203 discriminates an electroencephalogram type corresponding to the electroencephalogram detected by the detection device 201 from among the electroencephalogram types included in the first dictionary 211. At this time, it discriminates an electroencephalogram type corresponding to the detected electroencephalogram from among the electroencephalogram types corresponding to the operation types limited by the operation type list application device 202. More specifically, the electroencephalogram type discrimination device 203 discriminates whether the electroencephalogram detected by the detection device 201 corresponds to any of a plurality of electroencephalogram types registered in advance in the first dictionary 211, or whether it does not correspond to any of the electroencephalogram types. At this time, the discriminated electroencephalogram types are limited to the electroencephalogram types corresponding to the operation types included in the applied operation type list.

[0035] The recollective operation recognition device 204 recognizes an operation recalled by the user 9 by referring to the first dictionary 211 based on the electroencephalogram type determined by the electroencephalogram type determination device 203. More specifically, the recollective operation recognition device 204 recognizes an operation type corresponding to the determined electroencephalogram type as an operation type input by the recollective input method, and outputs information on the recognized operation type to the smartphone 3.

[0036] The learning device 205 has a learning function for improving the recognition accuracy when recognizing an operation type from electroencephalograms. The learning device 205 accepts an evaluation by the user 9 for an operation recognized by the recall operation recognition device 204, and adjusts the waveform of the electroencephalogram type corresponding to the operation type included in the first dictionary 211 based on the operation to be evaluated and the electroencephalogram used to recognize the operation. More specifically, when an operation is performed on the smartphone 3 according to an operation recalled by the user, the user inputs an evaluation indicating that fact. The learning device 205 adjusts the electroencephalogram waveform or pattern of the electroencephalogram type in the first dictionary 211 based on the detected electroencephalogram and the input evaluation so as to improve the discrimination accuracy of the electroencephalogram type. In addition, the learning device 205 sequentially displays multiple types of pointers for the pointer that is displayed on the screen of the smartphone 3 and serves as a reference or starting point of the operation by the recall input method, and outputs an instruction for the operation. The user recalls the operation according to the instruction. The learning device 205 receives a user's evaluation of the recognized and executed operation, and based on the evaluation, ranks the multiple types of pointers in descending order of operation recognition accuracy.

[0037] The first control device 206 comprehensively controls each device or each functional block so that the headset 2 operates properly.

[0038] <Smartphone function blocks> Each functional block of the smartphone 3 will be described.

[0039] The second storage device 310 stores data necessary for the smartphone 3 to operate properly. In this embodiment, the second storage device 310 stores an operating system 311, a first application 312, a second application 313, a second dictionary (second table) 315, and a headset control tool 316. The first and second applications are various application software that run on the operating system 311. The first application 312 is an Internet browser application, and the second application 313 is a fighting game application. The headset control tool 316 is software for controlling a headset. Therefore, it is not included in the software used by the smartphone 3 here.

[0040] The second dictionary 315 is configured by a table in which software used by the smartphone 3 and an operation type list corresponding to the software are associated with each software. The operation type list is a list that lists a limited number of operation types that should be recognized while using the software. The second dictionary 315 will be described in detail later.

[0041] The communication device 300 is connected to the communication device 200 on the headset 2 side wirelessly or via a wired connection, and performs two-way communication.

[0042] The software-in-use detection device 302 detects software being used in the smartphone 3. In this embodiment, the above-mentioned operating system 311, first application 312, and second application 313 are considered as examples of the software being used. The operating system is, for example, Android (registered trademark), iOS (a trademark used by Apple Inc.), etc. Furthermore, the first and second applications are, for example, application software associated with icons displayed on the screen of the smartphone 3.

[0043] The operation type list selection device 301 selects an operation type list corresponding to the detected software in use by referring to the second dictionary 315. The operation type list is a list of operation types that is determined for each software by narrowing down only the operation types that should be recognized.

[0044] The input mode detection device 303 detects the input mode being used in the smartphone 3, and outputs input mode identification information, which is information for identifying the detected input mode, to the headset 2. Note that the input mode is selected by the software being used according to the execution state of the software. In this embodiment, a pointer input mode, a first text input mode, a second text input mode, a map operation input mode, and a character operation input mode are considered as examples of the input mode.

[0045] The pointer input mode is an input mode for moving the pointer and performing operations based on or starting from the position of the pointer. The first text input mode is an input mode for inputting text including letters, numbers, symbols, etc. using a virtual keyboard. The second text input mode is an input mode for directly specifying and inputting text including letters, numbers, symbols, etc. based on the user's brain waves. The map operation input mode is an input mode for performing operations on a displayed map. The character operation input mode is an input mode for performing operations on a character appearing in a game or the like.

[0046] When information (data) on the type of operation recognized by the recall operation recognition device 204 of the headset 2 is sent to the recall operation reception device 304, the recall operation reception device 304 accepts the type of operation as an operation using the recall input method (hereinafter also referred to as a recall operation) (response processing).

[0047] When a touch operation by the user 9 is detected on the touch panel image display device 32, the touch operation reception device 305 receives an operation type corresponding to the touch operation as an operation by a touch input method (hereinafter also referred to as a touch operation).

[0048] When a voice from the user 9 is detected through the microphone 35, the voice operation reception device 306 receives the type of operation corresponding to the voice as an operation by a voice input method (hereinafter also referred to as a voice operation).

[0049] The operation reception control device 307 controls the reception of operations so that, when operations using the operation input methods are received in duplicate, the operation using the operation input method with the highest priority is adopted and executed.

[0050] Fig. 4 is a diagram showing an example of the priority order of operation input methods. In this embodiment, as shown in Fig. 4, the priority order is, from highest to lowest, the touch input method, the voice input method, and the recall input method. In this example, if there is no operation by the touch input method or the voice input method, the operation reception control device 307 executes an operation by the recall input method, that is, an operation recognized by the recall operation recognition device 204. At this time, an operation is executed with a pointer displayed on the screen of the smartphone 3 as a reference or starting point.

[0051] The second control device 308 comprehensively controls each device or each functional block so that the smartphone 3 operates properly.

[0052] "First and Second Dictionary" FIG. 5 is a diagram illustrating an example of the first dictionary. As shown in FIG. 5, the first dictionary 211 is composed of a table in which the operation types used by the user when using the smartphone 3 are associated with the electroencephalogram types of the user's electroencephalograms that are thought to occur when the user recalls performing the operation of that operation type.

[0053] The electroencephalogram type is, for example, an electroencephalogram pattern having a characteristic waveform, amplitude, time length, etc. The operation type includes, for example, pointer up movement, pointer down movement, pointer left movement, pointer right movement, swipe up, swipe down, swipe left, swipe right, flick up, flick down, flick left, flick right, drag up movement, drag down movement, drag left movement, drag right movement, tap, double tap, long press, pinch in, pinch out, direct input of text, i.e., character type, etc. Character types include, for example, alphabet, hiragana, numbers, symbols, etc.

[0054] It should be noted that swiping, flicking, dragging, tapping, double tapping, long press, pinch-in, and pinch-out are performed based on the position of the pointer as a reference or starting point.

[0055] Here, the meaning of each operation type will be briefly explained. A tap is an action of touching the screen with one finger, and is the same as a click on a PC. A double tap is an action of quickly touching the screen twice with one finger, and is the same as a double click on a PC. A long press is an action of touching the screen with one finger for a while, and an action corresponding to the long press occurs. A pinch in is an action of touching the screen with two fingers and narrowing the finger width, and is mainly used to reduce the screen. A pinch out is an action of touching the screen with two fingers and widening the finger width, and is mainly used to enlarge the screen. A drag is an action of touching the screen with one finger and moving it up, down, left and right, and an action according to the finger movement occurs. A swipe is an action of touching the screen with one finger and sliding the finger up, down, left and right, and is mainly used to scroll the screen. A flick is an action of touching the screen with one finger and sweeping the finger up, down, left and right in a short period of time, and is often used as an input method called flick input when inputting characters using a software keyboard.

[0056] Incidentally, the initial settings of the electroencephalogram patterns for each electroencephalogram type contained in the first dictionary 211 are obtained from neurological research, experiments and analyses related to electroencephalograms, and the like carried out in advance.

[0057] FIG. 6 is a diagram showing an example of the second dictionary. As shown in FIG. 6, the second dictionary 315 is configured by a table in which the type of software executed on the smartphone 3 and a software-specific operation type list that lists the operation types used when the software is executed are associated with each type of software. The software-specific operation type list is further configured by one or more input mode-specific operation type lists. The input mode-specific operation type list is prepared for each input mode selected according to the execution state of the software. The input mode-specific operation type list lists the operation types used in the input mode being used.

[0058] As shown in FIG. 6, in this embodiment, as operation type lists by software, an operating system operation type list LA associated with the operating system 311, a first application operation type list LB associated with the first application 312, and a second application operation type list LC associated with the second application 313 are prepared.

[0059] FIG. 7 is a diagram showing an example of an operation type list by input mode. As shown in FIG. 7, in this embodiment, the types of input modes are assumed to be a pointer input mode, a first text input mode, a second text input mode, a map operation input mode, a character operation input mode, and the like. The pointer input mode is a mode that accepts input of various operations based on the position of a pointer. The first text input mode is a mode that accepts input of various operations for inputting text using a virtual keyboard. The second text input mode is a mode that accepts input of various operations for inputting text directly from electroencephalograms. The map operation input mode is a mode that accepts input of an operation for moving / enlarging / reducing a displayed map area, or various operations for causing an action on the map. The character operation input mode is a mode that accepts input of various operations for moving a character such as an avatar on the screen or causing an action. In addition, as operation type lists for each input mode, an operation type list Lp for the pointer input mode, an operation type list Lt1 for the first text input mode, an operation type list Lt2 for the second text input mode, an operation type list Lm for the map operation input mode, and an operation type list Lf for the character operation input mode are prepared.

[0060] As shown in Fig. 6, in this embodiment, the operating system operation type list LA includes an operation type list Lp for the pointer input mode and an operation type list Lt1 for the first text input mode. The first application operation type list LB includes an operation type list Lp for the pointer input mode, an operation type list Lt1 for the first text input mode, and an operation type list Lm for the map operation input mode. The second application operation type list LC includes an operation type list Lp for the pointer input mode, an operation type list Lt1 for the first text input mode, and an operation type list Lf for the character operation input mode.

[0061] FIG. 8 is a diagram showing an example of an operation type list by input mode. As shown in FIG. 8, the operation type list Lp for the pointer input mode includes, for example, pointer up / down / left / right movement, swipe up / down / left / right, drag up / down / left / right movement, tap, long press, etc. as operation types. The operation type list Lt1 for the first text input mode includes, for example, pointer up / down / left / right movement, drag up / down movement, flick up / down / left / right, tap, etc. The operation type list Lt2 for the second text input mode includes, for example, operations for directly inputting individual alphabets, numbers, and symbols. The operation type list Lm for the map operation input mode includes, for example, pointer up / down / left / right movement, swipe up / down / left / right, drag up / down / left / right movement, pinch in, pinch out, tap, double tap, etc. The operation type list Lf for the character operation input mode includes, for example, pointer right movement, pointer left movement, flick up / down / left, tap, double tap, etc.

[0062] <<Operation of Information Processing System>> Example 1 Next, a description will be given of the operation of the information processing system according to the embodiment 1. In the embodiment 1, it is assumed that the software and input mode used in the smartphone 3 are the operating system 311 and the pointer input mode at first, and are switched to the first application 312 (browser) and the first text input mode midway through.

[0063] 9 is a diagram showing an example of a display of the smartphone in a state where only the operating system is being used. As shown in FIG. 9, the smartphone 3 is provided with a touch panel display screen 3d and a microphone 35.

[0064] On the touch panel display screen 3d, a plurality of icons 51 corresponding to the individual applications and a pointer 90 are displayed. The interior of the pointer 90 is transparent or semi-transparent so that an image in an area overlapping with the pointer 90 can be seen. In addition, an icon 91 for returning to the home screen, an icon 92 for returning to the previous state, and an icon 93 for displaying a menu screen or the like are displayed at the bottom of the touch panel display screen 3d. A "Like!" button 96 is also displayed on the touch panel display screen 3d. When the recall operation is successfully performed, that is, when the operation is performed as recalled by the user, the user presses the "Like!" button. As a result, an evaluation indicating that the current recall operation was successfully performed is input, and the evaluation information is utilized for recognizing the operation type, that is, for improving the accuracy of determining the electroencephalogram type.

[0065] The user can perform touch operations by touching the touch panel display screen 3d. The user can also perform voice operations by inputting voice into the microphone 35. If the user is temporarily unable to perform a good recall operation or if a quick operation is required, the user can use touch operations or voice operations as necessary.

[0066] 10A to 10D are diagrams showing the process flow of the information processing system according to embodiment 1. Note that the flow diagrams shown in Fig. 10A to 10D show one flow divided into four.

[0067] 10A mainly shows a process flow for preparing the recall operation. This process flow includes a process for establishing a link between the headset 2 and the smartphone 3. It also includes a process for limiting the types of operations to be recognized in the recall operation according to the software and input mode currently being used in the smartphone 3.

[0068] 10B mainly shows a process flow for executing a recall operation. This process flow includes a process for detecting the user's brain waves in the headset 2, recognizing the type of operation, and outputting the result to the smartphone 3. It also includes a process for accepting operations by each operation input method in the smartphone 3, and adopting and executing an operation with a high priority.

[0069] 10C mainly shows a process flow in the learning mode. This process flow includes a process of learning a form of a pointer to be displayed on the screen of the smartphone 3 that improves the recognition accuracy of the recall operation.

[0070] 10D mainly shows a process flow for ending the cooperation. This process flow includes a process related to a procedure for ending the cooperation between the headset 2 and the smartphone 3.

[0071] <Processing flow for preparing for recall operation> First, a flow of a process for establishing cooperation between the headset 2 and the smartphone 3, and a process for limiting the type of operation to be recognized by a recall operation depending on the software used in the smartphone 3 and the operation input mode will be described.

[0072] 10A, first, the second control device 308 outputs a link request signal to the headset 2 (S101). The first control device 206 outputs a link permission signal to the smartphone 3 in response to the link request signal (S102). The second control device 308 acquires the link permission signal (S103). As a result, linkage between the headset 2 and the smartphone 3 is established.

[0073] When the link is established, the second control device 308 starts up the headset control tool 316 stored in the second storage device 310 (S104), making it possible to control data transmission / reception or operation of the headset 2.

[0074] Next, the used software detection device 302 detects software used in the smartphone 3 (S105). The operation type list selection device 301 refers to the second dictionary 315 shown in Fig. 6 and selects a software-specific operation type list associated with the detected software (S106).

[0075] Here, since it is assumed that the operating system 311 is being used, the operating system 311 is detected as the software in use, and the operating system operation type list LA is selected as the software-specific operation type list. 11 is a conceptual diagram of the operating system operation type list LA. As shown in FIG 11, the operating system operation type list LA is made up of a pointer input mode operation type list Lp and a first text input mode operation type list Lt1.

[0076] When the software-specific operation type list is selected, the second control device 308 outputs the selected software-specific operation type list to the headset 2 (S107). Here, the operating system operation type list LA is output.

[0077] When the selected software-specific operation type list is output to the headset 2, the first control device 206 of the headset 2 acquires the software-specific operation type list (S108).

[0078] The input mode detection device 303 of the smartphone 3 detects the input mode currently being used in the smartphone 3 (S109). The second control device 308 outputs input mode identification information indicating the detected input mode to the headset 2 (S110).

[0079] Here, since it is assumed that the pointer input mode is used, input mode identification information that is information for identifying the pointer input mode is output to the headset 2.

[0080] The first control device 206 of the headset 2 acquires the input mode identification information (S111). The operation type list application device 202 selects an input mode-specific operation type list corresponding to the input mode identified by the acquired input mode identification information from the software-specific operation type list, and applies the selected input mode-specific operation type list to the first dictionary 211 (S112). That is, a setting is performed to limit the operation types recognized from the electroencephalogram to the operation types included in the applied input mode-specific operation type list among the operation types registered in the first dictionary 211.

[0081] Here, the pointer input mode operation type list Lp is selected from the operating system operation type list LA. Then, a setting is made to limit the operation types to be recognized to those included in this list. As a result, the operation types to be recognized from the electroencephalogram are narrowed down to only those operation types considered necessary in the pointer input mode. That is, operation types considered unnecessary in the pointer input mode, specifically, operations such as flick, pinch in, pinch out, or double tap, are excluded from the operation types to be recognized. Therefore, the number of operation types to be recognized is reduced, erroneous recognition of operation types is suppressed, and recognition accuracy is improved.

[0082] <<Execution process flow of recall operation>> Next, a process of detecting the user's brain waves in the headset 2 to recognize the type of operation and outputting the result to the smartphone 3, and a process of executing the operation in the smartphone 3 while taking into account the priority order of each operation input method will be described.

[0083] As shown in Fig. 10B, the detection device 201 detects the user's brain waves (S201) and temporarily records the waveform in the first storage device 210. The brain wave type discrimination device 203 discriminates the brain wave type based on the recorded brain wave waveform, etc. (S202). At this time, among the brain wave types registered in the first dictionary 211, discrimination is limited to any of the brain wave types corresponding to the operation types included in the input mode-specific operation type list currently applied, and other brain wave types.

[0084] For example, suppose that the user recalls an operation of moving the pointer left. Then, the detection device 201 detects and records an electroencephalogram corresponding to the operation of moving the pointer left. Then, the electroencephalogram type discrimination device 203 compares the recorded electroencephalogram waveform with only the electroencephalogram type corresponding to the operation type included in the pointer input mode operation type list Lp among the electroencephalogram types registered in the first dictionary 211. At this time, the electroencephalogram type corresponding to the operation type not included in the pointer input mode operation type list Lp is excluded from the comparison target. For example, the electroencephalogram types S1 to S4 corresponding to swipes shown in FIG. 5 are excluded from the comparison target. Then, as a result of the comparison, the electroencephalogram type having the highest similarity, likelihood, or correlation value with respect to the recorded electroencephalogram is identified. If the discrimination of the electroencephalogram type is good, the electroencephalogram type P3 corresponding to the pointer left movement is identified from the electroencephalogram types shown in FIG. 5.

[0085] In this embodiment, the determination is limited to electroencephalogram types corresponding to operation types included in the pointer input mode operation type list Lp as shown in FIG.

[0086] The recollection operation recognition device 204 recognizes that the recollection operation is not performed when the detected electroencephalogram is not determined to be one of the limited electroencephalogram types. The electroencephalogram type is determined, for example, by selecting the electroencephalogram type having the highest similarity to the recorded electroencephalogram.

[0087] The recall operation recognition device 204 refers to the first dictionary 211, identifies an operation type corresponding to the determined electroencephalogram type, and recognizes the identified operation type as the operation type recalled by the user (S203). When the operation type is recognized, the first control device 206 outputs recognized operation type information representing the recognized operation type to the smartphone 3 (S204).

[0088] The second control device 308 acquires recognition operation type information indicating the recognized operation type (S205). Here, the recall operation reception device 304 accepts the operation type indicated by the acquired recognition operation type information as an operation by the recall input method. In addition, the touch operation reception device 305 accepts an operation by the touch input method, and the voice operation reception device 306 accepts an operation by the voice input method (S206).

[0089] The operation reception control device 307 detects the presence or absence of an operation by each operation input method from the reception status of the operation by each operation input method. Specifically, it detects the presence or absence of an operation by each of the touch input method, the voice input method, and the recall input method. Then, it identifies the operation by the operation input method with the highest priority among the detected operations (S207), and executes the identified operation (S208).

[0090] Here, the second control device 308 outputs a learning evaluation for the recollecting operation to the headset 2 (S209). The learning evaluation includes, for example, information on whether or not pressing of the "Things!" button by the user is detected. The learning device 205 executes a learning process based on the information so as to improve the recognition accuracy of the recollecting operation (S210). For example, if it is recognized from the information that pressing of the "Things!" button is detected, it is determined that the recollecting operation is performed properly, and the characteristics of the electroencephalogram detected at that time are fed back to the first dictionary 211 and reflected in the electroencephalogram pattern corresponding to the electroencephalogram type to be determined.

[0091] When the learning process is executed, the second control device 308 judges whether the input mode being used has changed (S211). If it is judged that the input mode being used has changed (S211: Yes), the process returns to step S109. If it is judged that the input mode has not changed (S211: No), the process proceeds to the next judgment. The second control device 308 judges whether the software being used has changed (S212). If it is judged that the software being used has changed (S212: Yes), the process returns to step S105. If it is judged that the software being used has not changed (S212: No), the process proceeds to the next judgment. The second control device 308 judges whether the learning mode has been selected by the user (S213). If it is judged that the learning mode has been selected (S213: Yes), the process proceeds to step S301. If it is judged that the learning mode has not been selected (S213: No), the process proceeds to the next judgment. The second control device 308 judges whether or not the linkage should be terminated (S214). Situations in which the linkage should be terminated are, for example, when an instruction to terminate the linkage is input, when a failure occurs in a part of the linkage, when the remaining battery charge is low, etc. If it is judged that the linkage should be terminated (S214: Yes), the process proceeds to step S401. If it is judged that the linkage should not be terminated (S214: No), the process returns to step S201.

[0092] Here, it is assumed that the user performs an operation of tapping, for example, icon B corresponding to the first application (Internet browser application) 312, and the software used by the smartphone 3 is switched from only the operating system 311 to the first application 312. In this case, when the process of step S212 is executed, it is determined that the software being used has changed (S212: Yes), and the process returns to step S105.

[0093] Fig. 12 is a diagram showing an example of a display of the smartphone in a state where the first application is being used. As shown in Fig. 12, a text box 54 for inputting a search keyword to be used in a Web search and a software keyboard 94 for inputting text are displayed on the touch panel display screen 3d of the smartphone 3.

[0094] The used software detection device 302 detects the first application 312 after the switch (S105). The operation type list selection device 301 refers to the second dictionary 315 shown in Fig. 6 and selects the first application operation type list LB corresponding to the first application 312 (S106). 13 is a conceptual diagram of the first application operation type list LB. As shown in FIG. 13, the first application operation type list LB is made up of an operation type list Lp for the pointer input mode, an operation type list Lt1 for the first text input mode, and an operation type list Lm for the map operation input mode.

[0095] The second control device 308 outputs the selected first application operation type list LB to the headset 2 (S107). The first control device 206 of the headset 2 acquires the first application operation type list LB (S108).

[0096] The input mode detection device 303 of the smartphone 3 detects the input mode currently selected in the smartphone 3 (S109). Here, it is assumed that the first text input mode is selected, so the first text input mode is detected. The second control device 308 outputs input mode identification information that identifies the detected first text input mode to the headset 2 (S110).

[0097] The first control device 206 of the headset 2 acquires the input mode specifying information (S111). The operation type list application device 202 identifies the operation type list Lt1 for the first text input mode corresponding to the first text input mode identified by the acquired input mode specifying information, and applies the operation type list Lt1 for the first text input mode to the first dictionary 211 (S112). That is, a setting is made to limit the operation types to be recognized from the electroencephalogram to those included in the operation type list Lt1 for the first text input mode, among the operation types included in the first dictionary 211.

[0098] As a result, the operation types to be recognized from the electroencephalogram are narrowed down to only the operation types considered necessary in the first text input mode. In other words, the operation types considered unnecessary in the first text input mode, specifically, operations such as swiping, pinching in, pinching out, long pressing, and double tapping, are excluded from the operation types to be recognized.

[0099] By such processing, unless the selection of the learning mode or the termination of the link occurs, the cycle of recognizing the recall operation by detecting the brain wave and executing the recognized recall operation continues. The user can realize the desired operation by manipulating the pointer 90 displayed on the screen of the smartphone 3 with the recall operation. However, if there is a touch operation or a voice operation, these operations are executed preferentially. Also, whenever the software used or the input mode is changed, a new operation type list is applied to the first dictionary 211. In other words, the operation types to be recognized, i.e., the brain wave types to be determined, are always appropriately narrowed down.

[0100] Learning mode processing flow Next, a process in the learning mode, that is, a learning process for improving the recognition accuracy of the operation type in the recall operation, will be described.

[0101] 10C, when the learning mode is selected, the second control device 308 outputs a learning request signal to the headset 2 (S301). The learning device 205 outputs a learning permission signal to the smartphone 3 in response to the learning request signal (S302). The second control device 308 acquires the learning permission signal. When the learning permission signal is acquired, the learning mode is established.

[0102] The second control device 308 selects one of multiple types of pointers prepared in advance (S304) and displays the selected pointer on the screen. The second control device 308 also outputs an operation instruction for the user (S305). The output of the operation instruction may be in the form of displaying an image or text, or in the form of outputting a sound.

[0103] Fig. 14 is a diagram showing a display example of a smartphone in a state where the learning mode is selected. As shown in Fig. 14, for example, a star-shaped pointer 90s is displayed on the touch panel display screen 3d, along with an instruction image 95s showing an operation instruction to move the pointer rightward. Also displayed is a "Like!" button 96 that the user presses when the recall operation is successfully performed. Here, the user recalls the operation according to the output operation instruction.

[0104] When the pointer is displayed and an operation instruction is output, the learning device 205 controls the detection device 201, the brain wave type discrimination device 203, and the recall operation recognition device 204 to detect the user's brain wave (S306), discriminate the brain wave type based on the detected brain wave (S307), and recognize and output the operation type (S308) by referring to the first dictionary 211 and the applied operation type list based on the discriminated brain wave type. The learning device 205 also stores the brain wave detected during the learning mode in the first storage device 210 in association with the timing of the detection. The second control device 308 receives the operation type output from the headset 2 and executes an operation corresponding to the operation type (S309).

[0105] If the executed operation is the same as the operation recalled by the user, the user presses the "Like!" button 96 and inputs an evaluation that means the recalled operation was successful. If an evaluation that the recalled operation was good is input, the second control device 308 accepts the evaluation (S310) and stores in the second storage device 310 the type of the operation instruction at that time, the time timing, and evaluation result information that associates the evaluation.

[0106] The second control device 308 judges whether to change the operation instruction (S311). The judgment criteria here are, for example, whether the output duration of the operation instruction has reached a certain time or more, whether the number of inputs of the evaluation that the recall operation is good has reached a certain number or more, etc. If it is judged that the operation instruction is to be changed (S311, Yes), the second control device 308 returns to step S305 and changes and outputs the operation instruction. If it is judged that the operation instruction is not to be changed (S311, No), the second control device 308 judges whether to change the displayed pointer (S312). The judgment criteria here are, for example, whether the number of changes of the operation instruction has reached a certain number or more, whether the number of inputs of the evaluation that the recall operation is good has reached a certain number or more, etc.

[0107] If it is determined that the displayed pointer is to be changed (S312, Yes), the second control device 308 returns to step S304 and selects another pointer.

[0108] Fig. 15 is a diagram showing a display example in which the displayed pointer is changed to another pointer and an operation instruction is output. In the example of Fig. 15, the displayed pointer is changed to a face-shaped pointer 90f, and an operation instruction image 95f meaning "flick downward" is displayed as the operation instruction.

[0109] Fig. 16 is a diagram showing a display example in which the displayed pointer is changed to another pointer and an operation instruction is output. In the example of Fig. 16, the displayed pointer is changed to a cross-shaped pointer 90c expressed in a conspicuous color such as red or blue, and an operation instruction image 95c meaning "drag to move left" is displayed as the operation instruction.

[0110] When it is determined that the pointer to be displayed is not to be changed (S312, No), the second control device 308 ends the learning mode and outputs the evaluation result information stored in the second storage device 310 to the headset 2 (S313). The learning device 205 adjusts the first dictionary 211 based on the acquired evaluation result information (S314). For example, based on the operation type when the recollection operation is evaluated as good and the waveform of the brain wave at that time, the brain wave pattern of the brain wave type registered in the first dictionary 211 is corrected and adjusted to one that better reflects the characteristics of the user's brain wave. This adjustment of the first dictionary 211 improves the discrimination accuracy of the brain wave type in the recollection operation, and improves the recognition accuracy of the operation type. In addition, the learning device 205 analyzes which pointer improves the discrimination accuracy of the brain wave, and determines the order of the pointers in order of improving accuracy (S315). The information on the ranking of the pointers is output to the smartphone 3. The smartphone 3 refers to the ranking of the pointers and allows any type of pointer to be selected as the pointer to be displayed. Alternatively, the pointer of the highest rank type is automatically selected.

[0111] When the learning mode is completed, the second control device 308 returns the execution step to S214 and continues the process.

[0112] <<Connection termination process flow>> When it is determined in step S214 that the collaboration should be ended (S214: Yes), the second control device 308 outputs a collaboration end request signal to the headset 2 (S401). Then, in response to the request signal, the first control device 206 outputs a collaboration end permission signal to the smartphone 3 (S402). The second control device 308 acquires the collaboration end permission signal (S403), and collaboration is ended.

[0113] According to the first embodiment, even if the software being used is changed midway, the operation types to be recognized by the recall operation are appropriately limited, so that erroneous recognition of the operation types is reliably suppressed and the recognition accuracy is improved.

[0114] Example 2 In the second embodiment, it is assumed that the software in use is the first application (Internet browser application) 312, and the input mode is initially the pointer input mode, which is switched to the map operation input mode midway.

[0115] Fig. 17 is a diagram showing an example of a display of a smartphone when the software in use is the second application and the input mode is the pointer input mode. In this embodiment, for example, as shown in Fig. 17, a window 52 and a window 53 including a map are displayed partially overlapping on the touch panel display screen 3d. A pointer 90 is also displayed at a position on the touch panel display screen 3d.

[0116] In this embodiment, in the flow shown in FIG. 10A, in step S105, the second application 313 is detected as the software in use, and in step S106, the first application operation type list LB (see FIG. 13) is selected as the software-specific operation type list.

[0117] In step S107, the selected first application operation type list LB is output to the headset 2. In step S108, the first application operation type list LB is acquired in the headset 2. In step S109, the pointer input mode is detected as the input mode, and in step S110, information specifying the pointer input mode is output to the headset 2. In step S111, information specifying the pointer input mode is acquired by the headset 2, and in step S112, the pointer input mode operation type list Lp (see FIG. 13) in the first application operation type list LB is applied to the first dictionary 211.

[0118] Thereafter, steps S201 to S214 are repeatedly executed, and the recall operation is performed. When steps S201 to S214 are repeatedly executed, the user operates pointer 90 on touch panel display screen 3d, for example, to select a more interesting back window and bring that window to the front or to enlarge the front window. To bring a back window to the front or to enlarge the front window, for example, pointer 90 is moved onto the window by the recall operation, and a tap is performed in that state.

[0119] Fig. 18 is a diagram showing a display example in which a front window is tapped and expanded. For example, as shown in Fig. 18, window 53 is expanded and the map is displayed in an enlarged state. At this time, the input mode is switched from the pointer input mode to the map operation input mode.

[0120] In this case, in the flow shown in Fig. 10B, in step S211, it is determined that the input mode has changed, and the process returns to step S109. In step S109, the map operation input mode is detected as the input mode, and in step S110, information specifying the map operation input mode is output to the headset 2. In step S111, information specifying the map operation input mode is acquired by the headset 2, and in step S112, the map operation input mode operation type list Lm (see Fig. 13) in the first application operation type list LB is applied to the first dictionary 211.

[0121] Thereafter, steps S201 to S214 are repeatedly executed, and the recalling operation continues. When steps S201 to S214 are repeatedly executed, the user operates the pointer 90 on the touch panel display screen 3d, for example, to move the display area of ​​the map or to enlarge the map. To move the display area of ​​the map, for example, the user drags the pointer 90 up, down, left, or right through the recalling operation. To enlarge the map, the user moves the pointer 90 to a position where the user wants to enlarge the display, and pinches out or double taps at that position through the recalling operation.

[0122] According to the second embodiment, even if the input mode being used is changed midway, the operation types to be recognized by the recall operation are appropriately limited, so that erroneous recognition of the operation types is reliably suppressed and the recognition accuracy is improved.

[0123] Example 3 In the third embodiment, it is assumed that the software being used is the second application (fighting game application) 313, and the input mode is the character operation input mode.

[0124] Fig. 19 is a diagram showing a display example of a smartphone when the software used is the second application and the input mode is the character operation input mode. In this embodiment, for example, as shown in Fig. 19, an image 55 including a character 54a corresponding to the user's avatar and a character 54b that is the opponent of the user is displayed on the touch panel display screen 3d. Note that a pointer may or may not be displayed.

[0125] In this embodiment, in the flow shown in FIG. 10A, in step S105, the second application 313 is detected as the software in use, and in step S106, the second application operation type list LC is selected as the software-specific operation type list.

[0126] Fig. 20 is a conceptual diagram of the second application operation type list LC. As shown in Fig. 20, the second application operation type list LC is composed of a pointer input mode operation type list Lp, a first text input mode operation type list Lt1, and a character operation input mode operation type list Lf.

[0127] In step S107, the selected second application operation type list LC is output to the headset 2. In step S108, the second application operation type list LC is acquired in the headset 2. In step S109, the character operation input mode is detected as the input mode, and in step S110, information specifying the character operation input mode is output to the headset 2. In step S111, information specifying the character operation input mode is acquired by the headset 2, and in step S112, the character operation input mode operation type list Lf (see FIG. 20) in the second application operation type list LC is applied to the first dictionary 211.

[0128] As shown in Fig. 20, the operation type list Lf for the character operation input mode includes moving the pointer left and right, flicking up and down and left, tapping, and double tapping. Moving the pointer right corresponds to the character's forward movement, and moving the pointer left corresponds to the character's backward movement. Additionally, a flick up corresponds to the character's jumping movement, a flick down corresponds to the character's prone movement, and a flick left corresponds to the character's guarding movement. Additionally, a tap corresponds to the character's punching movement, and a double tap corresponds to the character's kicking movement.

[0129] By such processing, the operation types to be recognized are set to be limited to the operation types included in the operation type list Lf for the character operation input mode. As a result, the operation types to be recognized from the electroencephalogram are narrowed down to only the operation types considered necessary in the character operation input mode. In other words, operation types considered unnecessary in the character operation input mode, specifically, operations such as moving the pointer up and down, swiping up and down and left and right, dragging up and down, pinching in and out, are excluded from the operation types to be recognized.

[0130] According to the third embodiment, the direction, movement, etc. of the operation in the game app are often similar to the target action in the game, and it is easy to understand that the recall operation can be used intuitively. Also, the number of operation types tends to be relatively small compared to other apps. Therefore, the information processing system according to the present embodiment is extremely useful for such game apps, especially for fighting, sports, rhythm, or card game apps.

[0131] If the recalling operation is not recognized for a certain period of time or more, the second control device 308 may perform control so as to return the position of the pointer 90 to the center. By performing such control, the position of the pointer is reset, making it easier to start the next recalling operation.

[0132] According to the above-described embodiment 1, depending on the software used in the smartphone 3, a software-specific operation type list corresponding to the software is applied to the first dictionary 211. Also, depending on the input mode used in the smartphone 3, an input mode-specific operation type list corresponding to the input mode is applied to the first dictionary 211. Therefore, the brain wave type discrimination candidates are narrowed down to brain wave types associated with the operation types used in the software or input mode used in the smartphone 3. That is, the operation types considered unnecessary for the software or input mode used are excluded from the operation types to be recognized, and the brain wave types corresponding to these operation types are also excluded from the brain wave types to be discriminated. That is, the number of brain wave types to be discriminated and the number of operation types to be recognized are reduced, making it easier to discriminate the brain wave types, suppressing erroneous recognition of the operation types, and improving the recognition accuracy. As a result, it is possible to realize a highly feasible and practical recollected operation in the operation of the information processing terminal. That is, it is possible to recognize the operation recollected by the user and apply it to the operation of the information processing terminal.

[0133] (Embodiment 2) The information processing system according to the second embodiment is configured to identify an operation type list according to only the input mode of the smartphone, and to cause the headset to apply the operation type list to the first dictionary.

[0134] <Functional configuration> The functional configuration of the information processing system according to the second embodiment will be described.

[0135] FIG. 21 is a diagram showing a configuration of functional blocks of a headset and a smartphone in an information processing system according to the second embodiment.

[0136] The information processing system 1a is based on the information processing system 1 according to the first embodiment, but the second storage device 310a of the smartphone 3a stores a third dictionary (third table) 317 instead of the second dictionary 315.

[0137] The third dictionary 317 is configured with a table in which the input modes used in the smartphone 3a are associated with operation type lists corresponding to the input modes.

[0138] Fig. 22 is a diagram showing an example of the third dictionary. As shown in Fig. 22, the third dictionary 317 is configured by a table in which the types of input modes used in the smartphone 3a and input mode-specific lists that limit and list the operation types used when the input modes are used are associated with each type of input mode.

[0139] As shown in FIG. 22, in this embodiment, as operation type lists for different input modes, an operation type list Lp for the pointer input mode, an operation type list Lt1 for the first text input mode, an operation type list Lm for the map operation input mode, and an operation type list Lf for the character operation input mode are prepared.

[0140] Example 1 The first embodiment is an example in which the headset acquires a list of operation types by input mode every time the input mode of the smartphone changes.

[0141] That is, in the first embodiment, the smartphone 3a detects the input mode being used, and outputs the input mode-specific operation type list corresponding to the detected input mode to the headset 2. Each time the input mode changes, the input mode is detected, and the input mode-specific operation type list corresponding to the input mode is applied to the first dictionary 211.

[0142] 23A to 23D are process flow diagrams according to Example 1 of the information processing system according to the embodiment 2. The parts different from the flow of the embodiment 1 are included in the process flows of Fig. 23A and Fig. 23B. Note that only the process flows related to the parts different from the embodiment 1 will be described here, and the description of the other parts will be omitted.

[0143] In the process flow of FIG. 23A, when cooperation between the smartphone 3a and the headset 2 is established (S103), the smartphone 3a starts up the headset control tool 316 (S104) and then detects the input mode (S109). When the input mode is detected, an input mode-specific operation type list corresponding to the detected input mode is selected (S110). Then, the selected input mode-specific operation type list is output to the headset 2 (S121). The headset 2 acquires the input mode-specific operation type list (S122) and applies it to the first dictionary 211 (S113).

[0144] In the process flow of Fig. 23B, a determination is made as to whether the input mode has changed (S210), and if it is determined that the input mode has not changed (S210, No), a determination is not made as to whether the software in use has changed, and a determination is immediately made as to whether the learning mode has been selected (S212). On the other hand, if it is determined that the input mode has changed (S210, Yes), the process returns to step S109, and the input mode is detected again.

[0145] Example 2 The second embodiment is an example in which all input mode-specific operation type lists are acquired in advance by the headset.

[0146] That is, in the second embodiment, the smartphone first outputs all prepared operation type lists for each input mode to the headset, and the headset acquires all of the operation type lists for each input mode. Then, the smartphone detects the input mode and outputs input mode identification information to the headset. The headset acquires the input mode identification information, identifies the input mode based on the acquired input mode identification information, and applies the operation type list for each input mode corresponding to the identified input mode to the first dictionary 211. Each time the input mode changes, the input mode is detected and the operation type list for each input mode corresponding to the input mode is applied to the first dictionary.

[0147] 24A to 24D are process flow diagrams according to Example 2 of the information processing system according to the embodiment 2. The parts different from the flow of the embodiment 1 are included in the process flows of Fig. 24A and Fig. 24B. Note that only the process flows related to the parts different from the embodiment 1 will be described here, and the description of the other parts will be omitted.

[0148] In the processing flow of FIG. 24A, when cooperation between the smartphone 3a and the headset 2 is established (S103), the smartphone 3a starts the headset control tool 316 (S104), and then outputs all prepared lists of operation types by input mode collectively (S131). The headset 2 acquires all prepared lists of operation types by input mode. The smartphone 3a detects the input mode (S109), and outputs input mode identification information to the headset 2 (S133). The headset 2 acquires the input mode identification information (S134), and applies the list of operation types by input mode corresponding to the input mode identified by the input mode identification information to the first dictionary 211 (S113).

[0149] In the process flow of Fig. 24B, a determination is made as to whether the input mode has changed (S210), and if it is determined that the input mode has not changed (S210, No), a determination is not made as to whether the software in use has changed, and a determination is immediately made as to whether the learning mode has been selected (S212). On the other hand, if it is determined that the input mode has changed (S210, Yes), the process returns to step S109, and the input mode is detected again.

[0150] According to the above-described second embodiment, depending on the input mode used in the smartphone 3a, the input mode-specific operation type list corresponding to the input mode is applied to the first dictionary 211. Therefore, the discrimination candidates of the electroencephalogram type are narrowed down to the electroencephalogram type associated with the operation type used in the input mode used in the smartphone 3a. That is, the operation types considered unnecessary in the input mode used are excluded from the operation types to be recognized, and the electroencephalogram types corresponding to these operation types are also excluded from the electroencephalogram types to be discriminated. That is, the number of electroencephalogram types to be discriminated and the operation types to be recognized is reduced, making it easier to discriminate the electroencephalogram types, suppressing erroneous recognition of the operation types, and improving the recognition accuracy. As a result, as in the first embodiment, it is possible to realize a highly feasible and practical recollected operation in the operation of the information processing terminal. That is, it is possible to recognize the operation recollected by the user and apply it to the operation of the information processing terminal.

[0151] Furthermore, according to the second embodiment, it is not necessary to detect the software used in the smartphone 3a or to prepare a list of operation types for each piece of software, so that the system can be constructed more easily.

[0152] (Embodiment 3) The information processing system according to the third embodiment is configured so that the discrimination of the type of brain wave, the recognition of the type of operation, etc. are performed on the smartphone side, not on the headset side.

[0153] <Functional configuration> The functional configuration of the information processing system according to the third embodiment will be described.

[0154] FIG. 25 is a diagram showing a configuration of functional blocks of a headset and a smartphone in an information processing system according to the third embodiment. As shown in FIG.

[0155] As shown in Fig. 25, the information processing system 1b according to the third embodiment includes a headset 2b and a smartphone 3b. The headset 2b includes a communication device 200, a detection device 201, a first control device 206, and a first storage device 210. The smartphone 3b includes a communication device 300, an operation type list selection device 301, a used software detection device 302, an input mode detection device 303, a recall operation reception device 304, a touch operation reception device 305, a voice operation reception device 306, an operation reception control device 307, a second control device 308, an operation type list application device 202, an electroencephalogram type discrimination device 203, a recall operation recognition device 204, a learning device 205, and a second storage device 310b.

[0156] The second storage device 310b includes an operating system 311, a first app 312, a second app 313, a second dictionary 315, a headset control tool 316, and the first dictionary 211.

[0157] In the third embodiment, the smartphone 3b detects the software in use and the input mode, and applies the input mode-specific operation type list corresponding to the detected software and input mode to the first dictionary 211. The headset 2b detects the user's brainwaves and outputs them to the smartphone 3b. Each time the software in use is changed, the software in use is detected and a software-specific operation type list corresponding to the software in use is selected. Also, each time the input mode is changed, the input mode is detected and the input mode-specific operation type list corresponding to the input mode is applied to the first dictionary 211.

[0158] 26A to 26D are diagrams showing the processing flows of an information processing system according to embodiment 3. The parts different from the processing flows of embodiment 1 are included in the processing flows of Figures 26A to 26C. Note that only the processing flows related to the parts different from embodiment 1 will be described here, and descriptions of the other parts will be omitted.

[0159] <<Preparation process flow for recall operation>> As shown in FIG. 26A, first, the second control device 308 outputs a link request signal to the headset 2b (S101). The first control device 206 outputs a link permission signal to the smartphone 3b in response to the link request signal (S102). The second control device 308 acquires the link permission signal (S103). This establishes a link between the smartphone 3b and the headset 2b.

[0160] When the link is established, the second control device 308 starts up the headset control tool 316 stored in the second storage device 310b (S104), making it possible to control data transmission / reception or operation of the headset 2b.

[0161] Next, the used software detection device 302 detects the software used in the smartphone 3b (S105). The operation type list selection device 301 refers to the second dictionary 315 shown in Fig. 6 and selects a software-specific operation type list associated with the identified software (S106).

[0162] In this embodiment, it is assumed that the software in use is the operating system 311. In this case, the operating system 311 is detected as the software in use, and the operating system operation type list LA is selected as the software-specific operation type list.

[0163] When the software-specific operation type list is selected, the second control device 308 detects the input mode currently being used in the smartphone 3b (S109), and selects the input mode-specific operation type list corresponding to the detected input mode (S110).

[0164] In this embodiment, it is assumed that the pointer input mode is selected as the input mode, and therefore the pointer input mode operation type list Lp is selected here.

[0165] The recall operation recognition device 204 applies the selected input mode-specific operation type list to the first dictionary 211. That is, the recall operation recognition device 204 performs a setting to limit the operation types to be recognized to the operation types included in the applied input mode-specific operation type list among the operation types registered in the first dictionary 211 (S113).

[0166] Here, the pointer input mode operation type list Lp is applied to the first dictionary 211. As a result, the operation types to be recognized are narrowed down to only those operation types considered necessary in the pointer input mode.

[0167] <<Execution process flow of recall operation>> As shown in FIG. 26B, the detection device 201 detects the user's brain wave and outputs the detected brain wave to the smartphone 3b (S231). The brain wave type discrimination device 203 of the smartphone 3b acquires the brain wave and discriminates the brain wave type based on the acquired brain wave waveform, etc. (S202). At this time, among the brain wave types registered in the first dictionary 211, the discrimination is limited to any of the brain wave types corresponding to the operation types included in the input mode-specific operation type list currently applied, and other brain wave types. In this embodiment, the discrimination is limited to the brain wave types corresponding to the operation types included in the pointer input mode operation type list Lp as shown in FIG. 8.

[0168] The recall operation recognition device 204 refers to the first dictionary 211 to identify an operation type corresponding to the identified electroencephalogram type, and recognizes the identified operation type as the operation type recalled by the user (S232). When the operation type is recognized, the second control device 308 acquires information representing the recognized operation type (S204). The operation reception control device 307 detects the presence or absence of an operation by each operation input method (S205). Specifically, it detects the presence or absence of an operation by each of the touch input method, the voice input method, and the recall input method. Then, the operation reception control device 307 identifies an operation by the operation input method with the highest priority among the detected operations (S206), and adopts and executes the identified operation (S207).

[0169] Here, the learning device 205 receives a learning evaluation for the recollecting operation (S208). The learning evaluation includes, for example, information on whether or not the user's pressing of the "Like!" button is detected. The learning device 205 executes a learning process based on the information so as to improve the recognition accuracy of the recollecting operation (S209).

[0170] When the learning process is executed, the second control device 308 judges whether the input mode has changed (S210). If it is judged that the input mode has changed (S210: Yes), the process returns to step S109. If it is judged that the input mode has not changed (S210: No), the process proceeds to the next judgment. The second control device 308 judges whether the software in use has changed (S211). If it is judged that the software in use has changed (S211: Yes), the process returns to step S105. If it is judged that the software in use has not changed (S211: No), the process proceeds to the next judgment. The second control device 308 judges whether the learning mode has been selected by the user (S212). If it is judged that the learning mode has been selected (S212: Yes), the process proceeds to step S331. If it is judged that the learning mode has not been selected (S212: No), the process proceeds to the next judgment. The second control device 308 judges whether the link should be terminated (S213). Examples of situations in which the link should be terminated include when an instruction to terminate the link is input, when a failure occurs in part of the link, when the battery level is low, etc. If it is determined that the link should be terminated (S213: Yes), the process proceeds to step S401. If it is determined that the link should not be terminated (S213: No), the process returns to step S231.

[0171] In this way, unless there is a change in the input mode being used, a change in the software being used, selection of the learning mode, or termination of the link, the cycle of recognizing the recall operation by detecting the brainwave and executing the recognized recall operation continues. The user performs the desired operation by manipulating the pointer 90 displayed on the screen of the smartphone 3b with the recall operation.

[0172] Learning mode processing flow Next, the process in the learning mode will be described.

[0173] As shown in Fig. 26C, when the learning mode is selected, learning device 205 starts processing in the learning mode. Learning device 205 selects one of multiple types of pointers prepared in advance (S304), and displays the selected pointer on the screen. Learning device 205 also outputs an operation instruction for the user (S305). The operation instruction may be in the form of displaying an image or text, or in the form of outputting sound.

[0174] When the pointer is displayed and an operation instruction is output, the learning device 205 controls the detection device 201 of the headset 2b to detect the user's brain waves and output them to the smartphone 3b (S306). The brain wave type discrimination device 203 discriminates the brain wave type based on the detected brain waves (S307), and the recall operation recognition device 204 refers to the first dictionary 211 and the applied operation type list based on the discriminated brain wave type to recognize the operation type (S333). In addition, the learning device 205 stores the brain waves detected during the learning mode in the second storage device 310b in association with their timing. The second control device 308 executes an operation corresponding to the recognized operation type (S309).

[0175] If the executed operation is exactly as the user had imagined, the user presses the "Like!" button and inputs an evaluation that indicates that the recalled operation was successful. If an evaluation that the recalled operation was good is input, the second control device 308 accepts the evaluation (S310) and stores the type of the operation instruction at that time, the time timing, and evaluation result information that associates the evaluation in the second storage device 310b.

[0176] The second control device 308 judges whether or not to change the operation instruction (S311). If it is judged that the operation instruction is to be changed (S311, Yes), the second control device 308 returns to step S305 and changes and outputs the operation instruction. If it is judged that the operation instruction is not to be changed (S311, No), the second control device 308 judges whether or not to change the pointer (S312).

[0177] If it is determined that the pointer is to be changed (S312, Yes), the second control device 308 returns to step S304 and selects another pointer as the pointer to be displayed.

[0178] When it is determined that the pointer is not to be changed (S312, No), the second control device 308 ends the learning mode and outputs the evaluation result information stored in the second storage device 310b to the headset 2 (S313). The learning device 205 adjusts the first dictionary 211 based on the acquired evaluation result information (S314). For example, based on the operation type when the recollection operation is evaluated as good and the waveform of the brain wave at that time, the brain wave pattern of the brain wave type registered in the first dictionary 211 is corrected and adjusted to one that better reflects the characteristics of the user's brain wave. This adjustment of the first dictionary 211 improves the discrimination accuracy of the brain wave type in the recollection operation, and improves the recognition accuracy of the operation type. In addition, the learning device 205 analyzes which pointer improves the discrimination accuracy of the brain wave, and determines the order of the pointers in order of improving accuracy (S315). The information on the ranking of the pointers is output to the smartphone 3b. The smartphone 3b refers to the ranking of the pointers and allows any pointer to be selected as the pointer to be displayed. Alternatively, the pointer of the highest rank type is automatically selected.

[0179] When the learning mode is completed, the second control device 308 returns the execution step to S214 and continues the process.

[0180] According to the above-described embodiment 3, it is possible to obtain the same effects as those of embodiment 1. Furthermore, since the processes other than the detection of brain waves are performed on the smartphone side, it is possible to reduce the processes to be executed by the headset, and thus it is possible to simplify the configuration of the headset.

[0181] (Embodiment 4) The information processing system of the fourth embodiment is configured such that the discrimination of the electroencephalogram type, the recognition of the operation type, etc. are performed on the smartphone side, not on the headset side, and the operation type list is identified according to the input mode of the smartphone and applied to the first dictionary.

[0182] <Functional configuration> The functional configuration of the information processing system according to the fourth embodiment will be described.

[0183] FIG. 27 is a diagram showing a configuration of functional blocks of a headset and a smartphone in an information processing system according to the fourth embodiment. As shown in FIG.

[0184] As shown in FIG. 27, the information processing system 1c of embodiment 4 is based on the information processing system 1b of embodiment 3, but the second storage device 310c of the smartphone 3c stores a third dictionary 317 instead of the second dictionary 315.

[0185] The third dictionary 317 is configured with a table in which an input mode selected on the smartphone 3c is associated with a list of operation types corresponding to the input mode.

[0186] 28A to 28D are diagrams showing the processing flow of the information processing system according to embodiment 4. The parts different from the processing flow of embodiment 3 are included in the processing flows of Figures 26A and 26B. Note that only the processing flow related to the parts different from embodiment 3 will be described here, and the description of the other parts will be omitted.

[0187] In the process flow of Fig. 28A, when cooperation between the smartphone 3c and the headset 2c is established (S103), the smartphone 3c starts up a headset control tool (S104), and then detects the input mode (S109) without detecting the software in use. When the input mode is detected, a list of operation types by input mode corresponding to the detected input mode is selected (S110) and applied to the first dictionary 211 (S113).

[0188] In the process flow of Fig. 28B, a determination is made as to whether the input mode has changed (S210), and if it is determined that the input mode has not changed (S210, No), a determination is not made as to whether the software in use has changed, and a determination is immediately made as to whether the learning mode has been selected (S212). On the other hand, if it is determined that the input mode has changed (S210, Yes), the process returns to step S109, and the input mode is detected again.

[0189] According to the above-described embodiment 4, it is possible to obtain the same effects as those of embodiment 2. Furthermore, since the processes other than the detection of brain waves are performed on the smartphone side, it is possible to reduce the processes to be executed by the headset, and thus it is possible to simplify the configuration of the headset.

[0190] (Embodiment 5) In each of the above embodiments, the headset is an example of the electrode device in the present invention, and the shape, configuration, usage form, etc. are arbitrary. Therefore, for example, a hat-type device, a non-wearable electrode attachment type device, etc. may be used as the electrode device in the present invention.

[0191] In addition, in each of the above embodiments, the smartphone is an example of an information processing terminal in the present invention, and the shape, size, specifications, etc. are arbitrary. Therefore, for example, a tablet terminal, a laptop computer with a touch panel, etc. may be used as the information processing terminal in the present invention instead of a smartphone.

[0192] In this way, according to each of the above-mentioned embodiments, the operation types to be recognized based on the user's electroencephalogram are limited to those necessary at that time, so that it is possible to easily distinguish the electroencephalogram type and to suppress erroneous recognition of the operation type. As a result, it is possible to realize highly feasible and practical recall operations in the operation of the information processing terminal.

[0193] Furthermore, since recollection operations are possible without requiring high accuracy in detecting brain waves or distinguishing between types of brain waves, it is possible to enjoy recollection operations on an information processing terminal even using a relatively inexpensive brain wave detection device.

[0194] In addition, a pointer is displayed on the touch panel display screen, and operations using the pointer as a reference or starting point are regarded as the type of operation to be recognized, so that recall operations can be used more intuitively as a substitute for touch operations.

[0195] Although various embodiments of the present invention have been described above, the present invention is not limited to the above-mentioned embodiments, and various modified examples are included. Furthermore, the above-mentioned embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these belong to the scope of the present invention. Furthermore, the numerical values, messages, etc. contained in the text and figures are merely examples, and the effect of the present invention is not impaired even if different ones are used.

[0196] In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration. In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in hardware by designing a part or all of them as an integrated circuit, for example. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor such as an MPU or a CPU interpreting and executing a program that realizes each function. In addition, the range of functions realized by software is not limited, and hardware and software may be used together. Information such as programs, tables, files, etc. that realize each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

[0197] Possible aspects of the present invention will be described below.

[0198] [Appendix 1] A method for recollecting a recall operation, comprising: A detection step of detecting an electroencephalogram of the user for use in recognizing an operation recalled by the user; a discrimination step of referring to a first table in which types of operations accepted by an information processing terminal and types of electroencephalograms corresponding to the types of operations are associated with each other, and discriminating a type of electroencephalogram corresponding to the electroencephalogram detected by the detection step from among the types of electroencephalograms included in the first table; a recognition step of recognizing an operation imagined by the user by referring to the first table based on the type of electroencephalogram determined by the determination step; an execution step of executing the operation recognized by the recognition step; a limiting step of limiting the types of operations to be recognized in the recognition step from among the types of operations included in the first table in accordance with the type of software or input mode currently being used in the information processing terminal; A method for recognizing a recall operation, wherein the discrimination step discriminates a type of brain wave corresponding to the detected brain wave from among types of brain waves corresponding to the types of operations limited by the limiting step.

[0199] [Appendix 2] In the method for recognizing a recall operation according to claim 1, A method for recognizing a recall operation, comprising a learning process of accepting an evaluation by the user of the recognized operation, and adjusting the waveform of the type of electroencephalogram corresponding to the type of operation included in the first table based on the operation to be evaluated and the electroencephalograms used to recognize the operation.

[0200] [Appendix 3] In the method for recognizing a recall operation according to claim 1, In the execution step, an operation is executed with a pointer displayed on a screen of the information processing terminal as a reference or starting point.

[0201] [Appendix 4] In the method for recollecting a recall operation according to claim 3, A recall operation recognition method comprising a learning process of outputting operation instructions to the user for each of multiple types of pointers displayed sequentially on the screen, accepting an evaluation by the user of the recognized operations, and ranking the multiple types of pointers based on the evaluation.

[0202] [Appendix 5] In the method for recognizing a recall operation according to claim 1, The software includes game software.

[0203] [Appendix 6] In the method for recognizing a recall operation according to claim 1, The input mode includes a text input mode.

[0204] [Appendix 7] In the method for recognizing a recall operation according to claim 1, The information processing terminal is a smartphone, a tablet terminal, or a notebook computer with a touch panel. [Explanation of symbols]

[0205] 1...information processing system, 2...headset (brain wave detection device), 3...smartphone (information processing terminal), 9...user, 9w...brain wave, 24...electrode device for detecting brain waves, 200...communication device (first communication device), 300...communication device (second communication device), 201...detection device (detection device), 202 operation type list application device (first processing device), 203...brain wave type discrimination device (first processing device), 204...recall operation recognition device, 205...learning device (learning device), 206...first control device (first processing device), 210...first storage device (first storage device), 211...first dictionary (conversion dictionary, first table), 301...operation type list selection device (second processing device), 302...usage software detection device (second processing device), 303...input mode detection device (second processing device), 304...recall operation reception device (second processing device), 305...touch operation reception device (second processing device), 306...voice operation reception device (second processing device), 307...operation reception control device (second processing device), 308...second control device (second processing device), 310...second storage device (second storage device), 311...operating system (software), 312...first application (software), 313...second application (software), 315...second dictionary (second table), 316...headset control tool.

Claims

1. The present invention includes an electroencephalogram (EEG) detection device and an information processing terminal. The electroencephalogram detection device includes: A detection device for detecting an electroencephalogram of a user; a first database for converting the detected electroencephalogram characteristics into data that can be processed by the information processing terminal; a first processing device that converts the electroencephalogram detected by the detection device into data based on the first database; a first communication device for transmitting and receiving data to and from the information processing terminal; The information processing terminal includes: a second communication device for transmitting and receiving data to and from the brain wave detection device; A second processing device that executes a corresponding process corresponding to the data based on the data received by the second communication device; a second database for storing a data group related to an instruction input used in the information processing terminal; The first processing device stores the data group received via the first communication device in the first database, and performs a conversion process using the data group; the first processing device limits data to be converted based on the received data group; Information processing system.

2. The present invention includes an electroencephalogram (EEG) detection device and an information processing terminal. The electroencephalogram detection device comprises: A detection device for detecting an electroencephalogram of a user; a first database for converting the detected electroencephalogram characteristics into data that can be processed by the information processing terminal; a first processing device that converts the electroencephalogram detected by the detection device into data based on the first database; a first communication device for transmitting and receiving data to and from the information processing terminal; The information processing terminal includes: a second communication device for transmitting and receiving data to and from the brain wave detection device; A second processing device that executes a corresponding process corresponding to the data based on the data received by the second communication device; a third database storing input mode specification information for specifying an input mode used by the information processing terminal and a data group related to an instruction input used in the input mode, the data group corresponding to each type of the input mode; The first processing device stores the data group received via the first communication device in the first database, and performs a conversion process using the data group; the first processing device limits data to be converted based on the received data group; Information processing system.

3. 2. The information processing system according to claim 1, The information processing terminal includes: a third database storing a third table in which specific information for specifying an input mode used by the information processing terminal and a data group related to an instruction input used in the input mode are associated with each type of the input mode, the second processing device refers to the third table, identifies the data group corresponding to the input mode being used in the information processing terminal, and transmits the identified data group to the brain wave detection device; the first processing device limits data to be converted based on the received data group; Information processing system.

4. 3. The information processing system according to claim 1, a learning device that receives an evaluation by the user on the converted data, and adjusts waveform characteristics of the electroencephalogram corresponding to the data included in the first database based on the data to be evaluated and the electroencephalogram used for converting the data; Information processing system.

5. 3. The information processing system according to claim 1, the second processing device executes, as the corresponding process, an operation based on or as a starting point of a pointer displayed on a screen of the information processing terminal; Information processing system.

6. 6. The information processing system according to claim 5, a learning device that outputs operation instructions to the user for each of a plurality of types of pointers sequentially displayed on the screen, receives an evaluation by the user of the converted data, and ranks the plurality of types of pointers based on the evaluation; Information processing system.

7. 4. The information processing system according to claim 2, The input mode includes a text input mode. Information processing system.

8. 3. The information processing system according to claim 1, The information processing terminal is a smartphone, a tablet terminal, or a laptop computer with a touch panel. Information processing system.

9. An information processing terminal capable of operating in cooperation with an electroencephalogram detection device that outputs data based on a detected electroencephalogram of a user, a second communication device for transmitting and receiving data to and from the brain wave detection device; A second processing device that executes a corresponding process corresponding to the data based on the data received by the second communication device; a second database for storing a data group related to an instruction input used in the information processing terminal; the second processing device controls to identify the data group by referring to the second database; The second communication device transmits the identified data group to the brain wave detection device. Information processing terminal.

10. An information processing terminal capable of operating in cooperation with an electroencephalogram detection device that outputs data based on a detected electroencephalogram of a user, a second communication device for transmitting and receiving data to and from the brain wave detection device; A second processing device that executes a corresponding process corresponding to the data based on the data received by the second communication device; a third database storing input mode specification information for specifying an input mode used by the information processing terminal and a data group related to an instruction input used in the input mode, the data group corresponding to each type of the input mode; the second processing device controls to refer to the third database and identify the data group corresponding to the input mode being used in the information processing terminal; The second communication device transmits the identified data group to the brain wave detection device. Information processing terminal.

11. 10. The information processing terminal according to claim 9, a third database storing a third table in which specific information for specifying an input mode used by the information processing terminal and a data group related to an instruction input used in the input mode are associated with each type of the input mode, the second processing device refers to the third table, identifies the data group corresponding to the input mode being used in the information processing terminal, and transmits the identified data group to the brain wave detection device; Information processing terminal.

12. 11. The information processing terminal according to claim 9, the second processing device executes, as the corresponding process, an operation based on or as a starting point of a pointer displayed on a screen of the information processing terminal; Information processing terminal.

Images