Information processing system

Abstract

To provide an information processing system, an information processing terminal, and a recall operation recognition method which recognize operation recalled by a user and apply the operation to operation of an information processing terminal.SOLUTION: An information processing system includes a brain wave detection device and an information processing terminal. The brain wave detection device detects a brain wave of a user, and converts the detected brain wave into data and transmits the data to the information processing terminal, on the basis of a conversion dictionary for converting the feature of the brain wave into data capable of being processed by the information processing terminal. The information processing terminal executes corresponding processing corresponding to the data, on the basis of the received data. The information processing terminal transmits a data group relating to an instruction input used in software during operation of the information processing terminal, to the brain wave detection device, during or after execution of the corresponding processing. The brain wave detection device stores the received data group in a storage device, and restricts the data to be converted on the basis of the data group.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing terminal, and a recall operation recognition method.

Background Art

[0002] A technique for changing the state of an application based on detected neurological intention data of a user is disclosed in Patent Document 1.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] On the other hand, in recent years, the opportunity for users to operate information processing terminals such as smartphones, tablet terminals, and notebook computers with touch panels has increased significantly.

[0005] However, in Patent Document 1, no consideration is given to the case where the operation recalled by the user is recognized and applied to the operation of the information processing terminal. That is, there is no description of a specific method or a feasible method regarding the method of recognizing the operation recalled by the user and applying it to the operation of the information processing terminal.

[0006] In particular, for users who have difficulty operating with their own fingers, the need for a method of recognizing the operation recalled by the user and applying it to the operation of the information processing terminal is very high.

[0007] Under such circumstances, it is desired to provide a technology that can recognize the operation recalled by the user and apply it to the operation of the information processing terminal.

Means for Solving the Problems

[0008] A brief overview of some of the representative inventions disclosed in this application is as follows:

[0009] An information processing system according to a typical embodiment of the present invention comprises an electroencephalogram (EEG) detection device and an information processing terminal, wherein the EEG detection device includes a detection device for detecting the user's brainwaves, a first storage device for storing a conversion dictionary for converting the characteristics of the detected brainwaves into data that can be processed by the information processing terminal, a first processing device for converting the brainwaves detected by the detection device into data based on the conversion dictionary, and a first communication device for sending and receiving data with the information processing terminal, wherein the information processing terminal includes a second communication device for sending and receiving data with the EEG detection device and a second processing device for executing corresponding processing based on the data received by the second communication device, the second processing device transmits a group of data related to instruction inputs used in software running on the information processing terminal to the EEG detection device via the second communication device during or after the execution of a cooperative processing that enables cooperative operation between the information processing terminal and the EEG detection device, and the first processing device stores the group of data received via the first communication device in the first storage device and executes conversion processing using the group of data. [Effects of the Invention]

[0010] The effects obtained by some of the representative inventions disclosed in this application can be briefly explained as follows.

[0011] According to a typical embodiment of the present invention, it is possible to recognize an operation recalled by the user and apply it to the operation of the information processing terminal. [Brief explanation of the drawing]

[0012] [Figure 1] This is an external view of the information processing system according to Embodiment 1. [Figure 2]It is a diagram showing an example of the hardware configuration of a headset and a smartphone in an information processing system. [Figure 3] It is a diagram showing the configuration by functional blocks of a headset and a smartphone in an information processing system. [Figure 4] It is a diagram showing an example of the priority order of operation input methods. [Figure 5] It is a diagram showing an example of the first dictionary. [Figure 6] It is a diagram showing an example of the second dictionary. [Figure 7] It is a diagram showing an example of an operation type list for each input mode. [Figure 8] It is a diagram showing an example of an operation type list for each input mode. [Figure 9] It is a diagram showing an example of the display of a smartphone in a state where only the operating system is being used. [Figure 10A] It is a diagram showing the processing flow of the information processing system according to Embodiment 1. [Figure 10B] It is a diagram showing the processing flow of the information processing system according to Embodiment 1. [Figure 10C] It is a diagram showing the processing flow of the information processing system according to Embodiment 1. [Figure 10D] It is a diagram showing the processing flow of the information processing system according to Embodiment 1. [Figure 11] It is an image diagram of an operation type list for the operating system. [Figure 12] It is a diagram showing an example of the display of a smartphone in a state where the first application is being used. [Figure 13] It is an image diagram of an operation type list for the first application. [Figure 14] It is a diagram showing an example of the display of a smartphone in a state where the learning mode is selected. [Figure 15] It is a diagram showing an example of the display in a state where the pointer to be displayed is changed to another pointer and an operation instruction is output. [Figure 16]FIG. is a diagram showing a display example of a state in which an operation instruction is output by changing a pointer to be displayed to another pointer. [Figure 17] FIG. is a diagram showing a display example of a smartphone when the software in use is the second application and the input mode is the pointer input mode. [Figure 18] FIG. is a diagram showing a display example of a state in which the front window is tapped and expanded. [Figure 19] FIG. is a diagram showing a display example of a smartphone when the software in use is the second application and the input mode is the character operation input mode. [Figure 20] FIG. is an image diagram of an operation type list for the second application. [Figure 21] FIG. shows a configuration by functional blocks of a headset and a smartphone in an information processing system according to Embodiment 2. [Figure 22] FIG. shows an example of a third dictionary. [Figure 23A] FIG. is a processing flowchart according to Example 1 of the information processing system according to Embodiment 2. [Figure 23B] FIG. is a processing flowchart according to Example 1 of the information processing system according to Embodiment 2. [Figure 23C] FIG. is a processing flowchart according to Example 1 of the information processing system according to Embodiment 2. [Figure 23D] FIG. is a processing flowchart according to Example 1 of the information processing system according to Embodiment 2. [Figure 24A] FIG. is a processing flowchart according to Example 2 of the information processing system according to Embodiment 2. [Figure 24B] FIG. is a processing flowchart according to Example 2 of the information processing system according to Embodiment 2. [Figure 24C] FIG. is a processing flowchart according to Example 2 of the information processing system according to Embodiment 2. [Figure 24D] FIG. is a processing flowchart according to Example 2 of the information processing system according to Embodiment 2. [Figure 25]This figure shows the configuration of the headset and smartphone functional blocks in the information processing system according to Embodiment 3. [Figure 26A] This diagram shows the processing flow of the information processing system according to Embodiment 3. [Figure 26B] This diagram shows the processing flow of the information processing system according to Embodiment 3. [Figure 26C] This diagram shows the processing flow of the information processing system according to Embodiment 3. [Figure 26D] This diagram shows the processing flow of the information processing system according to Embodiment 3. [Figure 27] This figure shows the configuration of the headset and smartphone functional blocks in the information processing system according to Embodiment 4. [Figure 28A] This figure shows the processing flow of the information processing system according to Embodiment 4. [Figure 28B] This figure shows the processing flow of the information processing system according to Embodiment 4. [Figure 28C] This figure shows the processing flow of the information processing system according to Embodiment 4. [Figure 28D] This figure shows the processing flow of the information processing system according to Embodiment 4. [Modes for carrying out the invention]

[0013] Embodiments of the present invention will now be described. Note that the embodiments described below are merely examples for realizing the present invention and do not limit the technical scope of the present invention.

[0014] Furthermore, in the following embodiments, components having the same function are denoted by the same reference numerals, and repeated descriptions thereof are omitted unless particularly necessary.

[0015] (Embodiment 1) An information processing system according to Embodiment 1 of the present invention will be described.

[0016] <Overview of the Information Processing System> The information processing system according to Embodiment 1 is a system that recognizes and executes operations recalled by the user on an information processing terminal used by the user by detecting the user's brainwaves. Here, this operation is referred to as an operation using the recall input method. In this information processing system, a dictionary is prepared and stored in advance, which is a table that associates the types of operations accepted by the information processing terminal (hereinafter also referred to as operation types) with the types of brainwaves generated when the user recalls those operation types (hereinafter also referred to as brainwave types). In addition, an operation type list is also prepared and stored in advance, which lists only the operation types considered necessary under the conditions for each software and input mode used on the information processing terminal. When the software and input mode being used are detected, the operation type list corresponding to those conditions is selected and applied to the dictionary, narrowing down the operation types to be recognized to those necessary at that time. This function reduces the processing burden of determining brainwave types, improves the accuracy of operation recognition, and enhances the practicality of operations using the recall input method.

[0017] <Appearance and Hardware Configuration> The appearance and hardware configuration of the information processing system according to Embodiment 1 will be described below.

[0018] Figure 1 is an external view of the information processing system according to Embodiment 1. As shown in Figure 1, the information processing system 1 according to Embodiment 1 comprises a headset (electroencephalogram detection device) 2 and a smartphone (information processing terminal) 3. The headset 2 is worn on the head of the user 9. The smartphone 3 is an information processing terminal operated by the user 9.

[0019] The headset 2 and the smartphone 3 are configured to enable bidirectional communication via a wireless connection using radio waves 4a and 4b. In other words, the headset 2 and the smartphone 3 can operate in cooperation with each other. Alternatively, the headset 2 and the smartphone 3 may be configured to enable bidirectional communication via a wired connection using a connector cable 5.

[0020] Figure 2 shows an example of the hardware configuration of a headset and a smartphone in an information processing system.

[0021] As shown in Figure 2, the headset 2 comprises an operating device 21, an antenna 22, a connector 23, an electroencephalogram detection electrode device 24, an interface 25, a processor 26, memory 27, storage 28, and a battery 29. These are connected to each other via a bus 20, enabling the transmission and reception of electrical signals or data. The headset 2 may also receive power from a commercial power source via an AC adapter or the like instead of the battery 29.

[0022] The smartphone 3 comprises an operating device 31, a touch panel image display device 32, an antenna 33, a connector 34, a microphone 35, a speaker 36, a camera 37, an interface 38, a processor 39, memory 40, storage 41, and a battery 42. These are connected to each other via a bus 30, enabling the transmission and reception of electrical signals or data.

[0023] Connectors 23 and 34 are used to connect the headset 2 and the smartphone 3 via a wired connection. Antennas 22 and 33 are used to connect the headset 2 and the smartphone 3 wirelessly. The electroencephalogram detection electrode device 24 receives signals representing the user's brainwaves, which are used to recognize operations recalled by the user. Interface 25 converts electrical signals from the operating device 21, antenna 22, connector 23, electroencephalogram detection electrode device 24, etc., into data and transmits it to the processor 26, etc., or converts data from the processor 26, memory 27, or storage 28, etc., into electrical signals and transmits it to the connector 23 or antenna 22. Interface 38 converts electrical signals from the operating device 31, touch panel image display device 32, antenna 33, connector 34, microphone 35, camera 37, etc., into data and transmits it to the processor 39, etc., or converts data from the processor 39, memory 40, or storage 41, etc., into electrical signals and transmits it to the connector 34, speaker 36, or antenna 33.

[0024] Processors 26 and 39 are devices that perform arithmetic or data processing, and are composed of, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). Memory 27 and 40 are devices that temporarily store data, and are composed of, for example, semiconductor memory devices such as RAM (Random Access Memory). Storage 28 and 41 are devices that store data and various types of data, and are composed of, for example, non-volatile memory devices such as SSDs (Solid State Drives), HDDs (Hard Disk Drives), and flash memory.

[0025] <Functional configuration> The functional configuration of the information processing system 1 according to Embodiment 1 will be described below.

[0026] Figure 3 shows the configuration of the headset and smartphone in the information processing system, broken down by functional blocks. Each functional block of the headset 2 is realized by executing a predetermined program stored in the storage 28 using the processor 26, memory 27, etc., and coordinating with the hardware. Similarly, each functional block of the smartphone 3 is realized by executing a predetermined program stored in the storage 41 using the processor 39 and memory 40, etc., and coordinating with the hardware.

[0027] As shown in Figure 3, the headset 2 comprises, as a functional block, a communication device (first communication device) 200, a detection device (detection device) 201, an operation type list application device 202, an electroencephalogram type discrimination device 203, a recall operation recognition device 204, a learning device (learning device) 205, a first control device 206, and a first memory device (first memory device) 210. Note that the operation type list application device 202, the electroencephalogram type discrimination device 203, the recall operation recognition device 204, and the first control device 206 are examples of the first processing device in the present invention.

[0028] As shown in Figure 3, the smartphone 3 includes, as a functional block, a communication device (second communication device) 300, an operation type list selection device 301, a software usage detection device 302, an input mode detection device 303, a recall operation reception device 304, a touch operation reception device 305, a voice operation reception device 306, an operation reception control device 307, a second control device 308, and a second storage device (second storage device, third storage device) 310. Note that the operation type list selection device 301, the software usage detection device 302, the input mode detection device 303, the recall operation reception device 304, the touch operation reception device 305, the voice operation reception device 306, the operation reception control device 307, and the second control device 308 are examples of the second processing device in the present invention.

[0029] Headset Function Blocks This section describes each functional block of Headset 2.

[0030] The first storage device 210 stores the data necessary for the headset 2 to operate properly. In this embodiment, the first storage device 210 stores the first dictionary (conversion dictionary, first table) 211. The first dictionary 211 consists of a table that associates the operation type accepted by the smartphone 3 with the corresponding brainwave type for each operation type.

[0031] The communication device 200 is connected to the communication device 300 of the smartphone 3 wirelessly or via a wired connection, and performs bidirectional communication. The signals to be communicated are the data input and output in each functional block that makes up the headset 2. For example, Bluetooth® can be used as a wireless communication standard.

[0032] The detection device 201 detects the user's brainwaves 9w based on electrical signals received by the electroencephalogram detection electrode device 24 installed on the user's head.

[0033] The operation type list application device 202 limits the operation types that the recall operation recognition device 204 should recognize from among the operation types included in the first dictionary 211, according to the type of software or input mode being used on the smartphone 3. More specifically, the operation type list application device 202 receives information representing an operation type list (data group) from the smartphone 3 and applies that operation type list as the operation types to be recognized in a limited manner. In this embodiment, software-specific operation type lists are prepared in advance for each software being used. Furthermore, the software-specific operation type lists include one or more input mode-specific operation type lists created for each input mode being used. Based on information representing the input mode being used on the smartphone 3, the operation type list application device 202 applies the input mode-specific operation type list corresponding to that input mode as the operation types to be recognized in a limited manner.

[0034] The electroencephalogram (EEG) type discrimination device 203 identifies the EEG type corresponding to the EEG detected by the detection device 201 from among the EEG types included in the first dictionary 211. At this time, it identifies the EEG type corresponding to the detected EEG from among the EEG types corresponding to the operation type limited by the operation type list application device 202. More specifically, the EEG type discrimination device 203 determines whether the EEG detected by the detection device 201 corresponds to one of the multiple EEG types pre-registered in the first dictionary 211, or whether it does not correspond to any of the EEG types. At this time, the EEG types to be identified are limited to the EEG types corresponding to the operation type included in the applied operation type list.

[0035] The recall operation recognition device 204 recognizes the operation recalled by user 9 by referring to the first dictionary 211 based on the brainwave type determined by the brainwave type discrimination device 203. More specifically, the recall operation recognition device 204 recognizes the operation type corresponding to the determined brainwave type as the operation type input by the recall input method, and outputs the information of the recognized operation type to the smartphone 3.

[0036] The learning device 205 is equipped with a learning function to improve the recognition accuracy when recognizing operation types from brainwaves. The learning device 205 receives an evaluation from the user 9 of an operation recognized by the recall operation recognition device 204, and adjusts the waveform of the brainwave type corresponding to the operation type of that operation in the first dictionary 211 based on the operation being evaluated and the brainwave used to recognize that operation. More specifically, for example, when an operation is performed on the smartphone 3 exactly as recalled by the user, the learning device 205 receives an evaluation indicating that. Based on the detected brainwave and the input evaluation, the learning device 205 adjusts the waveform or pattern of the brainwave type in the first dictionary 211 to improve the accuracy of brainwave type discrimination. The learning device 205 also displays multiple types of pointers sequentially on the screen of the smartphone 3, which serve as the reference or starting point for operations using the recall input method, and outputs operation instructions. The user recalls the operation according to those instructions. The learning device 205 receives user evaluations of recognized and executed operations, and based on these evaluations, ranks the above-mentioned multiple types of pointers in order of the accuracy of operation recognition.

[0037] The first control unit 206 comprehensively controls each device or function block so that the headset 2 operates properly.

[0038] 《Smartphone Function Blocks》 This section explains each functional block of Smartphone 3.

[0039] The second storage device 310 stores data necessary for the proper operation of the smartphone 3. In this embodiment, the second storage device 310 stores the operating system 311, the first application 312, the second application 313, the second dictionary (second table) 315, and the headset control tool 316. The first and second applications are various application software that run on the operating system 311. The first application 312 is an internet browser application, and the second application 313 is a fighting game application. The headset control tool 316 is software for controlling the headset. Therefore, it is not included in the software used by the smartphone 3 in this embodiment.

[0040] The second dictionary 315 consists of a table that associates software used by smartphone 3 with a list of operation types corresponding to that software. The operation type list is a limited list of operation types that should be recognized while using the software. Details of the second dictionary 315 will be described later.

[0041] The communication device 300 is connected to the communication device 200 on the headset 2 side wirelessly or via a wired connection, and performs bidirectional communication.

[0042] The software detection device 302 detects the software being used on the smartphone 3. In this embodiment, examples of the software being used include the operating system 311, the first application 312, and the second application 313, as described above. The operating system is, for example, Android (registered trademark) or iOS (a trademark used by Apple Inc.). The first and second applications are application software associated with icons displayed on the screen of the smartphone 3, for example.

[0043] The operation type list selection device 301 selects an operation type list corresponding to the detected software currently in use by referring to the second dictionary 315. The operation type list is a list of operation types that are narrowed down to only those operation types that should be recognized for each software.

[0044] The input mode detection device 303 detects the input mode being used in the smartphone 3 and outputs input mode identification information, which is information that identifies the detected input mode, to the headset 2. The input mode is selected by the software being used, according to the execution state of that software. In this embodiment, examples of input modes include pointer input mode, first text input mode, second text input mode, map operation input mode, and character operation input mode.

[0045] The pointer input mode is an input mode for moving the pointer and performing operations based on or originating from the pointer's position. The first text input mode is an input mode for inputting text including characters, numbers, and symbols using a virtual keyboard. The second text input mode is an input mode for inputting text including characters, numbers, and symbols by directly identifying them based on the user's brainwaves. The map operation input mode is an input mode for performing operations on the displayed map. The character operation input mode is an input mode for performing operations on characters appearing in games, etc.

[0046] When the recall operation receiving device 304 receives information (data) of the operation type recognized by the recall operation recognition device 204 of the headset 2, it accepts that operation type as an operation using the recall input method (hereinafter also referred to as a recall operation) (corresponding processing).

[0047] When the touch operation receiving device 305 detects a touch operation by the user 9 on the touch panel image display device 32, it accepts the operation type corresponding to that touch operation as an operation using the touch input method (hereinafter also referred to as a touch operation).

[0048] When the voice control receiver 306 detects voice from the user 9 through the microphone 35, it accepts the operation type corresponding to that voice as an operation using the voice input method (hereinafter also referred to as voice operation).

[0049] The operation reception control device 307 controls the reception of operations so that, when operations using multiple operation input methods are received simultaneously, it adopts and executes the operation using the operation input method with the highest priority among them.

[0050] Figure 4 shows an example of the priority order of operation input methods. In this embodiment, as shown in Figure 4, the priority order from highest to lowest is touch input method, voice input method, and recall input method. In this example, if there are no operations using the touch input method or voice input method, the operation reception control device 307 executes an operation using the recall input method, that is, an operation recognized by the recall operation recognition device 204. At this time, the operation is executed using the pointer displayed on the screen of the smartphone 3 as the reference or starting point.

[0051] The second control unit 308 comprehensively controls each device or function block so that the smartphone 3 operates properly.

[0052] 《Dictionary 1 and Dictionary 2》 Figure 5 shows an example of the first dictionary. As shown in Figure 5, the first dictionary 211 consists of a table that associates the types of operations a user uses when using the smartphone 3 with the types of brainwaves that are thought to occur when the user recalls that type of operation.

[0053] An electroencephalogram (EEG) type is, for example, an EEG pattern that has a characteristic waveform, amplitude, duration, etc. Operation types include, for example, as shown in Figure 5, pointer up movement, pointer down movement, pointer left movement, pointer right movement, swipe up, swipe down, swipe left, swipe right, flick up, flick down, flick left, flick right, drag up movement, drag down movement, drag left movement, drag right movement, tap, double tap, long press, pinch in, pinch out, and direct input of text, i.e., characters. Character types include, for example, the alphabet, hiragana, numbers, and symbols.

[0054] Note that swiping, flicking, dragging, tapping, double-tapping, long-pressing, pinching in, and pinching out are all performed relative to or originating from the pointer's position.

[0055] Here's a brief explanation of each operation type. A tap is the action of touching the screen with one finger, similar to a click on a computer. A double tap is the action of quickly touching the screen twice with one finger, similar to a double-click on a computer. A long press is the action of touching the screen with one finger for a while, and an action corresponding to the long press will occur. A pinch in is the action of touching the screen with two fingers and narrowing the distance between them, mainly used to shrink the screen. A pinch out is the action of touching the screen with two fingers and widening the distance between them, mainly used to enlarge the screen. A drag is the action of touching the screen with one finger and moving it up, down, left, or right, and an action corresponding to the finger movement will occur. A swipe is the action of touching the screen with one finger and sliding it up, down, left, or right, mainly used for scrolling the screen. A flick is the action of touching the screen with one finger and quickly sweeping it up, down, left, or right, and is often used in a method called flick input when entering text using a software keyboard.

[0056] The initial settings for the electroencephalogram (EEG) patterns of each EEG type included in the first dictionary 211 are determined through prior neurological research, EEG experiments, and analyses.

[0057] Figure 6 shows an example of the second dictionary. As shown in Figure 6, the second dictionary 315 consists of a table that associates the types of software executed on the smartphone 3 with software-specific operation type lists, which are lists that limit the types of operations used when that software is running. The software-specific operation type lists are further composed of one or more input mode-specific operation type lists. An input mode-specific operation type list is prepared for each input mode selected according to the execution state of the software. The input mode-specific operation type list lists only the types of operations used in the input mode being used.

[0058] As shown in Figure 6, in this embodiment, the following software-specific operation type lists are provided: an operating system operation type list LA associated with the operating system 311, a first application operation type list LB associated with the first application 312, and a second application operation type list LC associated with the second application 313.

[0059] Figure 7 shows an example of a list of operation types by input mode. As shown in Figure 7, in this embodiment, the types of input modes are assumed to be pointer input mode, first text input mode, second text input mode, map operation input mode, character operation input mode, etc. Pointer input mode is a mode that accepts input of various operations based on the position of a pointer. First text input mode is a mode that accepts input of various operations for inputting text using a virtual keyboard. Second text input mode is a mode that accepts input of various operations for inputting text directly from brain waves. Map operation input mode is a mode that accepts input of operations to move and resize the displayed map area, or various operations to perform actions on the map. Character operation input mode is a mode that accepts input of various operations to move or perform actions on a character such as an avatar on the screen. Furthermore, the following operation type lists are provided for each input mode: Lp for pointer input mode, Lt1 for first text input mode, Lt2 for second text input mode, Lm for map operation input mode, and Lf for character operation input mode.

[0060] As shown in Figure 6, in this embodiment, the operation type list LA for the operating system includes the operation type list Lp for pointer input mode and the operation type list Lt1 for first text input mode. The operation type list LB for the first application includes the operation type list Lp for pointer input mode, the operation type list Lt1 for first text input mode, and the operation type list Lm for map operation input mode. The operation type list LC for the second application includes the operation type list Lp for pointer input mode, the operation type list Lt1 for first text input mode, and the operation type list Lf for character operation input mode.

[0061] Figure 8 shows an example of an operation type list for each input mode. As shown in Figure 8, the operation type list Lp for pointer input mode includes, for example, pointer up / down / left / right movement, swipe up / down / left / right movement, drag up / down / left / right movement, tap, long press, etc. The operation type list Lt1 for the first text input mode includes, for example, pointer up / down / left / right movement, drag up / down movement, flick up / down / left / right movement, tap, etc. The operation type list Lt2 for the second text input mode includes, for example, operations to directly input individual letters, numbers, and symbols. The operation type list Lm for map operation input mode includes, for example, pointer up / down / left / right movement, swipe up / down / left / right movement, drag up / down / left / right movement, pinch in, pinch out, tap, double tap, etc. The operation type list Lf for character operation input mode includes, for example, pointer right movement, pointer left movement, flick up / down / left movement, tap, double tap, etc.

[0062] Operation of Information Processing Systems [Example 1] The operation of the information processing system according to Embodiment 1 will now be described. In Embodiment 1, the software and input mode used on the smartphone 3 are initially the operating system 311 and pointer input mode, and it is assumed that the system switches to the first application 312 (browser) and first text input mode midway through.

[0063] Figure 9 shows an example of a smartphone display when only the operating system is being used. As shown in Figure 9, the smartphone 3 is equipped with a touch panel display screen 3d and a microphone 35.

[0064] The touch panel display screen 3d shows multiple icons 51 corresponding to individual apps, and a pointer 90. The pointer 90 is transparent or semi-transparent inside so that the image in the area overlapping with the pointer 90 is visible. At the bottom of the touch panel display screen 3d, there are icons 91 for returning to the home screen, 92 for returning to the previous state, and 93 for displaying menu screens, etc. The touch panel display screen 3d also displays a "Like!" button 96. When a recall operation is successfully performed, that is, when the operation is performed exactly as the user recalled, the user presses this "Like!" button. This inputs an evaluation indicating that the recall operation was successful, and this evaluation information is used to improve the accuracy of operation type recognition, i.e., the discrimination of brainwave types.

[0065] Users can operate the device by touching the 3D touch panel display screen. They can also operate it by voice input through the microphone 35. If good recall is temporarily impossible, or if quick operation is required, users can use either touch or voice operation as needed.

[0066] Figures 10A to 10D show the processing flow of the information processing system according to Embodiment 1. Note that the flow diagrams shown in Figures 10A to 10D are diagrams that divide one flow into four sections.

[0067] Figure 10A primarily shows the processing flow for preparing for the recall operation. This processing flow includes the process of establishing cooperation between the headset 2 and the smartphone 3. It also includes the process of limiting the types of operations to be recognized in the recall operation depending on the software and input mode being used on the smartphone 3.

[0068] Figure 10B primarily shows the processing flow for performing a recall operation. This processing flow includes the headset 2 detecting the user's brainwaves to recognize the type of operation and outputting the result to the smartphone 3. It also includes the smartphone 3 accepting operations from each input method, selecting the operation with the highest priority, and executing it.

[0069] Figure 10C primarily shows the processing flow in learning mode. This processing flow includes a process to learn the shape of the pointer displayed on the screen of smartphone 3 that improves the accuracy of recognition of recall operations.

[0070] Figure 10D primarily shows the processing flow for terminating the connection. This processing flow includes the steps for terminating the connection between headset 2 and smartphone 3.

[0071] Processing flow for preparing for recall operations First, we will explain the flow of the process for establishing cooperation between headset 2 and smartphone 3, and the process for limiting the types of operations to be recognized by recall operations according to the software used and the operation input mode on smartphone 3.

[0072] As shown in Figure 10A, first, the second control unit 308 outputs a cooperation request signal to the headset 2 (S101). The first control unit 206 outputs a cooperation permission signal to the smartphone 3 in response to the cooperation request signal (S102). The second control unit 308 receives the cooperation permission signal (S103). This establishes cooperation between the headset 2 and the smartphone 3.

[0073] Once the connection is established, the second control device 308 activates the headset control tool 316 stored in the second storage device 310 (S104), enabling data transmission and reception or operation control for the headset 2.

[0074] Next, the software detection device 302 detects the software being used on the smartphone 3 (S105). The operation type list selection device 301 refers to the second dictionary 315 shown in Figure 6 and selects a software-specific operation type list associated with the detected software (S106).

[0075] Here, it is assumed that operating system 311 is being used, so operating system 311 is detected as the software being used, and the operating system operation type list LA is selected as the software-specific operation type list. Figure 11 is an illustrative diagram of the operation type list for the operating system. As shown in Figure 11, the operation type list LA for the operating system consists of the operation type list Lp for pointer input mode and the operation type list Lt1 for first text input mode.

[0076] When a software-specific operation type list is selected, the second control unit 308 outputs the selected software-specific operation type list to the headset 2 (S107). In this case, the operating system operation type list LA is output.

[0077] When the selected software-specific operation type list is output to headset 2, the first control unit 206 of headset 2 acquires that software-specific operation type list (S108).

[0078] The input mode detection device 303 of the smartphone 3 detects the input mode currently in use on the smartphone 3 (S109). The second control device 308 outputs input mode identification information representing the detected input mode to the headset 2 (S110).

[0079] In this case, since it is assumed that pointer input mode is being used, input mode identification information, which identifies the pointer input mode, is output to headset 2.

[0080] The first control device 206 of the headset 2 acquires input mode identification information (S111). The operation type list application device 202 selects an input mode-specific operation type list corresponding to the input mode identified by the acquired input mode identification information from the software-specific operation type list, and applies the selected input mode-specific operation type list to the first dictionary 211 (S112). In other words, the operation types recognized from the brainwaves are limited to those included in the applied input mode-specific operation type list among the operation types registered in the first dictionary 211.

[0081] Here, the operation type list Lp for pointer input mode is selected from the operation type list LA for the operating system. Then, the settings are made to limit the operation types to be recognized to those included in this list. As a result, the operation types to be recognized from the brainwaves are narrowed down to only those that are considered necessary in pointer input mode. In other words, operation types that are considered unnecessary in pointer input mode, specifically flick, pinch in, pinch out, or double tap, are excluded from the operation types to be recognized. Therefore, the number of operation types to be recognized is reduced, misrecognition of operation types is suppressed, and recognition accuracy is improved.

[0082] 《Processing flow for recall operations》 Next, we will explain the process of detecting the user's brainwaves in headset 2 to recognize the type of operation and outputting the result to smartphone 3, and the process of executing the operation in smartphone 3 while considering the priority of each operation input method.

[0083] As shown in Figure 10B, the detection device 201 detects the user's brainwaves (S201) and temporarily records the waveform in the first memory device 210. The brainwave type discrimination device 203 determines the brainwave type based on the recorded brainwave waveform, etc. (S202). At this time, the discrimination is limited to brainwave types that correspond to the operation types included in the currently applied input mode operation type list from among the brainwave types registered in the first dictionary 211, and brainwave types outside of these categories.

[0084] For example, suppose a user recalls the operation of moving the pointer to the left. Then, the detection device 201 detects and records the brainwave corresponding to that operation of moving the pointer to the left. The brainwave type discrimination device 203 then compares the recorded brainwave waveform with brainwave types that correspond to operation types included in the pointer input mode operation type list Lp, which are registered in the first dictionary 211. At this time, brainwave types corresponding to operation types not included in the pointer input mode operation type list Lp are excluded from the comparison. For example, brainwave types S1 to S4, which correspond to swiping, as shown in Figure 5, are excluded from the comparison. As a result of the comparison, the brainwave type with the highest similarity, likelihood, or correlation value to the recorded brainwave is identified. If the brainwave type discrimination is good, brainwave type P3, which corresponds to moving the pointer to the left, will be identified from the brainwave types shown in Figure 5.

[0085] In this embodiment, discrimination is limited to electroencephalogram (EEG) types corresponding to the operation types included in the operation type list Lp for pointer input mode, as shown in Figure 8.

[0086] The recall operation recognition device 204 recognizes that no recall operation has been performed if the detected brainwaves are not classified into any of the limited brainwave types. The classification of brainwave types is performed, for example, by selecting the brainwave type with the highest similarity to the recorded brainwaves.

[0087] The recall operation recognition device 204 refers to the first dictionary 211 to identify the operation type corresponding to the identified brainwave type, and recognizes the identified operation type as the operation type recalled by the user (S203). Once the operation type is recognized, the first control device 206 outputs the recognized operation type information representing the recognized operation type to the smartphone 3 (S204).

[0088] The second control device 308 acquires recognized operation type information representing the recognized operation type (S205). Here, the recall operation receiving device 304 accepts the operation type represented by the acquired recognized operation type information as an operation using the recall input method. The touch operation receiving device 305 accepts operations using the touch input method, and the voice operation receiving device 306 accepts operations using the voice input method (S206).

[0089] The operation reception control device 307 detects whether an operation has been performed using each operation input method based on the reception status of operations using each operation input method. Specifically, it detects whether an operation has been performed using the touch input method, the voice input method, and the recall input method. Then, it identifies the operation using the operation input method with the highest priority among the detected operations (S207) and executes the identified operation (S208).

[0090] Here, the second control device 308 outputs a learning evaluation for the recall operation to the headset 2 (S209). The learning evaluation includes, for example, information on whether or not the user has pressed the "OK!" button. Based on this information, the learning device 205 performs a learning process to improve the recognition accuracy of the recall operation (S210). For example, if it is determined from this information that the "OK!" button has been pressed, it is considered that the recall operation is being performed correctly, and the characteristics of the brainwaves detected at that time are fed back to the first dictionary 211 and reflected in the brainwave pattern corresponding to the type of brainwave to be distinguished.

[0091] When the learning process is executed, the second control unit 308 determines whether the input mode being used has changed (S211). If it is determined that the input mode being used has changed (S211:Yes), the process returns to step S109. If it is determined that the input mode has not changed (S211:No), the process proceeds to the next determination. The second control unit 308 determines whether the software being used has changed (S212). If it is determined that the software being used has changed (S212:Yes), the process returns to step S105. If it is determined that the software being used has not changed (S212:No), the process proceeds to the next determination. The second control unit 308 determines whether the learning mode has been selected by the user (S213). If it is determined that the learning mode has been selected (S213:Yes), the process proceeds to step S301. If it is determined that the learning mode has not been selected (S213:No), the process proceeds to the next determination. The second control device 308 determines whether or not the cooperation should be terminated (S214). Situations in which the cooperation should be terminated include, for example, when an instruction to terminate the cooperation is input, when a failure occurs in part of the cooperation, or when the battery level becomes low. If it is determined that the cooperation should be terminated (S214: Yes), the process proceeds to step S401. If it is determined that the cooperation should not be terminated (S214: No), the process returns to step S201.

[0092] Here, let's assume that the user performs an operation, for example, by tapping icon B corresponding to the first application (internet browser application) 312, and the software used on the smartphone 3 switches from only the operating system 311 to the first application 312. In this case, when the process in step S212 is executed, it is determined that the software being used has changed (S212: Yes), and the process returns to step S105.

[0093] Figure 12 shows an example of a smartphone display when the first application is in use. As shown in Figure 12, the touch panel display screen 3d of the smartphone 3 shows a text box 54 for entering search keywords used for web searches and a software keyboard 94 for typing text.

[0094] The software detection device 302 detects the first application 312 after the switch (S105). The operation type list selection device 301 refers to the second dictionary 315 shown in Figure 6 and selects the operation type list LB for the first application that corresponds to the first application 312 (S106). Figure 13 is an illustrative diagram of the operation type list LB for the first application. As shown in Figure 13, the operation type list LB for the first application consists of the operation type list Lp for pointer input mode, the operation type list Lt1 for first text input mode, and the operation type list Lm for map operation input mode.

[0095] The second control unit 308 outputs the selected operation type list LB for the first application to the headset 2 (S107). The first control unit 206 of the headset 2 acquires the operation type list LB for the first application (S108).

[0096] The input mode detection device 303 of the smartphone 3 detects the input mode currently selected on the smartphone 3 (S109). Here, it is assumed that the first text input mode is selected, so the first text input mode is detected. The second control device 308 outputs input mode identification information that identifies the detected first text input mode to the headset 2 (S110).

[0097] The first control device 206 of the headset 2 acquires input mode identification information (S111). The operation type list application device 202 identifies the operation type list Lt1 for the first text input mode corresponding to the first text input mode identified by the acquired input mode identification information, and applies the operation type list Lt1 for the first text input mode to the first dictionary 211 (S112). In other words, the operation types recognized from the brainwaves are limited to those included in the operation type list Lt1 for the first text input mode from among the operation types included in the first dictionary 211.

[0098] As a result, the types of operations to be recognized from the brainwaves are narrowed down to only those deemed necessary in the first text input mode. In other words, operations that are deemed unnecessary in the first text input mode, specifically swiping, pinching in, pinching out, long pressing, or double tapping, are excluded from the types of operations to be recognized.

[0099] Through this process, unless a learning mode is selected or the connection is terminated, the cycle of recognizing recall operations through brainwave detection and executing the recognized recall operations continues. The user can perform desired operations by manipulating the pointer 90 displayed on the smartphone 3 screen with recall operations. However, if touch operations or voice operations are available, these operations will be executed preferentially. Furthermore, whenever the software being used or the input mode changes, a new list of operation types is applied to the first dictionary 211. In other words, the operation types to be recognized, i.e., the brainwave types to be distinguished, are always appropriately narrowed down.

[0100] Learning Mode Processing Flow Next, we will explain the processing in learning mode, that is, the learning process to improve the accuracy of recognizing the type of operation in recall operations.

[0101] As shown in Figure 10C, when the learning mode is selected, the second control device 308 outputs a learning request signal to the headset 2 (S301). The learning device 205 outputs a learning permission signal to the smartphone 3 in response to the learning request signal (S302). The second control device 308 acquires this learning permission signal. Once the learning permission signal is acquired, the learning mode is established.

[0102] The second control device 308 selects one of several types of pointers that are prepared in advance (S304) and displays the selected pointer on the screen. The second control device 308 also outputs an operation instruction to the user (S305). The operation instruction may be output in the form of an image or text, or in the form of an audio output.

[0103] Figure 14 shows an example of a smartphone display when learning mode is selected. As shown in Figure 14, the touch panel display screen 3d displays, for example, a star-shaped pointer 90s, along with an instruction image 95s indicating an instruction to move the pointer to the right. A "Like!" button 96 is also displayed, which the user presses when the recall operation is successfully performed. At this point, the user recalls the operation according to the outputted instruction.

[0104] When a pointer is displayed and an operation instruction is output, the learning device 205 controls the detection device 201, the EEG type discrimination device 203, and the recall operation recognition device 204 to detect the user's brainwaves (S306), determine the EEG type based on the detected brainwaves (S307), and, based on the determined EEG type, refer to the first dictionary 211 and the applied operation type list to recognize and output the operation type (S308). The learning device 205 also stores the brainwaves detected during the learning mode in the first memory device 210, associating them with the timing of their detection. The second control device 308 receives the operation type output from the headset 2 and executes the operation corresponding to that operation type (S309).

[0105] If the user's performed operation matches the operation they recalled, they press the "Like!" button 96 and input an evaluation indicating that the recall operation was successful. When the second control device 308 receives an evaluation indicating that the recall operation was successful, it accepts the evaluation (S310) and stores evaluation result information in the second storage device 310, which associates the type of operation instruction, the timing, and the evaluation.

[0106] The second control device 308 determines whether or not to change the operation instruction (S311). The criteria for this determination are, for example, whether the duration of output of the operation instruction has exceeded a certain period of time, or whether the number of inputs indicating that the recall operation is good has exceeded a certain number. If the second control device 308 determines to change the operation instruction (S311, Yes), it returns to step S305 and changes and outputs the operation instruction. If the second control device 308 determines not to change the operation instruction (S311, No), it determines whether or not to change the pointer to be displayed (S312). The criteria for this determination are, for example, whether the number of times the operation instruction has been changed has exceeded a certain number, or whether the number of inputs indicating that the recall operation is good has exceeded a certain number.

[0107] If the second control device 308 determines that it is necessary to change the pointer to be displayed (S312, Yes), it returns to step S304 and selects a different pointer.

[0108] Figure 15 shows an example of a display where the displayed pointer has been changed to a different pointer and an operation instruction has been output. In the example in Figure 15, the displayed pointer has been changed to a face-shaped pointer 90f, and an operation instruction image 95f meaning "flick down" is displayed as the operation instruction.

[0109] Figure 16 shows an example of a display where the displayed pointer has been changed to another pointer and an operation instruction has been output. In the example in Figure 16, the displayed pointer has been changed to a cross-shaped pointer 90c, which is represented by a conspicuous color such as red or blue, and an operation instruction image 95c meaning "drag to move to the left" is displayed as the operation instruction.

[0110] If the second control device 308 determines that the displayed pointer should not be changed (S312, No), it terminates the learning mode and outputs the evaluation result information stored in the second storage device 310 to the headset 2 (S313). Based on the acquired evaluation result information, the learning device 205 adjusts the first dictionary 211 (S314). For example, based on the type of operation and the electroencephalogram waveform at the time when the recall operation was evaluated as good, it modifies the electroencephalogram pattern of the electroencephalogram type registered in the first dictionary 211 to adjust it to better reflect the user's electroencephalogram characteristics. This adjustment of the first dictionary 211 improves the accuracy of discriminating the electroencephalogram type in the recall operation and improves the accuracy of recognizing the operation type. The learning device 205 also analyzes which pointer improves the accuracy of electroencephalogram discrimination and determines the ranking of the pointers in order of increasing accuracy (S315). The information on the pointer ranking is output to the smartphone 3. The smartphone 3 refers to the pointer ranking and allows the user to select any type of pointer to display. Alternatively, the system could automatically select the pointer of the highest-ranking type.

[0111] Once the learning mode is finished, the second control unit 308 returns to the execution step S214 and continues processing.

[0112] 《Collaboration Termination Processing Flow》 In step S214, if it is determined that the connection should be terminated (S214: Yes), the second control unit 308 outputs a connection termination request signal to the headset 2 (S401). Then, the first control unit 206 outputs a connection termination permission signal to the smartphone 3 in response to that request signal (S402). The second control unit 308 receives the connection termination permission signal (S403), and the connection termination is established.

[0113] According to this embodiment 1, even if the software being used is changed midway through, the types of operations to be recognized in the recall operation are appropriately limited, thereby reliably suppressing misrecognition of operation types and improving recognition accuracy.

[0114] [Example 2] In Example 2, the software used is the first application (internet browser application) 312, and the input mode is initially pointer input mode, with a switch to map operation input mode midway through.

[0115] Figure 17 shows an example of a smartphone display when the software being used is the second application and the input mode is pointer input mode. In this embodiment, for example, as shown in Figure 17, the touch panel display screen 3d displays window 52 and window 53 containing the map, partially overlapping. In addition, a pointer 90 is displayed at one of the positions on the touch panel display screen 3d.

[0116] In this embodiment, in the flow shown in Figure 10A, in step S105, the second application 313 is detected as the software to be used, and in step S106, the operation type list LB for the first application (see Figure 13) is selected as the software-specific operation type list.

[0117] In step S107, the selected operation type list LB for the first application is output to headset 2. In step S108, headset 2 acquires the operation type list LB for the first application. In step S109, the input mode is detected as a pointer input mode, and in step S110, information identifying the pointer input mode is output to headset 2. In step S111, the information identifying the pointer input mode is acquired by headset 2, and in step S112, the operation type list Lp for the pointer input mode (see Figure 13) in the operation type list LB for the first application is applied to the first dictionary 211.

[0118] Subsequently, steps S201 to S214 are repeatedly executed, and recall operations are performed. While steps S201 to S214 are being repeatedly executed, the user manipulates the pointer 90 on the touch panel display screen 3d to, for example, select a window of greater interest in the background and bring it to the front, or expand a window in the foreground. To bring a window in the background to the front or expand a window in the foreground, for example, the user moves the pointer 90 to that window through a recall operation and then taps it.

[0119] Figure 18 shows an example of the display when the front window is tapped to expand it. For example, as shown in Figure 18, window 53 is expanded and the map is displayed in an enlarged state. At this time, the input mode switches from pointer input mode to map operation input mode.

[0120] In this case, in the flow shown in Figure 10B, step S211 determines that the input mode has changed, and the process returns to step S109. In step S109, the map operation input mode is detected as the input mode, and in step S110, information identifying the map operation input mode is output to headset 2. In step S111, the information identifying the map operation input mode is acquired by headset 2, and in step S112, the operation type list Lm for map operation input mode (see Figure 13) in the operation type list LB for the first application is applied to the first dictionary 211.

[0121] Subsequently, steps S201 to S214 are repeatedly executed, and the recall operation continues. While steps S201 to S214 are being repeatedly executed, the user operates the pointer 90 on the touch panel display screen 3d, for example, to move the map display area or zoom in on the map. To move the map display area, for example, the user drags the pointer 90 up, down, left, or right using the recall operation. To zoom in on the map, the user moves the pointer 90 to the desired zoom position using the recall operation and then pinches out or double-taps at that position.

[0122] According to this embodiment 2, even if the input mode being used is changed midway through, the types of operations to be recognized in the recall operation are appropriately limited, thereby reliably suppressing misrecognition of operation types and improving recognition accuracy.

[0123] [Example 3] In Example 3, we assume that the software being used is the second application (fighting game application) 313, and that the input mode is the character control input mode.

[0124] Figure 19 shows an example of a smartphone display when the software used is the second application and the input mode is character operation input mode. In this embodiment, for example, as shown in Figure 19, the touch panel display screen 3d displays an image 55 that includes a character 54a corresponding to the user's avatar and a character 54b that is the opponent. Note that a pointer may or may not be displayed.

[0125] In this embodiment, in the flow shown in Figure 10A, in step S105, the second application 313 is detected as the software to be used, and in step S106, the operation type list LC for the second application is selected as the software-specific operation type list.

[0126] Figure 20 is an illustrative diagram of the operation type list for the second application. As shown in Figure 20, the operation type list LC for the second application consists of the operation type list Lp for pointer input mode, the operation type list Lt1 for first text input mode, and the operation type list Lf for character operation input mode.

[0127] In step S107, the selected operation type list LC for the second application is output to the headset 2. In step S108, the headset 2 acquires the operation type list LC for the second application. In step S109, the character operation input mode is detected as the input mode, and in step S110, information identifying the character operation input mode is output to the headset 2. In step S111, the information identifying the character operation input mode is acquired by the headset 2, and in step S112, the operation type list Lf for the character operation input mode (see Figure 20) in the operation type list LC for the second application is applied to the first dictionary 211.

[0128] The character control input mode operation type list Lf includes left / right pointer movement, up / down / left flick, tap, and double tap, as shown in Figure 20. Moving the pointer to the right corresponds to the character moving forward, and moving the pointer to the left corresponds to the character moving backward. Furthermore, flicking up corresponds to the character jumping, flicking down corresponds to the character crouching, and flicking left corresponds to the character guarding. In addition, tapping corresponds to the character punching, and double tapping corresponds to the character kicking.

[0129] This process limits the types of operations to be recognized to those included in the character input mode operation type list Lf. As a result, the types of operations to be recognized from the brainwaves are narrowed down to only those deemed necessary for character input mode. In other words, operations that are considered unnecessary for character input mode, specifically, pointer up / down movement, swiping up / down / left / right, dragging up / down, pinch-in, and pinch-out, are excluded from the types of operations to be recognized.

[0130] According to this embodiment 3, it is easy to understand that the direction and actions of operations in game apps often resemble the target actions within the game, allowing for intuitive use of recall operations. Furthermore, the number of operation types tends to be relatively small compared to other apps. Therefore, the information processing system according to this embodiment is extremely practical for such game apps, especially fighting, sports, rhythm, or card-based game apps.

[0131] Furthermore, the second control device 308 may control the pointer 90 to return to the center position if no recall operation is recognized for a certain period of time or longer. This control resets the pointer position, making it easier to start the next recall operation.

[0132] According to Embodiment 1 described above, a software-specific operation type list corresponding to the software used in the smartphone 3 is applied to the first dictionary 211. Similarly, a command-line operation type list corresponding to the input mode used in the smartphone 3 is applied to the first dictionary 211. Therefore, the candidates for brainwave type identification are narrowed down to those associated with operation types used in the software or input mode used in the smartphone 3. In other words, operation types considered unnecessary for the software or input mode are excluded from the operation types to be recognized, and brainwave types corresponding to these operation types are also excluded from the brainwave types to be identified. This reduces the number of brainwave types to be identified and operation types to be recognized, making brainwave type identification easier, suppressing misrecognition of operation types, and improving recognition accuracy. As a result, highly feasible and practical recall operations can be realized in the operation of the information processing terminal. That is, the operation recalled by the user can be recognized and applied to the operation of the information processing terminal.

[0133] (Embodiment 2) The information processing system according to Embodiment 2 is configured to identify an operation type list only according to the input mode on the smartphone, and to apply that operation type list to the first dictionary on the headset.

[0134] <Functional configuration> The functional configuration of the information processing system according to Embodiment 2 will be described.

[0135] Figure 21 is a diagram showing the configuration of the headset and smartphone functional blocks in the information processing system according to Embodiment 2.

[0136] The information processing system 1a is based on the information processing system 1 according to Embodiment 1, but the second storage device 310a of the smartphone 3a stores a third dictionary (third table) 317 instead of the second dictionary 315.

[0137] The third dictionary 317 consists of a table that associates the input modes used in the smartphone 3a with a list of operation types corresponding to those input modes, organized by input mode.

[0138] Figure 22 shows an example of the third dictionary. As shown in Figure 22, the third dictionary 317 consists of a table that associates each input mode type with a list of input mode types, which are limited lists of input mode types and the types of operations used when that input mode is in use.

[0139] As shown in Figure 22, in this embodiment, the following operation type lists are provided for each input mode: Lp for pointer input mode, Lt1 for first text input mode, Lm for map operation input mode, and Lf for character operation input mode.

[0140] [Example 1] Example 1 is an example in which the headset obtains a list of operation types for each input mode each time the input mode of the smartphone changes.

[0141] In other words, in Embodiment 1, the smartphone 3a detects the input mode being used and outputs a list of operation types for each input mode corresponding to the detected input mode to the headset 2. Whenever the input mode changes, the input mode is detected and the list of operation types for each input mode corresponding to that input mode is applied to the first dictionary 211.

[0142] Figures 23A to 23D are processing flow diagrams according to Embodiment 2 of the information processing system in Example 1. The parts that differ from the flow of Embodiment 1 are included in the processing flow diagrams in Figures 23A and 23B. Here, only the processing flow related to the parts that differ from Embodiment 1 will be explained, and the explanation of other parts will be omitted.

[0143] In the processing flow shown in Figure 23A, once the connection between the smartphone 3a and the headset 2 is established (S103), the headset control tool 316 is activated on the smartphone 3a (S104), and then the input mode is detected (S109). Once the input mode is detected, a list of operation types corresponding to that detected input mode is selected (S110). The selected list of operation types corresponding to the input mode is then output to the headset 2 (S121). The headset 2 acquires this list of operation types corresponding to the input mode (S122) and applies it to the first dictionary 211 (S113).

[0144] Furthermore, in the processing flow shown in Figure 23B, a determination is made as to whether the input mode has changed (S210). If it is determined that the input mode has not changed (S210, No), the determination of whether the software being used has changed is not made, and the determination of whether the learning mode has been selected is made immediately (S212). On the other hand, if it is determined that the input mode has changed (S210, Yes), the process returns to step S109, and the input mode is detected again.

[0145] [Example 2] Example 2 is an example in which the headset is pre-configured to acquire a list of all operation types for each input mode.

[0146] In other words, in Embodiment 2, the smartphone first outputs all the prepared input mode-specific operation type lists to the headset, and the headset acquires all of these input mode-specific operation type lists. Then, the smartphone detects the input mode and outputs input mode identification information to the headset. The headset acquires this input mode identification information, identifies the input mode based on the acquired input mode identification information, and applies the input mode-specific operation type list corresponding to the identified input mode to the first dictionary 211. Whenever the input mode changes, the input mode is detected and the input mode-specific operation type list corresponding to that input mode is applied to the first dictionary.

[0147] Figures 24A to 24D are processing flow diagrams according to Embodiment 2 of the information processing system. The parts that differ from the flow of Embodiment 1 are included in the processing flow diagrams of Figures 24A and 24B. Here, only the processing flow related to the parts that differ from Embodiment 1 will be explained, and the explanation of other parts will be omitted.

[0148] In the processing flow of Figure 24A, once the cooperation between the smartphone 3a and the headset 2 is established (S103), the headset control tool 316 is launched on the smartphone 3a (S104), and then all the prepared input mode-specific operation type lists are output together (S131). The headset 2 acquires all of these prepared input mode-specific operation type lists. The input mode is detected on the smartphone 3a (S109), and input mode identification information is output to the headset 2 (S133). The headset 2 acquires the input mode identification information (S134), and the input mode-specific operation type list corresponding to the input mode identified by that input mode identification information is applied to the first dictionary 211 (S113).

[0149] Furthermore, in the processing flow of Figure 24B, a determination is made as to whether the input mode has changed (S210). If it is determined that the input mode has not changed (S210, No), the determination of whether the software being used has changed is not made, and the determination of whether the learning mode has been selected is made immediately (S212). On the other hand, if it is determined that the input mode has changed (S210, Yes), the process returns to step S109, and the input mode is detected again.

[0150] According to Embodiment 2 described above, a list of operation types for each input mode corresponding to the input mode used in the smartphone 3a is applied to the first dictionary 211. As a result, the candidates for determining brainwave types are narrowed down to brainwave types associated with the operation types used in the input mode used in the smartphone 3a. In other words, operation types that are considered unnecessary in the input mode being used are excluded from the operation types to be recognized, and brainwave types corresponding to these operation types are also excluded from the brainwave types to be determined. This reduces the number of brainwave types to be determined and operation types to be recognized, making it easier to determine brainwave types, suppressing misrecognition of operation types, and improving recognition accuracy. As a result, similar to Embodiment 1, it is possible to realize highly feasible and practical recall operations in the operation of the information processing terminal. That is, it is possible to recognize the operation recalled by the user and apply it to the operation of the information processing terminal.

[0151] Furthermore, according to Embodiment 2, there is no need to detect the software used in the smartphone 3a or to prepare a list of operation types for each software, making it easier to build the system.

[0152] (Embodiment 3) The information processing system according to Embodiment 3 is configured so that the identification of brainwave types, recognition of operation types, etc., are performed on the smartphone side, rather than on the headset side.

[0153] <Functional configuration> The functional configuration of the information processing system according to Embodiment 3 will be described below.

[0154] Figure 25 is a diagram showing the configuration of the headset and smartphone functional blocks in the information processing system according to Embodiment 3.

[0155] As shown in Figure 25, the information processing system 1b according to Embodiment 3 comprises a headset 2b and a smartphone 3b. The headset 2b has a communication device 200, a detection device 201, a first control device 206, and a first storage device 210. The smartphone 3b has a communication device 300, an operation type list selection device 301, a software usage detection device 302, an input mode detection device 303, a recall operation reception device 304, a touch operation reception device 305, a voice operation reception device 306, an operation reception control device 307, a second control device 308, an operation type list application device 202, an electroencephalogram type discrimination device 203, a recall operation recognition device 204, a learning device 205, and a second storage device 310b.

[0156] The second storage device 310b includes an operating system 311, a first application 312, a second application 313, a second dictionary 315, a headset control tool 316, and a first dictionary 211.

[0157] In Embodiment 3, the smartphone 3b detects the software being used and the input mode, and applies the input mode-specific operation type list corresponding to the detected software and input mode to the first dictionary 211. The headset 2b is responsible for detecting the user's brainwaves and outputting them to the smartphone 3b. When the software being used changes, the software being used is detected and the software-specific operation type list corresponding to that software is selected each time. Similarly, when the input mode changes, the input mode is detected and the input mode-specific operation type list corresponding to that input mode is applied to the first dictionary 211 each time.

[0158] Figures 26A to 26D show the processing flow of the information processing system according to Embodiment 3. The parts that differ from the processing flow of Embodiment 1 are included in the processing flows of Figures 26A to 26C. Here, only the processing flows related to the parts that differ from Embodiment 1 will be explained, and the explanation of other parts will be omitted.

[0159] 《Preparation Process Flow for Recall Operations》 As shown in Figure 26A, first, the second control device 308 outputs a cooperation request signal to the headset 2b (S101). The first control device 206 outputs a cooperation permission signal to the smartphone 3b in response to the cooperation request signal (S102). The second control device 308 receives the cooperation permission signal (S103). This establishes cooperation between the smartphone 3b and the headset 2b.

[0160] Once the connection is established, the second control device 308 activates the headset control tool 316 stored in the second storage device 310b (S104), enabling data transmission and reception or operation control for the headset 2b.

[0161] Next, the software detection device 302 detects the software being used on the smartphone 3b (S105). The operation type list selection device 301 refers to the second dictionary 315 shown in Figure 6 and selects a software-specific operation type list associated with the identified software (S106).

[0162] In this embodiment, it is assumed that the software used is the operating system 311. In this case, the operating system 311 is detected as the software used, and the operating system operation type list LA is selected as the software-specific operation type list.

[0163] When a software-specific operation type list is selected, the second control device 308 detects the input mode currently in use on the smartphone 3b (S109) and selects an input mode-specific operation type list corresponding to the detected input mode (S110).

[0164] In this embodiment, we assume that the pointer input mode is selected as the input mode. Therefore, the operation type list Lp for pointer input mode is selected.

[0165] The recall operation recognition device 204 applies the selected input mode-specific operation type list to the first dictionary 211. That is, it sets the operation types to be recognized to be limited to those included in the applied input mode-specific operation type list from among the operation types registered in the first dictionary 211 (S113).

[0166] Here, the operation type list Lp for pointer input mode is applied to the first dictionary 211. As a result, the operation types to be recognized are narrowed down to only those that are considered necessary in pointer input mode.

[0167] 《Processing flow for recall operations》 As shown in Figure 26B, the detection device 201 detects the user's brainwaves and outputs the detected brainwaves to the smartphone 3b (S231). The brainwave type discrimination device 203 of the smartphone 3b acquires the brainwaves and determines the brainwave type based on the waveform of the acquired brainwaves (S202). At this time, the discrimination is limited to brainwave types that correspond to the operation types included in the input mode-specific operation type list currently applied, and brainwave types outside of these categories, from among the brainwave types registered in the first dictionary 211. In this embodiment, the discrimination is limited to brainwave types that correspond to the operation types included in the pointer input mode operation type list Lp, as shown in Figure 8.

[0168] The recall operation recognition device 204 refers to the first dictionary 211 to identify the operation type corresponding to the determined brainwave type and recognizes the identified operation type as the operation type recalled by the user (S232). Once the operation type is recognized, the second control device 308 acquires information representing the recognized operation type (S204). The operation reception control device 307 detects whether or not an operation has been performed using each operation input method (S205). Specifically, it detects whether or not an operation has been performed using the touch input method, the voice input method, and the recall input method. The operation reception control device 307 then identifies the operation using the operation input method with the highest priority among the detected operations (S206) and adopts and executes the identified operation (S207).

[0169] Here, the learning device 205 receives a learning evaluation for the recall operation (S208). The learning evaluation includes, for example, information on whether or not the user has pressed the "OK!" button. Based on this information, the learning device 205 performs a learning process to improve the recognition accuracy of the recall operation (S209).

[0170] When the learning process is executed, the second control unit 308 determines whether the input mode has changed (S210). If it is determined that the input mode has changed (S210: Yes), it returns to step S109. If it is determined that the input mode has not changed (S210: No), it proceeds to the next determination. The second control unit 308 determines whether the software being used has changed (S211). If it is determined that the software being used has changed (S211: Yes), it returns to step S105. If it is determined that the software being used has not changed (S211: No), it proceeds to the next determination. The second control unit 308 determines whether the learning mode has been selected by the user (S212). If it is determined that the learning mode has been selected (S212: Yes), it proceeds to step S331. If it is determined that the learning mode has not been selected (S212: No), it proceeds to the next determination. The second control unit 308 determines whether the cooperation should be terminated (S213). Situations in which the connection should be terminated include, for example, when a command to terminate the connection is entered, when a failure occurs in part of the connection, or when the battery level becomes low. If it is determined that the connection should be terminated (S213:Yes), the process proceeds to step S401. If it is determined that the connection should not be terminated (S213:No), the process returns to step S231.

[0171] Thus, unless there is a change in the input mode being used, a change in the software being used, a selection of a learning mode, or an termination of the connection, the cycle of recognizing recall operations through brainwave detection and executing the recognized recall operations will continue. The user manipulates the pointer 90 displayed on the screen of the smartphone 3b using recall operations to achieve the desired operation.

[0172] Learning Mode Processing Flow Next, we will explain the processing in learning mode.

[0173] As shown in Figure 26C, when the learning mode is selected, the learning device 205 starts processing the learning mode. The learning device 205 selects one of several types of pointers that are prepared in advance (S304) and displays the selected pointer on the screen. The learning device 205 also outputs an operation instruction to the user (S305). The operation instruction may be in the form of displaying an image or text, or it may be in the form of outputting audio.

[0174] When a pointer is displayed and an operation instruction is output, the learning device 205 controls the detection device 201 of the headset 2b to detect the user's brainwaves and output them to the smartphone 3b (S306). The brainwave type discrimination device 203 discriminates the brainwave type based on the detected brainwaves (S307), and the recall operation recognition device 204 recognizes the operation type by referring to the first dictionary 211 and the applied operation type list based on the discriminated brainwave type (S333). The learning device 205 also stores the brainwaves detected during the learning mode in the second memory device 310b in association with their timing. The second control device 308 executes the operation corresponding to the recognized operation type (S309).

[0175] If the user's executed operation matches the operation they recalled, they press the "Like!" button and input an evaluation indicating that the recall operation was successful. When the second control device 308 receives an evaluation indicating that the recall operation was successful, it accepts the evaluation (S310) and stores the type of operation instruction, the timing, and the evaluation result information associated with that evaluation in the second storage device 310b.

[0176] The second control device 308 determines whether or not to change the operation instruction (S311). If the second control device 308 determines to change the operation instruction (S311, Yes), it returns to step S305, changes the operation instruction, and outputs it. If the second control device 308 determines not to change the operation instruction (S311, No), it determines whether or not to change the pointer (S312).

[0177] If the second control device 308 determines that the pointer should be changed (S312, Yes), it returns to step S304 and selects a different pointer to display.

[0178] If the second control device 308 determines that the pointer should not be changed (S312, No), it terminates the learning mode and outputs the evaluation result information stored in the second storage device 310b to the headset 2 (S313). Based on the acquired evaluation result information, the learning device 205 adjusts the first dictionary 211 (S314). For example, based on the type of operation and the electroencephalogram waveform at the time when the recall operation was evaluated as good, it modifies the electroencephalogram pattern of the electroencephalogram type registered in the first dictionary 211 to adjust it to better reflect the user's electroencephalogram characteristics. This adjustment of the first dictionary 211 improves the accuracy of discriminating the electroencephalogram type in the recall operation and improves the accuracy of recognizing the operation type. The learning device 205 also analyzes which pointer improves the accuracy of electroencephalogram discrimination and determines the ranking of the pointers in order of increasing accuracy (S315). The information on the pointer ranking is output to the smartphone 3b. The smartphone 3b refers to the pointer ranking and allows the user to select any pointer to display. Alternatively, the system could automatically select the pointer of the highest-ranking type.

[0179] Once the learning mode is finished, the second control unit 308 returns to the execution step S214 and continues processing.

[0180] According to Embodiment 3 described above, the same effects as Embodiment 1 can be obtained. Furthermore, since processing other than brainwave detection is performed on the smartphone side, the amount of processing that must be performed on the headset can be reduced, and the configuration of the headset can be simplified.

[0181] (Embodiment 4) The information processing system according to Embodiment 4 is configured to perform brainwave type identification, operation type recognition, etc., on the smartphone side rather than the headset side, and to identify an operation type list according to the input mode of the smartphone and apply it to the first dictionary.

[0182] <Functional configuration> The functional configuration of the information processing system according to Embodiment 4 will be described below.

[0183] Figure 27 is a diagram showing the configuration of the headset and smartphone functional blocks in the information processing system according to Embodiment 4.

[0184] As shown in Figure 27, the information processing system 1c according to Embodiment 4 is based on the information processing system 1b according to Embodiment 3, but the second storage device 310c of the smartphone 3c stores a third dictionary 317 instead of a second dictionary 315.

[0185] The third dictionary 317 consists of a table that associates the input mode selected on the smartphone 3c with a list of operation types corresponding to that input mode, for each input mode.

[0186] Figures 28A to 28D show the processing flow of the information processing system according to Embodiment 4. The parts that differ from the processing flow of Embodiment 3 are included in the processing flows of Figures 26A and 26B. Here, only the processing flows related to the parts that differ from Embodiment 3 will be explained, and the explanation of other parts will be omitted.

[0187] In the processing flow shown in Figure 28A, once the cooperation between the smartphone 3c and the headset 2c is established (S103), the headset control tool is launched on the smartphone 3c (S104). After this, the software being used is not detected, and the input mode is detected (S109). Once the input mode is detected, a list of operation types corresponding to the detected input mode is selected (S110) and applied to the first dictionary 211 (S113).

[0188] Furthermore, in the processing flow of Figure 28B, a determination is made as to whether or not the input mode has changed (S210). If it is determined that the input mode has not changed (S210, No), the determination of whether the software being used has changed is not made, and the determination of whether or not the learning mode has been selected is made immediately (S212). On the other hand, if it is determined that the input mode has changed (S210, Yes), the process returns to step S109, and the input mode is detected again.

[0189] According to Embodiment 4 described above, the same effects as in Embodiment 2 can be obtained. Furthermore, since processing other than brainwave detection is performed on the smartphone side, the amount of processing that must be performed on the headset can be reduced, and the configuration of the headset can be simplified.

[0190] (Embodiment 5) In each of the embodiments described above, the headset is an example of an electrode device in the present invention, and its shape, configuration, and usage are arbitrary. Therefore, as the electrode device in the present invention, for example, a hat-type device, a non-wearable electrode-attached device, etc., may be used.

[0191] Furthermore, in each of the above embodiments, the smartphone is merely an example of an information processing terminal in the present invention, and its shape, size, specifications, etc., are arbitrary. Therefore, instead of a smartphone, for example, a tablet terminal, a notebook computer with a touch panel, etc., may be used as the information processing terminal in the present invention.

[0192] Thus, according to each of the above embodiments, the types of operations to be recognized based on the user's brainwaves are limited to those necessary at that time, making it easier to distinguish between brainwave types and suppressing misrecognition of operation types. As a result, it becomes possible to realize highly feasible and practical recall operations in the operation of an information processing terminal.

[0193] Furthermore, since recall operations are possible even without requiring high precision in detecting or identifying brainwave types, users can enjoy recall operations on information processing terminals even using relatively inexpensive brainwave detection devices.

[0194] Furthermore, by displaying a pointer on the touch panel screen and designating operations that use that pointer as a reference or starting point as the type of operation to be recognized, recall operations can be used more intuitively as an alternative to touch operations.

[0195] Although various embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above and includes various modifications. Furthermore, the embodiments described above are described in detail for the purpose of explaining the present invention in an easy-to-understand manner and are not necessarily limited to those having all the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these are within the scope of the present invention. In addition, the numbers and messages included in the text and figures are merely examples, and using different ones will not impair the effects of the present invention.

[0196] Furthermore, it is possible to add, delete, or replace some of the configurations in each embodiment with other configurations. Also, some or all of the above configurations, functions, processing units, processing means, etc., may be implemented in hardware, for example, by designing them as integrated circuits. Alternatively, the above configurations, functions, etc., may be implemented in software by a processor such as an MPU or CPU interpreting and executing programs that realize each function. Furthermore, the scope of functions implemented by software is not limited, and hardware and software may be used in combination. Information such as programs, tables, and files that realize each function can be stored in memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0197] Possible embodiments of the present invention are described below.

[0198] [Note 1] A method for recognizing recall operations, A detection step for detecting the user's brainwaves used to recognize the operation recalled by the user, A determination step involves referring to a first table in which the types of operations accepted by the information processing terminal and the types of brainwaves corresponding to those operations are associated for each type of operation, and determining the type of brainwave that corresponds to the brainwave detected in the detection step from among the types of brainwaves included in the first table. A recognition step is performed to recognize the operation recalled by the user by referring to the first table based on the type of brainwave determined by the discrimination step, An execution step which performs the operation recognized by the recognition step, The process includes a limiting step which limits the types of operations to be recognized in the recognition step from among the types of operations included in the first table, according to the type of software or input mode being used in the information processing terminal, The recall operation recognition method is a method in which the discrimination step determines the type of brainwave that corresponds to the detected brainwave among the types of brainwaves that correspond to the types of operations limited by the limitation step.

[0199] [Note 2] In the recall operation recognition method described in Appendix 1, A recall operation recognition method comprising a learning step of receiving an evaluation by the user for the recognized operation, and adjusting the waveform of the type of brainwave corresponding to the type of operation included in the first table based on the operation subject to evaluation and the brainwave used to recognize the operation.

[0200] [Note 3] In the recall operation recognition method described in Appendix 1, The recall operation recognition method, in the execution step, performs an operation based on or originating from a pointer displayed on the screen of the information processing terminal.

[0201] [Note 4] In the recall operation recognition method described in Appendix 3, A recall operation recognition method comprising a learning process that outputs an operation instruction to the user for each of several types of pointers that are sequentially displayed on the screen, receives an evaluation from the user for the recognized operation, and ranks the several types of pointers based on the evaluation.

[0202] [Note 5] In the recall operation recognition method described in Appendix 1, The software includes a game software and a method for recognizing recall operations.

[0203] [Note 6] In the recall operation recognition method described in Appendix 1, The input mode includes a text input mode, and the method is for recognizing recall operations.

[0204] [Note 7] In the recall operation recognition method described in Appendix 1, The recall operation recognition method wherein the information processing terminal is a smartphone, tablet device, or laptop computer with a touch panel. [Explanation of symbols]

[0205] 1...Information processing system, 2...Headset (EEG detection device), 3...Smartphone (Information processing terminal), 9...User, 9w...Electrical wave, 24...Electrode device for EEG detection, 200...Communication device (First communication device), 300...Communication device (Second communication device), 201...Detection device (Detection device), 202 Operation type list application device (First processing device), 203...Electrical wave type discrimination device (First processing device), 204...Recall operation recognition device, 205...Learning device (Learning device), 206...First control device (First processing device), 210...First memory device (First memory device), 211...First dictionary (Conversion dictionary, First table), 301...Operation type list selection device (Second 301...Processing device), 302...Software usage detection device (second processing device), 303...Input mode detection device (second processing device), 304...Recall operation reception device (second processing device), 305...Touch operation reception device (second processing device), 306...Voice operation reception device (second processing device), 307...Operation reception control device (second processing device), 308...Second control device (second processing device), 310...Second storage device (second storage device), 311...Operating system (software), 312...First application (software), 313...Second application (software), 315...Second dictionary (second table), 316...Headset control tool.

Claims

1. It is equipped with an electroencephalogram detection device and an information processing terminal. The electroencephalogram detection device is, A detection device that detects the user's brainwaves, A first database for converting the detected brainwave characteristics into data that can be processed by the information processing terminal, A first processing device that converts the brain waves detected by the detection device into data based on the first database, The system includes the aforementioned information processing terminal and a first communication device for sending and receiving data, The aforementioned information processing terminal is The electroencephalogram detection device and a second communication device that transmits and receives data, A second processing device that performs corresponding processing based on data received by the second communication device, It has a second database that stores a set of data related to instruction input used in the aforementioned information processing terminal, The first processing device stores the data group received via the first communication device in the first database, and performs a conversion process to associate it with the input instruction using the data group. The first processing device, based on the received data group, limits the data to be used as data to be converted according to the instruction input. Information processing system.

2. It is equipped with an electroencephalogram detection device and an information processing terminal. The electroencephalogram detection device is, A detection device that detects the user's brainwaves, A first database for converting the detected brainwave characteristics into data that can be processed by the information processing terminal, A first processing device that converts the brain waves detected by the detection device into data based on the first database, The system includes the aforementioned information processing terminal and a first communication device for sending and receiving data, The aforementioned information processing terminal is The electroencephalogram detection device and a second communication device that transmits and receives data, A second processing device that performs corresponding processing based on data received by the second communication device, The system includes an input mode identification information that identifies the input mode used by the information processing terminal, and a third database that stores the correspondence between the data set related to the instruction input used in that input mode and each type of input mode. The first processing device stores the data group received via the first communication device in the first database, and performs a conversion process using the data group to associate it with each type of input mode. The first processing device, based on the received data group, limits the data to be used as data to be converted according to the type of input mode. Information processing system.

3. In the information processing system described in claim 1, The aforementioned information processing terminal is The system includes a third database that stores a third table, each of which specific information that identifies the input mode used by the information processing terminal and a group of data related to instruction inputs used in that input mode are associated for each type of input mode. The second processing device refers to the third table to identify the data group corresponding to the input mode currently in use at the information processing terminal, and transmits the identified data group to the electroencephalogram detection device. The first processing device limits the data to be converted based on the received data set. Information processing system.

4. In the information processing system according to claim 1 or claim 2, The learning device receives an evaluation from the user of the converted data and adjusts the waveform characteristics of the brainwaves corresponding to the data in the first database based on the data subject to evaluation and the brainwaves used to convert the data. Information processing system.

5. In the information processing system according to claim 1 or claim 2, The second processing device executes an operation based on or originating from a pointer displayed on the screen of the information processing terminal as the corresponding process. Information processing system.

6. In the information processing system described in claim 5, The learning device includes an operation instruction for the user for each of the multiple types of pointers that are sequentially displayed on the screen, accepts the user's evaluation of the converted data, and ranks the multiple types of pointers based on the evaluation. Information processing system.

7. In the information processing system according to claim 2 or claim 3, The aforementioned input mode includes a text input mode. Information processing system.

8. In the information processing system according to claim 1 or claim 2, The aforementioned information processing terminal is a smartphone, tablet device, or laptop computer with a touchscreen. Information processing system.

Images