Information processing device, method, and program

Abstract

An information processing device according to one embodiment comprises: a measurement unit that measures brain waves of a user who is imagining in accordance with information presented to the user; a first acquisition unit that acquires the information presented to the user; a second acquisition unit that acquires identification information of the user, the acquired identification information indicating which user the presented information is for; an estimation unit that inputs the measured brain waves into a model to estimate information intended for the user, and inputs the measured brain waves into a model to estimate identification information of the user, the estimated identification information indicating which user the presented information is intended for; an error calculation unit that calculates an error for a combination of the information acquired by the first acquisition unit and the identification information of the user acquired by the second acquisition unit, and a combination of the estimated information and the estimated identification information of the user; and an updating unit that updates parameters of the model so that the calculated error is reduced.

Description

Information processing device, method and program

[0001] FIELD Embodiments of the present invention relate to an information processing device, method, and program.

[0002] Based on electroencephalogram (EEG) signals measured from electrodes placed on the user's scalp, a motor image can be estimated as the user's intended movement, and the operation of a device such as a wheelchair used by the user can be controlled based on this motor image, thereby supporting the movement of people who are unable to move their bodies as they wish. There is also a technology using a deep learning approach that estimates motor intentions based on EEG signal strength (see, for example, Non-Patent Document 1).

[0003] Wei, Chun-Shu, Toshiaki Koike-Akino, and Ye Wang. "Spatial component-wise convolutional network (SCCNet) for motor-imagery EEG classification." 2019 9th International IEEE / EMBS Conference on Neural Engineering (NER). IEEE, 2019.

[0004] However, even if multiple users perform similar exercises or imagine similar movements, the same brain regions are not necessarily activated in the same way for each user. Also, due to differences in scalp impedance or head shape, user characteristics, as indicated by the tendency of EEG features, differ from user to user.

[0005] As an approach that takes user characteristics into account, a fine-tuning approach has been proposed in which a target user-independent estimation model is constructed to classify motor images from the target user's EEG using a dataset in which EEG and motor images of users other than the target user are paired, and the parameters of the target user-independent model are trained using this dataset in which the target user's EEG and motor images are paired.

[0006] However, while EEG user characteristics vary from person to person, there is a possibility that certain users may have common features. If such common features could be incorporated into an estimation model, a generalized estimation model could be constructed across users. However, current fine-tuning approaches do not take into account which user's data is used when building the model, making it difficult to extract such common features.

[0007] The present invention has been made in light of the above circumstances, and an object of the present invention is to provide an information processing device, method, and program that are capable of performing generalized estimation.

[0008] An information processing device according to one aspect of the present invention includes a measurement unit that measures the electroencephalogram of a user who is imagining something in line with information presented to the user, a first acquisition unit that acquires the information presented to the user, a second acquisition unit that acquires identification information of the user that indicates to which user the presented information is intended, an estimation unit that inputs the electroencephalograms measured by the measurement unit into a model to estimate information intended by the user, and inputs the electroencephalograms measured by the measurement unit into a model to estimate identification information of the user that indicates to which user the estimated information is intended, an error calculation unit that calculates an error between a combination of the information acquired by the first acquisition unit and the identification information of the user acquired by the second acquisition unit, and a combination of the information estimated by the estimation unit and the identification information of the user estimated by the estimation unit, and an update unit that updates parameters of the model so as to reduce the error calculated by the error calculation unit.

[0009] An information processing method according to one aspect of the present invention is a method performed by an information processing device, the method including: measuring, by a measurement unit of the information processing device, electroencephalograms of a user who is imagining information that is to be presented to the user; acquiring, by a first acquisition unit of the information processing device, the information to be presented to the user; acquiring, by a second acquisition unit of the information processing device, identification information of the user that indicates to which user the presented information is intended; and inputting, by an estimation unit of the information processing device, the electroencephalograms measured by the measurement unit into a model, thereby estimating the information intended by the user; The method comprises inputting the measured electroencephalogram into a model to estimate the user's identification information, which indicates which user the estimated information is intended by; calculating, by an error calculation unit of the information processing device, an error between a combination of the information acquired by the first acquisition unit and the user's identification information acquired by the second acquisition unit, and a combination of the information estimated by the estimation unit and the user's identification information estimated by the estimation unit; and updating, by an update unit of the information processing device, parameters of the model so as to reduce the error calculated by the error calculation unit.

[0010] According to the present invention, generalized estimation can be performed.

[0011] Fig. 1 is a diagram showing an application example of an information processing device according to an embodiment of the present invention. Fig. 2 is a diagram explaining an example of a model for estimating motor imagery and user information from EEG. Fig. 3 is a flowchart showing an example of a processing operation procedure by the information processing device. Fig. 4 is a block diagram showing an example of the hardware configuration of an information processing device according to an embodiment of the present invention.

[0012] An embodiment of the present invention will be described below. In this embodiment, in order to have an estimation model that estimates a user's intended movement image from EEG take into consideration which user's EEG is inputted into the estimation model, a user ID (identifier) ​​is output as user information in addition to the movement image, thereby realizing the construction of a movement image estimation model that takes into consideration user characteristics that are different and common among users.

[0013] FIG. 1 is a diagram showing an application example of an information processing device according to an embodiment of the present invention. As shown in FIG. 1, the information processing device 100 according to this embodiment includes a data collection unit 10, a data combination unit 20, a motor imagery calculation unit 30, a user ID calculation unit 40, an error calculation unit 50, and a parameter update unit 60. The data collection unit 10 includes a brain activity measurement unit 11, a motor imagery acquisition unit 12, and a user ID acquisition unit 13. The motor imagery calculation unit 30 and the user ID calculation unit 40 can be realized, for example, by an estimation model that estimates motor imagery and user information from EEG. The functions of each unit will be described in detail below.

[0014] In this embodiment, the information processing device 100 acquires electroencephalograms, motor images, and a user ID, synchronizes the acquired data in a chronological order, calculates a motor image from the electroencephalogram data using an estimation model, calculates the user ID from the electroencephalogram data using the estimation model, calculates the error between the acquired motor image and the user ID from the calculated motor image and the user ID, and updates the parameters used in the estimation model to calculate the motor image and the user ID based on this error.

[0015] Regarding the point of view of storing information about the target user within the model, for example, the disclosure in "Matsumoto Kentaro, Shimada Kazutaka, Estimating Review Rating Values ​​Using User Information, Information Processing Society of Japan Kyushu Branch, Hinokuni Information Symposium 2018, (2018)" is related, but in this embodiment, there is no need to collect the characteristics of the target user in advance, and only observational data (EEG) is used as data to input into the model.

[0016] In this embodiment, the information processing device 100 uses a presentation device such as a display to present to the user movement images, such as right hand movements, left hand movements, and both foot movements, and the user then performs these presented movement images in their minds, i.e., they imagine the presented movement images or movements that are in line with the images.

[0017] It is also assumed that an internal memory (not shown) of the information processing device 100 stores in advance information indicating which user the information presented to the user belongs to, that is, a user ID.

[0018] 2 is a diagram illustrating an example of a model for estimating motor imagery and user information from EEG. The structure of the model for estimating motor imagery and user information from EEG is not particularly limited. In this embodiment, as shown in FIG. 2, for example, input data to the Vision Transformer disclosed in "Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929 (2020)" is converted from an image to EEG, and the motor image token and user ID token can be stored as features within the model for estimating motor imagery and user information from EEG.

[0019] Next, the functions of each unit of the information processing device 100 shown in Fig. 1 will be described. Fig. 3 is a flowchart showing an example of the procedure of processing operations by the information processing device. (Brain Activity Measurement Unit) The brain activity measurement unit 11 of the data collection unit 10 measures the EEG of the user while he or she is performing the exercise image presented as described above in his or her mind, using an external device capable of measuring the user's brain activity. The brain activity measurement unit 11 outputs the measured EEG to the data combination unit 20.

[0020] (Exercise Image Acquisition Unit) The exercise image acquisition unit 12 of the data collection unit 10 acquires the exercise image presented to the user. The exercise image acquisition unit 12 outputs the acquired exercise image to the data combination unit 20.

[0021] (User ID Acquisition Unit) The user ID acquisition unit 13 of the data collection unit 10 acquires a user ID stored in the internal memory of the information processing device 100. The user ID acquisition unit 13 outputs the acquired user ID to the data combination unit 20. As described above, each unit of the data collection unit 10 performs data collection processing (S10).

[0022] (Data Combining Unit) The data combining unit 20 receives the EEG output from the brain activity measuring unit 11 , the movement image output from the movement image acquiring unit 12 , and the user ID output from the user ID acquiring unit 13 .

[0023] The data combination unit 20 chronologically aligns the input data and combines them, and outputs the combined data to the movement image calculation unit 30 and user ID calculation unit 40 within the estimation model, and also outputs the combined data to the error calculation unit 50 (S20).

[0024] (Movement Image Calculation Unit) The movement image calculation unit 30 receives the combined data from the data combination unit 20 .

[0025] The motor image calculation unit 30 calculates features from the EEG signal of the input combined data, estimates the user's motor image based on the calculated features and the motor image token of the features within the estimation model, and outputs this estimation result to the error calculation unit 50 (S30).

[0026] (User ID Calculation Unit) The user ID calculation unit 40 receives the combined data from the data combination unit 20. The user ID calculation unit 40 calculates features from the EEG signals of the received combined data, estimates the user's user ID based on the calculated features and the user ID token of the features within the estimation model, and outputs the estimation result to the error calculation unit 50 (S40). The estimation result is information indicating which user intended the movement image estimated by the movement image calculation unit 30.

[0027] (Error Calculation Unit) The error calculation unit 50 receives the combined data output from the data combination unit 20. The error calculation unit 50 also receives the movement image estimated by the movement image calculation unit 30 and the user ID estimated by the user ID calculation unit 40.

[0028] The error calculation unit 50 calculates an error as a difference between the combination of the motor image and the user ID included in the combined data and the combination of the estimated motor image and the user ID (S50). The error calculation unit 50 outputs the error calculation result to the parameter update unit 60.

[0029] (Parameter Update Unit) The parameter update unit 60 receives the error output from the error calculation unit 50. The parameter update unit 60 updates the parameters of the estimation model related to the movement image calculation unit 30 and the parameters of the estimation model related to the user ID calculation unit 40 so as to reduce the calculated error (S60).

[0030] For example, the parameter update unit 60 updates the parameters of the estimation model related to the motor image calculation unit 30 by subtracting from the motor image calculation parameters the values ​​of the errors calculated as described above that are differentiated by the motor image calculation parameters, which are parameters of the estimation model related to the motor image calculation unit 30 and are held in the motor image calculation unit 30. Similarly, the parameter update unit 60 updates the parameters of the estimation model related to the user ID calculation unit 40 by subtracting from the user ID calculation parameters the values ​​of the errors calculated as described above that are differentiated by the user ID calculation parameters, which are parameters of the estimation model related to the user ID calculation unit 40 and are held in the user ID calculation unit 40.

[0031] In the embodiment described above, when constructing a model to estimate an exercise image, information regarding which user's data it is is included in the estimation model, making it possible to construct an exercise image estimation model that takes user characteristics into account.

[0032] As another example, the exercise image to be presented to the user may not only be displayed on a display, but may also be presented visually through audio presentation, vibration presentation, or a game, etc. Even with such a presentation method, it is possible to assume that there will be differences in the brain wave tendencies of users, and an instruction token indicating the type of instruction may be added, or other user attributes such as gender or age may be added as tokens.

[0033] 4 is a block diagram showing an example of the hardware configuration of an information processing device according to an embodiment of the present invention. In the example shown in FIG. 4, the information processing device 100 according to the embodiment is configured, for example, by a server computer or a personal computer, and has a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115.

[0034] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from a communication network. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0035] An input device 500 and an output device 600, which are attached to the information processing device 100 and used by a user or the like, are connected to the input / output interface 113. The input / output interface 113 can take in operation data input by a user or the like through the input device 500, such as a keyboard, a touch panel, or a touchpad, and can output and display output data to an output device 600, which includes a display device using a liquid crystal or an organic electroluminescence (EL) display or the like. The input device 500 and the output device 600 may be devices built into the information processing device 100, or may be input devices and output devices of other information terminals that can communicate with the information processing device 100 via a network.

[0036] The program memory 111B is a non-transitory tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a read only memory (ROM), and can store programs necessary to execute various control processes, etc., according to one embodiment.

[0037] The data memory 112 is a tangible storage medium that is, for example, a combination of the above-mentioned nonvolatile memory and a volatile memory such as RAM (Random Access Memory), and can be used to store various data or information acquired and created during various processes.

[0038] An information processing apparatus 100 according to one embodiment of the present invention can be configured as an information processing apparatus having the units shown in FIG. 1 as software-based processing function units.

[0039] The information storage unit used as a work memory or the like by each unit of the information processing device 100 can be configured by using the data memory 112 shown in Fig. 4. However, these configured storage areas are not essential components within the information processing device 100, and may be areas provided in, for example, an external storage medium such as a USB (Universal Serial Bus) memory, or a storage device such as a database server located in the cloud.

[0040] The processing function units in each of the above units can be realized by reading and executing a program stored in the program memory 111B by the hardware processor 111A. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0041] The methods described in each embodiment can be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, MO, etc.), or semiconductor memory (ROM, RAM, flash memory, etc.), and can also be distributed by transmitting it via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only execution programs but also tables or data structures) that the computer executes. The computer that realizes this device reads the program stored on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-mentioned processing by controlling the operation of this software means. The term "recording medium" as used herein is not limited to a storage medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory installed inside the computer or in a device connected via a network.

[0042] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0043] REFERENCE SIGNS LIST 100... Information processing device 10... Data collection unit 11... Brain activity measurement unit 12... Movement image acquisition unit 13... User ID acquisition unit 20... Data combination unit 30... Movement image calculation unit 40... User ID calculation unit 50... Error calculation unit 60... Parameter update unit

Claims

1. An information processing device comprising: a measurement unit that measures the brain waves of a user who is imagining something in line with information presented to the user; a first acquisition unit that acquires the information presented to the user; a second acquisition unit that acquires identification information of the user that indicates to which user the presented information is intended; an estimation unit that inputs the brain waves measured by the measurement unit into a model to estimate information intended by the user, and inputs the brain waves measured by the measurement unit into a model to estimate identification information of the user that indicates to which user the estimated information is intended; an error calculation unit that calculates an error between a combination of the information acquired by the first acquisition unit and the identification information of the user acquired by the second acquisition unit, and a combination of the information estimated by the estimation unit and the identification information of the user estimated by the estimation unit; and an update unit that updates parameters of the model so as to reduce the error calculated by the error calculation unit.

2. The information processing device of claim 1, wherein the information presented to the user is the user's motor image, the measurement unit measures the user's brain waves as the user imagines an exercise in accordance with the user's motor image presented to the user, the first acquisition unit acquires the user's motor image presented to the user, the second acquisition unit acquires the user's identification information indicating which user the presented motor image is for, the estimation unit estimates the motor image intended by the user by inputting the brain waves measured by the measurement unit into a model, and estimates the user's identification information indicating which user the estimated motor image is for by inputting the brain waves measured by the measurement unit into a model, and the error calculation unit calculates an error between a combination of the motor image acquired by the first acquisition unit and the user's identification information acquired by the second acquisition unit, and a combination of the motor image estimated by the estimation unit and the user's identification information estimated by the estimation unit.

3. A method performed by an information processing device, comprising: measuring, by a measurement unit of the information processing device, electroencephalograms of the user who is imagining something in line with information presented to the user; acquiring, by a first acquisition unit of the information processing device, the information presented to the user; acquiring, by a second acquisition unit of the information processing device, identification information of the user that indicates to which user the presented information is intended; inputting, by an estimation unit of the information processing device, the electroencephalograms measured by the measurement unit into a model to estimate information intended by the user, and inputting, by a model, the electroencephalograms measured by the measurement unit to estimate identification information of the user that indicates to which user the estimated information is intended; calculating, by an error calculation unit of the information processing device, an error between a combination of the information acquired by the first acquisition unit and the identification information of the user acquired by the second acquisition unit, and a combination of the information estimated by the estimation unit and the identification information of the user estimated by the estimation unit; updating, by an update unit of the information processing device, parameters of the model so as to reduce the error calculated by the error calculation unit; An information processing method comprising:

4. An information processing program that causes a processor to function as each part of the information processing device according to claim 1 or 2.

Images