Information processing device, method, and program

Abstract

An information processing device according to one embodiment includes: a measurement unit that measures brain waves of a user who is imagining intended exercise in accordance with an exercise presented to a the user; an estimation unit that estimates the number of body parts operated by the exercise intended by the user by inputting the brain waves measured by the measurement unit into the model; a calculation unit that calculates discrepancy between the number of body parts estimated by the estimation unit and the number of body parts operated by the actual exercise of the user in accordance with the presented exercise; and an update unit that updates parameters of the model such that the discrepancy calculated by the calculation unit becomes small.

Description

Information processing device, method and program

[0001] FIELD Embodiments of the present invention relate to an information processing device, method, and program.

[0002] Based on electroencephalogram (EEG) signals measured from electrodes placed on the scalp of the subject user, the system estimates the user's intended movement image, and then controls the operation of the wheelchair or other equipment used by the user based on this estimation, thereby supporting the exercise of people who are unable to move their bodies as they wish.

[0003] As for motor imagery tasks, as disclosed in Non-Patent Document 1, for example, relatively many of them involve single body part movements such as right hand movements, left hand movements, both feet movements, or tongue movements.

[0004] Wei, Chun-Shu, Toshiaki Koike-Akino, and Ye Wang. "Spatial component-wise convolutional network (SCCNet) for motor-imagery EEG classification." 2019 9th International IEEE / EMBS Conference on Neural Engineering (NER). IEEE, 2019.

[0005] The method disclosed in Non-Patent Document 1 utilizes the fact that the brain activation areas differ between motor images in the EEG when the user is performing motor imagery for a single body part movement, and by capturing the difference in the brain activation areas between motor images from the EEG, it is possible to estimate which motor image the user is performing.

[0006] However, the differences in brain activation areas between motor imageries are minute. Furthermore, to accurately estimate the source of the electrical signals emitted from these minute differences in brain activation areas from EEG, which is measured as bioelectrical signals propagating through the user's skull, requires the precise placement of a high-density EEG sensor.

[0007] Furthermore, brain activation areas differ between users, and even for the same user, the electrode position may change due to misalignment of the EEG sensor installation location when the sensor is attached or detached, which limits the ability to accurately estimate motor imagery based on differences in brain activation areas.

[0008] In view of the above issues, it is necessary to estimate motor imagery from EEG in a way that does not require a large number of sensors to detect minute differences in brain activation areas and is robust to sensor position.

[0009] The present invention has been made in light of the above circumstances, and its purpose is to provide an information processing device, method, and program that are capable of appropriately estimating the user's intended exercise.

[0010] An information processing device according to one aspect of the present invention includes a measurement unit that measures the brain waves of a user imagining an intended movement that is in line with the movement presented to the user; an estimation unit that estimates the number of body parts that will be moved by the movement intended by the user by inputting the brain waves measured by the measurement unit into a model; a calculation unit that calculates the error between the number of body parts estimated by the estimation unit and the number of body parts that will be moved by the user's actual movement that is in line with the presented movement; and an update unit that updates the parameters of the model so that the error calculated by the calculation unit is reduced.

[0011] An information processing method according to one aspect of the present invention is a method performed by an information processing device, and includes: a measurement unit of the information processing device measuring the brain waves of the user who is imagining an intended movement in line with the movement presented to the user; an estimation unit of the information processing device inputting the brain waves measured by the measurement unit into a model to estimate the number of body parts that will be moved by the movement intended by the user; a calculation unit of the information processing device calculating an error between the number of body parts estimated by the estimation unit and the number of body parts that will be moved by the user's actual movement in line with the presented movement; and an update unit of the information processing device updating parameters of the model so as to reduce the error calculated by the calculation unit.

[0012] According to the present invention, it is possible to appropriately estimate the exercise intended by the user.

[0013] Fig. 1 is a diagram showing an application example of a model training device according to one embodiment of the present invention. Fig. 2 is a flowchart showing an example of a procedure for processing operations by the model training device. Fig. 3 is a flowchart showing an example of a procedure for processing operations by a brain activity measurement unit. Fig. 4 is a flowchart showing an example of a procedure for processing operations related to regeneration of a movement combination number estimator by a combination movement imagery selection unit. Fig. 5 is a block diagram showing an example of the hardware configuration of a model training device according to one embodiment of the present invention.

[0014] An embodiment of the present invention will now be described. Fig. 1 is a diagram showing an application example of a model training device according to one embodiment of the present invention. As shown in Fig. 1, a model training device 100, which is an information processing device according to one embodiment of the present invention, includes a motor imagery combination database (DB) 10, a brain activity measurement unit 20, a motor combination number estimation unit 30, an error calculation unit 40, a parameter update unit 50, and a combined motor imagery selection unit 60.

[0015] Considering that specific brain regions of a user are activated when the user imagines the movement of a single body part, it can be assumed that multiple specific brain regions of a user are activated when the user imagines the movement of multiple body parts.

[0016] If the number of movement combinations, which is the number of body parts that move when the user imagines a movement, increases, and more specific brain regions are activated, then the EEG patterns that arise from the user's motor imagery, such as event-related desynchronization and synchronization, should become stronger and more pronounced. By focusing on the strength of such EEG patterns, it is no longer necessary to detect minute differences in the brain regions activated by the user, and it is expected that the method will be robust to changes in electrode position.

[0017] In this embodiment, a presentation device such as a display is used to present the user with an exercise image, such as a hand movement, a left hand movement, or a both-foot movement, and the user then performs the presented exercise image in their mind.

[0018] In this embodiment, in the task of estimating motor imagery from a user's electroencephalogram, right hand movement, left hand movement, and both foot movement, which are often taken up, are described as one motor imagery each, but this is not limiting, and for example, both foot movement may be described as right foot movement and left foot movement. Furthermore, up to Max_M combined motor images are classified based on brain activity.

[0019] 2 is a flowchart showing an example of the processing procedure of the model training device. The brain activity measurement unit 20 acquires a motor image for each motor combination from the motor imagery combination DB 10. The number of combinations of the user's motor images acquired is increased, for example, from one combination, e.g., a right-hand or left-hand movement, to two combinations, e.g., a right-hand and left-hand movement, to three combinations, and the brain activity measurement unit 20 measures the user's EEG during each combination of motor imagery while the user is imagining simultaneously performing up to Max_M body part exercises (S10). That is, the brain activity measurement unit 20 acquires the user's brain waves while the user is imagining the intended exercises in accordance with the exercises presented to the user.

[0020] The exercise combination number estimation unit 30 estimates the number of exercise combinations intended by the user from the EEG measured by the brain activity measurement unit 20 (S20). That is, the exercise combination number estimation unit 30 estimates the number of body parts that will be moved by the exercises intended by the user by inputting the brain waves measured by the brain activity measurement unit 20 into a model. The error calculation unit 40 calculates the error between the number of exercise combinations estimated by the exercise combination number estimation unit 30 and the number of exercise combinations that the user actually imagines (S30). That is, the error calculation unit 40 calculates the error between the number of body parts estimated by the exercise combination number estimation unit 30 and the number of body parts that will be moved by the user's actual exercises that are in line with the exercises presented to the user.

[0021] The parameter update unit 50 updates the parameters of the model used to calculate the number of exercise combinations for the user based on the error calculated by the error calculation unit 40 so as to reduce this error (S40).

[0022] The combined movement image selection unit 60 selects a combination of movement images for each movement combination number so that the error calculated by the error calculation unit 40 is minimized (S50).

[0023] The exercise combination number estimation unit 30, whose model parameters have been appropriately updated as described above, estimates the exercise combination number for any EEG of the user, and the estimation result can be used as a command from the user to an external device.

[0024] Next, we will explain specific examples of the processing of each unit in the model training device 100. (Brain activity measurement unit) The brain activity measurement unit 20 acquires a motor image for each number of motor combinations from the motor imagery combination DB 10. For example, when the number of motor combinations M is 1, the acquired motor images are motor images of one body part, such as [left hand, right hand, both feet], i.e., three types of motor images are acquired as a unit of acquisition, where the motor image is a motor image of a single body part movement.

[0025] When the number of movement combinations M is 2, the acquired movement images are three types of movement images, each of which involves two body parts, such as [left hand and right hand, right hand and both feet, both feet and left hand].

[0026] When the number of movement combinations M=3, this movement imagery is one type of movement imagery, in which the movement imagery of three parts, such as [left hand, right hand, and both feet], is acquired as a unit.

[0027] It should be noted that the multiple-body movement imagery refers to the user imagining the movement of multiple body parts simultaneously. The body parts intended by the above movement imagery are just one example, and the movement of more specific body parts, such as the fingers or shoulders, may also be included. The brain activity measurement unit 20 uses an external device capable of measuring the user's brain activity to measure the user's EEG while the presented movement imagery is being performed in the user's brain.

[0028] 3 is a flowchart showing an example of the procedure of the processing operation by the brain activity measurement unit. The brain activity measurement unit 20 sets the number of exercise combinations M to M=1 (S10-1), and if the set number of exercise combinations M does not exceed Max_M (No in S10-2), the brain activity measurement unit 20 measures the EEG of the user N times while the user is performing in his or her mind exercise images corresponding to the set number of exercise combinations presented to the user (S10-3).

[0029] Then, the brain activity measurement unit 20 updates the set number of exercise combinations M by incrementing it by 1 (S10-4), returns to S10-2, and when the set number of exercise combinations M exceeds Max_M (Yes in S10-2), outputs the EEG measured up to this point to the exercise combination number estimation unit 30. With this output, a series of processes by the brain activity measurement unit 20 ends.

[0030] (Movement combination number estimation unit) The movement combination number estimation unit 30 inputs EEG measured at the time of each movement combination for each number of movement combinations (M=1, 2, ..., Max_M) by the brain activity measurement unit 20. Furthermore, when updated parameters of the model of the movement combination number estimator are output from the parameter update unit 50, the movement combination number estimation unit 30 inputs the updated parameters and updates the parameters of the model of the movement combination number estimator.

[0031] The movement combination number estimation unit 30 selects one representative movement image for each movement combination number, and generates a model of a movement combination number estimator that estimates the movement combination number using EEG up to each movement combination number (M = 1, 2, ..., Max_M).

[0032] For example, the exercise combination number estimation unit 30 selects the exercise image of "left hand" for the exercise combination number "1", selects the exercise image of "left hand and both hands" for the exercise combination number "2", and selects the exercise image of "left hand, right hand, and both feet" for the exercise combination number "3". This selection may be performed randomly.

[0033] The model of the movement combination number estimator in the movement combination number estimator 30 calculates statistical features such as amplitude features or variance from the EEG input from the brain activity measurement unit 20, and estimates the number of movement combinations from the features using a support vector machine consisting of multiple parameters. The movement combination number estimator model may also be generated using a deep learning approach. The movement combination number estimator 30 outputs the estimated result of the number of movement combinations for the EEG to the error calculation unit 40.

[0034] (Error Calculation Unit) The error calculation unit 40 receives the estimated number of exercise combinations from the exercise combination number estimation unit 30 and receives information on the actual number of exercise combinations of the user who received the exercise imagery from, for example, an external device. The error calculation unit 40 compares this estimated result with the actual number of exercise combinations of the user who received the exercise imagery and calculates the error. That is, the error calculation unit 40 calculates the error between the estimated number of body parts to be moved by the user who received the exercise imagery and the number of body parts actually moved by the user who received the exercise imagery. Note that the error calculation method may use cross entropy, which is commonly used in classification tasks, or another method. The error calculation unit 40 outputs the calculated error to the parameter update unit 50 and the combined exercise image selection unit 60.

[0035] (Parameter Update Unit) The parameter update unit 50 receives the error from the error calculation unit 40. The parameter update unit 50 calculates updated parameters for the model of the exercise combination number estimator so as to reduce this error, i.e., the error between the number of exercise combinations estimated by the model of the exercise combination number estimator in the exercise combination number estimation unit 30 and the actual number of exercise combinations. For example, the parameter update unit 50 calculates the updated parameters by subtracting a value obtained by differentiating the error value from the error calculation unit 40 with the parameters of the model of the exercise combination number estimator from the parameters of the model of the exercise combination number estimator. The parameter update unit 50 outputs the updated parameters to the exercise combination number estimation unit 30.

[0036] (Combined Motor Image Selection Unit) The combined motor image selection unit 60 inputs the error related to the number of motor combinations calculated by the error calculation unit 40. Here, when the calculated error is large, the combined motor image selection unit 60 determines that the motor image selected as a representative by the motor combination number estimation unit 30, such as "left hand" or "both feet and right hand," is inappropriate, and switches the representative motor image to another motor image to improve accuracy.

[0037] For example, suppose the average calculated error is 0.5 when the number of motor images is "1," the average calculated error is 0.2 when the number of motor images is "2," and the average calculated error is 0.3 when the number of motor images is "3."

[0038] 4 is a flowchart showing an example of a processing procedure for regenerating the number of movement combinations estimator by the combined movement image selection unit 60. The combined movement image selection unit 60 attempts to select the number of movement images that is the number of movement combinations with the largest error (S50-1), selecting "1" here. If all numbers of movement images have not yet been selected (No in S50-2), it determines that the current representative movement image, for example, "left hand," is inappropriate for this number of movement images.

[0039] The combined movement image selection unit 60 selects another movement image related to the selected number of movement images, for example, "right hand" and "both feet," regenerates a model of the movement combination number estimator based on the selected movement image, verifies the error related to the number of movement images estimated by this model, selects the movement image with the smallest error as a new representative movement image for the selected number of movement images (S50-3), regenerates a model of the movement combination number estimator based on the result of this selection (S50-4), and returns to S50-1.

[0040] Next, the combined motor image selection unit 60 selects a new representative motor image for each of the selected motor images (here, for each number of motor images other than "1") in the same procedure, and regenerates a model for the number of motor combinations estimator. Then, if there are no selectable motor images in S50-1, that is, if all motor images have been selected (Yes in S50-2), the combined motor image selection unit 60 outputs the regenerated model to the number of motor combinations estimator 30, and the series of processes ends. The motor combination number estimator 30 performs subsequent processes using the regenerated model.

[0041] 5 is a block diagram showing an example of the hardware configuration of a model training apparatus according to an embodiment of the present invention. In the example shown in FIG. 5, the model training apparatus 100 according to the embodiment is configured, for example, as a server computer or a personal computer, and includes a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to the hardware processor 111A via a bus 115.

[0042] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from a communication network. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0043] An input device 500 and an output device 600 attached to the model training apparatus 100 and used by a user or the like are connected to the input / output interface 113. The input / output interface 113 can receive operation data input by a user or the like through the input device 500, such as a keyboard, a touch panel, a touchpad, or a mouse, and can output and display output data to an output device 600, such as a display device using a liquid crystal or an organic electroluminescence (EL) display. The input device 500 and the output device 600 may be devices built into the model training apparatus 100, or may be input devices and output devices of other information terminals that can communicate with the model training apparatus 100 via a network.

[0044] The program memory 111B is a non-transitory tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a read only memory (ROM), and can store programs necessary to execute various control processes, etc., according to one embodiment.

[0045] The data memory 112 is a tangible storage medium that is, for example, a combination of the above-mentioned nonvolatile memory and a volatile memory such as RAM (Random Access Memory), and can be used to store various data or information acquired and created during various processing steps.

[0046] A model training device 100 according to an embodiment of the present invention can be configured as a data processing device having the units shown in FIG. 1 as software processing functional units.

[0047] The information storage unit used as a work memory or the like by each unit of model training device 100 can be configured using data memory 112 shown in Fig. 5. However, these configured storage areas are not essential components within model training device 100, and may be areas provided in, for example, an external storage medium such as a USB (Universal Serial Bus) memory, or a storage device such as a database server located in the cloud.

[0048] The processing function units in each of the above units can be realized by reading and executing a program stored in the program memory 111B by the hardware processor 111A. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0049] The methods described in each embodiment can be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, MO, etc.), or semiconductor memory (ROM, RAM, flash memory, etc.), and can also be distributed by transmitting it via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only execution programs but also tables or data structures) that the computer executes. The computer that realizes this device reads the program stored on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-mentioned processing by controlling the operation of this software means. The term "recording medium" as used herein is not limited to a storage medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory installed inside the computer or in a device connected via a network.

[0050] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0051] 100... Model training device 10... Motor imagery combination DB 20... Brain activity measurement unit 30... Motor combination number estimation unit 40... Error calculation unit 50... Parameter update unit 60... Combined motor imagery selection unit

Claims

1. An information processing device comprising: a measurement unit that measures the brain waves of a user who is imagining an intended movement in line with the movement presented to the user; an estimation unit that estimates the number of body parts that will be moved by the movement intended by the user by inputting the brain waves measured by the measurement unit into a model; a calculation unit that calculates the error between the number of body parts estimated by the estimation unit and the number of body parts that will be moved by the user's actual movement in line with the presented movement; and an update unit that updates the parameters of the model so that the error calculated by the calculation unit is reduced.

2. The information processing device described in claim 1, wherein the estimation unit selects a type of exercise to be presented, generates a model that estimates the number of body parts that will be moved by the selected exercise, and generates a new model that estimates the number of body parts that will be moved by a type of exercise different from the selected type of exercise so as to reduce the error calculated by the calculation unit.

3. A method performed by an information processing device, comprising: a measurement unit of the information processing device measuring the brain waves of the user who is imagining an intended movement in line with the movement presented to the user; an estimation unit of the information processing device inputting the brain waves measured by the measurement unit into a model to estimate the number of body parts that will be moved by the movement intended by the user; a calculation unit of the information processing device calculating the error between the number of body parts estimated by the estimation unit and the number of body parts that will be moved by the user's actual movement in line with the presented movement; and an update unit of the information processing device updating the parameters of the model so as to reduce the error calculated by the calculation unit.

4. An information processing program that causes a processor to function as each part of the information processing device according to claim 1 or 2.

Images