Conversion method, conversion device, and conversion program
The conversion method and device address the limitations of small sample data in biosignal analysis by converting brain wave information into feature space data, allowing for the generation of various images and texts, thus expanding the applicability of electroencephalogram data.
Patent Information
- Application Number
- PCT/JP2023/018334
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-16
- Publication Date
- 2024-11-21
AI Technical Summary
Conventional methods face challenges in generating models using brain wave information due to overlearning and limitations from small sample data in biosignal analysis, hindering the application of biological signals like electroencephalogram data.
A conversion method and device that acquire brain wave information, convert it into feature space data using a predetermined model learned from large-scale datasets, and apply this conversion to external generation models for image or text generation, expanding the expressible information and applicability of brain wave data.
Enables the generation of diverse images and texts using brain wave information with a small number of samples by converting brain wave data into latent space representations, thereby expanding the range of applications for electroencephalogram information.
Smart Images

Figure JP2023018334_21112024_PF_FP_ABST
Abstract
Description
Conversion method, conversion device, and conversion program
[0001] The present invention relates to a conversion method, a conversion device, and a conversion program.
[0002] In recent years, in the fields of natural language processing and image recognition, highly accurate models have been generated by training using large-scale data (see Non-Patent Document 1). On the other hand, in the field of biosignal analysis, large-scale data sets have not yet been established, so training from a small number of samples is the norm (see Non-Patent Document 2).
[0003] Alec Radford et al., “Learning Transferable Visual Models From Natural Language Supervision”, arXiv, 2021Uzair Shah et al., “The Role of Artificial Intelligence in Decoding Speech from EEG Signals: A Scoping Review”, Sensors, 2022
[0004] However, conventional technologies have had difficulty generating models using EEG information. For example, in biosignal analysis, learning is performed from a small amount of sample data, which can lead to problems such as overfitting and limitations on the content that can be expressed. This can be a major obstacle to considering services that use biosignals such as EEG information.
[0005] The present invention has been made in view of the above, and aims to expand the amount of information that can be expressed by electroencephalogram information and to expand the range of models to which electroencephalogram information can be applied.
[0006] In order to solve the above-mentioned problems and achieve the objectives, the conversion method of the present invention is a conversion method executed by a conversion device, and is characterized by including an acquisition step of acquiring electroencephalogram information, and a conversion step of converting the acquired electroencephalogram information into data in the feature space of a predetermined model.
[0007] According to the present invention, it is possible to expand the amount of information that can be expressed by electroencephalogram information and to expand the range of models to which electroencephalogram information can be applied.
[0008] FIG. 1 is a diagram for explaining an overview of a conversion device of this embodiment. FIG. 2 is a diagram for explaining an overview of a conversion device of this embodiment. FIG. 3 is a schematic diagram illustrating an example of a general configuration of a conversion device of this embodiment. FIG. 4 is a diagram for explaining processing by the conversion device. FIG. 5 is a flowchart showing a conversion processing procedure. FIG. 6 is a flowchart showing a conversion processing procedure. FIG. 7 is a diagram showing an example of a computer that executes a conversion program.
[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are designated by the same reference numerals.
[0010] 1 and 2 are diagrams for explaining an overview of the conversion device of this embodiment. As shown in Fig. 1, the conversion device acquires an electroencephalogram (EEG) from a subject (S1), converts the electroencephalogram information after denoising processing into a feature vector in a feature space of a model trained using an existing large-scale dataset (S2), and outputs the converted feature vector (S3).
[0011] Specifically, in step S2, the conversion device generates, through training, a conversion model 14b that outputs a feature vector when electroencephalogram information is input. At this time, as illustrated in FIG. 2 , the conversion device generates the conversion model 14b using a trained model 14a and its correct answer data. Here, the model 14a is an existing model trained to output feature vectors of modalities other than electroencephalograms, such as images and text, using a large-scale data set of images or text. That is, the conversion device trains the conversion model 14b so that it outputs a feature vector according to the feature space representation of the trained model 14a.
[0012] Then, in step S3, the feature vector output from the trained conversion model 14b is applied to an external generation model, such as an image generation model or a sentence generation model. In this way, the conversion device converts EEG information into the latent space representation of the model 14a trained using a large-scale dataset, thereby expanding the amount of information that can be expressed by the EEG information. Furthermore, by combining the converted latent space representation with various image and sentence generation models, it becomes possible to generate a variety of images and sentences. Therefore, it becomes possible to generate images, sentences, etc. using EEG information with a small number of samples.
[0013] [Configuration of the Conversion Device] Fig. 3 is a schematic diagram illustrating the general configuration of the conversion device of this embodiment. As illustrated in Fig. 3, the conversion device 10 of this embodiment is realized by a general-purpose computer such as a personal computer, and includes an input unit 11, an output unit 12, a communication control unit 13, a storage unit 14, and a control unit 15.
[0014] The input unit 11 is realized using input devices such as a keyboard, a mouse, an electroencephalograph, etc., and inputs various instruction information such as a command to start processing to the control unit 15 in response to input operations by an operator. The output unit 12 is realized by a storage device such as a hard disk, a display device such as a liquid crystal display, a printing device such as a printer, etc. For example, the output unit 12 outputs a feature vector generated by a conversion process described below.
[0015] The communication control unit 13 is realized by a NIC (Network Interface Card) or the like, and controls communication between the control unit 15 and external devices via telecommunication lines such as a LAN (Local Area Network) or the Internet. For example, the communication control unit 13 controls communication between the control unit 15 and a management device or the like that manages various types of information.
[0016] The storage unit 14 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 14 stores in advance the processing program that operates the conversion device 10, data used during execution of the processing program, and the like, or temporarily stores the data each time processing is performed. The storage unit 14 may be configured to communicate with the control unit 15 via the communication control unit 13. In this embodiment, the storage unit 14 stores a model 14a used in the conversion process described below, a conversion model 14b generated in the conversion process, and the like.
[0017] The control unit 15 is realized using a CPU (Central Processing Unit) or the like, and executes a processing program stored in memory. As a result, the control unit 15 functions as an acquisition unit 15a, a conversion unit 15b, and a presentation unit 15c, as illustrated in FIG. 3, to perform the conversion process. Note that these functional units may be implemented individually or in part on different hardware. For example, the presentation unit 15c may be implemented on hardware different from the other functional units. The control unit 15 may also include other functional units.
[0018] The acquisition unit 15a acquires brain wave information of a user to be processed via the input unit 11 or via the communication control unit 13 from a user terminal or the like, for example.
[0019] Specifically, the acquisition unit 15a acquires brain waves measured by an electroencephalograph such as a high-precision electroencephalograph that acquires brain waves from the whole head or a simple electroencephalograph that uses a small number of electrodes, etc. Then, the acquisition unit 15a performs a denoising process on the measurement data to generate brain wave information.
[0020] For example, the acquisition unit 15a extracts information on a target frequency band to remove components other than brain waves from the measurement data. For example, the acquisition unit 15a extracts information so as to include one or more of the following: delta waves of 1-4 Hz, theta waves of 4-7 Hz, alpha waves of 8-13 Hz, beta waves of 14-30 Hz, and gamma waves of 30 Hz or higher. Furthermore, independent component analysis is performed to remove biological noise other than brain waves. This allows the acquisition unit 15a to acquire brain wave information from which noise components other than brain waves, such as noise derived from the eyeballs and muscles, have been removed. The brain wave information after noise removal is not limited to a waveform format, and may be converted into a time-frequency format using a method such as Fourier transform or wavelet transform.
[0021] The converter 15b converts the acquired electroencephalogram information into a feature vector in the feature space of a predetermined model 14a. Specifically, the model 14a is a model trained using data other than electroencephalogram information. For example, the model 14a is a model trained using large-scale data such as image data or text data, and outputs a feature vector corresponding to the input data.
[0022] 2, the conversion unit 15b uses the electroencephalogram information and the correct answer data of the model 14a to train a conversion model 14b that converts the input electroencephalogram information into a feature vector in the feature space of the model 14a and outputs the converted feature vector. The conversion model 14b may use a machine learning method such as a convolutional neural network or a recurrent neural network, a linear regression model, or a time-direction attention mechanism.
[0023] The presentation unit 15c presents the feature vector converted by the conversion unit 15b. For example, the presentation unit 15c outputs the feature vector output when the user's electroencephalogram information is input to the trained conversion model 14b to a device having a generation model such as an external image generation model or a sentence generation model via the communication control unit 13. The output feature vector is applied to the generation model such as the external image generation model or the sentence generation model.
[0024] In this way, the feature vector converted by the conversion unit 15b is applied to an external generation model such as an image generation model or a sentence generation model. In this case, when the conversion device 10 learns the conversion model 14b that outputs the feature vector to be used in the generation model, the acquisition unit 15a acquires a small amount of electroencephalogram data and correct answer data from the subject in advance.
[0025] For example, when training a conversion model 14b that outputs a feature vector to be used in an image generation model, the subject gazes at or recalls an image, and the conversion unit 15b uses the pair of brain wave information and image at that time to train the conversion model 14b.
[0026] The correct answer data is, for example, the feature vector of the image presented in the feature space of model 14a learned from a large-scale dataset of images, and the features of the electroencephalogram information are used as training data, and a conversion model 14b, which is a converter that maps the electroencephalogram information into the feature space, is generated by learning.
[0027] Alternatively, the feature space may be an intermediate layer of VGG16, a trained model for a large-scale image dataset, or CLIP (Contrastive Language-Image Pre-training), which represents images and text in the same feature space. By using such feature vectors in an image generation model, it becomes possible to generate images with high accuracy. By using feature vectors generated from EEG information and an existing trained model 14a, it becomes possible to generate a variety of images from EEG data with a small number of samples.
[0028] Furthermore, when training the conversion model 14b that outputs feature vectors to be used in the sentence generation model, the subject silently reads the presented words, and the conversion unit 15b uses the pair of brain wave information and sentences (words) at that time to train the conversion model 14b.
[0029] The correct answer data is, for example, the feature vector of a sentence presented in the feature space of model 14a, which has been trained from a large-scale data set such as sentences, and the features of the EEG information are used as training data to generate conversion model 14b, which is a converter that maps the EEG information into the feature space of a word, through training.
[0030] Furthermore, GPT-3, a trained model of a large-scale natural language dataset, may be used as the feature space. By using such feature vectors in a sentence generation model, sentences can be generated with high accuracy. By using feature vectors generated from EEG information and an existing trained model 14a, it becomes possible to generate a variety of sentences even from EEG data with a small number of samples.
[0031] 4 is a diagram for explaining the processing of the conversion device. Fig. 4 illustrates an example in which CLIP, a model trained with large-scale image and text data and representing images and text in the same feature space, is used as the trained model 14a. First, the acquisition unit 15a acquires electroencephalograms (EEG) from the subject when the subject mentally recalls a presented image, and the conversion unit 15b creates a small number of sample data sets consisting of pairs of recalled images and electroencephalograms.
[0032] By utilizing the CLIP model, the conversion unit 15b uses data obtained by converting an imaginary image into a CLIP feature vector as correct data and the electroencephalogram data as training data to generate a conversion model 14b using a convolutional neural network through learning.
[0033] The presentation unit 15c outputs the CLIP feature vector, which is output when the user's electroencephalogram information is input to the trained conversion model 14b, to an external device having an existing image generation model. The external device generates an image similar to the image imagined by the user using the existing image generation model using the CLIP feature vector and the feature vector output from the conversion device 10.
[0034] [Conversion Processing] Next, the conversion processing by the conversion device 10 according to this embodiment will be described with reference to Fig. 5 and Fig. 6. Fig. 5 and Fig. 6 are flowcharts showing the procedure of the conversion processing. The conversion processing according to this embodiment includes a learning process and a generation process.
[0035] First, the learning process procedure is shown in Fig. 5. The flowchart in Fig. 5 starts, for example, when the user performs an operation input to instruct the start of the process.
[0036] First, the acquisition unit 15a acquires electroencephalogram information (step S1).
[0037] Next, the conversion unit 15b converts the acquired electroencephalogram information into feature vectors in the feature space of a predetermined model 14a (step S2). Specifically, the model 14a is a model trained using data other than electroencephalogram information. For example, the model 14a is a model trained using large-scale data such as image data or text data, and outputs feature vectors corresponding to the input data.
[0038] Then, the conversion unit 15b uses the electroencephalogram information and the correct answer data of the model 14a to train a conversion model 14b that converts the input electroencephalogram information into a feature vector in the feature space of the model 14a and outputs the converted feature vector. The conversion unit 15b stores the trained conversion model 14b in the storage unit 14. This completes the series of learning processes.
[0039] Next, the procedure of the generation process is shown in Fig. 6. The flowchart in Fig. 6 starts, for example, when the user performs an operation input to instruct the start of the process.
[0040] First, the acquisition unit 15a acquires the user's electroencephalogram information (step S11).
[0041] The presenting unit 15c then outputs the converted feature vector to an external device (step S12). Specifically, the converting unit 15b inputs the user's electroencephalogram information into the trained conversion model 14b and converts it into a feature vector. The presenting unit 15c then presents the converted feature vector so that it can be used in a generation model such as an image generation model or a sentence generation model. For example, the presenting unit 15c outputs the feature vector output when the user's electroencephalogram information is input into the trained conversion model 14b to an external device having a generation model such as an image generation model or a sentence generation model via the communication control unit 13 or the output unit 12.
[0042] The feature vector output here is applied to an external generation model such as an image generation model or a text generation model, thereby completing the series of generation processes.
[0043] [Effects] As described above, in the conversion device 10 of this embodiment, the acquisition unit 15a acquires electroencephalogram information. The conversion unit 15b converts the acquired electroencephalogram information into feature vectors in the feature space of a predetermined model 14a.
[0044] Specifically, the model 14a is a model trained using data other than electroencephalogram information. The conversion unit 15b uses the electroencephalogram information and the correct answer data of the model 14a to train a conversion model 14b that converts the input electroencephalogram information into a feature vector in the feature space of the model 14a and outputs the converted feature vector.
[0045] This makes it possible to convert EEG information with a small number of samples into feature vectors in the feature space of model 14a, which has been trained using existing large-scale data, thereby expanding the amount of information that can be expressed by EEG information and the range of models to which EEG information can be applied.
[0046] The presenting unit 15c presents the converted feature vector. This makes it possible to apply the converted feature vector to an external generation model such as an image generation model or a sentence generation model. This makes it possible to generate images and sentences using electroencephalogram information with a small number of samples.
[0047] [Program] A program written in a computer-executable language may be created to execute the processes performed by the conversion device 10 according to the above embodiment. In one embodiment, the conversion device 10 can be implemented by installing a conversion program that executes the above conversion process as package software or online software on a desired computer. For example, by executing the conversion program on an information processing device, the information processing device can function as the conversion device 10. The information processing device referred to here includes desktop and notebook personal computers. Other information processing devices also include mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone System) phones, as well as slate terminals such as PDAs (Personal Digital Assistants). The functions of the conversion device 10 may also be implemented on a cloud server.
[0048] 7 is a diagram showing an example of a computer that executes a conversion program. The computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.
[0049] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1031. The disk drive interface 1040 is connected to a disk drive 1041. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1041. The serial port interface 1050 is connected to a mouse 1051 and a keyboard 1052, for example. The video adapter 1060 is connected to a display 1061, for example.
[0050] Here, the hard disk drive 1031 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. The various pieces of information described in the above embodiments are stored in the hard disk drive 1031 or memory 1010, for example.
[0051] The conversion program is stored in the hard disk drive 1031 as a program module 1093 in which instructions to be executed by the computer 1000 are written. Specifically, the program module 1093 in which each process executed by the conversion device 10 described in the above embodiment is written is stored in the hard disk drive 1031.
[0052] Data used for information processing by the conversion program is stored as program data 1094, for example, in the hard disk drive 1031. Then, the CPU 1020 reads the program module 1093 and the program data 1094 stored in the hard disk drive 1031 into the RAM 1012 as necessary, and executes each of the above-described procedures.
[0053] The program module 1093 and program data 1094 related to the conversion program are not limited to being stored in the hard disk drive 1031, and may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1041. Alternatively, the program module 1093 and program data 1094 related to the conversion program may be stored in another computer connected via a network such as a LAN or a WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070.
[0054] Although the present invention has been described above as an embodiment, the present invention is not limited to the description and drawings that form part of the disclosure of the present invention. In other words, other embodiments, examples, and operational techniques that can be made by those skilled in the art based on the present invention are all included in the scope of the present invention.
[0055] REFERENCE SIGNS LIST 10 Conversion device 11 Input unit 12 Output unit 13 Communication control unit 14 Storage unit 14a Model 14b Conversion model 15 Control unit 15a Acquisition unit 15b Conversion unit 15c Presentation unit
Claims
1. A conversion method executed by a conversion device, comprising: an acquisition step of acquiring electroencephalogram information; and a conversion step of converting the acquired electroencephalogram information into a feature vector in a feature space of a predetermined model.
2. The method of claim 1, wherein the model is a model trained using data other than electroencephalogram information.
3. The conversion method according to claim 1, characterized in that the conversion step involves learning a conversion model that uses the EEG information and correct answer data of the model to convert the input EEG information into a feature vector in the feature space of the model and output the converted information.
4. The method according to claim 1, further comprising a presentation step of presenting the feature vector converted by the conversion step.
5. A conversion device comprising: an acquisition unit that acquires electroencephalogram information; and a conversion unit that converts the acquired electroencephalogram information into a feature vector in a feature space of a predetermined model.
6. A conversion program for causing a computer to execute the conversion method according to any one of claims 1 to 4.
