Image generation method, image generation device, and image generation program

Abstract

In an image generation device (10), an acquisition unit (15a) acquires electroencephalogram information in at least one of the cases where the user perceives or imagines an image, and at least one of category information representing a classification category of the image perceived by the user or category information about the image imagined by the user. A generation unit (15b) uses the acquired electroencephalogram information and category information about each image to generate an image corresponding to the category information.

Description

[Technical Field]

[0001] The present invention relates to an image generation method, an image generation device, and an image generation program. [Background technology]

[0002] In recent years, advances in image generation technology have made it easier to generate high-quality images. Furthermore, in the field of research into decoding perception and imagination using brain information as input, progress is being made in the development of technologies to generate images seen and imagined content (see Non-Patent Documents 1 and 2). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Ryohei Fukuma, Takufumi Yanagisawa, Shinji Nishimoto, Hidenori Sugano, Kentaro Tamura, Shota Yamamoto, Yasushi Iimura, Yuya Fujita, Satoru Oshino, Naoki Tani, Naoko Koide Majima, Yukiyasu Kamitani and Haruhiko Kishima, “Voluntary control of semantic neural representations by imagery with conflicting visual stimulation”, Communications Biology 5, 214, 2022 [Non-patent document 2] Pan Wang, Danlin Peng, Simiao Yu, Chao Wu, Peter Childs, Yike Guo and Ling Li, “Verifying Design Through Generative Visualization of Neural Activities”, Design Computing and Cognition'20, 2022, pp 555-573 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional technology, it is difficult to generate both perceived and imagined images from brain waves. In other words, conventional technologies that use brain information are methods for generating either perceived or imagined images, and have not achieved a technology for decoding information that is being imagined while looking at something, nor a technology for decoding imagination in a situation where visual information is being input. This is because the mechanisms of perception and imagination have many similarities, making them difficult to separate.

[0005] Furthermore, with technology that decodes only either perception or imagination, it is difficult to determine whether the generated image is perception or imagination, and generating both perception and imagination images leads to an increase in the amount of calculations, resulting in high processing and learning costs.

[0006] The present invention has been made in view of the above, and has as its object to efficiently generate both perceived images and recalled images from electroencephalograms. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the object, the image generation method of the present invention is an image generation method executed by an image generation device, and is characterized by including: an acquisition step of acquiring electroencephalogram information at least when a user perceives or imagines an image, category information representing a classification category of the image perceived by the user, and at least one of category information of the image imagined by the user; and a generation step of generating an image corresponding to the category information using the acquired electroencephalogram information and the category information of each image. [Effects of the Invention]

[0008] According to the present invention, it is possible to efficiently generate both perceived images and imagined images from electroencephalograms. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram for explaining an overview of the image generating apparatus of this embodiment. [Figure 2] FIG. 2 is a schematic diagram illustrating the general configuration of the image generating apparatus of this embodiment. [Figure 3] FIG. 3 is a diagram for explaining the processing of the generation unit. [Figure 4] FIG. 4 is a diagram for explaining the processing of the learning unit. [Figure 5] FIG. 5 is a flowchart showing the procedure of the image generation process. [Figure 6] FIG. 6 is a diagram illustrating an example of a computer that executes an image generating program. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are designated by the same reference numerals.

[0011] [Image Generation Device Overview] Fig. 1 is a diagram for explaining an overview of the image generating device of this embodiment. As shown in Fig. 1, the image generating device acquires electroencephalogram information of a user (S1), and acquires category information indicating the classification category of each image for a perceived image perceived by the user and an imaginary image imagined by the user (S2).

[0012] Then, the image generating device generates images so as to reconstruct a perceived image and an imaginary image using the electroencephalogram information and category information of each image (S3).

[0013] Specifically, the image generating device trains a model using, as training data, electroencephalogram information and perceptual category information of the user when perceiving an image and electroencephalogram information and imagination category information when imagining an image.The image generating device then uses the trained model to generate an image from the electroencephalogram information, perceptual category information, and imagination category information.

[0014] Furthermore, when acquiring category information for each image (S2), the image generating device may estimate perceptual category information or imagined category information from electroencephalogram information using the trained category estimation model (S21). In this case, the image generating device trains the category estimation model using, as training data, the electroencephalogram information and perceptual category information when the user perceives an image and the electroencephalogram information and imagined category information when the user imagines an image.

[0015] The image generating device then presents the generated image to a user by outputting it to an output unit such as a display. In this way, the image generating device can easily generate both a perceptual image and an imaginary image from electroencephalograms by separately using the perceptual category information and the imaginary category information.

[0016] [Image generation device configuration] Fig. 2 is a schematic diagram illustrating the general configuration of an image generating device according to this embodiment. As illustrated in Fig. 2, an image generating device 10 according to this embodiment is realized by a general-purpose computer such as a personal computer, and includes an input unit 11, an output unit 12, a communication control unit 13, a storage unit 14, and a control unit 15.

[0017] The input unit 11 is realized using input devices such as a keyboard, a mouse, an electroencephalograph, etc., and inputs various instruction information such as a command to start processing to the control unit 15 in response to input operations by an operator. The output unit 12 is realized by a display device such as a liquid crystal display, a printing device such as a printer, etc. For example, the output unit 12 displays a facial image with an expression generated in an image generation process described below.

[0018] The communication control unit 13 is realized by a NIC (Network Interface Card) or the like, and controls communication between the control unit 15 and an external device via a telecommunication line such as a LAN (Local Area Network) or the Internet. For example, the communication control unit 13 controls communication between the control unit 15 and a management device or the like that manages various types of information.

[0019] The storage unit 14 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 14 stores in advance processing programs for operating the image generation device 10, data used during execution of the processing programs, and the like, or temporarily stores them each time processing is performed. The storage unit 14 may be configured to communicate with the control unit 15 via the communication control unit 13. In this embodiment, the storage unit 14 stores a model 14a, a category estimation model 14b, and the like used in the image generation process described below.

[0020] The control unit 15 is realized using a CPU (Central Processing Unit) or the like, and executes a processing program stored in a memory. As a result, the control unit 15 functions as an acquisition unit 15a, a generation unit 15b, a learning unit 15c, and a presentation unit 15d, as exemplified in FIG. 2, to perform image generation processing. Note that these functional units may be implemented individually or in part in different hardware. For example, the learning unit 15c may be implemented in hardware different from the other functional units. The control unit 15 may also include other functional units.

[0021] The acquiring unit 15a acquires electroencephalogram information at least when the user perceived or imagined an image, and at least one of category information indicating a classification category of the image perceived by the user and category information of the image imagined by the user. For example, the acquiring unit 15a acquires, via the input unit 11 or from a user terminal or the like via the communication control unit 13, electroencephalogram information at the time the user to be processed perceived or imagined an image, and at least one of perceptual category information of the perceived image perceived by the user or imaginary category information of the imaginary image imagined by the user.

[0022] Specifically, the acquiring unit 15a acquires time-series data of electroencephalogram and magnetoencephalogram information measured by an electroencephalograph such as a high-precision electroencephalograph that acquires electroencephalograms from the whole head or a simple electroencephalograph that uses a small number of electrodes, etc. Then, the acquiring unit 15a performs preprocessing on the time-series data to reduce noise contained therein and improve the signal-to-noise ratio, and treats the preprocessed data as electroencephalogram information.

[0023] For example, the acquisition unit 15a performs at least one of extracting information from a target frequency band or applying a denoising technique to remove components other than brain waves from the time-series data. For example, the acquisition unit 15a extracts information so as to include one or more of delta waves (1-4 Hz), theta waves (4-7 Hz), alpha waves (8-13 Hz), beta waves (14-30 Hz), and gamma waves (30 Hz or higher). Furthermore, independent component analysis is used as a denoising technique to remove biological noise other than brain waves. This allows the acquisition unit 15a to acquire brain wave information from which noise components other than brain waves have been removed.

[0024] The acquiring unit 15a also acquires category information representing a classification category of the perceived image or the imaginary image. For example, the acquiring unit 15a accepts input of language information representing the user's perception or imagination content by the user, and converts the information into perception category information or imagination category information.

[0025] Alternatively, the acquiring unit 15a may acquire category information using the trained category estimation model 14b. That is, the acquiring unit 15a may acquire category information estimated by the category estimation model 14b from the acquired electroencephalogram information using the category estimation model 14b trained using as training data electroencephalogram information at least when the user perceived or imagined an image and at least one of category information indicating the classification category of the image perceived by the user and category information of the image imagined by the user. In this case, the learning unit 15c, which will be described later, trains the category estimation model 14b.

[0026] The acquiring unit 15a may store the acquired electroencephalogram information and category information in the storage unit 14, or may transfer them to the following functional units without storing them in the storage unit 14.

[0027] The generating unit 15b uses the acquired electroencephalogram information and the category information of each image to generate an image corresponding to the category information.

[0028] Specifically, the generation unit 15b uses the model 14a trained using as learning data the electroencephalogram information at least when the user perceived or imagined an image, and the category information representing the classification category of the image perceived by the user and the category information of the image imagined by the user, to generate an image corresponding to the category information from the acquired electroencephalogram information and category information.

[0029] Here, FIG. 3 is a diagram for explaining the processing of the generation unit. As illustrated in FIG. 3, the model 14a specifically includes a feature conversion mechanism and a decoder. The feature conversion mechanism converts electroencephalogram information and perceptual category information into perceptual features, and converts electroencephalogram information and imaginary category information into imaginary features. The decoder converts perceptual features into perceptual images, and converts imaginary features into imaginary images. The decoder that converts perceptual features into perceptual images and the decoder that converts imaginary features into imaginary images may be the same decoder, or may be different decoders specialized for each. Using the same decoder reduces calculation costs. On the other hand, using different decoders improves the accuracy with which perceptual images and imaginary images are generated.

[0030] The learning unit 15c learns the model 14a. For example, the learning unit 15c learns the model 14a using previously acquired electroencephalogram information and correct perceptual category information and correct imaginative category information. The electroencephalogram information may be information converted into electroencephalogram feature quantities such as power spectral density and instantaneous phase. In this case, the generation unit 15b generates an image using the same electroencephalogram feature quantities as the electroencephalogram information.

[0031] Here, Fig. 4 is a diagram for explaining the processing of the learning unit. As exemplified in Fig. 4, the learning unit 15c acquires (a) electroencephalogram information when the user perceives an image, (b) electroencephalogram information when the user imagines an image, and (c) electroencephalogram information when the perceived image and the imagined image are different as learning data of electroencephalogram information. Furthermore, the learning unit 15c acquires perceptual category information of the perceived image or imagined category information of the imagined image of (a) to (c) as learning data of category information.

[0032] Then, the learning unit 15c learns the model 14a so as to minimize the difference between the generated perceived image and the correct image presented for perception, and the difference between the generated imaginary image and the correct image for imagination.

[0033] In this case, the learning unit 15c adjusts the parameters only in the feature conversion mechanism, and does not change the parameters of the decoder. Furthermore, by using a decoder that has been trained in advance to be able to generate images from a latent space, it is possible to reduce the learning cost.

[0034] The learning unit 15c may change the parameters of the decoder. In this case, the calculation cost increases because the number of parameters to be learned increases, but the accuracy of the generated image improves.

[0035] Furthermore, the learning unit 15c may train the category estimation model 14b as described above. In this case, the learning unit 15c acquires, as training data for the electroencephalogram information, (a) electroencephalogram information when the user perceives an image, (b) electroencephalogram information when the user imagines an image, and (c) electroencephalogram information when the perceived image and the imagined image differ, in the same manner as in the case of training the model 14a shown in FIG. In this case, the electroencephalogram information may also be information converted into electroencephalogram feature quantities such as power spectral density and instantaneous phase.

[0036] Furthermore, the learning unit 15c acquires the perceptual category information of the perceptual images (a) to (c) or the imaginary category information of the imaginary images as correct answer data. Then, the learning unit 15c trains a category estimation model 14b that estimates category information from electroencephalogram information. Here, the acquiring unit 15a acquires the estimated category information and supplies it to the generating unit 15b. This makes it possible to generate images more efficiently just by acquiring electroencephalogram information.

[0037] Returning to the description of Fig. 2, the presentation unit 15d presents the generated perceptual image and imaginary image to the user via the output unit 12.

[0038] [Image generation processing] Next, the image generation process by the image generation device 10 according to this embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the procedure of the image generation process. The flowchart in Fig. 5 starts, for example, when the user performs an operation input to instruct the start of the process.

[0039] First, the acquiring unit 15a acquires electroencephalogram information at least when the user perceives or imagines an image (step S1). The acquiring unit 15a also acquires category information at least either perceptual category information of the perceived image perceived by the user or imaginative category information of the imaginative image imagined by the user (step S2).

[0040] In this case, the acquisition unit 15a may acquire category information estimated by the category estimation model 14b from the acquired electroencephalogram information, using the category estimation model 14b trained using as training data electroencephalogram information at least when the user perceived or imagined an image and at least one of perceptual category information of the perceptual image perceived by the user and imaginary category information of the imaginary image imagined by the user.

[0041] Next, the generating unit 15b generates a perceptual image or an imaginary image corresponding to the category information using the acquired electroencephalogram information and at least one of the category information of the perceptual category information of the perceptual image and the imaginary category information of the imaginary image (step S3).

[0042] Specifically, the generation unit 15b uses the model 14a trained using as learning data brain wave information at least when the user perceived or imagined an image, and at least one of perceptual category information of the perceptual image perceived by the user and imaginary category information of the imaginary image imagined by the user, to generate a perceptual image or imaginary image corresponding to the category information from the acquired brain wave information and category information.

[0043] Then, the presentation unit 15d presents the generated perceptual image and imaginary image to the user via the output unit 12 (step S4), thereby completing a series of image generation processes.

[0044] [effect] As described above, in the image generating device 10 of this embodiment, the acquiring unit 15a acquires electroencephalogram information at least when the user perceives or imagines an image, and at least one of perceptual category information indicating a classification category of the perceived image perceived by the user and imaginary category information of the imaginary image imagined by the user. The generating unit 15b uses the acquired electroencephalogram information and the category information of each image to generate an image corresponding to the category information.

[0045] Specifically, the generation unit 15b uses the model 14a trained using as learning data brain wave information at least when the user perceived or imagined an image, and imaginary category information representing the classification category of the perceived image perceived by the user, and at least one of imaginary category information of the imaginary image imagined by the user, to generate an image corresponding to the category information from the acquired brain wave information and category information.

[0046] In this way, the image generating device 10 can easily generate both a perceptual image and an imaginary image from an electroencephalogram by separately using the perceptual category information and the imaginary category information.

[0047] Furthermore, the learning unit 15c learns the model 14a, which makes it possible to generate a perceived image and an imaginary image with high accuracy.

[0048] The acquiring unit 15a may acquire category information estimated by the category estimation model 14b from the acquired electroencephalogram information, using the category estimation model 14b trained using as training data electroencephalogram information at least when the user perceived or imagined an image and at least one of perceptual category information indicating a classification category of a perceived image perceived by the user and imaginary category information of an imaginary image imagined by the user. This makes it possible to generate an image corresponding to the category information more efficiently by simply acquiring electroencephalogram information.

[0049] In this case, the learning unit 15c learns the category estimation model 14b, which makes it possible to estimate category information with even higher accuracy.

[0050] [program] A program written in a computer-executable language may be created to execute the processes executed by the image generation device 10 according to the above embodiment. In one embodiment, the image generation device 10 can be implemented by installing an image generation program that executes the image generation process described above as package software or online software on a desired computer. For example, by executing the image generation program on an information processing device, the information processing device can function as the image generation device 10. The information processing device referred to here includes desktop and notebook personal computers. Other examples of information processing devices include mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants). The functions of the image generation device 10 may also be implemented on a cloud server.

[0051] 6 is a diagram showing an example of a computer that executes an image generation program. The computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0052] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1031. The disk drive interface 1040 is connected to a disk drive 1041. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1041. The serial port interface 1050 is connected to, for example, a mouse 1051 and a keyboard 1052. The video adapter 1060 is connected to, for example, a display 1061.

[0053] Here, the hard disk drive 1031 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. Each piece of information described in the above embodiment is stored in the hard disk drive 1031 or memory 1010, for example.

[0054] The image generation program is stored in the hard disk drive 1031 as, for example, a program module 1093 in which instructions to be executed by the computer 1000 are written. Specifically, the program module 1093 in which each process executed by the image generation device 10 described in the above embodiment is written is stored in the hard disk drive 1031.

[0055] Furthermore, data used for information processing by the image generation program is stored as program data 1094, for example, in the hard disk drive 1031. Then, the CPU 1020 reads the program module 1093 and the program data 1094 stored in the hard disk drive 1031 into the RAM 1012 as necessary, and executes each of the above-described procedures.

[0056] The program module 1093 and program data 1094 related to the image generation program are not limited to being stored in the hard disk drive 1031, but may be stored in a removable storage medium, for example, and read by the CPU 1020 via the disk drive 1041. Alternatively, the program module 1093 and program data 1094 related to the image generation program may be stored in another computer connected via a network such as a LAN or a WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070.

[0057] Although the present invention has been described above as an embodiment, the present invention is not limited to the description and drawings that form part of the disclosure of the present invention. In other words, other embodiments, examples, and operational techniques that can be made by those skilled in the art based on the present invention are all included in the scope of the present invention. [Explanation of symbols]

[0058] 10 Image generation device 11 Input section 12 Output section 13 Communication control section 14 Storage section 14a model 14b Categorical Estimation Model 15 Control Unit 15a Acquisition part 15b Generator 15c Learning Department 15d Presentation part

Claims

1. An image generation method executed by an image generation device, comprising: an acquisition step of acquiring electroencephalogram information at least when a user perceives or imagines an image, and category information indicating a classification category of the image perceived by the user, or category information of the image imagined by the user; a generating step of generating an image corresponding to the category information by using the acquired electroencephalogram information and the category information of each image; An image generating method comprising:

2. The image generation method according to claim 1, characterized in that the generation step uses a model trained using electroencephalogram information at least when the user perceived or imagined an image, and category information representing a classification category of the image perceived by the user, and category information of the image imagined by the user, as training data, to generate an image corresponding to the category information from the acquired electroencephalogram information and category information.

3. The image generating method according to claim 2 , further comprising a learning step of learning the model.

4. 2. The image generating method according to claim 1, wherein the acquiring step uses a category estimation model trained using electroencephalogram information at least when the user perceived or imagined an image and category information representing a classification category of the image perceived by the user and / or category information of the image imagined by the user as training data, and acquires category information estimated by the category estimation model from the acquired electroencephalogram information.

5. 5. The image generating method according to claim 4, further comprising a learning step of learning the category estimation model.

6. an acquisition unit that acquires electroencephalogram information at least when a user perceives or imagines an image, and category information indicating a classification category of the image perceived by the user, and category information of the image imagined by the user; a generating unit that generates an image corresponding to the category information by using the acquired electroencephalogram information and the category information of each image; An image generating device comprising:

7. An image generating program for causing a computer to execute the image generating method according to any one of claims 1 to 5.

Images