Generation apparatus, generation method, and generation program

Abstract

A generation apparatus (10) includes: a brain wave acquisition unit (11) that acquires brain wave data of a user; a conversion unit (12) that converts the brain wave data of the user acquired by the brain wave acquisition unit (11) into auxiliary information related to generation of an image, by using a first model that learns the relationship between the brain wave data of the user and the auxiliary information; a text input unit (13) that inputs text information related to a description of an image by the user; an image generation unit (14) that generates image data on the basis of the auxiliary information obtained by conversion by the conversion unit (12) and the text information; and an output control unit (15) that causes an output unit to output the image data generated by the image generation unit (14).

Description

Technical Field

[0001] The present invention relates to a generation device, a generation method, and a generation program.

Background Art

[0002] There is provided a system that automatically generates an image from input text information.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional method, even when the user inputs a text description of the image they want to generate, there is a problem that it is difficult to generate an image close to what the user envisioned in a single process.

[0005] The present invention has been made in view of the above, and an object thereof is to provide a generation device, a generation method, and a generation program that enable generation of an image closer to the image imagined by the user.

Means for Solving the Problems

[0006] To solve the above-mentioned problems and achieve the objective, the generation apparatus according to the present invention is characterized by comprising: an acquisition unit that acquires the user's brainwave data; a conversion unit that uses a first model that has learned the relationship between the user's brainwave data and auxiliary information related to image generation to convert the user's brainwave data acquired by the acquisition unit into the auxiliary information; an input unit that inputs text information relating to the user's description of the image; a generation unit that generates image data based on the auxiliary information converted by the conversion unit and the text information; and an output control unit that causes the image data generated by the generation unit to be output from an output unit. [Effects of the Invention]

[0007] According to the present invention, it is possible to generate images that are closer to the image imagined by the user. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows an example of the configuration of a generating apparatus according to an embodiment. [Figure 2] Figure 2 is a flowchart showing the processing procedure of the generation method in the embodiment. [Figure 3] Figure 3 shows an example of a computer in which a generation device is realized when a program is executed. [Modes for carrying out the invention]

[0009] Embodiments of the generation apparatus, generation method, and generation program according to the present invention will be described in detail below with reference to the drawings. Furthermore, the present invention is not limited to the embodiments described below.

[0010] [Embodiment] In this embodiment, when generating an image from text information, the summary image (layout, color, style, etc.) of the image converted from electroencephalogram data is used as constraint information, enabling the generation of an image that is closer to the image imagined by the user.

[0011] [Generation device] Figure 1 shows an example of the configuration of a generation device according to an embodiment. The generation device 10 is realized by loading a predetermined program into a computer, for example, which includes ROM (Read Only Memory), RAM (Random Access Memory), CPU (Central Processing Unit), etc., and having the CPU execute the predetermined program. The generation device 10 also has a communication interface for sending and receiving various information with other devices connected via a network or the like.

[0012] The generation device 10 includes an electroencephalogram acquisition unit 11 (acquisition unit), a conversion unit 12 (conversion unit), a text input unit 13 (input unit), an image generation unit 14 (generation unit), an output control unit 15, and an output unit 16.

[0013] The electroencephalogram (EEG) acquisition unit 11 acquires EEG data from a user who wishes to generate an image, and transmits the acquired EEG data to the conversion unit 12. For example, the EEG data is EEG time-series data.

[0014] The conversion unit 12 uses the first model 121 to convert the user's brainwave data acquired by the brainwave acquisition unit 11 into auxiliary information related to image generation. The conversion unit 12 transmits the converted auxiliary information to the image generation unit 14.

[0015] The auxiliary information includes layout features related to the image layout, color features related to the image's colors, and / or style features related to the image's rendering style.

[0016] The first model 121 takes the user's electroencephalogram (EEG) data as input and outputs auxiliary information. The first model 121 is a model that has been pre-trained on the relationship between the user's EEG data and the auxiliary information.

[0017] In the first model 121, for example, when viewing images with various layouts, the brain wave data of the user is acquired for each image. Then, the first model 121 is trained to convert from the brain wave data to layout feature amounts using the layout information of the image that the user actually viewed as teacher labels (see, for example, Reference 1). Reference 1: Pan Wang, Danlin Peng, Simiao Yu, Chao Wu, Peter Childs, Yike Guo and Ling Li, “Verifying Design through Generative Visualization of Neural Activities”, arXiv preprint arXiv:2103.15182, 2021.

[0018] Note that the brain wave data may be the acquired brain wave time series data, or may be data converted into a power spectrum, an instantaneous phase, or the like.

[0019] The text input unit 13 inputs text information regarding the description of an image by the user according to the operation of the user on the input device. The user inputs text information regarding the image to be output.

[0020] The image generation unit 14 generates image data based on the auxiliary information converted by the conversion unit 12 and the text information input from the text input unit 13. The image generation unit 14 uses the second model 141 to generate image data based on the auxiliary information and the text information. The image generation unit 14 inputs the text feature amounts obtained by feature conversion of the text information and the auxiliary information (for example, layout feature amounts) into the second model 141 and converts them into image data.

[0021] The second model 141 is a decoder that converts into image data when the text feature amounts obtained by feature conversion of the text information and the auxiliary information (for example, layout feature amounts) are input.

[0022] The second model 141 is a model trained to generate image data from the latent space. The second model 141 is a model that has learned the relationship between the auxiliary information, the text information, and the image imagined by the user.

[0023] The second model 141 is a model trained using text features, auxiliary information (e.g., layout features), and images as training data.

[0024] The output control unit 15 outputs the image data generated by the image generation unit 14 to the output unit 16 for output from the output unit 16.

[0025] The output unit 16 outputs the image data received from the image generation unit 14. The output unit 16 is, for example, a display that outputs an image or a terminal device having a display.

[0026] [Generation Process] Next, the processing procedure of the generation method executed by the generation device 10 of the embodiment will be described. FIG. 2 is a flowchart showing the processing procedure of the generation method in the embodiment.

[0027] As shown in FIG. 2, in the generation device 10, the electroencephalogram acquisition unit 11 acquires the electroencephalogram of the user who desires to generate an image (step S11), and the conversion unit 12 converts the electroencephalogram data of the user acquired by the electroencephalogram acquisition unit 11 into auxiliary information using the first model 121 (step S12).

[0028] The text input unit 13 inputs text information regarding the description of the image by the user (step S13). The image generation unit 14 generates image data based on the auxiliary information and the text information using the second model 141 (step S14).

[0029] The output control unit 15 causes the output unit 16 to output the image data received from the image generation unit 14 (step S15).

[0030] [Effects of the Embodiment] The generation device 10 according to the embodiment acquires the user's electroencephalogram (EEG) data and converts the user's EEG data into auxiliary information using a first model that has learned the relationship between the user's EEG data and auxiliary information related to image generation. The generation device 10 then receives text information regarding the user's description of the image, generates image data based on the auxiliary information and the text information, and outputs it.

[0031] Thus, the generation device 10 does not generate images from text information alone, but rather uses text information and auxiliary information, which is an outline image of the image converted from electroencephalogram data, as constraint information to generate images. Therefore, when generating images, the generation device 10 uses auxiliary information, which is an outline image of the image converted from electroencephalogram data, as constraint information in addition to text information, enabling the generation of images that are closer to the image imagined by the user.

[0032] The generation device 10 can smoothly generate an image that is closer to the image imagined by the user by applying, as auxiliary information, layout features related to the image layout, color features related to the image color, and / or style features related to the image drawing style, which are converted from electroencephalogram data.

[0033] The generation device 10 generates image data based on auxiliary information and text information using a second model that has been trained to generate image data from latent space, and can accurately generate images that are close to the image imagined by the user.

[0034] [Regarding the system configuration of the embodiment] Each component of the generation device 10 shown above is a functional concept and does not necessarily have to be physically configured as shown in the diagram. In other words, the specific forms of distribution and integration of the functions of the generation device 10 are not limited to those shown in the diagram, and all or part of it can be configured by functionally or physically distributing or integrating it in any unit according to various loads and usage conditions.

[0035] Furthermore, each process performed in the generation device 10 may be implemented entirely or partially by a CPU and a program that is analyzed and executed by the CPU. Alternatively, each process performed in the generation device 10 may be implemented as hardware using wired logic.

[0036] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters described above and illustrated may be changed as appropriate unless otherwise specified.

[0037] [program] Figure 3 shows an example of a computer in which the generation device 10 is realized when a program is executed. The computer 1000 has, for example, memory 1010 and CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0038] Memory 1010 includes ROM 1011 and RAM 1012. ROM 1011 stores, for example, a boot program such as the BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, the mouse 1110 and the keyboard 1120. The video adapter 1060 is connected to, for example, the display 1130.

[0039] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. That is, the program that defines each process of the generation device 10 is implemented as a program module 1093 in which code executable by the computer 1000 is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for performing the same processes as the functional configuration in the generation device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0040] Furthermore, the configuration data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes them.

[0041] Furthermore, the program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via a network interface 1070.

[0042] Although embodiments applying the invention made by the present inventors have been described above, the present invention is not limited by the descriptions and drawings that constitute part of the disclosure of the present invention in this embodiment. That is, all other embodiments, examples, and operational techniques made by those skilled in the art based on this embodiment are included in the scope of the present invention. [Explanation of Symbols]

[0043] 10 Generator 11. Electroencephalogram (EEG) acquisition unit 12 Conversion section 13 Text input section 14 Image generation unit 15 Output control unit 16 Output section 121 First Model 141 Second Model

Claims

1. An acquisition unit that acquires the user's brainwave data, A conversion unit that uses a first model that has learned the relationship between the user's electroencephalogram data and auxiliary information related to image generation to convert the user's electroencephalogram data acquired by the acquisition unit into the auxiliary information, An input unit for inputting text information regarding the image description provided by the user, A generation unit generates image data based on the auxiliary information and text information converted by the conversion unit, An output control unit that outputs the image data generated by the generation unit from the output unit, A generating apparatus characterized by having the following features.

2. The generation apparatus according to claim 1, characterized in that the auxiliary information is a layout feature quantity relating to the layout of the image, a color feature quantity relating to the color of the image, and / or a style feature quantity relating to the drawing style of the image.

3. The generation device according to claim 1, characterized in that the generation unit generates image data based on the auxiliary information and the text information using a second model that has been trained to generate image data from latent space.

4. The generation apparatus according to claim 3, characterized in that the second model is a model that has learned the relationship between the auxiliary information and the text information and the image.

5. A generation method performed by a generation device, The process of acquiring the user's brainwave data, Using a first model that has learned the relationship between the user's electroencephalogram (EEG) data and auxiliary information related to image generation, the process includes converting the user's EEG data acquired in the acquisition step into the auxiliary information, An input unit for inputting text information regarding the image description provided by the user, A step of generating image data based on the auxiliary information and text information converted in the conversion step, The process of outputting the image data generated in the above generation process from the output unit, A method for generating a product that includes the following:

6. A generation program for causing a computer to function as a generation device according to any one of claims 1 to 4.

Images