Program, image generation support device, image generation support system, and image generation support method

Abstract

The goal is to streamline the generation of images with arbitrary aspect ratios. [Solution] A program that causes a computer to function as a first acquisition unit (control unit 21) for acquiring reference image data, and a prompt generation unit (control unit 21) for generating prompts that cause an AI to generate an image based on the aspect ratio of the reference image data.

Description

Technical Field

[0001] The present disclosure relates to a program, an image generation support apparatus, an image generation support system, and an image generation support method.

Background Art

[0002] Conventionally, designs for various media such as posters, catalogs, presentation data, postcards, flyers, and advertisements have been created. In the creation of a design, first, a salesperson of a printing company draws out the client's requests while considering the aspect ratio adapted to the medium, creates a draft (rough image), and passes the draft to a designer. Thereafter, the designer completes a formal image based on the draft. In such a process, there is a problem that it takes a great deal of time for a salesperson of a printing company to create a desired draft from a blank state. In addition, drafts created by salespersons of printing companies are not always preferable in terms of design.

[0003] On the other hand, in recent years, it is known to generate images from text data using generative AI (Artificial Intelligence). Further, as described in Patent Document 1, it is known to resize and clip an image to a size and aspect ratio corresponding to a template for each medium.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] For salespeople who are not familiar with using generation AI, it is difficult to generate desired drafts using only text data. In particular, for ordinary users, it is difficult and cumbersome to specify an arbitrary aspect ratio to the generation AI. Furthermore, since Patent Document 1 does not generate new images, it cannot solve these problems.

[0006] Therefore, the objective of this disclosure is to improve the efficiency of image generation at any aspect ratio. [Means for solving the problem]

[0007] To solve the aforementioned problems, the program relating to this disclosure is: Computers First acquisition unit for acquiring reference image data, A prompt generation unit that generates prompts for the AI ​​to generate an image based on the aspect ratio of the aforementioned reference image data. To make it function as such.

[0008] Furthermore, the image generation support device related to this disclosure is A first acquisition unit for acquiring reference image data, A prompt generation unit generates a prompt to cause the AI ​​to generate an image based on the aspect ratio of the aforementioned reference image data, It is equipped with.

[0009] Furthermore, the image generation support system related to this disclosure is An image generation support system including an image generation support device, an image generation AI device, and a terminal device, The image generation support device is A first acquisition unit acquires reference image data from the aforementioned terminal device, The system includes a prompt generation unit that generates prompts for an AI to generate an image based on the aspect ratio of the aforementioned reference image data, The aforementioned image generation AI device is comprising an image generation unit that generates an image based on the prompt generated by the prompt generation unit, The terminal device comprises a display unit that displays the image generated by the image generation unit.

[0010] Further, an image generation support method according to the present disclosure is an image generation support method executed by an image generation support system, the method comprising: a first acquisition step of acquiring reference image data; a prompt generation step of generating a prompt that causes a generative AI to generate an image based on the aspect ratio of the reference image data; an image generation step of generating an image based on the generated prompt; . [Advantageous Effects of Invention]

[0011] According to the present disclosure, efficiency of image generation at an arbitrary aspect ratio can be improved. [Brief Description of Drawings]

[0012] [Figure 1] This is a block diagram illustrating the configuration of an image generation support system. [Figure 2] This is a flow diagram of image generation support processing. [Figure 3] This is a schematic diagram of an image generation support screen. [Figure 4] This is a schematic diagram of a reference image and a generated image. [Figure 5] This is a schematic diagram of a reference image and a generated image. [Mode for Carrying Out the Invention]

[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the scope of the present disclosure is not limited to what is described in the following embodiments and drawings.

[0014] <Image Generation Support System 100> First, the configuration of the image generation support system 100 will be explained using Figure 1. As shown in Figure 1, the image generation support system 100 comprises a terminal device 1, a cloud server 2, and an external device 3. The terminal device 1, the cloud server 2, and the external device 3 send and receive information via the communication network 1000.

[0015] Terminal device 1 is a device that supports image generation by users such as sales representatives of printing companies and their customers. Specifically, terminal device 1 displays a generated image created by cloud server 2 based on text data (first text data) entered by the user, reference image data, and fidelity, and accepts various operations on the generated image from the user. Here, "reference image data" refers to image data used as a reference during image generation. "Fidelity" refers to the degree to which the reference image data is referenced during image generation. More specifically, if the layout and aspect ratio of the reference image data are emphasized, the fidelity will be high. Conversely, if the content of the text data entered (color tone, seasonal feel, national characteristics, depth, etc.) is emphasized, the fidelity will be low. As an indicator of fidelity, for example, one could select one of five levels, or a parameter value (0-100%) could be set.

[0016] Cloud Server 2 is an image generation AI device that uses text generation AI to generate second text data from first text data, reference image data, and fidelity, and uses image generation AI to generate images from the second text data. Here, the second text data is what is known as a prompt. Therefore, cloud server 2 also functions as an image generation support device. Cloud server 2 may be an on-premise device, and the image generation support device and the image generation AI device may be configured as separate devices.

[0017] External device 3 is a device that allows the designer to display generated images and other similar content.

[0018] Communication networks 1000 include LANs (Local Area Networks), WANs (Wide Area Networks), and the Internet.

[0019] <Terminal device 1> Next, the configuration of terminal device 1 will be explained using Figure 1. As shown in Figure 1, terminal device 1 comprises a control unit 11, a display unit 12, an operation unit 13, a communication unit 14, and a storage unit 15.

[0020] The control unit 11 is composed of a CPU (Central Processing Unit), RAM (Random Access Memory), etc. The CPU of the control unit 11 reads various programs stored in the memory unit 15, loads them into the RAM, executes various processes according to the loaded programs, and controls the operation of each part of the terminal device 1.

[0021] The display unit 12 is composed of a monitor such as an LCD (Liquid Crystal Display) and displays various screens, etc., according to the instructions of the display signals input from the control unit 11.

[0022] The operation unit 13 includes a keyboard equipped with cursor keys, number input keys, various function keys, a pointing device such as a mouse, and a touch panel laminated on the surface of the display unit 12. The operation unit 13 is configured to be operated by the operator. The operation unit 13 also outputs various signals to the control unit 11 based on the operations performed by the operator.

[0023] The communication unit 14 is capable of sending and receiving various signals and data with other devices connected via the communication network 1000.

[0024] The memory unit 15 is composed of non-volatile semiconductor memory or a hard disk, and stores various programs executed by the control unit 11, parameters necessary for program execution, and various data.

[0025] <Cloud Server 2> Next, the configuration of the cloud server 2 will be explained using Figure 1. As shown in Figure 1, the cloud server 2 comprises a control unit 21, a communication unit 22, and a storage unit 23.

[0026] The control unit 21 is composed of a CPU, RAM, etc. The CPU of the control unit 21 reads various programs stored in the memory unit 23, loads them into RAM, executes various processes according to the loaded programs, and controls the operation of each part of the cloud server 2. The control unit 21 generates second text data from first text data, reference image data, and fidelity using a text generation AI, and generates an image from the second text data using an image generation AI. Examples of image generation AIs include Stable Diffusion, DALL·E2, Midjourney, starryai, and Dream by WOMBO. Examples of text generation AIs include ChatGPT, Bard, Gemini, and Bloom.

[0027] The control unit 21 functions as a first acquisition unit that acquires reference image data.

[0028] The control unit 21 functions as a prompt generation unit that generates second text data, which serves as a prompt for the AI ​​to generate an image based on the aspect ratio of the reference image data. The control unit 21 may, as a prompt generation unit, generate second text data as a prompt to cause the image generation AI to generate an image based on the layout of the reference image data.

[0029] The control unit 21 functions as a second acquisition unit that acquires text data. The control unit 21 may, as a prompt generation unit, generate prompts that cause the image generation AI to generate an image based on the aspect ratio of the text data and reference image data.

[0030] The control unit 21 functions as a third acquisition unit that acquires fidelity. The control unit 21 may, as a prompt generation unit, generate prompts that cause the image generation AI to generate an image based on the text data, the aspect ratio of the reference image data, and the fidelity.

[0031] The communication unit 22 is capable of sending and receiving various signals and data with other devices connected via the communication network 1000.

[0032] The memory unit 23 is composed of non-volatile semiconductor memory or a hard disk, and stores various programs executed by the control unit 11, parameters necessary for program execution, and various data.

[0033] <Image generation support processing> Next, we will explain the image generation support process using the flow shown in Figure 2. Image generation support processing is a process that assists users in generating images based on first text data, reference image data, and fidelity settings entered by users such as sales representatives of printing companies and their customers. Although omitted in the flow shown in Figure 2, the generated image data is ultimately sent to the designer's terminal (external device 3). The image generation support screen D1 shown in Figure 3 is assumed to be initially displayed on the display unit 12.

[0034] Here, we will explain the image generation support screen D1 shown in Figure 3. Area A1 is where the user inputs the first text data, reference image data, and fidelity. In area A1, the first text data consists of the type of medium, product, target audience, design content, and style. In area A1, reference image data is uploaded by clicking the image upload button. In region A1, the fidelity is entered as a numerical value, for example. Area A2 is the area that displays the generated image data generated by cloud server 2. Button B1 is a button that causes the cloud server 2 to generate an image.

[0035] First, the user inputs the first text data using the operation unit 13. The control unit 11 acquires the first text data (step S1).

[0036] Next, the user inputs reference image data using the operation unit 13. The control unit 11 acquires the reference image data (step S2).

[0037] Next, the user inputs the fidelity value using the operation unit 13. The control unit 11 acquires the fidelity value (step S3).

[0038] Next, the user presses button B1 using the operation unit 13. The control unit 11 transmits the first text data, reference image data, and fidelity to the cloud server 2 via the communication unit 14 (step S4).

[0039] Next, the control unit 21 receives data via the communication unit 22 and acquires the first text data, reference image data, and fidelity (step S5; first acquisition step, second acquisition step, third acquisition step).

[0040] Next, the control unit 21 generates second text data based on the first text data, reference image data, and fidelity (step S6; prompt generation unit). At this time, the control unit 21 extracts the aspect ratio of the reference image data and includes the extracted aspect ratio as a prompt in the second text data. Furthermore, the control unit 21 includes in the second text data, as a prompt, which prompt to be used to generate the image, based on fidelity: the prompt generated from the first text data or the prompt generated from the reference image data. Furthermore, the control unit 21 may extract layout information from the reference image data and include the extracted layout information as a prompt in the second text data.

[0041] Next, the control unit 21 generates an image using the generation AI based on the second text data (step S7; image generation step). The generated image data is designated as the generated image data.

[0042] Next, the control unit 21 transmits the generated image data to the terminal device 1 via the communication unit 22 (step S8).

[0043] Next, the control unit 11 receives data via the communication unit 14 and acquires the generated image data (step S9).

[0044] Next, the control unit 11 displays the generated image on the display unit 12 (step S10) and terminates the image generation support process.

[0045] In this way, users do not need to set the aspect ratio in the prompt, making it easy to generate the desired image. Furthermore, users do not need to set in the prompt how faithfully the generated image should be based on the reference image data, making it easy to generate the desired image.

[0046] <Examples> Figures 4 and 5 illustrate examples of reference images and generated images. Figure 4 is an illustrative diagram showing how an image is generated based on the aspect ratio of the reference image data. The aspect ratio of the generated image will be approximately the same as that of the reference image. Figure 5 is an illustrative diagram showing how an image is generated based on the aspect ratio and layout of the reference image data. The aspect ratio and layout of the generated image will be approximately the same as those of the reference image. The reason the images above are nearly identical is because they were generated by a generative AI. Because the images are generated by a generative AI, they are not always identical, and some slight discrepancies exist.

[0047] <Other> The above image generation support process assumes that steps S1 to S3 are all executed, but this example is not limited to this case. For example, step S1 does not necessarily have to be performed. In this case, the second text data will be generated as a prompt based solely on the reference image data. Furthermore, for example, step S3 does not necessarily have to be executed. In this case, the fidelity will not be set to the prompt in steps S6 and S7.

[0048] In the above explanation, the image generation support process was described with terminal device 1 and cloud server 2 as separate devices. However, it is also possible to equip terminal device 1 with the various functions of cloud server 2 so that the image generation support process is executed on terminal device 1. In this case, the processing content of each step is the same as that of each step in the image generation support process, but the transmission and reception of data between terminal device 1 and cloud server 2 is unnecessary. For example, instead of the control unit 11 receiving data via network 1000, the control unit 11 acquires the data directly.

[0049] <Effects> As explained above, the program causes the computer to function as a first acquisition unit (control unit 21) that acquires reference image data, and a prompt generation unit (control unit 21) that generates prompts for the AI ​​to generate an image based on the aspect ratio of the reference image data. Therefore, it is possible to efficiently generate images with any aspect ratio.

[0050] Furthermore, the image generation support device (cloud server 2) includes a first acquisition unit (control unit 21) that acquires reference image data, and a prompt generation unit (control unit 21) that generates prompts for the image generation AI to generate an image based on the aspect ratio of the reference image data. Therefore, it is possible to efficiently generate images with any aspect ratio.

[0051] Furthermore, the image generation support system 100 is an image generation support system including an image generation support device (cloud server 2), an image generation AI device (cloud server 2), and a terminal device 1. The image generation support device includes a first acquisition unit (control unit 21) that acquires reference image data from the terminal device, and a prompt generation unit (control unit 21) that generates prompts for the image generation AI to generate an image based on the aspect ratio of the reference image data. The image generation AI device includes an image generation unit (control unit 21) that generates an image based on the prompts generated by the prompt generation unit, and the terminal device includes a display unit 12 that displays the image generated by the image generation unit. Therefore, it is possible to efficiently generate images with any aspect ratio.

[0052] Furthermore, the image generation support method is an image generation support method executed by the image generation support system 100, and includes a first acquisition step (step S5) of acquiring reference image data, a prompt generation step (step S6) of generating a prompt to cause the image generation AI to generate an image based on the aspect ratio of the reference image data, and an image generation step (step S7) of generating an image based on the generated prompt. Therefore, it is possible to efficiently generate images with any aspect ratio.

[0053] Although the present disclosure has been described in detail based on embodiments, it goes without saying that the present disclosure is not limited to the above embodiments and can be modified as appropriate without departing from its essence. For example, the above description disclosed examples in which hard disks or semiconductor non-volatile memory were used as computer-readable media for the program relating to this disclosure, but the disclosure is not limited to these examples. Portable recording media such as CD-ROMs can also be used as other computer-readable media.

[0054] Furthermore, the detailed configuration and operation of each device can be modified as appropriate, without departing from the spirit of the invention. [Explanation of Symbols]

[0055] 100 Image Generation Support System 1. Terminal device 11 Control Unit 12 Display section 13 Control section 14 Communications Department 15 Storage section 2. Cloud Server (Image Generation Support Device) 21 Control Unit (First Acquisition Unit, Image Generation Unit, Second Acquisition Unit, Third Acquisition Unit) 22 Communications Department 23 storage section 3 External device 1000 Communication Networks

Claims

1. Computers First acquisition unit for acquiring reference image data, A prompt generation unit generates a prompt to cause the AI ​​to generate an image based on the aspect ratio of the aforementioned reference image data. A program that makes it function as such.

2. The program according to claim 1, wherein the prompt generation unit generates a prompt to cause the AI ​​to generate an image based on the layout of the reference image data.

3. To further, Second acquisition unit for acquiring text data, To make it function as, The program according to claim 1, wherein the prompt generation unit generates a prompt that causes the AI ​​to generate an image based on the aspect ratio of the text data and the reference image data.

4. To further, Third acquisition unit for obtaining fidelity, To make it function as, The prompt generation unit generates a prompt that causes the AI ​​to generate an image based on the text data, the aspect ratio of the reference image data, and the fidelity. The program according to claim 3, wherein the fidelity is the degree to which the image is generated in accordance with the aspect ratio of the text data or the reference image data.

5. A first acquisition unit for acquiring reference image data, A prompt generation unit generates a prompt to cause the AI ​​to generate an image based on the aspect ratio of the aforementioned reference image data, An image generation support device equipped with the following features.

6. An image generation support system including an image generation support device, an image generation AI device, and a terminal device, The image generation support device is A first acquisition unit acquires reference image data from the aforementioned terminal device, The system includes a prompt generation unit that generates prompts for an AI to generate an image based on the aspect ratio of the aforementioned reference image data, The aforementioned image generation AI device is The system includes an image generation unit that generates an image based on a prompt generated by the prompt generation unit, The aforementioned terminal device is An image generation support system comprising a display unit that displays the image generated by the image generation unit.

7. An image generation support method performed by an image generation support system, The first acquisition step involves obtaining reference image data, A prompt generation step that generates a prompt to cause the AI ​​to generate an image based on the aspect ratio of the aforementioned reference image data, An image generation step that generates an image based on the generated prompt, Image generation support method including

Images