Program, image generation support system, image generation support device, and image generation support method

Abstract

The goal is to generate the image the user intended more quickly. [Solution] A program that causes a computer to perform the following steps: a text generation step (step S4) in which the AI ​​generates multiple different image generation text data from input data so that the degree of difference between multiple generated images generated by the image generation AI differs according to predetermined conditions; and an image generation step (step S5) in which the image generation AI generates multiple generated images from the multiple different image generation text data.

Description

Technical Field

[0001] The present invention relates to a program, an image generation support system, an image generation support device, and an image generation support method.

Background Art

[0002] Conventionally, when a salesperson at a printing company discusses a printed matter image with a customer, the customer often does not have a clear image of what they want to embody. Therefore, the salesperson has to draw out an image from a "nonexistent" situation. Therefore, the salesperson generates the image several times and asks the customer to confirm it. Thereby, the salesperson finds the direction of the design, and then passes the image to a designer to complete a formal image.

[0003] On the other hand, image generation AIs (Artificial Intelligence) such as Stable Diffusion that generate images by inputting text are known. And image generation support devices using such image generation AIs have been developed. For example, such an image generation support device can generate a plurality of different prompts with a single drawing instruction such as text input and generate a plurality of images with different directions.

[0004] Also, Patent Document 1 describes that a user selects an image similar to / negates a desired image from exemplary images, creates search conditions from their feature amounts, and performs a search.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] As described above, salespeople can now quickly grasp the customer's preferences by showing them multiple images with different orientations. However, if the degree of directional difference between the multiple images remains constant, salespeople may have difficulty generating the desired image. For example, as the salesperson and the customer develop a shared understanding of the customer's needs and preferences, and as the decision-making phase progresses, the images required by the image generation support device will no longer be images with completely different orientations, but rather images that are moderately different. Although Patent Document 1 assumes repeated searches by re-specifying the searched and extracted images, the method for creating the search conditions remains constant regardless of the number of repetitions. Therefore, the degree of difference between the searched and extracted images does not fundamentally change.

[0007] Therefore, the objective of the present invention is to generate the image intended by the user more quickly. [Means for solving the problem]

[0008] To solve the aforementioned problems, the program according to the present invention is: On the computer, A text generation step in which an AI generates multiple different image generation text data from input data, such that the degree of difference between multiple generated images generated by the image generation AI differs according to predetermined conditions. Image generation step: An image generation AI generates multiple generated images from the aforementioned multiple different text data for image generation, Make it run.

[0009] Furthermore, the image generation support system according to the present invention is A text generation unit that uses AI to generate multiple different image generation text data from input data, according to predetermined conditions, so that the degree of difference between multiple generated images generated by the image generation AI varies; An image generation unit that generates multiple generated images using an image generation AI from the aforementioned multiple different text data for image generation, It is equipped with.

[0010] Furthermore, the image generation support device according to the present invention is A text generation unit that uses AI to generate multiple different image generation text data from input data, according to predetermined conditions, so that the degree of difference between multiple generated images generated by the image generation AI varies; An image generation unit that generates multiple generated images using an image generation AI from the aforementioned multiple different text data for image generation, It is equipped with.

[0011] Furthermore, the image generation support method according to the present invention is An image generation support method performed by an image generation support system, A text generation step in which an AI generates multiple different image generation text data from input data, such that the degree of difference between multiple generated images generated by the image generation AI differs according to predetermined conditions. Image generation step: An image generation AI generates multiple generated images from the aforementioned multiple different text data for image generation, Includes. [Effects of the Invention]

[0012] According to the present invention, the user can generate the image they intended more quickly. [Brief explanation of the drawing]

[0013] [Figure 1] This is a block diagram showing the configuration of the image generation support system. [Figure 2] This is a flowchart of the image generation support process. [Figure 3] This is an illustrative diagram of the image generation support screen. [Figure 4] This is an illustrative diagram of the image generation support screen. [Figure 5] This is an illustrative diagram of the image generation support screen. [Figure 6] This is an illustrative diagram of the image generation support screen. [Figure 7]This is an image diagram of the image generation support screen. [Figure 8] This is an image diagram of the image generation support screen. [Figure 9] This is an image diagram of the image generation support screen. [Figure 10] This is an example of correspondence between the decision-making phase and the changeability of items of text data for image generation. [Figure 11] This is a flowchart of the image generation support process. [Figure 12] This is an image diagram of the history screen. [Figure 13] This is an image diagram of the difference degree between the decision-making phase and the generated image.

Mode for Carrying Out the Invention

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the scope of the present invention is not limited to those described in the following embodiments and drawings.

[0015] <Image Generation Support System 100> First, the configuration of the image generation support system 100 will be described using FIG. 1. As shown in FIG. 1, the image generation support system 100 includes an image generation support device 1, a cloud server 2, and an external device 3. The image generation support device 1, the cloud server 2, and the external device 3 transmit and receive information via a communication network N.

[0016] The image generation support device 1 is a device that supports image generation by users such as salespersons of printing companies and their customers. Specifically, the image generation support device 1 displays a generated image generated by the cloud server 2 based on text data and / or image data input by the user, and accepts various operations on the generated image by the user. Note that the text data and / or image data are used as input data.

[0017] Cloud Server 2 is an AI device that uses text generation AI to generate text data for image generation from input data, and uses image generation AI to generate images from the text data for image generation. The text data for image generation is what is known as a prompt. Note that Cloud Server 2 may be an on-premises device.

[0018] External device 3 is a device that displays generated images and other data generated by image generation support device 1.

[0019] Communication networks N include LANs (Local Area Networks), WANs (Wide Area Networks), and the Internet.

[0020] <Image generation support device 1> Next, the configuration of the image generation support device 1 will be explained using Figure 1. As shown in Figure 1, the image generation support device 1 includes a control unit 11, a display unit 12, an operation unit 13, a communication unit 14, and a storage unit 15.

[0021] The control unit 11 includes a CPU (Central Processing Unit), RAM (Random Access Memory), etc. The CPU of the control unit 11 reads various programs stored in the memory unit 15, loads them into the RAM, executes various processes according to the loaded programs, and controls the operation of each part of the image generation support device 1.

[0022] The display unit 12 is equipped with a monitor such as an LCD (Liquid Crystal Display) and displays various screens (such as the image generation support screen described later) according to the instructions of the display signals input from the control unit 11.

[0023] The operation unit 13 includes a keyboard equipped with cursor keys, number input keys, various function keys, a pointing device such as a mouse, and a touch panel laminated on the surface of the display unit 12. The operation unit 13 can be operated by the user. The operation unit 13 also outputs various signals to the control unit 11 based on the operations performed by the user.

[0024] The communication unit 14 is capable of sending and receiving various signals and data with other devices connected via the communication network N.

[0025] The memory unit 15 is equipped with non-volatile semiconductor memory or a hard disk, and stores various programs executed by the control unit 11, parameters necessary for program execution, and various data.

[0026] <Cloud Server 2> Next, the configuration of the cloud server 2 will be explained using Figure 1. As shown in Figure 1, the cloud server 2 comprises a control unit 21, a communication unit 22, and a storage unit 23.

[0027] The control unit 21 includes a CPU, RAM, etc. The CPU of the control unit 21 reads various programs stored in the memory unit 23, loads them into RAM, executes various processes according to the loaded programs, and controls the operation of each part of the cloud server 2.

[0028] The control unit 21 functions as a text generation unit that uses AI to generate multiple different image generation text data from input data, according to predetermined conditions, so that the degree of difference between multiple generated images generated by the image generation AI varies. The specified conditions include the decision-making phase. The decision-making phase refers to the degree of progress in establishing a shared understanding of customer needs and preferences between the salesperson and the customer. There should be two or more decision-making phases, and three or more is even better. For example, the decision-making phase corresponds to the information on the input screen where the user enters input data. For example, the information on the input screen is obtained from various screens operated by the user (such as the image generation support screen described later). Specifically, the information on the input screen includes numerical values, words (such as "initial / mid-stage / late-stage," "divergence / convergence," "exploration / narrowing," and "key image / details"), and symbols that correspond to the progress of decision-making or the order of the decision-making phases. For example, the information on the input screen is user operation information on various screens (such as the image generation support screen described later). Specifically, the information on the input screen is information such as what was changed or not changed among the user-configurable inputs on various screens. User-configurable inputs include, for example, setting parameters that represent the degree of difference (numerical values ​​or selections such as large / medium / small), whether specific options were changed or not (such as photo / illustration / watercolor in "Style"), and turning specific modes ON / OFF (such as "Fine-tuning mode").

[0029] The control unit 21 functions as an image generation unit that generates multiple generated images using an image generation AI from multiple different text data for image generation. If the control unit 21 determines that the decision-making phase is in the early stages, it is preferable to make the degree of difference between the generated images larger than if the decision-making phase is in the later stages. To vary the degree of difference between generated images includes varying the magnitude of the differences and / or varying the number of differences. Varying the magnitude of the differences means varying the degree of difference in a particular item, such as a large or small difference in color tone. Varying the number of differences means varying the number of differences among multiple items, such as differences in motif (cat, cow, etc.), color tone, and style (photographic, watercolor, etc.), or differences in color tone only. Examples of image generation AI include Stable Diffusion, DALL·E2, Midjourney, starryai, and Dream by WOMBO. Examples of text generation AI include ChatGPT, Bard, Gemini, and Bloom.

[0030] The communication unit 22 is capable of sending and receiving various signals and data with other devices connected via the communication network N.

[0031] The memory unit 23 is equipped with non-volatile semiconductor memory or a hard disk, and stores various programs executed by the control unit 21, parameters necessary for program execution, and various data.

[0032] <Image generation support processing> Next, we will explain the image generation support process using the flow shown in Figure 2. Image generation support processing is a process that assists users in generating images based on input data entered by users such as sales representatives of printing companies and their customers. Although omitted in the flow shown in Figure 2, the generated data is ultimately sent to the designer's terminal (external device 3).

[0033] First, the user uses the operation unit 13 to operate various screens displayed on the display unit 12 (such as the image generation support screen described later) and input data. The control unit 11 receives the input data. Then, the user uses the operation unit 13 to press the image generation button described later. The control unit 11 transmits the input data to the cloud server 2 via the network N (step S1).

[0034] Here, we will explain the image generation support screen using Figures 3 to 9. First, we will explain the image generation support screen using Figure 3. Figure 3 is an image diagram of the image generation support screen D1 that is initially displayed on the display unit 12. Area A1 is the area where the image entered (uploaded) by the user is displayed. Area A2 is the area where the user enters text data. Button B1 is a button for the user to input (upload) an image. Button B2 is used to generate an image (key image) via Cloud Server 2. The key image is the key image used for subsequent image generation, and subsequent image generation will be based on the key image. If the fine-tuning mode described later is selected, the degree of derivation will be narrower than usual; in other words, images that are closer to the key image and more similar to each other will be generated.

[0035] Next, we will explain the image generation support screen D2 shown in Figure 4. Area A3 is the area for displaying the generated images. If multiple generated images are generated, the selected image will be displayed larger in the upper row, and the other generated images will be displayed smaller and side-by-side in the lower row. The initial selection of the generated image will be the one on the left in the lower row. Area A4 is the area for editing the generated image. The content displayed in area A4 changes depending on which tab TB1 to TB4 the user selects. In Figure 4, the design elements tab TB1 is selected, and in area A4, the design elements are displayed vertically for each noun, with modifiers for each noun displayed horizontally to the right of each noun. Design elements are elements that represent the generated image, formed by combining nouns and their modifiers, and are extracted from the generated image generated by the control unit 21 using image generation AI. The contents of the other tabs will be explained later. Button B3 is a button that causes the cloud server 2 to regenerate the image. Button B8 is used to record the user's rating (OK (good), NG (bad)) of the generated image. Button B11 is used to generate an image by making fine adjustments (fine-tuning mode) based on the image selected in area A3, rather than generating the image from scratch when button B3 is pressed and the image is generated again. Figure 5 is an image of the image generation support screen D2 when button B11 is pressed, and the screen enters fine-tuning mode.

[0036] Next, we will explain the image generation support screen D2 shown in Figure 6. The image generation support screen D2 shown in Figure 6 has the layout tab TB2 selected. The layout refers to the layout applied to the image displayed in area A3, and includes data such as the generation area mask, display area mask, and background-transparent text. The generation area mask is a mask that indicates the area (shaded area) where you want the image to be generated on Cloud Server 2. The generation area mask is selected by the user from a dropdown menu. The contents of the dropdown menu include, for example, "None," "Circular," "Circular (Inverted)," "Circular Gradient," and "Gradient." A display area mask is a mask that indicates the area (shaded area) of an image that you want to display. The display area mask is selected by the user from a dropdown menu. The contents of the dropdown menu include, for example, "None," "Circular," "Circular (Inverted)," "Circular Gradient," and "Gradient." Transparent background text refers to an image where the background is transparent except for the text itself. For example, the text portion might be a catchphrase. The user can select the transparent background text option from a dropdown menu.

[0037] Next, we will explain the image generation support screen D2 shown in Figure 7. The image generation support screen D2 shown in Figure 7 has the Tone Manner tab TB4 selected. In the Tone Manner tab TB4, Style, Composition, and Hue can be selected from the dropdown menus. Style refers to the elements that characterize an image, such as color expression, texture, and illustration techniques. Composition refers to design elements such as the arrangement of elements within an image, the overall balance of the image, and the relative positions of objects and elements. Hue refers to the quality and characteristics of colors, including color scheme, color balance, and color contrast.

[0038] Next, we will explain the image generation support screen D1 shown in Figure 8. The image generation support screen D1 shown in Figure 8 is the same as the image generation support screen D1 shown in Figure 3, but with a screen name added.

[0039] Next, we will explain the image generation support screen D2 shown in Figure 9. The image generation support screen D2 shown in Figure 9 is the same as the image generation support screen D2 shown in Figure 4, but with the addition of decision-making phase information.

[0040] Next, the control unit 21 receives input data via the network N (step S2).

[0041] Next, the control unit 21 acquires a predetermined condition (step S3). For example, the control unit 21 determines the decision-making phase from the screen information of various screens operated by the user (such as the image generation support screen described later) and acquires it as a predetermined condition. Specifically, the control unit 21 acquires screen information of the image generation support screen D1 shown in Figure 3, and since the image generation support screen D1 shown in Figure 3 is the initial screen, it may determine that the decision-making phase is the initial phase. Also, since the image generation button is button B2 (key image generation), the control unit 21 may determine that the decision-making phase is the initial phase. Specifically, the control unit 21 may determine that the decision-making phase is in the medium term because it has acquired screen information from the image generation support screen D2 shown in Figure 4 and found that the design element tab TB1 is selected. Alternatively, the control unit 21 may also determine that the decision-making phase is in the medium term because the image generation button is button B3 (normal image generation). Specifically, the control unit 21 acquires screen information from the image generation support screen D2 shown in Figure 5, and since button B11 (fine-tuning mode) is selected, it may determine that the decision-making phase is in the later stages. Specifically, the control unit 21 acquires screen information from the image generation support screen D2 shown in Figure 6, and since the layout tab TB2 is selected, it may determine that the decision-making phase is in the medium term. Specifically, the control unit 21 acquires screen information from the image generation support screen D2 shown in Figure 7, and since the tone manner tab TB4 is selected, it may determine that the decision-making phase is in the later stage. Specifically, the control unit 21 may acquire screen information from the image generation support screen D1 shown in Figure 8 and determine the decision-making phase to be the initial phase based on the screen name. Specifically, the control unit 21 may acquire screen information from the image generation support screen D2 shown in Figure 9 and determine the decision-making phase to be the medium term based on the decision-making phase information.

[0042] Next, the control unit 21 uses the input data to generate multiple different text data for image generation using a text generation AI (step S4; text generation step). At this time, the control unit 21 generates multiple different text data for image generation so that the degree of difference between the multiple generated images differs according to predetermined conditions obtained in step S3. Here, we will explain the method for generating text data for image generation using Figure 10. Figure 10 shows an example of the correspondence between the decision-making phase and whether the items in the text data for image generation can be changed (fixed, changeable). When the decision-making phase is in its early stages, the desired degree of difference between the multiple generated images produced in step S5 (described later) is to be large; therefore, the predetermined condition is one in which all items can be changed. If the decision-making phase is medium-term, we want the degree of difference between the multiple generated images produced in step S5 (described later) to be moderate. Therefore, the predetermined conditions are to fix the motif and style items, while allowing the other items to be changed. If the decision-making phase is in the later stages, in order to minimize the degree of difference between the multiple generated images produced in step S5 described later, the predetermined conditions are to fix the items of motif, style, color, and composition, and allow only the description of the motif to be changed. In other words, it's best to fix the settings in order of their impact on the generated image, starting with those that have the greatest impact.

[0043] Furthermore, as shown in Figure 10, instead of simply "change / fix," you can also change the degree of change. Furthermore, if an item has multiple elements, the number of elements to be changed may be adjusted. For example, if the item "motif" has five elements, the number of elements to be changed could be set to five if the decision-making phase is early, and to one if the decision-making phase is late. Furthermore, the SEED value of the random numbers that the control unit 21 provides to the image generation AI is not fixed. For example, the SEED value of the random numbers is a value that, when changed, allows the image generation AI to generate multiple images that are similar but different. In the later stages of the decision-making phase, one example is to change only the SEED value so that the degree of difference becomes smaller.

[0044] Next, the control unit 21 generates images using the image generation AI with the text data for image generation (step S5; image generation step). The control unit 21 generates as many images as there are text data for image generation.

[0045] Next, the control unit 21 transmits the data generated in step S6 (such as the generated image) to the image generation support device 1 via the network N (step S6).

[0046] Next, the control unit 11 receives data (such as generated images) via the network N (step S7).

[0047] Next, the control unit 11 displays the data received via the network N on the display unit 12 (step S8). For example, the control unit 11 updates the image generation support screen D2 and displays it on the display unit 12.

[0048] <Other> If a reference image exists, the control unit 21 may change the image generation method in step S5. The control unit 21 shall generate a similar generated image based on the reference image. If the decision-making phase is in its early stages, the control unit 21 may analyze the reference image, translate it into words, and generate a generated image using the "text to image" function of the image generation AI without using the image itself. If the decision-making phase is in the mid-term, the control unit 21 may use the "image to image" function of the image generation AI, reduce the "fidelity to the reference image" parameter provided, and generate the generated image. If the decision-making phase is in a later stage, the control unit 21 may use the "image to image" function of the image generation AI, increase the "fidelity to the reference image" parameter provided, and generate the generated image. In other words, it is advisable to vary the degree of difference between the reference image and the generated image depending on the decision-making phase. Furthermore, when using the "image to image" function of the image generation AI, and when a negative prompt is used, the "fidelity to reference image" parameter provided may be smaller than when a negative prompt is not used. Here, a negative prompt is used when, for example, the user uses the operation unit 13 to specify elements that they do not want to be included in the image.

[0049] Furthermore, Figure 11 shows the flow of the image generation support process when the control unit 11 of the image generation support device 1 is equipped with the various functions of the control unit 21, and the image generation support device 1 operates independently. Each step is the same as those shown in Figure 2, except for the transmission and reception of data between devices, so the explanation is omitted.

[0050] Figure 12 is an image of the history screen D3. Area A5 is the area that displays basic information related to image formation. Area A6 is a region that displays a list of generated images for each phase of decision-making. Note that the image generation support screen D2 mentioned above includes a button B8 (OK / NG button) for the user to input an evaluation, and in area A6, images with an OK evaluation (OK images) and images with an NG evaluation (NG images) may be displayed in separate areas. Area A7 is a region that displays user-entered input data, generated images, and user-entered comments on the generated images in chronological order.

[0051] Furthermore, while the above primarily describes the predetermined conditions as a phase of decision-making, this example is not the only one that applies. For example, the predetermined conditions may be information entered by the user using the operation unit 13 or the like. For example, the predetermined conditions may be information about what is being image-formed. Specifically, the control unit 21 may make the degree of difference between the generated images significantly different if the image-formed object is a "beer bottle," and may make the degree of difference between the generated images small if the image-formed object is a "car."

[0052] <Effects> As described above, the program causes the computer to perform a text generation step (steps S4 and S13) in which the AI ​​generates multiple different image generation text data from input data so that the degree of difference between the multiple generated images generated by the image generation AI differs according to predetermined conditions, and an image generation step (steps S5 and S14) in which the image generation AI generates multiple generated images from the multiple different image generation text data. Therefore, it is possible to generate the image intended by the user more quickly. In particular, as shown in the image diagram in Figure 14, a large degree of difference between images is preferable when the decision-making phase is in the early stages, and a small degree of difference between images is preferable when the decision-making phase is in the later stages.

[0053] Furthermore, the image generation support system 100 includes a text generation unit (control unit 21) that uses an AI to generate multiple different image generation text data from input data so that the degree of difference between multiple generated images generated by the image generation AI differs according to predetermined conditions, and an image generation unit (control unit 21) that uses an image generation AI to generate multiple generated images from the multiple different image generation text data. Therefore, it is possible to generate the image intended by the user more quickly.

[0054] Furthermore, the image generation support device 1 includes a text generation unit (control unit 11) that uses an AI to generate multiple different image generation text data from input data so that the degree of difference between multiple generated images generated by the image generation AI differs according to predetermined conditions, and an image generation unit (control unit 11) that uses an image generation AI to generate multiple generated images from the multiple different image generation text data. Therefore, it is possible to generate the image intended by the user more quickly.

[0055] Furthermore, the image generation support method is an image generation support method performed by an image generation support system, and includes a text generation step (steps S4 and S13) in which the AI ​​generates multiple different image generation text data from input data so as to vary the degree of difference between multiple generated images generated by the image generation AI according to predetermined conditions, and an image generation step (steps S5 and S14) in which the image generation AI generates multiple generated images from the multiple different image generation text data. Therefore, it is possible to generate the image intended by the user more quickly.

[0056] Although the present invention has been described in detail based on embodiments above, it goes without saying that the present invention is not limited to the above embodiments and can be modified as appropriate without departing from the spirit of the invention. For example, the above description disclosed an example in which a hard disk or semiconductor non-volatile memory was used as a computer-readable medium for the program according to the present invention, but the invention is not limited to this example. Portable recording media such as CD-ROMs can be used as other computer-readable media.

[0057] Furthermore, the detailed configuration and operation of each device can be modified as appropriate, without departing from the spirit of the invention. [Explanation of Symbols]

[0058] 100 Image Generation Support System 1. Image generation support device 11 Control Unit (Text Generation Unit, Image Generation Unit) 12 Display section 13 Control section 14 Communications Department 15 Storage section 2 Cloud Server 21 Control Unit (Text Generation Unit, Image Generation Unit) 22 Communications Department 23 Memory section 3 External device N Communication Network

Claims

1. On the computer, A text generation step in which an AI generates multiple different image generation text data from input data, such that the degree of difference between multiple generated images generated by the image generation AI differs according to predetermined conditions. Image generation step: An image generation AI generates multiple generated images from the aforementioned multiple different text data for image generation. A program that executes the command.

2. The program according to claim 1, wherein the predetermined conditions are in the decision-making phase.

3. The program according to claim 2, wherein the decision-making phase corresponds to information on an input screen for the user to input the input data.

4. The program according to claim 2, wherein the text generation step is configured such that the degree of difference is significantly different when it is determined that the decision-making phase is in the early stages compared to when it is determined that the decision-making phase is in the later stages.

5. The program according to claim 1, wherein varying the degree of difference means varying the magnitude of the difference and / or varying the number of differences.

6. The program according to claim 1, wherein the text generation step generates a plurality of different text data for image generation by changing the items to be changed among the items of the text data.

7. If the aforementioned input data includes a reference image, The program according to claim 1, wherein the image generation step generates a plurality of generated images such that the degree of difference differs, either without using the reference image or by using the reference image.

8. A text generation unit generates multiple different image generation text data using AI, based on input data, so that the degree of difference between multiple generated images generated by the image generation AI varies according to predetermined conditions. An image generation unit generates multiple generated images using an image generation AI from the aforementioned multiple different text data for image generation, An image generation support system equipped with the following features.

9. A text generation unit generates multiple different image generation text data using AI, based on input data, so that the degree of difference between multiple generated images generated by the image generation AI varies according to predetermined conditions. An image generation unit generates multiple generated images using an image generation AI from the aforementioned multiple different text data for image generation, An image generation support device equipped with the following features.

10. An image generation support method performed by an image generation support system, A text generation step in which an AI generates multiple different image generation text data from input data, such that the degree of difference between multiple generated images generated by the image generation AI differs according to predetermined conditions. Image generation step: An image generation AI generates multiple generated images from the aforementioned multiple different text data for image generation. Image generation support method including

Images