Prompt generation device

Abstract

To provide a prompt generation device for automatically providing an appropriate adjustment prompt to be input to an image generation model.SOLUTION: A user prompt acquisition unit 21 acquires a user prompt input by a user. A system prompt selection unit 22 selects a system prompt preset for an image application specified by the user from among system prompt presets including preselected image characteristic setting values for multiple image characteristic items for specific image applications. An input prompt generation unit 23 generates an input prompt based on the user prompt and the image characteristic setting value of the selected system prompt preset. A generate image acquisition unit 24 acquires a generated image corresponding to the input prompt with the use of an image generation model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a prompt generating device. [Background technology]

[0002] In a machine-learned image generation model, input text (prompt) is converted into a feature vector, and an image corresponding to the feature vector is generated (see, for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] "Learning Transferable Visual Models From Natural Language Supervision" (Radford et.al, Proceedings of the 38th International Conference on Machine Learning, PMLR 139:8748-8763, 2021) Summary of the Invention [Problem to be solved by the invention]

[0004] Generally, when a user uses an image generation model such as that described above to obtain a generated image as desired by the user, the user needs to include adjustment prompts (setting values ​​(modifiers) for items such as brightness, precision, and composition) in the prompts input to the image generation model to adjust image characteristics so that a generated image having the characteristics desired by the user is obtained.

[0005] Therefore, every time a user generates an image, the user must specify appropriate adjustment prompts to generate an image with desired image characteristics, which is time-consuming.

[0006] The present invention has been made in consideration of the above problems, and aims to provide a prompt generation device that automatically provides appropriate adjustment prompts for input into an image generation model with simple user operations. [Means for solving the problem]

[0007] The prompt generation device of the present invention includes a user prompt acquisition unit that acquires a user prompt input by a user, a system prompt selection unit that selects a system prompt preset for an image application specified by the user from among system prompt presets that include image characteristic setting values ​​preselected for a plurality of image characteristic items for a specific image application, an input prompt generation unit that generates an input prompt based on the user prompt and the image characteristic setting values ​​in the selected system prompt preset, and a generated image acquisition unit that acquires a generated image corresponding to the input prompt in an image generation model. [Effects of the Invention]

[0008] According to the present invention, a prompt generation device is provided that automatically provides appropriate adjustment prompts for inputting into an image generation model with simple user operation.

[0009] The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing the configuration of a prompt generation device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of the system prompt definition data 13a. [Figure 3] FIG. 3 is a diagram illustrating an example of system prompt presets for object types. [Figure 4]FIG. 4 illustrates an example of a system prompt preset for an image application. [Figure 5] FIG. 5 is a flowchart illustrating the generation of an input prompt for image generation in the prompt generation device 1 shown in FIG. [Figure 6] FIG. 6 is a flowchart illustrating the registration of system prompt presets in the prompt generation device 1 shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] Fig. 1 is a block diagram showing the configuration of a prompt generation device according to an embodiment of the present invention. The prompt generation device 1 shown in Fig. 1 is an electronic device such as a multifunction peripheral or a personal computer, and includes a processing unit 11 as a computer, a communication unit 12, a storage unit 13, a display unit 14, and an input unit 15.

[0013] The communication device 12 is a device (such as a network interface) capable of data communication with other devices via a computer network 2 such as the Internet or an intranet.

[0014] Storage device 13 is a non-volatile storage device such as a hard disk drive or flash memory, and stores programs, data, etc. Storage device 13 stores system prompt definition data 13a, system prompt preset data 13b, etc.

[0015] 2 is a diagram showing an example of the system prompt definition data 13a. For example, as shown in FIG. 2, the system prompt definition data 13a includes, for each of a plurality of image characteristic items, an image characteristic item and a set of setting value candidates for that image characteristic item, in a mutually associated state. The system prompt definition data 13a is used to generate the system prompt preset data 13b. Note that the system prompt is the adjustment prompt described above, and the system prompt preset is a data set that specifies a single setting value for each image characteristic item so that conflicting setting values ​​for each image characteristic item are not included in the input prompt.

[0016] Figure 3 shows examples of system prompt presets for object types, and Figure 4 shows examples of system prompt presets for image applications.

[0017] For example, as shown in FIG. 3, a system prompt preset for an object type is a data set containing preselected image property settings for a number of image properties for that object type.

[0018] Also, as shown in FIG. 4, for example, a system prompt preset for a certain image application is a data set containing preselected image characteristic setting values ​​(character strings) for a plurality of image characteristic items for that image application.

[0019] The display device 14 is a device such as a liquid crystal display that displays an operation screen, generated images (described later), etc. The input device 15 is a device such as a hard key or a touch panel that detects user operations.

[0020] Here, the processing unit 11 operates as a user prompt acquisition unit 21, a system prompt selection unit 22, an input prompt generation unit 23, a generated image acquisition unit 24, and a system prompt management unit 25 by executing a program in the storage unit 13.

[0021] The user prompt acquisition unit 21 acquires a user prompt (a character string such as a sentence) input by the user via the input device 15. Furthermore, the user prompt acquisition unit 21 detects a character string of an object type included in the user prompt.

[0022] The system prompt selector 22 selects a system prompt preset for the image application specified by the user, and also selects a system prompt preset for the object type of the character string detected in the user prompt.

[0023] The input prompt generator 23 generates an input prompt based on the above-mentioned user prompt and the image characteristic setting values ​​in the selected system prompt preset.

[0024] In this embodiment, the input prompt generator 23 generates an input prompt based on the above-described user prompt, the image characteristic settings in the system prompt preset for the selected image application, and the image characteristic settings in the system prompt preset for the selected object type.

[0025] Here, the user prompt acquisition unit 21 detects a character string of an image characteristic included in the acquired user prompt, and if the character string detected in the user prompt conflicts with an image characteristic setting value in a system prompt preset for a selected image use or object type, the input prompt generation unit 23 prioritizes the detected character string. Also, if the image characteristic setting value in the system prompt preset for a selected image use conflicts with an image characteristic setting value in a system prompt preset for a selected object type, the input prompt generation unit 23 selects the image characteristic setting value in the system prompt preset for the selected image use as a priority and generates an input prompt.

[0026] For example, if the character string "Illustration-style beautiful mountain" is entered as a user prompt, syntactic analysis will divide the character string into three characters: "Illustration-style," "Beautiful," and "Mountain." Of these characters, the noun "Mountain" is identified as the object type. Furthermore, from "Illustration-style" and "Beautiful," the modifier "Illustration," registered in the system prompt definition data 13a shown in Figure 2, is detected as a character string of an image characteristic.

[0027] Then, the system prompt preset for the object type "mountain" in the system prompt preset data 13b is selected.

[0028] In the system prompt preset for the object type "mountain," the setting value for the item "style" is "watercolor," which conflicts with the "illustration" specified in the user prompt. In this case, because the user prompt takes precedence over the system prompt preset, "illustration" is used as the setting value for the item "style" rather than "watercolor." For items with mismatched settings, the item specified based on the user prompt is given first priority, the item specified based on the system prompt preset for image use is given second priority, and the item specified based on the system prompt preset for object type is given third priority. In this case, the input prompt generated is "beautiful mountain, 4K, illustration, nature, distant, sunlight." In this way, the "beautiful" setting in the user prompt, which does not match the system prompt definition data 13a, may be included in the input prompt or excluded.

[0029] The generated image acquisition unit 24 acquires a generated image corresponding to the above-mentioned input prompt using an image generation model.

[0030] The image generation model is a learning device that has been machine-learned using an existing method, and generates image data (generated image) corresponding to the input prompt. The generated image acquisition unit 24 may have the image generation model built in, or may access an external server or the like that has the image generation model implemented using the communication device 12, send the input prompt to the external server, and acquire the generated image from the external server.

[0031] The system prompt management unit 25 generates system prompt preset data 13b indicating a system prompt preset based on the image use or object type and image characteristic setting values ​​specified by the user, and stores the generated data in the storage device 13. The system prompt selection unit 22 reads the system prompt preset data 13b stored in the storage device 13 to obtain the selected system prompt preset.

[0032] Specifically, the system prompt management unit 25 displays, for example, the system prompt definition data 13a shown in FIG. 2 on the display device 14 for the image use or object type specified by the user, and generates a system prompt preset for the image use or object type by selecting one of the setting values ​​from the setting value candidate group for each image characteristic item in accordance with the user's operation on the input device 15. Note that "(Invalid)" in FIG. 2 indicates a setting that does not include the setting value of that image characteristic item in the system prompt. value.

[0033] For example, if a user inputs the object type "mountain" as a string, selects "(disabled)" for the item "texture," selects "4K" for the item "image quality," selects "watercolor" for the item "style," selects "nature" for the item "background," selects "long distance" for the item "composition," selects "sunlight" for the item "lighting," and selects "(disabled)" for the item "people," then a system prompt preset for the object type "mountain" is generated, as shown in Figure 3.

[0034] In addition, the system prompt management unit 25 may detect object type strings (nouns) in strings such as sentences entered by the user, and may also search for setting value candidate strings registered in the system prompt definition data 13a, and generate a system prompt preset based on the detected object type strings and setting value candidate strings.

[0035] Next, the operation of the prompt generation device 1 according to this embodiment will be described.

[0036] (a) Generation of input prompts for image generation

[0037] FIG. 5 is a flowchart illustrating the generation of an input prompt for image generation in the prompt generation device 1 shown in FIG.

[0038] The user prompt acquisition unit 21 displays a screen (dialog) including, for example, an input field for a user prompt and a list of image uses (a list of image uses included in the system prompt presets registered in the system prompt preset data 13b) on the display device 14, detects the character string entered into the input field as a user prompt using the input device 15, and when a user operates to specify one of the image uses from the image use list, detects the specified image use using the input device 15 (step S1).

[0039] Furthermore, the user prompt acquiring unit 21 extracts the character string of the object type and the character string of the image characteristic (modifier) ​​from the character string of the user prompt, as described above (step S2).

[0040] Then, the system prompt selection unit 22 reads the system prompt preset data 13b and selects a system prompt preset corresponding to the extracted object type from the system prompt preset data 13b (step S3).

[0041] In addition, the system prompt selection unit 22 determines whether or not an image use has been specified as described above (step S4), and if an image use has been specified, selects a system prompt preset corresponding to the specified image use in the system prompt preset data 13b (step S5).

[0042] Then, the input prompt generation unit 23 generates an input prompt based on the character string of the image characteristics in the user prompt obtained in this manner, the system prompt preset corresponding to the object type, and the system prompt preset corresponding to the image use (step S6).

[0043] On the other hand, if the image use is not specified, the input prompt generation unit 23 generates an input prompt based on the character string of the image characteristics in the user prompt obtained as described above and the system prompt preset corresponding to the object type (step S7).

[0044] When the input prompt is generated in this way, the generated image acquisition unit 24 causes the image generation model to generate an image corresponding to the input prompt, and acquires the generated image (image data) (step S8).

[0045] (b) Registering system prompt presets

[0046] FIG. 6 is a flowchart illustrating the registration of system prompt presets in the prompt generation device 1 shown in FIG.

[0047] The system prompt management unit 25 displays a screen (dialog) on ​​the display device 14 (step S21), which includes an input field (first input field) for a new object or a new image, individual input fields (second input fields) for setting values ​​for a plurality of image characteristic items, and an input field (third input field) for inputting setting values ​​for a plurality of image characteristic items using character strings in a natural language. Note that the second input field for the setting value displays a list of setting value candidates in a menu, for example, a pull-down menu, and a setting value can be selected from the menu.

[0048] Then, the system prompt management unit 25 detects the character string of the input field for the new object or new image input into the above-mentioned input field by the input device 15 (step S22).

[0049] In addition, the system prompt management unit 25 determines whether or not input of a character string into the third input field is detected by the input device 15 (step S23), and also determines whether or not a selection operation into the second input field is detected by the input device 15 (step S24).

[0050] When the input of a character string into the third input field is detected by the input device 15, the system prompt management unit 25 searches for a character string in the character string that is the same as a setting value candidate registered in the system prompt definition data 13a (step S25), and uses the found character string as an image characteristic setting value to generate a system prompt preset for the character string in the input field for the above-mentioned new object or new image use, and registers it in the system prompt preset data 13b (step S26).

[0051] On the other hand, if a selection operation on the second input field is detected by the input device 15, the system prompt management unit 25 identifies the selected setting value candidate as the image characteristic setting value (step S27), and generates a system prompt preset for the character string of the input field for the above-mentioned new object or new image use using that image characteristic setting value, and registers it in the system prompt preset data 13b (step S28).

[0052] As described above, according to the above embodiment, the user prompt acquisition unit 21 acquires a user prompt input by a user. The system prompt selection unit 22 selects a system prompt preset for an image application designated by the user from among system prompt presets including preselected image characteristic setting values ​​for a plurality of image characteristic items for a specific image application. The input prompt generation unit 23 generates an input prompt based on the user prompt and the image characteristic setting values ​​in the selected system prompt preset. The generated image acquisition unit 24 acquires a generated image corresponding to the input prompt using an image generation model.

[0053] This allows the user to simply enter a user prompt and select the image purpose, and the appropriate adjustment prompts for input into the image generation model are automatically provided.

[0054] It should be noted that various changes and modifications to the above-described embodiments will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the subject matter and without diminishing its intended advantages. In other words, it is intended that such changes and modifications be included within the scope of the claims.

[0055] For example, in the above embodiment, instead of selecting a system prompt preset for an object type, a system prompt preset for an image application may be selected. [Industrial Applicability]

[0056] The present invention is applicable, for example, to prompt generation for image generation models. [Explanation of symbols]

[0057] 13b System prompt preset data 21 User prompt acquisition unit 22 System prompt selection section 23 Input prompt generation unit 24 Generated image acquisition unit 25 System Prompt Management Department

Claims

1. a user prompt acquisition unit that acquires a user prompt input by a user; a system prompt selection unit that selects a system prompt preset for an image application designated by the user from among system prompt presets including preselected image characteristic setting values ​​for a plurality of image characteristic items for a specific image application; an input prompt generator that generates an input prompt based on the user prompt and the image characteristic settings in the selected system prompt preset; a generated image acquisition unit that acquires a generated image corresponding to the input prompt in an image generation model; 1. A prompt generating device comprising:

2. the user prompt acquisition unit detects a character string of an image characteristic included in the user prompt; the input prompt generation unit generates the input prompt by prioritizing the detected character string when the detected character string conflicts with the image characteristic setting value in the selected system prompt preset; 2. The prompt generating device of claim 1, wherein:

3. The system prompt selection unit (a) selects a system prompt preset for an image use designated by the user, and (b) detects an object type character string included in the user prompt, and selects a system prompt preset for the object type of the detected character string from among system prompt presets including preselected image characteristic setting values ​​for a plurality of image characteristic items for a specific object type; 2. The prompt generating device of claim 1, wherein the input prompt generating unit generates an input prompt based on the user prompt, the image characteristic setting values ​​in the system prompt preset for the selected image use, and the image characteristic setting values ​​in the system prompt preset for the selected object type.

4. the user prompt acquisition unit detects a character string of an image characteristic included in the user prompt; the input prompt generation unit generates the input prompt by giving priority to the detected character string when the detected character string conflicts with the image characteristic setting value in the system prompt preset for the selected image use or the selected object type, and by giving priority to the image characteristic setting value in the system prompt preset for the selected image use when the image characteristic setting value in the system prompt preset for the selected object type conflicts with the image characteristic setting value in the system prompt preset for the selected image use; 4. The prompt generating device according to claim 3, wherein:

5. further comprising a system prompt management unit; the system prompt management unit generates system prompt preset data indicating the system prompt preset based on the image use or object type and image characteristic setting value designated by the user, and stores the system prompt preset data in a storage device; the system prompt selection unit reads out the system prompt preset data stored in the storage device and acquires the selected system prompt preset; 5. The prompt generating device according to claim 1, wherein:

Images