Prompt generation system

Abstract

To provide a prompt generation system for providing an appropriate adjustment prompt to be input to an image generation model.SOLUTION: A user attribute identification unit 22 identifies a user attribute of a user. A prompt estimation unit 23 estimates an adjustment prompt corresponding to the identified user attribute with the use of a machine-learned prompt estimation model. A generate image acquisition unit 24 acquires a generated image corresponding to an input prompt including the estimated adjustment prompt with the use of an image generation model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a prompt generation system. [Background technology]

[0002] In a machine-learned image generation model, input text (prompt) is converted into a feature vector, and an image corresponding to the feature vector is generated (see, for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] "Learning Transferable Visual Models From Natural Language Supervision" (Radford et.al, Proceedings of the 38th International Conference on Machine Learning, PMLR 139:8748-8763, 2021) Summary of the Invention [Problem to be solved by the invention]

[0004] Generally, when a user uses an image generation model such as that described above to obtain a generated image that the user desires, the user needs to include adjustment prompts (settings for brightness, precision, composition, etc.) in the prompts input to the image generation model so that the generated image has the characteristics the user desires.

[0005] However, depending on the user's skill level and knowledge, it may be difficult to specify an appropriate adjustment prompt.

[0006] The present invention has been made in view of the above problems, and an object of the present invention is to provide a prompt generation system that provides appropriate adjustment prompts to be input into an image generation model. [Means for solving the problem]

[0007] The prompt generation system of the present invention includes a user attribute identification unit that identifies user attributes, a prompt estimation unit that estimates an adjustment prompt corresponding to the user attributes using a machine-learned prompt estimation model, and a generated image acquisition unit that acquires a generated image corresponding to an input prompt including the adjustment prompt using an image generation model. [Effects of the Invention]

[0008] The present invention provides a prompt generation system that provides appropriate calibration prompts for input into an image generation model.

[0009] The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing the configuration of a prompt generation system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating user attributes and tailoring prompts. [Figure 3] FIG. 3 is a diagram illustrating the generation of a plurality of generated images. [Figure 4] FIG. 4 is a flowchart illustrating the operation of the image forming apparatus 1 in FIG. [Figure 5] FIG. 5 is a flowchart illustrating the operation of the management server 3 in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] 1 is a block diagram showing the configuration of a prompt generation system according to an embodiment of the present invention. The prompt generation system shown in Fig. 1 includes an image forming device 1 and a management server 3 capable of data communication with the image forming device 1 via a computer network 2.

[0013] The image forming apparatus 1 is an electronic device such as a multifunction peripheral, and includes a processing unit 11 as a computer, a communication unit 12, a storage unit 13, a display unit 14, and an input unit 15.

[0014] The communication device 12 is a device (such as a network interface) capable of communicating data with other devices (such as the management server 3 in this example) via a computer network 2, such as the Internet or an intranet. The storage device 13 is a non-volatile storage device, such as a hard disk drive or flash memory, that stores programs, data, and the like. The storage device 13 stores setting data 13a and user registration data 13b, which will be described later. The user registration data 13b includes a user ID and user attributes for each registered user. For example, the user registration data 13b is used for user authentication at login. The display device 14 is a device such as an LCD display that displays an operation screen and generated images, which will be described later. The input device 15 is a device such as a hard key or touch panel that detects user operations.

[0015] Here, the arithmetic processing unit 11 executes the program in the storage device 13, It operates as an object setting unit 21 , a user attribute specifying unit 22 , a prompt estimation unit 23 , a generated image obtaining unit 24 , a generated image selecting unit 25 , a training data transmitting unit 26 , and a prompt estimation model updating unit 27 .

[0016] The object setting unit 21 sets the type of object to be included in the generated image. The type of object is specified by the user who requested image generation. For example, if "orange" is specified as the type of object, "orange" is included in the input prompt, and a generated image including an image of an orange is generated.

[0017] The user attribute specification unit 22 specifies the user attribute of the user who has requested image generation. Specifically, the user attribute specification unit 22 specifies the user attribute by referring to the user registration data 13b.

[0018] The prompt estimation unit 23 estimates an adjustment prompt corresponding to a user attribute using a prompt estimation model that has been machine-learned. The prompt estimation model is a learning device (e.g., a deep neural network) that has parameter values ​​obtained by machine learning, which will be described later. These parameter values ​​are stored in the storage device 13 as setting data 13a.

[0019] Fig. 2 is a diagram illustrating user attributes and adjustment prompts. For example, as shown in Fig. 2, user attributes include the user's country (the country where the user lives, the country of the user's nationality, etc.), the user's industry, the user's occupation, etc., and adjustment prompts specify image characteristics (setting values ​​for items such as brightness, precision, composition, etc.) of objects specified by object types in the generated image.

[0020] The generated image acquisition unit 24 acquires, in the image generation model, a generated image corresponding to the input prompt including the adjustment prompt obtained by the prompt estimation unit 23. The input prompt includes the adjustment prompt as well as the object types described above.

[0021] The image generation model is a learning device that has been machine-learned using an existing method, and generates image data (generated image) corresponding to the input prompt. The generated image acquisition unit 24 may have the image generation model built-in, or may access an external server on which the image generation model is implemented, send the input prompt to the external server, and acquire the generated image from the external server.

[0022] Here, the prompt estimation unit 23 estimates a plurality of adjustment prompts corresponding to user attributes for a certain prompt type (i.e., the above-mentioned image feature items) using a prompt estimation model, and the generated image acquisition unit 24 acquires a plurality of generated images corresponding to a plurality of input prompts, each of which includes a plurality of adjustment prompts, using an image generation model.

[0023] FIG. 3 is a diagram illustrating the generation of multiple generated images. For example, as shown in FIG. 3, the prompt estimation unit 23 derives confidence levels (values ​​ranging from 0 to 1) for a predetermined number of adjustment prompts (setting values) for each prompt type using a prompt estimation model. The multiple adjustment prompts used in the multiple generated images are selected from the predetermined number of adjustment prompts based on the confidence levels derived by the prompt estimation model. For example, as shown in FIG. 3, for a prompt type (item) called brightness, two setting values, "bright" and "medium," are selected from three setting values: "bright" (confidence level 0.6), "medium" (confidence level 0.3), and "dark" (confidence level 0.1). Two generated images corresponding to the two setting values ​​are generated. That is, generated image #1 with high brightness and generated image #2 with a medium brightness are generated.

[0024] The generated image selection unit 25 selects a generated image designated by the user from among the plurality of generated images obtained by the generated image acquisition unit 24 .

[0025] When a generated image specified by the user is selected, the learning data transmission unit 26 identifies the user attributes of the user who requested image generation and the adjustment prompt corresponding to the selected generated image as a pair of learning data (a pair of explanatory variables and objective variables, i.e., a pair of input data and output data of the model), and transmits them to the management server 3 via the communication device 12.

[0026] For example, the user who has requested image generation is identified by a user ID when logging in to the image forming apparatus 1, or by a user ID included in a request received from an external host device.

[0027] Also, for example, as shown in Figure 2, if user A selects an image generated with an adjustment prompt in which the item "brightness" is "dark," the item "precision" is "medium," and the item "composition" is "close," learning data is generated and transmitted in which the user attributes (country, industry, occupation) are ("Japan," "caregiving," "staff") and the adjustment parameters (brightness, precision, composition) are ("dark," "medium," "close").

[0028] The prompt estimation model update unit 27 (a) acquires initial values ​​of the parameters of the prompt estimation model from the management server 3 using the communication device 12, stores the initial values ​​as setting data 13a in the storage device 3, and sets the initial values ​​in the prompt estimation model, and (b) upon receiving updated values ​​of the parameters of the prompt estimation model from the management server 3, updates the setting data 13a with the updated values, thereby updating the prompt estimation model.

[0029] The management server 3 also includes a processing unit 31 as a computer, a communication unit 32, and a storage unit 33.

[0030] The communication device 32 is a device (such as a network interface) capable of communicating data with other devices (such as the image forming device 1 in this example) via a computer network 2 such as the Internet or an intranet. The storage device 33 is a non-volatile storage device such as a hard disk drive or flash memory, and stores programs, data, etc. The storage device 33 stores a learning database 33a (described later) and the like.

[0031] Here, the arithmetic processing unit 31 executes the program in the storage device 33, It operates as a learning data receiving unit 41, a machine learning processing unit 42, and a prompt estimation model sending unit 43.

[0032] The training data receiving unit 41 uses the communication device 32 to repeatedly receive the above-mentioned pairs of training data from one or more image forming devices 1 via the computer network 2, performs embedding processing on the received training data to convert it into pairs of feature vectors, and stores the pairs of feature vectors in the training database 33a of the storage device 33.

[0033] In this embedding process, the user attributes as input data in the training data and the adjustment prompts (setting values) as output data are converted (for each item) into one-hot feature vectors. This embedding process may be performed by the training data accepting unit 41 or by the training data transmitting unit 26 before transmission.

[0034] The machine learning processing unit 42 uses the user attributes stored in the learning database 33a and the adjustment prompt corresponding to the selected generated image as pairs of learning data (specifically, the pair of the feature vector of the user attribute and the feature vector of the adjustment prompt as learning data), and performs machine learning of the prompt estimation model using an existing method with a predetermined number or more pairs of learning data, and derives parameter values ​​for the prompt estimation model.

[0035] The prompt estimation model sending unit 43 sends the parameter values ​​of the prompt estimation model derived by machine learning to the image forming device 1 (prompt estimation model update unit 27) in the communication device 32, and causes the prompt estimation model update unit 27 to update the prompt estimation model.

[0036] Here, the prompt estimation unit 23 estimates an adjustment prompt corresponding to a user attribute for each of a predetermined number of prompt types using a prompt estimation model for each of the prompt types, and the generated image acquisition unit 24 acquires generated images corresponding to input prompts including adjustment prompts of the multiple prompt types using an image generation model. Therefore, the machine learning processing unit 42 performs machine learning of a prompt estimation model for each of the prompt types.

[0037] The management server 3 receives the above-mentioned learning data from a plurality of image forming apparatuses 1.

[0038] Next, the operation of the above prompt generation system will be described.

[0039] (a) Operation of the image forming apparatus 1

[0040] FIG. 4 is a flowchart illustrating the operation of the image forming apparatus 1 in FIG.

[0041] In the image forming device 1, when a predetermined user operation (image generation request) is detected by the input device 15, the object setting unit 21 sets the type of object based on the user operation (step S1) and identifies the user attributes of the user (step S2).

[0042] The prompt estimation unit 23 estimates adjustment prompts corresponding to user attributes for each of a plurality of prompt types using a prompt estimation model for each prompt type to generate a plurality of input prompt candidates (step S3), and the generated image acquisition unit 24 acquires a plurality of generated images corresponding to the plurality of input prompt candidates using an image generation model (step S4).

[0043] Next, generated image selection unit 25 displays the acquired generated images on display device 14, and when input device 15 detects a user operation specifying a desired generated image from the displayed generated images, selects the desired generated image from the plurality of generated images (step S5). The selected generated image is stored in storage device 13 or used in subsequent processing.

[0044] Once the generated image is selected in this manner, the learning data transmission unit 26 transmits a pair of user attribute information indicating the above-mentioned user attributes and the adjustment prompt used in the selected generated image as learning data to the management server 3 via the communication device 12 (step S6).

[0045] In this way, each time a generated image is selected by the user, learning data is transmitted from the image forming apparatus 1 to the management server 3.

[0046] (b) Operation of Management Server 3

[0047] FIG. 5 is a flowchart illustrating the operation of the management server 3 in FIG.

[0048] In the management server 3, when the learning data is sent from the image forming device 1 to the management server 3, the learning data receiving unit 41 receives the learning data using the communication device 32 (step S11), performs embedding processing on the learning data (step S12), and then stores the data in the learning database 33a (step S13).

[0049] The machine learning processing unit 42 determines whether the number of data pairs stored in the learning database 33a has reached a predetermined number (step S14).

[0050] When the number of data pairs stored in the learning database 33a reaches a predetermined number, the machine learning processing unit 42 performs machine learning on the prompt inference model based on the data pairs stored in the learning database 33a at that time, and derives parameter values ​​of the prompt inference model (step S15). For example, machine learning is performed every time the number of data pairs increases by a predetermined number.

[0051] Then, the prompt estimation model sending unit 43 uses the communication device 32 to send the derived parameter values ​​of the prompt estimation model to the image forming device 1 (step S16). In the image forming device 1, the prompt estimation model updating unit 27 receives the parameter values ​​via the communication device 12 and updates the prompt estimation model with the received parameter values.

[0052] In this way, every time learning data is received from any of the image forming apparatuses 1, the learning data is accumulated, and machine learning of the prompt estimation model is repeatedly performed as appropriate, thereby updating the prompt estimation model.

[0053] As described above, according to the embodiment, the user attribute identification unit 22 identifies a user attribute. The prompt estimation unit 23 estimates an adjustment prompt corresponding to the identified user attribute using a machine-learned prompt estimation model. The generated image acquisition unit 24 acquires a generated image corresponding to an input prompt including the estimated adjustment prompt using an image generation model.

[0054] This provides appropriate adjustment prompts to input into the image generation model, resulting in a generated image with image features that correspond to the user attributes (i.e., a generated image that is likely to be liked by users of the used user attributes).

[0055] It should be noted that various changes and modifications to the above-described embodiments will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the subject matter and without diminishing its intended advantages. In other words, it is intended that such changes and modifications be included within the scope of the claims. [Industrial Applicability]

[0056] The present invention is applicable, for example, to prompt generation for image generation models. [Explanation of symbols]

[0057] 22 User attribute identification section 23 Prompt Estimation Section 24 Generated image acquisition unit 25 Generated image selection section 42 Machine learning processing section

Claims

1. a user attribute identification unit that identifies user attributes; a prompt estimation unit that estimates an adjustment prompt corresponding to the user attribute using a machine-learned prompt estimation model; a generated image acquisition unit that acquires, in an image generation model, a generated image corresponding to an input prompt including the adjustment prompt; A prompt generation system comprising:

2. Further comprising a generated image selection unit, the prompt estimation unit estimates, for a certain prompt type, a plurality of adjustment prompts corresponding to the user attributes using the prompt estimation model; the generated image acquisition unit acquires, from the image generation model, a plurality of generated images corresponding to a plurality of input prompts each including the plurality of adjustment prompts; the generated image selection unit selects a generated image designated by a user from among the plurality of generated images; 2. The prompt generation system of claim 1.

3. 3. The method of claim 2, wherein the plurality of calibration prompts are selected from a predetermined number of calibration prompts based on confidence levels derived from the prompt estimation model. Prompt generation system.

4. further comprising a machine learning processing unit; the machine learning processing unit performs machine learning of the prompt estimation model using the user attributes of the user and the selected adjustment prompt corresponding to the generated image as a pair of learning data; 3. The prompt generation system of claim 2.

5. the prompt estimation unit estimates, for each of a plurality of predetermined prompt types, an adjustment prompt corresponding to the user attribute using a prompt estimation model for each of the prompt types; the generated image acquisition unit acquires, from the image generation model, generated images corresponding to input prompts including the adjustment prompts of the plurality of prompt types; the machine learning processing unit performs machine learning of the prompt estimation model for each of the prompt types; 5. The prompt generation system of claim 4.

Images