Prompt word optimization method for image generation model of professional knowledge distillation

Abstract

The invention relates to the technical field of computer vision and multi-modal generation, in particular to an image generation model cue word optimization method and device for professional knowledge distillation, equipment and a computer storage medium. According to the method for optimizing the cue word of the image generation model, the training data of the professional knowledge text is constructed, and the cue word rewriter is trained, so that the character cue word of the user is successfully optimized into the text with richer content and more professional knowledge in subsequent application, and the image generation effect is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and multimodal generation technology, and in particular to a method, device, equipment and computer storage medium for optimizing prompt words of an image generation model for professional knowledge distillation. Background Art

[0002] With the rapid development of text-to-image generation technology, people can input text of various contents and styles and let the model generate the images they want. We found that the generation effect of the text-to-image model is greatly affected by the input text. Specifically, for many abstract, brief, or vague needs of humans, the model cannot generate images well. Some experiments show that properly optimizing and rewriting the text input by humans will significantly improve the effect of model generation and better meet human needs.

[0003] However, we found that in the image generation needs of professional fields (such as architecture, art, etc.), general optimization rewriting, such as adding some prompt words to make the picture more beautiful, is still far from meeting the needs of users. Architectural picture drawing involves many design concepts and corresponding design details. If the prompt word optimization model does not learn relevant content during the training process, it will not be reflected when the prompt word is rewritten. In order for the model to generate some pictures that meet the concepts and tastes of the professional field, this requires prompt word optimization to have professional knowledge.

[0004] Therefore, how to optimize and rewrite the prompt words of the image generation model to meet the needs of professional fields is a problem that needs to be solved at present. Summary of the invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problem in the prior art that the image generation model prompt words are not optimized and rewritten in accordance with professional field requirements, resulting in inaccurate image generation.

[0006] In order to solve the above technical problems, the present invention provides a method for optimizing prompt words of an image generation model, comprising:

[0007] Get user input text;

[0008] Input the user input text into the trained prompt word rewriter to obtain the target text,

[0009] The training process of the prompt word rewriter includes:

[0010] Collect multiple professional field pictures, and obtain picture descriptions corresponding to the multiple professional field pictures;

[0011] Input multiple image descriptions into the image generation model to generate multiple generated images;

[0012] The picture description corresponding to the generated image that meets the preset conditions is defined as the professional knowledge text;

[0013] Generate multiple training data pairs based on the professional knowledge text and its corresponding generated image to construct a training set;

[0014] Based on the training set, the prompt word rewriter is trained until the loss function converges to obtain a trained prompt word rewriter.

[0015] Preferably, the collecting of multiple professional field pictures and obtaining picture descriptions corresponding to the multiple professional field pictures includes:

[0016] Collecting multiple professional field images with quality scores greater than a first preset threshold;

[0017] The multiple professional field images are input into a visual understanding model based on prompt engineering to generate multiple image descriptions.

[0018] Preferably, defining the picture description corresponding to the generated image that meets the preset conditions as professional knowledge text includes:

[0019] The picture descriptions corresponding to the generated images whose quality scores are greater than the second preset threshold are collected to obtain a plurality of professional knowledge texts.

[0020] Preferably, after defining the picture description corresponding to the generated image that meets the preset conditions as professional knowledge text, the method further includes:

[0021] Each professional knowledge text is input into a large language model based on prompt engineering to generate multiple simplified texts.

[0022] Preferably, generating a plurality of training data pairs according to the professional knowledge text and the corresponding generated image, and constructing a training set comprises:

[0023] According to each simplified text and its corresponding generated image, multiple training data pairs are constructed, and a training set is constructed.

[0024] Preferably, the step of generating a plurality of training data pairs further comprises:

[0025] Collect real user feedback scores, simulated user feedback scores, and image quality scores for each generated image to generate a preference score for each generated image;

[0026] The training data pair corresponding to the generated image whose preference score is greater than the third preset threshold is defined as a preferred training data pair.

[0027] Preferably, the training of the prompt word rewriter based on the training set includes:

[0028] Based on the training set constructed by the preference training data, the prompt word rewriter is subjected to reinforcement learning training using a direct preference optimization algorithm.

[0029] The present invention also provides an image generation model prompt word optimization device, comprising:

[0030] A prompt word acquisition module is used to obtain user input text;

[0031] The prompt word optimization module is used to input the user input text into the trained prompt word rewriter to obtain the target text.

[0032] The training process of the prompt word rewriter includes:

[0033] Collect multiple professional field pictures, and obtain picture descriptions corresponding to the multiple professional field pictures;

[0034] Input multiple image descriptions into the image generation model to generate multiple generated images;

[0035] The picture description corresponding to the generated image that meets the preset conditions is defined as the professional knowledge text;

[0036] Generate multiple training data pairs based on the professional knowledge text and its corresponding generated image to construct a training set;

[0037] Based on the training set, the prompt word rewriter is trained until the loss function converges to obtain a trained prompt word rewriter.

[0038] The present invention also provides an image generation model prompt word optimization device, comprising:

[0039] Memory for storing computer programs;

[0040] A processor is used to implement the steps of the above-mentioned image generation model prompt word optimization method when executing the computer program.

[0041] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned image generation model prompt word optimization method are implemented.

[0042] The above technical solution of the present invention has the following advantages compared with the prior art:

[0043] The image generation model prompt word optimization method described in the present invention constructs training data of professional knowledge text and trains a prompt word rewriter so that in subsequent applications, the user's text prompt words can be successfully optimized into texts with richer content and more professional knowledge, thereby significantly improving the image generation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0045] Figure 1 It is a flow chart for implementing a prompt word optimization method for an image generation model provided by the present invention. DETAILED DESCRIPTION

[0046] The core of the present invention is to provide an image generation model prompt word optimization method, device, equipment and computer storage medium, which can optimize the user's input text into a text that conforms to professional knowledge and taste, so that the generated image can better meet the user's needs.

[0047] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0048] Please refer to Figure 1 , Figure 1 This is a flowchart of an implementation method of the image generation model prompt word optimization method provided by the present invention; the specific operation steps are as follows:

[0049] S101: Obtain user input text;

[0050] S102: Input the user input text into the trained prompt word rewriter to obtain the target text.

[0051] The training process of the prompt word rewriter includes:

[0052] Collect multiple professional field pictures, and obtain picture descriptions corresponding to the multiple professional field pictures;

[0053] Input multiple image descriptions into the image generation model to generate multiple generated images;

[0054] The picture description corresponding to the generated image that meets the preset conditions is defined as the professional knowledge text;

[0055] Generate multiple training data pairs based on the professional knowledge text and its corresponding generated image to construct a training set;

[0056] Based on the training set, the prompt word rewriter is trained until the loss function converges to obtain a trained prompt word rewriter.

[0057] Based on the above embodiments, this embodiment describes step S102 in detail:

[0058] Based on the above embodiment, the collecting of multiple professional field pictures and obtaining picture descriptions corresponding to the multiple professional field pictures includes:

[0059] Collect multiple professional field images with quality scores greater than a first preset threshold;

[0060] The present invention cooperated with experts in related professional fields, selected several public open source picture websites, and collected 600,000 professional field pictures.

[0061] There are important insights behind the pictures in professional fields. That is, we need to use the specific design details of professional pictures to obtain the corpus that the prompt word optimization model needs to learn.

[0062] We then used a variety of image evaluators, including an aesthetic scorer, to screen the images, obtaining 200,000 high-quality professional images with quality scores greater than the first preset threshold.

[0063] Input the plurality of professional field images into a visual understanding model based on prompt engineering to generate a plurality of image descriptions.

[0064] Given an original dataset D = {img}, where img is a professional field picture, we use the visual understanding model to automatically generate high-quality text description txt. In the experiment, we use the currently popular visual understanding models with the best performance, including GPT-4V, CogVLM, LLaVa, etc., and use prompt engineering to guide the visual understanding model to generate high-quality image descriptions.

[0065] Based on the above embodiment, defining the picture description corresponding to the generated image that meets the preset conditions as professional knowledge text includes:

[0066] The picture descriptions corresponding to the generated images whose quality scores are greater than the second preset threshold are collected to obtain a plurality of professional knowledge texts.

[0067] Since high-quality image descriptions may not directly lead to suitable input texts for the image generation model, the present invention needs to screen out truly suitable image descriptions. Therefore, the present invention inputs a variety of different image descriptions into the image generation model to obtain different images.

[0068] For these different pictures, the present invention obtains the quality of the pictures that can be obtained by multiple different texts by organizing a professional annotation process composed of experts, thereby obtaining high-quality professional knowledge text with a quality score greater than a second preset threshold.

[0069] Based on the above embodiment, after defining the picture description corresponding to the generated image that meets the preset conditions as professional knowledge text, the method further includes:

[0070] Each professional knowledge text is input into a large language model based on prompt engineering to generate multiple simplified texts.

[0071] As mentioned above, the prompt word rewriter rewrites the abbreviated or ambiguous text input by people into specific text with professional knowledge. Now that we have the latter, we need to get the former to form a training data pair. We use popular language models such as GLM-4 and GPT-4 to perform prompt engineering, automatically prompting the language model to simplify the latter to get the former. Since simplification only requires general language skills, this process can be automatically completed by the language model.

[0072] Based on the above embodiment, generating multiple training data pairs according to the professional knowledge text and its corresponding generated image, and constructing a training set includes:

[0073] According to each simplified text and its corresponding generated image, multiple training data pairs are constructed, and a training set is constructed.

[0074] Based on the above embodiment, the generating of multiple training data pairs further includes:

[0075] Collect real user feedback scores, simulated user feedback scores, and image quality scores for each generated image to generate a preference score for each generated image;

[0076] In order to train the reinforcement learning model, the present invention needs to construct preference training data pairs. This step involves collecting real user feedback from the generated images and simulating user feedback using an automated evaluation system.

[0077] Specifically, the present invention collects user preference data for generated images. For each image, we ask the user to evaluate and record it. We use the text descriptions corresponding to the images selected by the user as "good" samples and the unselected ones as "bad" samples to form preliminary preference training data.

[0078] In order to make up for the shortcomings of human annotation, the present invention has developed an automated image evaluation system that can simulate the human evaluation process, conduct preliminary evaluation of the generated images, and screen out candidate high-quality images. Specifically, the present invention uses different prompt engineering methods to perform diversified rewriting of 20,000 groups of fuzzy prompt texts randomly selected from the data set, and generate new prompts from different directions. The purpose of this step is to explore the impact of different rewriting strategies on the image generation effect, and how to better guide the model to generate images that meet user preferences.

[0079] The training data pairs corresponding to the generated images whose preference scores are greater than the third preset threshold are defined as the preferred training data pairs.

[0080] For each rewritten prompt, we apply the image generation model to generate the corresponding image. Based on advanced image evaluation models such as LAION-AES, MPS, and ImageReward, we score the generated sample images and collect preference data according to the score sequence. The present invention also uses this part of the automatically generated preference results as training data to enhance the generalization ability of the model.

[0081] Based on the above embodiment, the training of the prompt word rewriter based on the training set includes:

[0082] Based on the training set constructed by the preference training data, the prompt word rewriter is subjected to reinforcement learning training using a direct preference optimization algorithm.

[0083] The present invention uses an open source language model for efficient fine-tuning. Specifically, the present invention selects the popular ChatGLM and Llama models in academia for training. We use the standard maximization loss function to fine-tune on the collected data set to obtain πφ. The form of the maximization loss function is as follows:

[0084]

[0085] After constructing the preference training data pairs, the present invention uses the direct preference optimization (DPO) algorithm to perform reinforcement learning training on the rewriter. The core of the DPO algorithm is to optimize a loss function that measures the difference between the text description generated by the model and the user's preference. The loss function is defined as shown in the following formula:

[0086]

[0087] Among them, L(θ) represents the loss function, θ is the model parameter, E represents the expected value, D is the data set, x is the input image, y is the w is a text description of the user's preference, y 1 is a textual description of disfavor, π θ 9y w |x) indicates that the model generates a preferred text description y given an image x. w The probability of ref (y w|x) represents the probability of the reference model generating the same text description, θ is the sigmoid function, ensuring that the output value is between 0 and 1, and β is a hyperparameter that balances the weights of positive and negative samples. By maximizing this loss function, the model learns how to generate text prompts that are more preferred by users. This process involves continuous iterative optimization until the text description generated by the model can significantly improve the quality of the generated image.

[0088] The present invention proposes a prompt word optimization method for an image generation model based on professional knowledge distillation, which can successfully optimize the user's text prompt words into text with richer content and more professional knowledge, thereby significantly improving the image generation effect.

[0089] An embodiment of the present invention provides a device for optimizing prompt words of an image generation model; the specific device may include:

[0090] A prompt word acquisition module is used to obtain user input text;

[0091] The prompt word optimization module is used to input the user input text into the trained prompt word rewriter to obtain the target text.

[0092] The training process of the prompt word rewriter includes:

[0093] Collect multiple professional field pictures, and obtain picture descriptions corresponding to the multiple professional field pictures;

[0094] Input multiple image descriptions into the image generation model to generate multiple generated images;

[0095] The picture description corresponding to the generated image that meets the preset conditions is defined as the professional knowledge text;

[0096] According to the professional knowledge text and its corresponding generated image, a plurality of training data pairs are generated to construct a training set; based on the training set, the prompt word rewriter is trained until the loss function converges to obtain a trained prompt word rewriter.

[0097] The image generation model prompt word optimization device of this embodiment is used to implement the aforementioned image generation model prompt word optimization method. Therefore, the specific implementation method of the image generation model prompt word optimization device can be seen in the embodiment part of the aforementioned image generation model prompt word optimization method. For example, the prompt word acquisition module and the prompt word optimization module are respectively used to implement steps S101, S102, S103, S104 and S105 in the aforementioned image generation model prompt word optimization method. Therefore, its specific implementation method can refer to the description of the corresponding various parts of the embodiment, which will not be repeated here.

[0098] A specific embodiment of the present invention further provides an image generation model prompt word optimization device, comprising: a memory for storing a computer program; and a processor for implementing the steps of the above-mentioned image generation model prompt word optimization method when executing the computer program.

[0099] A specific embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned image generation model prompt word optimization method are implemented.

[0100] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0101] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0102] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0103] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1A step that specifies a function in one or more boxes.

[0104] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A prompt word optimization method for an image generation model, characterized in that: include: Get user input text; Input the user input text into the trained prompt word rewriter to obtain the target text, The training process of the prompt word rewriter includes: Collect multiple professional field pictures, and obtain picture descriptions corresponding to the multiple professional field pictures; Input multiple image descriptions into the image generation model to generate multiple generated images; The picture description corresponding to the generated image that meets the preset conditions is defined as the professional knowledge text; Generate multiple training data pairs based on the professional knowledge text and its corresponding generated image to construct a training set; Based on the training set, the prompt word rewriter is trained until the loss function converges to obtain a trained prompt word rewriter.

2. The image generation model prompt word optimization method according to claim 1, characterized in that: The collecting of multiple professional field pictures and obtaining picture descriptions corresponding to the multiple professional field pictures includes: Collecting multiple professional field images with quality scores greater than a first preset threshold; The multiple professional field images are input into a visual understanding model based on prompt engineering to generate multiple image descriptions.

3. The image generation model prompt word optimization method according to claim 1, characterized in that: The step of defining the picture description corresponding to the generated image that meets the preset conditions as professional knowledge text includes: The picture descriptions corresponding to the generated images whose quality scores are greater than the second preset threshold are collected to obtain a plurality of professional knowledge texts.

4. The image generation model prompt word optimization method according to claim 1, characterized in that: The step of defining the picture description corresponding to the generated image that meets the preset conditions as professional knowledge text further includes: Each professional knowledge text is input into a large language model based on prompt engineering to generate multiple simplified texts.

5. The image generation model prompt word optimization method according to claim 4, characterized in that: The step of generating a plurality of training data pairs according to the professional knowledge text and the corresponding generated image, and constructing a training set includes: According to each simplified text and its corresponding generated image, multiple training data pairs are constructed, and a training set is constructed.

6. The image generation model prompt word optimization method according to claim 1, characterized in that: The generating of a plurality of training data pairs further comprises: Collect real user feedback scores, simulated user feedback scores, and image quality scores for each generated image to generate a preference score for each generated image; The training data pair corresponding to the generated image whose preference score is greater than the third preset threshold is defined as a preferred training data pair.

7. The image generation model prompt word optimization method according to claim 6, characterized in that: The training of the prompt word rewriter based on the training set includes: Based on the training set constructed by the preference training data, the prompt word rewriter is subjected to reinforcement learning training using a direct preference optimization algorithm.

8. An image generation model prompt word optimization device, characterized in that: include: A prompt word acquisition module is used to obtain user input text; The prompt word optimization module is used to input the user input text into the trained prompt word rewriter to obtain the target text. The training process of the prompt word rewriter includes: Collect multiple professional field pictures, and obtain picture descriptions corresponding to the multiple professional field pictures; Input multiple image descriptions into the image generation model to generate multiple generated images; The picture description corresponding to the generated image that meets the preset conditions is defined as the professional knowledge text; Generate multiple training data pairs based on the professional knowledge text and its corresponding generated image to construct a training set; Based on the training set, the prompt word rewriter is trained until the loss function converges to obtain a trained prompt word rewriter.

9. An image generation model prompt word optimization device, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of the image generation model prompt word optimization method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image generation model prompt word optimization method as described in any one of claims 1 to 7 are implemented.

Images