Prompt word intelligent optimization method and system for text-to-image generation

Abstract

The application discloses a prompt word intelligent optimization method and system for text generated images. The method comprises the following steps: obtaining input text input by a user, inputting the input text into a text classification model to obtain a main intention of the input text, wherein the main intention is used to represent a drawing intention of the user, and the drawing intention includes a character or does not include a character; inputting the input text into a pre-trained text-to-text generation model, and outputting a supplemented prompt text based on generation parameters of the text-to-text generation model; if the main intention does not include a character, removing artist names with a tendency to draw characters in the prompt text according to a pre-generated artist list, and obtaining an optimized input prompt. The application solves the technical problems that the related art has low efficiency, high trial and error cost and cannot accurately supplement the intention of the user input prompt, and realizes high efficiency, low trial and error cost and accurate supplement of the intention of the user input prompt.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and specifically relates to a method and system for intelligent optimization of prompt words for text-to-image generation. Background Technology

[0002] In recent years, AI's ability to generate images from text has matured significantly, especially with the Stable Diffusion model, which has greatly improved both the speed and quality of image generation. However, in AI's ability to generate images from text, prompts (input hints) have a significant impact on the quality of the generated images (prompt engineering is a formal search for hints from a language model to retrieve the desired result, where the desired result depends on the final task and the end user). In particular, some descriptive prompts can greatly enhance the aesthetic appeal of the generated image. Moreover, most user-input prompts are insufficient and cannot fully utilize the creative capabilities of Stable Diffusion, making supplementary prompts essential.

[0003] In related technologies, methods for supplementing and optimizing prompts are inefficient, have high trial-and-error costs, and cannot accurately supplement the user's intention in inputting a prompt.

[0004] There is currently no effective solution to the technical problems of inefficient and costly methods for supplementing and optimizing prompts in related technologies, which fail to accurately supplement the intent of user-input prompts. Summary of the Invention

[0005] Therefore, the present application provides a method and system for intelligent optimization of prompt words for text-to-image generation, aiming to solve at least one problem existing in the prior art.

[0006] To achieve the above objectives, in a first aspect, this application provides a method for intelligent optimization of prompt words for text-to-image generation, comprising:

[0007] The user input text is obtained and fed into a text classification model to obtain the main intent of the input text. The main intent is used to characterize whether the user's drawing intent includes people or not.

[0008] The input text is fed into a pre-trained text-to-text generation model, and the supplemented prompt text is output based on the generation parameters of the text-to-text generation model.

[0009] If the intended subject is not to include figures, then the names of artists with a tendency to draw figures are removed from the prompt text according to the pre-generated list of artists, resulting in an optimized input prompt; wherein, the list of artists includes multiple artist names with a tendency to draw figures.

[0010] In one embodiment, the method further includes: acquiring a training sample set, extracting the main intent words of each input prompt word text in the sample set, and determining the main intent of each input prompt word text based on the main intent words;

[0011] The target input prompt text is obtained to obtain pre-training sample data, wherein the target input prompt text is all input prompt texts whose sentence begins with the main intent word; the pre-training sample data is processed using undersampling technology to make the proportion of samples with main intent including people and those without people consistent, thereby obtaining training sample data;

[0012] The pre-trained text-to-text generation model is obtained by using the main intent word of each input prompt word in the training sample data as input to the text-to-text generation model.

[0013] In one embodiment, the step of using the main intent word of each input prompt word text in the training sample data as input to the text-to-text generation model to obtain the pre-trained text-to-text generation model includes: using the main intent word of each input prompt word text in the training sample data as input to the text-to-text generation model; the text-to-text generation model continues the main intent word to obtain a complete training prompt word; calculating the cross-entropy loss between the training prompt word and each word of the corresponding input prompt word text; backfeeding the cross-entropy loss to the text-to-text generation model and using an optimizer to correct the model weights; stopping training to obtain the text-to-text generation model when the cross-entropy loss no longer decreases within N steps, where N is a preset value and is a positive integer.

[0014] In one embodiment, extracting the main intent word of each input prompt text in the sample set includes: dividing each input prompt text or prompt text into multiple segments using symbols as dividing points, counting the frequency of each segment, and taking the segment whose frequency meets the threshold and is the beginning of the sentence of the input prompt text as the main intent word.

[0015] In one embodiment, the text classification model is a zero-shot text classification model.

[0016] In one embodiment, the method further includes: using a decision tree to fit the artist names and corresponding subject intentions in the input prompt text of the sample set to obtain the influence value of each artist name on the generation of images with human attributes; filtering artist names that appear less than a preset value in the sample set; calculating the drawing frequency of each artist name based on the number of times each artist name draws a human and the number of times it appears in the sample set; and adding artist names with influence values ​​greater than the mean influence value and drawing frequency higher than the preset value to the list to generate the artist list.

[0017] In one embodiment, if the subject's intent is to include a person, then the prompt text is directly used as the input prompt.

[0018] Secondly, this application also provides a prompt word intelligent optimization system for text-to-image generation, comprising:

[0019] The acquisition unit is used to acquire the input text input by the user, input the input text into the text classification model to obtain the main intent of the input text, and the main intent is used to characterize whether the user's drawing intent includes people or not.

[0020] The processing unit is used to input the input text into a pre-trained text-to-text generation model and output supplemented prompt text based on the generation parameters of the text-to-text generation model.

[0021] The generation unit is configured to, if the subject's intent is not to include figures, remove artist names with a tendency to draw figures from the prompt text based on a pre-generated artist list, thereby obtaining an optimized input prompt; wherein the artist list includes multiple interfering artist names with a tendency to draw figures.

[0022] This application provides an intelligent optimization method and system for prompts in text-to-image generation. The method involves acquiring user-input text, inputting it into a text classification model to obtain the main intent of the input text, which characterizes whether the user's drawing intention includes or does not include figures. The input text is then input into a pre-trained text-to-text generation model, which outputs supplementary prompt text based on the model's generation parameters. If the main intent does not include figures, artist names with a tendency to draw figures are removed from the prompt text according to a pre-generated artist list, resulting in optimized prompts. The artist list includes multiple interfering artist names with a tendency to draw figures. This method solves the technical problems of low efficiency, high trial-and-error costs, and inaccurate supplementation of user input prompt intent in related technologies. It achieves the following beneficial effects: by analyzing the user's main drawing intent in the input text, the supplementary prompts accurately express the user's intent; by pre-training an advanced text generation model, the user's trial-and-error costs are reduced, improving the efficiency and accuracy of text supplementation; and by filtering artist names, user intent is prevented from being altered by artist style. Overall, it achieves high efficiency, low trial-and-error costs, and accurate supplementation of the user's input prompt intent. Attached Figure Description

[0023] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application. In the drawings:

[0024] Figure 1 The implementation flow of the intelligent optimization method for prompt words in text-to-image generation provided in the embodiments of this application;

[0025] Figure 2 A flowchart illustrating the intelligent optimization method for generating images from text provided in this application embodiment;

[0026] Figure 3 A schematic diagram of the main modules of the intelligent optimization system for generating images from text provided in this application embodiment;

[0027] Figure 4 An exemplary system architecture diagram provided for application in the embodiments of this application;

[0028] Figure 5 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers in the embodiments of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] In this application, the terms "upper," "lower," "left," "right," "front," "rear," "top," "bottom," "inner," "outer," "middle," "vertical," "horizontal," "lateral," and "longitudinal" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are primarily for the purpose of better describing this application and its embodiments, and are not intended to limit the indicated device, element, or component to having a specific orientation, or to be constructed and operated in a specific orientation.

[0032] Furthermore, in addition to indicating location or positional relationship, some of the aforementioned terms may also have other meanings. For example, the term "above" may also be used in some cases to indicate a certain dependency or connection relationship. Those skilled in the art can understand the specific meaning of these terms in this application based on the specific circumstances.

[0033] In addition, the term "multiple" should mean two or more.

[0034] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0035] Figure 1 This paper illustrates the implementation flow of a method for intelligent optimization of prompt words for text-to-image generation, provided in an embodiment of this application. Figure 2The diagram illustrates the overall flowchart of the intelligent optimization method for generating images from text provided in this application. For ease of explanation, only the parts relevant to the embodiments of this application are shown, and are detailed below:

[0036] In recent years, AI's ability to generate images from text has matured, especially the Stable Diffusion model (StableDiffusion is an open-source model from Stability AI used for image generation), which has greatly improved both the speed and quality of image generation. However, in AI's ability to generate images from text, prompts (input hints) have a significant impact on the quality of the generated images (prompt engineering is a formal search for hints from a language model to retrieve the desired result, where the desired result depends on the final task and the end user). In particular, some descriptive hints can greatly enhance the aesthetics of the generated image; moreover, most user-input prompts are insufficient and cannot fully utilize the creative capabilities of Stable Diffusion. Supplementing the prompt is essential. For example, the original prompt input text is "Nezha," while the optimized prompt is "Nezha, D&D, fantasy, portrait, highly detailed, digital painting, trending on artstation, concept art, sharp focus, illustration, art by Artgermand Greg Rutkowski and Alphonse Mucha." The resulting image style differs significantly between the original and optimized prompt text. Therefore, refined image embellishments can substantially improve the quality of the generated image. This application provides a smart prompt supplementation and optimization capability for Stable Diffusion, enabling it to create higher-quality images.

[0037] Here, the optimization and supplementation of the prompt can be divided into the following categories:

[0038] 1. Provide supplementary words selection: After the user enters the original prompt, provide a list of supplementary modifiers, allowing the user to choose the prompt words to fill in (such as prompttomania prompt-builder);

[0039] 2. Finding near-high-quality prompts: On some text-to-image prompt collection websites, search for prompts that are relevant to the user's intent as input for the current generation task;

[0040] 3. Generative model supplementation of prompts: Train a prompt generation model (such as the open-source model MagicPrompt) on a publicly available text-to-image prompt dataset to supplement basic user input descriptions.

[0041] The above methods can gradually solve the predicament of users lacking supplementary words when drawing, but they also have different problems:

[0042] 1. Provide a method for selecting supplementary words:

[0043] a. Some tools can directly provide all available suggestions, often offering users thousands of words to choose from. Different word combinations also have a significant impact on the generated quality, thus increasing the trial-and-error cost for users.

[0044] b. Some tools have organized and categorized modifiers and supplementary artists, which can make the provided vocabulary more organized. However, organizing and categorizing requires a lot of human knowledge and processing. Moreover, this method is not an end-to-end one-step word supplementation and also requires some trial and error costs.

[0045] 2. Find methods to approximate high-quality prompts, providing users with readily available and high-quality prompts while:

[0046] a. On the one hand, it limits or alters the user's original intention to varying degrees. For example, the user may want "Iron Man with Venom", but the approximate prompt that can be found is "venom in a venom-inspired ironmansuit, black and red, dynamic lighting, photorealistic fantasy concept art, trending on art station, stunning visuals, terrifying, creative, cinematic".

[0047] b. On the other hand, it also faces the problem of cold start when there is no approximate prompt;

[0048] 3. Generative model prompt supplementation methods (such as the open-source model MagicPrompt), while largely able to supplement with appropriate words to suit the intent and ensure the quality of the generated image, suffer from imbalances in the intent of the training set data and the main intent being buried in the middle or end of the text, leading to:

[0049] a. The prompt output by the model may deviate from the user's original intent, adding unnecessary elements. For example, if the original input is "An apple", the model output may be: "An apple, green forest, clearing. Beige leaves. In style of Yoji Shinkawa and Hyung-tae Kim, trending on ArtStation, dark fantasy, great composition, concept art, highly detailed." This will reduce the intent of "An apple" in the generated image and add elements such as "green forest".

[0050] b. Additional supplementary words by artists skilled in drawing people were added. In the example of "An apple," the addition of artists such as "Yoji Shinkawa" caused the model generating the image to be more inclined to draw a person rather than "apple." This kind of discrepancy between random text input and generated image results is quite common. Therefore, the solution disclosed in this application can be a prompt intelligent supplementation optimization capability. It uses an advanced text generation model, which has been fine-tuned on our massive collection of prompt word datasets, and includes data preprocessing and post-processing processes, correcting the two shortcomings of other technologies in changing intent and drawing person tendencies.

[0051] The intelligent supplementation optimization method for prompts in the application may include the following steps:

[0052] 1. Data analysis and cleaning: distinguish the drawing intention type of the prompt in the dataset, whether the drawing intention includes people or not;

[0053] 2. Based on the intention type of the prompt, analyze the tendency of each artist to generate characters in the results;

[0054] 3. Based on the intent type of the prompt, perform data balancing for prompts that contain people and those that do not;

[0055] 4. On the cleaned and balanced data, the main intention, i.e. the painting intention, is used as the model input, and the complete prompt is used as the model training target. The T5 model (T5: Text-to-Text Transfer Transformer) is used for fine-tuning and training.

[0056] 5. In the model application stage, first distinguish the type of user input intent. For the prompt results generated for intents that do not contain people, filter out the names of artists with a high probability of depicting people.

[0057] In one embodiment of this application, a method for intelligent optimization of prompt words for text-to-image generation is provided, comprising the following steps:

[0058] S101: Obtain the input text input by the user, input the input text into the text classification model to obtain the main intent of the input text, the main intent being used to characterize whether the user's drawing intent includes people or not;

[0059] S102: Input the input text into a pre-trained text-to-text generation model, and output the supplemented prompt text based on the generation parameters of the text-to-text generation model;

[0060] S103: If the subject's intention is not to include people, then according to the pre-generated artist list, the names of artists with a tendency to draw people are removed from the prompt text to obtain the optimized input prompt; wherein, the artist list includes multiple interfering artist names with a tendency to draw people.

[0061] In step S101: The user's input text is obtained, and the input text is input into a text classification model to obtain the main intent of the input text. The main intent is used to characterize whether the user's drawing intention includes or does not include people. The input text is the original prompt input text entered by the user. After the user enters the original prompt input text before image generation, the user's input text is obtained and input into a preset text classification model to obtain the main intent of the user's input text. The main intent is the drawing intention of the user's input text, which includes whether the user's drawing intention is to draw an image containing people or an image not containing people.

[0062] Here, a zero-shot text classification model (which requires no training) can be used to distinguish the intent of sentences in the input text. The target categories for a zero-shot text classification model include the following:

[0063] 1["human",

[0064] 2"portrait",

[0065] 3"comics animation movie characters",

[0066] 4 "goods",

[0067] 5 "food"

[0068] 6"fruit",

[0069] 7"electronic product",

[0070] 8"product",

[0071] 9"traffic",

[0072] 10 "city",

[0073] 11"natural",

[0074] 12 "animal",

[0075] 13"tools"];

[0076] When the model determines that the subject's intent is one of ["human", "portrait", "comics animation movie characters"], the subject's intent can be determined to include people; other categories can be determined to not include people.

[0077] In step S102: The input text is fed into a pre-trained text-to-text generation model, and supplemented prompt text is output based on the generation parameters of the text-to-text generation model. After classifying the intent of the user's input text, the input text is fed into a pre-trained text generation model. Based on the pre-set generation parameters, the user's input text is intelligently supplemented and optimized to obtain the optimized complete prompt text, i.e., the optimized prompt input text, so as to generate a high-quality image that matches the user's drawing intent. Here, the generation parameters of the text-to-text generation model are the settings when the deep learning model samples the sequence logits of the model output results into a language sequence during training for inference. Given an input, the model weights are calculated, and the model will obtain the final output matrix, and the model's work is complete. At this time, the final complete output sentence cannot be obtained yet. The matrix output by the model is given to the sequence sampler, and a greedy search strategy or a beam search strategy is used to decode the vector at each position into a specific character, finally obtaining the final output text sequence. The generation parameters are the parameters of the strategy during sequence sampling.

[0078] In one embodiment, the generation parameters of the text-to-text generation model are tuned based on the model's accuracy performance. Specifically, the generation parameters of the model are set as follows:

[0079] i.do_sample:True (This increases the diversity of generated results);

[0080] ii.early_stopping: True (Only generate results within the range with the highest probability);

[0081] iii.length_penalty: 1.0 (The model will be more inclined to generate shorter and more accurate results);

[0082] Furthermore, by combining the subject's intent classification, the length of the model's generated data is controlled:

[0083] i. When the intent is ["human","portrait","comics animation movie characters"], the minimum complement length is set to 70 (Stable Diffusion's text encoder can accept a maximum of 77 tokens), which can supplement more portrait type artists and embellishments;

[0084] ii. When the intent is ["food","fruit","electronic product","product","tools"], the minimum complement length is set to 30. The model outputs fewer and more accurate modifiers, which can make stable diffusion avoid people and rarely add artists.

[0085] iii. Other intentions, such as "city", etc., with the minimum complement length set to 50, and balanced complement modifiers and artists.

[0086] Therefore, after obtaining the user's input text and distinguishing the main intent of the input text, parameters can be generated directly according to the model settings to supplement the input text with prompts. That is, the user's input text (such as "An apple") is fed into the trained T5 model, and the model supplements the input text to complete the prompt.

[0087] It should be noted that the pre-trained text-to-text generation model (T5 model) can be obtained by fine-tuning the T5 model (T5: Text-to-Text Transfer Transformer).

[0088] Before training the model, data preparation was carried out: In this application, a large amount of pre-collected prompt word data was used, including 140,000+ prompt texts that can generate high-quality images. Examples: prompt 1 "Anthropomorphized pineapple pizza, D&D, fantasy, cinematic lighting, highly detailed, digital painting, artstation, concept art, smooth, sharp focus, illustration, warm light, cozy warm tint, magic the gathering artwork, volumetric lighting, 8k, no gold, no gold colours, art by Akihiko Yoshida, Greg Rutkowski"; prompt 2 "a fox fursona, trending on pixiv, by kawacy, furry art, digital art, cyberpunk, high quality, backlighting".

[0089] Next, perform data analysis on the prepared data: In the example above, a high-quality prompt generally consists of three parts (taking the first example, prompt1, as an example):

[0090] 1. The main intent, which dominates the main content of the generated result and is also the main intent of the user input, such as: "Anthropomorphized pineapple pizza";

[0091] 2. Image style enhancement: Adding to the image style can improve the image quality. For example: "D&D, fantasy, cinematic lighting, highly detailed, digital painting, artstation, concept art, smooth, sharp focus, illustration, warm light, cozy warm tint, magic the gathering artwork, volumetric lighting, 8k, no gold, no gold colors";

[0092] 3. Artist's supplement: The artist's name can dominate the overall style of the painting, for example: "art by Akihiko Yoshida, Greg Rutkowski".

[0093] Here, the prompt dataset of over 140,000 characters exhibits the following pattern:

[0094] 1. The stylistic modifications in the images are mostly a combination of multiple descriptive words for the images, with each word appearing repeatedly throughout the dataset;

[0095] 2. Artist supplementation is also a concatenation of multiple artist names, with each artist name appearing multiple times throughout the dataset;

[0096] 3. In the main intent descriptions of this dataset, the intent to draw people accounts for more than 50%.

[0097] To avoid the output of the text-generating model being biased towards the prompts for drawing people, some preprocessing and cleaning operations are needed for the data.

[0098] Specifically, through the preparation and analysis of the sample dataset, the T5 model can be trained using the following method: Obtain a training sample set; extract the main intent word from each input prompt text in the sample set; determine the main intent of each input prompt text based on the main intent word; obtain target input prompt texts to obtain pre-training sample data, where the target input prompt texts are all input prompt texts whose sentence begins with the main intent word; process the pre-training sample data using undersampling techniques to ensure that the proportion of samples with and without main intent words is consistent, obtaining training sample data, maintaining a 1:1 ratio of prompts containing and without main intent words; use the main intent word of each input prompt text in the training sample data as input to the text-to-text generation model for training to obtain the pre-trained text-to-text generation model. Here, only prompt data with main intent words at the beginning of sentences are taken to avoid the generative model supplementing additional main vocabulary during training.

[0099] Furthermore, the step of using the main intent word of each input prompt text in the training sample data as input to the text-to-text generation model to obtain the pre-trained text-to-text generation model includes: using the main intent word of each input prompt text in the training sample data as input to the text-to-text generation model; the text-to-text generation model continues the main intent word to obtain a complete training prompt word; calculating the cross-entropy loss between the training prompt word and each word of the corresponding input prompt text; determining whether the current cross-entropy loss has decreased compared to the previous cross-entropy loss; if so, the cross-entropy loss is backpropagated to the text-to-text generation model, and the model weights are corrected using an optimizer, and the above steps are repeated until the cross-entropy loss no longer decreases within N steps, at which point training stops to obtain the text-to-text generation model, where N is a preset value and a positive integer. Here, model weights refer to the model weight optimization strategy in deep learning training methods, mainly using the AdamW optimizer to calculate the loss gradient and correct the model weights.

[0100] In one embodiment, extracting the main intent word of each input prompt text in the sample set includes: dividing each input prompt text or prompt text into multiple segments using symbols as dividing points, counting the frequency of each segment, and taking the segment whose frequency meets the threshold and is the beginning of the sentence of the input prompt text as the main intent word.

[0101] In one embodiment, the method further includes: fitting the artist names and corresponding subject intentions in the input prompt text of the sample set using a decision tree to obtain the influence value of each artist name on the generation of images with human attributes; filtering artist names whose frequency of occurrence in the sample set is less than a preset value; calculating the drawing frequency of each artist name based on the number of times each artist name draws a human and the number of times it appears in the sample set; and adding artist names with influence values ​​greater than the mean influence value and drawing frequency of human higher than the preset value to a list to generate the artist list. Here, fitting the artist names and corresponding subject intentions in the input prompt text of the sample set using a decision tree can be done by first vectorizing the artist names using a TF-IDF Vectorizer (based on TF-IDF (Term Frequency-Inverse Document Frequency) feature vector extraction), and then inputting the vectorized artist names into the decision tree. The classification objective of the decision tree is to classify the images as containing human figures or not containing human figures. The influence value can range from 0 to 1, with 1 being the highest influence value.

[0102] In a specific implementation, after preparing and analyzing the sample dataset, the intent of the prompts in the sample dataset is classified. Specifically, to identify the intent type of the prompt, the main intent portion of the prompt must first be extracted, and the process is as follows:

[0103] 1. Divide each prompt into smaller segments using symbols;

[0104] 2. Count the frequency of each segment;

[0105] 3. Select the segment that appears only 5 times or less and is the beginning of the prompt; this is the main intent.

[0106] Next, the subject's intent is categorized as follows: A zero-shot text classification model (no training required) is used to distinguish the intent of the sentence; when the model determines that the subject's intent is one of ["human", "portrait", "comics animation movie characters"], the intent is classified as containing people; the other categories are classified as not containing people.

[0107] In addition, the sample dataset needs to be balanced and cleaned. Specifically, after classifying the intent of the prompt, the data is processed as follows:

[0108] 1. Only take prompt data where the main intent is at the beginning of the sentence to avoid the generative model supplementing additional main vocabulary during training and learning;

[0109] 2. In the cleaned data as described above, undersampling is used to maintain a 1:1 ratio of prompts containing people to those not containing people. This yields over 30,000 cleaned and balanced high-quality prompts.

[0110] Thus, the aforementioned training sample data was generated.

[0111] It should be noted that the bone intervention training language model used in this application is Google's open-source generative model: T5 (Text-to-Text Transfer Transformer). This model is currently the most advanced generative model. It is a generative model with an Encoder and a Decoder, which can expand or translate the input text into new text. It can be applied to downstream tasks such as classification, summarization, question answering, translation, and continuation writing.

[0112] Specifically, the training process and supervision methods of this application are as follows:

[0113] 1. In the cleaned balanced dataset, use the main intent of each prompt as the model input;

[0114] 2. The T5 model will continue writing the main intent to a complete prompt;

[0115] 3. The supervised process evaluates the cross-entropy loss between the prompt generated by the model in each step and each token (word) of the real complete prompt. The loss is then fed back to the model to use a weight optimization method to correct the model weights until the model loss no longer decreases within N steps, at which point training stops.

[0116] Furthermore, during the model training phase, the artists in the sample dataset can be analyzed simultaneously. Experiments revealed that for Stable Diffusion (a diffusion model), even if the input prompt does not contain information about painting figures, as long as the prompt includes the name of an artist who frequently paints figures, the generated result has a high probability of producing an image with a figure. Therefore, it is necessary to analyze the tendency of artist names to paint figures, and the process is as follows:

[0117] 1. By fitting a decision tree to the artist names and intention categories (including people and not including people) in the prompt, we can obtain the feature_importance of each artist, that is, the degree of influence of the artist on painting people.

[0118] 2. Count the frequency of artists drawing figures: Number of times figures were drawn / Number of times the artist appeared;

[0119] 3. Filter artists who appear less than 30 times in the statistics (too low a frequency is not statistically significant);

[0120] 4. When an artist's feature_importance is greater than the mean and the frequency of drawing people is higher than 50%, that is, when it is determined that the prompt contains the artist, an image with people will be output.

[0121] 5. Adding the artist names obtained in step 4 to the list will give you a list of artists who tend to paint figures.

[0122] In step S103: If the main intent is not to include figures, then the names of artists with a tendency to paint figures are removed from the prompt text according to the pre-generated artist list, resulting in an optimized input prompt; wherein, the artist list includes multiple interfering artist names with a tendency to paint figures. If the main intent includes figures, then the prompt text is directly used as the input prompt. Therefore, based on the main intent category, if the user input is determined not to include figures, the prompt result generated by the model is further post-processed to filter out artists with a tendency to paint figures obtained in the artist analysis step; if the user input is determined to include figures, no processing is performed. Thus, the final prompt result is output.

[0123] The intelligent prompt optimization method in this application is an end-to-end prompt optimization method compared to providing optional words and finding approximate prompts. Compared to existing open-source generative prompt optimization methods, it trains a generative model through balanced sampling of data, ensuring that the model does not change the original painting intent. Furthermore, it supplements prompts that improve the Stable Diffusion painting effect without altering the intent, adding a post-processing step to make the Stable Diffusion painting result more stable and prevents the output of images with figures when there is no input intent for figures. Moreover, it distinguishes various parts of the prompt through segmentation and frequency statistics: the main intent, image style modification, and artist supplementation. Using an open-source pre-trained model, it differentiates the main intent of the prompt and defines 13 categories of painting main intents, analyzes the influence of each artist on the inclusion of figures in the image generation result, and applies this analysis in the post-processing process.

[0124] Therefore, the intelligent optimization method for prompt words in text-to-image generation provided in this application obtains the user's input text, inputs the input text into a text classification model to obtain the main intent of the input text, which represents whether the user's drawing intent includes or does not include figures; inputs the input text into a pre-trained text-to-text generation model, and outputs supplemented prompt word text based on the generation parameters of the text-to-text generation model; if the main intent does not include figures, then according to a pre-generated artist list, artist names with a tendency to draw figures are removed from the prompt word text to obtain optimized input prompt words; wherein, the artist list includes multiple artist names with a tendency to draw figures. This solves the technical problems of low efficiency, high trial-and-error cost, and inability to accurately supplement the user's input prompt intent in related technologies, and achieves the following beneficial effects: by analyzing the user's main drawing intent in the input text, the supplemented input prompt words accurately express the user's intent; by pre-training an advanced text generation model, the user's trial-and-error cost is reduced, and the efficiency and accuracy of text supplementation are improved; by filtering artist names, the user's intent is avoided due to changes in artist style. Overall, it achieves high efficiency, low trial-and-error costs, and accurate supplementation of the user's input prompt intent.

[0125] Figure 3 The diagram illustrates the main modules of the intelligent optimization system for generating images from text provided in this application. For ease of explanation, only the parts relevant to this application are shown, and are described in detail below:

[0126] A prompt word intelligent optimization system 200 for text-to-image generation includes:

[0127] The acquisition unit 201 is used to acquire the input text input by the user, input the input text into the text classification model to obtain the main intent of the input text, and the main intent is used to characterize whether the user's drawing intent includes people or not.

[0128] The processing unit 202 is used to input the input text into a pre-trained text-to-text generation model and output supplemented prompt text based on the generation parameters of the text-to-text generation model.

[0129] The generation unit 203 is used to remove artist names with a tendency to draw people from the prompt text according to a pre-generated artist list if the subject's intention is not to include people, thereby obtaining an optimized input prompt; wherein the artist list includes multiple interfering artist names with a tendency to draw people.

[0130] For the acquisition unit 201: it is used to acquire the input text input by the user, and input the input text into a text classification model to obtain the main intent of the input text. The main intent is used to characterize whether the user's drawing intention includes people or not. The input text is the original prompt input text input by the user. After the user inputs the original prompt input text before image generation, the input text input by the user is acquired and input into a preset text classification model to obtain the main intent of the user's input text. The main intent is the drawing intention of the user's input text, which includes whether the user's drawing intention is to draw an image containing people or an image not containing people.

[0131] Here, a zero-shot text classification model (which requires no training) can be used to distinguish the intent of sentences in the input text. The target categories for the zero-shot text classification model include the following:

[0132] 1["human",

[0133] 2"portrait",

[0134] 3"comics animation movie characters",

[0135] 4 "goods",

[0136] 5 "food"

[0137] 6"fruit",

[0138] 7"electronic product",

[0139] 8"product",

[0140] 9"traffic",

[0141] 10 "city",

[0142] 11"natural",

[0143] 12 "animal",

[0144] 13"tools"];

[0145] When the model determines that the subject's intent is one of ["human", "portrait", "comics animation movie characters"], the subject's intent can be determined to include people; other categories can be determined to not include people.

[0146] For processing unit 202: it is used to input the input text into a pre-trained text-to-text generation model, and output supplemented prompt text based on the generation parameters of the text-to-text generation model. After classifying the intent of the user's input text, the input text is input into a pre-trained text generation model, and the user's input text is intelligently supplemented and optimized based on the pre-set generation parameters to obtain optimized prompt text, that is, optimized prompt input text, so as to generate a high-quality image that matches the user's drawing intent.

[0147] It should be noted that the pre-trained text-to-text generation model (T5 model) can be obtained by fine-tuning the T5 model (T5: Text-to-Text Transfer Transformer).

[0148] Before training the model, data preparation was carried out: In this application, a large amount of pre-collected prompt word data was used, including 140,000+ prompt texts that can generate high-quality images. Examples: prompt 1 "Anthropomorphized pineapple pizza, D&D, fantasy, cinematic lighting, highly detailed, digital painting, artstation, concept art, smooth, sharp focus, illustration, warm light, cozy warm tint, magic the gathering artwork, volumetric lighting, 8k, no gold, no gold colours, art by Akihiko Yoshida, Greg Rutkowski"; prompt 2 "a fox fursona, trending on pixiv, by kawacy, furry art, digital art, cyberpunk, high quality, backlighting".

[0149] Next, perform data analysis on the prepared data: In the example above, a high-quality prompt generally consists of three parts (taking the first example, prompt1, as an example):

[0150] 1. The main intent, which dominates the main content of the generated result and is also the main intent of the user input, such as: "Anthropomorphized pineapple pizza";

[0151] 2. Image style enhancement: Adding to the image style can improve the image quality. For example: "D&D, fantasy, cinematic lighting, highly detailed, digital painting, artstation, concept art, smooth, sharp focus, illustration, warm light, cozy warm tint, magic the gathering artwork, volumetric lighting, 8k, no gold, no gold colors";

[0152] 3. Artist's supplement: The artist's name can dominate the overall style of the painting, for example: "art by Akihiko Yoshida, Greg Rutkowski".

[0153] Here, the prompt dataset of over 140,000 characters exhibits the following pattern:

[0154] 1. The stylistic modifications in the images are mostly a combination of multiple descriptive words for the images, with each word appearing repeatedly throughout the dataset;

[0155] 2. Artist supplementation is also a concatenation of multiple artist names, with each artist name appearing multiple times throughout the dataset;

[0156] 3. In the main intent descriptions of this dataset, the intent to draw people accounts for more than 50%.

[0157] To avoid the output of the text-generating model being biased towards the prompts for drawing people, some preprocessing and cleaning operations are needed for the data.

[0158] Specifically, through the preparation and analysis of the sample dataset, the T5 model can be trained using the following method: Obtain a training sample set; extract the main intent word from each input prompt text in the sample set; determine the main intent of each input prompt text based on the main intent word; obtain target input prompt texts to obtain pre-training sample data, where the target input prompt texts are all input prompt texts whose sentence begins with the main intent word; process the pre-training sample data using undersampling techniques to ensure that the proportion of samples with and without main intent words is consistent, obtaining training sample data, maintaining a 1:1 ratio of prompts containing and without main intent words; use the main intent word of each input prompt text in the training sample data as input to the text-to-text generation model for training to obtain the pre-trained text-to-text generation model. Here, only prompt data with main intent words at the beginning of sentences are taken to avoid the generative model supplementing additional main vocabulary during training.

[0159] Furthermore, the step of using the main intent word of each input prompt word text in the training sample data as input to the text-to-text generation model to obtain the pre-trained text-to-text generation model includes: using the main intent word of each input prompt word text in the training sample data as input to the text-to-text generation model; the text-to-text generation model continues the main intent word to obtain a complete training prompt word; calculating the cross-entropy loss between the training prompt word and each word of the corresponding input prompt word text; determining whether the current cross-entropy loss has decreased compared to the previous cross-entropy loss; if so, the cross-entropy loss is fed back to the text-to-text generation model and the generation parameters are corrected, and the above steps are repeated until the cross-entropy loss no longer decreases within N steps, at which point training stops to obtain the text-to-text generation model, where N is a preset value and is a positive integer.

[0160] In one embodiment, extracting the main intent word of each input prompt text in the sample set includes: dividing each input prompt text or prompt text into multiple segments using symbols as dividing points, counting the frequency of each segment, and taking the segment whose frequency meets the threshold and is the beginning of the sentence of the input prompt text as the main intent word.

[0161] In one embodiment, the method further includes: using a decision tree to fit the artist names and corresponding subject intentions in the input prompt text of the sample set to obtain the influence value of each artist name on the generation of images with human attributes; filtering artist names that appear less than a preset value in the sample set; calculating the drawing frequency of each artist name based on the number of times each artist name draws a human and the number of times it appears in the sample set; and adding artist names with influence values ​​greater than the mean influence value and drawing frequency higher than the preset value to the list to generate the artist list.

[0162] In a specific implementation, after preparing and analyzing the sample dataset, the intent of the prompts in the sample dataset is classified. Specifically, to identify the intent type of the prompt, the main intent portion of the prompt must first be extracted, and the process is as follows:

[0163] 1. Divide each prompt into smaller segments using symbols;

[0164] 2. Count the frequency of each segment;

[0165] 3. Select the segment that appears only 5 times or less and is the beginning of the prompt; this is the main intent.

[0166] Next, the subject's intent is categorized as follows: A zero-shot text classification model (no training required) is used to distinguish the intent of the sentence; when the model determines that the subject's intent is one of ["human", "portrait", "comics animation movie characters"], the intent is classified as containing people; the other categories are classified as not containing people.

[0167] In addition, the sample dataset needs to be balanced and cleaned. Specifically, after classifying the intent of the prompt, the data is processed as follows:

[0168] 1. Only take prompt data where the main intent is at the beginning of the sentence to avoid the generative model supplementing additional main vocabulary during training and learning;

[0169] 2. In the cleaned data as described above, undersampling is used to maintain a 1:1 ratio of prompts containing people to those not containing people. This yields over 30,000 cleaned and balanced high-quality prompts.

[0170] Thus, the aforementioned training sample data was generated.

[0171] It should be noted that the bone intervention training language model used in this application is Google's open-source generative model: T5 (Text-to-Text Transfer Transformer). This model is currently the most advanced generative model, featuring both an Encoder and a Decoder. It can expand or translate input text into new text and is applied in classification, summarization, question answering, translation, and other fields.

[0172] Continue writing and other downstream tasks.

[0173] Specifically, the training process and supervision methods of this application are as follows:

[0174] 1. In the cleaned balanced dataset, use the main intent of each prompt as the model input;

[0175] 2. The T5 model will continue writing the main intent to a complete prompt;

[0176] 3. The supervised process evaluates the cross-entropy loss between the prompt generated by the model in each step and each token (word) of the real complete prompt. The loss is then fed back to the model to use a weight optimization method to correct the model weights until the model loss no longer decreases within N steps, at which point training stops.

[0177] Furthermore, during the model training phase, the artists in the sample dataset can be analyzed simultaneously. Experiments revealed that for Stable Diffusion, even if the input prompt does not contain information about painting figures, as long as the prompt includes the name of an artist who frequently paints figures, the generated result has a high probability of producing an image with a figure. Therefore, it is necessary to analyze the tendency of artist names to paint figures, and the process is as follows:

[0178] 1. By fitting a decision tree to the artist names and intention categories (including people and not including people) in the prompt, we can obtain the feature_importance of each artist, that is, the degree of influence of the artist on painting people.

[0179] 2. Count the frequency of artists drawing figures: Number of times figures were drawn / Number of times the artist appeared;

[0180] 3. Filter artists who appear less than 30 times in the statistics (too low a frequency is not statistically significant);

[0181] 4. When an artist's feature_importance is greater than the mean and the frequency of drawing people is higher than 50%, that is, when it is determined that the prompt contains the artist, an image with people will be output.

[0182] 5. Adding the artist names obtained in step 4 to the list will give you a list of artists who tend to paint figures.

[0183] In one embodiment, the generation parameters of the text-to-text generation model are tuned during model training based on the model's accuracy performance. Specifically, the generation parameters of the model are set as follows:

[0184] i.do_sample:True (This increases the diversity of generated results);

[0185] ii.early_stopping: True (Only generate results within the range with the highest probability);

[0186] iii.length_penalty: 1.0 (The model will be more inclined to generate shorter and more accurate results);

[0187] Furthermore, by combining the subject's intent classification, the length of the model's generated data is controlled:

[0188] i. When the intent is ["human","portrait","comics animation movie characters"], the minimum complement length is set to 70 (Stable Diffusion's text encoder can accept a maximum of 77 tokens), which can supplement more portrait type artists and embellishments;

[0189] ii. When the intent is ["food","fruit","electronic product","product","tools"], the minimum complement length is set to 30. The model outputs fewer and more accurate modifiers, which can make stable diffusion avoid people and rarely add artists.

[0190] iii. Other intentions, such as "city", etc., with the minimum complement length set to 50, and balanced complement modifiers and artists.

[0191] Therefore, after obtaining the user's input text and distinguishing the main intent of the input text, parameters can be generated directly according to the model settings to supplement the input text with prompts. That is, the user's input text (such as "An apple") is fed into the trained T5 model, and the model supplements the input text to complete the prompt.

[0192] For generation unit 203: if the main intent is not to include figures, it removes artist names with a tendency to paint figures from the prompt text based on a pre-generated artist list to obtain an optimized input prompt; wherein the artist list includes multiple interfering artist names with a tendency to paint figures. If the main intent includes figures, the prompt text is directly used as the input prompt. Therefore, based on the main intent category, if the user input is determined not to include figures, the prompt result generated by the model is further post-processed to filter out artists with a tendency to paint figures obtained in the artist analysis step; if the user input is determined to include figures, no processing is performed. Thus, the final prompt result is output.

[0193] Therefore, the intelligent prompt word optimization system for text-to-image generation provided in this application ensures that the supplemented input prompt words accurately express the user's intent by analyzing the painting subject intent of the user's input text; it reduces the user's trial-and-error cost and improves the efficiency and accuracy of text supplementation by pre-training an advanced text generation model; and it avoids changes to the user's intent due to the artist's style by filtering the artist's name. Overall, it achieves high efficiency, low trial-and-error cost, and accurate supplementation of the user's input prompt intent.

[0194] This application also provides an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the intelligent optimization method for generating images from text according to the embodiments of this application.

[0195] This application also provides a computer-readable medium storing a computer program thereon, which, when executed by a processor, implements the intelligent optimization method for prompt words in text-to-image generation according to this application.

[0196] Figure 4 An exemplary system architecture 300 for intelligent optimization of prompt words for text-to-image generation, which can be applied according to embodiments of this application, is shown.

[0197] like Figure 4 As shown, system architecture 300 may include terminal devices 301, 302, and 303, a network 304, and a server 305. Network 304 serves as the medium for providing communication links between terminal devices 301, 302, and 303 and server 305. Network 304 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0198] Users can use terminal devices 301, 302, and 303 to interact with server 305 via network 304 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 301, 302, and 303, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0199] Terminal devices 301, 302, and 303 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0200] Server 305 can be a server that provides various services, such as a backend management server that supports messages sent by users using terminal devices 301, 302, and 303. The backend management server can analyze and process requests received from terminal devices and then return the processing results to the terminal devices.

[0201] It should be noted that the intelligent optimization method for prompt words in text-to-image generation provided in this application embodiment is generally executed by terminal devices 301, 302, 303 or server 305. Correspondingly, the intelligent optimization system for prompt words in text-to-image generation is generally set in terminal devices 301, 302, 303 or server 305.

[0202] It should be understood that Figure 4 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0203] The following is for reference. Figure 5 It shows a schematic diagram of the structure of a computer system 400 suitable for implementing electronic devices according to embodiments of the present application. Figure 5 The computer system shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0204] like Figure 5 As shown, the computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 402 or programs loaded from storage section 408 into random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the system 400. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0205] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to I / O interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 410 as needed so that computer programs read from it can be installed into storage section 408 as needed.

[0206] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit (CPU) 401, it performs the functions defined above in the system of this application.

[0207] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0208] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0209] The modules described in the embodiments of this application can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including a determining module, an extraction module, a training module, and a filtering module. The names of these modules do not necessarily limit the module itself; for example, the determining module can also be described as a "module for determining a candidate user set".

[0210] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

[0211] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for intelligent optimization of prompt words for text-to-image generation, characterized in that, include: The user input text is obtained and fed into a text classification model to obtain the main intent of the input text. The main intent is used to characterize whether the user's drawing intent includes people or not. The input text is fed into a pre-trained text-to-text generation model, and the supplemented prompt text is output based on the generation parameters of the text-to-text generation model. If the intended subject is not to include figures, then the names of artists with a tendency to draw figures are removed from the prompt text according to the pre-generated list of artists, resulting in an optimized input prompt; wherein, the list of artists includes multiple artist names with a tendency to draw figures. The training method for text-to-text generation models is as follows: Obtain a training sample set, extract the main intent words of each input prompt word text in the sample set, and determine the main intent of each input prompt word text based on the main intent words; The target input prompt text is obtained to obtain pre-training sample data, wherein the target input prompt text is all input prompt texts whose sentence begins with the main intent word; the pre-training sample data is processed using undersampling technology to make the proportion of samples with main intent including people and those without people consistent, thereby obtaining training sample data; The pre-trained text-to-text generation model is obtained by using the main intent word of each input prompt word in the training sample data as input to the text-to-text generation model. The list of artists is generated as follows: The decision tree is used to fit the artist names and corresponding subject intentions in the input prompt text of the sample set to obtain the influence value of each artist name on the generation of images with human attributes; artist names that appear less than a preset value in the sample set are filtered out; the frequency of drawing human figures for each artist name is calculated based on the number of times each artist name is drawn and the number of times it appears in the sample set; artist names with an influence value greater than the mean influence value and a frequency of drawing human figures higher than the preset value are added to the list to generate the artist list.

2. The intelligent optimization method for prompt words in text-to-image generation according to claim 1, characterized in that, The step of training a pre-trained text-to-text generation model by using the main intent word of each input prompt word text in the training sample data as input to the text-to-text generation model includes: using the main intent word of each input prompt word text in the training sample data as input to the text-to-text generation model; the text-to-text generation model continues the main intent word to obtain a complete training prompt word; calculating the cross-entropy loss between the training prompt word and each word of the corresponding input prompt word text; backfeeding the cross-entropy loss to the text-to-text generation model and using an optimizer to correct the model weights; stopping training when the cross-entropy loss no longer decreases within N steps to obtain the text-to-text generation model, where N is a preset value and is a positive integer.

3. The intelligent optimization method for prompt words in text-to-image generation according to claim 1, characterized in that, The extraction of the main intent word for each input prompt text in the sample set includes: dividing each input prompt text into multiple segments using symbols as dividing points, counting the frequency of each segment, and taking the segment whose frequency meets the threshold and is the beginning of the sentence of the input prompt text as the main intent word.

4. The intelligent optimization method for prompt words in text-to-image generation according to claim 2, characterized in that, The text classification model is a zero-sample text classification model.

5. The intelligent optimization method for prompt words in text-to-image generation according to claim 1, characterized in that, If the subject's intent is to include a person, then the prompt text will be directly used as the input prompt.

6. A system for intelligently optimizing prompts for text-to-image generation, characterized in that, include: The acquisition unit is used to acquire the input text input by the user, input the input text into the text classification model to obtain the main intent of the input text, and the main intent is used to characterize whether the user's drawing intent includes people or not. The processing unit is used to input the input text into a pre-trained text-to-text generation model and output supplemented prompt text based on the generation parameters of the text-to-text generation model. The generation unit is used to remove artist names with a tendency to draw people from the prompt text according to a pre-generated artist list if the subject's intention is not to include people, thereby obtaining an optimized input prompt; wherein, the artist list includes multiple interfering artist names with a tendency to draw people. The training method for text-to-text generation models is as follows: Obtain a training sample set, extract the main intent words of each input prompt word text in the sample set, and determine the main intent of each input prompt word text based on the main intent words; The target input prompt text is obtained to obtain pre-training sample data, wherein the target input prompt text is all input prompt texts whose sentence begins with the main intent word; the pre-training sample data is processed using undersampling technology to make the proportion of samples with main intent including people and those without people consistent, thereby obtaining training sample data; The pre-trained text-to-text generation model is obtained by using the main intent word of each input prompt word in the training sample data as input to the text-to-text generation model. The list of artists is generated as follows: The decision tree is used to fit the artist names and corresponding subject intentions in the input prompt text of the sample set to obtain the influence value of each artist name on the generation of images with human attributes; artist names that appear less than a preset value in the sample set are filtered out; the frequency of drawing human figures for each artist name is calculated based on the number of times each artist name is drawn and the number of times it appears in the sample set; artist names with an influence value greater than the mean influence value and a frequency of drawing human figures higher than the preset value are added to the list to generate the artist list.

Images