Image generation method and device, computer device, storage medium and program product

Abstract

The present application relates to the technical field of intelligent model, and discloses an image generation method and device, computer equipment, storage medium and program product, the present application determines similar preset prompt words, potential classification list of to-be-generated images and confusion category pairs based on original prompt words, so as to utilize the confusion category pairs for classification correction, avoid misjudging the theme category of the to-be-generated images, and improve the accuracy of the theme category. If the theme category is a person theme, the original prompt word is optimized through the scene type of the to-be-generated images to obtain a target prompt word, so that the image generation model generates images by utilizing the target prompt word rich in details and the scene type. When the image generation model has not ended iteration, the intermediate image generated by the image generation model is subjected to face repair, so as to avoid image repair after the iteration ends, thereby prolonging the generation time, improving the image generation efficiency, avoiding the phenomenon of face collapse, and improving the image generation quality.

Description

Technical Field

[0001] This invention relates to the field of intelligent model technology, specifically to image generation methods, apparatus, computer equipment, storage media, and program products. Background Technology

[0002] With the development of artificial intelligence technology, intelligent model-generated images have been widely used in most scenarios. Currently, mainstream image generation models (such as diffusion models) mostly adopt an end-to-end architecture, where the model directly outputs the generated image based on the text prompts input by the user.

[0003] In the image generation process, a Large Language Model (LLM) is needed to convert user input into target prompts. However, LLM is prone to misjudging the type of prompts, resulting in images generated based on the target prompts that do not correspond to the type of image requested by the user. Furthermore, when generating images related to people, mainstream diffusion models struggle to restore details such as eyebrows and pupils during the decoding process, easily leading to image quality issues such as face distortion, which affects the visual effect of the final image. Summary of the Invention

[0004] In view of this, the present invention provides an image generation method, apparatus, computer device, storage medium, and program product to solve the problems of the prior art generating images that do not match user needs and generating images of low quality.

[0005] In a first aspect, the present invention provides an image generation method, the method comprising:

[0006] Obtain the original prompt words input by the user, which include descriptive information about the image to be generated;

[0007] Based on the original prompt words, determine the preset prompt words similar to the original prompt words, the potential classification list of the image to be generated, and the confusion category pairs, and determine the subject category of the image to be generated based on the preset prompt words, the potential classification list, and the confusion category pairs;

[0008] If the detected theme category is a people theme, the scene type of the image to be generated is determined based on the original prompt words, and the original prompt words are optimized according to the scene type to obtain the target prompt words;

[0009] The target prompt and topic category are input into the pre-trained image generation model. Before the iteration of the image generation model ends, face inpainting is performed on the intermediate image generated by the image generation model to obtain the target image. The intermediate image is the image generated by the image generation model before the iteration ends.

[0010] This application, based on the original prompt words, determines similar preset prompt words, a potential classification list of the image to be generated, and confusing category pairs. These confusing category pairs are then used for classification correction to avoid misclassifying the subject category of the image to be generated, thus improving the accuracy of subject category classification. If the subject category is a person, the original prompt words are optimized based on the scene type of the image to be generated to obtain target prompt words. This allows the image generation model to utilize the detailed target prompt words and scene type for image generation. Before the image generation model finishes its iteration, face restoration is performed on the intermediate images generated by the model. This avoids prolonged generation time due to image restoration after iteration, improving image generation efficiency and preventing face distortion, thereby improving image generation quality.

[0011] In some optional implementations, there are multiple preset prompt words, each preset prompt word is associated with a preset category, and the potential category list includes multiple potential categories and the confidence level of each potential category; based on the preset prompt words, the potential category list, and the obfuscated category pairs, the subject category of the image to be generated is determined, including:

[0012] Determine multiple candidate topic categories for the image to be generated; wherein, the multiple candidate topic categories include the preset category associated with each preset prompt word and multiple potential categories;

[0013] Determine the frequency of occurrence of preset prompts associated with the same preset category among multiple preset prompts. Based on the frequency of occurrence for each preset category and the confidence level of each potential category, obtain the evaluation score for each candidate topic category.

[0014] The topic category of the image to be generated is obtained by correcting the evaluation score based on the confusion category and the corrected evaluation score corresponding to each candidate topic category.

[0015] This application comprehensively determines multiple candidate topic categories for the image to be generated based on the retrieved preset categories and the latent categories generated by the large language model. Each candidate topic category is evaluated by the frequency of occurrence of the preset categories in the search results and the confidence level of the latent categories generated by the large language model, resulting in a corresponding evaluation score. Finally, the evaluation score is corrected by checking whether any two candidate topic categories match a confused category pair, thus avoiding excessively high evaluation scores for misclassified categories and improving the accuracy of topic classification.

[0016] In some optional implementations, the confusion category pair includes a first category and a second category that is confused with the first category; any two candidate topic categories from a plurality of candidate topic categories include the first candidate topic category and the second candidate topic category; the evaluation score is adjusted according to the confusion category pair, including:

[0017] If the first candidate topic category is found to be consistent with the first category, and the second candidate topic category is found to be consistent with the second category, then the evaluation score of the first candidate topic category is increased.

[0018] When any two candidate topic categories among multiple candidate topic categories match the confused category pair, this application corrects the evaluation score of the corresponding candidate topic category to avoid the misclassified category having an excessively high evaluation score that affects the classification result when a misclassification occurs, thereby improving the accuracy of topic classification.

[0019] In some optional implementations, based on the original prompt words, preset prompt words similar to the original prompt words, a potential classification list of the image to be generated, and obfuscated category pairs are determined, including:

[0020] Based on the original prompt words, a search is performed in the preset request database to obtain preset prompt words that are similar to the original prompt words;

[0021] Classification prompts are constructed based on the original prompts, and the classification prompts are input into the target large language model to obtain a potential classification list of the image to be generated;

[0022] Based on the original prompts and potential category list, a search is performed in the preset confusion category library to obtain confusion category pairs.

[0023] This application retrieves preset prompts similar to the original prompts from a preset request library to help predict the possible categories of the image to be generated. The classification prompts constructed based on the original prompts are input into the target large language model to cover as many potential categories as possible for the image to be generated, resulting in a list of potential categories. Furthermore, by retrieving confusion category pairs from a preset confusion category library, the classification results are corrected using these confusion category pairs.

[0024] In some optional implementations, before the iteration of the image generation model ends, face inpainting is performed on the intermediate images generated by the image generation model to obtain the target image, including:

[0025] The first face region in the intermediate image is repaired, and the initial image generated by the image generation model is acquired simultaneously; the initial image is the image generated by the image generation model after the iteration is completed.

[0026] The target image is generated based on the repaired first face region and the initial image.

[0027] The image generation model in this application iterates normally until it finishes generating a detailed initial image. Simultaneously, it acquires intermediate images generated during the iteration process. Before the iteration ends, it repairs the first face region in the intermediate image to prevent face distortion. The repaired first face region and the detailed initial image are then merged to generate the target image, thus improving both image generation efficiency and quality.

[0028] In some optional implementations, a target image is generated based on the repaired first face region and the initial image, including:

[0029] Identify the second face region in the initial image;

[0030] The repaired first face region is used to replace the second face region in the initial image to generate the target image.

[0031] This application replaces the second face region in the initial image with the repaired first face region, thereby fusing the repaired first face region with the initial image, so that the target image retains the background details in the initial image while improving the visual effect of the face region.

[0032] In some alternative implementations, the method further includes:

[0033] Determine the resolution of the image to be generated, and determine the target iteration steps when violation detection is enabled based on the resolution and subject category;

[0034] If the iterative step of the image generation model is detected to have reached the target iterative step, violation detection is performed on the intermediate image generated by the image generation model to obtain the violation detection result;

[0035] If the violation detection result indicates that the intermediate image is in violation, then the generation of the target image is stopped.

[0036] This application eliminates the need to wait for the image to be fully generated, performing violation detection during the intermediate iterations of the image generation model. Since the intermediate images generated later in the image generation model already contain subject semantic information, determining the target iteration step for enabling violation detection based on the resolution and subject category of the image to be generated allows for adaptation to image generation tasks with varying resolutions and scene complexities. Performing violation detection earlier in the target iteration step, and parallelizing it with subsequent image generation, reduces generation latency and improves the flexibility of violation detection.

[0037] In some optional implementations, violation detection is performed on the intermediate images generated by the image generation model to obtain violation detection results, including:

[0038] The pre-trained violation detection model is used to detect violations in the intermediate images generated by the image generation model to obtain the first detection result;

[0039] The intermediate image generated by the image generation model is uploaded to the violation detection application so that the violation detection application can perform violation detection on the intermediate image and obtain a second detection result;

[0040] The results of the violation are obtained based on the first test result and / or the second test result.

[0041] This application performs violation detection on intermediate images generated by the image generation model by asynchronously calling the violation detection model and the violation detection application, saving detection time. It also avoids process blocking caused by the unavailability of the violation detection application, ensuring system robustness. Parallel processing of image generation and violation detection improves the utilization of computational resources.

[0042] In some optional implementations, the scene type of the image to be generated is determined based on the original prompt words, and the original prompt words are optimized according to the scene type to obtain target prompt words, including:

[0043] Input the original prompt words into the target large language model to obtain the scene type of the image to be generated;

[0044] If the scene type is detected as a preset scene type, then facial detail optimization prompts are added to the original prompts to obtain the target prompts.

[0045] This application utilizes a target large language model to identify the semantic and logical information of the original prompt words, thereby determining the scene type of the image to be generated. If the scene type requires highlighting facial details, prompt words that optimize facial details are added to the original prompt words to obtain the target prompt words. This allows the image generation model to generate images of people based on richer prompt words, avoiding facial distortion.

[0046] In some alternative implementations, the method further includes:

[0047] Obtain candidate large language models and candidate configuration methods for these models; among them, candidate configuration methods include candidate acceleration technologies and candidate hardware resource allocation methods.

[0048] Based on candidate large language models and candidate configuration methods, construct the objective function and constraints;

[0049] Based on the objective function and constraints, a target large language model is selected from the candidate large language models, and a target configuration method for the target large language model is selected from the candidate configuration methods. The target large language model is then configured based on the target configuration method.

[0050] This application dynamically selects the target large language model and the corresponding target acceleration technology and target hardware resource configuration method through objective function and constraints, thereby reducing image generation latency, improving concurrent processing capability and reducing hardware resource cost while ensuring the correct semantic understanding of the large language model.

[0051] In some alternative implementations, the method further includes:

[0052] Based on the original prompt words, generate extended prompt words that are semantically consistent with the original prompt words;

[0053] The original prompt words, expanded prompt words, and target image are stored together in association;

[0054] If the similarity between the new prompt word entered by the user and the original prompt word or the expanded prompt word exceeds the similarity threshold, the target image is output.

[0055] This application associates and stores the original prompt words, extended prompt words, and corresponding target images, enabling the rapid return of the target image when the user inputs similar new prompt words. This reduces the number of calls to the image generation model and the consumption of computational resources, lowering the cost of image generation. Furthermore, by caching the results and quickly returning the target image, user waiting is avoided, improving user satisfaction.

[0056] In a second aspect, the present invention provides an image generation apparatus, the apparatus comprising:

[0057] The acquisition module is used to acquire the original prompt words input by the user. The original prompt words include descriptive information of the image to be generated.

[0058] The first processing module is used to determine, based on the original prompt words, preset prompt words similar to the original prompt words, a potential classification list of the image to be generated, and confusion category pairs, and to determine the subject category of the image to be generated based on the preset prompt words, the potential classification list, and the confusion category pairs;

[0059] The second processing module is used to determine the scene type of the image to be generated based on the original prompt words if the detected theme category is a person theme, and optimize the original prompt words according to the scene type to obtain the target prompt words;

[0060] The third processing module is used to input the target prompt words and topic categories into the pre-trained image generation model, and to perform face restoration on the intermediate images generated by the image generation model before the iteration of the image generation model ends, so as to obtain the target image; the intermediate images are the images generated by the image generation model before the iteration ends.

[0061] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the image generation method described in the first aspect or any corresponding embodiment thereof.

[0062] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the image generation method described in the first aspect or any corresponding embodiment thereof.

[0063] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the image generation method described in the first aspect or any corresponding embodiment thereof.

[0064] The beneficial effects of this application are as follows:

[0065] This application maximizes resource utilization by configuring a large language model for inference, selecting an appropriate large language model, acceleration technology, and hardware resource allocation method. It dynamically selects a lightweight large language model and employs similarity prompt word retrieval, latent classification generation, and obfuscated category pair retrieval to achieve accurate classification of the image to be generated and optimize prompt words.

[0066] This application addresses the potential face collapse problem during image generation. First, it utilizes a large language model to parse the user-inputted prompts, accurately determining the subject category of the image to be generated, and then optimizing the original prompts. During the iterative process of the image generation model, face restoration is performed on intermediate images beforehand, and the restored faces are then integrated into the initial image after the image generation model iteration, resulting in a high-quality target image.

[0067] In the iterative process of the image generation model, this application performs violation detection on the content of intermediate images to ensure that the content of the generated target image is compliant.

[0068] This application caches the generated target image and its corresponding prompt words so that when the user enters a similar new prompt word, the corresponding image can be quickly returned. Attached Figure Description

[0069] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0070] Figure 1 This is a schematic flowchart of an image generation method according to an embodiment of the present invention;

[0071] Figure 2 This is a flowchart illustrating another image generation method according to an embodiment of the present invention;

[0072] Figure 3 This is a flowchart illustrating a method for determining a topic category according to an embodiment of the present invention;

[0073] Figure 4 This is a schematic diagram of a face restoration process according to an embodiment of the present invention;

[0074] Figure 5A This is a schematic diagram of an initial image showing a collapsed face according to an embodiment of the present invention;

[0075] Figure 5B This is a schematic diagram of a target image after face restoration according to an embodiment of the present invention;

[0076] Figure 6 This is a schematic diagram of a violation detection process according to an embodiment of the present invention;

[0077] Figure 7 This is a schematic diagram of a process for extracting a target image based on a cache according to an embodiment of the present invention;

[0078] Figure 8 This is a flowchart illustrating another image generation method according to an embodiment of the present invention;

[0079] Figure 9 This is a structural block diagram of an image generation apparatus according to an embodiment of the present invention;

[0080] Figure 10 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0081] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0082] According to an embodiment of the present invention, an image generation method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0083] This embodiment provides an image generation method. Figure 1 This is a flowchart of an image generation method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0084] Step S101: Obtain the original prompt words input by the user.

[0085] Specifically, the system can obtain the original prompts entered by the user through the corresponding input interface. These original prompts include descriptive information about the image to be generated, such as the scene or people in the image.

[0086] Step S102: Based on the original prompt words, determine the preset prompt words similar to the original prompt words, the potential classification list of the image to be generated, and the confusion category pairs, and determine the subject category of the image to be generated based on the preset prompt words, the potential classification list, and the confusion category pairs.

[0087] Specifically, multiple basic categories can be pre-defined, and theme categories can be selected from these categories. These basic categories include, but are not limited to: people themes, landscape themes, popular IP themes, car themes, and other themes, which can be set according to actual needs. Specifically, people themes mainly involve human figures, portraits, and characters; landscape themes describe natural or urban landscapes; popular IP themes involve well-known film, game, and anime characters or elements; car themes involve vehicle-related descriptions; and other themes are used to cover topics that cannot be categorized into the above categories.

[0088] In step S102, preset prompts similar to the original prompts are retrieved to predict the possible categories of the image to be generated based on the preset categories associated with the preset prompts. A potential classification list for the image to be generated is determined to cover as many classification possibilities as possible and avoid omissions. Since LLM is prone to misclassification when determining the type of prompts, this embodiment retrieves confused category pairs related to the original prompts to comprehensively determine the subject category of the image to be generated using the preset prompts, the potential classification list, and the confused category pairs, thereby improving classification accuracy.

[0089] Step S103: If the detected theme category is a person theme, determine the scene type of the image to be generated based on the original prompt words, optimize the original prompt words according to the scene type, and obtain the target prompt words.

[0090] Specifically, it determines whether the subject category of the image to be generated is a person subject as defined in the basic category. If the subject category is a person subject, it means that the image generation model needs to generate a face-related image. Because existing text-to-image generation techniques struggle to reproduce details such as eyebrows and pupils, face distortion is common. To address this issue, this embodiment first optimizes the prompts input into the image generation model.

[0091] In step S103, the scene type of the image to be generated is determined so that the original prompt words can be dynamically optimized according to the scene category, thereby enriching the details of the prompt words. This enables the image generation model to generate images using the richly detailed target prompt words, avoiding the omission of semantic information.

[0092] Step S104: Input the target prompt word and topic category into the pre-trained image generation model, and before the iteration of the image generation model ends, perform face restoration on the intermediate image generated by the image generation model to obtain the target image; the intermediate image is the image generated by the image generation model before the iteration ends.

[0093] Specifically, detailed target prompts and the topic category of the image to be generated are input into a pre-trained image generation model. Before the image generation model finishes its iterations, face inpainting is performed on the intermediate images generated by the model. This avoids prolonged generation time caused by image inpainting after the iteration ends, improving image generation efficiency and preventing face distortion, thus enhancing image generation quality. The image generation model can employ a diffusion model. The training process of the image generation model can be described in detail in relevant technical descriptions, and will not be elaborated upon here.

[0094] In some embodiments, if the subject category of the image to be generated is not a person, the normal generation process is entered, the original prompt words and subject category are directly input into the pre-trained image generation model, and the face restoration is performed on the image generated by the image generation model to obtain the target image.

[0095] Traditional diffusion models convert user-input text into latent vectors and embed spatial image features with random noise into these vectors. They then use a variational autoencoder (VAE) to reconstruct the image from the denoised latent vectors. However, this process is prone to misclassification of subject categories and can result in facial distortion, making it difficult to guarantee the quality of the generated image.

[0096] The image generation method provided in this embodiment, based on the original prompt words, determines similar preset prompt words, a potential classification list of the image to be generated, and confusing category pairs. This allows for classification correction using the confusing category pairs, avoiding misjudgment of the subject category of the image to be generated and improving the accuracy of subject category classification. If the subject category is a person, the original prompt words are optimized based on the scene type of the image to be generated to obtain target prompt words. This enables the image generation model to generate images using detailed target prompt words and scene types. Before the image generation model finishes its iteration, face restoration is performed on the generated images to improve image generation efficiency and avoid face collapse, thus improving image generation quality.

[0097] This embodiment provides an image generation method. Figure 2 This is a flowchart of an image generation method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0098] Step S201: Obtain the original prompt words input by the user. The original prompt words include descriptive information about the image to be generated. See details below. Figure 1 The detailed description of step S201 in the illustrated embodiment will not be repeated here.

[0099] Step S202: Based on the original prompt words, determine the preset prompt words similar to the original prompt words, the potential classification list of the image to be generated, and the confusion category pairs, and determine the subject category of the image to be generated based on the preset prompt words, the potential classification list, and the confusion category pairs.

[0100] Specifically, step S202 includes:

[0101] Step S2021: Based on the original prompt words, determine the preset prompt words similar to the original prompt words, the potential classification list of the image to be generated, and the confused category pairs.

[0102] In some optional implementations, step S2021 above includes:

[0103] Step a1: Based on the original prompt words, search the preset request database to obtain preset prompt words that are similar to the original prompt words.

[0104] Specifically, test data and user history requests are collected to construct preset prompt words, and each preset prompt word is labeled with a corresponding preset category. The preset categories can be selected from several pre-defined basic categories (such as people themes, scenery themes, popular IP themes, car themes, and other themes). This creates a preset request library with preset category labels. Each data entry in the preset request library contains a preset prompt word, the preset category associated with the preset prompt word, and the semantic vector corresponding to the preset prompt word.

[0105] In step a1, as Figure 3 As shown, a vectorization model such as the BGE (BAAI General Embedding) model can be used to convert the original prompt words input by the user into semantic vectors, thereby vectorizing the user input. Then, multiple (e.g., 5-10) preset prompt words with the highest similarity to the original prompt words are retrieved from a preset request library, and the preset category associated with each preset prompt word is determined, so as to use the preset category for auxiliary classification.

[0106] For example, the original prompt is "generate cyberpunk sports car," and its corresponding semantic vector is [0.3, -0.1, ..., 0.8]. Based on this semantic vector, a search is performed in a preset request library. The resulting preset prompts with the highest semantic vector similarity and the preset categories associated with each preset prompt are: "Design a futuristic Tesla concept car (car theme)," "Generate a sci-fi vehicle under neon lights (car theme)," and "Create a cyberpunk city street scene (landscape theme)." Thus, by analyzing the category distribution of the preset prompts, the possible category of the image to be generated can be predicted. For example, if most of the search results are "car theme," then it is highly likely that a car theme image will be generated.

[0107] Step a2: Construct classification prompts based on the original prompts, input the classification prompts into the target large language model, and obtain the potential classification list of the image to be generated.

[0108] In step a2, see again Figure 3 A target large language model is constructed to classify information based on the original prompt words. These classification prompt words are then input into the target large language model. By adjusting the temperature parameter of the target large language model, diverse classification hypotheses are generated, resulting in a potential classification list. This potential classification list includes multiple potential classifications and the confidence level of each potential classification.

[0109] It should be noted that the large language model is an intelligent system capable of understanding user descriptions and generating text that conforms to logic and context. The temperature parameter is used to control the randomness of the text generated by the large language model. By adjusting the temperature parameter, the potential classifications of the image to be generated can be covered as much as possible. For details, please refer to the detailed description of the relevant technologies, which will not be repeated here.

[0110] For example, the classification prompt could be "Please determine the potential category of the image to be generated based on user input (candidate categories: people, landscape, popular IP, car, other), and output them in order of confidence." For instance, if the user input is "generate a cyberpunk sports car" and the temperature parameter is 0.7, the generated potential category list would be "car (confidence 0.92), landscape (confidence 0.65), popular IP (confidence 0.3), car (confidence 0.88), other (confidence 0.1)". Further, the potential category list is post-processed by merging duplicate potential categories and taking the highest confidence level. For example, "car (confidence 0.92)" and "car (confidence 0.88)" are merged into "car (confidence 0.92)".

[0111] Step a3: Based on the original prompt words and the potential category list, search the preset confusion category library to obtain confusion category pairs.

[0112] Specifically, a pre-built confusion category library is constructed in advance. This library includes multiple confusion category pairs. Each confusion category pair includes a first category and a second category that is easily confused with the first category; that is, the model is likely to misclassify the first category as the second category. For example, some confusion category pairs can be shown in Table 1:

[0113] Table 1

[0114]

[0115] In step a3, see again Figure 3 The system transforms the user-inputted prompts and potential categories into BGE vectors. Based on these BGE vectors, it retrieves multiple related obfuscation category pairs from a pre-defined obfuscation category library. For example, if the original prompt is "generate cyberpunk sports car," its potential categories include car theme, landscape theme, popular IP theme, and other themes. Based on the transformed BGE vectors, the obfuscation category pairs retrieved from the pre-defined obfuscation category library would be "cyberpunk vehicles (car theme) misclassified as landscape theme" and "futuristic car design (car theme) obfuscated as popular IP theme."

[0116] This embodiment retrieves preset prompts similar to the original prompts from a preset request library to help predict the possible categories of the image to be generated. The classification prompts constructed based on the original prompts are input into the target large language model to cover as many potential categories as possible for the image to be generated, resulting in a potential category list. Furthermore, confusion category pairs are retrieved from a preset confusion category library to correct the classification results using these pairs.

[0117] Step S2022: Determine multiple candidate topic categories for the image to be generated; wherein, the multiple candidate topic categories include the preset category associated with each preset prompt word and multiple potential categories.

[0118] Specifically, by statistically analyzing the preset categories associated with each preset prompt and the potential categories in the potential category list, multiple candidate theme categories for the image to be generated are obtained. Using the example above, the original prompt is "Generate a cyberpunk sports car," and its corresponding preset prompts and their associated preset categories are: "Design a futuristic Tesla concept car (car theme)," "Generate a sci-fi vehicle under neon lights (car theme)," and "Create a cyberpunk city street scene (landscape theme)." The potential category list is: "Car theme (confidence 0.92), Landscape theme (confidence 0.65), Popular IP theme (confidence 0.3), Other themes (confidence 0.1)." Therefore, the multiple candidate theme categories for the image to be generated include car theme, landscape theme, popular IP theme, and other themes.

[0119] Step S2023: Determine the number of times preset prompts associated with the same preset category appear in multiple preset prompts. Based on the number of occurrences corresponding to each preset category and the confidence level of each potential category, obtain the evaluation score of each candidate topic category.

[0120] Specifically, see again Figure 3 The algorithm counts the occurrence counts of predefined suggestion words associated with the same predefined category across multiple predefined suggestion words. Specifically, it counts the occurrence counts of each predefined category across multiple retrieved predefined categories. A first score is obtained based on the occurrence count of each predefined category, and a second score is obtained based on the confidence level of the potential classification. For each candidate topic category, a weighted sum of the first and second scores is calculated to obtain the evaluation score for that candidate topic category, thus evaluating each candidate topic category. The scoring rules are shown in Table 2 below.

[0121] Table 2

[0122]

[0123] Using the above examples as an illustration, the first score for the car theme is 2×2=4 points, and the second score is 5 points; the first score for the scenery theme is 1×2=2 points, and the second score is 3 points; the first score for the popular IP theme is 0×2=0 points, and the second score is 1 point; and the first score for other themes is 0×2=0 points, and the second score is 0 points. Furthermore, for each candidate theme category, the first and second scores of that candidate theme category are weighted and calculated to obtain the evaluation score for the car theme. The assessment score for the landscape theme is The evaluation score for popular IP themes is The assessment scores for other topics are .in, This indicates the weight of the first score. This indicates the weight of the second score. This increases the authority of preset prompts that are similar to the original prompts and reduces the randomness of potential categories.

[0124] Step S2024: Correct the evaluation score according to the confusion category, and obtain the topic category of the image to be generated according to the corrected evaluation score corresponding to each candidate topic category.

[0125] Specifically, any two candidate topic categories among multiple candidate topic categories are represented as the first candidate topic category and the second candidate topic category. If the first candidate topic category is found to be consistent with the first category, and the second candidate topic category is found to be consistent with the second category, then the evaluation score of the first candidate topic category is increased.

[0126] Using the above example as an illustration, the original prompt is "generate cyberpunk sports car". The retrieved confusion category pairs are "cyberpunk vehicles (car theme) misjudged as landscape theme" and "future car design (car theme) confused as popular IP theme". Therefore, the evaluation score of the car theme needs to be increased.

[0127] For example, for each confused category pair that supports the first category and is easily confused with the second category, a corrected score for the first category is obtained based on the number of entries that support the first category. This corrected score is then used to adjust the evaluation score of the first category. In the example above, if two confused category pairs supporting the topic of automobiles are found, then based on... Obtain the corrected score for the car-related topic, and sum this corrected score with the original evaluation score for the car-related topic to obtain the corrected evaluation score. .in, This indicates the weight corresponding to the corrected score. .

[0128] In step S2024, the candidate topic category with the highest corrected evaluation score is determined as the topic category of the image to be generated. In the example above, the final topic category of the image to be generated is the car topic.

[0129] In some embodiments, see again Figure 3 Users can correct errors in topic categories and generate new obfuscation category pairs, storing them in a preset obfuscation category library for continuous updating. They can also use these new obfuscation category pairs to fine-tune the BGE model, improving the distinction between the first and second categories.

[0130] In this embodiment, when any two candidate topic categories among multiple candidate topic categories match the confused category pair, the evaluation score of the corresponding candidate topic category is corrected. This avoids the misclassified category having an excessively high evaluation score, which would affect the classification result, and thus improves the accuracy of topic classification.

[0131] The above embodiments comprehensively determine multiple candidate topic categories for the image to be generated based on the retrieved preset categories and the latent classifications generated by the large language model. Each candidate topic category is evaluated by the frequency of occurrence of the preset category in the search results and the confidence level of the latent classifications generated by the large language model, resulting in a corresponding evaluation score. Finally, the evaluation score is corrected by checking whether any two candidate topic categories match a confused category pair, thus avoiding excessively high evaluation scores for misclassified categories and improving the accuracy of topic classification.

[0132] This application comprehensively determines the subject category of the image to be generated by using preset categories, potential categories, and confusion categories, significantly reducing the misclassification rate in easily confused scenes. Furthermore, it dynamically updates the preset confusion category library and fine-tunes the BGE model based on user feedback, forming a self-optimizing closed-loop system. Moreover, the preset request library does not require large-scale labeled data, effectively reducing computational and human resource costs.

[0133] Step S203: If the detected theme category is a person theme, determine the scene type of the image to be generated based on the original prompt words, optimize the original prompt words according to the scene type, and obtain the target prompt words.

[0134] Specifically, step S203 includes:

[0135] Step S2031: If the detected topic category is a person topic, input the original prompt words into the target large language model to obtain the scene type of the image to be generated.

[0136] Specifically, the target large language model is used to perform semantic understanding of the original prompt words to determine whether the image to be generated is a multi-person scene or a long-distance scene, thus obtaining the scene type of the image to be generated.

[0137] Step S2032: If the scene type is detected as a preset scene type, then add face detail optimization prompts to the original prompts to obtain the target prompts.

[0138] Specifically, the preset scene types are those that require highlighting facial details. These preset scene types include, but are not limited to, non-long-distance scenes in multi-person scenarios, long-distance scenes in single-person scenarios, and non-long-distance scenes in single-person scenarios. The specific settings can be configured according to actual needs.

[0139] For example, if the image to be generated is a non-long-distance scene in a multi-person scenario, or a long-distance scene in a single-person scenario, and conforms to the preset scene type, then facial detail optimization prompts are forcibly added to the original prompts to obtain the target prompts. Furthermore, if the user does not specify the camera distance in the original prompts, causing the target large language model to be unable to determine whether the image to be generated is a long-distance scene, then facial detail optimization prompts are forcibly added to the original prompts by default to obtain the target prompts. The facial detail optimization prompts are prompts that optimize facial details, such as "medium-close range image, highlighting facial details" or "medium-close range image, high-definition face," etc., and can be specifically set according to actual needs.

[0140] For example, if the image to be generated is a distant scene in a multi-person scenario and does not conform to the preset scene type, the original prompt words are not added, and the original prompt words are directly used as the target prompt words of the input image generation model to avoid face restoration distortion.

[0141] This embodiment utilizes a target large language model to identify the semantic and logical information of the original prompt words, thereby determining the scene type of the image to be generated. If the scene type requires highlighting facial details, prompt words that optimize facial details are added to the original prompt words to obtain the target prompt words. This allows the image generation model to generate images of people based on richer prompt words, avoiding facial distortion.

[0142] Step S204: Input the target prompt word and topic category into the pre-trained image generation model, and before the iteration of the image generation model ends, perform face restoration on the intermediate image generated by the image generation model to obtain the target image; the intermediate image is the image generated by the image generation model before the iteration ends.

[0143] Specifically, step S204 includes:

[0144] Step S2041: Input the target prompt word and topic category into the pre-trained image generation model to repair the first face region in the intermediate image, and simultaneously acquire the initial image generated by the image generation model; the initial image is the image generated by the image generation model after the iteration.

[0145] In this embodiment, the image generation model can be a diffusion model, such as a denoising diffusion probabilistic model (DDPM). The diffusion model generates images following a Markov chain, progressively denoising and generating the image. The diffusion model requires a total of T iterations, and its performance in the early and middle stages... In each iterative step, normal denoising is performed to generate the overall image structure. At this point, the generated intermediate image already contains the main semantic information. In the final step... Each iterative step adds details to the image, such as textures and small objects.

[0146] Specifically, such as Figure 4 As shown, the image generation model iterates on the main thread, and when the image generation model enters a critical iteration node (the... When the image generation model generates an intermediate image in this step, it copies the image and starts an asynchronous repair thread to perform face restoration on the intermediate image. Meanwhile, the image generation model continues iterating on the main thread, executing the final step. This process involves several iterations, after which the initial image generated by the image generation model is obtained. For example, T is 50. It is 5.

[0147] Specifically, see again Figure 4 The trained face recognition model can be used to locate the first face region in an intermediate image. For the detected first face region, a pre-trained face restoration model is used to restore it, resulting in a restored first face region that is then saved. It should be noted that the face recognition model can be a lightweight YOLO (You Only Look Once) model, and the face restoration model can be a Generative Facial Prior-Guided GAN (GFPGAN). For detailed training processes of YOLO and GFPGAN models, please refer to the relevant technical descriptions; they will not be elaborated upon here.

[0148] Step S2042: Generate the target image based on the repaired first face region and the initial image.

[0149] See you again Figure 4 After the iteration is complete, the main thread generates the initial image. At this time, it determines whether the asynchronous repair is complete. If it is complete, it identifies the second face region in the initial image, replaces the second face region in the initial image with the repaired first face region, generates the target image, and then merges the repaired first face region into the initial image to obtain the target image and output it. If it is not complete, the main thread waits. If the wait times out, it skips the repair process and directly outputs the initial image.

[0150] In some embodiments, when replacing the second face region in the initial image with the repaired first face region, Poisson fusion technology can be used to seamlessly paste the repaired first face region back, thereby preserving the rich background details of the initial image, ensuring background consistency, and improving the visual effect of the face region.

[0151] For example, Figure 5A The image is the initial image generated by the image generation model without face restoration, which contains facial distortion. Figure 5B By comparing the target images generated by the image generation model after face restoration, it is clear that the visual effect of the target images obtained after face restoration is significantly better than that of the initial images without face restoration.

[0152] In this embodiment, the image generation model iterates normally until it finishes, generating a detailed initial image. Simultaneously, intermediate images generated during the iteration process are acquired. Before the iteration ends, the first face region in the intermediate image is repaired to prevent face distortion. The repaired first face region and the detailed initial image are then merged to generate the target image, improving both image generation efficiency and quality.

[0153] In some alternative implementations, the following steps are performed during the iteration of the image generation model:

[0154] Step b1: Determine the resolution of the image to be generated, and determine the target iteration steps when violation detection is enabled based on the resolution and subject category.

[0155] Specifically, intermediate images corresponding to different iteration steps can be generated in advance using an image generation model on multiple violation datasets, and the t iteration steps that can identify violations can be manually labeled. The violation datasets can be Common Objects in Context (COCO) datasets, custom violation image libraries, etc., and this application is not limited to these.

[0156] Further, the structural similarity index (SSIM) and the intersection-over-union (IoU) ratio of the illegal regions between the intermediate images and the final generated initial image are calculated after t iterations. From the t iterations, the smallest iteration step in which the structural similarity index exceeds the structural similarity index threshold (e.g., SSIM > 0.7) and the illegal region IoU ratio exceeds the IoU threshold (e.g., IoU > 0.6) is determined and saved.

[0157] In this embodiment, a pre-stored minimum iteration step for enabling violation detection is obtained, and the minimum iteration step is dynamically adjusted based on the resolution and subject category of the image to be generated to obtain the target iteration step. For example, high-resolution images (such as 1024x1024) need to be detected earlier, and violation detection should be performed in the last 15 steps, while low-resolution images (256x256) can start violation detection in the last 5 steps; images of people need to be detected earlier (e.g., violation detection in the last 10 steps), while violation detection of images of landscapes can be delayed (e.g., violation detection in the last 5 steps).

[0158] Step b2: If the iterative step of the image generation model is detected to have reached the target iterative step, violation detection is performed on the intermediate image generated by the image generation model to obtain the violation detection result.

[0159] Specifically, a pre-trained violation detection model is used to detect violations in the intermediate image generated by the image generation model, resulting in a first detection result. The intermediate image generated by the image generation model is then uploaded to a violation detection application, which performs violation detection on the intermediate image, resulting in a second detection result. The final violation detection result is obtained based on the first and / or second detection results.

[0160] In this embodiment, a violation detection model can be built based on the YOLO model. During the training process of the violation detection model, intermediate image samples that are "missed in the intermediate image but violate the rules in the final image" are mixed into the intermediate image dataset, forcing the violation detection model to learn blurred features. Furthermore, Gaussian noise matching the noise distribution of the intermediate images in the diffusion model is added to the intermediate images to simulate image noise. Gaussian blur, pixelation (scaling to 25% and then enlarging), and random occlusion (occluding 50% of the intermediate image area to simulate an incomplete region) can be applied to the intermediate images to specifically train the violation detection model. In addition, a very low probability of missing violation images is allowed, and the generation log is recorded, with periodic sampling for manual review.

[0161] In some embodiments, during the iteration of the image generation model, asynchronous calls and timeout control mechanisms are employed to achieve parallel processing of violation detection and image generation, thereby reducing end-to-end latency. For example... Figure 6As shown, violation detection is performed using both a violation detection model and an application-level violation detection mechanism (API). A thread pool is created, with the main thread executing the violation detection tasks based on the violation detection model, while API detection tasks are submitted to a separate thread to avoid blocking the main thread. After asynchronously submitting an API detection request, the main thread immediately continues executing the violation detection model's operations without waiting for the API to return results.

[0162] Specifically, see again Figure 6 After the image generation model reaches the target iteration step and the API detection task is submitted, the main thread synchronously executes the violation detection model to detect intermediate images, making full use of CPU / GPU resources and reducing idle waiting time. In the iteration steps near the target iteration step, the violation detection model continuously captures a preset number of neighboring intermediate images (e.g., 3-5 frames) and performs violation detection on these neighboring intermediate images. If the number of neighboring intermediate images with violations exceeds the preset number (e.g., 2 out of 3 frames have violations), then a violation is identified, and the first detection result is obtained.

[0163] Furthermore, after the main thread completes the violation detection model's detection, it attempts to wait for the API's second detection result, but sets the maximum waiting time to the remaining generation time of the image to be generated (i.e., it stops waiting after the image generation model iteration ends). If the API returns a result within the maximum waiting time, the first and second detection results are integrated to obtain the violation detection result; if no result is returned within the maximum waiting time (e.g., due to slow API response or network latency), the process degrades to relying solely on the second detection result to avoid blocking the image generation process.

[0164] Related technologies typically perform image security checks on the generated images after the image generation process is complete. This not only increases the system's processing time but also makes it difficult to make quick and effective adaptive adjustments when faced with diverse detection needs and scenarios.

[0165] This embodiment eliminates the need to wait for the image to be fully generated, performing violation detection during the intermediate iteration stage of the image generation model. Since the intermediate images generated later in the image generation model already contain subject semantic information, determining the target iteration step for enabling violation detection based on the resolution and subject category of the image to be generated allows for adaptation to image generation tasks with varying resolutions and scene complexities. Performing violation detection earlier in the target iteration step, and parallelizing it with subsequent image generation, reduces generation latency and improves the flexibility of violation detection.

[0166] Step b3: If the violation detection result indicates that the intermediate image is in violation, then stop generating the target image.

[0167] Specifically, see again Figure 6 If the first detection result and / or the second detection result indicate that the intermediate image is in violation, then stop generating the target image; otherwise, output the target image directly.

[0168] This embodiment performs violation detection on intermediate images generated by the image generation model by asynchronously calling the violation detection model and the violation detection application, saving detection time. It also avoids process blocking caused by the unavailability of the violation detection application, ensuring system robustness. Parallel processing of image generation and violation detection improves the utilization of computing resources.

[0169] In some optional implementations, extended prompts with semantically consistent meaning are generated based on the original prompts. The original prompts, extended prompts, and target image are then stored together. If the similarity between a new prompt input by the user and the original or extended prompts exceeds a similarity threshold, the target image is output.

[0170] Specifically, using a target large language model, the original prompt words input by the user are expanded into multiple semantically similar extended prompt words, ensuring that the extended prompt words are semantically consistent with the original prompt words but have diverse expressions. For example, the target large language model is instructed to "generate 5 semantically similar extended prompt words based on the original prompt words, keeping the core requirement unchanged." The user inputs the original prompt word "generate a landscape painting," and the target large language model outputs "draw a Chinese-style landscape painting," "create a landscape painting in the style of ink wash painting," and "generate a digital artwork of a traditional landscape painting."

[0171] Furthermore, the extended prompt words are converted into vectors, and the original prompt words, extended prompt words, and the storage path of the target image generated based on the original prompt words, user request frequency, etc., are stored in a preset request library. For example... Figure 7 As shown, after the user enters a new prompt word, the system quickly matches the new prompt word based on a preset request library, and quickly returns the corresponding target image based on the storage path of the associated target image.

[0172] For example, if a user inputs the new suggestion "draw a Chinese-style landscape painting", the new suggestion is converted into a semantic vector and searched in a preset request library. If the similarity between the new suggestion and the original or extended suggestion stored in the preset request library exceeds the similarity threshold (e.g., 0.9), the match is successful, and the target image associated with the original or extended suggestion is quickly returned.

[0173] In some embodiments, the image generation model is trained by generating cue word-image pairs using the original cue word, expanded cue word, and target image, thereby improving the generation quality of the image generation model.

[0174] This embodiment associates and stores the original prompt words, expanded prompt words, and corresponding target images, so that when the user inputs similar new prompt words, the target image can be quickly returned. This reduces the number of calls to the image generation model and the consumption of computational resources, thereby reducing the cost of image generation. Furthermore, by caching the results and quickly returning the target image, user waiting is avoided, improving user satisfaction.

[0175] The above embodiments all use a target large language model when identifying the potential classification list, the scene type of the image to be generated, and generating extended prompt words. In some embodiments, the following steps are performed before identifying the potential classification list, the scene type of the image to be generated, or generating extended prompt words:

[0176] Step c1: Obtain candidate large language models and candidate configuration methods for the candidate large language models. The candidate configuration methods include candidate acceleration technologies and candidate hardware resource allocation methods.

[0177] Specifically, multiple candidate large language models are obtained, such as Llama and DeepSeek, each with different parameter counts, computational complexity, and GPU memory requirements. Applicable candidate acceleration techniques, such as distillation and quantization, are identified, as these can significantly reduce computation time and GPU memory usage. Furthermore, candidate hardware resource allocation methods, such as GPU resource allocation methods, are determined to balance the needs of different tasks.

[0178] Step c2: Based on the candidate large language model and candidate configuration method, construct the objective function and constraints.

[0179] Specifically, based on the objectives of minimizing inference latency (ensuring the system response time is as short as possible), maximizing throughput (increasing the number of tasks processed per unit time), and minimizing cost (reducing hardware and operating costs as much as possible while ensuring performance), the objective function is constructed as follows:

[0180]

[0181] in, This represents the base latency of the i-th candidate large language model without any acceleration techniques. Indicating targeting Decision variables (e.g., whether to choose) (1 indicates selection, 0 indicates no selection). This represents the latency reduction brought about by the j-th candidate acceleration technology. Indicating targeting Decision variables (e.g., whether to choose) (1 indicates selection, 0 indicates no selection). express In application The resulting throughput gain This represents the throughput contribution corresponding to the k-th hardware resource. Indicating targeting The decision variables (e.g., whether to select the kth hardware resource and the corresponding allocation method, 1 indicates selection, 0 indicates no selection), express use The overall cost afterwards Indicates that the k-th hardware resource uses The unit cost after that.

[0182] It should be noted that differentiated adaptation strategies can be used for in-depth optimization and testing of different hardware resources and candidate large language models. This allows for testing the performance of candidate large language models under different hardware resource allocation methods and acceleration technologies, yielding measurement parameters such as latency, throughput (the number of requests that can be processed per unit time), and cost (e.g., runtime, electricity costs, hardware costs, power consumption). Specific testing procedures can be found in the detailed descriptions of relevant technologies and will not be elaborated upon here.

[0183] Furthermore, based on the principle that the total video memory cannot exceed the GPU's maximum video memory resources, a video memory constraint is constructed: ,in, express use The subsequent video memory, express In application Subsequent video memory, This represents the GPU's maximum video memory resources. Based on the principle that total computing power cannot exceed the GPU's maximum computing power, a computing power constraint is established: ,in, express use The subsequent computing power This indicates the GPU's maximum computing power.

[0184] Furthermore, some acceleration technologies can only be used with specific models or hardware. Therefore, when selecting a particular acceleration technology (i.e....) When doing so, it is also necessary to select a specific large language model (i.e., ) or hardware (i.e. Therefore, for acceleration technologies, large language models, or hardware that need to be used in conjunction, there are compatibility constraints between the acceleration technology and the model: To ensure that no acceleration technology is selected (i.e., However, it did not select the large language model that needs to be used in conjunction with it (i.e. The situation is as follows: and compatibility constraints between hardware resources and acceleration technologies are established. To ensure that no acceleration technology is selected (i.e., However, the required hardware was not selected (i.e., In addition, since large language models must maintain a certain reception rate or accuracy, a reception rate constraint is constructed: ,in, express use The subsequent reception rate This represents the receiver rate threshold. Therefore, by combining the aforementioned constraints on video memory, computing power, compatibility, and receiver rate, we obtain the constraint conditions.

[0185] In some embodiments, a large language model can be used to perform semantic understanding of user input, and the large language model can be used to construct corresponding objective functions and constraints.

[0186] Step c3: Based on the objective function and constraints, select the target large language model from the candidate large language models, select the target configuration method from the candidate configuration methods, and configure the target large language model based on the target configuration method.

[0187] For example, in the topic classification stage, a large language model is needed to generate a list of potential categories. Using the test data from this application as an illustration, the original solution uses a Llama model for potential classification, resulting in high latency and large memory consumption. Optimization is performed based on the objective function and constraints. The target large language model is replaced with a distilled TinyLlama-1B for semantic understanding (reducing the number of parameters), thus lowering memory requirements. The target acceleration technique uses INT8 quantization to further reduce latency. Furthermore, constraints are used to ensure classification accuracy, reducing costs and increasing throughput.

[0188] For example, in the extended prompt generation stage, the original solution consistently uses the Llama2-70B model to generate extended prompts, which is time-consuming and fully utilizes the GPU memory of a single card. Optimization is performed based on the objective function and constraints. If the topic category complexity is low, the target large language model is switched to Llama2-7B, and FlashAttention is used for acceleration. For complex topics, Llama2-70B is still used, and dynamic batch processing is employed for acceleration. The target hardware resource allocation method is to deploy the Llama2-70B model on an NVIDIA H100 graphics card and the Llama2-7B model on an NVIDIA RTX4090 graphics card to avoid resource contention. This reduces latency and improves hardware resource utilization.

[0189] For example, in the image generation stage, the original solution adopted a single-task, single-card strategy, which resulted in high generation time and the inability to process user requests concurrently. The optimized solution merges multiple user prompt word requests and leverages the H100's Transformer engine to accelerate the generation of multiple images in batches.

[0190] Because the computing architectures of graphics cards from different manufacturers and models vary greatly, different acceleration technologies are required to maximize their performance. However, existing text-based image generation models are not optimized for different models or deployments on different hardware resources. Furthermore, image generation quality is strongly positively correlated with the number of parameters in a large language model, and existing text-based image generation models do not optimize for model adaptability, making it difficult to balance the contradictions between the number of parameters in a large language model, computing resources, and image generation quality.

[0191] This embodiment dynamically selects the target large language model and corresponding target acceleration technology and target hardware resource configuration method through objective function and constraints, thereby reducing image generation latency, improving concurrent processing capability, and reducing hardware resource costs while ensuring the correct semantic understanding of the large language model.

[0192] The image generation scheme of the present invention will be described in detail below with reference to a specific application example.

[0193] like Figure 8 As shown, after the user inputs a prompt word, the system first searches the preset request library to determine whether there is a prompt word in the preset request library that has a similarity to the user's input exceeding the similarity threshold. If such a prompt word exists, the match is successful, the cached target image is retrieved, and the image is output.

[0194] If the match fails, the pre-defined category associated with the retrieved similar prompt words is determined. Using the target large language model based on the user-input prompt words, the potential category of the image to be generated is determined, resulting in a list of potential categories. Furthermore, a search is performed in a pre-defined confusion category library to obtain confusion category pairs. Based on the confusion category pairs, the potential category list, and the pre-defined categories, the topic category of the image to be generated is determined.

[0195] If the topic category is people, the target large language model is used to optimize the user input prompts, and the optimized target prompts are then input into the image generation model. Simultaneously, during the iterative process of the image generation model, face restoration and violation detection are performed on the intermediate images. If the violation detection passes, the restored face images are fused to obtain the target image.

[0196] This application maximizes resource utilization by configuring a large language model for inference, selecting an appropriate large language model, acceleration technology, and hardware resource allocation method. It dynamically selects a lightweight large language model and employs similarity prompt word retrieval, latent classification generation, and obfuscated category pair retrieval to achieve accurate classification of the image to be generated and optimize prompt words.

[0197] This application addresses the potential face collapse problem during image generation. First, it utilizes a large language model to parse the user-inputted prompts, accurately determining the subject category of the image to be generated, and then optimizing the original prompts. During the iterative process of the image generation model, face restoration is performed on intermediate images beforehand, and the restored faces are then integrated into the initial image after the image generation model iteration, resulting in a high-quality target image.

[0198] In the iterative process of the image generation model, this application performs violation detection on the content of intermediate images to ensure that the content of the generated target image is compliant.

[0199] This application caches the generated target image and its corresponding prompt words so that when the user enters a similar new prompt word, the corresponding image can be quickly returned.

[0200] This embodiment also provides an image generation apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0201] This embodiment provides an image generation device, such as... Figure 9 As shown, it includes:

[0202] The acquisition module 901 is used to acquire the original prompt words input by the user, which include descriptive information of the image to be generated;

[0203] The first processing module 902 is used to determine, based on the original prompt words, a preset prompt word similar to the original prompt word, a potential classification list of the image to be generated, and a pair of confused categories, and to determine the subject category of the image to be generated based on the preset prompt words, the potential classification list, and the pair of confused categories;

[0204] The second processing module 903 is used to determine the scene type of the image to be generated based on the original prompt words if the detected theme category is a person theme, and optimize the original prompt words according to the scene type to obtain the target prompt words;

[0205] The third processing module 904 is used to input the target prompt word and topic category into the pre-trained image generation model, and to perform face restoration on the intermediate image generated by the image generation model before the iteration of the image generation model ends, so as to obtain the target image; the intermediate image is the image generated by the image generation model before the iteration ends.

[0206] In some optional implementations, the number of preset prompt words is multiple, each preset prompt word is associated with a preset category, and the potential category list includes multiple potential categories and the confidence level of each potential category; the first processing module 902 is further configured to:

[0207] Determine multiple candidate topic categories for the image to be generated; wherein, the multiple candidate topic categories include the preset category associated with each preset prompt word and multiple potential categories;

[0208] Determine the frequency of occurrence of preset prompts associated with the same preset category among multiple preset prompts. Based on the frequency of occurrence for each preset category and the confidence level of each potential category, obtain the evaluation score for each candidate topic category.

[0209] The topic category of the image to be generated is obtained by correcting the evaluation score based on the confusion category and the corrected evaluation score corresponding to each candidate topic category.

[0210] In some optional implementations, the obfuscated category pair includes a first category and a second category obfuscated with the first category, and any two candidate topic categories from a plurality of candidate topic categories include the first candidate topic category and the second candidate topic category; the first processing module 902 is further configured to:

[0211] If the first candidate topic category is found to be consistent with the first category, and the second candidate topic category is found to be consistent with the second category, then the evaluation score of the first candidate topic category is increased.

[0212] In some optional implementations, the first processing module 902 is further configured to:

[0213] Based on the original prompt words, a search is performed in the preset request database to obtain preset prompt words that are similar to the original prompt words;

[0214] Classification prompts are constructed based on the original prompts, and the classification prompts are input into the target large language model to obtain a potential classification list of the image to be generated;

[0215] Based on the original prompts and potential category list, a search is performed in the preset confusion category library to obtain confusion category pairs.

[0216] In some alternative implementations, the second processing module 903 is further configured to include:

[0217] Input the original prompt words into the target large language model to obtain the scene type of the image to be generated;

[0218] If the scene type is detected as a preset scene type, then facial detail optimization prompts are added to the original prompts to obtain the target prompts.

[0219] In some optional implementations, the third processing module 904 is further configured to:

[0220] The first face region in the intermediate image is repaired, and the initial image generated by the image generation model is acquired simultaneously; the initial image is the image generated by the image generation model after the iteration is completed.

[0221] The target image is generated based on the repaired first face region and the initial image.

[0222] In some optional implementations, the third processing module 904 is further configured to:

[0223] Identify the second face region in the initial image;

[0224] The repaired first face region is used to replace the second face region in the initial image to generate the target image.

[0225] In some optional implementations, the third processing module 904 is further configured to:

[0226] Determine the resolution of the image to be generated, and determine the target iteration steps when violation detection is enabled based on the resolution and subject category;

[0227] If the iterative step of the image generation model is detected to have reached the target iterative step, violation detection is performed on the intermediate image generated by the image generation model to obtain the violation detection result;

[0228] If the violation detection result indicates that the intermediate image is in violation, then the generation of the target image is stopped.

[0229] In some optional implementations, the third processing module 904 is further configured to:

[0230] The pre-trained violation detection model is used to detect violations in the intermediate images generated by the image generation model to obtain the first detection result;

[0231] The intermediate image generated by the image generation model is uploaded to the violation detection application so that the violation detection application can perform violation detection on the intermediate image and obtain a second detection result;

[0232] The results of the violation are obtained based on the first test result and / or the second test result.

[0233] In some alternative implementations, the device is also used for:

[0234] Obtain candidate large language models and candidate configuration methods for these models; among them, candidate configuration methods include candidate acceleration technologies and candidate hardware resource allocation methods.

[0235] Based on candidate large language models and candidate configuration methods, construct the objective function and constraints;

[0236] Based on the objective function and constraints, a target large language model is selected from the candidate large language models, and a target configuration method for the target large language model is selected from the candidate configuration methods. The target large language model is then configured based on the target configuration method.

[0237] In some alternative implementations, the device is also used for:

[0238] Based on the original prompt words, generate extended prompt words that are semantically consistent with the original prompt words;

[0239] The original prompt words, expanded prompt words, and target image are stored together in association;

[0240] If the similarity between the new prompt word entered by the user and the original prompt word or the expanded prompt word exceeds the similarity threshold, the target image is output.

[0241] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0242] In this embodiment, the image generation device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0243] This invention also provides a computer device having the above-described features. Figure 9 The image generation device shown.

[0244] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 10 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 10 Take a processor 10 as an example.

[0245] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0246] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0247] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0248] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0249] The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 10 Taking the example of a connection between China and Israel via a bus.

[0250] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the computer device, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0251] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0252] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0253] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. An image generation method characterized by, The method comprises: obtaining a user inputted original prompt word, the original prompt word comprising description information of an image to be generated; determining, based on the original prompt word, a preset prompt word similar to the original prompt word, a potential classification list of the image to be generated, and a confusion category pair, and determining, based on the preset prompt word, the potential classification list, and the confusion category pair, a theme category of the image to be generated; if it is detected that the theme category is a person theme, determining, based on the original prompt word, a scene type of the image to be generated, optimizing the original prompt word according to the scene type to obtain a target prompt word; inputting the target prompt word and the theme category into a pre-trained image generation model, and performing face restoration on an intermediate image generated by the image generation model when iteration of the image generation model has not ended to obtain a target image; the intermediate image is an image generated by the image generation model when iteration of the image generation model has not ended.

2. The image generation method according to claim 1, characterized by, The number of preset prompt words is a plurality, each preset prompt word is associated with a preset category, the potential classification list comprises a plurality of potential categories and a confidence degree of each potential category; the determination of the theme category of the image to be generated based on the preset prompt word, the potential classification list, and the confusion category pair comprises: determining a plurality of candidate theme categories of the image to be generated; wherein the plurality of candidate theme categories comprise the preset category associated with each preset prompt word and the plurality of potential categories; determining a number of occurrences of the preset prompt word associated with the same preset category in the plurality of preset prompt words, and obtaining an evaluation score of each candidate theme category according to the number of occurrences corresponding to each preset category and the confidence degree of each potential category; correcting the evaluation score according to the confusion category pair, and obtaining the theme category of the image to be generated according to the corrected evaluation score corresponding to each candidate theme category.

3. The image generation method according to claim 2, characterized by, The confusion category pair comprises a first category and a second category confused with the first category, and any two candidate theme categories in the plurality of candidate theme categories comprise a first candidate theme category and a second candidate theme category; the correction of the evaluation score according to the confusion category pair comprises: if it is detected that the first candidate theme category is consistent with the first category, and the second candidate theme category is consistent with the second category, the evaluation score of the first candidate theme category is increased.

4. The image generation method of claim 2, wherein, The determination of the preset prompt word similar to the original prompt word, the potential classification list of the image to be generated, and the confusion category pair based on the original prompt word comprises: performing retrieval on the original prompt word in a preset request library to obtain a preset prompt word similar to the original prompt word; constructing a classification prompt word based on the original prompt word, inputting the classification prompt word into a target large language model to obtain a potential classification list of the image to be generated; performing retrieval on the original prompt word and the potential classification list in a preset confusion category library to obtain a confusion category pair.

5. The image generation method of claim 1, wherein, The face restoration on the intermediate image generated by the image generation model when iteration of the image generation model has not ended to obtain the target image comprises: repair the first face region in the intermediate image, and synchronously acquire an initial image generated by the image generation model; the initial image is an image generated by the image generation model after iteration ends; generate a target image according to the repaired first face region and the initial image.

6. The image generation method of claim 5, wherein, The generating a target image according to the repaired first face region and the initial image comprises: identifying a second face region in the initial image; replacing the second face region in the initial image with the repaired first face region to generate a target image.

7. The image generation method of claim 1, wherein, The method further comprises: determining a resolution of the image to be generated, determining a target iteration step when violation detection is started based on the resolution and the theme category; if it is detected that the iteration step of the image generation model reaches the target iteration step, performing violation detection on the intermediate image generated by the image generation model to obtain a violation detection result; if the violation detection result indicates that the intermediate image is in violation, stopping the generation of the target image.

8. The image generation method of claim 7, wherein, The performing violation detection on the intermediate image generated by the image generation model to obtain a violation detection result comprises: performing violation detection on the intermediate image generated by the image generation model by using a pre-trained violation detection model to obtain a first detection result; uploading the intermediate image generated by the image generation model to a violation detection application to enable the violation detection application to perform violation detection on the intermediate image to obtain a second detection result; obtaining a violation detection result according to the first detection result and / or the second detection result.

9. The image generation method of claim 1, wherein, The determining a scene type of the image to be generated based on the original prompt word, optimizing the original prompt word according to the scene type to obtain a target prompt word comprises: inputting the original prompt word into a target large language model to obtain a scene type of the image to be generated; if it is detected that the scene type is a preset scene type, adding a face detail optimization prompt word in the original prompt word to obtain a target prompt word.

10. The image generation method according to any one of claims 1-9, characterized by, The method further comprises: acquiring a candidate large language model and a candidate configuration mode of the candidate large language model; wherein the candidate configuration mode comprises a candidate acceleration technology and a candidate hardware resource allocation mode; constructing a target function and a constraint condition based on the candidate large language model and the candidate configuration mode; selecting a target large language model from the candidate large language model and a target configuration mode of the target large language model from the candidate configuration mode according to the target function and the constraint condition, and configuring the target large language model based on the target configuration mode.

11. The image generation method of any one of claims 1-9, wherein, The method further comprises: generating an extended prompt word consistent with the semantic of the original prompt word based on the original prompt word; storing the original prompt word, the extended prompt word and the target image in association; if it is detected that the similarity between a new prompt word input by a user and the original prompt word or the extended prompt word exceeds a similarity threshold, outputting the target image.

12. An image generation apparatus characterized by comprising: The device comprises: an acquisition module configured to acquire an original prompt word input by a user, the original prompt word comprising description information of an image to be generated; The first processing module is configured to determine, based on the original prompt word, a preset prompt word similar to the original prompt word, a potential classification list of an image to be generated, and a confusion category pair, and determine, based on the preset prompt word, the potential classification list, and the confusion category pair, a subject category of the image to be generated. The second processing module is configured to, if it is detected that the subject category is a person subject, determine, based on the original prompt word, a scene type of the image to be generated, and optimize the original prompt word according to the scene type to obtain a target prompt word. The third processing module is configured to input the target prompt word and the subject category into a pre-trained image generation model, and perform face repair on an intermediate image generated by the image generation model when iteration of the image generation model is not completed to obtain a target image; the intermediate image is an image generated by the image generation model when iteration of the image generation model is not completed.

13. A computer device, comprising: The memory and the processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the image generation method in any one of claims 1 to 11. The computer readable storage medium stores computer instructions for causing a computer to perform the image generation method in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer instructions are used to cause a computer to perform the image generation method in any one of claims 1 to 11.

15. A computer program product, characterised in that, ​

Images